[PATCH v2 0/4] VIRTIO: Provision maximum MSI-X vectors for a VF

[PATCH v2 0/4] VIRTIO: Provision maximum MSI-X vectors for a VF

[Max Gurtovoy]

Hi,

In a PCI SR-IOV configuration, MSI-X vectors of the device is precious
device resource. Hence making efficient use of it based on the use case
that aligns to the VM configuration is desired for best system
performance.

For example, today's static assignment of the amount of MSI-X vectors
doesn't allow sophisticated utilization of resources.

A typical cloud provider SR-IOV use case is to create many VFs for
use by guest VMs. Each VM might have a different purpose and different
amount of resources accordingly (e.g. number of CPUs). A common driver
usage of device's MSI-X vectors is proportional to the number of CPUs in
the VM. Since the system administrator might know the amount of CPUs in
the requested VM, he can also configure the VF's MSI-X vectors amount
proportional to the number of CPUs in the VM. In this way, the
utilization of the physical hardware will be improved.

Today we have some operating systems that support provisioning MSI-X
vectors for PCI VFs.

Update the specification to have a method to change the number of MSI-X
vectors supported by a VF using the PF admin virtqueue interface. For that,
create a generic infrastructure for managing PCI resources of the managed
VF by its parent PF.

The admin virtqueue interface will be extended in the future with more and more
features that some of them already in discussions. Some of these features don't
fit to MMIO/config_space characteristics, therefore a queue is selected to address
admin commands.

Motivation for choosing admin queue:
1. It is anticipated that admin queue will be used for managing and configuring
   many different type of resources. For example,
   a. PCI PF configuring PCI VF attributes in [1].
   b. virtio device creating/destroying/configuring subfunctions discussed in [2]
   c. composing device config space of VF or SF such as mac address, number of VQs, virtio features

Mapping all of them as configuration registers to MMIO will require large MMIO space, if done for
each VF/SF. Such MMIO implementation in physical devices such as PCI PF and VF requires on-chip
resources to complete within MMIO access latencies. Such resources are very expensive.

2. Such limitation can be overcome by having smaller MMIO register set to build
   a command request response interface. However, such MMIO based command interface
   will be limited to serve single outstanding command execution. Such limitation can
   resulting in high device creation and composing time which can affect VM startup time.
   Often device can queue and service multiple commands in parallel, such command interface
   cannot use parallelism offered by the device.

3. When a command wants to DMA data from one or more physical addresses, for example in the future a
   live migration command may need to fetch device state consist of config space, tens of
   VQs state, VLAN and MAC table, per VQ partial outstanding block IO list database and more.
   Packing one or more DMA addresses over new command interface will be burden some and continue
   to suffer single outstanding command execution latencies. Such limitation is not good for time
   sensitive live migration use cases.

4. A virtio queue overcomes all the above limitations. It also supports DMA and multiple outstanding
   descriptors. Similar mechanism exist today for device specific configuration - the control VQ.

[1] This proposal
[2] https://lists.oasis-open.org/archives/virtio-comment/202108/msg00025.html

Patches (1/4) introduce the admin virtqueue concept and feature bit.
Patches (2/4)-(3/4) add the admin virtq to virtio-blk and virtio-net
devices.
Patch (4/4) introduce MSI-X control support.

---

changes from V1:

1. Addressed below comments from MST:
 - updated cover letter for admin queue motivation
 - removed VIRTIO_F_ADMIN_PCI_VIRT_MANAGER
 - simplified admin queue interface by removing 
   VIRTIO_F_ADMIN_VQ_INDIRECT_DESC/VIRTIO_F_ADMIN_VQ_IN_ORDER feature bits
 - added a subsection for VF MSI-X control capability in PCI section
 - re-designed interrupt vector management admin commands
 - added a mandatory admin command to expose admin capabilities
 - improved commit messages
 - described error code for unsupported command
 - described error code for errors on invalid VF
 - described system software requirements for supporting MSI-X configuration

2. Addressed comments from Parav
 - extended command opcode to 16-bit
 - improve commit messages

3. Added more command specific error codes

---

Max Gurtovoy (4):
  Add virtio Admin Virtqueue
  virtio-blk: add support for VIRTIO_F_ADMIN_VQ
  virtio-net: add support for VIRTIO_F_ADMIN_VQ
  Add support for MSI-X vectors configuration for PCI VFs

 admin-virtq.tex | 235 ++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex     |  67 +++++++++++++-
 2 files changed, 299 insertions(+), 3 deletions(-)
 create mode 100644 admin-virtq.tex

-- 
2.21.0

[PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

In one of the many use cases a user wants to manipulate features and
configuration of the virtio devices regardless of the device type
(net/block/console). Some of this configuration is generic enough. i.e
Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
such features query and manipulation by its parent PCI PF.

Currently virtio specification defines control virtqueue to manipulate
features and configuration of the device it operates on. However,
control virtqueue commands are device type specific, which makes it very
difficult to extend for device agnostic commands.

To support this requirement in elegant way, this patch introduces a new
admin virtqueue. Admin virtqueue will use the same command format for all
types of virtio devices.

Manipulate features via admin virtqueue is asynchronous, scalable, easy
to extend and doesn't require additional and expensive on-die resources
to be allocated for every new feature that will be added in the future.

Subsequent patches make use of this admin virtqueue.

Reviewed-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
 content.tex     |  9 +++--
 2 files changed, 96 insertions(+), 2 deletions(-)
 create mode 100644 admin-virtq.tex

diff --git a/admin-virtq.tex b/admin-virtq.tex
new file mode 100644
index 0000000..1a41c22
--- /dev/null
+++ b/admin-virtq.tex
@@ -0,0 +1,89 @@
+\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
+
+Admin virtqueue is used to send administrative commands to manipulate
+various features of the device and/or to manipulate various features,
+if possible, of another device within the same group (e.g. PCI VFs of
+a parent PCI PF device are grouped together. These devices can be
+optionally managed by its parent PCI PF using its admin virtqueue.).
+
+Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
+feature bit.
+
+Admin virtqueue index may vary among different device types.
+
+Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
+are used by the device in out of order manner.
+
+The Admin command set defines the commands that may be issued only to the admin
+virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ feature, MUST
+support all the mandatory admin commands. A device MAY support also one or more
+optional admin commands. All commands are of the following form:
+
+\begin{lstlisting}
+struct virtio_admin_cmd {
+        /* Device-readable part */
+        u16 command;
+        u8 command_specific_data[];
+
+        /* Device-writable part */
+        u8 status;
+        u8 command_specific_error;
+        u8 command_specific_result[];
+};
+\end{lstlisting}
+
+The following table describes the generic Admin status codes:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Opcode & Status & Description \\
+\hline \hline
+00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
+\hline
+01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
+\hline
+02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or invalid opcode  \\
+\hline
+\end{tabular}
+
+The \field{command} and \field{command_specific_data} are
+set by the driver, and the device sets the \field{status}, the
+\field{command_specific_error} and the \field{command_specific_result},
+if needed.
+
+The \field{command_specific_error} should be inspected by the driver only if \field{status} is set to
+VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
+holds the command specific error. If \field{status} is not set to VIRTIO_ADMIN_STATUS_CS_ERR, the
+\field{command_specific_error} value is undefined.
+
+The following table describes the Admin command set:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Opcode & Command & M/O \\
+\hline \hline
+0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
+\hline
+0001h - 7FFFh   & Generic admin cmds    & -  \\
+\hline
+8000h - FFFFh   & Reserved    & - \\
+\hline
+\end{tabular}
+
+\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}
+
+The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data set by the driver.
+This command upon success, returns a data buffer that describes information about the supported
+admin capabilities by the device. This information is of form:
+\begin{lstlisting}
+struct virtio_admin_caps_identify_result {
+       /*
+        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
+        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
+        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
+        * ....
+        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
+        */
+       u8 caps[8192];
+};
+\end{lstlisting}
diff --git a/content.tex b/content.tex
index 32de668..c524fab 100644
--- a/content.tex
+++ b/content.tex
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23] Feature bits for the specific device type
 
-\item[24 to 40] Feature bits reserved for extensions to the queue and
+\item[24 to 41] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[41 and above] Feature bits reserved for future extensions.
+\item[42 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\input{admin-virtq.tex}
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   that the driver can reset a queue individually.
   See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
 
+  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
+  the device supports administration virtqueue negotiation.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
2.21.0

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
> +are used by the device in out of order manner.

Instead of special-casing AQ I'd like to see a generic capability
addressing this need. For example, TX for virtio net might benefit from
this too. And I'd like to mention, again, VIRTIO_F_PARTIAL_ORDER
proposal as one, arguably cleaner and more generic way to address this.
And if that's not adequate I'd like to address that as part of the
PARTIAL_ORDER proposal, this kind of per-queue in order was definitely
on the radar as it was formulated.

-- 
MST

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/26/2022 4:40 PM, Michael S. Tsirkin wrote:
> On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
>> +are used by the device in out of order manner.
> Instead of special-casing AQ I'd like to see a generic capability
> addressing this need. For example, TX for virtio net might benefit from
> this too. And I'd like to mention, again, VIRTIO_F_PARTIAL_ORDER
> proposal as one, arguably cleaner and more generic way to address this.

We already suggested a special capability for IN_ORDER for AQ and you 
asked to drop it. We drop it and agreed that AQ will be OOO.

Why are we going back here ?

You also mentioned that this patch set is already big enough so why 
solve all the problems we can think of in this one ? Why mixing 
VIRTIO_F_PARTIAL_ORDER here ?

> And if that's not adequate I'd like to address that as part of the
> PARTIAL_ORDER proposal, this kind of per-queue in order was definitely
> on the radar as it was formulated.
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Wed, Jan 26, 2022 at 04:54:22PM +0200, Max Gurtovoy wrote:
> 
> On 1/26/2022 4:40 PM, Michael S. Tsirkin wrote:
> > On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
> > > +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
> > > +are used by the device in out of order manner.
> > Instead of special-casing AQ I'd like to see a generic capability
> > addressing this need. For example, TX for virtio net might benefit from
> > this too. And I'd like to mention, again, VIRTIO_F_PARTIAL_ORDER
> > proposal as one, arguably cleaner and more generic way to address this.
> 
> We already suggested a special capability for IN_ORDER for AQ and you asked
> to drop it. We drop it and agreed that AQ will be OOO.
> 
> Why are we going back here ?

That's a misunderstanding. Currently all VQs of a device follow same
ordering. So I suggested making AQ just follow ordering of rest of VQs
of the device.

> You also mentioned that this patch set is already big enough so why solve
> all the problems we can think of in this one ?

Right. So just leave the ordering be for now, driver can just skip
IN_ORDER and use AQ without it if it wants to.

> Why mixing
> VIRTIO_F_PARTIAL_ORDER here ?

PARTIAL_ORDER has its own patch, no need to mix it in. Reusing that
seems cleaner than special-casing here though, we do try to have
reusable modules, otherwise things just get out of hand quickly.

> > And if that's not adequate I'd like to address that as part of the
> > PARTIAL_ORDER proposal, this kind of per-queue in order was definitely
> > on the radar as it was formulated.
> > 

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/26/2022 5:03 PM, Michael S. Tsirkin wrote:
> On Wed, Jan 26, 2022 at 04:54:22PM +0200, Max Gurtovoy wrote:
>> On 1/26/2022 4:40 PM, Michael S. Tsirkin wrote:
>>> On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
>>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
>>>> +are used by the device in out of order manner.
>>> Instead of special-casing AQ I'd like to see a generic capability
>>> addressing this need. For example, TX for virtio net might benefit from
>>> this too. And I'd like to mention, again, VIRTIO_F_PARTIAL_ORDER
>>> proposal as one, arguably cleaner and more generic way to address this.
>> We already suggested a special capability for IN_ORDER for AQ and you asked
>> to drop it. We drop it and agreed that AQ will be OOO.
>>
>> Why are we going back here ?
> That's a misunderstanding. Currently all VQs of a device follow same
> ordering. So I suggested making AQ just follow ordering of rest of VQs
> of the device.
Can we say the the AQ is by nature OOO so if AQ is set then IN_ORDER 
can't be set ?
>
>> You also mentioned that this patch set is already big enough so why solve
>> all the problems we can think of in this one ?
> Right. So just leave the ordering be for now, driver can just skip
> IN_ORDER and use AQ without it if it wants to.

Without mentioning this in the spec ? can we control all the drivers in 
the world ?

>
>> Why mixing
>> VIRTIO_F_PARTIAL_ORDER here ?
> PARTIAL_ORDER has its own patch, no need to mix it in. Reusing that
> seems cleaner than special-casing here though, we do try to have
> reusable modules, otherwise things just get out of hand quickly.

re-using modules that haven't yet merged ? This require mixing 2 proposals.

I think best we can do is to say:

"A device MUST NOT propose VIRTIO_F_IN_ORDER if VIRTIO_F_ADMIN_VQ is 
proposed."

>
>>> And if that's not adequate I'd like to address that as part of the
>>> PARTIAL_ORDER proposal, this kind of per-queue in order was definitely
>>> on the radar as it was formulated.
>>>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, January 26, 2022 8:34 PM
> 
> On Wed, Jan 26, 2022 at 04:54:22PM +0200, Max Gurtovoy wrote:
> >
> > On 1/26/2022 4:40 PM, Michael S. Tsirkin wrote:
> > > On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
> > > > +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue
> > > > +command buffers are used by the device in out of order manner.
> > > Instead of special-casing AQ I'd like to see a generic capability
> > > addressing this need. For example, TX for virtio net might benefit
> > > from this too. And I'd like to mention, again,
> > > VIRTIO_F_PARTIAL_ORDER proposal as one, arguably cleaner and more
> generic way to address this.
> >
> > We already suggested a special capability for IN_ORDER for AQ and you
> > asked to drop it. We drop it and agreed that AQ will be OOO.
> >
> > Why are we going back here ?
> 
> That's a misunderstanding. Currently all VQs of a device follow same ordering.
> So I suggested making AQ just follow ordering of rest of VQs of the device.
> 
> > You also mentioned that this patch set is already big enough so why
> > solve all the problems we can think of in this one ?
> 
> Right. So just leave the ordering be for now, driver can just skip IN_ORDER and
> use AQ without it if it wants to.
>
Ok. I read your same suggestion as what I thought now.
It answers my previous email.

Thanks.

RE: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, January 26, 2022 8:11 PM
> 
> On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
> > +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
> > +buffers are used by the device in out of order manner.
> 
> Instead of special-casing AQ I'd like to see a generic capability addressing this
> need. For example, TX for virtio net might benefit from this too. And I'd like to
> mention, again, VIRTIO_F_PARTIAL_ORDER proposal as one, arguably cleaner
> and more generic way to address this.
> And if that's not adequate I'd like to address that as part of the
> PARTIAL_ORDER proposal, this kind of per-queue in order was definitely on
> the radar as it was formulated.
As we dropped other less important items from this proposal because it was too big.
I am going to keep the PARTIAL_ORDER also out of this one. It falls in same bucket.

So AQ follows same ordering rules as other queues.
Are you ok with this in v3?

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

On Thu, Jan 27, 2022 at 11:55 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, January 26, 2022 8:11 PM
> >
> > On Mon, Jan 24, 2022 at 11:39:15AM +0200, Max Gurtovoy wrote:
> > > +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
> > > +buffers are used by the device in out of order manner.
> >
> > Instead of special-casing AQ I'd like to see a generic capability addressing this
> > need. For example, TX for virtio net might benefit from this too. And I'd like to
> > mention, again, VIRTIO_F_PARTIAL_ORDER proposal as one, arguably cleaner
> > and more generic way to address this.
> > And if that's not adequate I'd like to address that as part of the
> > PARTIAL_ORDER proposal, this kind of per-queue in order was definitely on
> > the radar as it was formulated.
> As we dropped other less important items from this proposal because it was too big.
> I am going to keep the PARTIAL_ORDER also out of this one. It falls in same bucket.
>
> So AQ follows same ordering rules as other queues.
> Are you ok with this in v3?

I'm fine with this.

Thanks

>

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

在 2022/1/24 下午5:39, Max Gurtovoy 写道:
> In one of the many use cases a user wants to manipulate features and
> configuration of the virtio devices regardless of the device type
> (net/block/console). Some of this configuration is generic enough. i.e
> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> such features query and manipulation by its parent PCI PF.
>
> Currently virtio specification defines control virtqueue to manipulate
> features and configuration of the device it operates on. However,
> control virtqueue commands are device type specific, which makes it very
> difficult to extend for device agnostic commands.
>
> To support this requirement in elegant way, this patch introduces a new
> admin virtqueue. Admin virtqueue will use the same command format for all
> types of virtio devices.
>
> Manipulate features via admin virtqueue is asynchronous, scalable, easy
> to extend and doesn't require additional and expensive on-die resources
> to be allocated for every new feature that will be added in the future.
>
> Subsequent patches make use of this admin virtqueue.
>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>   admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>   content.tex     |  9 +++--
>   2 files changed, 96 insertions(+), 2 deletions(-)
>   create mode 100644 admin-virtq.tex
>
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> new file mode 100644
> index 0000000..1a41c22
> --- /dev/null
> +++ b/admin-virtq.tex
> @@ -0,0 +1,89 @@
> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> +
> +Admin virtqueue is used to send administrative commands to manipulate
> +various features of the device and/or to manipulate various features,
> +if possible, of another device within the same group (e.g. PCI VFs of
> +a parent PCI PF device are grouped together. These devices can be
> +optionally managed by its parent PCI PF using its admin virtqueue.).
> +
> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> +feature bit.
> +
> +Admin virtqueue index may vary among different device types.
> +
> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command buffers
> +are used by the device in out of order manner.
> +
> +The Admin command set defines the commands that may be issued only to the admin
> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ feature, MUST
> +support all the mandatory admin commands. A device MAY support also one or more
> +optional admin commands. All commands are of the following form:


Do we need a way for advertising the supported optional commands (or 
features bits)? Otherwise dealing with the compatibility will result of 
per command probing.


> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd {
> +        /* Device-readable part */
> +        u16 command;


Two questions:

1) It looks to me it's better to have a explicit device_id here other 
than embed it in each command

2) virtio-net cvq use class/command which seems more elegant, e.g all 
commands belongs to MSI could be grouped into the same class


> +        u8 command_specific_data[];
> +
> +        /* Device-writable part */
> +        u8 status;
> +        u8 command_specific_error;


I wonder the reason why not using a single field for the above two 
fields. (Or any value for separating command specific error out, we 
don't do that for virtio-net cvq).


> +        u8 command_specific_result[];
> +};
> +\end{lstlisting}
> +
> +The following table describes the generic Admin status codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> +\hline
> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
> +\hline
> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or invalid opcode  \\
> +\hline
> +\end{tabular}
> +
> +The \field{command} and \field{command_specific_data} are
> +set by the driver, and the device sets the \field{status}, the
> +\field{command_specific_error} and the \field{command_specific_result},
> +if needed.
> +
> +The \field{command_specific_error} should be inspected by the driver only if \field{status} is set to
> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content of \field{command_specific_error}
> +holds the command specific error. If \field{status} is not set to VIRTIO_ADMIN_STATUS_CS_ERR, the
> +\field{command_specific_error} value is undefined.
> +
> +The following table describes the Admin command set:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Command & M/O \\
> +\hline \hline
> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
> +\hline
> +0001h - 7FFFh   & Generic admin cmds    & -  \\
> +\hline
> +8000h - FFFFh   & Reserved    & - \\
> +\hline
> +\end{tabular}


See above, I'd suggest to use class/command as virtio-net cvq.


> +
> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}
> +
> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data set by the driver.
> +This command upon success, returns a data buffer that describes information about the supported
> +admin capabilities by the device. This information is of form:
> +\begin{lstlisting}
> +struct virtio_admin_caps_identify_result {
> +       /*
> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
> +        * ....
> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
> +        */
> +       u8 caps[8192];
> +};


Ok, so I see the identify command. But I wonder if we can do that via 
the feature bits?

Thanks


> +\end{lstlisting}
> diff --git a/content.tex b/content.tex
> index 32de668..c524fab 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>   \begin{description}
>   \item[0 to 23] Feature bits for the specific device type
>   
> -\item[24 to 40] Feature bits reserved for extensions to the queue and
> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>     feature negotiation mechanisms
>   
> -\item[41 and above] Feature bits reserved for future extensions.
> +\item[42 and above] Feature bits reserved for future extensions.
>   \end{description}
>   
>   \begin{note}
> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>   types. It is RECOMMENDED that devices generate version 4
>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>   
> +\input{admin-virtq.tex}
> +
>   \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>   
>   We start with an overview of device initialization, then expand on the
> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>     that the driver can reset a queue individually.
>     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>   
> +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> +  the device supports administration virtqueue negotiation.
> +
>   \end{description}
>   
>   \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/27/2022 5:14 AM, Jason Wang wrote:
>
> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
>> In one of the many use cases a user wants to manipulate features and
>> configuration of the virtio devices regardless of the device type
>> (net/block/console). Some of this configuration is generic enough. i.e
>> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
>> such features query and manipulation by its parent PCI PF.
>>
>> Currently virtio specification defines control virtqueue to manipulate
>> features and configuration of the device it operates on. However,
>> control virtqueue commands are device type specific, which makes it very
>> difficult to extend for device agnostic commands.
>>
>> To support this requirement in elegant way, this patch introduces a new
>> admin virtqueue. Admin virtqueue will use the same command format for 
>> all
>> types of virtio devices.
>>
>> Manipulate features via admin virtqueue is asynchronous, scalable, easy
>> to extend and doesn't require additional and expensive on-die resources
>> to be allocated for every new feature that will be added in the future.
>>
>> Subsequent patches make use of this admin virtqueue.
>>
>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>   admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>>   content.tex     |  9 +++--
>>   2 files changed, 96 insertions(+), 2 deletions(-)
>>   create mode 100644 admin-virtq.tex
>>
>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>> new file mode 100644
>> index 0000000..1a41c22
>> --- /dev/null
>> +++ b/admin-virtq.tex
>> @@ -0,0 +1,89 @@
>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio 
>> Device / Admin Virtqueues}
>> +
>> +Admin virtqueue is used to send administrative commands to manipulate
>> +various features of the device and/or to manipulate various features,
>> +if possible, of another device within the same group (e.g. PCI VFs of
>> +a parent PCI PF device are grouped together. These devices can be
>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>> +
>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>> +feature bit.
>> +
>> +Admin virtqueue index may vary among different device types.
>> +
>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command 
>> buffers
>> +are used by the device in out of order manner.
>> +
>> +The Admin command set defines the commands that may be issued only 
>> to the admin
>> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ 
>> feature, MUST
>> +support all the mandatory admin commands. A device MAY support also 
>> one or more
>> +optional admin commands. All commands are of the following form:
>
>
> Do we need a way for advertising the supported optional commands (or 
> features bits)? Otherwise dealing with the compatibility will result 
> of per command probing.
>
>
>> +
>> +\begin{lstlisting}
>> +struct virtio_admin_cmd {
>> +        /* Device-readable part */
>> +        u16 command;
>
>
> Two questions:
>
> 1) It looks to me it's better to have a explicit device_id here other 
> than embed it in each command

this was already discussed.

We agreed that some commands are not referring to different device so 
no, we don't need to put it in generic structure.

I'm not against putting such id but we don't have an exact idea how will 
SF device_id will look like.

Maybe it will be some 128 bit uuid ? maybe u64 ? how can we guess ?


>
> 2) virtio-net cvq use class/command which seems more elegant, e.g all 
> commands belongs to MSI could be grouped into the same class

why can't we do it in u16 command opcode ? are you open for changes or 
should we copy/paste from net cvq ?

Let's ask differently. Why we need class/command structure ? why is it 
more elegant ?

>
>
>> +        u8 command_specific_data[];
>> +
>> +        /* Device-writable part */
>> +        u8 status;
>> +        u8 command_specific_error;
>
>
> I wonder the reason why not using a single field for the above two 
> fields. (Or any value for separating command specific error out, we 
> don't do that for virtio-net cvq).

each command have its own specific errors.

virtio net cvq is not generic - it's a net device cvqueue.

To make it generic we need to separate.

>
>
>> +        u8 command_specific_result[];
>> +};
>> +\end{lstlisting}
>> +
>> +The following table describes the generic Admin status codes:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Opcode & Status & Description \\
>> +\hline \hline
>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>> +\hline
>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
>> +\hline
>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or 
>> invalid opcode  \\
>> +\hline
>> +\end{tabular}
>> +
>> +The \field{command} and \field{command_specific_data} are
>> +set by the driver, and the device sets the \field{status}, the
>> +\field{command_specific_error} and the \field{command_specific_result},
>> +if needed.
>> +
>> +The \field{command_specific_error} should be inspected by the driver 
>> only if \field{status} is set to
>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content 
>> of \field{command_specific_error}
>> +holds the command specific error. If \field{status} is not set to 
>> VIRTIO_ADMIN_STATUS_CS_ERR, the
>> +\field{command_specific_error} value is undefined.
>> +
>> +The following table describes the Admin command set:
>> +
>> +\begin{tabular}{|l|l|l|}
>> +\hline
>> +Opcode & Command & M/O \\
>> +\hline \hline
>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
>> +\hline
>> +0001h - 7FFFh   & Generic admin cmds    & -  \\
>> +\hline
>> +8000h - FFFFh   & Reserved    & - \\
>> +\hline
>> +\end{tabular}
>
>
> See above, I'd suggest to use class/command as virtio-net cvq.

see my comment above.

>
>
>> +
>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic 
>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS 
>> IDENTIFY command}
>> +
>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data 
>> set by the driver.
>> +This command upon success, returns a data buffer that describes 
>> information about the supported
>> +admin capabilities by the device. This information is of form:
>> +\begin{lstlisting}
>> +struct virtio_admin_caps_identify_result {
>> +       /*
>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>> +        * ....
>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
>> +        */
>> +       u8 caps[8192];
>> +};
>
>
> Ok, so I see the identify command. But I wonder if we can do that via 
> the feature bits?

It doesn't scale. I tried it. It doesn't work.


Thanks.

>
> Thanks
>
>
>> +\end{lstlisting}
>> diff --git a/content.tex b/content.tex
>> index 32de668..c524fab 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities 
>> of a Virtio Device / Feature B
>>   \begin{description}
>>   \item[0 to 23] Feature bits for the specific device type
>>   -\item[24 to 40] Feature bits reserved for extensions to the queue and
>> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>>     feature negotiation mechanisms
>>   -\item[41 and above] Feature bits reserved for future extensions.
>> +\item[42 and above] Feature bits reserved for future extensions.
>>   \end{description}
>>     \begin{note}
>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic 
>> Facilities of a Virtio Device / Expo
>>   types. It is RECOMMENDED that devices generate version 4
>>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>   +\input{admin-virtq.tex}
>> +
>>   \chapter{General Initialization And Device 
>> Operation}\label{sec:General Initialization And Device Operation}
>>     We start with an overview of device initialization, then expand 
>> on the
>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature 
>> Bits}\label{sec:Reserved Feature Bits}
>>     that the driver can reset a queue individually.
>>     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / 
>> Virtqueue Reset}.
>>   +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>> +  the device supports administration virtqueue negotiation.
>> +
>>   \end{description}
>>     \drivernormative{\section}{Reserved Feature Bits}{Reserved 
>> Feature Bits}
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SW%2FM98QZ7Av2wsv1NoekveDdQmlfWMYcvZQiFvpLcAE%3D&amp;reserved=0
> Feedback License: 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=C5D9qmFqbNrx7CoTzsx%2B9qalts534XPaImHlcszwm1M%3D&amp;reserved=0
> List Guidelines: 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bi7OJDuH9CvoKdxtA01dn5SpZwDp3Bna4%2B00bVOuz2o%3D&amp;reserved=0
> Committee: 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SdGNrxTUZVIe2WJwb%2BdbT2UxagM9AdEzDWXccFvRE3s%3D&amp;reserved=0
> Join OASIS: 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ynKmp30vKJMw%2FL4IHbBFY9EebT%2BsVX2%2Boz7IIcO%2F1xg%3D&amp;reserved=0
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>
> On 1/27/2022 5:14 AM, Jason Wang wrote:
> >
> > 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
> >> In one of the many use cases a user wants to manipulate features and
> >> configuration of the virtio devices regardless of the device type
> >> (net/block/console). Some of this configuration is generic enough. i.e
> >> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> >> such features query and manipulation by its parent PCI PF.
> >>
> >> Currently virtio specification defines control virtqueue to manipulate
> >> features and configuration of the device it operates on. However,
> >> control virtqueue commands are device type specific, which makes it very
> >> difficult to extend for device agnostic commands.
> >>
> >> To support this requirement in elegant way, this patch introduces a new
> >> admin virtqueue. Admin virtqueue will use the same command format for
> >> all
> >> types of virtio devices.
> >>
> >> Manipulate features via admin virtqueue is asynchronous, scalable, easy
> >> to extend and doesn't require additional and expensive on-die resources
> >> to be allocated for every new feature that will be added in the future.
> >>
> >> Subsequent patches make use of this admin virtqueue.
> >>
> >> Reviewed-by: Parav Pandit <parav@nvidia.com>
> >> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >> ---
> >>   admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
> >>   content.tex     |  9 +++--
> >>   2 files changed, 96 insertions(+), 2 deletions(-)
> >>   create mode 100644 admin-virtq.tex
> >>
> >> diff --git a/admin-virtq.tex b/admin-virtq.tex
> >> new file mode 100644
> >> index 0000000..1a41c22
> >> --- /dev/null
> >> +++ b/admin-virtq.tex
> >> @@ -0,0 +1,89 @@
> >> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
> >> Device / Admin Virtqueues}
> >> +
> >> +Admin virtqueue is used to send administrative commands to manipulate
> >> +various features of the device and/or to manipulate various features,
> >> +if possible, of another device within the same group (e.g. PCI VFs of
> >> +a parent PCI PF device are grouped together. These devices can be
> >> +optionally managed by its parent PCI PF using its admin virtqueue.).
> >> +
> >> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >> +feature bit.
> >> +
> >> +Admin virtqueue index may vary among different device types.
> >> +
> >> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
> >> buffers
> >> +are used by the device in out of order manner.
> >> +
> >> +The Admin command set defines the commands that may be issued only
> >> to the admin
> >> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ
> >> feature, MUST
> >> +support all the mandatory admin commands. A device MAY support also
> >> one or more
> >> +optional admin commands. All commands are of the following form:
> >
> >
> > Do we need a way for advertising the supported optional commands (or
> > features bits)? Otherwise dealing with the compatibility will result
> > of per command probing.
> >
> >
> >> +
> >> +\begin{lstlisting}
> >> +struct virtio_admin_cmd {
> >> +        /* Device-readable part */
> >> +        u16 command;
> >
> >
> > Two questions:
> >
> > 1) It looks to me it's better to have a explicit device_id here other
> > than embed it in each command
>
> this was already discussed.
>
> We agreed that some commands are not referring to different device so
> no, we don't need to put it in generic structure.
>
> I'm not against putting such id but we don't have an exact idea how will
> SF device_id will look like.
>
> Maybe it will be some 128 bit uuid ? maybe u64 ?

How do you know u16 is sufficient for command?

>  how can we guess ?

You know it's you but not me that tries to propose this solution, right?

>
>
> >
> > 2) virtio-net cvq use class/command which seems more elegant, e.g all
> > commands belongs to MSI could be grouped into the same class
>
> why can't we do it in u16 command opcode ?

Where did you see I say you can't use u16?

> are you open for changes or
> should we copy/paste from net cvq ?

Net cvq comes first, it's natural to ask why you do things differently.

>
> Let's ask differently. Why we need class/command structure ? why is it
> more elegant ?

Don't you see my reply above?

"
commands belongs to MSI could be grouped into the same class
"

>
> >
> >
> >> +        u8 command_specific_data[];
> >> +
> >> +        /* Device-writable part */
> >> +        u8 status;
> >> +        u8 command_specific_error;
> >
> >
> > I wonder the reason why not using a single field for the above two
> > fields. (Or any value for separating command specific error out, we
> > don't do that for virtio-net cvq).
>
> each command have its own specific errors.
>
> virtio net cvq is not generic - it's a net device cvqueue.

We're talking about the command format, not its semantics, right?

It's command format is pretty general.

>
> To make it generic we need to separate.
>
> >
> >
> >> +        u8 command_specific_result[];
> >> +};
> >> +\end{lstlisting}
> >> +
> >> +The following table describes the generic Admin status codes:
> >> +
> >> +\begin{tabular}{|l|l|l|}
> >> +\hline
> >> +Opcode & Status & Description \\
> >> +\hline \hline
> >> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> >> +\hline
> >> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
> >> +\hline
> >> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or
> >> invalid opcode  \\
> >> +\hline
> >> +\end{tabular}
> >> +
> >> +The \field{command} and \field{command_specific_data} are
> >> +set by the driver, and the device sets the \field{status}, the
> >> +\field{command_specific_error} and the \field{command_specific_result},
> >> +if needed.
> >> +
> >> +The \field{command_specific_error} should be inspected by the driver
> >> only if \field{status} is set to
> >> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
> >> of \field{command_specific_error}
> >> +holds the command specific error. If \field{status} is not set to
> >> VIRTIO_ADMIN_STATUS_CS_ERR, the
> >> +\field{command_specific_error} value is undefined.
> >> +
> >> +The following table describes the Admin command set:
> >> +
> >> +\begin{tabular}{|l|l|l|}
> >> +\hline
> >> +Opcode & Command & M/O \\
> >> +\hline \hline
> >> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
> >> +\hline
> >> +0001h - 7FFFh   & Generic admin cmds    & -  \\
> >> +\hline
> >> +8000h - FFFFh   & Reserved    & - \\
> >> +\hline
> >> +\end{tabular}
> >
> >
> > See above, I'd suggest to use class/command as virtio-net cvq.
>
> see my comment above.
>
> >
> >
> >> +
> >> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
> >> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS
> >> IDENTIFY command}
> >> +
> >> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data
> >> set by the driver.
> >> +This command upon success, returns a data buffer that describes
> >> information about the supported
> >> +admin capabilities by the device. This information is of form:
> >> +\begin{lstlisting}
> >> +struct virtio_admin_caps_identify_result {
> >> +       /*
> >> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
> >> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
> >> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
> >> +        * ....
> >> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
> >> +        */
> >> +       u8 caps[8192];
> >> +};
> >
> >
> > Ok, so I see the identify command. But I wonder if we can do that via
> > the feature bits?
>
> It doesn't scale. I tried it. It doesn't work.

What prevents you from improving the scalability instead of trying to
revinting a duplicated mechanism?

Thanks

>
>
> Thanks.
>
> >
> > Thanks
> >
> >
> >> +\end{lstlisting}
> >> diff --git a/content.tex b/content.tex
> >> index 32de668..c524fab 100644
> >> --- a/content.tex
> >> +++ b/content.tex
> >> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities
> >> of a Virtio Device / Feature B
> >>   \begin{description}
> >>   \item[0 to 23] Feature bits for the specific device type
> >>   -\item[24 to 40] Feature bits reserved for extensions to the queue and
> >> +\item[24 to 41] Feature bits reserved for extensions to the queue and
> >>     feature negotiation mechanisms
> >>   -\item[41 and above] Feature bits reserved for future extensions.
> >> +\item[42 and above] Feature bits reserved for future extensions.
> >>   \end{description}
> >>     \begin{note}
> >> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
> >> Facilities of a Virtio Device / Expo
> >>   types. It is RECOMMENDED that devices generate version 4
> >>   UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >>   +\input{admin-virtq.tex}
> >> +
> >>   \chapter{General Initialization And Device
> >> Operation}\label{sec:General Initialization And Device Operation}
> >>     We start with an overview of device initialization, then expand
> >> on the
> >> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
> >> Bits}\label{sec:Reserved Feature Bits}
> >>     that the driver can reset a queue individually.
> >>     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> >> Virtqueue Reset}.
> >>   +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> >> +  the device supports administration virtqueue negotiation.
> >> +
> >>   \end{description}
> >>     \drivernormative{\section}{Reserved Feature Bits}{Reserved
> >> Feature Bits}
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SW%2FM98QZ7Av2wsv1NoekveDdQmlfWMYcvZQiFvpLcAE%3D&amp;reserved=0
> > Feedback License:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=C5D9qmFqbNrx7CoTzsx%2B9qalts534XPaImHlcszwm1M%3D&amp;reserved=0
> > List Guidelines:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=bi7OJDuH9CvoKdxtA01dn5SpZwDp3Bna4%2B00bVOuz2o%3D&amp;reserved=0
> > Committee:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=SdGNrxTUZVIe2WJwb%2BdbT2UxagM9AdEzDWXccFvRE3s%3D&amp;reserved=0
> > Join OASIS:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ca448108b88d84d87023d08d9e143238e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788501921358048%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=ynKmp30vKJMw%2FL4IHbBFY9EebT%2BsVX2%2Boz7IIcO%2F1xg%3D&amp;reserved=0
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/27/2022 3:03 PM, Jason Wang wrote:
> On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>
>> On 1/27/2022 5:14 AM, Jason Wang wrote:
>>> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
>>>> In one of the many use cases a user wants to manipulate features and
>>>> configuration of the virtio devices regardless of the device type
>>>> (net/block/console). Some of this configuration is generic enough. i.e
>>>> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
>>>> such features query and manipulation by its parent PCI PF.
>>>>
>>>> Currently virtio specification defines control virtqueue to manipulate
>>>> features and configuration of the device it operates on. However,
>>>> control virtqueue commands are device type specific, which makes it very
>>>> difficult to extend for device agnostic commands.
>>>>
>>>> To support this requirement in elegant way, this patch introduces a new
>>>> admin virtqueue. Admin virtqueue will use the same command format for
>>>> all
>>>> types of virtio devices.
>>>>
>>>> Manipulate features via admin virtqueue is asynchronous, scalable, easy
>>>> to extend and doesn't require additional and expensive on-die resources
>>>> to be allocated for every new feature that will be added in the future.
>>>>
>>>> Subsequent patches make use of this admin virtqueue.
>>>>
>>>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>> ---
>>>>    admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>    content.tex     |  9 +++--
>>>>    2 files changed, 96 insertions(+), 2 deletions(-)
>>>>    create mode 100644 admin-virtq.tex
>>>>
>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>> new file mode 100644
>>>> index 0000000..1a41c22
>>>> --- /dev/null
>>>> +++ b/admin-virtq.tex
>>>> @@ -0,0 +1,89 @@
>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
>>>> Device / Admin Virtqueues}
>>>> +
>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>> +various features of the device and/or to manipulate various features,
>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>> +a parent PCI PF device are grouped together. These devices can be
>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>> +
>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>> +feature bit.
>>>> +
>>>> +Admin virtqueue index may vary among different device types.
>>>> +
>>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
>>>> buffers
>>>> +are used by the device in out of order manner.
>>>> +
>>>> +The Admin command set defines the commands that may be issued only
>>>> to the admin
>>>> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ
>>>> feature, MUST
>>>> +support all the mandatory admin commands. A device MAY support also
>>>> one or more
>>>> +optional admin commands. All commands are of the following form:
>>>
>>> Do we need a way for advertising the supported optional commands (or
>>> features bits)? Otherwise dealing with the compatibility will result
>>> of per command probing.
>>>
>>>
>>>> +
>>>> +\begin{lstlisting}
>>>> +struct virtio_admin_cmd {
>>>> +        /* Device-readable part */
>>>> +        u16 command;
>>>
>>> Two questions:
>>>
>>> 1) It looks to me it's better to have a explicit device_id here other
>>> than embed it in each command
>> this was already discussed.
>>
>> We agreed that some commands are not referring to different device so
>> no, we don't need to put it in generic structure.
>>
>> I'm not against putting such id but we don't have an exact idea how will
>> SF device_id will look like.
>>
>> Maybe it will be some 128 bit uuid ? maybe u64 ?
> How do you know u16 is sufficient for command?

it's sufficient for VFs and this is the subject of the proposed commands.

When you'll add SF to virtio spec we'll create new commands for it with 
SF_id.

Again, this was discussed with Parav in V1 and was close. Please don't 
re-open.

>
>>   how can we guess ?
> You know it's you but not me that tries to propose this solution, right?
I'm proposing a solution but you ask questions that we answer but 
somehow they are asked again.
>
>>
>>> 2) virtio-net cvq use class/command which seems more elegant, e.g all
>>> commands belongs to MSI could be grouped into the same class
>> why can't we do it in u16 command opcode ?
> Where did you see I say you can't use u16?

you suggest to replace it to class + command.

I don't understand why.

If it's not mandatory, please ack.


>
>> are you open for changes or
>> should we copy/paste from net cvq ?
> Net cvq comes first, it's natural to ask why you do things differently.

I answered why.

Admin command set has a different purpose.

>
>> Let's ask differently. Why we need class/command structure ? why is it
>> more elegant ?
> Don't you see my reply above?
>
> "
> commands belongs to MSI could be grouped into the same class
> "

It's doesn't explain why class A + commands 1, 2 is better than opcodes 
1 + 2 without a class.


>
>>>
>>>> +        u8 command_specific_data[];
>>>> +
>>>> +        /* Device-writable part */
>>>> +        u8 status;
>>>> +        u8 command_specific_error;
>>>
>>> I wonder the reason why not using a single field for the above two
>>> fields. (Or any value for separating command specific error out, we
>>> don't do that for virtio-net cvq).
>> each command have its own specific errors.
>>
>> virtio net cvq is not generic - it's a net device cvqueue.
> We're talking about the command format, not its semantics, right?
>
> It's command format is pretty general.

in the Admin command set we would like to have better format that will 
allow users/drivers to better understand device error.

It is very common in many HW devices and specs out there.

I hope it doesn't critical for you as a concept.

>
>> To make it generic we need to separate.
>>
>>>
>>>> +        u8 command_specific_result[];
>>>> +};
>>>> +\end{lstlisting}
>>>> +
>>>> +The following table describes the generic Admin status codes:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Opcode & Status & Description \\
>>>> +\hline \hline
>>>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>>>> +\hline
>>>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
>>>> +\hline
>>>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or
>>>> invalid opcode  \\
>>>> +\hline
>>>> +\end{tabular}
>>>> +
>>>> +The \field{command} and \field{command_specific_data} are
>>>> +set by the driver, and the device sets the \field{status}, the
>>>> +\field{command_specific_error} and the \field{command_specific_result},
>>>> +if needed.
>>>> +
>>>> +The \field{command_specific_error} should be inspected by the driver
>>>> only if \field{status} is set to
>>>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
>>>> of \field{command_specific_error}
>>>> +holds the command specific error. If \field{status} is not set to
>>>> VIRTIO_ADMIN_STATUS_CS_ERR, the
>>>> +\field{command_specific_error} value is undefined.
>>>> +
>>>> +The following table describes the Admin command set:
>>>> +
>>>> +\begin{tabular}{|l|l|l|}
>>>> +\hline
>>>> +Opcode & Command & M/O \\
>>>> +\hline \hline
>>>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
>>>> +\hline
>>>> +0001h - 7FFFh   & Generic admin cmds    & -  \\
>>>> +\hline
>>>> +8000h - FFFFh   & Reserved    & - \\
>>>> +\hline
>>>> +\end{tabular}
>>>
>>> See above, I'd suggest to use class/command as virtio-net cvq.
>> see my comment above.
I answered above.
>>>
>>>> +
>>>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
>>>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS
>>>> IDENTIFY command}
>>>> +
>>>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data
>>>> set by the driver.
>>>> +This command upon success, returns a data buffer that describes
>>>> information about the supported
>>>> +admin capabilities by the device. This information is of form:
>>>> +\begin{lstlisting}
>>>> +struct virtio_admin_caps_identify_result {
>>>> +       /*
>>>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
>>>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>>>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>>>> +        * ....
>>>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
>>>> +        */
>>>> +       u8 caps[8192];
>>>> +};
>>>
>>> Ok, so I see the identify command. But I wonder if we can do that via
>>> the feature bits?
>> It doesn't scale. I tried it. It doesn't work.
> What prevents you from improving the scalability instead of trying to
> revinting a duplicated mechanism?

It was already discussed.

Spec doesn't scale, too many constrains between command-A,B,C and 
feature 44,45, 46 and for transport X, Y, Z.

>
> Thanks
>
>>
>> Thanks.
>>
>>> Thanks
>>>
>>>
>>>> +\end{lstlisting}
>>>> diff --git a/content.tex b/content.tex
>>>> index 32de668..c524fab 100644
>>>> --- a/content.tex
>>>> +++ b/content.tex
>>>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities
>>>> of a Virtio Device / Feature B
>>>>    \begin{description}
>>>>    \item[0 to 23] Feature bits for the specific device type
>>>>    -\item[24 to 40] Feature bits reserved for extensions to the queue and
>>>> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>>>>      feature negotiation mechanisms
>>>>    -\item[41 and above] Feature bits reserved for future extensions.
>>>> +\item[42 and above] Feature bits reserved for future extensions.
>>>>    \end{description}
>>>>      \begin{note}
>>>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
>>>> Facilities of a Virtio Device / Expo
>>>>    types. It is RECOMMENDED that devices generate version 4
>>>>    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>    +\input{admin-virtq.tex}
>>>> +
>>>>    \chapter{General Initialization And Device
>>>> Operation}\label{sec:General Initialization And Device Operation}
>>>>      We start with an overview of device initialization, then expand
>>>> on the
>>>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
>>>> Bits}\label{sec:Reserved Feature Bits}
>>>>      that the driver can reset a queue individually.
>>>>      See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
>>>> Virtqueue Reset}.
>>>>    +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>>>> +  the device supports administration virtqueue negotiation.
>>>> +
>>>>    \end{description}
>>>>      \drivernormative{\section}{Reserved Feature Bits}{Reserved
>>>> Feature Bits}
>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&amp;reserved=0
>>> Feedback License:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&amp;reserved=0
>>> List Guidelines:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&amp;reserved=0
>>> Committee:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&amp;reserved=0
>>> Join OASIS:
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&amp;reserved=0
>>>
>
> This publicly archived list offers a means to provide input to the
>
> OASIS Virtual I/O Device (VIRTIO) TC.
>
>
>
> In order to verify user consent to the Feedback License terms and
>
> to minimize spam in the list archive, subscription is required
>
> before posting.
>
>
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>
> List help: virtio-comment-help@lists.oasis-open.org
>
> List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&amp;reserved=0
>
> Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&amp;reserved=0
>
> List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&amp;reserved=0
>
> Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&amp;reserved=0
>
> Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&amp;reserved=0
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

On Thu, Jan 27, 2022 at 10:47 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>
> On 1/27/2022 3:03 PM, Jason Wang wrote:
> > On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >>
> >> On 1/27/2022 5:14 AM, Jason Wang wrote:
> >>> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
> >>>> In one of the many use cases a user wants to manipulate features and
> >>>> configuration of the virtio devices regardless of the device type
> >>>> (net/block/console). Some of this configuration is generic enough. i.e
> >>>> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> >>>> such features query and manipulation by its parent PCI PF.
> >>>>
> >>>> Currently virtio specification defines control virtqueue to manipulate
> >>>> features and configuration of the device it operates on. However,
> >>>> control virtqueue commands are device type specific, which makes it very
> >>>> difficult to extend for device agnostic commands.
> >>>>
> >>>> To support this requirement in elegant way, this patch introduces a new
> >>>> admin virtqueue. Admin virtqueue will use the same command format for
> >>>> all
> >>>> types of virtio devices.
> >>>>
> >>>> Manipulate features via admin virtqueue is asynchronous, scalable, easy
> >>>> to extend and doesn't require additional and expensive on-die resources
> >>>> to be allocated for every new feature that will be added in the future.
> >>>>
> >>>> Subsequent patches make use of this admin virtqueue.
> >>>>
> >>>> Reviewed-by: Parav Pandit <parav@nvidia.com>
> >>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> >>>> ---
> >>>>    admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
> >>>>    content.tex     |  9 +++--
> >>>>    2 files changed, 96 insertions(+), 2 deletions(-)
> >>>>    create mode 100644 admin-virtq.tex
> >>>>
> >>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
> >>>> new file mode 100644
> >>>> index 0000000..1a41c22
> >>>> --- /dev/null
> >>>> +++ b/admin-virtq.tex
> >>>> @@ -0,0 +1,89 @@
> >>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
> >>>> Device / Admin Virtqueues}
> >>>> +
> >>>> +Admin virtqueue is used to send administrative commands to manipulate
> >>>> +various features of the device and/or to manipulate various features,
> >>>> +if possible, of another device within the same group (e.g. PCI VFs of
> >>>> +a parent PCI PF device are grouped together. These devices can be
> >>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
> >>>> +
> >>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >>>> +feature bit.
> >>>> +
> >>>> +Admin virtqueue index may vary among different device types.
> >>>> +
> >>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
> >>>> buffers
> >>>> +are used by the device in out of order manner.
> >>>> +
> >>>> +The Admin command set defines the commands that may be issued only
> >>>> to the admin
> >>>> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ
> >>>> feature, MUST
> >>>> +support all the mandatory admin commands. A device MAY support also
> >>>> one or more
> >>>> +optional admin commands. All commands are of the following form:
> >>>
> >>> Do we need a way for advertising the supported optional commands (or
> >>> features bits)? Otherwise dealing with the compatibility will result
> >>> of per command probing.
> >>>
> >>>
> >>>> +
> >>>> +\begin{lstlisting}
> >>>> +struct virtio_admin_cmd {
> >>>> +        /* Device-readable part */
> >>>> +        u16 command;
> >>>
> >>> Two questions:
> >>>
> >>> 1) It looks to me it's better to have a explicit device_id here other
> >>> than embed it in each command
> >> this was already discussed.
> >>
> >> We agreed that some commands are not referring to different device so
> >> no, we don't need to put it in generic structure.
> >>
> >> I'm not against putting such id but we don't have an exact idea how will
> >> SF device_id will look like.
> >>
> >> Maybe it will be some 128 bit uuid ? maybe u64 ?
> > How do you know u16 is sufficient for command?
>
> it's sufficient for VFs and this is the subject of the proposed commands.

So VIRTIO_ADMIN_CAPS_IDENTIFY maps to 3 commands:

VIRTIO_ADMIN_PCI_SRIOV_ATTRS,
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,
VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET

Unless you do 1:1 mapping between cap bit and command, you will be
short of commands if you're using u16.

>
> When you'll add SF to virtio spec we'll create new commands for it with
> SF_id.
>
> Again, this was discussed with Parav in V1 and was close. Please don't
> re-open.

I don't think so, you can reserve a special device id for the PF itself.

>
> >
> >>   how can we guess ?
> > You know it's you but not me that tries to propose this solution, right?
> I'm proposing a solution but you ask questions that we answer but
> somehow they are asked again.
> >
> >>
> >>> 2) virtio-net cvq use class/command which seems more elegant, e.g all
> >>> commands belongs to MSI could be grouped into the same class
> >> why can't we do it in u16 command opcode ?
> > Where did you see I say you can't use u16?
>
> you suggest to replace it to class + command.
>
> I don't understand why.
>
> If it's not mandatory, please ack.
>
>
> >
> >> are you open for changes or
> >> should we copy/paste from net cvq ?
> > Net cvq comes first, it's natural to ask why you do things differently.
>
> I answered why.
>
> Admin command set has a different purpose.
>
> >
> >> Let's ask differently. Why we need class/command structure ? why is it
> >> more elegant ?
> > Don't you see my reply above?
> >
> > "
> > commands belongs to MSI could be grouped into the same class
> > "
>
> It's doesn't explain why class A + commands 1, 2 is better than opcodes
> 1 + 2 without a class.

Well, if you read how cvq work it's not hard to get the conclusion,

It's easier to be mapped to a feature bit. You just map a class and that's all.

Otherwise, you need to feature X:  command A,B,C,D ....

>
>
> >
> >>>
> >>>> +        u8 command_specific_data[];
> >>>> +
> >>>> +        /* Device-writable part */
> >>>> +        u8 status;
> >>>> +        u8 command_specific_error;
> >>>
> >>> I wonder the reason why not using a single field for the above two
> >>> fields. (Or any value for separating command specific error out, we
> >>> don't do that for virtio-net cvq).
> >> each command have its own specific errors.
> >>
> >> virtio net cvq is not generic - it's a net device cvqueue.
> > We're talking about the command format, not its semantics, right?
> >
> > It's command format is pretty general.
>
> in the Admin command set we would like to have better format that will
> allow users/drivers to better understand device error.
>
> It is very common in many HW devices and specs out there.
>
> I hope it doesn't critical for you as a concept.
>
> >
> >> To make it generic we need to separate.
> >>
> >>>
> >>>> +        u8 command_specific_result[];
> >>>> +};
> >>>> +\end{lstlisting}
> >>>> +
> >>>> +The following table describes the generic Admin status codes:
> >>>> +
> >>>> +\begin{tabular}{|l|l|l|}
> >>>> +\hline
> >>>> +Opcode & Status & Description \\
> >>>> +\hline \hline
> >>>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
> >>>> +\hline
> >>>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
> >>>> +\hline
> >>>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or
> >>>> invalid opcode  \\
> >>>> +\hline
> >>>> +\end{tabular}
> >>>> +
> >>>> +The \field{command} and \field{command_specific_data} are
> >>>> +set by the driver, and the device sets the \field{status}, the
> >>>> +\field{command_specific_error} and the \field{command_specific_result},
> >>>> +if needed.
> >>>> +
> >>>> +The \field{command_specific_error} should be inspected by the driver
> >>>> only if \field{status} is set to
> >>>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
> >>>> of \field{command_specific_error}
> >>>> +holds the command specific error. If \field{status} is not set to
> >>>> VIRTIO_ADMIN_STATUS_CS_ERR, the
> >>>> +\field{command_specific_error} value is undefined.
> >>>> +
> >>>> +The following table describes the Admin command set:
> >>>> +
> >>>> +\begin{tabular}{|l|l|l|}
> >>>> +\hline
> >>>> +Opcode & Command & M/O \\
> >>>> +\hline \hline
> >>>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
> >>>> +\hline
> >>>> +0001h - 7FFFh   & Generic admin cmds    & -  \\
> >>>> +\hline
> >>>> +8000h - FFFFh   & Reserved    & - \\
> >>>> +\hline
> >>>> +\end{tabular}
> >>>
> >>> See above, I'd suggest to use class/command as virtio-net cvq.
> >> see my comment above.
> I answered above.
> >>>
> >>>> +
> >>>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
> >>>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS
> >>>> IDENTIFY command}
> >>>> +
> >>>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data
> >>>> set by the driver.
> >>>> +This command upon success, returns a data buffer that describes
> >>>> information about the supported
> >>>> +admin capabilities by the device. This information is of form:
> >>>> +\begin{lstlisting}
> >>>> +struct virtio_admin_caps_identify_result {
> >>>> +       /*
> >>>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
> >>>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
> >>>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
> >>>> +        * ....
> >>>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
> >>>> +        */
> >>>> +       u8 caps[8192];
> >>>> +};
> >>>
> >>> Ok, so I see the identify command. But I wonder if we can do that via
> >>> the feature bits?
> >> It doesn't scale. I tried it. It doesn't work.
> > What prevents you from improving the scalability instead of trying to
> > revinting a duplicated mechanism?
>
> It was already discussed.
>
> Spec doesn't scale, too many constrains between command-A,B,C and
> feature 44,45, 46 and for transport X, Y, Z.

What I meant is, if you feel the current feature negotiation doesn't
scale, you can improve it instead of reinventing it partially (e.g no
negotiation) .

Thanks

>
> >
> > Thanks
> >
> >>
> >> Thanks.
> >>
> >>> Thanks
> >>>
> >>>
> >>>> +\end{lstlisting}
> >>>> diff --git a/content.tex b/content.tex
> >>>> index 32de668..c524fab 100644
> >>>> --- a/content.tex
> >>>> +++ b/content.tex
> >>>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities
> >>>> of a Virtio Device / Feature B
> >>>>    \begin{description}
> >>>>    \item[0 to 23] Feature bits for the specific device type
> >>>>    -\item[24 to 40] Feature bits reserved for extensions to the queue and
> >>>> +\item[24 to 41] Feature bits reserved for extensions to the queue and
> >>>>      feature negotiation mechanisms
> >>>>    -\item[41 and above] Feature bits reserved for future extensions.
> >>>> +\item[42 and above] Feature bits reserved for future extensions.
> >>>>    \end{description}
> >>>>      \begin{note}
> >>>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
> >>>> Facilities of a Virtio Device / Expo
> >>>>    types. It is RECOMMENDED that devices generate version 4
> >>>>    UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >>>>    +\input{admin-virtq.tex}
> >>>> +
> >>>>    \chapter{General Initialization And Device
> >>>> Operation}\label{sec:General Initialization And Device Operation}
> >>>>      We start with an overview of device initialization, then expand
> >>>> on the
> >>>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
> >>>> Bits}\label{sec:Reserved Feature Bits}
> >>>>      that the driver can reset a queue individually.
> >>>>      See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
> >>>> Virtqueue Reset}.
> >>>>    +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> >>>> +  the device supports administration virtqueue negotiation.
> >>>> +
> >>>>    \end{description}
> >>>>      \drivernormative{\section}{Reserved Feature Bits}{Reserved
> >>>> Feature Bits}
> >>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive:
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&amp;reserved=0
> >>> Feedback License:
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&amp;reserved=0
> >>> List Guidelines:
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&amp;reserved=0
> >>> Committee:
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&amp;reserved=0
> >>> Join OASIS:
> >>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&amp;reserved=0
> >>>
> >
> > This publicly archived list offers a means to provide input to the
> >
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> >
> >
> > In order to verify user consent to the Feedback License terms and
> >
> > to minimize spam in the list archive, subscription is required
> >
> > before posting.
> >
> >
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >
> > List help: virtio-comment-help@lists.oasis-open.org
> >
> > List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hkU2cd8o0F%2FTH%2F%2BHA8GQzVKZHPIDpuHBLMgVbjQSvN4%3D&amp;reserved=0
> >
> > Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=jqtoxmxY5oMIJ8X4R%2FCADdAwYHFNP%2Fix%2Frue8xqf8%2F0%3D&amp;reserved=0
> >
> > List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=a3bGp1iZqbK8dy2zMV3Umypft1kIL%2Bugjc8sZRZaX1M%3D&amp;reserved=0
> >
> > Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=H5cMIMFYWJT1AVO9%2FLrJ8KDidyuFhuE%2Bb7xFTlKNDLI%3D&amp;reserved=0
> >
> > Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cc8efa048a41e4be4d35508d9e195694e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637788855745609706%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=RY2TmLGYxJLQkX6GQVUxXOL7GeoWR6EqL8kStesrmD0%3D&amp;reserved=0
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/28/2022 5:18 AM, Jason Wang wrote:
> On Thu, Jan 27, 2022 at 10:47 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>
>> On 1/27/2022 3:03 PM, Jason Wang wrote:
>>> On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>> On 1/27/2022 5:14 AM, Jason Wang wrote:
>>>>> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
>>>>>> In one of the many use cases a user wants to manipulate features and
>>>>>> configuration of the virtio devices regardless of the device type
>>>>>> (net/block/console). Some of this configuration is generic enough. i.e
>>>>>> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
>>>>>> such features query and manipulation by its parent PCI PF.
>>>>>>
>>>>>> Currently virtio specification defines control virtqueue to manipulate
>>>>>> features and configuration of the device it operates on. However,
>>>>>> control virtqueue commands are device type specific, which makes it very
>>>>>> difficult to extend for device agnostic commands.
>>>>>>
>>>>>> To support this requirement in elegant way, this patch introduces a new
>>>>>> admin virtqueue. Admin virtqueue will use the same command format for
>>>>>> all
>>>>>> types of virtio devices.
>>>>>>
>>>>>> Manipulate features via admin virtqueue is asynchronous, scalable, easy
>>>>>> to extend and doesn't require additional and expensive on-die resources
>>>>>> to be allocated for every new feature that will be added in the future.
>>>>>>
>>>>>> Subsequent patches make use of this admin virtqueue.
>>>>>>
>>>>>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>> ---
>>>>>>     admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>     content.tex     |  9 +++--
>>>>>>     2 files changed, 96 insertions(+), 2 deletions(-)
>>>>>>     create mode 100644 admin-virtq.tex
>>>>>>
>>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>>> new file mode 100644
>>>>>> index 0000000..1a41c22
>>>>>> --- /dev/null
>>>>>> +++ b/admin-virtq.tex
>>>>>> @@ -0,0 +1,89 @@
>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
>>>>>> Device / Admin Virtqueues}
>>>>>> +
>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>> +various features of the device and/or to manipulate various features,
>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>> +
>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>> +feature bit.
>>>>>> +
>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>> +
>>>>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue command
>>>>>> buffers
>>>>>> +are used by the device in out of order manner.
>>>>>> +
>>>>>> +The Admin command set defines the commands that may be issued only
>>>>>> to the admin
>>>>>> +virtqueue. Each virtio device that advertises the VIRTIO_F_ADMIN_VQ
>>>>>> feature, MUST
>>>>>> +support all the mandatory admin commands. A device MAY support also
>>>>>> one or more
>>>>>> +optional admin commands. All commands are of the following form:
>>>>> Do we need a way for advertising the supported optional commands (or
>>>>> features bits)? Otherwise dealing with the compatibility will result
>>>>> of per command probing.
>>>>>
>>>>>
>>>>>> +
>>>>>> +\begin{lstlisting}
>>>>>> +struct virtio_admin_cmd {
>>>>>> +        /* Device-readable part */
>>>>>> +        u16 command;
>>>>> Two questions:
>>>>>
>>>>> 1) It looks to me it's better to have a explicit device_id here other
>>>>> than embed it in each command
>>>> this was already discussed.
>>>>
>>>> We agreed that some commands are not referring to different device so
>>>> no, we don't need to put it in generic structure.
>>>>
>>>> I'm not against putting such id but we don't have an exact idea how will
>>>> SF device_id will look like.
>>>>
>>>> Maybe it will be some 128 bit uuid ? maybe u64 ?
>>> How do you know u16 is sufficient for command?
>> it's sufficient for VFs and this is the subject of the proposed commands.
> So VIRTIO_ADMIN_CAPS_IDENTIFY maps to 3 commands:
>
> VIRTIO_ADMIN_PCI_SRIOV_ATTRS,
> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,
> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET
>
> Unless you do 1:1 mapping between cap bit and command, you will be
> short of commands if you're using u16.

2^16 admin commands are more than enough.


>
>> When you'll add SF to virtio spec we'll create new commands for it with
>> SF_id.
>>
>> Again, this was discussed with Parav in V1 and was close. Please don't
>> re-open.
> I don't think so, you can reserve a special device id for the PF itself.

can you propose something concrete ?

are you sure by 100% that sf_id is 32 bit ? can the TG ack on that ?

>
>>>>    how can we guess ?
>>> You know it's you but not me that tries to propose this solution, right?
>> I'm proposing a solution but you ask questions that we answer but
>> somehow they are asked again.
>>>>> 2) virtio-net cvq use class/command which seems more elegant, e.g all
>>>>> commands belongs to MSI could be grouped into the same class
>>>> why can't we do it in u16 command opcode ?
>>> Where did you see I say you can't use u16?
>> you suggest to replace it to class + command.
>>
>> I don't understand why.
>>
>> If it's not mandatory, please ack.
>>
>>
>>>> are you open for changes or
>>>> should we copy/paste from net cvq ?
>>> Net cvq comes first, it's natural to ask why you do things differently.
>> I answered why.
>>
>> Admin command set has a different purpose.
>>
>>>> Let's ask differently. Why we need class/command structure ? why is it
>>>> more elegant ?
>>> Don't you see my reply above?
>>>
>>> "
>>> commands belongs to MSI could be grouped into the same class
>>> "
>> It's doesn't explain why class A + commands 1, 2 is better than opcodes
>> 1 + 2 without a class.
> Well, if you read how cvq work it's not hard to get the conclusion,
>
> It's easier to be mapped to a feature bit. You just map a class and that's all.
>
> Otherwise, you need to feature X:  command A,B,C,D ....

But I just added one feature bit: the existence of the AQ.

in your suggestion I'll need to add for each feature:

Feature X can be negotiated only if AQ feature is negotiated.

And if feature X will have a small extension so a new feature will be 
needed and it will say:

Feature Y can be negotiated in feature X and feature AQ are negotiated.

or if feature X and feature Y are mutual exclusive then:

feature X can't be negotiated if feature Y is negotiated.

feature X can be negotiated only if feature AQ is negotiated.


and this is a small example. We will have more and more Admin 
functionality and this will be impossible to scale.

A simple identify command solve this.

>
>>
>>>>>> +        u8 command_specific_data[];
>>>>>> +
>>>>>> +        /* Device-writable part */
>>>>>> +        u8 status;
>>>>>> +        u8 command_specific_error;
>>>>> I wonder the reason why not using a single field for the above two
>>>>> fields. (Or any value for separating command specific error out, we
>>>>> don't do that for virtio-net cvq).
>>>> each command have its own specific errors.
>>>>
>>>> virtio net cvq is not generic - it's a net device cvqueue.
>>> We're talking about the command format, not its semantics, right?
>>>
>>> It's command format is pretty general.
>> in the Admin command set we would like to have better format that will
>> allow users/drivers to better understand device error.
>>
>> It is very common in many HW devices and specs out there.
>>
>> I hope it doesn't critical for you as a concept.
>>
>>>> To make it generic we need to separate.
>>>>
>>>>>> +        u8 command_specific_result[];
>>>>>> +};
>>>>>> +\end{lstlisting}
>>>>>> +
>>>>>> +The following table describes the generic Admin status codes:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Opcode & Status & Description \\
>>>>>> +\hline \hline
>>>>>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>>>>>> +\hline
>>>>>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
>>>>>> +\hline
>>>>>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & unsupported or
>>>>>> invalid opcode  \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>>> +
>>>>>> +The \field{command} and \field{command_specific_data} are
>>>>>> +set by the driver, and the device sets the \field{status}, the
>>>>>> +\field{command_specific_error} and the \field{command_specific_result},
>>>>>> +if needed.
>>>>>> +
>>>>>> +The \field{command_specific_error} should be inspected by the driver
>>>>>> only if \field{status} is set to
>>>>>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the content
>>>>>> of \field{command_specific_error}
>>>>>> +holds the command specific error. If \field{status} is not set to
>>>>>> VIRTIO_ADMIN_STATUS_CS_ERR, the
>>>>>> +\field{command_specific_error} value is undefined.
>>>>>> +
>>>>>> +The following table describes the Admin command set:
>>>>>> +
>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>> +\hline
>>>>>> +Opcode & Command & M/O \\
>>>>>> +\hline \hline
>>>>>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
>>>>>> +\hline
>>>>>> +0001h - 7FFFh   & Generic admin cmds    & -  \\
>>>>>> +\hline
>>>>>> +8000h - FFFFh   & Reserved    & - \\
>>>>>> +\hline
>>>>>> +\end{tabular}
>>>>> See above, I'd suggest to use class/command as virtio-net cvq.
>>>> see my comment above.
>> I answered above.
>>>>>> +
>>>>>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
>>>>>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS
>>>>>> IDENTIFY command}
>>>>>> +
>>>>>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific data
>>>>>> set by the driver.
>>>>>> +This command upon success, returns a data buffer that describes
>>>>>> information about the supported
>>>>>> +admin capabilities by the device. This information is of form:
>>>>>> +\begin{lstlisting}
>>>>>> +struct virtio_admin_caps_identify_result {
>>>>>> +       /*
>>>>>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
>>>>>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>>>>>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>>>>>> +        * ....
>>>>>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
>>>>>> +        */
>>>>>> +       u8 caps[8192];
>>>>>> +};
>>>>> Ok, so I see the identify command. But I wonder if we can do that via
>>>>> the feature bits?
>>>> It doesn't scale. I tried it. It doesn't work.
>>> What prevents you from improving the scalability instead of trying to
>>> revinting a duplicated mechanism?
>> It was already discussed.
>>
>> Spec doesn't scale, too many constrains between command-A,B,C and
>> feature 44,45, 46 and for transport X, Y, Z.
> What I meant is, if you feel the current feature negotiation doesn't
> scale, you can improve it instead of reinventing it partially (e.g no
> negotiation) .

We're adding an identify command for the admin command set.

did you review it ?

>
> Thanks
>
>>> Thanks
>>>
>>>> Thanks.
>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>> +\end{lstlisting}
>>>>>> diff --git a/content.tex b/content.tex
>>>>>> index 32de668..c524fab 100644
>>>>>> --- a/content.tex
>>>>>> +++ b/content.tex
>>>>>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities
>>>>>> of a Virtio Device / Feature B
>>>>>>     \begin{description}
>>>>>>     \item[0 to 23] Feature bits for the specific device type
>>>>>>     -\item[24 to 40] Feature bits reserved for extensions to the queue and
>>>>>> +\item[24 to 41] Feature bits reserved for extensions to the queue and
>>>>>>       feature negotiation mechanisms
>>>>>>     -\item[41 and above] Feature bits reserved for future extensions.
>>>>>> +\item[42 and above] Feature bits reserved for future extensions.
>>>>>>     \end{description}
>>>>>>       \begin{note}
>>>>>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
>>>>>> Facilities of a Virtio Device / Expo
>>>>>>     types. It is RECOMMENDED that devices generate version 4
>>>>>>     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>     +\input{admin-virtq.tex}
>>>>>> +
>>>>>>     \chapter{General Initialization And Device
>>>>>> Operation}\label{sec:General Initialization And Device Operation}
>>>>>>       We start with an overview of device initialization, then expand
>>>>>> on the
>>>>>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
>>>>>> Bits}\label{sec:Reserved Feature Bits}
>>>>>>       that the driver can reset a queue individually.
>>>>>>       See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues /
>>>>>> Virtqueue Reset}.
>>>>>>     +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>>>>>> +  the device supports administration virtqueue negotiation.
>>>>>> +
>>>>>>     \end{description}
>>>>>>       \drivernormative{\section}{Reserved Feature Bits}{Reserved
>>>>>> Feature Bits}
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive:
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0
>>>>> Feedback License:
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0
>>>>> List Guidelines:
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0
>>>>> Committee:
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0
>>>>> Join OASIS:
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>>
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>>
>>>
>>> In order to verify user consent to the Feedback License terms and
>>>
>>> to minimize spam in the list archive, subscription is required
>>>
>>> before posting.
>>>
>>>
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>
>>> List help: virtio-comment-help@lists.oasis-open.org
>>>
>>> List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0
>>>
>>> Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0
>>>
>>> List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0
>>>
>>> Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0
>>>
>>> Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0
>>>
>
> This publicly archived list offers a means to provide input to the
>
> OASIS Virtual I/O Device (VIRTIO) TC.
>
>
>
> In order to verify user consent to the Feedback License terms and
>
> to minimize spam in the list archive, subscription is required
>
> before posting.
>
>
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>
> List help: virtio-comment-help@lists.oasis-open.org
>
> List archive: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0
>
> Feedback License: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0
>
> List Guidelines: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0
>
> Committee: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0
>
> Join OASIS: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

在 2022/1/30 下午6:31, Max Gurtovoy 写道:
>
> On 1/28/2022 5:18 AM, Jason Wang wrote:
>> On Thu, Jan 27, 2022 at 10:47 PM Max Gurtovoy <mgurtovoy@nvidia.com> 
>> wrote:
>>>
>>> On 1/27/2022 3:03 PM, Jason Wang wrote:
>>>> On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy <mgurtovoy@nvidia.com> 
>>>> wrote:
>>>>> On 1/27/2022 5:14 AM, Jason Wang wrote:
>>>>>> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
>>>>>>> In one of the many use cases a user wants to manipulate features 
>>>>>>> and
>>>>>>> configuration of the virtio devices regardless of the device type
>>>>>>> (net/block/console). Some of this configuration is generic 
>>>>>>> enough. i.e
>>>>>>> Number of MSI-X vectors of a virtio PCI VF device. There is a 
>>>>>>> need to do
>>>>>>> such features query and manipulation by its parent PCI PF.
>>>>>>>
>>>>>>> Currently virtio specification defines control virtqueue to 
>>>>>>> manipulate
>>>>>>> features and configuration of the device it operates on. However,
>>>>>>> control virtqueue commands are device type specific, which makes 
>>>>>>> it very
>>>>>>> difficult to extend for device agnostic commands.
>>>>>>>
>>>>>>> To support this requirement in elegant way, this patch 
>>>>>>> introduces a new
>>>>>>> admin virtqueue. Admin virtqueue will use the same command 
>>>>>>> format for
>>>>>>> all
>>>>>>> types of virtio devices.
>>>>>>>
>>>>>>> Manipulate features via admin virtqueue is asynchronous, 
>>>>>>> scalable, easy
>>>>>>> to extend and doesn't require additional and expensive on-die 
>>>>>>> resources
>>>>>>> to be allocated for every new feature that will be added in the 
>>>>>>> future.
>>>>>>>
>>>>>>> Subsequent patches make use of this admin virtqueue.
>>>>>>>
>>>>>>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>> ---
>>>>>>>     admin-virtq.tex | 89 
>>>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>     content.tex     |  9 +++--
>>>>>>>     2 files changed, 96 insertions(+), 2 deletions(-)
>>>>>>>     create mode 100644 admin-virtq.tex
>>>>>>>
>>>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>>>> new file mode 100644
>>>>>>> index 0000000..1a41c22
>>>>>>> --- /dev/null
>>>>>>> +++ b/admin-virtq.tex
>>>>>>> @@ -0,0 +1,89 @@
>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
>>>>>>> Device / Admin Virtqueues}
>>>>>>> +
>>>>>>> +Admin virtqueue is used to send administrative commands to 
>>>>>>> manipulate
>>>>>>> +various features of the device and/or to manipulate various 
>>>>>>> features,
>>>>>>> +if possible, of another device within the same group (e.g. PCI 
>>>>>>> VFs of
>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>> +optionally managed by its parent PCI PF using its admin 
>>>>>>> virtqueue.).
>>>>>>> +
>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>> +feature bit.
>>>>>>> +
>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>> +
>>>>>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue 
>>>>>>> command
>>>>>>> buffers
>>>>>>> +are used by the device in out of order manner.
>>>>>>> +
>>>>>>> +The Admin command set defines the commands that may be issued only
>>>>>>> to the admin
>>>>>>> +virtqueue. Each virtio device that advertises the 
>>>>>>> VIRTIO_F_ADMIN_VQ
>>>>>>> feature, MUST
>>>>>>> +support all the mandatory admin commands. A device MAY support 
>>>>>>> also
>>>>>>> one or more
>>>>>>> +optional admin commands. All commands are of the following form:
>>>>>> Do we need a way for advertising the supported optional commands (or
>>>>>> features bits)? Otherwise dealing with the compatibility will result
>>>>>> of per command probing.
>>>>>>
>>>>>>
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_cmd {
>>>>>>> +        /* Device-readable part */
>>>>>>> +        u16 command;
>>>>>> Two questions:
>>>>>>
>>>>>> 1) It looks to me it's better to have a explicit device_id here 
>>>>>> other
>>>>>> than embed it in each command
>>>>> this was already discussed.
>>>>>
>>>>> We agreed that some commands are not referring to different device so
>>>>> no, we don't need to put it in generic structure.
>>>>>
>>>>> I'm not against putting such id but we don't have an exact idea 
>>>>> how will
>>>>> SF device_id will look like.
>>>>>
>>>>> Maybe it will be some 128 bit uuid ? maybe u64 ?
>>>> How do you know u16 is sufficient for command?
>>> it's sufficient for VFs and this is the subject of the proposed 
>>> commands.
>> So VIRTIO_ADMIN_CAPS_IDENTIFY maps to 3 commands:
>>
>> VIRTIO_ADMIN_PCI_SRIOV_ATTRS,
>> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,
>> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET
>>
>> Unless you do 1:1 mapping between cap bit and command, you will be
>> short of commands if you're using u16.
>
> 2^16 admin commands are more than enough.


You introduce 65536 cap bits via identify. And you use u16 for command.

But you map 3 commands to a single cap bit. You will be short of 
commands. Isn't it?


>
>
>>
>>> When you'll add SF to virtio spec we'll create new commands for it with
>>> SF_id.
>>>
>>> Again, this was discussed with Parav in V1 and was close. Please don't
>>> re-open.
>> I don't think so, you can reserve a special device id for the PF itself.
>
> can you propose something concrete ?
>
> are you sure by 100% that sf_id is 32 bit ? can the TG ack on that ?


I meant for command without and destination. You can reserve a special 
ID for that.

Regarding the bit, if you think 32bit is not sufficient, let's just use 
64bit, or reserving some bytes in the command.


>
>>
>>>>>    how can we guess ?
>>>> You know it's you but not me that tries to propose this solution, 
>>>> right?
>>> I'm proposing a solution but you ask questions that we answer but
>>> somehow they are asked again.
>>>>>> 2) virtio-net cvq use class/command which seems more elegant, e.g 
>>>>>> all
>>>>>> commands belongs to MSI could be grouped into the same class
>>>>> why can't we do it in u16 command opcode ?
>>>> Where did you see I say you can't use u16?
>>> you suggest to replace it to class + command.
>>>
>>> I don't understand why.
>>>
>>> If it's not mandatory, please ack.
>>>
>>>
>>>>> are you open for changes or
>>>>> should we copy/paste from net cvq ?
>>>> Net cvq comes first, it's natural to ask why you do things 
>>>> differently.
>>> I answered why.
>>>
>>> Admin command set has a different purpose.
>>>
>>>>> Let's ask differently. Why we need class/command structure ? why 
>>>>> is it
>>>>> more elegant ?
>>>> Don't you see my reply above?
>>>>
>>>> "
>>>> commands belongs to MSI could be grouped into the same class
>>>> "
>>> It's doesn't explain why class A + commands 1, 2 is better than opcodes
>>> 1 + 2 without a class.
>> Well, if you read how cvq work it's not hard to get the conclusion,
>>
>> It's easier to be mapped to a feature bit. You just map a class and 
>> that's all.
>>
>> Otherwise, you need to feature X:  command A,B,C,D ....
>
> But I just added one feature bit: the existence of the AQ.
>
> in your suggestion I'll need to add for each feature:
>
> Feature X can be negotiated only if AQ feature is negotiated.
>
> And if feature X will have a small extension so a new feature will be 
> needed and it will say:
>
> Feature Y can be negotiated in feature X and feature AQ are negotiated.
>
> or if feature X and feature Y are mutual exclusive then:
>
> feature X can't be negotiated if feature Y is negotiated.
>
> feature X can be negotiated only if feature AQ is negotiated.


This is how virtqueue works for all the existing device. It's as simple 
as depending the feature to another feature which indicates the existing 
of a specific type of virtqueue.


>
>
> and this is a small example. We will have more and more Admin 
> functionality and this will be impossible to scale.
>
> A simple identify command solve this.
>
>>
>>>
>>>>>>> +        u8 command_specific_data[];
>>>>>>> +
>>>>>>> +        /* Device-writable part */
>>>>>>> +        u8 status;
>>>>>>> +        u8 command_specific_error;
>>>>>> I wonder the reason why not using a single field for the above two
>>>>>> fields. (Or any value for separating command specific error out, we
>>>>>> don't do that for virtio-net cvq).
>>>>> each command have its own specific errors.
>>>>>
>>>>> virtio net cvq is not generic - it's a net device cvqueue.
>>>> We're talking about the command format, not its semantics, right?
>>>>
>>>> It's command format is pretty general.
>>> in the Admin command set we would like to have better format that will
>>> allow users/drivers to better understand device error.
>>>
>>> It is very common in many HW devices and specs out there.
>>>
>>> I hope it doesn't critical for you as a concept.
>>>
>>>>> To make it generic we need to separate.
>>>>>
>>>>>>> +        u8 command_specific_result[];
>>>>>>> +};
>>>>>>> +\end{lstlisting}
>>>>>>> +
>>>>>>> +The following table describes the generic Admin status codes:
>>>>>>> +
>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>> +\hline
>>>>>>> +Opcode & Status & Description \\
>>>>>>> +\hline \hline
>>>>>>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>>>>>>> +\hline
>>>>>>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific error  \\
>>>>>>> +\hline
>>>>>>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & 
>>>>>>> unsupported or
>>>>>>> invalid opcode  \\
>>>>>>> +\hline
>>>>>>> +\end{tabular}
>>>>>>> +
>>>>>>> +The \field{command} and \field{command_specific_data} are
>>>>>>> +set by the driver, and the device sets the \field{status}, the
>>>>>>> +\field{command_specific_error} and the 
>>>>>>> \field{command_specific_result},
>>>>>>> +if needed.
>>>>>>> +
>>>>>>> +The \field{command_specific_error} should be inspected by the 
>>>>>>> driver
>>>>>>> only if \field{status} is set to
>>>>>>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the 
>>>>>>> content
>>>>>>> of \field{command_specific_error}
>>>>>>> +holds the command specific error. If \field{status} is not set to
>>>>>>> VIRTIO_ADMIN_STATUS_CS_ERR, the
>>>>>>> +\field{command_specific_error} value is undefined.
>>>>>>> +
>>>>>>> +The following table describes the Admin command set:
>>>>>>> +
>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>> +\hline
>>>>>>> +Opcode & Command & M/O \\
>>>>>>> +\hline \hline
>>>>>>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M \\
>>>>>>> +\hline
>>>>>>> +0001h - 7FFFh   & Generic admin cmds    & - \\
>>>>>>> +\hline
>>>>>>> +8000h - FFFFh   & Reserved    & - \\
>>>>>>> +\hline
>>>>>>> +\end{tabular}
>>>>>> See above, I'd suggest to use class/command as virtio-net cvq.
>>>>> see my comment above.
>>> I answered above.
>>>>>>> +
>>>>>>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN 
>>>>>>> CAPS
>>>>>>> IDENTIFY command}
>>>>>>> +
>>>>>>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific 
>>>>>>> data
>>>>>>> set by the driver.
>>>>>>> +This command upon success, returns a data buffer that describes
>>>>>>> information about the supported
>>>>>>> +admin capabilities by the device. This information is of form:
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_caps_identify_result {
>>>>>>> +       /*
>>>>>>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>> +        * ....
>>>>>>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>> +        */
>>>>>>> +       u8 caps[8192];
>>>>>>> +};
>>>>>> Ok, so I see the identify command. But I wonder if we can do that 
>>>>>> via
>>>>>> the feature bits?
>>>>> It doesn't scale. I tried it. It doesn't work.
>>>> What prevents you from improving the scalability instead of trying to
>>>> revinting a duplicated mechanism?
>>> It was already discussed.
>>>
>>> Spec doesn't scale, too many constrains between command-A,B,C and
>>> feature 44,45, 46 and for transport X, Y, Z.
>> What I meant is, if you feel the current feature negotiation doesn't
>> scale, you can improve it instead of reinventing it partially (e.g no
>> negotiation) .
>
> We're adding an identify command for the admin command set.
>
> did you review it ?


Yes, but what I meant is that we've already had a feature negotiation 
mechanism. We should improve it instead of reinventing another one.

Thanks


>
>>
>> Thanks
>>
>>>> Thanks
>>>>
>>>>> Thanks.
>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>> +\end{lstlisting}
>>>>>>> diff --git a/content.tex b/content.tex
>>>>>>> index 32de668..c524fab 100644
>>>>>>> --- a/content.tex
>>>>>>> +++ b/content.tex
>>>>>>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic 
>>>>>>> Facilities
>>>>>>> of a Virtio Device / Feature B
>>>>>>>     \begin{description}
>>>>>>>     \item[0 to 23] Feature bits for the specific device type
>>>>>>>     -\item[24 to 40] Feature bits reserved for extensions to the 
>>>>>>> queue and
>>>>>>> +\item[24 to 41] Feature bits reserved for extensions to the 
>>>>>>> queue and
>>>>>>>       feature negotiation mechanisms
>>>>>>>     -\item[41 and above] Feature bits reserved for future 
>>>>>>> extensions.
>>>>>>> +\item[42 and above] Feature bits reserved for future extensions.
>>>>>>>     \end{description}
>>>>>>>       \begin{note}
>>>>>>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
>>>>>>> Facilities of a Virtio Device / Expo
>>>>>>>     types. It is RECOMMENDED that devices generate version 4
>>>>>>>     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>>     +\input{admin-virtq.tex}
>>>>>>> +
>>>>>>>     \chapter{General Initialization And Device
>>>>>>> Operation}\label{sec:General Initialization And Device Operation}
>>>>>>>       We start with an overview of device initialization, then 
>>>>>>> expand
>>>>>>> on the
>>>>>>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
>>>>>>> Bits}\label{sec:Reserved Feature Bits}
>>>>>>>       that the driver can reset a queue individually.
>>>>>>>       See \ref{sec:Basic Facilities of a Virtio Device / 
>>>>>>> Virtqueues /
>>>>>>> Virtqueue Reset}.
>>>>>>>     +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>>>>>>> +  the device supports administration virtqueue negotiation.
>>>>>>> +
>>>>>>>     \end{description}
>>>>>>>       \drivernormative{\section}{Reserved Feature Bits}{Reserved
>>>>>>> Feature Bits}
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive:
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0 
>>>>>>
>>>>>> Feedback License:
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0 
>>>>>>
>>>>>> List Guidelines:
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0 
>>>>>>
>>>>>> Committee:
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0 
>>>>>>
>>>>>> Join OASIS:
>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0 
>>>>>>
>>>>>>
>>>> This publicly archived list offers a means to provide input to the
>>>>
>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>
>>>>
>>>>
>>>> In order to verify user consent to the Feedback License terms and
>>>>
>>>> to minimize spam in the list archive, subscription is required
>>>>
>>>> before posting.
>>>>
>>>>
>>>>
>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>
>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>
>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>
>>>> List archive: 
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0
>>>>
>>>> Feedback License: 
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0
>>>>
>>>> List Guidelines: 
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0
>>>>
>>>> Committee: 
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0
>>>>
>>>> Join OASIS: 
>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0
>>>>
>>
>> This publicly archived list offers a means to provide input to the
>>
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>>
>>
>> In order to verify user consent to the Feedback License terms and
>>
>> to minimize spam in the list archive, subscription is required
>>
>> before posting.
>>
>>
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>
>> List help: virtio-comment-help@lists.oasis-open.org
>>
>> List archive: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=I8TmvBEUVIj3BqZfjX9IG3JGchDPYH2fyoWQq4MEElk%3D&amp;reserved=0
>>
>> Feedback License: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=22N0rKTDw%2FzO7LZjAVjU5FXKfAiF9wm%2BoKAqNk5vhg4%3D&amp;reserved=0
>>
>> List Guidelines: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=hVC8t%2FpBFqyJNUJnXw5pUFyr2TIfc7Y8o3Tj%2F3DSkj4%3D&amp;reserved=0
>>
>> Committee: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=AuNIOPqOlcH%2BB3F6cZ7cNYLFazmvRvhugTy%2B4PU4fIM%3D&amp;reserved=0
>>
>> Join OASIS: 
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Cecb99411085446c7d06708d9e20d260e%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637789368821237568%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=W2yLv9d5YcfQ%2F5GXV9uvATbpQdamK4cRCLRlWiFLvmI%3D&amp;reserved=0
>>
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 2/9/2022 4:45 AM, Jason Wang wrote:
>
> 在 2022/1/30 下午6:31, Max Gurtovoy 写道:
>>
>> On 1/28/2022 5:18 AM, Jason Wang wrote:
>>> On Thu, Jan 27, 2022 at 10:47 PM Max Gurtovoy <mgurtovoy@nvidia.com> 
>>> wrote:
>>>>
>>>> On 1/27/2022 3:03 PM, Jason Wang wrote:
>>>>> On Thu, Jan 27, 2022 at 6:25 PM Max Gurtovoy 
>>>>> <mgurtovoy@nvidia.com> wrote:
>>>>>> On 1/27/2022 5:14 AM, Jason Wang wrote:
>>>>>>> 在 2022/1/24 下午5:39, Max Gurtovoy 写道:
>>>>>>>> In one of the many use cases a user wants to manipulate 
>>>>>>>> features and
>>>>>>>> configuration of the virtio devices regardless of the device type
>>>>>>>> (net/block/console). Some of this configuration is generic 
>>>>>>>> enough. i.e
>>>>>>>> Number of MSI-X vectors of a virtio PCI VF device. There is a 
>>>>>>>> need to do
>>>>>>>> such features query and manipulation by its parent PCI PF.
>>>>>>>>
>>>>>>>> Currently virtio specification defines control virtqueue to 
>>>>>>>> manipulate
>>>>>>>> features and configuration of the device it operates on. However,
>>>>>>>> control virtqueue commands are device type specific, which 
>>>>>>>> makes it very
>>>>>>>> difficult to extend for device agnostic commands.
>>>>>>>>
>>>>>>>> To support this requirement in elegant way, this patch 
>>>>>>>> introduces a new
>>>>>>>> admin virtqueue. Admin virtqueue will use the same command 
>>>>>>>> format for
>>>>>>>> all
>>>>>>>> types of virtio devices.
>>>>>>>>
>>>>>>>> Manipulate features via admin virtqueue is asynchronous, 
>>>>>>>> scalable, easy
>>>>>>>> to extend and doesn't require additional and expensive on-die 
>>>>>>>> resources
>>>>>>>> to be allocated for every new feature that will be added in the 
>>>>>>>> future.
>>>>>>>>
>>>>>>>> Subsequent patches make use of this admin virtqueue.
>>>>>>>>
>>>>>>>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>>>>>>>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>>>>>>>> ---
>>>>>>>>     admin-virtq.tex | 89 
>>>>>>>> +++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>>     content.tex     |  9 +++--
>>>>>>>>     2 files changed, 96 insertions(+), 2 deletions(-)
>>>>>>>>     create mode 100644 admin-virtq.tex
>>>>>>>>
>>>>>>>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>>>>>>>> new file mode 100644
>>>>>>>> index 0000000..1a41c22
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/admin-virtq.tex
>>>>>>>> @@ -0,0 +1,89 @@
>>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio
>>>>>>>> Device / Admin Virtqueues}
>>>>>>>> +
>>>>>>>> +Admin virtqueue is used to send administrative commands to 
>>>>>>>> manipulate
>>>>>>>> +various features of the device and/or to manipulate various 
>>>>>>>> features,
>>>>>>>> +if possible, of another device within the same group (e.g. PCI 
>>>>>>>> VFs of
>>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>>> +optionally managed by its parent PCI PF using its admin 
>>>>>>>> virtqueue.).
>>>>>>>> +
>>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>>> +feature bit.
>>>>>>>> +
>>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>>> +
>>>>>>>> +Regardless of device offering VIRTIO_F_IN_ORDER, admin queue 
>>>>>>>> command
>>>>>>>> buffers
>>>>>>>> +are used by the device in out of order manner.
>>>>>>>> +
>>>>>>>> +The Admin command set defines the commands that may be issued 
>>>>>>>> only
>>>>>>>> to the admin
>>>>>>>> +virtqueue. Each virtio device that advertises the 
>>>>>>>> VIRTIO_F_ADMIN_VQ
>>>>>>>> feature, MUST
>>>>>>>> +support all the mandatory admin commands. A device MAY support 
>>>>>>>> also
>>>>>>>> one or more
>>>>>>>> +optional admin commands. All commands are of the following form:
>>>>>>> Do we need a way for advertising the supported optional commands 
>>>>>>> (or
>>>>>>> features bits)? Otherwise dealing with the compatibility will 
>>>>>>> result
>>>>>>> of per command probing.
>>>>>>>
>>>>>>>
>>>>>>>> +
>>>>>>>> +\begin{lstlisting}
>>>>>>>> +struct virtio_admin_cmd {
>>>>>>>> +        /* Device-readable part */
>>>>>>>> +        u16 command;
>>>>>>> Two questions:
>>>>>>>
>>>>>>> 1) It looks to me it's better to have a explicit device_id here 
>>>>>>> other
>>>>>>> than embed it in each command
>>>>>> this was already discussed.
>>>>>>
>>>>>> We agreed that some commands are not referring to different 
>>>>>> device so
>>>>>> no, we don't need to put it in generic structure.
>>>>>>
>>>>>> I'm not against putting such id but we don't have an exact idea 
>>>>>> how will
>>>>>> SF device_id will look like.
>>>>>>
>>>>>> Maybe it will be some 128 bit uuid ? maybe u64 ?
>>>>> How do you know u16 is sufficient for command?
>>>> it's sufficient for VFs and this is the subject of the proposed 
>>>> commands.
>>> So VIRTIO_ADMIN_CAPS_IDENTIFY maps to 3 commands:
>>>
>>> VIRTIO_ADMIN_PCI_SRIOV_ATTRS,
>>> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET,
>>> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET
>>>
>>> Unless you do 1:1 mapping between cap bit and command, you will be
>>> short of commands if you're using u16.
>>
>> 2^16 admin commands are more than enough.
>
>
> You introduce 65536 cap bits via identify. And you use u16 for command.
>
> But you map 3 commands to a single cap bit. You will be short of 
> commands. Isn't it?

No we just don't use all the bits in the identify. Identify has 2^16 
bits to be able to cover all we need.

>
>
>>
>>
>>>
>>>> When you'll add SF to virtio spec we'll create new commands for it 
>>>> with
>>>> SF_id.
>>>>
>>>> Again, this was discussed with Parav in V1 and was close. Please don't
>>>> re-open.
>>> I don't think so, you can reserve a special device id for the PF 
>>> itself.
>>
>> can you propose something concrete ?
>>
>> are you sure by 100% that sf_id is 32 bit ? can the TG ack on that ?
>
>
> I meant for command without and destination. You can reserve a special 
> ID for that.
>
> Regarding the bit, if you think 32bit is not sufficient, let's just 
> use 64bit, or reserving some bytes in the command.

So 64bit size is acked by TG for vdev_id ?


>
>
>>
>>>
>>>>>>    how can we guess ?
>>>>> You know it's you but not me that tries to propose this solution, 
>>>>> right?
>>>> I'm proposing a solution but you ask questions that we answer but
>>>> somehow they are asked again.
>>>>>>> 2) virtio-net cvq use class/command which seems more elegant, 
>>>>>>> e.g all
>>>>>>> commands belongs to MSI could be grouped into the same class
>>>>>> why can't we do it in u16 command opcode ?
>>>>> Where did you see I say you can't use u16?
>>>> you suggest to replace it to class + command.
>>>>
>>>> I don't understand why.
>>>>
>>>> If it's not mandatory, please ack.
>>>>
>>>>
>>>>>> are you open for changes or
>>>>>> should we copy/paste from net cvq ?
>>>>> Net cvq comes first, it's natural to ask why you do things 
>>>>> differently.
>>>> I answered why.
>>>>
>>>> Admin command set has a different purpose.
>>>>
>>>>>> Let's ask differently. Why we need class/command structure ? why 
>>>>>> is it
>>>>>> more elegant ?
>>>>> Don't you see my reply above?
>>>>>
>>>>> "
>>>>> commands belongs to MSI could be grouped into the same class
>>>>> "
>>>> It's doesn't explain why class A + commands 1, 2 is better than 
>>>> opcodes
>>>> 1 + 2 without a class.
>>> Well, if you read how cvq work it's not hard to get the conclusion,
>>>
>>> It's easier to be mapped to a feature bit. You just map a class and 
>>> that's all.
>>>
>>> Otherwise, you need to feature X:  command A,B,C,D ....
>>
>> But I just added one feature bit: the existence of the AQ.
>>
>> in your suggestion I'll need to add for each feature:
>>
>> Feature X can be negotiated only if AQ feature is negotiated.
>>
>> And if feature X will have a small extension so a new feature will be 
>> needed and it will say:
>>
>> Feature Y can be negotiated in feature X and feature AQ are negotiated.
>>
>> or if feature X and feature Y are mutual exclusive then:
>>
>> feature X can't be negotiated if feature Y is negotiated.
>>
>> feature X can be negotiated only if feature AQ is negotiated.
>
>
> This is how virtqueue works for all the existing device. It's as 
> simple as depending the feature to another feature which indicates the 
> existing of a specific type of virtqueue.

It is not simple and not scale.

>
>
>>
>>
>> and this is a small example. We will have more and more Admin 
>> functionality and this will be impossible to scale.
>>
>> A simple identify command solve this.
>>
>>>
>>>>
>>>>>>>> +        u8 command_specific_data[];
>>>>>>>> +
>>>>>>>> +        /* Device-writable part */
>>>>>>>> +        u8 status;
>>>>>>>> +        u8 command_specific_error;
>>>>>>> I wonder the reason why not using a single field for the above two
>>>>>>> fields. (Or any value for separating command specific error out, we
>>>>>>> don't do that for virtio-net cvq).
>>>>>> each command have its own specific errors.
>>>>>>
>>>>>> virtio net cvq is not generic - it's a net device cvqueue.
>>>>> We're talking about the command format, not its semantics, right?
>>>>>
>>>>> It's command format is pretty general.
>>>> in the Admin command set we would like to have better format that will
>>>> allow users/drivers to better understand device error.
>>>>
>>>> It is very common in many HW devices and specs out there.
>>>>
>>>> I hope it doesn't critical for you as a concept.
>>>>
>>>>>> To make it generic we need to separate.
>>>>>>
>>>>>>>> +        u8 command_specific_result[];
>>>>>>>> +};
>>>>>>>> +\end{lstlisting}
>>>>>>>> +
>>>>>>>> +The following table describes the generic Admin status codes:
>>>>>>>> +
>>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>>> +\hline
>>>>>>>> +Opcode & Status & Description \\
>>>>>>>> +\hline \hline
>>>>>>>> +00h   & VIRTIO_ADMIN_STATUS_OK    & successful completion  \\
>>>>>>>> +\hline
>>>>>>>> +01h   & VIRTIO_ADMIN_STATUS_CS_ERR    & command specific 
>>>>>>>> error  \\
>>>>>>>> +\hline
>>>>>>>> +02h   & VIRTIO_ADMIN_STATUS_COMMAND_UNSUPPORTED    & 
>>>>>>>> unsupported or
>>>>>>>> invalid opcode  \\
>>>>>>>> +\hline
>>>>>>>> +\end{tabular}
>>>>>>>> +
>>>>>>>> +The \field{command} and \field{command_specific_data} are
>>>>>>>> +set by the driver, and the device sets the \field{status}, the
>>>>>>>> +\field{command_specific_error} and the 
>>>>>>>> \field{command_specific_result},
>>>>>>>> +if needed.
>>>>>>>> +
>>>>>>>> +The \field{command_specific_error} should be inspected by the 
>>>>>>>> driver
>>>>>>>> only if \field{status} is set to
>>>>>>>> +VIRTIO_ADMIN_STATUS_CS_ERR by the device. In this case, the 
>>>>>>>> content
>>>>>>>> of \field{command_specific_error}
>>>>>>>> +holds the command specific error. If \field{status} is not set to
>>>>>>>> VIRTIO_ADMIN_STATUS_CS_ERR, the
>>>>>>>> +\field{command_specific_error} value is undefined.
>>>>>>>> +
>>>>>>>> +The following table describes the Admin command set:
>>>>>>>> +
>>>>>>>> +\begin{tabular}{|l|l|l|}
>>>>>>>> +\hline
>>>>>>>> +Opcode & Command & M/O \\
>>>>>>>> +\hline \hline
>>>>>>>> +0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M \\
>>>>>>>> +\hline
>>>>>>>> +0001h - 7FFFh   & Generic admin cmds    & - \\
>>>>>>>> +\hline
>>>>>>>> +8000h - FFFFh   & Reserved    & - \\
>>>>>>>> +\hline
>>>>>>>> +\end{tabular}
>>>>>>> See above, I'd suggest to use class/command as virtio-net cvq.
>>>>>> see my comment above.
>>>> I answered above.
>>>>>>>> +
>>>>>>>> +\subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic
>>>>>>>> Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN 
>>>>>>>> CAPS
>>>>>>>> IDENTIFY command}
>>>>>>>> +
>>>>>>>> +The VIRTIO_ADMIN_CAPS_IDENTIFY command has no command specific 
>>>>>>>> data
>>>>>>>> set by the driver.
>>>>>>>> +This command upon success, returns a data buffer that describes
>>>>>>>> information about the supported
>>>>>>>> +admin capabilities by the device. This information is of form:
>>>>>>>> +\begin{lstlisting}
>>>>>>>> +struct virtio_admin_caps_identify_result {
>>>>>>>> +       /*
>>>>>>>> +        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>>> +        * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>>> +        * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>>> +        * ....
>>>>>>>> +        * caps[8191] - Bit 0x0 - Bit 0x7 are reserved
>>>>>>>> +        */
>>>>>>>> +       u8 caps[8192];
>>>>>>>> +};
>>>>>>> Ok, so I see the identify command. But I wonder if we can do 
>>>>>>> that via
>>>>>>> the feature bits?
>>>>>> It doesn't scale. I tried it. It doesn't work.
>>>>> What prevents you from improving the scalability instead of trying to
>>>>> revinting a duplicated mechanism?
>>>> It was already discussed.
>>>>
>>>> Spec doesn't scale, too many constrains between command-A,B,C and
>>>> feature 44,45, 46 and for transport X, Y, Z.
>>> What I meant is, if you feel the current feature negotiation doesn't
>>> scale, you can improve it instead of reinventing it partially (e.g no
>>> negotiation) .
>>
>> We're adding an identify command for the admin command set.
>>
>> did you review it ?
>
>
> Yes, but what I meant is that we've already had a feature negotiation 
> mechanism. We should improve it instead of reinventing another one.

I've explained the scale issue.


>
> Thanks
>
>
>>
>>>
>>> Thanks
>>>
>>>>> Thanks
>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>> +\end{lstlisting}
>>>>>>>> diff --git a/content.tex b/content.tex
>>>>>>>> index 32de668..c524fab 100644
>>>>>>>> --- a/content.tex
>>>>>>>> +++ b/content.tex
>>>>>>>> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic 
>>>>>>>> Facilities
>>>>>>>> of a Virtio Device / Feature B
>>>>>>>>     \begin{description}
>>>>>>>>     \item[0 to 23] Feature bits for the specific device type
>>>>>>>>     -\item[24 to 40] Feature bits reserved for extensions to 
>>>>>>>> the queue and
>>>>>>>> +\item[24 to 41] Feature bits reserved for extensions to the 
>>>>>>>> queue and
>>>>>>>>       feature negotiation mechanisms
>>>>>>>>     -\item[41 and above] Feature bits reserved for future 
>>>>>>>> extensions.
>>>>>>>> +\item[42 and above] Feature bits reserved for future extensions.
>>>>>>>>     \end{description}
>>>>>>>>       \begin{note}
>>>>>>>> @@ -449,6 +449,8 @@ \section{Exporting Objects}\label{sec:Basic
>>>>>>>> Facilities of a Virtio Device / Expo
>>>>>>>>     types. It is RECOMMENDED that devices generate version 4
>>>>>>>>     UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>>>>>>>>     +\input{admin-virtq.tex}
>>>>>>>> +
>>>>>>>>     \chapter{General Initialization And Device
>>>>>>>> Operation}\label{sec:General Initialization And Device Operation}
>>>>>>>>       We start with an overview of device initialization, then 
>>>>>>>> expand
>>>>>>>> on the
>>>>>>>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature
>>>>>>>> Bits}\label{sec:Reserved Feature Bits}
>>>>>>>>       that the driver can reset a queue individually.
>>>>>>>>       See \ref{sec:Basic Facilities of a Virtio Device / 
>>>>>>>> Virtqueues /
>>>>>>>> Virtqueue Reset}.
>>>>>>>>     +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>>>>>>>> +  the device supports administration virtqueue negotiation.
>>>>>>>> +
>>>>>>>>     \end{description}
>>>>>>>>       \drivernormative{\section}{Reserved Feature Bits}{Reserved
>>>>>>>> Feature Bits}
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive:
>>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=pJ5nr6uphZoehnGnA0VSMX5eu2x%2Fgy%2BGp6eYNz6Pi0g%3D&amp;reserved=0 
>>>>>>>
>>>>>>> Feedback License:
>>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Xkaqrd0XlxyYl%2FLmxGS%2BylV7DYZr%2F7AeEQF5ht1EzAI%3D&amp;reserved=0 
>>>>>>>
>>>>>>> List Guidelines:
>>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=WJfH8kSdB3z14hiThyhKSU1m3xmGLRQ0UDvgwW0ITLI%3D&amp;reserved=0 
>>>>>>>
>>>>>>> Committee:
>>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=lzexk08M1jvX3RXvA%2B56wlYc9wPV79JmSNPgvw0F8EU%3D&amp;reserved=0 
>>>>>>>
>>>>>>> Join OASIS:
>>>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=GoUCz7VeXW9JC47bBETePKDTAAyWqlHsjnCOAhhys1k%3D&amp;reserved=0 
>>>>>>>
>>>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>>
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>>
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>>
>>>>> to minimize spam in the list archive, subscription is required
>>>>>
>>>>> before posting.
>>>>>
>>>>>
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>
>>>>> List archive: 
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=pJ5nr6uphZoehnGnA0VSMX5eu2x%2Fgy%2BGp6eYNz6Pi0g%3D&amp;reserved=0
>>>>>
>>>>> Feedback License: 
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Xkaqrd0XlxyYl%2FLmxGS%2BylV7DYZr%2F7AeEQF5ht1EzAI%3D&amp;reserved=0
>>>>>
>>>>> List Guidelines: 
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=WJfH8kSdB3z14hiThyhKSU1m3xmGLRQ0UDvgwW0ITLI%3D&amp;reserved=0
>>>>>
>>>>> Committee: 
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=lzexk08M1jvX3RXvA%2B56wlYc9wPV79JmSNPgvw0F8EU%3D&amp;reserved=0
>>>>>
>>>>> Join OASIS: 
>>>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=GoUCz7VeXW9JC47bBETePKDTAAyWqlHsjnCOAhhys1k%3D&amp;reserved=0
>>>>>
>>>
>>> This publicly archived list offers a means to provide input to the
>>>
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>>
>>>
>>> In order to verify user consent to the Feedback License terms and
>>>
>>> to minimize spam in the list archive, subscription is required
>>>
>>> before posting.
>>>
>>>
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>
>>> List help: virtio-comment-help@lists.oasis-open.org
>>>
>>> List archive: 
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.oasis-open.org%2Farchives%2Fvirtio-comment%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=pJ5nr6uphZoehnGnA0VSMX5eu2x%2Fgy%2BGp6eYNz6Pi0g%3D&amp;reserved=0
>>>
>>> Feedback License: 
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fwho%2Fipr%2Ffeedback_license.pdf&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=Xkaqrd0XlxyYl%2FLmxGS%2BylV7DYZr%2F7AeEQF5ht1EzAI%3D&amp;reserved=0
>>>
>>> List Guidelines: 
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fpolicies-guidelines%2Fmailing-lists&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=WJfH8kSdB3z14hiThyhKSU1m3xmGLRQ0UDvgwW0ITLI%3D&amp;reserved=0
>>>
>>> Committee: 
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fcommittees%2Fvirtio%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=lzexk08M1jvX3RXvA%2B56wlYc9wPV79JmSNPgvw0F8EU%3D&amp;reserved=0
>>>
>>> Join OASIS: 
>>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.oasis-open.org%2Fjoin%2F&amp;data=04%7C01%7Cmgurtovoy%40nvidia.com%7Ceac245eba2584b108f4308d9eb763a85%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637799715297571729%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=GoUCz7VeXW9JC47bBETePKDTAAyWqlHsjnCOAhhys1k%3D&amp;reserved=0
>>>
>>
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:

> In one of the many use cases a user wants to manipulate features and
> configuration of the virtio devices regardless of the device type
> (net/block/console). Some of this configuration is generic enough. i.e
> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> such features query and manipulation by its parent PCI PF.
>
> Currently virtio specification defines control virtqueue to manipulate
> features and configuration of the device it operates on. However,
> control virtqueue commands are device type specific, which makes it very
> difficult to extend for device agnostic commands.
>
> To support this requirement in elegant way, this patch introduces a new
> admin virtqueue. Admin virtqueue will use the same command format for all
> types of virtio devices.
>
> Manipulate features via admin virtqueue is asynchronous, scalable, easy
> to extend and doesn't require additional and expensive on-die resources
> to be allocated for every new feature that will be added in the future.
>
> Subsequent patches make use of this admin virtqueue.
>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>  content.tex     |  9 +++--
>  2 files changed, 96 insertions(+), 2 deletions(-)
>  create mode 100644 admin-virtq.tex
>
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> new file mode 100644
> index 0000000..1a41c22
> --- /dev/null
> +++ b/admin-virtq.tex
> @@ -0,0 +1,89 @@
> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> +
> +Admin virtqueue is used to send administrative commands to manipulate
> +various features of the device and/or to manipulate various features,
> +if possible, of another device within the same group (e.g. PCI VFs of
> +a parent PCI PF device are grouped together. These devices can be
> +optionally managed by its parent PCI PF using its admin virtqueue.).
> +
> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> +feature bit.
> +
> +Admin virtqueue index may vary among different device types.

So, my understanding is:
- any device type may or may not support the admin vq
- if the device type wants to be able to accommodate the admin vq, it
  also needs to specify where it shows up when the feature is negotiated

Do we expect that eventually all device types will need to support the
admin vq (if some use case comes along that will require all devices to
participate, for example?)

Anyway, I'd reword the two sentences above:

"An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is
negotiated. The index of the admin virtqueue is device type specific."

(...)

> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    that the driver can reset a queue individually.
>    See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>  
> +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> +  the device supports administration virtqueue negotiation.

Maybe

"This feature indicates that an administration virtqueue is supported." ?

> +
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> 
> > In one of the many use cases a user wants to manipulate features and
> > configuration of the virtio devices regardless of the device type
> > (net/block/console). Some of this configuration is generic enough. i.e
> > Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> > such features query and manipulation by its parent PCI PF.
> >
> > Currently virtio specification defines control virtqueue to manipulate
> > features and configuration of the device it operates on. However,
> > control virtqueue commands are device type specific, which makes it very
> > difficult to extend for device agnostic commands.
> >
> > To support this requirement in elegant way, this patch introduces a new
> > admin virtqueue. Admin virtqueue will use the same command format for all
> > types of virtio devices.
> >
> > Manipulate features via admin virtqueue is asynchronous, scalable, easy
> > to extend and doesn't require additional and expensive on-die resources
> > to be allocated for every new feature that will be added in the future.
> >
> > Subsequent patches make use of this admin virtqueue.
> >
> > Reviewed-by: Parav Pandit <parav@nvidia.com>
> > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > ---
> >  admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  content.tex     |  9 +++--
> >  2 files changed, 96 insertions(+), 2 deletions(-)
> >  create mode 100644 admin-virtq.tex
> >
> > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > new file mode 100644
> > index 0000000..1a41c22
> > --- /dev/null
> > +++ b/admin-virtq.tex
> > @@ -0,0 +1,89 @@
> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > +
> > +Admin virtqueue is used to send administrative commands to manipulate
> > +various features of the device and/or to manipulate various features,
> > +if possible, of another device within the same group (e.g. PCI VFs of
> > +a parent PCI PF device are grouped together. These devices can be
> > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > +
> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > +feature bit.
> > +
> > +Admin virtqueue index may vary among different device types.
> 
> So, my understanding is:
> - any device type may or may not support the admin vq
> - if the device type wants to be able to accommodate the admin vq, it
>   also needs to specify where it shows up when the feature is negotiated
> 
> Do we expect that eventually all device types will need to support the
> admin vq (if some use case comes along that will require all devices to
> participate, for example?)

I suspect yes. And that's one of the reasons why I'd rather we had a
device independent way to locate the admin queue. There are less
transports than device types.

> Anyway, I'd reword the two sentences above:
> 
> "An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is
> negotiated. The index of the admin virtqueue is device type specific."
> 
> (...)
> 
> > @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >    that the driver can reset a queue individually.
> >    See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> >  
> > +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> > +  the device supports administration virtqueue negotiation.
> 
> Maybe
> 
> "This feature indicates that an administration virtqueue is supported." ?
> 
> > +
> >  \end{description}
> >  
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> > +
>> > +Admin virtqueue is used to send administrative commands to manipulate
>> > +various features of the device and/or to manipulate various features,
>> > +if possible, of another device within the same group (e.g. PCI VFs of
>> > +a parent PCI PF device are grouped together. These devices can be
>> > +optionally managed by its parent PCI PF using its admin virtqueue.).
>> > +
>> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>> > +feature bit.
>> > +
>> > +Admin virtqueue index may vary among different device types.
>> 
>> So, my understanding is:
>> - any device type may or may not support the admin vq
>> - if the device type wants to be able to accommodate the admin vq, it
>>   also needs to specify where it shows up when the feature is negotiated
>> 
>> Do we expect that eventually all device types will need to support the
>> admin vq (if some use case comes along that will require all devices to
>> participate, for example?)
>
> I suspect yes. And that's one of the reasons why I'd rather we had a
> device independent way to locate the admin queue. There are less
> transports than device types.

So, do we want to bite the bullet now and simply say that every device
type has the admin vq as the last vq if the feature is negotiated?
Should be straightforward for the device types that have a fixed number
of vqs, and doable for those that have a variable amount (two device
types are covered by this series anyway.) I think we need to put it with
the device types, as otherwise the numbering of virtqueues could change
in unpredictable ways with the admin vq off/on.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> >> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> >> > +
> >> > +Admin virtqueue is used to send administrative commands to manipulate
> >> > +various features of the device and/or to manipulate various features,
> >> > +if possible, of another device within the same group (e.g. PCI VFs of
> >> > +a parent PCI PF device are grouped together. These devices can be
> >> > +optionally managed by its parent PCI PF using its admin virtqueue.).
> >> > +
> >> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >> > +feature bit.
> >> > +
> >> > +Admin virtqueue index may vary among different device types.
> >> 
> >> So, my understanding is:
> >> - any device type may or may not support the admin vq
> >> - if the device type wants to be able to accommodate the admin vq, it
> >>   also needs to specify where it shows up when the feature is negotiated
> >> 
> >> Do we expect that eventually all device types will need to support the
> >> admin vq (if some use case comes along that will require all devices to
> >> participate, for example?)
> >
> > I suspect yes. And that's one of the reasons why I'd rather we had a
> > device independent way to locate the admin queue. There are less
> > transports than device types.
> 
> So, do we want to bite the bullet now and simply say that every device
> type has the admin vq as the last vq if the feature is negotiated?
> Should be straightforward for the device types that have a fixed number
> of vqs, and doable for those that have a variable amount (two device
> types are covered by this series anyway.) I think we need to put it with
> the device types, as otherwise the numbering of virtqueues could change
> in unpredictable ways with the admin vq off/on.

Well that only works once. The next thing we'll need we won't be able to
make the last one ;) So I am inclined to add a per-transport field that
gives the admin queue number. Another advantage to this approach is that
we can make sure admin queue gets a page by itself (which can be good if
we want to allow access to regular vqs but not to the admin queue to
guest) even if regular vqs share a page. Will help devices use less
memory space.

-- 
MST

[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>> >> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> >> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> >> > +
>> >> > +Admin virtqueue is used to send administrative commands to manipulate
>> >> > +various features of the device and/or to manipulate various features,
>> >> > +if possible, of another device within the same group (e.g. PCI VFs of
>> >> > +a parent PCI PF device are grouped together. These devices can be
>> >> > +optionally managed by its parent PCI PF using its admin virtqueue.).
>> >> > +
>> >> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>> >> > +feature bit.
>> >> > +
>> >> > +Admin virtqueue index may vary among different device types.
>> >> 
>> >> So, my understanding is:
>> >> - any device type may or may not support the admin vq
>> >> - if the device type wants to be able to accommodate the admin vq, it
>> >>   also needs to specify where it shows up when the feature is negotiated
>> >> 
>> >> Do we expect that eventually all device types will need to support the
>> >> admin vq (if some use case comes along that will require all devices to
>> >> participate, for example?)
>> >
>> > I suspect yes. And that's one of the reasons why I'd rather we had a
>> > device independent way to locate the admin queue. There are less
>> > transports than device types.
>> 
>> So, do we want to bite the bullet now and simply say that every device
>> type has the admin vq as the last vq if the feature is negotiated?
>> Should be straightforward for the device types that have a fixed number
>> of vqs, and doable for those that have a variable amount (two device
>> types are covered by this series anyway.) I think we need to put it with
>> the device types, as otherwise the numbering of virtqueues could change
>> in unpredictable ways with the admin vq off/on.
>
> Well that only works once. The next thing we'll need we won't be able to
> make the last one ;) So I am inclined to add a per-transport field that
> gives the admin queue number. Another advantage to this approach is that
> we can make sure admin queue gets a page by itself (which can be good if
> we want to allow access to regular vqs but not to the admin queue to
> guest) even if regular vqs share a page. Will help devices use less
> memory space.

I only meant to make it the last one _now_ :)

But admin-vq-gets-its-own-page is a good point.

Maybe pci gets a new entry in common_cfg, mmio gets a new register, and
ccw gets a new command? (Although I'd prefer to be conservative with new
commands for ccw, maybe it's time to introduce a "get misc config" type
command that can be reused for other things. There's a generic ccw for
that, but adding new stuff to it would require an s390 architecture
change AFAIK, so I had decided not to go down that path for virtio.)


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Fri, Jan 28, 2022 at 05:14:37PM +0100, Cornelia Huck wrote:
> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> >> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> 
> >> > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> >> >> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >> >> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> >> >> > +
> >> >> > +Admin virtqueue is used to send administrative commands to manipulate
> >> >> > +various features of the device and/or to manipulate various features,
> >> >> > +if possible, of another device within the same group (e.g. PCI VFs of
> >> >> > +a parent PCI PF device are grouped together. These devices can be
> >> >> > +optionally managed by its parent PCI PF using its admin virtqueue.).
> >> >> > +
> >> >> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >> >> > +feature bit.
> >> >> > +
> >> >> > +Admin virtqueue index may vary among different device types.
> >> >> 
> >> >> So, my understanding is:
> >> >> - any device type may or may not support the admin vq
> >> >> - if the device type wants to be able to accommodate the admin vq, it
> >> >>   also needs to specify where it shows up when the feature is negotiated
> >> >> 
> >> >> Do we expect that eventually all device types will need to support the
> >> >> admin vq (if some use case comes along that will require all devices to
> >> >> participate, for example?)
> >> >
> >> > I suspect yes. And that's one of the reasons why I'd rather we had a
> >> > device independent way to locate the admin queue. There are less
> >> > transports than device types.
> >> 
> >> So, do we want to bite the bullet now and simply say that every device
> >> type has the admin vq as the last vq if the feature is negotiated?
> >> Should be straightforward for the device types that have a fixed number
> >> of vqs, and doable for those that have a variable amount (two device
> >> types are covered by this series anyway.) I think we need to put it with
> >> the device types, as otherwise the numbering of virtqueues could change
> >> in unpredictable ways with the admin vq off/on.
> >
> > Well that only works once. The next thing we'll need we won't be able to
> > make the last one ;) So I am inclined to add a per-transport field that
> > gives the admin queue number. Another advantage to this approach is that
> > we can make sure admin queue gets a page by itself (which can be good if
> > we want to allow access to regular vqs but not to the admin queue to
> > guest) even if regular vqs share a page. Will help devices use less
> > memory space.
> 
> I only meant to make it the last one _now_ :)
> 
> But admin-vq-gets-its-own-page is a good point.
> 
> Maybe pci gets a new entry in common_cfg, mmio gets a new register, and
> ccw gets a new command?

That's fine.

> (Although I'd prefer to be conservative with new
> commands for ccw, maybe it's time to introduce a "get misc config" type
> command that can be reused for other things. There's a generic ccw for
> that, but adding new stuff to it would require an s390 architecture
> change AFAIK, so I had decided not to go down that path for virtio.)

You mean a generic "get misc config" for all transports? That's fine
by me too. I suspect nvidia guys will need help building
something like this though ...

-- 
MST

[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Fri, Jan 28, 2022 at 05:14:37PM +0100, Cornelia Huck wrote:
>> Maybe pci gets a new entry in common_cfg, mmio gets a new register, and
>> ccw gets a new command?
>
> That's fine.
>
>> (Although I'd prefer to be conservative with new
>> commands for ccw, maybe it's time to introduce a "get misc config" type
>> command that can be reused for other things. There's a generic ccw for
>> that, but adding new stuff to it would require an s390 architecture
>> change AFAIK, so I had decided not to go down that path for virtio.)
>
> You mean a generic "get misc config" for all transports? That's fine
> by me too. I suspect nvidia guys will need help building
> something like this though ...

I was more thinking about ccw only... not sure if we can define
something useful for all transports.

I'm willing to help with defining a ccw mechanism; I realize that this
will be hard for most people.

(Let's get agreement on whether we want to go down that route first.)


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > >> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > >> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > >> > +
> > >> > +Admin virtqueue is used to send administrative commands to manipulate
> > >> > +various features of the device and/or to manipulate various features,
> > >> > +if possible, of another device within the same group (e.g. PCI VFs of
> > >> > +a parent PCI PF device are grouped together. These devices can be
> > >> > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > >> > +
> > >> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > >> > +feature bit.
> > >> > +
> > >> > +Admin virtqueue index may vary among different device types.
> > >>
> > >> So, my understanding is:
> > >> - any device type may or may not support the admin vq
> > >> - if the device type wants to be able to accommodate the admin vq, it
> > >>   also needs to specify where it shows up when the feature is negotiated
> > >>
> > >> Do we expect that eventually all device types will need to support the
> > >> admin vq (if some use case comes along that will require all devices to
> > >> participate, for example?)
> > >
> > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > device independent way to locate the admin queue. There are less
> > > transports than device types.
> >
> > So, do we want to bite the bullet now and simply say that every device
> > type has the admin vq as the last vq if the feature is negotiated?
> > Should be straightforward for the device types that have a fixed number
> > of vqs, and doable for those that have a variable amount (two device
> > types are covered by this series anyway.) I think we need to put it with
> > the device types, as otherwise the numbering of virtqueues could change
> > in unpredictable ways with the admin vq off/on.
>
> Well that only works once. The next thing we'll need we won't be able to
> make the last one ;) So I am inclined to add a per-transport field that
> gives the admin queue number.

Technically, there's no need to use the same namespace for admin
virtqueue if it has a dedicated notification area. If we go this way,
we can simply use 0 as queue index for admin virtqueue.

Thanks

> Another advantage to this approach is that
> we can make sure admin queue gets a page by itself (which can be good if
> we want to allow access to regular vqs but not to the admin queue to
> guest) even if regular vqs share a page. Will help devices use less
> memory space.
>
> --
> MST
>

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/29/2022 5:53 AM, Jason Wang wrote:
> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>
>>>> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>>>>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>> +
>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>> +various features of the device and/or to manipulate various features,
>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>> +
>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>> +feature bit.
>>>>>> +
>>>>>> +Admin virtqueue index may vary among different device types.
>>>>> So, my understanding is:
>>>>> - any device type may or may not support the admin vq
>>>>> - if the device type wants to be able to accommodate the admin vq, it
>>>>>    also needs to specify where it shows up when the feature is negotiated
>>>>>
>>>>> Do we expect that eventually all device types will need to support the
>>>>> admin vq (if some use case comes along that will require all devices to
>>>>> participate, for example?)
>>>> I suspect yes. And that's one of the reasons why I'd rather we had a
>>>> device independent way to locate the admin queue. There are less
>>>> transports than device types.
>>> So, do we want to bite the bullet now and simply say that every device
>>> type has the admin vq as the last vq if the feature is negotiated?
>>> Should be straightforward for the device types that have a fixed number
>>> of vqs, and doable for those that have a variable amount (two device
>>> types are covered by this series anyway.) I think we need to put it with
>>> the device types, as otherwise the numbering of virtqueues could change
>>> in unpredictable ways with the admin vq off/on.
>> Well that only works once. The next thing we'll need we won't be able to
>> make the last one ;) So I am inclined to add a per-transport field that
>> gives the admin queue number.
> Technically, there's no need to use the same namespace for admin
> virtqueue if it has a dedicated notification area. If we go this way,
> we can simply use 0 as queue index for admin virtqueue.

Or we can use index 0xFFFF for admin virtqueue for compatibility.


> Thanks
>
>> Another advantage to this approach is that
>> we can make sure admin queue gets a page by itself (which can be good if
>> we want to allow access to regular vqs but not to the admin queue to
>> guest) even if regular vqs share a page. Will help devices use less
>> memory space.
>>
>> --
>> MST
>>

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> 
> On 1/29/2022 5:53 AM, Jason Wang wrote:
> > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > 
> > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > > +
> > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > > > > > > +various features of the device and/or to manipulate various features,
> > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > > > > > +
> > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > > +feature bit.
> > > > > > > +
> > > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > So, my understanding is:
> > > > > > - any device type may or may not support the admin vq
> > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > > > > >    also needs to specify where it shows up when the feature is negotiated
> > > > > > 
> > > > > > Do we expect that eventually all device types will need to support the
> > > > > > admin vq (if some use case comes along that will require all devices to
> > > > > > participate, for example?)
> > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > > device independent way to locate the admin queue. There are less
> > > > > transports than device types.
> > > > So, do we want to bite the bullet now and simply say that every device
> > > > type has the admin vq as the last vq if the feature is negotiated?
> > > > Should be straightforward for the device types that have a fixed number
> > > > of vqs, and doable for those that have a variable amount (two device
> > > > types are covered by this series anyway.) I think we need to put it with
> > > > the device types, as otherwise the numbering of virtqueues could change
> > > > in unpredictable ways with the admin vq off/on.
> > > Well that only works once. The next thing we'll need we won't be able to
> > > make the last one ;) So I am inclined to add a per-transport field that
> > > gives the admin queue number.
> > Technically, there's no need to use the same namespace for admin
> > virtqueue if it has a dedicated notification area. If we go this way,
> > we can simply use 0 as queue index for admin virtqueue.
> 
> Or we can use index 0xFFFF for admin virtqueue for compatibility.

I think I'd prefer a register with the #. For example we might want
to limit the # of VQs in order to pass extra data with the kick write.


> 
> > Thanks
> > 
> > > Another advantage to this approach is that
> > > we can make sure admin queue gets a page by itself (which can be good if
> > > we want to allow access to regular vqs but not to the admin queue to
> > > guest) even if regular vqs share a page. Will help devices use less
> > > memory space.
> > > 
> > > --
> > > MST
> > > 

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
>> On 1/29/2022 5:53 AM, Jason Wang wrote:
>>> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>>>>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>
>>>>>> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>>>>>>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>>>> +
>>>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>>>> +various features of the device and/or to manipulate various features,
>>>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>>>> +
>>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>>> +feature bit.
>>>>>>>> +
>>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>> So, my understanding is:
>>>>>>> - any device type may or may not support the admin vq
>>>>>>> - if the device type wants to be able to accommodate the admin vq, it
>>>>>>>     also needs to specify where it shows up when the feature is negotiated
>>>>>>>
>>>>>>> Do we expect that eventually all device types will need to support the
>>>>>>> admin vq (if some use case comes along that will require all devices to
>>>>>>> participate, for example?)
>>>>>> I suspect yes. And that's one of the reasons why I'd rather we had a
>>>>>> device independent way to locate the admin queue. There are less
>>>>>> transports than device types.
>>>>> So, do we want to bite the bullet now and simply say that every device
>>>>> type has the admin vq as the last vq if the feature is negotiated?
>>>>> Should be straightforward for the device types that have a fixed number
>>>>> of vqs, and doable for those that have a variable amount (two device
>>>>> types are covered by this series anyway.) I think we need to put it with
>>>>> the device types, as otherwise the numbering of virtqueues could change
>>>>> in unpredictable ways with the admin vq off/on.
>>>> Well that only works once. The next thing we'll need we won't be able to
>>>> make the last one ;) So I am inclined to add a per-transport field that
>>>> gives the admin queue number.
>>> Technically, there's no need to use the same namespace for admin
>>> virtqueue if it has a dedicated notification area. If we go this way,
>>> we can simply use 0 as queue index for admin virtqueue.
>> Or we can use index 0xFFFF for admin virtqueue for compatibility.
> I think I'd prefer a register with the #. For example we might want
> to limit the # of VQs in order to pass extra data with the kick write.

So you are suggesting adding a new cfg_type (#define 
VIRTIO_PCI_CAP_ADMIN_CFG 10) ?

that will look something like:

struct virtio_pci_admin_cfg {

     le32 queue_index; /* read only for the driver */

     le16 queue_size; /* read-write */
     le16 queue_msix_vector; /* read-write */
     le16 queue_enable; /* read-write */
     le16 queue_notify_off; /* read-only for driver */
     le64 queue_desc; /* read-write */
     le64 queue_driver; /* read-write */
     le64 queue_device; /* read-write */
     le16 queue_notify_data; /* read-only for driver */
     le16 queue_reset; /* read-write */

};

instead of re-using the struct virtio_pci_common_cfg ?


or do you prefer extending the struct virtio_pci_common_cfg with "le16 
admin_queue_index; /* read only for the driver */ ?

>
>
>>> Thanks
>>>
>>>> Another advantage to this approach is that
>>>> we can make sure admin queue gets a page by itself (which can be good if
>>>> we want to allow access to regular vqs but not to the admin queue to
>>>> guest) even if regular vqs share a page. Will help devices use less
>>>> memory space.
>>>>
>>>> --
>>>> MST
>>>>

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
> 
> On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> > > On 1/29/2022 5:53 AM, Jason Wang wrote:
> > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > 
> > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > > > > +
> > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > > > > > > > > +various features of the device and/or to manipulate various features,
> > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > > > > > > > +
> > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > > > > +feature bit.
> > > > > > > > > +
> > > > > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > > > So, my understanding is:
> > > > > > > > - any device type may or may not support the admin vq
> > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > > > > > > >     also needs to specify where it shows up when the feature is negotiated
> > > > > > > > 
> > > > > > > > Do we expect that eventually all device types will need to support the
> > > > > > > > admin vq (if some use case comes along that will require all devices to
> > > > > > > > participate, for example?)
> > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > > > > device independent way to locate the admin queue. There are less
> > > > > > > transports than device types.
> > > > > > So, do we want to bite the bullet now and simply say that every device
> > > > > > type has the admin vq as the last vq if the feature is negotiated?
> > > > > > Should be straightforward for the device types that have a fixed number
> > > > > > of vqs, and doable for those that have a variable amount (two device
> > > > > > types are covered by this series anyway.) I think we need to put it with
> > > > > > the device types, as otherwise the numbering of virtqueues could change
> > > > > > in unpredictable ways with the admin vq off/on.
> > > > > Well that only works once. The next thing we'll need we won't be able to
> > > > > make the last one ;) So I am inclined to add a per-transport field that
> > > > > gives the admin queue number.
> > > > Technically, there's no need to use the same namespace for admin
> > > > virtqueue if it has a dedicated notification area. If we go this way,
> > > > we can simply use 0 as queue index for admin virtqueue.
> > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
> > I think I'd prefer a register with the #. For example we might want
> > to limit the # of VQs in order to pass extra data with the kick write.
> 
> So you are suggesting adding a new cfg_type (#define
> VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> 
> that will look something like:
> 
> struct virtio_pci_admin_cfg {
> 
>     le32 queue_index; /* read only for the driver */
> 
>     le16 queue_size; /* read-write */
>     le16 queue_msix_vector; /* read-write */
>     le16 queue_enable; /* read-write */
>     le16 queue_notify_off; /* read-only for driver */
>     le64 queue_desc; /* read-write */
>     le64 queue_driver; /* read-write */
>     le64 queue_device; /* read-write */
>     le16 queue_notify_data; /* read-only for driver */
>     le16 queue_reset; /* read-write */
> 
> };
> 
> instead of re-using the struct virtio_pci_common_cfg ?
> 
> 
> or do you prefer extending the struct virtio_pci_common_cfg with "le16
> admin_queue_index; /* read only for the driver */ ?

The later. Other transports will need this too.


Cornelia has another idea which is that instead of
adding just the admin queue register to all transports,
we instead add a misc_config structure to all
transports. Working basically like device specific config,
but being device independent. For now it will only have
a single le16 admin_queue_index register.

For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG

The point here is that we are making it easier to add
more fields just like admin queue index in the future.



> > 
> > 
> > > > Thanks
> > > > 
> > > > > Another advantage to this approach is that
> > > > > we can make sure admin queue gets a page by itself (which can be good if
> > > > > we want to allow access to regular vqs but not to the admin queue to
> > > > > guest) even if regular vqs share a page. Will help devices use less
> > > > > memory space.
> > > > > 
> > > > > --
> > > > > MST
> > > > > 

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
> On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
>> On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
>>> On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
>>>> On 1/29/2022 5:53 AM, Jason Wang wrote:
>>>>> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>>>>>>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>>>
>>>>>>>> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>>>>>>>>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>>>>>> +
>>>>>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>>>>>> +various features of the device and/or to manipulate various features,
>>>>>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>>>>>> +
>>>>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>>>>> +feature bit.
>>>>>>>>>> +
>>>>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>>>> So, my understanding is:
>>>>>>>>> - any device type may or may not support the admin vq
>>>>>>>>> - if the device type wants to be able to accommodate the admin vq, it
>>>>>>>>>      also needs to specify where it shows up when the feature is negotiated
>>>>>>>>>
>>>>>>>>> Do we expect that eventually all device types will need to support the
>>>>>>>>> admin vq (if some use case comes along that will require all devices to
>>>>>>>>> participate, for example?)
>>>>>>>> I suspect yes. And that's one of the reasons why I'd rather we had a
>>>>>>>> device independent way to locate the admin queue. There are less
>>>>>>>> transports than device types.
>>>>>>> So, do we want to bite the bullet now and simply say that every device
>>>>>>> type has the admin vq as the last vq if the feature is negotiated?
>>>>>>> Should be straightforward for the device types that have a fixed number
>>>>>>> of vqs, and doable for those that have a variable amount (two device
>>>>>>> types are covered by this series anyway.) I think we need to put it with
>>>>>>> the device types, as otherwise the numbering of virtqueues could change
>>>>>>> in unpredictable ways with the admin vq off/on.
>>>>>> Well that only works once. The next thing we'll need we won't be able to
>>>>>> make the last one ;) So I am inclined to add a per-transport field that
>>>>>> gives the admin queue number.
>>>>> Technically, there's no need to use the same namespace for admin
>>>>> virtqueue if it has a dedicated notification area. If we go this way,
>>>>> we can simply use 0 as queue index for admin virtqueue.
>>>> Or we can use index 0xFFFF for admin virtqueue for compatibility.
>>> I think I'd prefer a register with the #. For example we might want
>>> to limit the # of VQs in order to pass extra data with the kick write.
>> So you are suggesting adding a new cfg_type (#define
>> VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
>>
>> that will look something like:
>>
>> struct virtio_pci_admin_cfg {
>>
>>      le32 queue_index; /* read only for the driver */
>>
>>      le16 queue_size; /* read-write */
>>      le16 queue_msix_vector; /* read-write */
>>      le16 queue_enable; /* read-write */
>>      le16 queue_notify_off; /* read-only for driver */
>>      le64 queue_desc; /* read-write */
>>      le64 queue_driver; /* read-write */
>>      le64 queue_device; /* read-write */
>>      le16 queue_notify_data; /* read-only for driver */
>>      le16 queue_reset; /* read-write */
>>
>> };
>>
>> instead of re-using the struct virtio_pci_common_cfg ?
>>
>>
>> or do you prefer extending the struct virtio_pci_common_cfg with "le16
>> admin_queue_index; /* read only for the driver */ ?
> The later. Other transports will need this too.
>
>
> Cornelia has another idea which is that instead of
> adding just the admin queue register to all transports,
> we instead add a misc_config structure to all
> transports. Working basically like device specific config,
> but being device independent. For now it will only have
> a single le16 admin_queue_index register.
>
> For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
>
> The point here is that we are making it easier to add
> more fields just like admin queue index in the future.

OK.

#define VIRTIO_PCI_CAP_MISC_CFG 10

and

struct virtio_pci_misc_cfg {
     le16 admin_queue_index; /* read-only for driver */
};

Is agreed by all for V3 ? instead of the net and blk AQ index definitions.


>
>
>>>
>>>>> Thanks
>>>>>
>>>>>> Another advantage to this approach is that
>>>>>> we can make sure admin queue gets a page by itself (which can be good if
>>>>>> we want to allow access to regular vqs but not to the admin queue to
>>>>>> guest) even if regular vqs share a page. Will help devices use less
>>>>>> memory space.
>>>>>>
>>>>>> --
>>>>>> MST
>>>>>>

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> 
> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
> > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
> > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:
> > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > 
> > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > > > > > > +
> > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > > > > > > > > > > +various features of the device and/or to manipulate various features,
> > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > > > > > > > > > +
> > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > > > > > > +feature bit.
> > > > > > > > > > > +
> > > > > > > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > > > > > So, my understanding is:
> > > > > > > > > > - any device type may or may not support the admin vq
> > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > > > > > > > > >      also needs to specify where it shows up when the feature is negotiated
> > > > > > > > > > 
> > > > > > > > > > Do we expect that eventually all device types will need to support the
> > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> > > > > > > > > > participate, for example?)
> > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > > > > > > device independent way to locate the admin queue. There are less
> > > > > > > > > transports than device types.
> > > > > > > > So, do we want to bite the bullet now and simply say that every device
> > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> > > > > > > > Should be straightforward for the device types that have a fixed number
> > > > > > > > of vqs, and doable for those that have a variable amount (two device
> > > > > > > > types are covered by this series anyway.) I think we need to put it with
> > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> > > > > > > > in unpredictable ways with the admin vq off/on.
> > > > > > > Well that only works once. The next thing we'll need we won't be able to
> > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> > > > > > > gives the admin queue number.
> > > > > > Technically, there's no need to use the same namespace for admin
> > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> > > > > > we can simply use 0 as queue index for admin virtqueue.
> > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
> > > > I think I'd prefer a register with the #. For example we might want
> > > > to limit the # of VQs in order to pass extra data with the kick write.
> > > So you are suggesting adding a new cfg_type (#define
> > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> > > 
> > > that will look something like:
> > > 
> > > struct virtio_pci_admin_cfg {
> > > 
> > >      le32 queue_index; /* read only for the driver */
> > > 
> > >      le16 queue_size; /* read-write */
> > >      le16 queue_msix_vector; /* read-write */
> > >      le16 queue_enable; /* read-write */
> > >      le16 queue_notify_off; /* read-only for driver */
> > >      le64 queue_desc; /* read-write */
> > >      le64 queue_driver; /* read-write */
> > >      le64 queue_device; /* read-write */
> > >      le16 queue_notify_data; /* read-only for driver */
> > >      le16 queue_reset; /* read-write */
> > > 
> > > };
> > > 
> > > instead of re-using the struct virtio_pci_common_cfg ?
> > > 
> > > 
> > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> > > admin_queue_index; /* read only for the driver */ ?
> > The later. Other transports will need this too.
> > 
> > 
> > Cornelia has another idea which is that instead of
> > adding just the admin queue register to all transports,
> > we instead add a misc_config structure to all
> > transports. Working basically like device specific config,
> > but being device independent. For now it will only have
> > a single le16 admin_queue_index register.
> > 
> > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> > 
> > The point here is that we are making it easier to add
> > more fields just like admin queue index in the future.
> 
> OK.
> 
> #define VIRTIO_PCI_CAP_MISC_CFG 10
> 
> and
> 
> struct virtio_pci_misc_cfg {
>     le16 admin_queue_index; /* read-only for driver */
> };
> 
> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.

We need to add it to MMIO and CCW I guess too.

This is Cornelia's idea, we'll need her response.


> 
> > 
> > 
> > > > 
> > > > > > Thanks
> > > > > > 
> > > > > > > Another advantage to this approach is that
> > > > > > > we can make sure admin queue gets a page by itself (which can be good if
> > > > > > > we want to allow access to regular vqs but not to the admin queue to
> > > > > > > guest) even if regular vqs share a page. Will help devices use less
> > > > > > > memory space.
> > > > > > > 
> > > > > > > --
> > > > > > > MST
> > > > > > > 

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/30/2022 5:30 PM, Michael S. Tsirkin wrote:
> On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
>> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
>>> On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
>>>> On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
>>>>> On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
>>>>>> On 1/29/2022 5:53 AM, Jason Wang wrote:
>>>>>>> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>>> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>>>>>>>>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>>>>>>>>>>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>>>>>>>> +
>>>>>>>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>>>>>>>> +various features of the device and/or to manipulate various features,
>>>>>>>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>>>>>>>> +
>>>>>>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>>>>>>> +feature bit.
>>>>>>>>>>>> +
>>>>>>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>>>>>> So, my understanding is:
>>>>>>>>>>> - any device type may or may not support the admin vq
>>>>>>>>>>> - if the device type wants to be able to accommodate the admin vq, it
>>>>>>>>>>>       also needs to specify where it shows up when the feature is negotiated
>>>>>>>>>>>
>>>>>>>>>>> Do we expect that eventually all device types will need to support the
>>>>>>>>>>> admin vq (if some use case comes along that will require all devices to
>>>>>>>>>>> participate, for example?)
>>>>>>>>>> I suspect yes. And that's one of the reasons why I'd rather we had a
>>>>>>>>>> device independent way to locate the admin queue. There are less
>>>>>>>>>> transports than device types.
>>>>>>>>> So, do we want to bite the bullet now and simply say that every device
>>>>>>>>> type has the admin vq as the last vq if the feature is negotiated?
>>>>>>>>> Should be straightforward for the device types that have a fixed number
>>>>>>>>> of vqs, and doable for those that have a variable amount (two device
>>>>>>>>> types are covered by this series anyway.) I think we need to put it with
>>>>>>>>> the device types, as otherwise the numbering of virtqueues could change
>>>>>>>>> in unpredictable ways with the admin vq off/on.
>>>>>>>> Well that only works once. The next thing we'll need we won't be able to
>>>>>>>> make the last one ;) So I am inclined to add a per-transport field that
>>>>>>>> gives the admin queue number.
>>>>>>> Technically, there's no need to use the same namespace for admin
>>>>>>> virtqueue if it has a dedicated notification area. If we go this way,
>>>>>>> we can simply use 0 as queue index for admin virtqueue.
>>>>>> Or we can use index 0xFFFF for admin virtqueue for compatibility.
>>>>> I think I'd prefer a register with the #. For example we might want
>>>>> to limit the # of VQs in order to pass extra data with the kick write.
>>>> So you are suggesting adding a new cfg_type (#define
>>>> VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
>>>>
>>>> that will look something like:
>>>>
>>>> struct virtio_pci_admin_cfg {
>>>>
>>>>       le32 queue_index; /* read only for the driver */
>>>>
>>>>       le16 queue_size; /* read-write */
>>>>       le16 queue_msix_vector; /* read-write */
>>>>       le16 queue_enable; /* read-write */
>>>>       le16 queue_notify_off; /* read-only for driver */
>>>>       le64 queue_desc; /* read-write */
>>>>       le64 queue_driver; /* read-write */
>>>>       le64 queue_device; /* read-write */
>>>>       le16 queue_notify_data; /* read-only for driver */
>>>>       le16 queue_reset; /* read-write */
>>>>
>>>> };
>>>>
>>>> instead of re-using the struct virtio_pci_common_cfg ?
>>>>
>>>>
>>>> or do you prefer extending the struct virtio_pci_common_cfg with "le16
>>>> admin_queue_index; /* read only for the driver */ ?
>>> The later. Other transports will need this too.
>>>
>>>
>>> Cornelia has another idea which is that instead of
>>> adding just the admin queue register to all transports,
>>> we instead add a misc_config structure to all
>>> transports. Working basically like device specific config,
>>> but being device independent. For now it will only have
>>> a single le16 admin_queue_index register.
>>>
>>> For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
>>>
>>> The point here is that we are making it easier to add
>>> more fields just like admin queue index in the future.
>> OK.
>>
>> #define VIRTIO_PCI_CAP_MISC_CFG 10
>>
>> and
>>
>> struct virtio_pci_misc_cfg {
>>      le16 admin_queue_index; /* read-only for driver */
>> };
>>
>> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> We need to add it to MMIO and CCW I guess too.
>
> This is Cornelia's idea, we'll need her response.

Ok, Cornelia please review the above.

For V3 I'll prepare the PCI transport and we can work off list on the 
MMIO and CCW patches for V4.

sounds reasonable ?

>
>
>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>> Another advantage to this approach is that
>>>>>>>> we can make sure admin queue gets a page by itself (which can be good if
>>>>>>>> we want to allow access to regular vqs but not to the admin queue to
>>>>>>>> guest) even if regular vqs share a page. Will help devices use less
>>>>>>>> memory space.
>>>>>>>>
>>>>>>>> --
>>>>>>>> MST
>>>>>>>>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sun, Jan 30, 2022 at 08:23:30PM +0200, Max Gurtovoy wrote:
> 
> On 1/30/2022 5:30 PM, Michael S. Tsirkin wrote:
> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> > > On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
> > > > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
> > > > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> > > > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> > > > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:
> > > > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > 
> > > > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > > > > > > > > > > > > +various features of the device and/or to manipulate various features,
> > > > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > > > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > > > > > > > > +feature bit.
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > > > > > > > So, my understanding is:
> > > > > > > > > > > > - any device type may or may not support the admin vq
> > > > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > > > > > > > > > > >       also needs to specify where it shows up when the feature is negotiated
> > > > > > > > > > > > 
> > > > > > > > > > > > Do we expect that eventually all device types will need to support the
> > > > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> > > > > > > > > > > > participate, for example?)
> > > > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > > > > > > > > device independent way to locate the admin queue. There are less
> > > > > > > > > > > transports than device types.
> > > > > > > > > > So, do we want to bite the bullet now and simply say that every device
> > > > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> > > > > > > > > > Should be straightforward for the device types that have a fixed number
> > > > > > > > > > of vqs, and doable for those that have a variable amount (two device
> > > > > > > > > > types are covered by this series anyway.) I think we need to put it with
> > > > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> > > > > > > > > > in unpredictable ways with the admin vq off/on.
> > > > > > > > > Well that only works once. The next thing we'll need we won't be able to
> > > > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> > > > > > > > > gives the admin queue number.
> > > > > > > > Technically, there's no need to use the same namespace for admin
> > > > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> > > > > > > > we can simply use 0 as queue index for admin virtqueue.
> > > > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
> > > > > > I think I'd prefer a register with the #. For example we might want
> > > > > > to limit the # of VQs in order to pass extra data with the kick write.
> > > > > So you are suggesting adding a new cfg_type (#define
> > > > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> > > > > 
> > > > > that will look something like:
> > > > > 
> > > > > struct virtio_pci_admin_cfg {
> > > > > 
> > > > >       le32 queue_index; /* read only for the driver */
> > > > > 
> > > > >       le16 queue_size; /* read-write */
> > > > >       le16 queue_msix_vector; /* read-write */
> > > > >       le16 queue_enable; /* read-write */
> > > > >       le16 queue_notify_off; /* read-only for driver */
> > > > >       le64 queue_desc; /* read-write */
> > > > >       le64 queue_driver; /* read-write */
> > > > >       le64 queue_device; /* read-write */
> > > > >       le16 queue_notify_data; /* read-only for driver */
> > > > >       le16 queue_reset; /* read-write */
> > > > > 
> > > > > };
> > > > > 
> > > > > instead of re-using the struct virtio_pci_common_cfg ?
> > > > > 
> > > > > 
> > > > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> > > > > admin_queue_index; /* read only for the driver */ ?
> > > > The later. Other transports will need this too.
> > > > 
> > > > 
> > > > Cornelia has another idea which is that instead of
> > > > adding just the admin queue register to all transports,
> > > > we instead add a misc_config structure to all
> > > > transports. Working basically like device specific config,
> > > > but being device independent. For now it will only have
> > > > a single le16 admin_queue_index register.
> > > > 
> > > > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> > > > 
> > > > The point here is that we are making it easier to add
> > > > more fields just like admin queue index in the future.
> > > OK.
> > > 
> > > #define VIRTIO_PCI_CAP_MISC_CFG 10
> > > 
> > > and
> > > 
> > > struct virtio_pci_misc_cfg {
> > >      le16 admin_queue_index; /* read-only for driver */
> > > };
> > > 
> > > Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> > We need to add it to MMIO and CCW I guess too.
> > 
> > This is Cornelia's idea, we'll need her response.
> 
> Ok, Cornelia please review the above.
> 
> For V3 I'll prepare the PCI transport and we can work off list on the MMIO
> and CCW patches for V4.
> 
> sounds reasonable ?


Yes except pls work on list.



> > 
> > 
> > > > 
> > > > > > > > Thanks
> > > > > > > > 
> > > > > > > > > Another advantage to this approach is that
> > > > > > > > > we can make sure admin queue gets a page by itself (which can be good if
> > > > > > > > > we want to allow access to regular vqs but not to the admin queue to
> > > > > > > > > guest) even if regular vqs share a page. Will help devices use less
> > > > > > > > > memory space.
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > MST
> > > > > > > > > 

[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
>> 
>> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
>> > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
>> > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
>> > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
>> > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:
>> > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>> > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>> > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > > > > > > > 
>> > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>> > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>> > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> > > > > > > > > > > +
>> > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
>> > > > > > > > > > > +various features of the device and/or to manipulate various features,
>> > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
>> > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
>> > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
>> > > > > > > > > > > +
>> > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>> > > > > > > > > > > +feature bit.
>> > > > > > > > > > > +
>> > > > > > > > > > > +Admin virtqueue index may vary among different device types.
>> > > > > > > > > > So, my understanding is:
>> > > > > > > > > > - any device type may or may not support the admin vq
>> > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
>> > > > > > > > > >      also needs to specify where it shows up when the feature is negotiated
>> > > > > > > > > > 
>> > > > > > > > > > Do we expect that eventually all device types will need to support the
>> > > > > > > > > > admin vq (if some use case comes along that will require all devices to
>> > > > > > > > > > participate, for example?)
>> > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
>> > > > > > > > > device independent way to locate the admin queue. There are less
>> > > > > > > > > transports than device types.
>> > > > > > > > So, do we want to bite the bullet now and simply say that every device
>> > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
>> > > > > > > > Should be straightforward for the device types that have a fixed number
>> > > > > > > > of vqs, and doable for those that have a variable amount (two device
>> > > > > > > > types are covered by this series anyway.) I think we need to put it with
>> > > > > > > > the device types, as otherwise the numbering of virtqueues could change
>> > > > > > > > in unpredictable ways with the admin vq off/on.
>> > > > > > > Well that only works once. The next thing we'll need we won't be able to
>> > > > > > > make the last one ;) So I am inclined to add a per-transport field that
>> > > > > > > gives the admin queue number.
>> > > > > > Technically, there's no need to use the same namespace for admin
>> > > > > > virtqueue if it has a dedicated notification area. If we go this way,
>> > > > > > we can simply use 0 as queue index for admin virtqueue.
>> > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
>> > > > I think I'd prefer a register with the #. For example we might want
>> > > > to limit the # of VQs in order to pass extra data with the kick write.
>> > > So you are suggesting adding a new cfg_type (#define
>> > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
>> > > 
>> > > that will look something like:
>> > > 
>> > > struct virtio_pci_admin_cfg {
>> > > 
>> > >      le32 queue_index; /* read only for the driver */
>> > > 
>> > >      le16 queue_size; /* read-write */
>> > >      le16 queue_msix_vector; /* read-write */
>> > >      le16 queue_enable; /* read-write */
>> > >      le16 queue_notify_off; /* read-only for driver */
>> > >      le64 queue_desc; /* read-write */
>> > >      le64 queue_driver; /* read-write */
>> > >      le64 queue_device; /* read-write */
>> > >      le16 queue_notify_data; /* read-only for driver */
>> > >      le16 queue_reset; /* read-write */
>> > > 
>> > > };
>> > > 
>> > > instead of re-using the struct virtio_pci_common_cfg ?
>> > > 
>> > > 
>> > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
>> > > admin_queue_index; /* read only for the driver */ ?
>> > The later. Other transports will need this too.
>> > 
>> > 
>> > Cornelia has another idea which is that instead of
>> > adding just the admin queue register to all transports,
>> > we instead add a misc_config structure to all
>> > transports. Working basically like device specific config,
>> > but being device independent. For now it will only have
>> > a single le16 admin_queue_index register.
>> > 
>> > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
>> > 
>> > The point here is that we are making it easier to add
>> > more fields just like admin queue index in the future.
>> 
>> OK.
>> 
>> #define VIRTIO_PCI_CAP_MISC_CFG 10
>> 
>> and
>> 
>> struct virtio_pci_misc_cfg {
>>     le16 admin_queue_index; /* read-only for driver */
>> };
>> 
>> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
>
> We need to add it to MMIO and CCW I guess too.

That seems ok for pci.

For ccw, I'd do something like

#define CCW_CMD_READ_MISC_CONF 0x82

struct virtio_misc_conf {
       be16 admin_queue_index;
};

bound to revision 3, which gets a payload data containing the length of
this structure (for future expansions).

Halil, do you think that would work?

For mmio, I'd need to think a bit more. Any mmio experts around?


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:
> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> >> 
> >> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
> >> > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
> >> > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> >> > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> >> > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:
> >> > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >> > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> >> > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > > > > 
> >> > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> >> > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> >> > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> >> > > > > > > > > > > +various features of the device and/or to manipulate various features,
> >> > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> >> > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> >> > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >> > > > > > > > > > > +feature bit.
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Admin virtqueue index may vary among different device types.
> >> > > > > > > > > > So, my understanding is:
> >> > > > > > > > > > - any device type may or may not support the admin vq
> >> > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> >> > > > > > > > > >      also needs to specify where it shows up when the feature is negotiated
> >> > > > > > > > > > 
> >> > > > > > > > > > Do we expect that eventually all device types will need to support the
> >> > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> >> > > > > > > > > > participate, for example?)
> >> > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> >> > > > > > > > > device independent way to locate the admin queue. There are less
> >> > > > > > > > > transports than device types.
> >> > > > > > > > So, do we want to bite the bullet now and simply say that every device
> >> > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> >> > > > > > > > Should be straightforward for the device types that have a fixed number
> >> > > > > > > > of vqs, and doable for those that have a variable amount (two device
> >> > > > > > > > types are covered by this series anyway.) I think we need to put it with
> >> > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> >> > > > > > > > in unpredictable ways with the admin vq off/on.
> >> > > > > > > Well that only works once. The next thing we'll need we won't be able to
> >> > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> >> > > > > > > gives the admin queue number.
> >> > > > > > Technically, there's no need to use the same namespace for admin
> >> > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> >> > > > > > we can simply use 0 as queue index for admin virtqueue.
> >> > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
> >> > > > I think I'd prefer a register with the #. For example we might want
> >> > > > to limit the # of VQs in order to pass extra data with the kick write.
> >> > > So you are suggesting adding a new cfg_type (#define
> >> > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> >> > > 
> >> > > that will look something like:
> >> > > 
> >> > > struct virtio_pci_admin_cfg {
> >> > > 
> >> > >      le32 queue_index; /* read only for the driver */
> >> > > 
> >> > >      le16 queue_size; /* read-write */
> >> > >      le16 queue_msix_vector; /* read-write */
> >> > >      le16 queue_enable; /* read-write */
> >> > >      le16 queue_notify_off; /* read-only for driver */
> >> > >      le64 queue_desc; /* read-write */
> >> > >      le64 queue_driver; /* read-write */
> >> > >      le64 queue_device; /* read-write */
> >> > >      le16 queue_notify_data; /* read-only for driver */
> >> > >      le16 queue_reset; /* read-write */
> >> > > 
> >> > > };
> >> > > 
> >> > > instead of re-using the struct virtio_pci_common_cfg ?
> >> > > 
> >> > > 
> >> > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> >> > > admin_queue_index; /* read only for the driver */ ?
> >> > The later. Other transports will need this too.
> >> > 
> >> > 
> >> > Cornelia has another idea which is that instead of
> >> > adding just the admin queue register to all transports,
> >> > we instead add a misc_config structure to all
> >> > transports. Working basically like device specific config,
> >> > but being device independent. For now it will only have
> >> > a single le16 admin_queue_index register.
> >> > 
> >> > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> >> > 
> >> > The point here is that we are making it easier to add
> >> > more fields just like admin queue index in the future.
> >> 
> >> OK.
> >> 
> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> 
> >> and
> >> 
> >> struct virtio_pci_misc_cfg {
> >>     le16 admin_queue_index; /* read-only for driver */
> >> };
> >> 
> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> >
> > We need to add it to MMIO and CCW I guess too.
> 
> That seems ok for pci.
> 
> For ccw, I'd do something like
> 
> #define CCW_CMD_READ_MISC_CONF 0x82
> 
> struct virtio_misc_conf {
>        be16 admin_queue_index;
> };
> 
> bound to revision 3, which gets a payload data containing the length of
> this structure (for future expansions).
> 
> Halil, do you think that would work?
> 
> For mmio, I'd need to think a bit more. Any mmio experts around?

Not an expert but I think we can rely on a feature
bit to be acked since admin vq is only needed
after feature negotiation is complete.

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:
>> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
>> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
>> >> 
>> >> and
>> >> 
>> >> struct virtio_pci_misc_cfg {
>> >>     le16 admin_queue_index; /* read-only for driver */
>> >> };
>> >> 
>> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
>> >
>> > We need to add it to MMIO and CCW I guess too.
>> 
>> That seems ok for pci.
>> 
>> For ccw, I'd do something like
>> 
>> #define CCW_CMD_READ_MISC_CONF 0x82
>> 
>> struct virtio_misc_conf {
>>        be16 admin_queue_index;
>> };
>> 
>> bound to revision 3, which gets a payload data containing the length of
>> this structure (for future expansions).
>> 
>> Halil, do you think that would work?
>> 
>> For mmio, I'd need to think a bit more. Any mmio experts around?
>
> Not an expert but I think we can rely on a feature
> bit to be acked since admin vq is only needed
> after feature negotiation is complete.

You mean a register that is valid conditionally? I don't see an easy way
to add some kind of "misc" interface for mmio, unlike for the other
transports.

So something like:

AdminQueueIndex/0x0c4/R
If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
returns the queue index of the administration virtqueue.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
> On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:
> >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> 
> >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> >> 
> >> >> and
> >> >> 
> >> >> struct virtio_pci_misc_cfg {
> >> >>     le16 admin_queue_index; /* read-only for driver */
> >> >> };
> >> >> 
> >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> >> >
> >> > We need to add it to MMIO and CCW I guess too.
> >> 
> >> That seems ok for pci.
> >> 
> >> For ccw, I'd do something like
> >> 
> >> #define CCW_CMD_READ_MISC_CONF 0x82
> >> 
> >> struct virtio_misc_conf {
> >>        be16 admin_queue_index;
> >> };
> >> 
> >> bound to revision 3, which gets a payload data containing the length of
> >> this structure (for future expansions).
> >> 
> >> Halil, do you think that would work?
> >> 
> >> For mmio, I'd need to think a bit more. Any mmio experts around?
> >
> > Not an expert but I think we can rely on a feature
> > bit to be acked since admin vq is only needed
> > after feature negotiation is complete.
> 
> You mean a register that is valid conditionally? I don't see an easy way
> to add some kind of "misc" interface for mmio, unlike for the other
> transports.
> 
> So something like:
> 
> AdminQueueIndex/0x0c4/R
> If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> returns the queue index of the administration virtqueue.

No, I mean a register that switches 100+ between device specific
and misc space.

-- 
MST

[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
>> On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> 
>> > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:
>> >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> 
>> >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
>> >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
>> >> >> 
>> >> >> and
>> >> >> 
>> >> >> struct virtio_pci_misc_cfg {
>> >> >>     le16 admin_queue_index; /* read-only for driver */
>> >> >> };
>> >> >> 
>> >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
>> >> >
>> >> > We need to add it to MMIO and CCW I guess too.
>> >> 
>> >> That seems ok for pci.
>> >> 
>> >> For ccw, I'd do something like
>> >> 
>> >> #define CCW_CMD_READ_MISC_CONF 0x82
>> >> 
>> >> struct virtio_misc_conf {
>> >>        be16 admin_queue_index;
>> >> };
>> >> 
>> >> bound to revision 3, which gets a payload data containing the length of
>> >> this structure (for future expansions).
>> >> 
>> >> Halil, do you think that would work?
>> >> 
>> >> For mmio, I'd need to think a bit more. Any mmio experts around?
>> >
>> > Not an expert but I think we can rely on a feature
>> > bit to be acked since admin vq is only needed
>> > after feature negotiation is complete.
>> 
>> You mean a register that is valid conditionally? I don't see an easy way
>> to add some kind of "misc" interface for mmio, unlike for the other
>> transports.
>> 
>> So something like:
>> 
>> AdminQueueIndex/0x0c4/R
>> If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
>> returns the queue index of the administration virtqueue.
>
> No, I mean a register that switches 100+ between device specific
> and misc space.

Do you want to make that dependent on an extra feature bit, then? We
certainly want to make that facility available if the admin vq is not
negotiated but another future feature using misc is. (We probably don't
want to bump the device version.)


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 31, 2022 at 04:48:10PM +0100, Cornelia Huck wrote:
> On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
> >> On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> 
> >> > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:
> >> >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> 
> >> >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> >> >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> >> >> 
> >> >> >> and
> >> >> >> 
> >> >> >> struct virtio_pci_misc_cfg {
> >> >> >>     le16 admin_queue_index; /* read-only for driver */
> >> >> >> };
> >> >> >> 
> >> >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> >> >> >
> >> >> > We need to add it to MMIO and CCW I guess too.
> >> >> 
> >> >> That seems ok for pci.
> >> >> 
> >> >> For ccw, I'd do something like
> >> >> 
> >> >> #define CCW_CMD_READ_MISC_CONF 0x82
> >> >> 
> >> >> struct virtio_misc_conf {
> >> >>        be16 admin_queue_index;
> >> >> };
> >> >> 
> >> >> bound to revision 3, which gets a payload data containing the length of
> >> >> this structure (for future expansions).
> >> >> 
> >> >> Halil, do you think that would work?
> >> >> 
> >> >> For mmio, I'd need to think a bit more. Any mmio experts around?
> >> >
> >> > Not an expert but I think we can rely on a feature
> >> > bit to be acked since admin vq is only needed
> >> > after feature negotiation is complete.
> >> 
> >> You mean a register that is valid conditionally? I don't see an easy way
> >> to add some kind of "misc" interface for mmio, unlike for the other
> >> transports.
> >> 
> >> So something like:
> >> 
> >> AdminQueueIndex/0x0c4/R
> >> If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> >> returns the queue index of the administration virtqueue.
> >
> > No, I mean a register that switches 100+ between device specific
> > and misc space.
> 
> Do you want to make that dependent on an extra feature bit, then? We
> certainly want to make that facility available if the admin vq is not
> negotiated but another future feature using misc is. (We probably don't
> want to bump the device version.)

I guess we could do that, yes. It's unfortunate that value of unused
registers is undefined for MMIO.

-- 
MST

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Halil Pasic]

On Mon, 31 Jan 2022 09:52:54 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >   
> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:  
> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>   
> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:  
> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> > >> >> 
> > >> >> and
> > >> >> 
> > >> >> struct virtio_pci_misc_cfg {
> > >> >>     le16 admin_queue_index; /* read-only for driver */
> > >> >> };
> > >> >> 
> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.  
> > >> >
> > >> > We need to add it to MMIO and CCW I guess too.  
> > >> 
> > >> That seems ok for pci.
> > >> 
> > >> For ccw, I'd do something like
> > >> 
> > >> #define CCW_CMD_READ_MISC_CONF 0x82
> > >> 
> > >> struct virtio_misc_conf {
> > >>        be16 admin_queue_index;
> > >> };
> > >> 
> > >> bound to revision 3, which gets a payload data containing the length of
> > >> this structure (for future expansions).
> > >> 
> > >> Halil, do you think that would work?
> > >> 
> > >> For mmio, I'd need to think a bit more. Any mmio experts around?  
> > >
> > > Not an expert but I think we can rely on a feature
> > > bit to be acked since admin vq is only needed
> > > after feature negotiation is complete.  
> > 
> > You mean a register that is valid conditionally? I don't see an easy way
> > to add some kind of "misc" interface for mmio, unlike for the other
> > transports.
> > 
> > So something like:
> > 
> > AdminQueueIndex/0x0c4/R
> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> > returns the queue index of the administration virtqueue.  
> 
> No, I mean a register that switches 100+ between device specific
> and misc space.
> 

Maybe adding a register that tells us where does the "misc config
start" is another option. I don't think we need an open ended
device-config in practice. I have no idea if there are any upper limits
on MMIO address space though. If we are constrained there, the switching
is certainly more efficient. Otherwise, I think having the misc config
somewhere after device specific config is simpler.

BTW I don't really like this "misc" as the name. FWIW it is less
miscellaneous that the device specific config, since it is common
for all devices. I don't have a good name for it, but I would prefer
"common" over "misc"

Regards,
Halil

[virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:

> On Mon, 31 Jan 2022 09:52:54 -0500
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
>> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
>> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >   
>> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:  
>> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> > >>   
>> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:  
>> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
>> > >> >> 
>> > >> >> and
>> > >> >> 
>> > >> >> struct virtio_pci_misc_cfg {
>> > >> >>     le16 admin_queue_index; /* read-only for driver */
>> > >> >> };
>> > >> >> 
>> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.  
>> > >> >
>> > >> > We need to add it to MMIO and CCW I guess too.  
>> > >> 
>> > >> That seems ok for pci.
>> > >> 
>> > >> For ccw, I'd do something like
>> > >> 
>> > >> #define CCW_CMD_READ_MISC_CONF 0x82
>> > >> 
>> > >> struct virtio_misc_conf {
>> > >>        be16 admin_queue_index;
>> > >> };
>> > >> 
>> > >> bound to revision 3, which gets a payload data containing the length of
>> > >> this structure (for future expansions).
>> > >> 
>> > >> Halil, do you think that would work?
>> > >> 
>> > >> For mmio, I'd need to think a bit more. Any mmio experts around?  
>> > >
>> > > Not an expert but I think we can rely on a feature
>> > > bit to be acked since admin vq is only needed
>> > > after feature negotiation is complete.  
>> > 
>> > You mean a register that is valid conditionally? I don't see an easy way
>> > to add some kind of "misc" interface for mmio, unlike for the other
>> > transports.
>> > 
>> > So something like:
>> > 
>> > AdminQueueIndex/0x0c4/R
>> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
>> > returns the queue index of the administration virtqueue.  
>> 
>> No, I mean a register that switches 100+ between device specific
>> and misc space.
>> 
>
> Maybe adding a register that tells us where does the "misc config
> start" is another option. I don't think we need an open ended
> device-config in practice. I have no idea if there are any upper limits
> on MMIO address space though. If we are constrained there, the switching
> is certainly more efficient. Otherwise, I think having the misc config
> somewhere after device specific config is simpler.

I think we first need to agree what the "misc" thing is actually
supposed to be. My idea was that we don't have an unlimited supply of
ccws to use for new features, so introducing one for reading "misc"
configuration would be a way to keep things extensible (it also might
make the config/register space for other transports less cluttered). The
same idea (save on ccws) would apply to the multiplexing "action" ccw I
mentioned in my other mail.

So, for the case here (simply relaying the location of the admin vq), we
don't really need a "misc" mechanism for pci/mmio, but I'd like to
introduce one for ccw. If we agree that it would be useful for pci/mmio
as well, we should introduce it now.

>
> BTW I don't really like this "misc" as the name. FWIW it is less
> miscellaneous that the device specific config, since it is common
> for all devices. I don't have a good name for it, but I would prefer
> "common" over "misc"

I'm not sure whether "common" would be more confusing, as pci already
has a "common" config (with things that are covered in the mmio
registers, and via various ccw commands.) Does anyone have a better idea
for a bikeshed colour?


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 31, 2022 at 06:10:39PM +0100, Cornelia Huck wrote:
> On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
> 
> > On Mon, 31 Jan 2022 09:52:54 -0500
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >
> >> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:
> >> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >   
> >> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:  
> >> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > >>   
> >> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:  
> >> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> > >> >> 
> >> > >> >> and
> >> > >> >> 
> >> > >> >> struct virtio_pci_misc_cfg {
> >> > >> >>     le16 admin_queue_index; /* read-only for driver */
> >> > >> >> };
> >> > >> >> 
> >> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.  
> >> > >> >
> >> > >> > We need to add it to MMIO and CCW I guess too.  
> >> > >> 
> >> > >> That seems ok for pci.
> >> > >> 
> >> > >> For ccw, I'd do something like
> >> > >> 
> >> > >> #define CCW_CMD_READ_MISC_CONF 0x82
> >> > >> 
> >> > >> struct virtio_misc_conf {
> >> > >>        be16 admin_queue_index;
> >> > >> };
> >> > >> 
> >> > >> bound to revision 3, which gets a payload data containing the length of
> >> > >> this structure (for future expansions).
> >> > >> 
> >> > >> Halil, do you think that would work?
> >> > >> 
> >> > >> For mmio, I'd need to think a bit more. Any mmio experts around?  
> >> > >
> >> > > Not an expert but I think we can rely on a feature
> >> > > bit to be acked since admin vq is only needed
> >> > > after feature negotiation is complete.  
> >> > 
> >> > You mean a register that is valid conditionally? I don't see an easy way
> >> > to add some kind of "misc" interface for mmio, unlike for the other
> >> > transports.
> >> > 
> >> > So something like:
> >> > 
> >> > AdminQueueIndex/0x0c4/R
> >> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> >> > returns the queue index of the administration virtqueue.  
> >> 
> >> No, I mean a register that switches 100+ between device specific
> >> and misc space.
> >> 
> >
> > Maybe adding a register that tells us where does the "misc config
> > start" is another option. I don't think we need an open ended
> > device-config in practice. I have no idea if there are any upper limits
> > on MMIO address space though. If we are constrained there, the switching
> > is certainly more efficient. Otherwise, I think having the misc config
> > somewhere after device specific config is simpler.
> 
> I think we first need to agree what the "misc" thing is actually
> supposed to be. My idea was that we don't have an unlimited supply of
> ccws to use for new features, so introducing one for reading "misc"
> configuration would be a way to keep things extensible (it also might
> make the config/register space for other transports less cluttered). The
> same idea (save on ccws) would apply to the multiplexing "action" ccw I
> mentioned in my other mail.
> 
> So, for the case here (simply relaying the location of the admin vq), we
> don't really need a "misc" mechanism for pci/mmio, but I'd like to
> introduce one for ccw. If we agree that it would be useful for pci/mmio
> as well, we should introduce it now.

OK, my idea was that it's a config that is just live device config but
both device and transport independent.  Adding it in the misc structure
would allow defining it in a single place.

> >
> > BTW I don't really like this "misc" as the name. FWIW it is less
> > miscellaneous that the device specific config, since it is common
> > for all devices. I don't have a good name for it, but I would prefer
> > "common" over "misc"
> 
> I'm not sure whether "common" would be more confusing, as pci already
> has a "common" config (with things that are covered in the mmio
> registers, and via various ccw commands.) Does anyone have a better idea
> for a bikeshed colour?

I'd say "shared" maybe but unfortunately pci already gained "shared memory"
so I think it would be confusing.
"transport independent" is too wordy...

-- 
MST

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Halil Pasic]

On Mon, 31 Jan 2022 18:10:39 +0100
Cornelia Huck <cohuck@redhat.com> wrote:

> On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
> 
> > On Mon, 31 Jan 2022 09:52:54 -0500
> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >  
> >> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:  
> >> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >     
> >> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:    
> >> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > >>     
> >> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:    
> >> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> > >> >> 
> >> > >> >> and
> >> > >> >> 
> >> > >> >> struct virtio_pci_misc_cfg {
> >> > >> >>     le16 admin_queue_index; /* read-only for driver */
> >> > >> >> };
> >> > >> >> 
> >> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.    
> >> > >> >
> >> > >> > We need to add it to MMIO and CCW I guess too.    
> >> > >> 
> >> > >> That seems ok for pci.
> >> > >> 
> >> > >> For ccw, I'd do something like
> >> > >> 
> >> > >> #define CCW_CMD_READ_MISC_CONF 0x82
> >> > >> 
> >> > >> struct virtio_misc_conf {
> >> > >>        be16 admin_queue_index;
> >> > >> };
> >> > >> 
> >> > >> bound to revision 3, which gets a payload data containing the length of
> >> > >> this structure (for future expansions).
> >> > >> 
> >> > >> Halil, do you think that would work?
> >> > >> 
> >> > >> For mmio, I'd need to think a bit more. Any mmio experts around?    
> >> > >
> >> > > Not an expert but I think we can rely on a feature
> >> > > bit to be acked since admin vq is only needed
> >> > > after feature negotiation is complete.    
> >> > 
> >> > You mean a register that is valid conditionally? I don't see an easy way
> >> > to add some kind of "misc" interface for mmio, unlike for the other
> >> > transports.
> >> > 
> >> > So something like:
> >> > 
> >> > AdminQueueIndex/0x0c4/R
> >> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> >> > returns the queue index of the administration virtqueue.    
> >> 
> >> No, I mean a register that switches 100+ between device specific
> >> and misc space.
> >>   
> >
> > Maybe adding a register that tells us where does the "misc config
> > start" is another option. I don't think we need an open ended
> > device-config in practice. I have no idea if there are any upper limits
> > on MMIO address space though. If we are constrained there, the switching
> > is certainly more efficient. Otherwise, I think having the misc config
> > somewhere after device specific config is simpler.  
> 
> I think we first need to agree what the "misc" thing is actually
> supposed to be. My idea was that we don't have an unlimited supply of
> ccws to use for new features, so introducing one for reading "misc"
> configuration would be a way to keep things extensible (it also might
> make the config/register space for other transports less cluttered). The
> same idea (save on ccws) would apply to the multiplexing "action" ccw I
> mentioned in my other mail.
> 

I agree with not wasting CCWs.

> So, for the case here (simply relaying the location of the admin vq), we
> don't really need a "misc" mechanism for pci/mmio, but I'd like to
> introduce one for ccw. If we agree that it would be useful for pci/mmio
> as well, we should introduce it now.

Please see Michael's response. My understanding was also that what we
want is something like config space for the virtio protocol stuff. The
current config space is entirely device specific, so if we would want a
common thing in _config space_, like _the index of the administration 
vq_ then each device would have to define it separately, in a device
specific location. Which is not nice.

AFAIU having a this protocol config could be sufficient in the sense
that we probably don't need another misc. My line of thinking is: with
this we basically get a read-write interface for exposing stuff. The
only other thing I can think of is _transport specific fields_. That
is if we needed something that ain't specific to the device, but ain't
common to all virtio (i.e. the virtio protocol). One idea, which would
allow us to remain flexible is to a make this new thing not
only this new _protocol config_ but state in some sort of a header
that _protocol config_ is a given range of addresses within the
space on which what you called MISC_CONF operates on.

This multiplexing "action" ccw sounds like an interesting idea to
explore. Maybe we only need that, and can integrate misc config
or protocol config into that interface.

Do you have a proposal somewhere? I do remember the other email you
mentioned it in, but I don't remember seeing anything akin to an
interface specification.

Regards,
Halil

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Tue, Feb 01 2022, Halil Pasic <pasic@linux.ibm.com> wrote:

> On Mon, 31 Jan 2022 18:10:39 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
>
>> On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
>> 
>> > On Mon, 31 Jan 2022 09:52:54 -0500
>> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >  
>> >> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:  
>> >> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> >     
>> >> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:    
>> >> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> >> > >>     
>> >> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:    
>> >> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
>> >> > >> >> 
>> >> > >> >> and
>> >> > >> >> 
>> >> > >> >> struct virtio_pci_misc_cfg {
>> >> > >> >>     le16 admin_queue_index; /* read-only for driver */
>> >> > >> >> };
>> >> > >> >> 
>> >> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.    
>> >> > >> >
>> >> > >> > We need to add it to MMIO and CCW I guess too.    
>> >> > >> 
>> >> > >> That seems ok for pci.
>> >> > >> 
>> >> > >> For ccw, I'd do something like
>> >> > >> 
>> >> > >> #define CCW_CMD_READ_MISC_CONF 0x82
>> >> > >> 
>> >> > >> struct virtio_misc_conf {
>> >> > >>        be16 admin_queue_index;
>> >> > >> };
>> >> > >> 
>> >> > >> bound to revision 3, which gets a payload data containing the length of
>> >> > >> this structure (for future expansions).
>> >> > >> 
>> >> > >> Halil, do you think that would work?
>> >> > >> 
>> >> > >> For mmio, I'd need to think a bit more. Any mmio experts around?    
>> >> > >
>> >> > > Not an expert but I think we can rely on a feature
>> >> > > bit to be acked since admin vq is only needed
>> >> > > after feature negotiation is complete.    
>> >> > 
>> >> > You mean a register that is valid conditionally? I don't see an easy way
>> >> > to add some kind of "misc" interface for mmio, unlike for the other
>> >> > transports.
>> >> > 
>> >> > So something like:
>> >> > 
>> >> > AdminQueueIndex/0x0c4/R
>> >> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
>> >> > returns the queue index of the administration virtqueue.    
>> >> 
>> >> No, I mean a register that switches 100+ between device specific
>> >> and misc space.
>> >>   
>> >
>> > Maybe adding a register that tells us where does the "misc config
>> > start" is another option. I don't think we need an open ended
>> > device-config in practice. I have no idea if there are any upper limits
>> > on MMIO address space though. If we are constrained there, the switching
>> > is certainly more efficient. Otherwise, I think having the misc config
>> > somewhere after device specific config is simpler.  
>> 
>> I think we first need to agree what the "misc" thing is actually
>> supposed to be. My idea was that we don't have an unlimited supply of
>> ccws to use for new features, so introducing one for reading "misc"
>> configuration would be a way to keep things extensible (it also might
>> make the config/register space for other transports less cluttered). The
>> same idea (save on ccws) would apply to the multiplexing "action" ccw I
>> mentioned in my other mail.
>> 
>
> I agree with not wasting CCWs.
>
>> So, for the case here (simply relaying the location of the admin vq), we
>> don't really need a "misc" mechanism for pci/mmio, but I'd like to
>> introduce one for ccw. If we agree that it would be useful for pci/mmio
>> as well, we should introduce it now.
>
> Please see Michael's response. My understanding was also that what we
> want is something like config space for the virtio protocol stuff. The
> current config space is entirely device specific, so if we would want a
> common thing in _config space_, like _the index of the administration 
> vq_ then each device would have to define it separately, in a device
> specific location. Which is not nice.

Ok, so for ccw, it will be more of a secondary "config space", and we
should reuse all the infrastructure that we already use for our normal
"config space" (put into quotes because it isn't really a config space.)

Maybe we can call this "protocol config space"?

>
> AFAIU having a this protocol config could be sufficient in the sense
> that we probably don't need another misc. My line of thinking is: with
> this we basically get a read-write interface for exposing stuff. The
> only other thing I can think of is _transport specific fields_. That
> is if we needed something that ain't specific to the device, but ain't
> common to all virtio (i.e. the virtio protocol). One idea, which would
> allow us to remain flexible is to a make this new thing not
> only this new _protocol config_ but state in some sort of a header
> that _protocol config_ is a given range of addresses within the
> space on which what you called MISC_CONF operates on.

Yeah, we can make transport config a subset of protocol
config... possibly just a command + address setup, and the transport can
multiplex on that?

>
> This multiplexing "action" ccw sounds like an interesting idea to
> explore. Maybe we only need that, and can integrate misc config
> or protocol config into that interface.

It would probably be better to try and integrate it into the protocol
config, if that's going to be read/write anyway. We can introduce like
queue reset we're still missing on ccw into the transport config and
already test out whether that makes actual sense.

>
> Do you have a proposal somewhere? I do remember the other email you
> mentioned it in, but I don't remember seeing anything akin to an
> interface specification.

No, nothing formal. I was thinking of a ccw with a command/length/data
payload, with successful conclusion of the I/O signifying successful
triggering of the action (or maybe conclusion, depending on the
command.) That might be more effective than the "config space"
read/write dance, but I'm not sure whether it's worth deviating from the
other transports.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Tue, Feb 01, 2022 at 06:01:34PM +0100, Cornelia Huck wrote:
> On Tue, Feb 01 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
> 
> > On Mon, 31 Jan 2022 18:10:39 +0100
> > Cornelia Huck <cohuck@redhat.com> wrote:
> >
> >> On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:
> >> 
> >> > On Mon, 31 Jan 2022 09:52:54 -0500
> >> > "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >  
> >> >> On Mon, Jan 31, 2022 at 03:26:36PM +0100, Cornelia Huck wrote:  
> >> >> > On Mon, Jan 31 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> >     
> >> >> > > On Mon, Jan 31, 2022 at 10:16:43AM +0100, Cornelia Huck wrote:    
> >> >> > >> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> >> > >>     
> >> >> > >> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:    
> >> >> > >> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> >> > >> >> 
> >> >> > >> >> and
> >> >> > >> >> 
> >> >> > >> >> struct virtio_pci_misc_cfg {
> >> >> > >> >>     le16 admin_queue_index; /* read-only for driver */
> >> >> > >> >> };
> >> >> > >> >> 
> >> >> > >> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.    
> >> >> > >> >
> >> >> > >> > We need to add it to MMIO and CCW I guess too.    
> >> >> > >> 
> >> >> > >> That seems ok for pci.
> >> >> > >> 
> >> >> > >> For ccw, I'd do something like
> >> >> > >> 
> >> >> > >> #define CCW_CMD_READ_MISC_CONF 0x82
> >> >> > >> 
> >> >> > >> struct virtio_misc_conf {
> >> >> > >>        be16 admin_queue_index;
> >> >> > >> };
> >> >> > >> 
> >> >> > >> bound to revision 3, which gets a payload data containing the length of
> >> >> > >> this structure (for future expansions).
> >> >> > >> 
> >> >> > >> Halil, do you think that would work?
> >> >> > >> 
> >> >> > >> For mmio, I'd need to think a bit more. Any mmio experts around?    
> >> >> > >
> >> >> > > Not an expert but I think we can rely on a feature
> >> >> > > bit to be acked since admin vq is only needed
> >> >> > > after feature negotiation is complete.    
> >> >> > 
> >> >> > You mean a register that is valid conditionally? I don't see an easy way
> >> >> > to add some kind of "misc" interface for mmio, unlike for the other
> >> >> > transports.
> >> >> > 
> >> >> > So something like:
> >> >> > 
> >> >> > AdminQueueIndex/0x0c4/R
> >> >> > If VIRTIO_F_ADMIN_VQ has been negotiated, reading from this register
> >> >> > returns the queue index of the administration virtqueue.    
> >> >> 
> >> >> No, I mean a register that switches 100+ between device specific
> >> >> and misc space.
> >> >>   
> >> >
> >> > Maybe adding a register that tells us where does the "misc config
> >> > start" is another option. I don't think we need an open ended
> >> > device-config in practice. I have no idea if there are any upper limits
> >> > on MMIO address space though. If we are constrained there, the switching
> >> > is certainly more efficient. Otherwise, I think having the misc config
> >> > somewhere after device specific config is simpler.  
> >> 
> >> I think we first need to agree what the "misc" thing is actually
> >> supposed to be. My idea was that we don't have an unlimited supply of
> >> ccws to use for new features, so introducing one for reading "misc"
> >> configuration would be a way to keep things extensible (it also might
> >> make the config/register space for other transports less cluttered). The
> >> same idea (save on ccws) would apply to the multiplexing "action" ccw I
> >> mentioned in my other mail.
> >> 
> >
> > I agree with not wasting CCWs.
> >
> >> So, for the case here (simply relaying the location of the admin vq), we
> >> don't really need a "misc" mechanism for pci/mmio, but I'd like to
> >> introduce one for ccw. If we agree that it would be useful for pci/mmio
> >> as well, we should introduce it now.
> >
> > Please see Michael's response. My understanding was also that what we
> > want is something like config space for the virtio protocol stuff. The
> > current config space is entirely device specific, so if we would want a
> > common thing in _config space_, like _the index of the administration 
> > vq_ then each device would have to define it separately, in a device
> > specific location. Which is not nice.
> 
> Ok, so for ccw, it will be more of a secondary "config space", and we
> should reuse all the infrastructure that we already use for our normal
> "config space" (put into quotes because it isn't really a config space.)
> 
> Maybe we can call this "protocol config space"?

Well config is not an english word ;)

We don't have many mentions of config space, those that are there
should probably be converted to device configuration.

I am fine with protocol configuration if others like it, though
I'm not 100% sure which "protocol" does this refer to, but there
are instances in the spec where "protocol" means "virtio spec".


> >
> > AFAIU having a this protocol config could be sufficient in the sense
> > that we probably don't need another misc. My line of thinking is: with
> > this we basically get a read-write interface for exposing stuff. The
> > only other thing I can think of is _transport specific fields_. That
> > is if we needed something that ain't specific to the device, but ain't
> > common to all virtio (i.e. the virtio protocol). One idea, which would
> > allow us to remain flexible is to a make this new thing not
> > only this new _protocol config_ but state in some sort of a header
> > that _protocol config_ is a given range of addresses within the
> > space on which what you called MISC_CONF operates on.
> 
> Yeah, we can make transport config a subset of protocol
> config... possibly just a command + address setup, and the transport can
> multiplex on that?

OK but we need size too? I'd like to avoid a ton of exits if possible,
people already complain about pci discovery causing a ton of exits,
an exit per value is possible for pci after all.

> >
> > This multiplexing "action" ccw sounds like an interesting idea to
> > explore. Maybe we only need that, and can integrate misc config
> > or protocol config into that interface.
> 
> It would probably be better to try and integrate it into the protocol
> config, if that's going to be read/write anyway. We can introduce like
> queue reset we're still missing on ccw into the transport config and
> already test out whether that makes actual sense.

The annoying thing is we added queue reset to pci already, ugh.
There are two ways to do it for pci then? Oh well ...


> >
> > Do you have a proposal somewhere? I do remember the other email you
> > mentioned it in, but I don't remember seeing anything akin to an
> > interface specification.
> 
> No, nothing formal. I was thinking of a ccw with a command/length/data
> payload, with successful conclusion of the I/O signifying successful
> triggering of the action (or maybe conclusion, depending on the
> command.) That might be more effective than the "config space"
> read/write dance, but I'm not sure whether it's worth deviating from the
> other transports.

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Halil Pasic]

On Mon, 31 Jan 2022 10:16:43 +0100
Cornelia Huck <cohuck@redhat.com> wrote:

> On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:  
> >> 
> >> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:  
> >> > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:  
> >> > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:  
> >> > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:  
> >> > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:  
> >> > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:  
> >> > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:  
> >> > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> > > > > > > >   
> >> > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:  
> >> > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:  
> >> > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> >> > > > > > > > > > > +various features of the device and/or to manipulate various features,
> >> > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> >> > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> >> > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> >> > > > > > > > > > > +feature bit.
> >> > > > > > > > > > > +
> >> > > > > > > > > > > +Admin virtqueue index may vary among different device types.  
> >> > > > > > > > > > So, my understanding is:
> >> > > > > > > > > > - any device type may or may not support the admin vq
> >> > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> >> > > > > > > > > >      also needs to specify where it shows up when the feature is negotiated
> >> > > > > > > > > > 
> >> > > > > > > > > > Do we expect that eventually all device types will need to support the
> >> > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> >> > > > > > > > > > participate, for example?)  
> >> > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> >> > > > > > > > > device independent way to locate the admin queue. There are less
> >> > > > > > > > > transports than device types.  
> >> > > > > > > > So, do we want to bite the bullet now and simply say that every device
> >> > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> >> > > > > > > > Should be straightforward for the device types that have a fixed number
> >> > > > > > > > of vqs, and doable for those that have a variable amount (two device
> >> > > > > > > > types are covered by this series anyway.) I think we need to put it with
> >> > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> >> > > > > > > > in unpredictable ways with the admin vq off/on.  
> >> > > > > > > Well that only works once. The next thing we'll need we won't be able to
> >> > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> >> > > > > > > gives the admin queue number.  
> >> > > > > > Technically, there's no need to use the same namespace for admin
> >> > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> >> > > > > > we can simply use 0 as queue index for admin virtqueue.  
> >> > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.  
> >> > > > I think I'd prefer a register with the #. For example we might want
> >> > > > to limit the # of VQs in order to pass extra data with the kick write.  
> >> > > So you are suggesting adding a new cfg_type (#define
> >> > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> >> > > 
> >> > > that will look something like:
> >> > > 
> >> > > struct virtio_pci_admin_cfg {
> >> > > 
> >> > >      le32 queue_index; /* read only for the driver */
> >> > > 
> >> > >      le16 queue_size; /* read-write */
> >> > >      le16 queue_msix_vector; /* read-write */
> >> > >      le16 queue_enable; /* read-write */
> >> > >      le16 queue_notify_off; /* read-only for driver */
> >> > >      le64 queue_desc; /* read-write */
> >> > >      le64 queue_driver; /* read-write */
> >> > >      le64 queue_device; /* read-write */
> >> > >      le16 queue_notify_data; /* read-only for driver */
> >> > >      le16 queue_reset; /* read-write */
> >> > > 
> >> > > };
> >> > > 
> >> > > instead of re-using the struct virtio_pci_common_cfg ?
> >> > > 
> >> > > 
> >> > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> >> > > admin_queue_index; /* read only for the driver */ ?  
> >> > The later. Other transports will need this too.
> >> > 
> >> > 
> >> > Cornelia has another idea which is that instead of
> >> > adding just the admin queue register to all transports,
> >> > we instead add a misc_config structure to all
> >> > transports. Working basically like device specific config,
> >> > but being device independent. For now it will only have
> >> > a single le16 admin_queue_index register.
> >> > 
> >> > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> >> > 
> >> > The point here is that we are making it easier to add
> >> > more fields just like admin queue index in the future.  
> >> 
> >> OK.
> >> 
> >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> >> 
> >> and
> >> 
> >> struct virtio_pci_misc_cfg {
> >>     le16 admin_queue_index; /* read-only for driver */
> >> };
> >> 
> >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.  
> >
> > We need to add it to MMIO and CCW I guess too.  
> 
> That seems ok for pci.
> 
> For ccw, I'd do something like
> 
> #define CCW_CMD_READ_MISC_CONF 0x82
> 
> struct virtio_misc_conf {
>        be16 admin_queue_index;
> };
> 
> bound to revision 3, which gets a payload data containing the length of
> this structure (for future expansions).
> 
> Halil, do you think that would work?


I think so. But I would like to review the actual proposal :)

Some of the questions I have in mind are:
* Are fields in this config protected with feature bits
* Do we ever need to write this config? Adding a writable field to pci
  is easier than to ccw (no need to invent a new ccw)
* Looks like we will have an addressability mismatch with
  ccw compared to pci and mmio: ccw can only read a prefix of this new
  config, while mmio and pci have access to the individual fields
* Is this static stuff only? If not, do we need notifications for config
  change like for the device specific config?

> 
> For mmio, I'd need to think a bit more. Any mmio experts around?
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> 

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Mon, Jan 31, 2022 at 04:47:29PM +0100, Halil Pasic wrote:
> On Mon, 31 Jan 2022 10:16:43 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
> 
> > On Sun, Jan 30 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > 
> > > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:  
> > >> 
> > >> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:  
> > >> > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:  
> > >> > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:  
> > >> > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:  
> > >> > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:  
> > >> > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:  
> > >> > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:  
> > >> > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >> > > > > > > >   
> > >> > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:  
> > >> > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:  
> > >> > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > >> > > > > > > > > > > +
> > >> > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > >> > > > > > > > > > > +various features of the device and/or to manipulate various features,
> > >> > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > >> > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > >> > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > >> > > > > > > > > > > +
> > >> > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > >> > > > > > > > > > > +feature bit.
> > >> > > > > > > > > > > +
> > >> > > > > > > > > > > +Admin virtqueue index may vary among different device types.  
> > >> > > > > > > > > > So, my understanding is:
> > >> > > > > > > > > > - any device type may or may not support the admin vq
> > >> > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > >> > > > > > > > > >      also needs to specify where it shows up when the feature is negotiated
> > >> > > > > > > > > > 
> > >> > > > > > > > > > Do we expect that eventually all device types will need to support the
> > >> > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> > >> > > > > > > > > > participate, for example?)  
> > >> > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > >> > > > > > > > > device independent way to locate the admin queue. There are less
> > >> > > > > > > > > transports than device types.  
> > >> > > > > > > > So, do we want to bite the bullet now and simply say that every device
> > >> > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> > >> > > > > > > > Should be straightforward for the device types that have a fixed number
> > >> > > > > > > > of vqs, and doable for those that have a variable amount (two device
> > >> > > > > > > > types are covered by this series anyway.) I think we need to put it with
> > >> > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> > >> > > > > > > > in unpredictable ways with the admin vq off/on.  
> > >> > > > > > > Well that only works once. The next thing we'll need we won't be able to
> > >> > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> > >> > > > > > > gives the admin queue number.  
> > >> > > > > > Technically, there's no need to use the same namespace for admin
> > >> > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> > >> > > > > > we can simply use 0 as queue index for admin virtqueue.  
> > >> > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.  
> > >> > > > I think I'd prefer a register with the #. For example we might want
> > >> > > > to limit the # of VQs in order to pass extra data with the kick write.  
> > >> > > So you are suggesting adding a new cfg_type (#define
> > >> > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> > >> > > 
> > >> > > that will look something like:
> > >> > > 
> > >> > > struct virtio_pci_admin_cfg {
> > >> > > 
> > >> > >      le32 queue_index; /* read only for the driver */
> > >> > > 
> > >> > >      le16 queue_size; /* read-write */
> > >> > >      le16 queue_msix_vector; /* read-write */
> > >> > >      le16 queue_enable; /* read-write */
> > >> > >      le16 queue_notify_off; /* read-only for driver */
> > >> > >      le64 queue_desc; /* read-write */
> > >> > >      le64 queue_driver; /* read-write */
> > >> > >      le64 queue_device; /* read-write */
> > >> > >      le16 queue_notify_data; /* read-only for driver */
> > >> > >      le16 queue_reset; /* read-write */
> > >> > > 
> > >> > > };
> > >> > > 
> > >> > > instead of re-using the struct virtio_pci_common_cfg ?
> > >> > > 
> > >> > > 
> > >> > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> > >> > > admin_queue_index; /* read only for the driver */ ?  
> > >> > The later. Other transports will need this too.
> > >> > 
> > >> > 
> > >> > Cornelia has another idea which is that instead of
> > >> > adding just the admin queue register to all transports,
> > >> > we instead add a misc_config structure to all
> > >> > transports. Working basically like device specific config,
> > >> > but being device independent. For now it will only have
> > >> > a single le16 admin_queue_index register.
> > >> > 
> > >> > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> > >> > 
> > >> > The point here is that we are making it easier to add
> > >> > more fields just like admin queue index in the future.  
> > >> 
> > >> OK.
> > >> 
> > >> #define VIRTIO_PCI_CAP_MISC_CFG 10
> > >> 
> > >> and
> > >> 
> > >> struct virtio_pci_misc_cfg {
> > >>     le16 admin_queue_index; /* read-only for driver */
> > >> };
> > >> 
> > >> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.  
> > >
> > > We need to add it to MMIO and CCW I guess too.  
> > 
> > That seems ok for pci.
> > 
> > For ccw, I'd do something like
> > 
> > #define CCW_CMD_READ_MISC_CONF 0x82
> > 
> > struct virtio_misc_conf {
> >        be16 admin_queue_index;
> > };
> > 
> > bound to revision 3, which gets a payload data containing the length of
> > this structure (for future expansions).
> > 
> > Halil, do you think that would work?
> 
> 
> I think so. But I would like to review the actual proposal :)
> 
> Some of the questions I have in mind are:
> * Are fields in this config protected with feature bits
> * Do we ever need to write this config? Adding a writable field to pci
>   is easier than to ccw (no need to invent a new ccw)
> * Looks like we will have an addressability mismatch with
>   ccw compared to pci and mmio: ccw can only read a prefix of this new
>   config, while mmio and pci have access to the individual fields
> * Is this static stuff only? If not, do we need notifications for config
>   change like for the device specific config?

I'd just reuse the generic config change if we ever add
something non-static.

Donnu about other questions.

> > 
> > For mmio, I'd need to think a bit more. Any mmio experts around?
> > 
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > 

Re: [virtio-dev] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Cornelia Huck]

On Mon, Jan 31 2022, Halil Pasic <pasic@linux.ibm.com> wrote:

> On Mon, 31 Jan 2022 10:16:43 +0100
> Cornelia Huck <cohuck@redhat.com> wrote:
>> For ccw, I'd do something like
>> 
>> #define CCW_CMD_READ_MISC_CONF 0x82
>> 
>> struct virtio_misc_conf {
>>        be16 admin_queue_index;
>> };
>> 
>> bound to revision 3, which gets a payload data containing the length of
>> this structure (for future expansions).
>> 
>> Halil, do you think that would work?
>
>
> I think so. But I would like to review the actual proposal :)
>
> Some of the questions I have in mind are:
> * Are fields in this config protected with feature bits

I'd say they exist, as long as they are contained in the length provided
via the revision data. Whether the data in the field is valid depends on
feature bits.

> * Do we ever need to write this config? Adding a writable field to pci
>   is easier than to ccw (no need to invent a new ccw)

If we need that, we need to define a companion write command. If we
already think we might need to write some fields in the future, we could
introduce that command already right now.

> * Looks like we will have an addressability mismatch with
>   ccw compared to pci and mmio: ccw can only read a prefix of this new
>   config, while mmio and pci have access to the individual fields

Yeah, that's basically like the config space access, which we're also
modelling with reading up to the requested field. I'm not sure whether
there's a better way to model this.

> * Is this static stuff only? If not, do we need notifications for config
>   change like for the device specific config?

This needs more thought :) To the others reading this thread: do you
think "a mechanism for the device to make various extra configuration
information available to the driver" (i.e. static and read-only) is
enough, or do you think we need more? I think "initiate an action" like
the queue reset is not neccessarily served best by an interface like
this (I had thought about a multiplexing "initiate action" ccw for that
purpose.) The different transports likely have different preferences
here.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Jason Wang]

在 2022/1/30 下午11:30, Michael S. Tsirkin 写道:
> On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
>> On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
>>> On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
>>>> On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
>>>>> On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
>>>>>> On 1/29/2022 5:53 AM, Jason Wang wrote:
>>>>>>> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>>>>> On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
>>>>>>>>> On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>>>>>
>>>>>>>>>> On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
>>>>>>>>>>> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>>>>>>>>>>>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>>>>>>>>>>>> +
>>>>>>>>>>>> +Admin virtqueue is used to send administrative commands to manipulate
>>>>>>>>>>>> +various features of the device and/or to manipulate various features,
>>>>>>>>>>>> +if possible, of another device within the same group (e.g. PCI VFs of
>>>>>>>>>>>> +a parent PCI PF device are grouped together. These devices can be
>>>>>>>>>>>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>>>>>>>>>>>> +
>>>>>>>>>>>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>>>>>>>>>>>> +feature bit.
>>>>>>>>>>>> +
>>>>>>>>>>>> +Admin virtqueue index may vary among different device types.
>>>>>>>>>>> So, my understanding is:
>>>>>>>>>>> - any device type may or may not support the admin vq
>>>>>>>>>>> - if the device type wants to be able to accommodate the admin vq, it
>>>>>>>>>>>       also needs to specify where it shows up when the feature is negotiated
>>>>>>>>>>>
>>>>>>>>>>> Do we expect that eventually all device types will need to support the
>>>>>>>>>>> admin vq (if some use case comes along that will require all devices to
>>>>>>>>>>> participate, for example?)
>>>>>>>>>> I suspect yes. And that's one of the reasons why I'd rather we had a
>>>>>>>>>> device independent way to locate the admin queue. There are less
>>>>>>>>>> transports than device types.
>>>>>>>>> So, do we want to bite the bullet now and simply say that every device
>>>>>>>>> type has the admin vq as the last vq if the feature is negotiated?
>>>>>>>>> Should be straightforward for the device types that have a fixed number
>>>>>>>>> of vqs, and doable for those that have a variable amount (two device
>>>>>>>>> types are covered by this series anyway.) I think we need to put it with
>>>>>>>>> the device types, as otherwise the numbering of virtqueues could change
>>>>>>>>> in unpredictable ways with the admin vq off/on.
>>>>>>>> Well that only works once. The next thing we'll need we won't be able to
>>>>>>>> make the last one ;) So I am inclined to add a per-transport field that
>>>>>>>> gives the admin queue number.
>>>>>>> Technically, there's no need to use the same namespace for admin
>>>>>>> virtqueue if it has a dedicated notification area. If we go this way,
>>>>>>> we can simply use 0 as queue index for admin virtqueue.
>>>>>> Or we can use index 0xFFFF for admin virtqueue for compatibility.
>>>>> I think I'd prefer a register with the #. For example we might want
>>>>> to limit the # of VQs in order to pass extra data with the kick write.
>>>> So you are suggesting adding a new cfg_type (#define
>>>> VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
>>>>
>>>> that will look something like:
>>>>
>>>> struct virtio_pci_admin_cfg {
>>>>
>>>>       le32 queue_index; /* read only for the driver */
>>>>
>>>>       le16 queue_size; /* read-write */
>>>>       le16 queue_msix_vector; /* read-write */
>>>>       le16 queue_enable; /* read-write */
>>>>       le16 queue_notify_off; /* read-only for driver */
>>>>       le64 queue_desc; /* read-write */
>>>>       le64 queue_driver; /* read-write */
>>>>       le64 queue_device; /* read-write */
>>>>       le16 queue_notify_data; /* read-only for driver */
>>>>       le16 queue_reset; /* read-write */
>>>>
>>>> };
>>>>
>>>> instead of re-using the struct virtio_pci_common_cfg ?
>>>>
>>>>
>>>> or do you prefer extending the struct virtio_pci_common_cfg with "le16
>>>> admin_queue_index; /* read only for the driver */ ?
>>> The later. Other transports will need this too.
>>>
>>>
>>> Cornelia has another idea which is that instead of
>>> adding just the admin queue register to all transports,
>>> we instead add a misc_config structure to all
>>> transports. Working basically like device specific config,
>>> but being device independent. For now it will only have
>>> a single le16 admin_queue_index register.
>>>
>>> For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
>>>
>>> The point here is that we are making it easier to add
>>> more fields just like admin queue index in the future.
>> OK.
>>
>> #define VIRTIO_PCI_CAP_MISC_CFG 10
>>
>> and
>>
>> struct virtio_pci_misc_cfg {
>>      le16 admin_queue_index; /* read-only for driver */
>> };
>>
>> Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> We need to add it to MMIO and CCW I guess too.


I wonder how much useful is this.

E.g for PCI we have an equation to calculate the queue notify address, 
if device choose to use dedicated notify for each queue it will probably 
end up with the last queue.

And I think the admin_queue_index should be stable regardless of the 
feature that has been negotiated?

Thanks


>
> This is Cornelia's idea, we'll need her response.
>
>
>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>> Another advantage to this approach is that
>>>>>>>> we can make sure admin queue gets a page by itself (which can be good if
>>>>>>>> we want to allow access to regular vqs but not to the admin queue to
>>>>>>>> guest) even if regular vqs share a page. Will help devices use less
>>>>>>>> memory space.
>>>>>>>>
>>>>>>>> --
>>>>>>>> MST
>>>>>>>>

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Wed, Feb 09, 2022 at 10:27:25AM +0800, Jason Wang wrote:
> 
> 在 2022/1/30 下午11:30, Michael S. Tsirkin 写道:
> > On Sun, Jan 30, 2022 at 05:12:46PM +0200, Max Gurtovoy wrote:
> > > On 1/30/2022 4:41 PM, Michael S. Tsirkin wrote:
> > > > On Sun, Jan 30, 2022 at 11:56:30AM +0200, Max Gurtovoy wrote:
> > > > > On 1/30/2022 11:40 AM, Michael S. Tsirkin wrote:
> > > > > > On Sun, Jan 30, 2022 at 11:13:38AM +0200, Max Gurtovoy wrote:
> > > > > > > On 1/29/2022 5:53 AM, Jason Wang wrote:
> > > > > > > > On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > > > > > > > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > > > > > 
> > > > > > > > > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > > > > > > > > > > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > > > > > > > > > > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Admin virtqueue is used to send administrative commands to manipulate
> > > > > > > > > > > > > +various features of the device and/or to manipulate various features,
> > > > > > > > > > > > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > > > > > > > > > > > +a parent PCI PF device are grouped together. These devices can be
> > > > > > > > > > > > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > > > > > > > > > > > +feature bit.
> > > > > > > > > > > > > +
> > > > > > > > > > > > > +Admin virtqueue index may vary among different device types.
> > > > > > > > > > > > So, my understanding is:
> > > > > > > > > > > > - any device type may or may not support the admin vq
> > > > > > > > > > > > - if the device type wants to be able to accommodate the admin vq, it
> > > > > > > > > > > >       also needs to specify where it shows up when the feature is negotiated
> > > > > > > > > > > > 
> > > > > > > > > > > > Do we expect that eventually all device types will need to support the
> > > > > > > > > > > > admin vq (if some use case comes along that will require all devices to
> > > > > > > > > > > > participate, for example?)
> > > > > > > > > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > > > > > > > > device independent way to locate the admin queue. There are less
> > > > > > > > > > > transports than device types.
> > > > > > > > > > So, do we want to bite the bullet now and simply say that every device
> > > > > > > > > > type has the admin vq as the last vq if the feature is negotiated?
> > > > > > > > > > Should be straightforward for the device types that have a fixed number
> > > > > > > > > > of vqs, and doable for those that have a variable amount (two device
> > > > > > > > > > types are covered by this series anyway.) I think we need to put it with
> > > > > > > > > > the device types, as otherwise the numbering of virtqueues could change
> > > > > > > > > > in unpredictable ways with the admin vq off/on.
> > > > > > > > > Well that only works once. The next thing we'll need we won't be able to
> > > > > > > > > make the last one ;) So I am inclined to add a per-transport field that
> > > > > > > > > gives the admin queue number.
> > > > > > > > Technically, there's no need to use the same namespace for admin
> > > > > > > > virtqueue if it has a dedicated notification area. If we go this way,
> > > > > > > > we can simply use 0 as queue index for admin virtqueue.
> > > > > > > Or we can use index 0xFFFF for admin virtqueue for compatibility.
> > > > > > I think I'd prefer a register with the #. For example we might want
> > > > > > to limit the # of VQs in order to pass extra data with the kick write.
> > > > > So you are suggesting adding a new cfg_type (#define
> > > > > VIRTIO_PCI_CAP_ADMIN_CFG 10) ?
> > > > > 
> > > > > that will look something like:
> > > > > 
> > > > > struct virtio_pci_admin_cfg {
> > > > > 
> > > > >       le32 queue_index; /* read only for the driver */
> > > > > 
> > > > >       le16 queue_size; /* read-write */
> > > > >       le16 queue_msix_vector; /* read-write */
> > > > >       le16 queue_enable; /* read-write */
> > > > >       le16 queue_notify_off; /* read-only for driver */
> > > > >       le64 queue_desc; /* read-write */
> > > > >       le64 queue_driver; /* read-write */
> > > > >       le64 queue_device; /* read-write */
> > > > >       le16 queue_notify_data; /* read-only for driver */
> > > > >       le16 queue_reset; /* read-write */
> > > > > 
> > > > > };
> > > > > 
> > > > > instead of re-using the struct virtio_pci_common_cfg ?
> > > > > 
> > > > > 
> > > > > or do you prefer extending the struct virtio_pci_common_cfg with "le16
> > > > > admin_queue_index; /* read only for the driver */ ?
> > > > The later. Other transports will need this too.
> > > > 
> > > > 
> > > > Cornelia has another idea which is that instead of
> > > > adding just the admin queue register to all transports,
> > > > we instead add a misc_config structure to all
> > > > transports. Working basically like device specific config,
> > > > but being device independent. For now it will only have
> > > > a single le16 admin_queue_index register.
> > > > 
> > > > For PCI we would thus add it with VIRTIO_PCI_CAP_MISC_CFG
> > > > 
> > > > The point here is that we are making it easier to add
> > > > more fields just like admin queue index in the future.
> > > OK.
> > > 
> > > #define VIRTIO_PCI_CAP_MISC_CFG 10
> > > 
> > > and
> > > 
> > > struct virtio_pci_misc_cfg {
> > >      le16 admin_queue_index; /* read-only for driver */
> > > };
> > > 
> > > Is agreed by all for V3 ? instead of the net and blk AQ index definitions.
> > We need to add it to MMIO and CCW I guess too.
> 
> 
> I wonder how much useful is this.
> 
> E.g for PCI we have an equation to calculate the queue notify address, if
> device choose to use dedicated notify for each queue it will probably end up
> with the last queue.
> 
> And I think the admin_queue_index should be stable regardless of the feature
> that has been negotiated?
> 
> Thanks


there's only one last queue though. I wouldn't just use it and hope
we don't need it for anything else.

> 
> > 
> > This is Cornelia's idea, we'll need her response.
> > 
> > 
> > > > 
> > > > > > > > Thanks
> > > > > > > > 
> > > > > > > > > Another advantage to this approach is that
> > > > > > > > > we can make sure admin queue gets a page by itself (which can be good if
> > > > > > > > > we want to allow access to regular vqs but not to the admin queue to
> > > > > > > > > guest) even if regular vqs share a page. Will help devices use less
> > > > > > > > > memory space.
> > > > > > > > > 
> > > > > > > > > --
> > > > > > > > > MST
> > > > > > > > > 

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sat, Jan 29, 2022 at 11:53:06AM +0800, Jason Wang wrote:
> On Fri, Jan 28, 2022 at 11:52 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Fri, Jan 28, 2022 at 04:49:34PM +0100, Cornelia Huck wrote:
> > > On Fri, Jan 28 2022, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >
> > > > On Fri, Jan 28, 2022 at 01:14:14PM +0100, Cornelia Huck wrote:
> > > >> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > > >> > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > >> > +
> > > >> > +Admin virtqueue is used to send administrative commands to manipulate
> > > >> > +various features of the device and/or to manipulate various features,
> > > >> > +if possible, of another device within the same group (e.g. PCI VFs of
> > > >> > +a parent PCI PF device are grouped together. These devices can be
> > > >> > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > >> > +
> > > >> > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > >> > +feature bit.
> > > >> > +
> > > >> > +Admin virtqueue index may vary among different device types.
> > > >>
> > > >> So, my understanding is:
> > > >> - any device type may or may not support the admin vq
> > > >> - if the device type wants to be able to accommodate the admin vq, it
> > > >>   also needs to specify where it shows up when the feature is negotiated
> > > >>
> > > >> Do we expect that eventually all device types will need to support the
> > > >> admin vq (if some use case comes along that will require all devices to
> > > >> participate, for example?)
> > > >
> > > > I suspect yes. And that's one of the reasons why I'd rather we had a
> > > > device independent way to locate the admin queue. There are less
> > > > transports than device types.
> > >
> > > So, do we want to bite the bullet now and simply say that every device
> > > type has the admin vq as the last vq if the feature is negotiated?
> > > Should be straightforward for the device types that have a fixed number
> > > of vqs, and doable for those that have a variable amount (two device
> > > types are covered by this series anyway.) I think we need to put it with
> > > the device types, as otherwise the numbering of virtqueues could change
> > > in unpredictable ways with the admin vq off/on.
> >
> > Well that only works once. The next thing we'll need we won't be able to
> > make the last one ;) So I am inclined to add a per-transport field that
> > gives the admin queue number.
> 
> Technically, there's no need to use the same namespace for admin
> virtqueue if it has a dedicated notification area. If we go this way,
> we can simply use 0 as queue index for admin virtqueue.
> 
> Thanks

Problem is 0 and 1 use the same page for notification unless
there's a page per vq (which is possible but expensive in memory space
usage term).

> > Another advantage to this approach is that
> > we can make sure admin queue gets a page by itself (which can be good if
> > we want to allow access to regular vqs but not to the admin queue to
> > guest) even if regular vqs share a page. Will help devices use less
> > memory space.
> >
> > --
> > MST
> >

[virtio-comment] Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Max Gurtovoy]

On 1/28/2022 2:14 PM, Cornelia Huck wrote:
> On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
>
>> In one of the many use cases a user wants to manipulate features and
>> configuration of the virtio devices regardless of the device type
>> (net/block/console). Some of this configuration is generic enough. i.e
>> Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
>> such features query and manipulation by its parent PCI PF.
>>
>> Currently virtio specification defines control virtqueue to manipulate
>> features and configuration of the device it operates on. However,
>> control virtqueue commands are device type specific, which makes it very
>> difficult to extend for device agnostic commands.
>>
>> To support this requirement in elegant way, this patch introduces a new
>> admin virtqueue. Admin virtqueue will use the same command format for all
>> types of virtio devices.
>>
>> Manipulate features via admin virtqueue is asynchronous, scalable, easy
>> to extend and doesn't require additional and expensive on-die resources
>> to be allocated for every new feature that will be added in the future.
>>
>> Subsequent patches make use of this admin virtqueue.
>>
>> Reviewed-by: Parav Pandit <parav@nvidia.com>
>> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
>> ---
>>   admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
>>   content.tex     |  9 +++--
>>   2 files changed, 96 insertions(+), 2 deletions(-)
>>   create mode 100644 admin-virtq.tex
>>
>> diff --git a/admin-virtq.tex b/admin-virtq.tex
>> new file mode 100644
>> index 0000000..1a41c22
>> --- /dev/null
>> +++ b/admin-virtq.tex
>> @@ -0,0 +1,89 @@
>> +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
>> +
>> +Admin virtqueue is used to send administrative commands to manipulate
>> +various features of the device and/or to manipulate various features,
>> +if possible, of another device within the same group (e.g. PCI VFs of
>> +a parent PCI PF device are grouped together. These devices can be
>> +optionally managed by its parent PCI PF using its admin virtqueue.).
>> +
>> +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
>> +feature bit.
>> +
>> +Admin virtqueue index may vary among different device types.
> So, my understanding is:
> - any device type may or may not support the admin vq
> - if the device type wants to be able to accommodate the admin vq, it
>    also needs to specify where it shows up when the feature is negotiated
>
> Do we expect that eventually all device types will need to support the
> admin vq (if some use case comes along that will require all devices to
> participate, for example?)
>
> Anyway, I'd reword the two sentences above:
>
> "An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is
> negotiated. The index of the admin virtqueue is device type specific."

Ok. I'll use this in V3, unless we'll find some other mechanism that is 
not so clear now.

>
> (...)
>
>> @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>     that the driver can reset a queue individually.
>>     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>>   
>> +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
>> +  the device supports administration virtqueue negotiation.
> Maybe
>
> "This feature indicates that an administration virtqueue is supported." ?

Ok.


>
>> +
>>   \end{description}
>>   
>>   \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [PATCH v2 1/4] Add virtio Admin Virtqueue

[Michael S. Tsirkin]

On Sun, Jan 30, 2022 at 01:21:41PM +0200, Max Gurtovoy wrote:
> 
> On 1/28/2022 2:14 PM, Cornelia Huck wrote:
> > On Mon, Jan 24 2022, Max Gurtovoy <mgurtovoy@nvidia.com> wrote:
> > 
> > > In one of the many use cases a user wants to manipulate features and
> > > configuration of the virtio devices regardless of the device type
> > > (net/block/console). Some of this configuration is generic enough. i.e
> > > Number of MSI-X vectors of a virtio PCI VF device. There is a need to do
> > > such features query and manipulation by its parent PCI PF.
> > > 
> > > Currently virtio specification defines control virtqueue to manipulate
> > > features and configuration of the device it operates on. However,
> > > control virtqueue commands are device type specific, which makes it very
> > > difficult to extend for device agnostic commands.
> > > 
> > > To support this requirement in elegant way, this patch introduces a new
> > > admin virtqueue. Admin virtqueue will use the same command format for all
> > > types of virtio devices.
> > > 
> > > Manipulate features via admin virtqueue is asynchronous, scalable, easy
> > > to extend and doesn't require additional and expensive on-die resources
> > > to be allocated for every new feature that will be added in the future.
> > > 
> > > Subsequent patches make use of this admin virtqueue.
> > > 
> > > Reviewed-by: Parav Pandit <parav@nvidia.com>
> > > Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> > > ---
> > >   admin-virtq.tex | 89 +++++++++++++++++++++++++++++++++++++++++++++++++
> > >   content.tex     |  9 +++--
> > >   2 files changed, 96 insertions(+), 2 deletions(-)
> > >   create mode 100644 admin-virtq.tex
> > > 
> > > diff --git a/admin-virtq.tex b/admin-virtq.tex
> > > new file mode 100644
> > > index 0000000..1a41c22
> > > --- /dev/null
> > > +++ b/admin-virtq.tex
> > > @@ -0,0 +1,89 @@
> > > +\section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues}
> > > +
> > > +Admin virtqueue is used to send administrative commands to manipulate
> > > +various features of the device and/or to manipulate various features,
> > > +if possible, of another device within the same group (e.g. PCI VFs of
> > > +a parent PCI PF device are grouped together. These devices can be
> > > +optionally managed by its parent PCI PF using its admin virtqueue.).
> > > +
> > > +Use of Admin virtqueue is negotiated by the VIRTIO_F_ADMIN_VQ
> > > +feature bit.
> > > +
> > > +Admin virtqueue index may vary among different device types.
> > So, my understanding is:
> > - any device type may or may not support the admin vq
> > - if the device type wants to be able to accommodate the admin vq, it
> >    also needs to specify where it shows up when the feature is negotiated
> > 
> > Do we expect that eventually all device types will need to support the
> > admin vq (if some use case comes along that will require all devices to
> > participate, for example?)
> > 
> > Anyway, I'd reword the two sentences above:
> > 
> > "An admin virtqueue exists for a certain device if VIRTIO_F_ADMIN_VQ is
> > negotiated. The index of the admin virtqueue is device type specific."
> 
> Ok. I'll use this in V3, unless we'll find some other mechanism that is not
> so clear now.

I suggest a transport specific register for the admin queue index.
What do others think?

> > 
> > (...)
> > 
> > > @@ -6847,6 +6849,9 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> > >     that the driver can reset a queue individually.
> > >     See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> > > +  \item[VIRTIO_F_ADMIN_VQ (41)] This feature indicates that
> > > +  the device supports administration virtqueue negotiation.
> > Maybe
> > 
> > "This feature indicates that an administration virtqueue is supported." ?
> 
> Ok.
> 
> 
> > 
> > > +
> > >   \end{description}
> > >   \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}

[virtio-comment] [PATCH v2 2/4] virtio-blk: add support for VIRTIO_F_ADMIN_VQ

[Max Gurtovoy]

Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation.

Reviewed-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 content.tex | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/content.tex b/content.tex
index c524fab..f5f95fb 100644
--- a/content.tex
+++ b/content.tex
@@ -4517,10 +4517,19 @@ \subsection{Device ID}\label{sec:Device Types / Block Device / Device ID}
   2
 
 \subsection{Virtqueues}\label{sec:Device Types / Block Device / Virtqueues}
+ if VIRTIO_F_ADMIN_VQ is not negotiated, the request queues layout is as follows:
 \begin{description}
 \item[0] requestq1
 \item[\ldots]
 \item[N-1] requestqN
+\end{description}
+
+ If VIRTIO_F_ADMIN_VQ is negotiated, the queues layout is as follows:
+\begin{description}
+\item[0] requestq1
+\item[\ldots]
+\item[N-1] requestqN
+\item[N] adminq
 \end{description}
 
  N=1 if VIRTIO_BLK_F_MQ is not negotiated, otherwise N is set by
@@ -4589,7 +4598,7 @@ \subsection{Device configuration layout}\label{sec:Device Types / Block Device /
 bits as indicated above.
 
 The field \field{num_queues} only exists if VIRTIO_BLK_F_MQ is set. This field specifies
-the number of queues.
+the number of request queues. This field doesn't account admin virtqueue.
 
 The parameters in the configuration space of the device \field{max_discard_sectors}
 \field{discard_sector_alignment} are expressed in 512-byte units if the
-- 
2.21.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[PATCH v2 3/4] virtio-net: add support for VIRTIO_F_ADMIN_VQ

[Max Gurtovoy]

Set the relevant index in case of VIRTIO_F_ADMIN_VQ negotiation.

Reviewed-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 content.tex | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/content.tex b/content.tex
index f5f95fb..18a5c66 100644
--- a/content.tex
+++ b/content.tex
@@ -3033,6 +3033,7 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 \item[2(N-1)] receiveqN
 \item[2(N-1)+1] transmitqN
 \item[2N] controlq
+\item[2N + 1] adminq (or \textbf{2N} in case VIRTIO_NET_F_CTRL_VQ is not set)
 \end{description}
 
  N=1 if neither VIRTIO_NET_F_MQ nor VIRTIO_NET_F_RSS are negotiated, otherwise N is set by
@@ -3040,6 +3041,8 @@ \subsection{Virtqueues}\label{sec:Device Types / Network Device / Virtqueues}
 
  controlq only exists if VIRTIO_NET_F_CTRL_VQ set.
 
+ adminq only exists if VIRTIO_F_ADMIN_VQ set.
+
 \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits}
 
 \begin{description}
-- 
2.21.0

[PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Max Gurtovoy]

A typical cloud provider SR-IOV use case is to create many VFs for
use by guest VMs. The VFs may not be assigned to a VM until a user
requests a VM of a certain size, e.g., number of CPUs. A VF may need
MSI-X vectors proportional to the number of CPUs in the VM, but there is
no standard way today in the spec to change the number of MSI-X vectors
supported by a VF, although there are some operating systems that
support this.

Introduce new admin commands for a generic interrupt vector management
for PCI VFs. For now, this mechanism will manage the MSI-X interrupt
vectors assignments of a VF by its parent PF.

These admin commands will be easily extended, if needed, for other types
of interrupt vectors in the future with backward compatibility to old
drivers and devices.

Reviewed-by: Parav Pandit <parav@nvidia.com>
Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
---
 admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++-
 content.tex     |  44 ++++++++++++++
 2 files changed, 192 insertions(+), 2 deletions(-)

diff --git a/admin-virtq.tex b/admin-virtq.tex
index 1a41c22..b57b57d 100644
--- a/admin-virtq.tex
+++ b/admin-virtq.tex
@@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin
 \hline \hline
 0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
 \hline
-0001h - 7FFFh   & Generic admin cmds    & -  \\
+0001h   & VIRTIO_ADMIN_PCI_SRIOV_ATTRS    & O  \\
+\hline
+0002h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET    & O  \\
+\hline
+0003h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET    & O  \\
+\hline
+0004h - 7FFFh   & Generic admin cmds    & -  \\
 \hline
 8000h - FFFFh   & Reserved    & - \\
 \hline
@@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
 \begin{lstlisting}
 struct virtio_admin_caps_identify_result {
        /*
-        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
+        * caps[0] - Bit 0x0 - if set, VF MSI-X control supported
+        *           Bit 0x1 - Bit 0x7 are reserved
         * caps[1] - Bit 0x0 - Bit 0x7 are reserved
         * caps[2] - Bit 0x0 - Bit 0x7 are reserved
         * ....
@@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
        u8 caps[8192];
 };
 \end{lstlisting}
+
+For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}.
+
+\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command}
+
+The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver.
+This command upon success, returns a data buffer that describes information about PCI SRIOV
+related capabilities and attributes for the device. This command can be supported only by
+PCI devices that supports Single Root I/O Virtualization.
+This information is of form:
+\begin{lstlisting}
+struct virtio_admin_pci_sriov_attrs_result {
+        /* For compatibility - indicates which of the below fields are valid (1 means valid):
+         * Bit 0x0 - vfs_free_msix_count
+         * Bit 0x1 - per_vf_max_msix_count
+         * Bits 0x2 - 0x3F - reserved for future fields
+         */
+        le64 attrs_mask;
+        /* Number of free msix vectors in the global msix pool for VFs */
+        le32 vfs_free_msix_count;
+        /* Max number of msix vectors that can be assigned for a single VF */
+        le16 per_vf_max_msix_count;
+};
+\end{lstlisting}
+
+\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}
+
+The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors
+count for a PCI virtual function. The command specific data set by the driver is of form:
+\begin{lstlisting}
+struct virtio_admin_pci_vf_interrupt_config_set_data {
+        /* The virtual function number */
+        le16 vf_number;
+        /* For compatibility - indicates which of the below properties should be
+         * modified (1 means that field should be modified):
+         * Bit 0x0 - msix_count
+         * Bits 0x1 - 0x3F - reserved for future fields
+         */
+        le64 attrs_mask;
+        /* The amount of MSI-X interrupt vectors */
+        le16 msix_count;
+};
+\end{lstlisting}
+
+The following table describes the command specific error codes codes:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Opcode & Status & Description \\
+\hline \hline
+00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\
+\hline
+01h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED    & MSI-X count exceed the max value per VF  \\
+\hline
+02h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC    & MSI-X count exceed the free pool resources   \\
+\hline
+03h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE    & VF is already in use, operation failed   \\
+\hline
+04h - FFh   & Reserved    & -  \\
+\hline
+\end{tabular}
+
+\begin{note}
+{vf_number can't be greater than NumVFs value as defined in the PCI specification
+or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
+is returned when vf_number is out of the range.}
+\end{note}
+
+\begin{note}
+{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED
+is returned when the amount of MSI-X to assign exceed the maximum value that can be
+assigned to a single VF.}
+\end{note}
+
+\begin{note}
+{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC
+is returned when the amount of MSI-X to assign exceed the amount of free MSI-X
+vectors in the global pool.}
+\end{note}
+
+This command has no command specific result set by the device. Upon success, the device guarantees
+that all the requested properties were modified to the given values. Otherwise, error will be returned.
+
+\begin{note}
+{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.
+Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.}
+\end{note}
+
+\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}
+
+The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs
+interrupt vectors configuration.
+The command specific data set by the driver is of form:
+\begin{lstlisting}
+struct virtio_admin_pci_vf_interrupt_config_get_data {
+        /* The virtual function number */
+        le16 vf_number;
+        /* For compatibility - indicates which of the below properties should be
+         * queried (1 means that field should be queried):
+         * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors)
+         * Bits 0x1 - 0x3F - reserved for future fields
+         */
+        le64 attrs_mask;
+};
+\end{lstlisting}
+
+The following table describes the command specific error codes codes:
+
+\begin{tabular}{|l|l|l|}
+\hline
+Opcode & Status & Description \\
+\hline \hline
+00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\
+\hline
+01h - FFh   & Reserved    & -  \\
+\hline
+\end{tabular}
+
+\begin{note}
+{vf_number can't be greater than NumVFs value as defined in the PCI specification
+or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
+is returned when vf_number is out of the range.}
+\end{note}
+
+This command, upon success, returns a data buffer that describes the properties that were requested
+and their values for the subject virtio VF device according to the given vf_number.
+This information is of form:
+\begin{lstlisting}
+struct virtio_admin_pci_vf_interrupt_config_get_result {
+        /* For compatibility - indicates which of the below fields were returned
+         * (1 means that field was returned):
+         * Bit 0x0 - msix_count
+         * Bits 0x1 - 0x3F - reserved for future fields
+         */
+        le64 attrs_mask;
+        /* The amount of MSI-X interrupt vectors currently assigned to the VF */
+        le16 msix_count;
+};
+\end{lstlisting}
diff --git a/content.tex b/content.tex
index 18a5c66..2dd1d46 100644
--- a/content.tex
+++ b/content.tex
@@ -1734,6 +1734,50 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
     \end{itemize}
 \end{itemize}
 
+\subsection{PCI-specific Admin capabilities}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin capabilities}
+
+This documents the group of Admin capabilities for PCI virtio devices. Each capability is
+implemented using one or more Admin queue commands.
+
+\subsubsection{VF MSI-X control}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}
+
+This capability enables a virtio PCI PF device to control the assignment of MSI-X interrupt vectors
+for its managed VFs. Capable devices will need to set bit 0x0 of caps[0] in the result of VIRTIO_ADMIN_CAPS_IDENTIFY
+admin command. See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}
+for more details.
+
+Capable devices will also need to implement VIRTIO_ADMIN_PCI_SRIOV_ATTRS, VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and
+VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin commands.
+
+In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a capable device should return the number of free msix vectors
+in the global msix pool for its VFs in \field{vfs_free_msix_count} field and also the maximal number of msix vectors that
+can be assigned for a single VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1 should be set
+to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer.
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more
+details.
+
+A PCI PF device may assign default number of MSI-X vectors to the enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
+it is possible to update the default MSI-X assignment for a specific VF.
+In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, a driver set the virtual function number in
+\field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
+In addition, bit 0x0 in the \field{attrs_mask} field should be set. A successful operation guarantees that the requested
+amount of MSI-X interrupt vectors was assigned to the subject virtual function.
+Also, a successful operation guarantees that the MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect
+the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control
+register should reflect a value of 7.
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
+
+It is beyond the scope of the virtio specification to define necessary synchronization in system software to ensure that a virtio PCI VF device
+interrupt configuration modification is reflected in the PCI device. However, it is expected that any modern system software implementing virtio
+drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or
+such configuration fails.
+
+In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin command, a driver should set the virtual function number in
+\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} field should be set to indicate requested output fields in
+the result from the device. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently
+assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit 0x0 in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field.
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
+
 \section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO}
 
 Virtual environments without PCI support (a common situation in
-- 
2.21.0

Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Michael S. Tsirkin]

On Mon, Jan 24, 2022 at 11:39:18AM +0200, Max Gurtovoy wrote:
> A typical cloud provider SR-IOV use case is to create many VFs for
> use by guest VMs. The VFs may not be assigned to a VM until a user
> requests a VM of a certain size, e.g., number of CPUs. A VF may need
> MSI-X vectors proportional to the number of CPUs in the VM, but there is
> no standard way today in the spec to change the number of MSI-X vectors
> supported by a VF, although there are some operating systems that
> support this.
> 
> Introduce new admin commands for a generic interrupt vector management
> for PCI VFs. For now, this mechanism will manage the MSI-X interrupt
> vectors assignments of a VF by its parent PF.
> 
> These admin commands will be easily extended, if needed, for other types
> of interrupt vectors in the future with backward compatibility to old
> drivers and devices.
> 
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>  admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++-
>  content.tex     |  44 ++++++++++++++
>  2 files changed, 192 insertions(+), 2 deletions(-)
> 
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> index 1a41c22..b57b57d 100644
> --- a/admin-virtq.tex
> +++ b/admin-virtq.tex
> @@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin
>  \hline \hline
>  0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
>  \hline
> -0001h - 7FFFh   & Generic admin cmds    & -  \\
> +0001h   & VIRTIO_ADMIN_PCI_SRIOV_ATTRS    & O  \\
> +\hline
> +0002h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET    & O  \\
> +\hline
> +0003h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET    & O  \\
> +\hline
> +0004h - 7FFFh   & Generic admin cmds    & -  \\
>  \hline
>  8000h - FFFFh   & Reserved    & - \\
>  \hline
> @@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
>  \begin{lstlisting}
>  struct virtio_admin_caps_identify_result {
>         /*
> -        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
> +        * caps[0] - Bit 0x0 - if set, VF MSI-X control supported
> +        *           Bit 0x1 - Bit 0x7 are reserved
>          * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>          * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>          * ....
> @@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
>         u8 caps[8192];
>  };
>  \end{lstlisting}
> +
> +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}.
> +
> +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command}
> +
> +The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver.
> +This command upon success, returns a data buffer that describes information about PCI SRIOV
> +related capabilities and attributes for the device. This command can be supported only by
> +PCI devices that supports Single Root I/O Virtualization.
> +This information is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_sriov_attrs_result {
> +        /* For compatibility - indicates which of the below fields are valid (1 means valid):
> +         * Bit 0x0 - vfs_free_msix_count
> +         * Bit 0x1 - per_vf_max_msix_count
> +         * Bits 0x2 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +        /* Number of free msix vectors in the global msix pool for VFs */
> +        le32 vfs_free_msix_count;
> +        /* Max number of msix vectors that can be assigned for a single VF */
> +        le16 per_vf_max_msix_count;
> +};
> +\end{lstlisting}
> +
> +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}
> +
> +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors
> +count for a PCI virtual function. The command specific data set by the driver is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_set_data {
> +        /* The virtual function number */
> +        le16 vf_number;
> +        /* For compatibility - indicates which of the below properties should be
> +         * modified (1 means that field should be modified):
> +         * Bit 0x0 - msix_count
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +        /* The amount of MSI-X interrupt vectors */
> +        le16 msix_count;
> +};
> +\end{lstlisting}
> +
> +The following table describes the command specific error codes codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\
> +\hline
> +01h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED    & MSI-X count exceed the max value per VF  \\
> +\hline
> +02h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC    & MSI-X count exceed the free pool resources   \\
> +\hline
> +03h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE    & VF is already in use, operation failed   \\
> +\hline
> +04h - FFh   & Reserved    & -  \\
> +\hline
> +\end{tabular}
> +
> +\begin{note}
> +{vf_number can't be greater than NumVFs value as defined in the PCI specification
> +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
> +is returned when vf_number is out of the range.}
> +\end{note}
> +
> +\begin{note}
> +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED
> +is returned when the amount of MSI-X to assign exceed the maximum value that can be
> +assigned to a single VF.}
> +\end{note}
> +
> +\begin{note}
> +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC
> +is returned when the amount of MSI-X to assign exceed the amount of free MSI-X
> +vectors in the global pool.}
> +\end{note}
> +
> +This command has no command specific result set by the device. Upon success, the device guarantees
> +that all the requested properties were modified to the given values. Otherwise, error will be returned.
> +
> +\begin{note}
> +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.
> +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.}
> +\end{note}
> +
> +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}
> +
> +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs
> +interrupt vectors configuration.
> +The command specific data set by the driver is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_get_data {
> +        /* The virtual function number */
> +        le16 vf_number;
> +        /* For compatibility - indicates which of the below properties should be
> +         * queried (1 means that field should be queried):
> +         * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors)
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +};
> +\end{lstlisting}
> +
> +The following table describes the command specific error codes codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\
> +\hline
> +01h - FFh   & Reserved    & -  \\
> +\hline
> +\end{tabular}
> +
> +\begin{note}
> +{vf_number can't be greater than NumVFs value as defined in the PCI specification
> +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
> +is returned when vf_number is out of the range.}
> +\end{note}
> +
> +This command, upon success, returns a data buffer that describes the properties that were requested
> +and their values for the subject virtio VF device according to the given vf_number.
> +This information is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_get_result {
> +        /* For compatibility - indicates which of the below fields were returned
> +         * (1 means that field was returned):
> +         * Bit 0x0 - msix_count
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +        /* The amount of MSI-X interrupt vectors currently assigned to the VF */
> +        le16 msix_count;
> +};
> +\end{lstlisting}
> diff --git a/content.tex b/content.tex
> index 18a5c66..2dd1d46 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -1734,6 +1734,50 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
>      \end{itemize}
>  \end{itemize}
>  
> +\subsection{PCI-specific Admin capabilities}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin capabilities}
> +
> +This documents the group of Admin capabilities for PCI virtio devices. Each capability is
> +implemented using one or more Admin queue commands.
> +
> +\subsubsection{VF MSI-X control}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}
> +
> +This capability enables a virtio PCI PF device to control the assignment of MSI-X interrupt vectors
> +for its managed VFs. Capable devices will need to set bit 0x0 of caps[0] in the result of VIRTIO_ADMIN_CAPS_IDENTIFY
> +admin command. See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}
> +for more details.
> +
> +Capable devices will also need to implement VIRTIO_ADMIN_PCI_SRIOV_ATTRS, VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and
> +VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin commands.
> +
> +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a capable device should return the number of free msix vectors
> +in the global msix pool for its VFs in \field{vfs_free_msix_count} field and also the maximal number of msix vectors that
> +can be assigned for a single VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1 should be set
> +to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more
> +details.
> +
> +A PCI PF device may assign default number of MSI-X vectors to the
> enabled PCI VFs.


MAY SHIULD etc all need to be capitalized and moved to a conformance sections.
Free text just using simple tense, e.g. "bit 0 is set".

So with a vague statement like this, how can the vectors be
managed?  I guess one basically has to start by querying all VFs?

> Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
> +it is possible to update the default MSI-X assignment for a specific VF.
> +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, a driver set the virtual function number in
> +\field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
> +In addition, bit 0x0 in the \field{attrs_mask} field

this use of hex for bit numbers is confusing. it's ok to use them
for values but bit numbers are clearer decimail imho.

> should be set. A successful operation guarantees that the requested
> +amount of MSI-X interrupt vectors was assigned to the subject virtual function.
> +Also, a successful operation guarantees that the MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect
> +the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control
> +register should reflect a value of 7.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.

Lifetime rules for these commands need to be discussed.
I am guessing this command is not allowed when driver
is accessing the VF?
VFs have lots of other resources besides MSI-X.
SRIOV is quite primitive in requiring knowing # of VFs enabled up front.
If we are going to this length to manage resources per VF
should we not start with a more basic thing, and allow
enabling/disabling individual VFs?


All this looks kind of messy.

I think Jason's previous comment where he suggested enabling the VF with
a single command begins to look more attractive.

IOW we would have two commands, enable and disable, an enabled VFs has
to be supplied with all of its configuration.

And naturally driver is supposed to attach only to enabled VFs.


> +
> +It is beyond the scope of the virtio specification to define
> necessary synchronization in system software to ensure that a virtio
> PCI VF device +interrupt configuration modification is reflected in
> the PCI device.

I don't see why this is beyond the scope of the specification,
seems quite pertinent. It's probably hard to do, so we can
say that we don't, but let's not pretend we are doing anyone
a favor here ;)

> However, it is expected that any modern system
> software implementing virtio +drivers and PCI subsystem should ensure
> that any changes occurring in the VF interrupt configuration is either
> updated in the PCI VF device or +such configuration fails.
> +
> +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin command, a driver should set the virtual function number in
> +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} field should be set to indicate requested output fields in
> +the result from the device. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently
> +assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit 0x0 in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
> +
>  \section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO}
>  
>  Virtual environments without PCI support (a common situation in
> -- 
> 2.21.0

This needs driver and device conformance sections.

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, January 25, 2022 5:23 PM
> 
> On Mon, Jan 24, 2022 at 11:39:18AM +0200, Max Gurtovoy wrote:

> > +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a
> > +capable device should return the number of free msix vectors in the
> > +global msix pool for its VFs in \field{vfs_free_msix_count} field and
> > +also the maximal number of msix vectors that can be assigned for a single
> VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1
> should be set to indicate on the validity of the other 2 fields in the
> \field{attrs_mask} field of the result buffer.
> > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin
> > +Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more details.
> > +
> > +A PCI PF device may assign default number of MSI-X vectors to the
> > enabled PCI VFs.
> 
> 
> MAY SHIULD etc all need to be capitalized and moved to a conformance
> sections.
Above line is an informative detail. It is not related to conformance.
Is current place better? Or?

> Free text just using simple tense, e.g. "bit 0 is set".
> 
Ok.
> So with a vague statement like this, how can the vectors be managed?  I guess
> one basically has to start by querying all VFs?
Need not be.
When deploying a VF, just pick a first free VF, assign the required vectors if not same, and use it.
But yes, first time query can be done too get all VFs info.

> 
> > Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
> > +it is possible to update the default MSI-X assignment for a specific VF.
> > +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin
> > +command, a driver set the virtual function number in \field{vf_number} and
> the amount of MSI-X interrupt vectors to configure to the subject virtual
> function in \field{msix_count}.
> > +In addition, bit 0x0 in the \field{attrs_mask} field
> 
> this use of hex for bit numbers is confusing. it's ok to use them for values but bit
> numbers are clearer decimail imho.
> 
Ok.
Need to update in v3.

> > should be set. A successful operation guarantees that the requested
> > +amount of MSI-X interrupt vectors was assigned to the subject virtual
> function.
> > +Also, a successful operation guarantees that the MSI-X capability
> > +access by the subject PCI VF defined by the PCI specification must
> > +reflect the new configuration in all relevant fields. For example, by default if
> the PCI VF has been assigned 4 MSI-X vectors, and
> VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors
> to 8; on this change, reading Table size field of the MSI-X message control
> register should reflect a value of 7.
> > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
> VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> 
> Lifetime rules for these commands need to be discussed.
It is described below

> I am guessing this command is not allowed when driver is accessing the VF?
> VFs have lots of other resources besides MSI-X.
> SRIOV is quite primitive in requiring knowing # of VFs enabled up front.
> If we are going to this length to manage resources per VF should we not start
> with a more basic thing, and allow enabling/disabling individual VFs?
> 
Sure, It is.
This is why we and others created scalable functions/subfunctions/adi to overcome some of these limitations over last 2-3 years.
Even though SIOV exists in some form, it has own deployment limitations from platform side.

However, this primitive SRIOV technology is exposed by the virtio spec using VIRTIO_F_SR_IOV for a while and also in use by users. :)
Here is an attempt to improve it.

> 
> All this looks kind of messy.
> 
> I think Jason's previous comment where he suggested enabling the VF with a
> single command begins to look more attractive.
> 
The device vendor who actually implements it can propose it extension and plumbing in the spec.
I have explored this more than 2 years ago before upstreaming the mlx5 SF solution. And its fairly complicated beast to achieve in PCI subsystem.
To our knowledge, there is no such device, PCI subsystem, and OS in existence that can actually do it.

And when that occurs, it will be able to make use of msix config command presented here on existing VFs.
It will be able to use it without disabling the whole VF on redeploying the VF second time.

> IOW we would have two commands, enable and disable, an enabled VFs has to
> be supplied with all of its configuration.
> 
I wish best to those who will attempt this.
Devices that we experience are able to handle SR-IOV scale for several hundreds of VFs without need to undergo enable-disable individual VFs.

> And naturally driver is supposed to attach only to enabled VFs.
> 
> 
> > +
> > +It is beyond the scope of the virtio specification to define
> > necessary synchronization in system software to ensure that a virtio
> > PCI VF device +interrupt configuration modification is reflected in
> > the PCI device.
> 
> I don't see why this is beyond the scope of the specification, seems quite
> pertinent. 

It is beyond the scope of specification, because virtio specification do not define what all system software modules to do.
It is very OS specific.
virtio specification doesn't know what those modules are either.
It can only provide the guidance. Cannot define how/what to do.

And this hasn't changed from our discussion in v1.

> > However, it is expected that any modern system software implementing
> > virtio +drivers and PCI subsystem should ensure that any changes
> > occurring in the VF interrupt configuration is either updated in the
> > PCI VF device or +such configuration fails.
> > +
> > +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin
> > +command, a driver should set the virtual function number in
> > +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask}
> > +field should be set to indicate requested output fields in the result from the
> device. In the result of a successful operation, the amount of MSI-X interrupt
> vectors that is currently assigned to the subject virtual function is returned by
> the device in \field{msix_count} field. In addition, bit 0x0 in the
> \field{attrs_mask} field should be set by the device to indicate on the validity of
> \field{msix_count} field.
> > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
> VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
> > +
> >  \section{Virtio Over MMIO}\label{sec:Virtio Transport Options /
> > Virtio Over MMIO}
> >
> >  Virtual environments without PCI support (a common situation in
> > --
> > 2.21.0
> 
> This needs driver and device conformance sections.
Ok. v3 to cover this too.

[virtio-comment] Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Michael S. Tsirkin]

On Wed, Jan 26, 2022 at 01:03:15PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, January 25, 2022 5:23 PM
> > 
> > On Mon, Jan 24, 2022 at 11:39:18AM +0200, Max Gurtovoy wrote:
> 
> > > +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a
> > > +capable device should return the number of free msix vectors in the
> > > +global msix pool for its VFs in \field{vfs_free_msix_count} field and
> > > +also the maximal number of msix vectors that can be assigned for a single
> > VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1
> > should be set to indicate on the validity of the other 2 fields in the
> > \field{attrs_mask} field of the result buffer.
> > > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin
> > > +Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more details.
> > > +
> > > +A PCI PF device may assign default number of MSI-X vectors to the
> > > enabled PCI VFs.
> > 
> > 
> > MAY SHIULD etc all need to be capitalized and moved to a conformance
> > sections.
> Above line is an informative detail. It is not related to conformance.
> Is current place better? Or?

the way we do it generally is this. write an informative text avoiding
use of MAY/SHOULD etc. Just describe the way it works. Add conformance
statements somewhat repeating the informative text using MAY/SHOULD/MUST
etc.


> > Free text just using simple tense, e.g. "bit 0 is set".
> > 
> Ok.
> > So with a vague statement like this, how can the vectors be managed?  I guess
> > one basically has to start by querying all VFs?
> Need not be.
> When deploying a VF, just pick a first free VF, assign the required vectors

management tools need to balance requirements versus availability

> if not same,

same as what?

> and use it.
> But yes, first time query can be done too get all VFs info.
> 
> > 
> > > Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
> > > +it is possible to update the default MSI-X assignment for a specific VF.
> > > +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin
> > > +command, a driver set the virtual function number in \field{vf_number} and
> > the amount of MSI-X interrupt vectors to configure to the subject virtual
> > function in \field{msix_count}.
> > > +In addition, bit 0x0 in the \field{attrs_mask} field
> > 
> > this use of hex for bit numbers is confusing. it's ok to use them for values but bit
> > numbers are clearer decimail imho.
> > 
> Ok.
> Need to update in v3.
> 
> > > should be set. A successful operation guarantees that the requested
> > > +amount of MSI-X interrupt vectors was assigned to the subject virtual
> > function.
> > > +Also, a successful operation guarantees that the MSI-X capability
> > > +access by the subject PCI VF defined by the PCI specification must
> > > +reflect the new configuration in all relevant fields. For example, by default if
> > the PCI VF has been assigned 4 MSI-X vectors, and
> > VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors
> > to 8; on this change, reading Table size field of the MSI-X message control
> > register should reflect a value of 7.
> > > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
> > VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> > 
> > Lifetime rules for these commands need to be discussed.
> It is described below

Couldn't find it sorry. Adding a reference might be helpful.

> > I am guessing this command is not allowed when driver is accessing the VF?
> > VFs have lots of other resources besides MSI-X.
> > SRIOV is quite primitive in requiring knowing # of VFs enabled up front.
> > If we are going to this length to manage resources per VF should we not start
> > with a more basic thing, and allow enabling/disabling individual VFs?
> > 
> Sure, It is.
> This is why we and others created scalable functions/subfunctions/adi to overcome some of these limitations over last 2-3 years.
> Even though SIOV exists in some form, it has own deployment limitations from platform side.
> 
> However, this primitive SRIOV technology is exposed by the virtio spec using VIRTIO_F_SR_IOV for a while and also in use by users. :)
> Here is an attempt to improve it.

Right. So how about enabling/disabling individual VFs then?

> > 
> > All this looks kind of messy.
> > 
> > I think Jason's previous comment where he suggested enabling the VF with a
> > single command begins to look more attractive.
> > 
> The device vendor who actually implements it can propose it extension and plumbing in the spec.
> I have explored this more than 2 years ago before upstreaming the mlx5 SF solution. And its fairly complicated beast to achieve in PCI subsystem.
> To our knowledge, there is no such device, PCI subsystem, and OS in existence that can actually do it.
> 
> And when that occurs, it will be able to make use of msix config command presented here on existing VFs.
> It will be able to use it without disabling the whole VF on redeploying the VF second time.

I'm not sure we are speaking about the same thing.
The way you propose is each VF gets some "default" config and then the
commands can be used to tweak that.  Which seems pretty complex for
management software if it ever does need to tweak things.

And alternative is to declare that VF is not set up
automatically, and ask driver to set it up before use.

It's strictly less work for the device so I find it hard to
believe it's impossible to implement.







> > IOW we would have two commands, enable and disable, an enabled VFs has to
> > be supplied with all of its configuration.
> > 
> I wish best to those who will attempt this.

It's possible that you will end up being that person, we need to
have the interface that's generic and clear enough even if
your hardware does not benefit.

> Devices that we experience are able to handle SR-IOV scale for several hundreds of VFs without need to undergo enable-disable individual VFs.

OK... and is it true that at the same time they do not support enough
MSI-X vectors per VF such that we need complex machinery to control
them?

What is the motivation for this proposal again?


> > And naturally driver is supposed to attach only to enabled VFs.
> > 
> > 
> > > +
> > > +It is beyond the scope of the virtio specification to define
> > > necessary synchronization in system software to ensure that a virtio
> > > PCI VF device +interrupt configuration modification is reflected in
> > > the PCI device.
> > 
> > I don't see why this is beyond the scope of the specification, seems quite
> > pertinent. 
> 
> It is beyond the scope of specification, because virtio specification do not define what all system software modules to do.
> It is very OS specific.
> virtio specification doesn't know what those modules are either.
> It can only provide the guidance. Cannot define how/what to do.
> 
> And this hasn't changed from our discussion in v1.

Well we define a bunch of PCI things like BARs, device ID etc.  How is
this one different? I think basically it's a lot of work and we do not
think it's srictly necessary to define it, but it's not because of
scope.


> > > However, it is expected that any modern system software implementing
> > > virtio +drivers and PCI subsystem should ensure that any changes
> > > occurring in the VF interrupt configuration is either updated in the
> > > PCI VF device or +such configuration fails.
> > > +
> > > +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin
> > > +command, a driver should set the virtual function number in
> > > +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask}
> > > +field should be set to indicate requested output fields in the result from the
> > device. In the result of a successful operation, the amount of MSI-X interrupt
> > vectors that is currently assigned to the subject virtual function is returned by
> > the device in \field{msix_count} field. In addition, bit 0x0 in the
> > \field{attrs_mask} field should be set by the device to indicate on the validity of
> > \field{msix_count} field.
> > > +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues /
> > VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
> > > +
> > >  \section{Virtio Over MMIO}\label{sec:Virtio Transport Options /
> > > Virtio Over MMIO}
> > >
> > >  Virtual environments without PCI support (a common situation in
> > > --
> > > 2.21.0
> > 
> > This needs driver and device conformance sections.
> Ok. v3 to cover this too.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, January 26, 2022 7:39 PM

> > > > +A PCI PF device may assign default number of MSI-X vectors to the
> > > > enabled PCI VFs.
> > >
> > >
> > > MAY SHIULD etc all need to be capitalized and moved to a conformance
> > > sections.
> > Above line is an informative detail. It is not related to conformance.
> > Is current place better? Or?
> 
> the way we do it generally is this. write an informative text avoiding use of
> MAY/SHOULD etc. 
Ok. 
> Just describe the way it works. Add conformance
> statements somewhat repeating the informative text using
> MAY/SHOULD/MUST etc.
> 
Got it. Will do in v3.

> 
> > > Free text just using simple tense, e.g. "bit 0 is set".
> > >
> > Ok.
> > > So with a vague statement like this, how can the vectors be managed?
> > > I guess one basically has to start by querying all VFs?
> > Need not be.
> > When deploying a VF, just pick a first free VF, assign the required
> > vectors
> 
> management tools need to balance requirements versus availability
> 
> > if not same,
> 
> same as what?
>
Basically the example is,
If say all the VFs has default of 4 vectors, and if system admin wants to assign 8 vectors to a VF,
It issues msix_config command for a VF A to change from 4 to 8, and uses the VF.

If the system admin wants to assign 4 vectors to a VF, the current value in VF and desired value are same.
So no need to assign.


> > > VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> > >
> > > Lifetime rules for these commands need to be discussed.
> > It is described below
>
Below is the snippet from the patch. Please let us know if its missing something for lifetime rules.
> Couldn't find it sorry. Adding a reference might be helpful.

A PCI PF device may assign default number of MSI-X vectors to the 
+enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
+it is possible to update the default MSI-X assignment for a specific VF.
+In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, 
+a driver set the virtual function number in \field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
+In addition, bit 0x0 in the \field{attrs_mask} field should be set. A 
+successful operation guarantees that the requested amount of MSI-X interrupt vectors was assigned to the subject virtual function.
+Also, a successful operation guarantees that the MSI-X capability 
+access by the subject PCI VF defined by the PCI specification must 
+reflect the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7.
+See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
+

> > However, this primitive SRIOV technology is exposed by the virtio spec
> > using VIRTIO_F_SR_IOV for a while and also in use by users. :) Here is an
> attempt to improve it.
> 
> Right. So how about enabling/disabling individual VFs then?
>
 
> > >
> > > All this looks kind of messy.
> > >
> > > I think Jason's previous comment where he suggested enabling the VF
> > > with a single command begins to look more attractive.
> > >
> > The device vendor who actually implements it can propose it extension and
> plumbing in the spec.
> > I have explored this more than 2 years ago before upstreaming the mlx5 SF
> solution. And its fairly complicated beast to achieve in PCI subsystem.
> > To our knowledge, there is no such device, PCI subsystem, and OS in
> existence that can actually do it.
> >
> > And when that occurs, it will be able to make use of msix config command
> presented here on existing VFs.
> > It will be able to use it without disabling the whole VF on redeploying the VF
> second time.
> 
> I'm not sure we are speaking about the same thing.
> The way you propose is each VF gets some "default" config and then the
> commands can be used to tweak that.  Which seems pretty complex for
> management software if it ever does need to tweak things.
>
A default configured vector can be zero, or device may have given some default value.

It is like a netdevice comes up default with some X queues and if user wants to modify it, it does ethtool and use the new value.
It is not that complex.
 
> And alternative is to declare that VF is not set up automatically, and ask driver
> to set it up before use.
> 
Yes. This is equally good idea.
May be a pci_msix_config on PF device will be good idea indicating that, always have all the VFs with zero default vector.
And user will setup the vector before giving the VF to the driver.

> It's strictly less work for the device so I find it hard to believe it's impossible to
> implement.
>
It is less work. This is fine.
Enabling/disabling individual VF is a completely different mechanics.
 
> 
> > > IOW we would have two commands, enable and disable, an enabled VFs
> > > has to be supplied with all of its configuration.
> > >
> > I wish best to those who will attempt this.
> 
> It's possible that you will end up being that person, we need to have the
> interface that's generic and clear enough even if your hardware does not
> benefit.
> 
> > Devices that we experience are able to handle SR-IOV scale for several
> hundreds of VFs without need to undergo enable-disable individual VFs.
> 
> OK... and is it true that at the same time they do not support enough MSI-X
> vectors per VF such that we need complex machinery to control them?
>
Yes.
 
> What is the motivation for this proposal again?
> 
Efficiently distribute device resources among large number of VFs.

> 
> > > And naturally driver is supposed to attach only to enabled VFs.
> > >
> > >
> > > > +
> > > > +It is beyond the scope of the virtio specification to define
> > > > necessary synchronization in system software to ensure that a
> > > > virtio PCI VF device +interrupt configuration modification is
> > > > reflected in the PCI device.
> > >
> > > I don't see why this is beyond the scope of the specification, seems
> > > quite pertinent.
> >
> > It is beyond the scope of specification, because virtio specification do not
> define what all system software modules to do.
> > It is very OS specific.
> > virtio specification doesn't know what those modules are either.
> > It can only provide the guidance. Cannot define how/what to do.
> >
> > And this hasn't changed from our discussion in v1.
> 
> Well we define a bunch of PCI things like BARs, device ID etc.  How is this one
> different? 
It is different in following way.
When to pci capabilities, where to store in driver structure, when to update them, when to mark those capabilities dirty and re-read it.
All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can give guidance of what to expect so that OS can implement above for msix config update.

Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Michael S. Tsirkin]

On Thu, Jan 27, 2022 at 03:40:22AM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, January 26, 2022 7:39 PM
> 
> > > > > +A PCI PF device may assign default number of MSI-X vectors to the
> > > > > enabled PCI VFs.
> > > >
> > > >
> > > > MAY SHIULD etc all need to be capitalized and moved to a conformance
> > > > sections.
> > > Above line is an informative detail. It is not related to conformance.
> > > Is current place better? Or?
> > 
> > the way we do it generally is this. write an informative text avoiding use of
> > MAY/SHOULD etc. 
> Ok. 
> > Just describe the way it works. Add conformance
> > statements somewhat repeating the informative text using
> > MAY/SHOULD/MUST etc.
> > 
> Got it. Will do in v3.
> 
> > 
> > > > Free text just using simple tense, e.g. "bit 0 is set".
> > > >
> > > Ok.
> > > > So with a vague statement like this, how can the vectors be managed?
> > > > I guess one basically has to start by querying all VFs?
> > > Need not be.
> > > When deploying a VF, just pick a first free VF, assign the required
> > > vectors
> > 
> > management tools need to balance requirements versus availability
> > 
> > > if not same,
> > 
> > same as what?
> >
> Basically the example is,
> If say all the VFs has default of 4 vectors, and if system admin wants to assign 8 vectors to a VF,
> It issues msix_config command for a VF A to change from 4 to 8, and uses the VF.
> 
> If the system admin wants to assign 4 vectors to a VF, the current value in VF and desired value are same.
> So no need to assign.

But there's no way to know how many vectors there are by default,
or even whether the default is same for all VFs,
without first issuing a ton of commands.

> 
> > > > VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> > > >
> > > > Lifetime rules for these commands need to be discussed.
> > > It is described below
> >
> Below is the snippet from the patch. Please let us know if its missing something for lifetime rules.
> > Couldn't find it sorry. Adding a reference might be helpful.
> 
> A PCI PF device may assign default number of MSI-X vectors to the 
> +enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
> +it is possible to update the default MSI-X assignment for a specific VF.
> +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, 
> +a driver set the virtual function number in \field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
> +In addition, bit 0x0 in the \field{attrs_mask} field should be set. A 
> +successful operation guarantees that the requested amount of MSI-X interrupt vectors was assigned to the subject virtual function.
> +Also, a successful operation guarantees that the MSI-X capability 
> +access by the subject PCI VF defined by the PCI specification must 
> +reflect the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control register should reflect a value of 7.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> +


Nothing here specifies the lifetime, e.g. nothing precludes tweaking the
number while driver is using the VF. Which will just be a mess to
support.


> > > However, this primitive SRIOV technology is exposed by the virtio spec
> > > using VIRTIO_F_SR_IOV for a while and also in use by users. :) Here is an
> > attempt to improve it.
> > 
> > Right. So how about enabling/disabling individual VFs then?
> >
>  
> > > >
> > > > All this looks kind of messy.
> > > >
> > > > I think Jason's previous comment where he suggested enabling the VF
> > > > with a single command begins to look more attractive.
> > > >
> > > The device vendor who actually implements it can propose it extension and
> > plumbing in the spec.
> > > I have explored this more than 2 years ago before upstreaming the mlx5 SF
> > solution. And its fairly complicated beast to achieve in PCI subsystem.
> > > To our knowledge, there is no such device, PCI subsystem, and OS in
> > existence that can actually do it.
> > >
> > > And when that occurs, it will be able to make use of msix config command
> > presented here on existing VFs.
> > > It will be able to use it without disabling the whole VF on redeploying the VF
> > second time.
> > 
> > I'm not sure we are speaking about the same thing.
> > The way you propose is each VF gets some "default" config and then the
> > commands can be used to tweak that.  Which seems pretty complex for
> > management software if it ever does need to tweak things.
> >
> A default configured vector can be zero, or device may have given some default value.
> 
> It is like a netdevice comes up default with some X queues and if user wants to modify it, it does ethtool and use the new value.
> It is not that complex.

A ton of commands will be needed to first query all VFs,
then configure them all the way management wants to.

> > And alternative is to declare that VF is not set up automatically, and ask driver
> > to set it up before use.
> > 
> Yes. This is equally good idea.
> May be a pci_msix_config on PF device will be good idea indicating that, always have all the VFs with zero default vector.
> And user will setup the vector before giving the VF to the driver.
> 
> > It's strictly less work for the device so I find it hard to believe it's impossible to
> > implement.
> >
> It is less work. This is fine.
> Enabling/disabling individual VF is a completely different mechanics.

Well. you seem to already allow allocating 0 vectors to a VF,
which will break most drivers effectively disabling it.
I just this it's best to formalize this.


> > 
> > > > IOW we would have two commands, enable and disable, an enabled VFs
> > > > has to be supplied with all of its configuration.
> > > >
> > > I wish best to those who will attempt this.
> > 
> > It's possible that you will end up being that person, we need to have the
> > interface that's generic and clear enough even if your hardware does not
> > benefit.
> > 
> > > Devices that we experience are able to handle SR-IOV scale for several
> > hundreds of VFs without need to undergo enable-disable individual VFs.
> > 
> > OK... and is it true that at the same time they do not support enough MSI-X
> > vectors per VF such that we need complex machinery to control them?
> >
> Yes.
>  
> > What is the motivation for this proposal again?
> > 
> Efficiently distribute device resources among large number of VFs.

So what's the total # of vectors supported and what's the # of VFs then?

> > 
> > > > And naturally driver is supposed to attach only to enabled VFs.
> > > >
> > > >
> > > > > +
> > > > > +It is beyond the scope of the virtio specification to define
> > > > > necessary synchronization in system software to ensure that a
> > > > > virtio PCI VF device +interrupt configuration modification is
> > > > > reflected in the PCI device.
> > > >
> > > > I don't see why this is beyond the scope of the specification, seems
> > > > quite pertinent.
> > >
> > > It is beyond the scope of specification, because virtio specification do not
> > define what all system software modules to do.
> > > It is very OS specific.
> > > virtio specification doesn't know what those modules are either.
> > > It can only provide the guidance. Cannot define how/what to do.
> > >
> > > And this hasn't changed from our discussion in v1.
> > 
> > Well we define a bunch of PCI things like BARs, device ID etc.  How is this one
> > different? 
> It is different in following way.
> When to pci capabilities, where to store in driver structure, when to update them, when to mark those capabilities dirty and re-read it.
> All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can give guidance of what to expect so that OS can implement above for msix config update.

All I'm saying it let's drop the "beyond the scope" thing and in fact
this whole paragraph does not seem to add anything useful to the spec.

Maybe you are trying to say that the driver or systems software
needs to do some kind of synchronization - is that it?
It is not clear from this description what are you talking about,
what is synched with what, or
what does "reflected in the PCI device" mean.

-- 
MST

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, January 27, 2022 6:10 PM

> > If the system admin wants to assign 4 vectors to a VF, the current value in VF
> and desired value are same.
> > So no need to assign.
> 
> But there's no way to know how many vectors there are by default, or even
> whether the default is same for all VFs, without first issuing a ton of commands.
Either query all and build database (likely sane thing to do) or 
Query the first free VF, configure the value to it and use it.

> 
> Nothing here specifies the lifetime, e.g. nothing precludes tweaking the
> number while driver is using the VF. Which will just be a mess to support.

It is described in the v2.

+{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.

Above should be changed to "must not be used by the driver."

> > It is like a netdevice comes up default with some X queues and if user wants
> to modify it, it does ethtool and use the new value.
> > It is not that complex.
> 
> A ton of commands will be needed to first query all VFs, then configure them
> all the way management wants to.
So, it that a problem? To configure some value, query is needed.
If user doesn't want to issue too many commands it can pick the free VF and configure it.

Other option is below, where HV says, ah I don't like to query all the VFs.
I want to manage each VF individually.
So HV does one command even before enabling SR-IOV to give 0 vectors by default to each VF.
So sequence will be like,

1. reset all vf vectors, count = 0.
2. query total vectors of the device
3. enable sriov (num vfs = N)
3. assign #A vectors to VF-A.
4. assign VF to VM
5. use VM
6. release VM and VF
7. assign #B vectors to VF-A.
8. Repeat 5 to 6

> 
> > > And alternative is to declare that VF is not set up automatically,
> > > and ask driver to set it up before use.
> > >
> > Yes. This is equally good idea.
> > May be a pci_msix_config on PF device will be good idea indicating that,
> always have all the VFs with zero default vector.
> > And user will setup the vector before giving the VF to the driver.
> >
> > > It's strictly less work for the device so I find it hard to believe
> > > it's impossible to implement.
> > >
> > It is less work. This is fine.
> > Enabling/disabling individual VF is a completely different mechanics.
> 
> Well. you seem to already allow allocating 0 vectors to a VF, which will break
> most drivers effectively disabling it.
Allocating 0 vectors is not to break driver. Not sure why you use the term break.

> I just this it's best to formalize this.
I liked the idea of above 8 steps example, this will be efficient for the user.

> > > > > IOW we would have two commands, enable and disable, an enabled
> > > > > VFs has to be supplied with all of its configuration.
> > > > >
> > > > I wish best to those who will attempt this.
> > >
> > > It's possible that you will end up being that person, we need to
> > > have the interface that's generic and clear enough even if your
> > > hardware does not benefit.
> > >
> > > > Devices that we experience are able to handle SR-IOV scale for
> > > > several
> > > hundreds of VFs without need to undergo enable-disable individual VFs.
> > >
> > > OK... and is it true that at the same time they do not support
> > > enough MSI-X vectors per VF such that we need complex machinery to
> control them?
> > >
> > Yes.
> >
> > > What is the motivation for this proposal again?
> > >
> > Efficiently distribute device resources among large number of VFs.
> 
> So what's the total # of vectors supported and what's the # of VFs then?

One example, 256 VFs. total of 2K MSI-X vectors among these 256 VFs.

> > When to pci capabilities, where to store in driver structure, when to update
> them, when to mark those capabilities dirty and re-read it.
> > All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can
> give guidance of what to expect so that OS can implement above for msix
> config update.
> 
> All I'm saying it let's drop the "beyond the scope" thing and in fact this whole
> paragraph does not seem to add anything useful to the spec.
> 
Oh, you suggested to describe this details in v1 in the spec. 

> Maybe you are trying to say that the driver or systems software needs to do
> some kind of synchronization - is that it?
Yes.
> It is not clear from this description what are you talking about, what is synched
> with what, or what does "reflected in the PCI device" mean.

Below is what is written in proposed patch:

"MSI-X capability access by the subject PCI VF defined by the PCI specification must 
reflect the new configuration in all relevant fields."

All the configuration relevant to the MSI-X should reflect in necessary registers defined by PCI spec.

Probably it should be written as "All MSI-X related registers, capabilities and fields must reflect the new configuration".
Is this better?

Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Michael S. Tsirkin]

On Thu, Jan 27, 2022 at 02:16:32PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, January 27, 2022 6:10 PM
> 
> > > If the system admin wants to assign 4 vectors to a VF, the current value in VF
> > and desired value are same.
> > > So no need to assign.
> > 
> > But there's no way to know how many vectors there are by default, or even
> > whether the default is same for all VFs, without first issuing a ton of commands.
> Either query all and build database (likely sane thing to do) or 
> Query the first free VF, configure the value to it and use it.

And not only that. Let's say you have 256 vectors split between 256 VFs
equally. If you want to increase vectors for VF0, you need to start
by reducing vectors for other VFs.

It's really a ton of round trips to the card just for startup.



> > 
> > Nothing here specifies the lifetime, e.g. nothing precludes tweaking the
> > number while driver is using the VF. Which will just be a mess to support.
> 
> It is described in the v2.
> 
> +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.
> 
> Above should be changed to "must not be used by the driver."

ok so there's an implicit uninitialized state.

This is why I felt that instead, we should make it implicit
calling the command INITIALIZE, with flags specifying
what is being initialized.

And an UINITIALIZE command for symmetry.

Alternatively - VFs should start in
a known state, with 0 vectors, and uninitialized - driver can not bind.
Following this, we'd do commands to assign resources to the VFs.
Finally, we make the VF initialized - after this point driver can bind,
but configuration can no longer be changed.

It looks like in your proposal, just assigning msix count makes
VF initialized, so this is implicit. One also can not put VF
back in uninitialized state which seems asymmetrical.


> > > It is like a netdevice comes up default with some X queues and if user wants
> > to modify it, it does ethtool and use the new value.
> > > It is not that complex.
> > 
> > A ton of commands will be needed to first query all VFs, then configure them
> > all the way management wants to.
> So, it that a problem? To configure some value, query is needed.

yes because it's not even one command. It can be like 200 commands.
That is excessive.

> If user doesn't want to issue too many commands it can pick the free VF and configure it.
> 
> Other option is below, where HV says, ah I don't like to query all the VFs.
> I want to manage each VF individually.
> So HV does one command even before enabling SR-IOV to give 0 vectors by default to each VF.



> So sequence will be like,
> 
> 1. reset all vf vectors, count = 0.

This one small step is like 200 commands though.

> 2. query total vectors of the device
> 3. enable sriov (num vfs = N)
> 3. assign #A vectors to VF-A.
> 4. assign VF to VM
> 5. use VM
> 6. release VM and VF
> 7. assign #B vectors to VF-A.
> 8. Repeat 5 to 6
> 
> > 
> > > > And alternative is to declare that VF is not set up automatically,
> > > > and ask driver to set it up before use.
> > > >
> > > Yes. This is equally good idea.
> > > May be a pci_msix_config on PF device will be good idea indicating that,
> > always have all the VFs with zero default vector.
> > > And user will setup the vector before giving the VF to the driver.
> > >
> > > > It's strictly less work for the device so I find it hard to believe
> > > > it's impossible to implement.
> > > >
> > > It is less work. This is fine.
> > > Enabling/disabling individual VF is a completely different mechanics.
> > 
> > Well. you seem to already allow allocating 0 vectors to a VF, which will break
> > most drivers effectively disabling it.
> Allocating 0 vectors is not to break driver. Not sure why you use the term break.

just that most drivers won't work if device can not drive any interrupt.

> > I just this it's best to formalize this.
> I liked the idea of above 8 steps example, this will be efficient for the user.

step 1 is problematic though.

> > > > > > IOW we would have two commands, enable and disable, an enabled
> > > > > > VFs has to be supplied with all of its configuration.
> > > > > >
> > > > > I wish best to those who will attempt this.
> > > >
> > > > It's possible that you will end up being that person, we need to
> > > > have the interface that's generic and clear enough even if your
> > > > hardware does not benefit.
> > > >
> > > > > Devices that we experience are able to handle SR-IOV scale for
> > > > > several
> > > > hundreds of VFs without need to undergo enable-disable individual VFs.
> > > >
> > > > OK... and is it true that at the same time they do not support
> > > > enough MSI-X vectors per VF such that we need complex machinery to
> > control them?
> > > >
> > > Yes.
> > >
> > > > What is the motivation for this proposal again?
> > > >
> > > Efficiently distribute device resources among large number of VFs.
> > 
> > So what's the total # of vectors supported and what's the # of VFs then?
> 
> One example, 256 VFs. total of 2K MSI-X vectors among these 256 VFs.

OK so 128 per VF by default. So pls think of a set of commands where
we can manage most reasonable tasks without resorting to hundreds of
commands.


> > > When to pci capabilities, where to store in driver structure, when to update
> > them, when to mark those capabilities dirty and re-read it.
> > > All above is not defined by PCI spec; it varies from OS to OS. Virtio spec can
> > give guidance of what to expect so that OS can implement above for msix
> > config update.
> > 
> > All I'm saying it let's drop the "beyond the scope" thing and in fact this whole
> > paragraph does not seem to add anything useful to the spec.
> > 
> Oh, you suggested to describe this details in v1 in the spec. 

Wall this does is try to say that you won't though.

> > Maybe you are trying to say that the driver or systems software needs to do
> > some kind of synchronization - is that it?
> Yes.
> > It is not clear from this description what are you talking about, what is synched
> > with what, or what does "reflected in the PCI device" mean.
> 
> Below is what is written in proposed patch:
> 
> "MSI-X capability access by the subject PCI VF defined by the PCI specification must 
> reflect the new configuration in all relevant fields."

MUST I guess and do the conformance paragraph dance then.

and I think you mean "after setting the msix_count field" right?

otherwise this is ok, but I was referring to the beyond scope part,
not to this sentence at all.

> 
> All the configuration relevant to the MSI-X should reflect in necessary registers defined by PCI spec.
> 
> Probably it should be written as "All MSI-X related registers, capabilities and fields must reflect the new configuration".
> Is this better?

no, that sentence was mostly ok.

I think this thread has outlived its usefulness.
Please just post a new version and we can restart as at this point it's no longer
clear who is responding to what.


-- 
MST

Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Jason Wang]

在 2022/1/24 下午5:39, Max Gurtovoy 写道:
> A typical cloud provider SR-IOV use case is to create many VFs for
> use by guest VMs. The VFs may not be assigned to a VM until a user
> requests a VM of a certain size, e.g., number of CPUs. A VF may need
> MSI-X vectors proportional to the number of CPUs in the VM, but there is
> no standard way today in the spec to change the number of MSI-X vectors
> supported by a VF, although there are some operating systems that
> support this.
>
> Introduce new admin commands for a generic interrupt vector management
> for PCI VFs. For now, this mechanism will manage the MSI-X interrupt
> vectors assignments of a VF by its parent PF.
>
> These admin commands will be easily extended, if needed, for other types
> of interrupt vectors in the future with backward compatibility to old
> drivers and devices.
>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Signed-off-by: Max Gurtovoy <mgurtovoy@nvidia.com>
> ---
>   admin-virtq.tex | 150 +++++++++++++++++++++++++++++++++++++++++++++++-
>   content.tex     |  44 ++++++++++++++
>   2 files changed, 192 insertions(+), 2 deletions(-)
>
> diff --git a/admin-virtq.tex b/admin-virtq.tex
> index 1a41c22..b57b57d 100644
> --- a/admin-virtq.tex
> +++ b/admin-virtq.tex
> @@ -64,7 +64,13 @@ \section{Admin Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Admin
>   \hline \hline
>   0000h   & VIRTIO_ADMIN_CAPS_IDENTIFY    & M  \\
>   \hline
> -0001h - 7FFFh   & Generic admin cmds    & -  \\
> +0001h   & VIRTIO_ADMIN_PCI_SRIOV_ATTRS    & O  \\
> +\hline
> +0002h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET    & O  \\
> +\hline
> +0003h   & VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET    & O  \\
> +\hline
> +0004h - 7FFFh   & Generic admin cmds    & -  \\
>   \hline
>   8000h - FFFFh   & Reserved    & - \\
>   \hline
> @@ -78,7 +84,8 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
>   \begin{lstlisting}
>   struct virtio_admin_caps_identify_result {
>          /*
> -        * caps[0] - Bit 0x0 - Bit 0x7 are reserved
> +        * caps[0] - Bit 0x0 - if set, VF MSI-X control supported
> +        *           Bit 0x1 - Bit 0x7 are reserved
>           * caps[1] - Bit 0x0 - Bit 0x7 are reserved
>           * caps[2] - Bit 0x0 - Bit 0x7 are reserved
>           * ....
> @@ -87,3 +94,142 @@ \subsection{VIRTIO ADMIN CAPS IDENTIFY command}\label{sec:Basic Facilities of a
>          u8 caps[8192];
>   };
>   \end{lstlisting}
> +
> +For more details on VF MSI-X configuration support see section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}.
> +
> +\subsection{VIRTIO ADMIN PCI SRIOV ATTRS command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command}
> +


So this is all for PCI transport, do we need to

1) move this to PCI transport

or

2) use non PCI specific termiology (PF/VF) here

?


> +The VIRTIO_ADMIN_PCI_SRIOV_ATTRS command has no command specific data set by the driver.
> +This command upon success, returns a data buffer that describes information about PCI SRIOV
> +related capabilities and attributes for the device. This command can be supported only by
> +PCI devices that supports Single Root I/O Virtualization.
> +This information is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_sriov_attrs_result {
> +        /* For compatibility - indicates which of the below fields are valid (1 means valid):
> +         * Bit 0x0 - vfs_free_msix_count
> +         * Bit 0x1 - per_vf_max_msix_count
> +         * Bits 0x2 - 0x3F - reserved for future fields
> +         */


So we have several layers of compatibilities:

1) feature bits
2) cap bitmap / identifiy
3) attrs_mask (no negotiation so it's unclear that how the compatibility 
can work here).

This looks like a burden for the management.

Is it really helpful to have a device that can only return 
free_msix_count but not per_vf_max_msix_count?

And I think it's better to explicit split the commands then we can have 
dedicated commands for MSI instead of trying to mix all SRIOV related 
attributes into the same command.


> +        le64 attrs_mask;
> +        /* Number of free msix vectors in the global msix pool for VFs */
> +        le32 vfs_free_msix_count;
> +        /* Max number of msix vectors that can be assigned for a single VF */
> +        le16 per_vf_max_msix_count;
> +};
> +\end{lstlisting}
> +
> +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command}
> +
> +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET command is used to modify the interrupt vectors
> +count for a PCI virtual function. The command specific data set by the driver is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_set_data {
> +        /* The virtual function number */
> +        le16 vf_number;
> +        /* For compatibility - indicates which of the below properties should be
> +         * modified (1 means that field should be modified):
> +         * Bit 0x0 - msix_count
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +        /* The amount of MSI-X interrupt vectors */
> +        le16 msix_count;
> +};
> +\end{lstlisting}


I think it's better to have a dedicated command to set the msix_vectors, 
then there's no need for the attrs_mask.


> +
> +The following table describes the command specific error codes codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\


Does it mean it exceeds the totalVFs? We need an accurate definition on 
"invalid" here.


> +\hline
> +01h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED    & MSI-X count exceed the max value per VF  \\


I guess it's actually per_vf_max_msix_count? What if the device doesn't give us per_vf_max_msix_count (it's odd but it's allowed in this propsal if I was not wrong).


> +\hline
> +02h   & VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC    & MSI-X count exceed the free pool resources   \\
> +\hline


Do we need to define "pool" here?


> +03h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE    & VF is already in use, operation failed   \\


How do you define "VF is already in use"? Is this command only 
applicable when sriov_numvfs in the capability is zero? How about the 
initial associated VF via initial_vfs?


> +\hline
> +04h - FFh   & Reserved    & -  \\
> +\hline
> +\end{tabular}
> +
> +\begin{note}
> +{vf_number can't be greater than NumVFs value as defined in the PCI specification
> +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
> +is returned when vf_number is out of the range.}


Should be TotalVFs in the SR-IOV extended capability.


> +\end{note}
> +
> +\begin{note}
> +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_COUNT_EXCEED
> +is returned when the amount of MSI-X to assign exceed the maximum value that can be
> +assigned to a single VF.}
> +\end{note}
> +
> +\begin{note}
> +{A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_MSIX_POOL_NO_RSC
> +is returned when the amount of MSI-X to assign exceed the amount of free MSI-X
> +vectors in the global pool.}
> +\end{note}
> +
> +This command has no command specific result set by the device. Upon success, the device guarantees
> +that all the requested properties were modified to the given values. Otherwise, error will be returned.
> +
> +\begin{note}
> +{Before setting msix_count property the virtual/managed device (VF) shall be un-initialized and may not be used by the driver.
> +Otherwise, a command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_IN_USE will be returned.}


How did the device know whether or not VF is not used by any driver?


> +\end{note}
> +
> +\subsection{VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}\label{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command}
> +
> +The VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET command is used to obtain the values of the VFs
> +interrupt vectors configuration.
> +The command specific data set by the driver is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_get_data {
> +        /* The virtual function number */
> +        le16 vf_number;
> +        /* For compatibility - indicates which of the below properties should be
> +         * queried (1 means that field should be queried):
> +         * Bit 0x0 - msix_count (The amount of MSI-X interrupt vectors)
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +};
> +\end{lstlisting}
> +
> +The following table describes the command specific error codes codes:
> +
> +\begin{tabular}{|l|l|l|}
> +\hline
> +Opcode & Status & Description \\
> +\hline \hline
> +00h   & VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID    & invalid VF number  \\
> +\hline
> +01h - FFh   & Reserved    & -  \\
> +\hline
> +\end{tabular}
> +
> +\begin{note}
> +{vf_number can't be greater than NumVFs value as defined in the PCI specification


I guess for NumVFs you meant the TotalVFs in SR-IOV extended capability.


> +or smaller than 1. A command specific error status code VIRTIO_ADMIN_CS_ERR_PCI_VF_NUM_INVALID
> +is returned when vf_number is out of the range.}
> +\end{note}
> +
> +This command, upon success, returns a data buffer that describes the properties that were requested
> +and their values for the subject virtio VF device according to the given vf_number.
> +This information is of form:
> +\begin{lstlisting}
> +struct virtio_admin_pci_vf_interrupt_config_get_result {
> +        /* For compatibility - indicates which of the below fields were returned
> +         * (1 means that field was returned):
> +         * Bit 0x0 - msix_count
> +         * Bits 0x1 - 0x3F - reserved for future fields
> +         */
> +        le64 attrs_mask;
> +        /* The amount of MSI-X interrupt vectors currently assigned to the VF */
> +        le16 msix_count;
> +};
> +\end{lstlisting}
> diff --git a/content.tex b/content.tex
> index 18a5c66..2dd1d46 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -1734,6 +1734,50 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
>       \end{itemize}
>   \end{itemize}
>   
> +\subsection{PCI-specific Admin capabilities}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin capabilities}
> +
> +This documents the group of Admin capabilities for PCI virtio devices. Each capability is
> +implemented using one or more Admin queue commands.
> +
> +\subsubsection{VF MSI-X control}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Admin command set / VF MSI-X control}
> +
> +This capability enables a virtio PCI PF device to control the assignment of MSI-X interrupt vectors
> +for its managed VFs. Capable devices will need to set bit 0x0 of caps[0] in the result of VIRTIO_ADMIN_CAPS_IDENTIFY
> +admin command. See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN CAPS IDENTIFY command}
> +for more details.
> +
> +Capable devices will also need to implement VIRTIO_ADMIN_PCI_SRIOV_ATTRS, VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET and
> +VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin commands.
> +
> +In the result of VIRTIO_ADMIN_PCI_SRIOV_ATTRS admin command, a capable device should return the number of free msix vectors
> +in the global msix pool for its VFs in \field{vfs_free_msix_count} field and also the maximal number of msix vectors that
> +can be assigned for a single VF in \field{per_vf_max_msix_count} field. In addition, bit 0x0 and bit 0x1 should be set
> +to indicate on the validity of the other 2 fields in the \field{attrs_mask} field of the result buffer.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI SRIOV ATTRS command} for more
> +details.
> +
> +A PCI PF device may assign default number of MSI-X vectors to the enabled PCI VFs. Also, using VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET
> +it is possible to update the default MSI-X assignment for a specific VF.
> +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET admin command, a driver set the virtual function number in
> +\field{vf_number} and the amount of MSI-X interrupt vectors to configure to the subject virtual function in \field{msix_count}.
> +In addition, bit 0x0 in the \field{attrs_mask} field should be set. A successful operation guarantees that the requested
> +amount of MSI-X interrupt vectors was assigned to the subject virtual function.
> +Also, a successful operation guarantees that the MSI-X capability access by the subject PCI VF defined by the PCI specification must reflect
> +the new configuration in all relevant fields. For example, by default if the PCI VF has been assigned 4 MSI-X vectors, and VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_SET increases the MSI-X vectors to 8; on this change, reading Table size field of the MSI-X message control
> +register should reflect a value of 7.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG SET command} for more details.
> +
> +It is beyond the scope of the virtio specification to define necessary synchronization in system software to ensure that a virtio PCI VF device
> +interrupt configuration modification is reflected in the PCI device. However, it is expected that any modern system software implementing virtio
> +drivers and PCI subsystem should ensure that any changes occurring in the VF interrupt configuration is either updated in the PCI VF device or
> +such configuration fails.
> +
> +In the data of VIRTIO_ADMIN_PCI_VF_INTERRUPT_CONFIG_GET admin command, a driver should set the virtual function number in
> +\field{vf_number}. In addition, bit 0x0 in the \field{attrs_mask} field should be set to indicate requested output fields in
> +the result from the device. In the result of a successful operation, the amount of MSI-X interrupt vectors that is currently
> +assigned to the subject virtual function is returned by the device in \field{msix_count} field. In addition, bit 0x0 in the \field{attrs_mask} field should be set by the device to indicate on the validity of \field{msix_count} field.
> +See section \ref{sec:Basic Facilities of a Virtio Device / Admin Virtqueues / VIRTIO ADMIN PCI VF INTERRUPT CONFIG GET command} for more details.
> +


I think we should have a guidance on how to provision #msix_vectors step 
by step. E.g:

1) Do we need to do that before or after the VF provisioning
2) Can the value survive when 0 is wrote to sriov_numvfs

etc.

Thanks


>   \section{Virtio Over MMIO}\label{sec:Virtio Transport Options / Virtio Over MMIO}
>   
>   Virtual environments without PCI support (a common situation in

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, January 27, 2022 9:06 AM

[...]

> 
> And I think it's better to explicit split the commands then we can have
> dedicated commands for MSI instead of trying to mix all SRIOV related
> attributes into the same command.

It is not mixed. Its clubbed together so that driver-device communication can be minimal and still flexible.

For example in [1] it has, command like,
ip link add vx0 type vxlan id 100 local 1.1.1.1 remote 2.2.2.2 dev eth0 dstport 4789

it would be inefficient do it as,
$ ip link add vx0 type vxlan id 100
$ ip link vx0 local 1.1.1.1 
$ ip link vx0 remote 2.2.2.2 
$ ip link vx0 dev eth0 
$ ip link vx0 dstport 4789

[1] https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking#macvlan

Similarly, we prefer the ability and flexibility to set more fields in one command.

> I think it's better to have a dedicated command to set the msix_vectors, then
> there's no need for the attrs_mask.
> 
Main design principle in having attr_mask is to avoid issuing multiple commands for related/adjacent attributes.
Lesser the number of commands serviced, more efficient the overall system is.
Hence the attribute mask gives the flexibility to set one or more.

[virtio-comment] Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Jason Wang]

On Thu, Jan 27, 2022 at 1:22 PM Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Thursday, January 27, 2022 9:06 AM
>
> [...]
>
> >
> > And I think it's better to explicit split the commands then we can have
> > dedicated commands for MSI instead of trying to mix all SRIOV related
> > attributes into the same command.
>
> It is not mixed. Its clubbed together so that driver-device communication can be minimal and still flexible.
>
> For example in [1] it has, command like,
> ip link add vx0 type vxlan id 100 local 1.1.1.1 remote 2.2.2.2 dev eth0 dstport 4789
>
> it would be inefficient do it as,
> $ ip link add vx0 type vxlan id 100
> $ ip link vx0 local 1.1.1.1
> $ ip link vx0 remote 2.2.2.2
> $ ip link vx0 dev eth0
> $ ip link vx0 dstport 4789
>
> [1] https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking#macvlan

vxlan part actually.

>
> Similarly, we prefer the ability and flexibility to set more fields in one command.

I understood this motivation. But what I want to say is:

struct virtio_cmd {
    u64 attr_mask;
    field_X;
    field_Y;
    field_Z;
}

What's the default value if one of the X,Y, Z is not specified? How to
get those default values?

And suppose we add a new field

struct virtio_cmd {
    u64 attr_mask;
    field_X;
    field_Y;
    field_Z;
    field_M;
}

How can the driver know it can use field_M? And if field_M is
supported in src but not dst, how can we keep the migration
compatibility?

Would it be simple to just mandate all the fields in this case?

Thanks


>
> > I think it's better to have a dedicated command to set the msix_vectors, then
> > there's no need for the attrs_mask.
> >
> Main design principle in having attr_mask is to avoid issuing multiple commands for related/adjacent attributes.
> Lesser the number of commands serviced, more efficient the overall system is.
> Hence the attribute mask gives the flexibility to set one or more.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Friday, January 28, 2022 8:54 AM
> vxlan part actually.

Right.
> 
> >
> > Similarly, we prefer the ability and flexibility to set more fields in one
> command.
> 
> I understood this motivation. But what I want to say is:
> 
> struct virtio_cmd {
>     u64 attr_mask;
>     field_X;
>     field_Y;
>     field_Z;
> }
> 
> What's the default value if one of the X,Y, Z is not specified? 
If minimum default values are not provided, HV cannot proceed to do configuration and ENOSUPPORT error code is returned by kernel to user.

> How to get those default values?
>
By doing identify or _get respective command.
 
> And suppose we add a new field
> 
> struct virtio_cmd {
>     u64 attr_mask;
>     field_X;
>     field_Y;
>     field_Z;
>     field_M;
> }
> 
> How can the driver know it can use field_M? 
If attribute mask is present for field_M, driver can use it. 

> And if field_M is supported in src
> but not dst, how can we keep the migration compatibility?
This is HV level command to query and provision a VF.
Migration compatibility is high level check where it will identify on which HV to migrate where I can provision X vectors for a VF.

> 
> Would it be simple to just mandate all the fields in this case?
>
May be, attribute mask is basically to let expand structure without inventing new fields.
For a given command few minimal attributes to be set and driver can verify, that if they are not, a given functionality is unsupported.

Re: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Jason Wang]

在 2022/1/28 上午11:30, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Friday, January 28, 2022 8:54 AM
>> vxlan part actually.
> Right.
>>> Similarly, we prefer the ability and flexibility to set more fields in one
>> command.
>>
>> I understood this motivation. But what I want to say is:
>>
>> struct virtio_cmd {
>>      u64 attr_mask;
>>      field_X;
>>      field_Y;
>>      field_Z;
>> }
>>
>> What's the default value if one of the X,Y, Z is not specified?
> If minimum default values are not provided, HV cannot proceed to do configuration and ENOSUPPORT error code is returned by kernel to user.


Do we need to document the minimum default value?


>
>> How to get those default values?
>>
> By doing identify or _get respective command.


This can only work if there's no attr_mask in get command. If the device 
can selectively provides us partial information we can't do 
compatibility check since we don't know its value.


>   
>> And suppose we add a new field
>>
>> struct virtio_cmd {
>>      u64 attr_mask;
>>      field_X;
>>      field_Y;
>>      field_Z;
>>      field_M;
>> }
>>
>> How can the driver know it can use field_M?
> If attribute mask is present for field_M, driver can use it.


So the driver need first use get then try to use set?


>
>> And if field_M is supported in src
>> but not dst, how can we keep the migration compatibility?
> This is HV level command to query and provision a VF.
> Migration compatibility is high level check where it will identify on which HV to migrate where I can provision X vectors for a VF.


By probing capability of attr_mask in each possible commands?


>
>> Would it be simple to just mandate all the fields in this case?
>>
> May be, attribute mask is basically to let expand structure without inventing new fields.


It could be done via new command.

Thanks


> For a given command few minimal attributes to be set and driver can verify, that if they are not, a given functionality is unsupported.
>

RE: [PATCH v2 4/4] Add support for MSI-X vectors configuration for PCI VFs

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Friday, January 28, 2022 9:06 AM
> 
> 在 2022/1/28 上午11:30, Parav Pandit 写道:
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Friday, January 28, 2022 8:54 AM vxlan part actually.
> > Right.
> >>> Similarly, we prefer the ability and flexibility to set more fields
> >>> in one
> >> command.
> >>
> >> I understood this motivation. But what I want to say is:
> >>
> >> struct virtio_cmd {
> >>      u64 attr_mask;
> >>      field_X;
> >>      field_Y;
> >>      field_Z;
> >> }
> >>
> >> What's the default value if one of the X,Y, Z is not specified?
> > If minimum default values are not provided, HV cannot proceed to do
> configuration and ENOSUPPORT error code is returned by kernel to user.
> 
> 
> Do we need to document the minimum default value?
>
I guess, not because attr_mask is expected to grow and there may be unrelated attributes to it.
 
> 
> >
> >> How to get those default values?
> >>
> > By doing identify or _get respective command.
> 
> 
> This can only work if there's no attr_mask in get command. If the device can
> selectively provides us partial information we can't do compatibility check
> since we don't know its value.
> 
Device will follow some sane implementation.
If device which wants to support msix config for a VF, it will report right values in generic identify and get cmd of device.
A simple error check of bit mask will be enough in driver.

> 
> >
> >> And suppose we add a new field
> >>
> >> struct virtio_cmd {
> >>      u64 attr_mask;
> >>      field_X;
> >>      field_Y;
> >>      field_Z;
> >>      field_M;
> >> }
> >>
> >> How can the driver know it can use field_M?
> > If attribute mask is present for field_M, driver can use it.
> 
> 
> So the driver need first use get then try to use set?
>
No. the flow is:
1. AQ feature bit is negotiated
2. Create AQ
3. Do identify cmd and cache the device capabilities
A bit says that VF msix modification supported.
4. user queried all VFs current vector distribution (optimize this bit later, but consider this for now)
If attr.mask.bitX not set, fail the user request.
4. user requests to modify VF-A vectors from (8 to 4) uses attr.mask.bitX.
 
> 
> >
> >> And if field_M is supported in src
> >> but not dst, how can we keep the migration compatibility?
> > This is HV level command to query and provision a VF.
> > Migration compatibility is high level check where it will identify on which HV
> to migrate where I can provision X vectors for a VF.
> 
> 
> By probing capability of attr_mask in each possible commands?
>
No. not in each possible commands.
Identify command gives the capabilities supported in step #3 above to see if PFs on source and destination HV support it or not.
 
> 
> >
> >> Would it be simple to just mandate all the fields in this case?
> >>
> > May be, attribute mask is basically to let expand structure without inventing
> new fields.
> 
> 
> It could be done via new command.
Sure, it can be.
We want to avoid too many commands, if they can be clubbed in single command when starting a VF.