* [virtio-dev] [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-06 0:01 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
This short series introduces legacy registers access commands for the owner
group member PCI PF to access the legacy registers of the member VFs.
If in future any SIOV devices to support legacy registers, they
can be easily supported using same commands by using the group
member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Patch summary:
--------------
patch-1 adds administrative virtuqueue commands
patch-2 adds its conformance section
This short series is on top of latest work [1] from Michael.
It uses the newly introduced administrative virtqueue facility with 3 new
commands which uses the existing virtio_admin_cmd.
[1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements can be solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
The third command suggested by Jason queries the VF device's driver
notification region.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it as native device in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
v0->v1:
- addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
- far more simpler design than MMR access
- removed complexities of MMR device ids
- removed complexities of MMR registers and extended capabilities
- dropped adding new extended capabilities because if if they are
added, a pci device still needs to have existing capabilities
in the legacy configuration space and hypervisor driver do not
need to access them
Parav Pandit (2):
transport-pci: Introduce legacy registers access commands
transport-pci: Add legacy register access conformance section
admin.tex | 5 +-
conformance.tex | 2 +
transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
transport-pci.tex | 2 +
4 files changed, 141 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-06 0:01 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
This short series introduces legacy registers access commands for the owner
group member PCI PF to access the legacy registers of the member VFs.
If in future any SIOV devices to support legacy registers, they
can be easily supported using same commands by using the group
member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Patch summary:
--------------
patch-1 adds administrative virtuqueue commands
patch-2 adds its conformance section
This short series is on top of latest work [1] from Michael.
It uses the newly introduced administrative virtqueue facility with 3 new
commands which uses the existing virtio_admin_cmd.
[1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements can be solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
The third command suggested by Jason queries the VF device's driver
notification region.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it as native device in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
v0->v1:
- addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
- far more simpler design than MMR access
- removed complexities of MMR device ids
- removed complexities of MMR registers and extended capabilities
- dropped adding new extended capabilities because if if they are
added, a pci device still needs to have existing capabilities
in the legacy configuration space and hypervisor driver do not
need to access them
Parav Pandit (2):
transport-pci: Introduce legacy registers access commands
transport-pci: Add legacy register access conformance section
admin.tex | 5 +-
conformance.tex | 2 +
transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
transport-pci.tex | 2 +
4 files changed, 141 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-06 0:01 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
This patch introduces legacy registers access commands for the owner
group member PCI PF to access the legacy registers of the member VFs.
If in future any SIOV devices to support legacy registers, they
can be easily supported using same commands by using the group
member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements can be solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
The third command suggested by Jason queries the VF device's driver
notification region.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it as native device in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
---
admin.tex | 5 +-
transport-pci-legacy-regs.tex | 106 ++++++++++++++++++++++++++++++++++
transport-pci.tex | 2 +
3 files changed, 112 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
diff --git a/admin.tex b/admin.tex
index 47556f3..8fef32d 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,10 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\hline \hline
0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
+0x0002 & VIRTIO_ADMIN_CMD_LREG_WRITE & Write legacy registers of a member device \\
+0x0003 & VIRTIO_ADMIN_CMD_LREG_READ & Read legacy registers of a member device \\
+0x0004 & VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY & Read the queue notification offset for legacy interface \\
+0x0005 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
\hline
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
new file mode 100644
index 0000000..9088fe6
--- /dev/null
+++ b/transport-pci-legacy-regs.tex
@@ -0,0 +1,106 @@
+\subsection{Legacy Interfaces: SR-IOV VFs Registers Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+As described in PCIe base specification \hyperref[intro:PCIe]{[PCIe]} PCI VFs
+do not support the I/O space BAR. A PCI VF hardware device that can support
+legacy interfaces cannot be passthrough to the guest virtual machine
+due to this limitation. To have a transitional virtio device in the guest
+virtual machine, a PCI PF device as a group owner may support accessing its
+group member PCI VF device's legacy registers using an administrative
+virtqueue. This enables minimal intervention in the hypervisor only for
+the purpose of legacy registers access.
+
+A hypervisor software accesses the legacy configuration and device specific
+registers requested by the guest virtual machine using an administrative
+virtqueue. Even though virtqueue driver notifications can be communicated
+through administrative virtqueue, if the PCI PF and VF devices support such
+notifications using a memory-mapped operation, such driver notifications
+are sent using a device defined notification region.
+
+The group owner PCI PF device can optionally enable the driver to access
+its member PCI VFs devices legacy common configuration and device configuration
+registers using an administration virtqueue. Such a PCI PF device supports
+the following commands:
+
+\begin{enumerate}
+\item Legacy Registers Write Command
+\item Legacy Registers Read Command
+\item Legacy Queue Notify Offset Query Command
+\end{enumerate}
+
+\subsubsection{Legacy Registers Write Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Registers Write Command}
+
+The Legacy Registers Write Command follows \field{struct virtio_admin_cmd}.
+This command writes legacy registers of a member VF device.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LREG_WRITE.
+
+The command VIRTIO_ADMIN_CMD_LREG_WRITE uses following structure for
+\field{command_specific_data}:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_wr_data {
+ u8 offset; /* Starting byte offset of the register(s) to write */
+ u8 register[];
+};
+\end{lstlisting}
+
+This command does not have any command specific result.
+
+\subsubsection{Legacy Registers Read Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Registers Read Command}
+
+The Legacy Registers Read Command follows \field{struct virtio_admin_cmd}.
+This command reads legacy registers of a member VF device.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LREG_READ.
+
+The command VIRTIO_ADMIN_CMD_LREG_READ uses following listed structure for
+\field{command_specific_data}:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_rd_data {
+ u8 offset; /* Starting byte offset of the register to read */
+};
+\end{lstlisting}
+
+When command completes successfully, \field{command_specific_result}
+uses following listed structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_rd_result {
+ u8 registers[];
+};
+\end{lstlisting}
+
+\subsubsection{Legacy Queue Notify Offset Query Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Queue Notify Offset Query Command}
+
+This command returns the notify offset of the member VF for virtqueue
+driver notifications. This command follows \field{struct virtio_admin_cmd}.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY.
+There is no command specific data for this command.
+
+When command completes successfully, \field{command_specific_result}
+uses following listed structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lq_notify_query_result {
+ u8 bar; /* PCI BAR number of the member VF */
+ u8 reserved[7];
+ le64 offset; /* Byte offset within the BAR */
+};
+\end{lstlisting}
+
+The driver that may use the driver notifications region of the VF device
+returned in this result likely attain higher performance or the drier may use
+the VIRTIO_ADMIN_CMD_LREG_WRITE command.
+
+\begin{note}
+The device and driver must encode and decode legacy device specific registers
+using little endian format. Per PCI VF device level big endian format support
+is left for the future.
+\end{note}
+
+\begin{note}
+The PCI VF device should not use PCI BAR 0 when it prefers to support
+legacy interface registers access using its group owner PF. This enables
+hypervisor software to operate with least complexities to compose a legacy
+interface I/O space BAR and passthrough other PCI BARs and PCI device
+capabilities to the guest virtual machine without any translation.
+\end{note}
diff --git a/transport-pci.tex b/transport-pci.tex
index ff889d3..9b58a57 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1179,3 +1179,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
re-examine the configuration space to see what changed.
\end{itemize}
\end{itemize}
+
+\input{transport-pci-legacy-regs.tex}
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 252+ messages in thread
* [virtio-comment] [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-06 0:01 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
This patch introduces legacy registers access commands for the owner
group member PCI PF to access the legacy registers of the member VFs.
If in future any SIOV devices to support legacy registers, they
can be easily supported using same commands by using the group
member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements can be solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
The third command suggested by Jason queries the VF device's driver
notification region.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it as native device in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
---
admin.tex | 5 +-
transport-pci-legacy-regs.tex | 106 ++++++++++++++++++++++++++++++++++
transport-pci.tex | 2 +
3 files changed, 112 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
diff --git a/admin.tex b/admin.tex
index 47556f3..8fef32d 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,10 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\hline \hline
0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
+0x0002 & VIRTIO_ADMIN_CMD_LREG_WRITE & Write legacy registers of a member device \\
+0x0003 & VIRTIO_ADMIN_CMD_LREG_READ & Read legacy registers of a member device \\
+0x0004 & VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY & Read the queue notification offset for legacy interface \\
+0x0005 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
\hline
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
new file mode 100644
index 0000000..9088fe6
--- /dev/null
+++ b/transport-pci-legacy-regs.tex
@@ -0,0 +1,106 @@
+\subsection{Legacy Interfaces: SR-IOV VFs Registers Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+As described in PCIe base specification \hyperref[intro:PCIe]{[PCIe]} PCI VFs
+do not support the I/O space BAR. A PCI VF hardware device that can support
+legacy interfaces cannot be passthrough to the guest virtual machine
+due to this limitation. To have a transitional virtio device in the guest
+virtual machine, a PCI PF device as a group owner may support accessing its
+group member PCI VF device's legacy registers using an administrative
+virtqueue. This enables minimal intervention in the hypervisor only for
+the purpose of legacy registers access.
+
+A hypervisor software accesses the legacy configuration and device specific
+registers requested by the guest virtual machine using an administrative
+virtqueue. Even though virtqueue driver notifications can be communicated
+through administrative virtqueue, if the PCI PF and VF devices support such
+notifications using a memory-mapped operation, such driver notifications
+are sent using a device defined notification region.
+
+The group owner PCI PF device can optionally enable the driver to access
+its member PCI VFs devices legacy common configuration and device configuration
+registers using an administration virtqueue. Such a PCI PF device supports
+the following commands:
+
+\begin{enumerate}
+\item Legacy Registers Write Command
+\item Legacy Registers Read Command
+\item Legacy Queue Notify Offset Query Command
+\end{enumerate}
+
+\subsubsection{Legacy Registers Write Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Registers Write Command}
+
+The Legacy Registers Write Command follows \field{struct virtio_admin_cmd}.
+This command writes legacy registers of a member VF device.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LREG_WRITE.
+
+The command VIRTIO_ADMIN_CMD_LREG_WRITE uses following structure for
+\field{command_specific_data}:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_wr_data {
+ u8 offset; /* Starting byte offset of the register(s) to write */
+ u8 register[];
+};
+\end{lstlisting}
+
+This command does not have any command specific result.
+
+\subsubsection{Legacy Registers Read Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Registers Read Command}
+
+The Legacy Registers Read Command follows \field{struct virtio_admin_cmd}.
+This command reads legacy registers of a member VF device.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LREG_READ.
+
+The command VIRTIO_ADMIN_CMD_LREG_READ uses following listed structure for
+\field{command_specific_data}:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_rd_data {
+ u8 offset; /* Starting byte offset of the register to read */
+};
+\end{lstlisting}
+
+When command completes successfully, \field{command_specific_result}
+uses following listed structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lreg_rd_result {
+ u8 registers[];
+};
+\end{lstlisting}
+
+\subsubsection{Legacy Queue Notify Offset Query Command}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access / Legacy Queue Notify Offset Query Command}
+
+This command returns the notify offset of the member VF for virtqueue
+driver notifications. This command follows \field{struct virtio_admin_cmd}.
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY.
+There is no command specific data for this command.
+
+When command completes successfully, \field{command_specific_result}
+uses following listed structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_lq_notify_query_result {
+ u8 bar; /* PCI BAR number of the member VF */
+ u8 reserved[7];
+ le64 offset; /* Byte offset within the BAR */
+};
+\end{lstlisting}
+
+The driver that may use the driver notifications region of the VF device
+returned in this result likely attain higher performance or the drier may use
+the VIRTIO_ADMIN_CMD_LREG_WRITE command.
+
+\begin{note}
+The device and driver must encode and decode legacy device specific registers
+using little endian format. Per PCI VF device level big endian format support
+is left for the future.
+\end{note}
+
+\begin{note}
+The PCI VF device should not use PCI BAR 0 when it prefers to support
+legacy interface registers access using its group owner PF. This enables
+hypervisor software to operate with least complexities to compose a legacy
+interface I/O space BAR and passthrough other PCI BARs and PCI device
+capabilities to the guest virtual machine without any translation.
+\end{note}
diff --git a/transport-pci.tex b/transport-pci.tex
index ff889d3..9b58a57 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1179,3 +1179,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
re-examine the configuration space to see what changed.
\end{itemize}
\end{itemize}
+
+\input{transport-pci-legacy-regs.tex}
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 252+ messages in thread
* [virtio-dev] [PATCH v2 2/2] transport-pci: Add legacy register access conformance section
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-06 0:01 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
Add device and driver conformanace section for legacy registers access
commands interface.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- removed duplicate and contradicting normatives to existing legacy
interface section
- added endianness normative for device specific region
- updated conformance links
---
conformance.tex | 2 ++
transport-pci-legacy-regs.tex | 27 +++++++++++++++++++++++++++
2 files changed, 29 insertions(+)
diff --git a/conformance.tex b/conformance.tex
index 01ccd69..3f2d49e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -109,6 +109,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes}
+\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
\end{itemize}
\conformance{\subsection}{MMIO Driver Conformance}\label{sec:Conformance / Driver Conformance / MMIO Driver Conformance}
@@ -194,6 +195,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
\end{itemize}
\conformance{\subsection}{MMIO Device Conformance}\label{sec:Conformance / Device Conformance / MMIO Device Conformance}
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
index 9088fe6..4db578f 100644
--- a/transport-pci-legacy-regs.tex
+++ b/transport-pci-legacy-regs.tex
@@ -104,3 +104,30 @@ \subsubsection{Legacy Queue Notify Offset Query Command}\label{sec:Virtio Transp
interface I/O space BAR and passthrough other PCI BARs and PCI device
capabilities to the guest virtual machine without any translation.
\end{note}
+
+\devicenormative{\paragraph}{SR-IOV VFs Legacy Registers Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+If the PCI PF device supports legacy registers access for its group members,
+the device MUST set all corresponding bits for commands VIRTIO_ADMIN_CMD_LREG_WRITE,
+VIRTIO_ADMIN_CMD_LREG_READ and VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY in
+command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
+\field{device_admin_cmd_opcodes}.
+
+The device MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LREG_WRITE and VIRTIO_ADMIN_CMD_LREG_READ
+commands for the invalid offset which is outside the legacy registers
+address range.
+
+The PCI VF device SHOULD NOT use PCI BAR 0 when it prefers to support
+legacy interface registers access.
+
+\drivernormative{\paragraph}{SR-IOV VFs Legacy Registers Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+The driver MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The driver SHOULD send commands VIRTIO_ADMIN_CMD_LREG_WRITE and
+VIRTIO_ADMIN_CMD_LREG_READ with a valid offset which is in the legacy
+registers address range.
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 252+ messages in thread
* [virtio-comment] [PATCH v2 2/2] transport-pci: Add legacy register access conformance section
@ 2023-05-06 0:01 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-06 0:01 UTC (permalink / raw)
To: mst, virtio-dev, cohuck, david.edmondson
Cc: sburla, jasowang, yishaih, maorg, virtio-comment, shahafs, Parav Pandit
Add device and driver conformanace section for legacy registers access
commands interface.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v1->v2:
- addressed comments from Michael
- removed duplicate and contradicting normatives to existing legacy
interface section
- added endianness normative for device specific region
- updated conformance links
---
conformance.tex | 2 ++
transport-pci-legacy-regs.tex | 27 +++++++++++++++++++++++++++
2 files changed, 29 insertions(+)
diff --git a/conformance.tex b/conformance.tex
index 01ccd69..3f2d49e 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -109,6 +109,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes}
+\item \ref{drivernormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
\end{itemize}
\conformance{\subsection}{MMIO Driver Conformance}\label{sec:Conformance / Driver Conformance / MMIO Driver Conformance}
@@ -194,6 +195,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Used Buffer Notifications}
\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Notification of Device Configuration Changes}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
\end{itemize}
\conformance{\subsection}{MMIO Device Conformance}\label{sec:Conformance / Device Conformance / MMIO Device Conformance}
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
index 9088fe6..4db578f 100644
--- a/transport-pci-legacy-regs.tex
+++ b/transport-pci-legacy-regs.tex
@@ -104,3 +104,30 @@ \subsubsection{Legacy Queue Notify Offset Query Command}\label{sec:Virtio Transp
interface I/O space BAR and passthrough other PCI BARs and PCI device
capabilities to the guest virtual machine without any translation.
\end{note}
+
+\devicenormative{\paragraph}{SR-IOV VFs Legacy Registers Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+If the PCI PF device supports legacy registers access for its group members,
+the device MUST set all corresponding bits for commands VIRTIO_ADMIN_CMD_LREG_WRITE,
+VIRTIO_ADMIN_CMD_LREG_READ and VIRTIO_ADMIN_CMD_LQ_NOTIFY_QUERY in
+command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
+\field{device_admin_cmd_opcodes}.
+
+The device MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LREG_WRITE and VIRTIO_ADMIN_CMD_LREG_READ
+commands for the invalid offset which is outside the legacy registers
+address range.
+
+The PCI VF device SHOULD NOT use PCI BAR 0 when it prefers to support
+legacy interface registers access.
+
+\drivernormative{\paragraph}{SR-IOV VFs Legacy Registers Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interfaces: SR-IOV VFs Registers Access}
+
+The driver MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The driver SHOULD send commands VIRTIO_ADMIN_CMD_LREG_WRITE and
+VIRTIO_ADMIN_CMD_LREG_READ with a valid offset which is in the legacy
+registers address range.
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-06 2:31 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-06 2:31 UTC (permalink / raw)
To: Parav Pandit
Cc: mst, virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
>
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
So as replied in V1, I think it's not a good idea to invent commands
for a partial transport just for legacy devices. It's better either:
1) rebase or collaborate this work on top of the transport virtqueue
or
2) having a PCI over admin virtqueue transport, since this proposal
has already had BAR access, we can add config space access then it is
self-contained so we don't need to go through every corner case like
inventing dedicated commands to accessing some function that is
duplicated with capabilities. It will become yet another transport and
legacy support is just a good byproduct.
Thanks
>
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
>
> Parav Pandit (2):
> transport-pci: Introduce legacy registers access commands
> transport-pci: Add legacy register access conformance section
>
> admin.tex | 5 +-
> conformance.tex | 2 +
> transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 +
> 4 files changed, 141 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-06 2:31 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-06 2:31 UTC (permalink / raw)
To: Parav Pandit
Cc: mst, virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
>
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
So as replied in V1, I think it's not a good idea to invent commands
for a partial transport just for legacy devices. It's better either:
1) rebase or collaborate this work on top of the transport virtqueue
or
2) having a PCI over admin virtqueue transport, since this proposal
has already had BAR access, we can add config space access then it is
self-contained so we don't need to go through every corner case like
inventing dedicated commands to accessing some function that is
duplicated with capabilities. It will become yet another transport and
legacy support is just a good byproduct.
Thanks
>
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
>
> Parav Pandit (2):
> transport-pci: Introduce legacy registers access commands
> transport-pci: Add legacy register access conformance section
>
> admin.tex | 5 +-
> conformance.tex | 2 +
> transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 +
> 4 files changed, 141 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-07 9:04 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-07 9:04 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:33AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
one things I still don't see addressed here is support for
legacy interrupts. legacy driver can disable msix and
interrupts will be then sent.
how about a special command that is used when device would
normally send INT#x? it can also return ISR to reduce latency.
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
>
> Parav Pandit (2):
> transport-pci: Introduce legacy registers access commands
> transport-pci: Add legacy register access conformance section
>
> admin.tex | 5 +-
> conformance.tex | 2 +
> transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 +
> 4 files changed, 141 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-07 9:04 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-07 9:04 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:33AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
one things I still don't see addressed here is support for
legacy interrupts. legacy driver can disable msix and
interrupts will be then sent.
how about a special command that is used when device would
normally send INT#x? it can also return ISR to reduce latency.
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
>
> Parav Pandit (2):
> transport-pci: Introduce legacy registers access commands
> transport-pci: Add legacy register access conformance section
>
> admin.tex | 5 +-
> conformance.tex | 2 +
> transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 +
> 4 files changed, 141 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-06 2:31 ` [virtio-comment] " Jason Wang
@ 2023-05-07 13:44 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-07 13:44 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> > This short series introduces legacy registers access commands for the owner
> > group member PCI PF to access the legacy registers of the member VFs.
> >
> > If in future any SIOV devices to support legacy registers, they
> > can be easily supported using same commands by using the group
> > member identifiers of the future SIOV devices.
> >
> > More details as overview, motivation, use case are further described
> > below.
> >
> > Patch summary:
> > --------------
> > patch-1 adds administrative virtuqueue commands
> > patch-2 adds its conformance section
> >
> > This short series is on top of latest work [1] from Michael.
> > It uses the newly introduced administrative virtqueue facility with 3 new
> > commands which uses the existing virtio_admin_cmd.
> >
> > [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> >
> > Usecase:
> > --------
> > 1. A hypervisor/system needs to provide transitional
> > virtio devices to the guest VM at scale of thousands,
> > typically, one to eight devices per VM.
> >
> > 2. A hypervisor/system needs to provide such devices using a
> > vendor agnostic driver in the hypervisor system.
> >
> > 3. A hypervisor system prefers to have single stack regardless of
> > virtio device type (net/blk) and be future compatible with a
> > single vfio stack using SR-IOV or other scalable device
> > virtualization technology to map PCI devices to the guest VM.
> > (as transitional or otherwise)
> >
> > Motivation/Background:
> > ----------------------
> > The existing virtio transitional PCI device is missing support for
> > PCI SR-IOV based devices. Currently it does not work beyond
> > PCI PF, or as software emulated device in reality. Currently it
> > has below cited system level limitations:
> >
> > [a] PCIe spec citation:
> > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> >
> > [b] cpu arch citiation:
> > Intel 64 and IA-32 Architectures Software Developer’s Manual:
> > The processor’s I/O address space is separate and distinct from
> > the physical-memory address space. The I/O address space consists
> > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> >
> > [c] PCIe spec citation:
> > If a bridge implements an I/O address range,...I/O address range will be
> > aligned to a 4 KB boundary.
> >
> > Overview:
> > ---------
> > Above usecase requirements can be solved by PCI PF group owner accessing
> > its group member PCI VFs legacy registers using an admin virtqueue of
> > the group owner PCI PF.
> >
> > Two new admin virtqueue commands are added which read/write PCI VF
> > registers.
> >
> > The third command suggested by Jason queries the VF device's driver
> > notification region.
> >
> > Software usage example:
> > -----------------------
> > One way to use and map to the guest VM is by using vfio driver
> > framework in Linux kernel.
> >
> > +----------------------+
> > |pci_dev_id = 0x100X |
> > +---------------|pci_rev_id = 0x0 |-----+
> > |vfio device |BAR0 = I/O region | |
> > | |Other attributes | |
> > | +----------------------+ |
> > | |
> > + +--------------+ +-----------------+ |
> > | |I/O BAR to AQ | | Other vfio | |
> > | |rd/wr mapper | | functionalities | |
> > | +--------------+ +-----------------+ |
> > | |
> > +------+-------------------------+-----------+
> > | |
> > +----+------------+ +----+------------+
> > | +-----+ | | PCI VF device A |
> > | | AQ |-------------+---->+-------------+ |
> > | +-----+ | | | | legacy regs | |
> > | PCI PF device | | | +-------------+ |
> > +-----------------+ | +-----------------+
> > |
> > | +----+------------+
> > | | PCI VF device N |
> > +---->+-------------+ |
> > | | legacy regs | |
> > | +-------------+ |
> > +-----------------+
> >
> > 2. Virtio pci driver to bind to the listed device id and
> > use it as native device in the host.
> >
> > 3. Use it in a light weight hypervisor to run bare-metal OS.
> >
> > Please review.
> >
> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> > Signed-off-by: Parav Pandit <parav@nvidia.com>
> >
> > ---
> > changelog:
> > v1->v2:
> > - addressed comments from Michael
> > - added theory of operation
> > - grammar corrections
> > - removed group fields description from individual commands as
> > it is already present in generic section
> > - added endianness normative for legacy device registers region
> > - renamed the file to drop vf and add legacy prefix
> > - added overview in commit log
> > - renamed subsection to reflect command
>
> So as replied in V1, I think it's not a good idea to invent commands
> for a partial transport just for legacy devices. It's better either:
>
> 1) rebase or collaborate this work on top of the transport virtqueue
>
> or
>
> 2) having a PCI over admin virtqueue transport, since this proposal
> has already had BAR access, we can add config space access then it is
> self-contained so we don't need to go through every corner case like
> inventing dedicated commands to accessing some function that is
> duplicated with capabilities. It will become yet another transport and
> legacy support is just a good byproduct.
>
> Thanks
I thought so too originally. Unfortunately I now think that no, legacy is not
going to be a byproduct of transport virtqueue for modern -
it is different enough that it needs dedicated commands.
Consider simplest case, multibyte fields. Legacy needs multibyte write,
modern does not even need multibyte read.
> >
> > v0->v1:
> > - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> > - far more simpler design than MMR access
> > - removed complexities of MMR device ids
> > - removed complexities of MMR registers and extended capabilities
> > - dropped adding new extended capabilities because if if they are
> > added, a pci device still needs to have existing capabilities
> > in the legacy configuration space and hypervisor driver do not
> > need to access them
> >
> >
> > Parav Pandit (2):
> > transport-pci: Introduce legacy registers access commands
> > transport-pci: Add legacy register access conformance section
> >
> > admin.tex | 5 +-
> > conformance.tex | 2 +
> > transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> > transport-pci.tex | 2 +
> > 4 files changed, 141 insertions(+), 1 deletion(-)
> > create mode 100644 transport-pci-legacy-regs.tex
> >
> > --
> > 2.26.2
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-07 13:44 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-07 13:44 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> > This short series introduces legacy registers access commands for the owner
> > group member PCI PF to access the legacy registers of the member VFs.
> >
> > If in future any SIOV devices to support legacy registers, they
> > can be easily supported using same commands by using the group
> > member identifiers of the future SIOV devices.
> >
> > More details as overview, motivation, use case are further described
> > below.
> >
> > Patch summary:
> > --------------
> > patch-1 adds administrative virtuqueue commands
> > patch-2 adds its conformance section
> >
> > This short series is on top of latest work [1] from Michael.
> > It uses the newly introduced administrative virtqueue facility with 3 new
> > commands which uses the existing virtio_admin_cmd.
> >
> > [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> >
> > Usecase:
> > --------
> > 1. A hypervisor/system needs to provide transitional
> > virtio devices to the guest VM at scale of thousands,
> > typically, one to eight devices per VM.
> >
> > 2. A hypervisor/system needs to provide such devices using a
> > vendor agnostic driver in the hypervisor system.
> >
> > 3. A hypervisor system prefers to have single stack regardless of
> > virtio device type (net/blk) and be future compatible with a
> > single vfio stack using SR-IOV or other scalable device
> > virtualization technology to map PCI devices to the guest VM.
> > (as transitional or otherwise)
> >
> > Motivation/Background:
> > ----------------------
> > The existing virtio transitional PCI device is missing support for
> > PCI SR-IOV based devices. Currently it does not work beyond
> > PCI PF, or as software emulated device in reality. Currently it
> > has below cited system level limitations:
> >
> > [a] PCIe spec citation:
> > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> >
> > [b] cpu arch citiation:
> > Intel 64 and IA-32 Architectures Software Developer’s Manual:
> > The processor’s I/O address space is separate and distinct from
> > the physical-memory address space. The I/O address space consists
> > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> >
> > [c] PCIe spec citation:
> > If a bridge implements an I/O address range,...I/O address range will be
> > aligned to a 4 KB boundary.
> >
> > Overview:
> > ---------
> > Above usecase requirements can be solved by PCI PF group owner accessing
> > its group member PCI VFs legacy registers using an admin virtqueue of
> > the group owner PCI PF.
> >
> > Two new admin virtqueue commands are added which read/write PCI VF
> > registers.
> >
> > The third command suggested by Jason queries the VF device's driver
> > notification region.
> >
> > Software usage example:
> > -----------------------
> > One way to use and map to the guest VM is by using vfio driver
> > framework in Linux kernel.
> >
> > +----------------------+
> > |pci_dev_id = 0x100X |
> > +---------------|pci_rev_id = 0x0 |-----+
> > |vfio device |BAR0 = I/O region | |
> > | |Other attributes | |
> > | +----------------------+ |
> > | |
> > + +--------------+ +-----------------+ |
> > | |I/O BAR to AQ | | Other vfio | |
> > | |rd/wr mapper | | functionalities | |
> > | +--------------+ +-----------------+ |
> > | |
> > +------+-------------------------+-----------+
> > | |
> > +----+------------+ +----+------------+
> > | +-----+ | | PCI VF device A |
> > | | AQ |-------------+---->+-------------+ |
> > | +-----+ | | | | legacy regs | |
> > | PCI PF device | | | +-------------+ |
> > +-----------------+ | +-----------------+
> > |
> > | +----+------------+
> > | | PCI VF device N |
> > +---->+-------------+ |
> > | | legacy regs | |
> > | +-------------+ |
> > +-----------------+
> >
> > 2. Virtio pci driver to bind to the listed device id and
> > use it as native device in the host.
> >
> > 3. Use it in a light weight hypervisor to run bare-metal OS.
> >
> > Please review.
> >
> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> > Signed-off-by: Parav Pandit <parav@nvidia.com>
> >
> > ---
> > changelog:
> > v1->v2:
> > - addressed comments from Michael
> > - added theory of operation
> > - grammar corrections
> > - removed group fields description from individual commands as
> > it is already present in generic section
> > - added endianness normative for legacy device registers region
> > - renamed the file to drop vf and add legacy prefix
> > - added overview in commit log
> > - renamed subsection to reflect command
>
> So as replied in V1, I think it's not a good idea to invent commands
> for a partial transport just for legacy devices. It's better either:
>
> 1) rebase or collaborate this work on top of the transport virtqueue
>
> or
>
> 2) having a PCI over admin virtqueue transport, since this proposal
> has already had BAR access, we can add config space access then it is
> self-contained so we don't need to go through every corner case like
> inventing dedicated commands to accessing some function that is
> duplicated with capabilities. It will become yet another transport and
> legacy support is just a good byproduct.
>
> Thanks
I thought so too originally. Unfortunately I now think that no, legacy is not
going to be a byproduct of transport virtqueue for modern -
it is different enough that it needs dedicated commands.
Consider simplest case, multibyte fields. Legacy needs multibyte write,
modern does not even need multibyte read.
> >
> > v0->v1:
> > - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> > - far more simpler design than MMR access
> > - removed complexities of MMR device ids
> > - removed complexities of MMR registers and extended capabilities
> > - dropped adding new extended capabilities because if if they are
> > added, a pci device still needs to have existing capabilities
> > in the legacy configuration space and hypervisor driver do not
> > need to access them
> >
> >
> > Parav Pandit (2):
> > transport-pci: Introduce legacy registers access commands
> > transport-pci: Add legacy register access conformance section
> >
> > admin.tex | 5 +-
> > conformance.tex | 2 +
> > transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> > transport-pci.tex | 2 +
> > 4 files changed, 141 insertions(+), 1 deletion(-)
> > create mode 100644 transport-pci-legacy-regs.tex
> >
> > --
> > 2.26.2
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-07 13:44 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-08 2:23 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-08 2:23 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> > On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> > >
> > > This short series introduces legacy registers access commands for the owner
> > > group member PCI PF to access the legacy registers of the member VFs.
> > >
> > > If in future any SIOV devices to support legacy registers, they
> > > can be easily supported using same commands by using the group
> > > member identifiers of the future SIOV devices.
> > >
> > > More details as overview, motivation, use case are further described
> > > below.
> > >
> > > Patch summary:
> > > --------------
> > > patch-1 adds administrative virtuqueue commands
> > > patch-2 adds its conformance section
> > >
> > > This short series is on top of latest work [1] from Michael.
> > > It uses the newly introduced administrative virtqueue facility with 3 new
> > > commands which uses the existing virtio_admin_cmd.
> > >
> > > [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> > >
> > > Usecase:
> > > --------
> > > 1. A hypervisor/system needs to provide transitional
> > > virtio devices to the guest VM at scale of thousands,
> > > typically, one to eight devices per VM.
> > >
> > > 2. A hypervisor/system needs to provide such devices using a
> > > vendor agnostic driver in the hypervisor system.
> > >
> > > 3. A hypervisor system prefers to have single stack regardless of
> > > virtio device type (net/blk) and be future compatible with a
> > > single vfio stack using SR-IOV or other scalable device
> > > virtualization technology to map PCI devices to the guest VM.
> > > (as transitional or otherwise)
> > >
> > > Motivation/Background:
> > > ----------------------
> > > The existing virtio transitional PCI device is missing support for
> > > PCI SR-IOV based devices. Currently it does not work beyond
> > > PCI PF, or as software emulated device in reality. Currently it
> > > has below cited system level limitations:
> > >
> > > [a] PCIe spec citation:
> > > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> > >
> > > [b] cpu arch citiation:
> > > Intel 64 and IA-32 Architectures Software Developer’s Manual:
> > > The processor’s I/O address space is separate and distinct from
> > > the physical-memory address space. The I/O address space consists
> > > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> > >
> > > [c] PCIe spec citation:
> > > If a bridge implements an I/O address range,...I/O address range will be
> > > aligned to a 4 KB boundary.
> > >
> > > Overview:
> > > ---------
> > > Above usecase requirements can be solved by PCI PF group owner accessing
> > > its group member PCI VFs legacy registers using an admin virtqueue of
> > > the group owner PCI PF.
> > >
> > > Two new admin virtqueue commands are added which read/write PCI VF
> > > registers.
> > >
> > > The third command suggested by Jason queries the VF device's driver
> > > notification region.
> > >
> > > Software usage example:
> > > -----------------------
> > > One way to use and map to the guest VM is by using vfio driver
> > > framework in Linux kernel.
> > >
> > > +----------------------+
> > > |pci_dev_id = 0x100X |
> > > +---------------|pci_rev_id = 0x0 |-----+
> > > |vfio device |BAR0 = I/O region | |
> > > | |Other attributes | |
> > > | +----------------------+ |
> > > | |
> > > + +--------------+ +-----------------+ |
> > > | |I/O BAR to AQ | | Other vfio | |
> > > | |rd/wr mapper | | functionalities | |
> > > | +--------------+ +-----------------+ |
> > > | |
> > > +------+-------------------------+-----------+
> > > | |
> > > +----+------------+ +----+------------+
> > > | +-----+ | | PCI VF device A |
> > > | | AQ |-------------+---->+-------------+ |
> > > | +-----+ | | | | legacy regs | |
> > > | PCI PF device | | | +-------------+ |
> > > +-----------------+ | +-----------------+
> > > |
> > > | +----+------------+
> > > | | PCI VF device N |
> > > +---->+-------------+ |
> > > | | legacy regs | |
> > > | +-------------+ |
> > > +-----------------+
> > >
> > > 2. Virtio pci driver to bind to the listed device id and
> > > use it as native device in the host.
> > >
> > > 3. Use it in a light weight hypervisor to run bare-metal OS.
> > >
> > > Please review.
> > >
> > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> > > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > >
> > > ---
> > > changelog:
> > > v1->v2:
> > > - addressed comments from Michael
> > > - added theory of operation
> > > - grammar corrections
> > > - removed group fields description from individual commands as
> > > it is already present in generic section
> > > - added endianness normative for legacy device registers region
> > > - renamed the file to drop vf and add legacy prefix
> > > - added overview in commit log
> > > - renamed subsection to reflect command
> >
> > So as replied in V1, I think it's not a good idea to invent commands
> > for a partial transport just for legacy devices. It's better either:
> >
> > 1) rebase or collaborate this work on top of the transport virtqueue
> >
> > or
> >
> > 2) having a PCI over admin virtqueue transport, since this proposal
> > has already had BAR access, we can add config space access then it is
> > self-contained so we don't need to go through every corner case like
> > inventing dedicated commands to accessing some function that is
> > duplicated with capabilities. It will become yet another transport and
> > legacy support is just a good byproduct.
> >
> > Thanks
>
>
> I thought so too originally. Unfortunately I now think that no, legacy is not
> going to be a byproduct of transport virtqueue for modern -
> it is different enough that it needs dedicated commands.
If you mean the transport virtqueue, I think some dedicated commands
for legacy are needed. Then it would be a transport that supports
transitional devices. It would be much better than having commands for
a partial transport like this patch did.
> Consider simplest case, multibyte fields. Legacy needs multibyte write,
> modern does not even need multibyte read.
I'm not sure I will get here, since we can't expose admin vq to
guests, it means we need some software mediation. So if we just
implement what PCI allows us, then everything would be fine (even if
some method is not used).
Thanks
>
> > >
> > > v0->v1:
> > > - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> > > - far more simpler design than MMR access
> > > - removed complexities of MMR device ids
> > > - removed complexities of MMR registers and extended capabilities
> > > - dropped adding new extended capabilities because if if they are
> > > added, a pci device still needs to have existing capabilities
> > > in the legacy configuration space and hypervisor driver do not
> > > need to access them
> > >
> > >
> > > Parav Pandit (2):
> > > transport-pci: Introduce legacy registers access commands
> > > transport-pci: Add legacy register access conformance section
> > >
> > > admin.tex | 5 +-
> > > conformance.tex | 2 +
> > > transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> > > transport-pci.tex | 2 +
> > > 4 files changed, 141 insertions(+), 1 deletion(-)
> > > create mode 100644 transport-pci-legacy-regs.tex
> > >
> > > --
> > > 2.26.2
> > >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-08 2:23 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-08 2:23 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> > On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> > >
> > > This short series introduces legacy registers access commands for the owner
> > > group member PCI PF to access the legacy registers of the member VFs.
> > >
> > > If in future any SIOV devices to support legacy registers, they
> > > can be easily supported using same commands by using the group
> > > member identifiers of the future SIOV devices.
> > >
> > > More details as overview, motivation, use case are further described
> > > below.
> > >
> > > Patch summary:
> > > --------------
> > > patch-1 adds administrative virtuqueue commands
> > > patch-2 adds its conformance section
> > >
> > > This short series is on top of latest work [1] from Michael.
> > > It uses the newly introduced administrative virtqueue facility with 3 new
> > > commands which uses the existing virtio_admin_cmd.
> > >
> > > [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> > >
> > > Usecase:
> > > --------
> > > 1. A hypervisor/system needs to provide transitional
> > > virtio devices to the guest VM at scale of thousands,
> > > typically, one to eight devices per VM.
> > >
> > > 2. A hypervisor/system needs to provide such devices using a
> > > vendor agnostic driver in the hypervisor system.
> > >
> > > 3. A hypervisor system prefers to have single stack regardless of
> > > virtio device type (net/blk) and be future compatible with a
> > > single vfio stack using SR-IOV or other scalable device
> > > virtualization technology to map PCI devices to the guest VM.
> > > (as transitional or otherwise)
> > >
> > > Motivation/Background:
> > > ----------------------
> > > The existing virtio transitional PCI device is missing support for
> > > PCI SR-IOV based devices. Currently it does not work beyond
> > > PCI PF, or as software emulated device in reality. Currently it
> > > has below cited system level limitations:
> > >
> > > [a] PCIe spec citation:
> > > VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> > >
> > > [b] cpu arch citiation:
> > > Intel 64 and IA-32 Architectures Software Developer’s Manual:
> > > The processor’s I/O address space is separate and distinct from
> > > the physical-memory address space. The I/O address space consists
> > > of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> > >
> > > [c] PCIe spec citation:
> > > If a bridge implements an I/O address range,...I/O address range will be
> > > aligned to a 4 KB boundary.
> > >
> > > Overview:
> > > ---------
> > > Above usecase requirements can be solved by PCI PF group owner accessing
> > > its group member PCI VFs legacy registers using an admin virtqueue of
> > > the group owner PCI PF.
> > >
> > > Two new admin virtqueue commands are added which read/write PCI VF
> > > registers.
> > >
> > > The third command suggested by Jason queries the VF device's driver
> > > notification region.
> > >
> > > Software usage example:
> > > -----------------------
> > > One way to use and map to the guest VM is by using vfio driver
> > > framework in Linux kernel.
> > >
> > > +----------------------+
> > > |pci_dev_id = 0x100X |
> > > +---------------|pci_rev_id = 0x0 |-----+
> > > |vfio device |BAR0 = I/O region | |
> > > | |Other attributes | |
> > > | +----------------------+ |
> > > | |
> > > + +--------------+ +-----------------+ |
> > > | |I/O BAR to AQ | | Other vfio | |
> > > | |rd/wr mapper | | functionalities | |
> > > | +--------------+ +-----------------+ |
> > > | |
> > > +------+-------------------------+-----------+
> > > | |
> > > +----+------------+ +----+------------+
> > > | +-----+ | | PCI VF device A |
> > > | | AQ |-------------+---->+-------------+ |
> > > | +-----+ | | | | legacy regs | |
> > > | PCI PF device | | | +-------------+ |
> > > +-----------------+ | +-----------------+
> > > |
> > > | +----+------------+
> > > | | PCI VF device N |
> > > +---->+-------------+ |
> > > | | legacy regs | |
> > > | +-------------+ |
> > > +-----------------+
> > >
> > > 2. Virtio pci driver to bind to the listed device id and
> > > use it as native device in the host.
> > >
> > > 3. Use it in a light weight hypervisor to run bare-metal OS.
> > >
> > > Please review.
> > >
> > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> > > Signed-off-by: Parav Pandit <parav@nvidia.com>
> > >
> > > ---
> > > changelog:
> > > v1->v2:
> > > - addressed comments from Michael
> > > - added theory of operation
> > > - grammar corrections
> > > - removed group fields description from individual commands as
> > > it is already present in generic section
> > > - added endianness normative for legacy device registers region
> > > - renamed the file to drop vf and add legacy prefix
> > > - added overview in commit log
> > > - renamed subsection to reflect command
> >
> > So as replied in V1, I think it's not a good idea to invent commands
> > for a partial transport just for legacy devices. It's better either:
> >
> > 1) rebase or collaborate this work on top of the transport virtqueue
> >
> > or
> >
> > 2) having a PCI over admin virtqueue transport, since this proposal
> > has already had BAR access, we can add config space access then it is
> > self-contained so we don't need to go through every corner case like
> > inventing dedicated commands to accessing some function that is
> > duplicated with capabilities. It will become yet another transport and
> > legacy support is just a good byproduct.
> >
> > Thanks
>
>
> I thought so too originally. Unfortunately I now think that no, legacy is not
> going to be a byproduct of transport virtqueue for modern -
> it is different enough that it needs dedicated commands.
If you mean the transport virtqueue, I think some dedicated commands
for legacy are needed. Then it would be a transport that supports
transitional devices. It would be much better than having commands for
a partial transport like this patch did.
> Consider simplest case, multibyte fields. Legacy needs multibyte write,
> modern does not even need multibyte read.
I'm not sure I will get here, since we can't expose admin vq to
guests, it means we need some software mediation. So if we just
implement what PCI allows us, then everything would be fine (even if
some method is not used).
Thanks
>
> > >
> > > v0->v1:
> > > - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> > > - far more simpler design than MMR access
> > > - removed complexities of MMR device ids
> > > - removed complexities of MMR registers and extended capabilities
> > > - dropped adding new extended capabilities because if if they are
> > > added, a pci device still needs to have existing capabilities
> > > in the legacy configuration space and hypervisor driver do not
> > > need to access them
> > >
> > >
> > > Parav Pandit (2):
> > > transport-pci: Introduce legacy registers access commands
> > > transport-pci: Add legacy register access conformance section
> > >
> > > admin.tex | 5 +-
> > > conformance.tex | 2 +
> > > transport-pci-legacy-regs.tex | 133 ++++++++++++++++++++++++++++++++++
> > > transport-pci.tex | 2 +
> > > 4 files changed, 141 insertions(+), 1 deletion(-)
> > > create mode 100644 transport-pci-legacy-regs.tex
> > >
> > > --
> > > 2.26.2
> > >
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-07 9:04 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-08 16:54 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-08 16:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On 5/7/2023 5:04 AM, Michael S. Tsirkin wrote:
>
> one things I still don't see addressed here is support for
> legacy interrupts. legacy driver can disable msix and
> interrupts will be then sent.
> how about a special command that is used when device would
> normally send INT#x? it can also return ISR to reduce latency.
I am not sure if this is a real issue. Because even the legacy guests
have msix enabled by default. In theory yes, it can fall back to intx.
There are few options.
1. A hypervisor driver can be conservative and steal an msix of the VF
for transporting intx.
Pros: Does not need special things in device
Cons:
a. Fairly intrusive in hypervisor vf driver.
b. May not be ever used as guest is unlikely to fail on msix
2. Since multiple VFs intx to be serviced, one command per VF in AQ is
too much overhead that device needs to map a request to,
A better way is to have an eventq of depth = num_vfs, like many other
virtio devices have it.
An eventq can hold per VF interrupt entry including the isr value that
you suggest above.
Something like,
union eventq_entry {
u8 raw_data[16];
struct intx_entry {
u8 event_opcode;
u8 group_type;
u8 reserved[6];
le64 group_identifier;
u8 isr_status;
};
};
This eventq resides on the owner parent PF.
isr_status is read on clear like today.
May be such eventq can be useful in future for wider case.
We may have to find a different name for it as other devices has device
specific eventq.
I am inclined to differ this to a later point if one can identify the
real failure with msix for the guest VM.
So far we don't see this ever happening.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-08 16:54 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-08 16:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On 5/7/2023 5:04 AM, Michael S. Tsirkin wrote:
>
> one things I still don't see addressed here is support for
> legacy interrupts. legacy driver can disable msix and
> interrupts will be then sent.
> how about a special command that is used when device would
> normally send INT#x? it can also return ISR to reduce latency.
I am not sure if this is a real issue. Because even the legacy guests
have msix enabled by default. In theory yes, it can fall back to intx.
There are few options.
1. A hypervisor driver can be conservative and steal an msix of the VF
for transporting intx.
Pros: Does not need special things in device
Cons:
a. Fairly intrusive in hypervisor vf driver.
b. May not be ever used as guest is unlikely to fail on msix
2. Since multiple VFs intx to be serviced, one command per VF in AQ is
too much overhead that device needs to map a request to,
A better way is to have an eventq of depth = num_vfs, like many other
virtio devices have it.
An eventq can hold per VF interrupt entry including the isr value that
you suggest above.
Something like,
union eventq_entry {
u8 raw_data[16];
struct intx_entry {
u8 event_opcode;
u8 group_type;
u8 reserved[6];
le64 group_identifier;
u8 isr_status;
};
};
This eventq resides on the owner parent PF.
isr_status is read on clear like today.
May be such eventq can be useful in future for wider case.
We may have to find a different name for it as other devices has device
specific eventq.
I am inclined to differ this to a later point if one can identify the
real failure with msix for the guest VM.
So far we don't see this ever happening.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-08 2:23 ` [virtio-comment] " Jason Wang
@ 2023-05-08 17:07 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-08 17:07 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/7/2023 10:23 PM, Jason Wang wrote:
> On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
>>> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
>>>>
>>>> This short series introduces legacy registers access commands for the owner
>>>> group member PCI PF to access the legacy registers of the member VFs.
>>>>
>>>> If in future any SIOV devices to support legacy registers, they
>>>> can be easily supported using same commands by using the group
>>>> member identifiers of the future SIOV devices.
>>>>
>>>> More details as overview, motivation, use case are further described
>>>> below.
>>>>
>>>> Patch summary:
>>>> --------------
>>>> patch-1 adds administrative virtuqueue commands
>>>> patch-2 adds its conformance section
>>>>
>>>> This short series is on top of latest work [1] from Michael.
>>>> It uses the newly introduced administrative virtqueue facility with 3 new
>>>> commands which uses the existing virtio_admin_cmd.
>>>>
>>>> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>>>>
>>>> Usecase:
>>>> --------
>>>> 1. A hypervisor/system needs to provide transitional
>>>> virtio devices to the guest VM at scale of thousands,
>>>> typically, one to eight devices per VM.
>>>>
>>>> 2. A hypervisor/system needs to provide such devices using a
>>>> vendor agnostic driver in the hypervisor system.
>>>>
>>>> 3. A hypervisor system prefers to have single stack regardless of
>>>> virtio device type (net/blk) and be future compatible with a
>>>> single vfio stack using SR-IOV or other scalable device
>>>> virtualization technology to map PCI devices to the guest VM.
>>>> (as transitional or otherwise)
>>>>
>>>> Motivation/Background:
>>>> ----------------------
>>>> The existing virtio transitional PCI device is missing support for
>>>> PCI SR-IOV based devices. Currently it does not work beyond
>>>> PCI PF, or as software emulated device in reality. Currently it
>>>> has below cited system level limitations:
>>>>
>>>> [a] PCIe spec citation:
>>>> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>>>>
>>>> [b] cpu arch citiation:
>>>> Intel 64 and IA-32 Architectures Software Developer’s Manual:
>>>> The processor’s I/O address space is separate and distinct from
>>>> the physical-memory address space. The I/O address space consists
>>>> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>>>>
>>>> [c] PCIe spec citation:
>>>> If a bridge implements an I/O address range,...I/O address range will be
>>>> aligned to a 4 KB boundary.
>>>>
>>>> Overview:
>>>> ---------
>>>> Above usecase requirements can be solved by PCI PF group owner accessing
>>>> its group member PCI VFs legacy registers using an admin virtqueue of
>>>> the group owner PCI PF.
>>>>
>>>> Two new admin virtqueue commands are added which read/write PCI VF
>>>> registers.
>>>>
>>>> The third command suggested by Jason queries the VF device's driver
>>>> notification region.
>>>>
>>>> Software usage example:
>>>> -----------------------
>>>> One way to use and map to the guest VM is by using vfio driver
>>>> framework in Linux kernel.
>>>>
>>>> +----------------------+
>>>> |pci_dev_id = 0x100X |
>>>> +---------------|pci_rev_id = 0x0 |-----+
>>>> |vfio device |BAR0 = I/O region | |
>>>> | |Other attributes | |
>>>> | +----------------------+ |
>>>> | |
>>>> + +--------------+ +-----------------+ |
>>>> | |I/O BAR to AQ | | Other vfio | |
>>>> | |rd/wr mapper | | functionalities | |
>>>> | +--------------+ +-----------------+ |
>>>> | |
>>>> +------+-------------------------+-----------+
>>>> | |
>>>> +----+------------+ +----+------------+
>>>> | +-----+ | | PCI VF device A |
>>>> | | AQ |-------------+---->+-------------+ |
>>>> | +-----+ | | | | legacy regs | |
>>>> | PCI PF device | | | +-------------+ |
>>>> +-----------------+ | +-----------------+
>>>> |
>>>> | +----+------------+
>>>> | | PCI VF device N |
>>>> +---->+-------------+ |
>>>> | | legacy regs | |
>>>> | +-------------+ |
>>>> +-----------------+
>>>>
>>>> 2. Virtio pci driver to bind to the listed device id and
>>>> use it as native device in the host.
>>>>
>>>> 3. Use it in a light weight hypervisor to run bare-metal OS.
>>>>
>>>> Please review.
>>>>
>>>> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
>>>> Signed-off-by: Parav Pandit <parav@nvidia.com>
>>>>
>>>> ---
>>>> changelog:
>>>> v1->v2:
>>>> - addressed comments from Michael
>>>> - added theory of operation
>>>> - grammar corrections
>>>> - removed group fields description from individual commands as
>>>> it is already present in generic section
>>>> - added endianness normative for legacy device registers region
>>>> - renamed the file to drop vf and add legacy prefix
>>>> - added overview in commit log
>>>> - renamed subsection to reflect command
>>>
>>> So as replied in V1, I think it's not a good idea to invent commands
>>> for a partial transport just for legacy devices. It's better either:
>>>
>>> 1) rebase or collaborate this work on top of the transport virtqueue
>>>
>>> or
>>>
>>> 2) having a PCI over admin virtqueue transport, since this proposal
>>> has already had BAR access, we can add config space access then it is
>>> self-contained so we don't need to go through every corner case like
>>> inventing dedicated commands to accessing some function that is
>>> duplicated with capabilities. It will become yet another transport and
>>> legacy support is just a good byproduct.
>>>
>>> Thanks
>>
>>
>> I thought so too originally. Unfortunately I now think that no, legacy is not
>> going to be a byproduct of transport virtqueue for modern -
>> it is different enough that it needs dedicated commands.
>
> If you mean the transport virtqueue, I think some dedicated commands
> for legacy are needed. Then it would be a transport that supports
> transitional devices. It would be much better than having commands for
> a partial transport like this patch did.
>
>> Consider simplest case, multibyte fields. Legacy needs multibyte write,
>> modern does not even need multibyte read.
>
> I'm not sure I will get here, since we can't expose admin vq to
> guests, it means we need some software mediation. So if we just
> implement what PCI allows us, then everything would be fine (even if
> some method is not used).
>
> Thanks
The fundamental reason for not accessing the 1.x VF and SIOV device
registers, config space, feature bits through PF is: it requires PF
device mediation. VF and SIOV devices are first class citizen in PCIe
spec and deserve direct interaction with the device.
Hence, the transport we built is to consider this in mind for the coming
future.
So if each VF has its own configq, or cmdq, it totally make sense to me
which is bootstrap interface to transport existing config space interface.
The problem is: it is not backward compatible;
Hence a device has no way of when to support both or only new configq.
So eve growing these fields and optionally placement on configq doesn't
really help and device builder to build it efficiently (without much
predictability).
Instead of we say, that what exists today in config space stays in
config space, anything additional on new q, than its deterministic
behavior to size up the scale.
For example, a PCI device who wants to support 100 VFs, can easily size
its memory to 30 bytes * 100 reserved for supporting config space.
And new 40 bytes * 100 fields doesn't have to be in the resident memory.
If we have optional configq/cmdq for transport, than 30*100 bytes are
used (reserved) as 3000/(30+40) = 42 VFs.
Only if some VFs use configq, more VFs can be deployed. It is hard to
build scale this way. Therefore suggestion is to place new attributes on
new config/cmd/transport q, and old to stay as-is.
The legacy infra is unfortunately is for the exception path due to the
history; hence they are different commands as Michael suggests.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-08 17:07 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-08 17:07 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/7/2023 10:23 PM, Jason Wang wrote:
> On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>
>> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
>>> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
>>>>
>>>> This short series introduces legacy registers access commands for the owner
>>>> group member PCI PF to access the legacy registers of the member VFs.
>>>>
>>>> If in future any SIOV devices to support legacy registers, they
>>>> can be easily supported using same commands by using the group
>>>> member identifiers of the future SIOV devices.
>>>>
>>>> More details as overview, motivation, use case are further described
>>>> below.
>>>>
>>>> Patch summary:
>>>> --------------
>>>> patch-1 adds administrative virtuqueue commands
>>>> patch-2 adds its conformance section
>>>>
>>>> This short series is on top of latest work [1] from Michael.
>>>> It uses the newly introduced administrative virtqueue facility with 3 new
>>>> commands which uses the existing virtio_admin_cmd.
>>>>
>>>> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>>>>
>>>> Usecase:
>>>> --------
>>>> 1. A hypervisor/system needs to provide transitional
>>>> virtio devices to the guest VM at scale of thousands,
>>>> typically, one to eight devices per VM.
>>>>
>>>> 2. A hypervisor/system needs to provide such devices using a
>>>> vendor agnostic driver in the hypervisor system.
>>>>
>>>> 3. A hypervisor system prefers to have single stack regardless of
>>>> virtio device type (net/blk) and be future compatible with a
>>>> single vfio stack using SR-IOV or other scalable device
>>>> virtualization technology to map PCI devices to the guest VM.
>>>> (as transitional or otherwise)
>>>>
>>>> Motivation/Background:
>>>> ----------------------
>>>> The existing virtio transitional PCI device is missing support for
>>>> PCI SR-IOV based devices. Currently it does not work beyond
>>>> PCI PF, or as software emulated device in reality. Currently it
>>>> has below cited system level limitations:
>>>>
>>>> [a] PCIe spec citation:
>>>> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>>>>
>>>> [b] cpu arch citiation:
>>>> Intel 64 and IA-32 Architectures Software Developer’s Manual:
>>>> The processor’s I/O address space is separate and distinct from
>>>> the physical-memory address space. The I/O address space consists
>>>> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>>>>
>>>> [c] PCIe spec citation:
>>>> If a bridge implements an I/O address range,...I/O address range will be
>>>> aligned to a 4 KB boundary.
>>>>
>>>> Overview:
>>>> ---------
>>>> Above usecase requirements can be solved by PCI PF group owner accessing
>>>> its group member PCI VFs legacy registers using an admin virtqueue of
>>>> the group owner PCI PF.
>>>>
>>>> Two new admin virtqueue commands are added which read/write PCI VF
>>>> registers.
>>>>
>>>> The third command suggested by Jason queries the VF device's driver
>>>> notification region.
>>>>
>>>> Software usage example:
>>>> -----------------------
>>>> One way to use and map to the guest VM is by using vfio driver
>>>> framework in Linux kernel.
>>>>
>>>> +----------------------+
>>>> |pci_dev_id = 0x100X |
>>>> +---------------|pci_rev_id = 0x0 |-----+
>>>> |vfio device |BAR0 = I/O region | |
>>>> | |Other attributes | |
>>>> | +----------------------+ |
>>>> | |
>>>> + +--------------+ +-----------------+ |
>>>> | |I/O BAR to AQ | | Other vfio | |
>>>> | |rd/wr mapper | | functionalities | |
>>>> | +--------------+ +-----------------+ |
>>>> | |
>>>> +------+-------------------------+-----------+
>>>> | |
>>>> +----+------------+ +----+------------+
>>>> | +-----+ | | PCI VF device A |
>>>> | | AQ |-------------+---->+-------------+ |
>>>> | +-----+ | | | | legacy regs | |
>>>> | PCI PF device | | | +-------------+ |
>>>> +-----------------+ | +-----------------+
>>>> |
>>>> | +----+------------+
>>>> | | PCI VF device N |
>>>> +---->+-------------+ |
>>>> | | legacy regs | |
>>>> | +-------------+ |
>>>> +-----------------+
>>>>
>>>> 2. Virtio pci driver to bind to the listed device id and
>>>> use it as native device in the host.
>>>>
>>>> 3. Use it in a light weight hypervisor to run bare-metal OS.
>>>>
>>>> Please review.
>>>>
>>>> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
>>>> Signed-off-by: Parav Pandit <parav@nvidia.com>
>>>>
>>>> ---
>>>> changelog:
>>>> v1->v2:
>>>> - addressed comments from Michael
>>>> - added theory of operation
>>>> - grammar corrections
>>>> - removed group fields description from individual commands as
>>>> it is already present in generic section
>>>> - added endianness normative for legacy device registers region
>>>> - renamed the file to drop vf and add legacy prefix
>>>> - added overview in commit log
>>>> - renamed subsection to reflect command
>>>
>>> So as replied in V1, I think it's not a good idea to invent commands
>>> for a partial transport just for legacy devices. It's better either:
>>>
>>> 1) rebase or collaborate this work on top of the transport virtqueue
>>>
>>> or
>>>
>>> 2) having a PCI over admin virtqueue transport, since this proposal
>>> has already had BAR access, we can add config space access then it is
>>> self-contained so we don't need to go through every corner case like
>>> inventing dedicated commands to accessing some function that is
>>> duplicated with capabilities. It will become yet another transport and
>>> legacy support is just a good byproduct.
>>>
>>> Thanks
>>
>>
>> I thought so too originally. Unfortunately I now think that no, legacy is not
>> going to be a byproduct of transport virtqueue for modern -
>> it is different enough that it needs dedicated commands.
>
> If you mean the transport virtqueue, I think some dedicated commands
> for legacy are needed. Then it would be a transport that supports
> transitional devices. It would be much better than having commands for
> a partial transport like this patch did.
>
>> Consider simplest case, multibyte fields. Legacy needs multibyte write,
>> modern does not even need multibyte read.
>
> I'm not sure I will get here, since we can't expose admin vq to
> guests, it means we need some software mediation. So if we just
> implement what PCI allows us, then everything would be fine (even if
> some method is not used).
>
> Thanks
The fundamental reason for not accessing the 1.x VF and SIOV device
registers, config space, feature bits through PF is: it requires PF
device mediation. VF and SIOV devices are first class citizen in PCIe
spec and deserve direct interaction with the device.
Hence, the transport we built is to consider this in mind for the coming
future.
So if each VF has its own configq, or cmdq, it totally make sense to me
which is bootstrap interface to transport existing config space interface.
The problem is: it is not backward compatible;
Hence a device has no way of when to support both or only new configq.
So eve growing these fields and optionally placement on configq doesn't
really help and device builder to build it efficiently (without much
predictability).
Instead of we say, that what exists today in config space stays in
config space, anything additional on new q, than its deterministic
behavior to size up the scale.
For example, a PCI device who wants to support 100 VFs, can easily size
its memory to 30 bytes * 100 reserved for supporting config space.
And new 40 bytes * 100 fields doesn't have to be in the resident memory.
If we have optional configq/cmdq for transport, than 30*100 bytes are
used (reserved) as 3000/(30+40) = 42 VFs.
Only if some VFs use configq, more VFs can be deployed. It is hard to
build scale this way. Therefore suggestion is to place new attributes on
new config/cmd/transport q, and old to stay as-is.
The legacy infra is unfortunately is for the exception path due to the
history; hence they are different commands as Michael suggests.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-08 17:07 ` [virtio-comment] " Parav Pandit
@ 2023-05-09 3:44 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-09 3:44 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Tue, May 9, 2023 at 1:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/7/2023 10:23 PM, Jason Wang wrote:
> > On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >>
> >> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> >>> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> >>>>
> >>>> This short series introduces legacy registers access commands for the owner
> >>>> group member PCI PF to access the legacy registers of the member VFs.
> >>>>
> >>>> If in future any SIOV devices to support legacy registers, they
> >>>> can be easily supported using same commands by using the group
> >>>> member identifiers of the future SIOV devices.
> >>>>
> >>>> More details as overview, motivation, use case are further described
> >>>> below.
> >>>>
> >>>> Patch summary:
> >>>> --------------
> >>>> patch-1 adds administrative virtuqueue commands
> >>>> patch-2 adds its conformance section
> >>>>
> >>>> This short series is on top of latest work [1] from Michael.
> >>>> It uses the newly introduced administrative virtqueue facility with 3 new
> >>>> commands which uses the existing virtio_admin_cmd.
> >>>>
> >>>> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> >>>>
> >>>> Usecase:
> >>>> --------
> >>>> 1. A hypervisor/system needs to provide transitional
> >>>> virtio devices to the guest VM at scale of thousands,
> >>>> typically, one to eight devices per VM.
> >>>>
> >>>> 2. A hypervisor/system needs to provide such devices using a
> >>>> vendor agnostic driver in the hypervisor system.
> >>>>
> >>>> 3. A hypervisor system prefers to have single stack regardless of
> >>>> virtio device type (net/blk) and be future compatible with a
> >>>> single vfio stack using SR-IOV or other scalable device
> >>>> virtualization technology to map PCI devices to the guest VM.
> >>>> (as transitional or otherwise)
> >>>>
> >>>> Motivation/Background:
> >>>> ----------------------
> >>>> The existing virtio transitional PCI device is missing support for
> >>>> PCI SR-IOV based devices. Currently it does not work beyond
> >>>> PCI PF, or as software emulated device in reality. Currently it
> >>>> has below cited system level limitations:
> >>>>
> >>>> [a] PCIe spec citation:
> >>>> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> >>>>
> >>>> [b] cpu arch citiation:
> >>>> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> >>>> The processor’s I/O address space is separate and distinct from
> >>>> the physical-memory address space. The I/O address space consists
> >>>> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> >>>>
> >>>> [c] PCIe spec citation:
> >>>> If a bridge implements an I/O address range,...I/O address range will be
> >>>> aligned to a 4 KB boundary.
> >>>>
> >>>> Overview:
> >>>> ---------
> >>>> Above usecase requirements can be solved by PCI PF group owner accessing
> >>>> its group member PCI VFs legacy registers using an admin virtqueue of
> >>>> the group owner PCI PF.
> >>>>
> >>>> Two new admin virtqueue commands are added which read/write PCI VF
> >>>> registers.
> >>>>
> >>>> The third command suggested by Jason queries the VF device's driver
> >>>> notification region.
> >>>>
> >>>> Software usage example:
> >>>> -----------------------
> >>>> One way to use and map to the guest VM is by using vfio driver
> >>>> framework in Linux kernel.
> >>>>
> >>>> +----------------------+
> >>>> |pci_dev_id = 0x100X |
> >>>> +---------------|pci_rev_id = 0x0 |-----+
> >>>> |vfio device |BAR0 = I/O region | |
> >>>> | |Other attributes | |
> >>>> | +----------------------+ |
> >>>> | |
> >>>> + +--------------+ +-----------------+ |
> >>>> | |I/O BAR to AQ | | Other vfio | |
> >>>> | |rd/wr mapper | | functionalities | |
> >>>> | +--------------+ +-----------------+ |
> >>>> | |
> >>>> +------+-------------------------+-----------+
> >>>> | |
> >>>> +----+------------+ +----+------------+
> >>>> | +-----+ | | PCI VF device A |
> >>>> | | AQ |-------------+---->+-------------+ |
> >>>> | +-----+ | | | | legacy regs | |
> >>>> | PCI PF device | | | +-------------+ |
> >>>> +-----------------+ | +-----------------+
> >>>> |
> >>>> | +----+------------+
> >>>> | | PCI VF device N |
> >>>> +---->+-------------+ |
> >>>> | | legacy regs | |
> >>>> | +-------------+ |
> >>>> +-----------------+
> >>>>
> >>>> 2. Virtio pci driver to bind to the listed device id and
> >>>> use it as native device in the host.
> >>>>
> >>>> 3. Use it in a light weight hypervisor to run bare-metal OS.
> >>>>
> >>>> Please review.
> >>>>
> >>>> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> >>>> Signed-off-by: Parav Pandit <parav@nvidia.com>
> >>>>
> >>>> ---
> >>>> changelog:
> >>>> v1->v2:
> >>>> - addressed comments from Michael
> >>>> - added theory of operation
> >>>> - grammar corrections
> >>>> - removed group fields description from individual commands as
> >>>> it is already present in generic section
> >>>> - added endianness normative for legacy device registers region
> >>>> - renamed the file to drop vf and add legacy prefix
> >>>> - added overview in commit log
> >>>> - renamed subsection to reflect command
> >>>
> >>> So as replied in V1, I think it's not a good idea to invent commands
> >>> for a partial transport just for legacy devices. It's better either:
> >>>
> >>> 1) rebase or collaborate this work on top of the transport virtqueue
> >>>
> >>> or
> >>>
> >>> 2) having a PCI over admin virtqueue transport, since this proposal
> >>> has already had BAR access, we can add config space access then it is
> >>> self-contained so we don't need to go through every corner case like
> >>> inventing dedicated commands to accessing some function that is
> >>> duplicated with capabilities. It will become yet another transport and
> >>> legacy support is just a good byproduct.
> >>>
> >>> Thanks
> >>
> >>
> >> I thought so too originally. Unfortunately I now think that no, legacy is not
> >> going to be a byproduct of transport virtqueue for modern -
> >> it is different enough that it needs dedicated commands.
> >
> > If you mean the transport virtqueue, I think some dedicated commands
> > for legacy are needed. Then it would be a transport that supports
> > transitional devices. It would be much better than having commands for
> > a partial transport like this patch did.
> >
> >> Consider simplest case, multibyte fields. Legacy needs multibyte write,
> >> modern does not even need multibyte read.
> >
> > I'm not sure I will get here, since we can't expose admin vq to
> > guests, it means we need some software mediation. So if we just
> > implement what PCI allows us, then everything would be fine (even if
> > some method is not used).
> >
> > Thanks
>
> The fundamental reason for not accessing the 1.x VF and SIOV device
> registers, config space, feature bits through PF is: it requires PF
> device mediation. VF and SIOV devices are first class citizen in PCIe
> spec and deserve direct interaction with the device.
Unless I miss something obvious, SIOV requires mediation (or
composition) for sure. Otherwise you break the compatibility.
>
> Hence, the transport we built is to consider this in mind for the coming
> future.
For transport virtqueue, it's not specific to PCI. It could be used in
a much broader use case.
> So if each VF has its own configq, or cmdq, it totally make sense to me
> which is bootstrap interface to transport existing config space interface.
> The problem is: it is not backward compatible;
> Hence a device has no way of when to support both or only new configq.
Providing compatibility in software is much more simpler than
inventing new hardware interfaces. Isn't it? (e.g if we want to
provide compatibility for VF on a SIOV device). And inventing a new
hardware interface for compatibility might not always work, it may
break the advantages of the new hardware (like scalability).
>
> So eve growing these fields and optionally placement on configq doesn't
> really help and device builder to build it efficiently (without much
> predictability).
Config queue is not the only choice, we have a lot of other choices
(for example PASID may help to reduce the on-chip resources).
>
> Instead of we say, that what exists today in config space stays in
> config space, anything additional on new q, than its deterministic
> behavior to size up the scale.
Just to be clear, if we have PCI over adminq, VF's config space could
be the minimal one for PCI spec complaint. The real config space is
accessed via the admin virtqueue.
>
> For example, a PCI device who wants to support 100 VFs, can easily size
> its memory to 30 bytes * 100 reserved for supporting config space.
Those capabilities (30 bytes) can be accessed via admin virtqueue. So
we don't need to place them in the config space.
> And new 40 bytes * 100 fields doesn't have to be in the resident memory.
>
> If we have optional configq/cmdq for transport, than 30*100 bytes are
> used (reserved) as 3000/(30+40) = 42 VFs.
>
> Only if some VFs use configq, more VFs can be deployed.
I don't understand here.
> It is hard to
> build scale this way. Therefore suggestion is to place new attributes on
> new config/cmd/transport q, and old to stay as-is.
Just to be sure we're on the same page. The proposal of both you and
mine are based on the adminq for PF not VF. The reason is obvious:
adminq per VF won't work without PASID, since it would have security
issues.
>
> The legacy infra is unfortunately is for the exception path due to the
> history; hence they are different commands as Michael suggests.
>
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-09 3:44 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-09 3:44 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Tue, May 9, 2023 at 1:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/7/2023 10:23 PM, Jason Wang wrote:
> > On Sun, May 7, 2023 at 9:44 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >>
> >> On Sat, May 06, 2023 at 10:31:30AM +0800, Jason Wang wrote:
> >>> On Sat, May 6, 2023 at 8:02 AM Parav Pandit <parav@nvidia.com> wrote:
> >>>>
> >>>> This short series introduces legacy registers access commands for the owner
> >>>> group member PCI PF to access the legacy registers of the member VFs.
> >>>>
> >>>> If in future any SIOV devices to support legacy registers, they
> >>>> can be easily supported using same commands by using the group
> >>>> member identifiers of the future SIOV devices.
> >>>>
> >>>> More details as overview, motivation, use case are further described
> >>>> below.
> >>>>
> >>>> Patch summary:
> >>>> --------------
> >>>> patch-1 adds administrative virtuqueue commands
> >>>> patch-2 adds its conformance section
> >>>>
> >>>> This short series is on top of latest work [1] from Michael.
> >>>> It uses the newly introduced administrative virtqueue facility with 3 new
> >>>> commands which uses the existing virtio_admin_cmd.
> >>>>
> >>>> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
> >>>>
> >>>> Usecase:
> >>>> --------
> >>>> 1. A hypervisor/system needs to provide transitional
> >>>> virtio devices to the guest VM at scale of thousands,
> >>>> typically, one to eight devices per VM.
> >>>>
> >>>> 2. A hypervisor/system needs to provide such devices using a
> >>>> vendor agnostic driver in the hypervisor system.
> >>>>
> >>>> 3. A hypervisor system prefers to have single stack regardless of
> >>>> virtio device type (net/blk) and be future compatible with a
> >>>> single vfio stack using SR-IOV or other scalable device
> >>>> virtualization technology to map PCI devices to the guest VM.
> >>>> (as transitional or otherwise)
> >>>>
> >>>> Motivation/Background:
> >>>> ----------------------
> >>>> The existing virtio transitional PCI device is missing support for
> >>>> PCI SR-IOV based devices. Currently it does not work beyond
> >>>> PCI PF, or as software emulated device in reality. Currently it
> >>>> has below cited system level limitations:
> >>>>
> >>>> [a] PCIe spec citation:
> >>>> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
> >>>>
> >>>> [b] cpu arch citiation:
> >>>> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> >>>> The processor’s I/O address space is separate and distinct from
> >>>> the physical-memory address space. The I/O address space consists
> >>>> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
> >>>>
> >>>> [c] PCIe spec citation:
> >>>> If a bridge implements an I/O address range,...I/O address range will be
> >>>> aligned to a 4 KB boundary.
> >>>>
> >>>> Overview:
> >>>> ---------
> >>>> Above usecase requirements can be solved by PCI PF group owner accessing
> >>>> its group member PCI VFs legacy registers using an admin virtqueue of
> >>>> the group owner PCI PF.
> >>>>
> >>>> Two new admin virtqueue commands are added which read/write PCI VF
> >>>> registers.
> >>>>
> >>>> The third command suggested by Jason queries the VF device's driver
> >>>> notification region.
> >>>>
> >>>> Software usage example:
> >>>> -----------------------
> >>>> One way to use and map to the guest VM is by using vfio driver
> >>>> framework in Linux kernel.
> >>>>
> >>>> +----------------------+
> >>>> |pci_dev_id = 0x100X |
> >>>> +---------------|pci_rev_id = 0x0 |-----+
> >>>> |vfio device |BAR0 = I/O region | |
> >>>> | |Other attributes | |
> >>>> | +----------------------+ |
> >>>> | |
> >>>> + +--------------+ +-----------------+ |
> >>>> | |I/O BAR to AQ | | Other vfio | |
> >>>> | |rd/wr mapper | | functionalities | |
> >>>> | +--------------+ +-----------------+ |
> >>>> | |
> >>>> +------+-------------------------+-----------+
> >>>> | |
> >>>> +----+------------+ +----+------------+
> >>>> | +-----+ | | PCI VF device A |
> >>>> | | AQ |-------------+---->+-------------+ |
> >>>> | +-----+ | | | | legacy regs | |
> >>>> | PCI PF device | | | +-------------+ |
> >>>> +-----------------+ | +-----------------+
> >>>> |
> >>>> | +----+------------+
> >>>> | | PCI VF device N |
> >>>> +---->+-------------+ |
> >>>> | | legacy regs | |
> >>>> | +-------------+ |
> >>>> +-----------------+
> >>>>
> >>>> 2. Virtio pci driver to bind to the listed device id and
> >>>> use it as native device in the host.
> >>>>
> >>>> 3. Use it in a light weight hypervisor to run bare-metal OS.
> >>>>
> >>>> Please review.
> >>>>
> >>>> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> >>>> Signed-off-by: Parav Pandit <parav@nvidia.com>
> >>>>
> >>>> ---
> >>>> changelog:
> >>>> v1->v2:
> >>>> - addressed comments from Michael
> >>>> - added theory of operation
> >>>> - grammar corrections
> >>>> - removed group fields description from individual commands as
> >>>> it is already present in generic section
> >>>> - added endianness normative for legacy device registers region
> >>>> - renamed the file to drop vf and add legacy prefix
> >>>> - added overview in commit log
> >>>> - renamed subsection to reflect command
> >>>
> >>> So as replied in V1, I think it's not a good idea to invent commands
> >>> for a partial transport just for legacy devices. It's better either:
> >>>
> >>> 1) rebase or collaborate this work on top of the transport virtqueue
> >>>
> >>> or
> >>>
> >>> 2) having a PCI over admin virtqueue transport, since this proposal
> >>> has already had BAR access, we can add config space access then it is
> >>> self-contained so we don't need to go through every corner case like
> >>> inventing dedicated commands to accessing some function that is
> >>> duplicated with capabilities. It will become yet another transport and
> >>> legacy support is just a good byproduct.
> >>>
> >>> Thanks
> >>
> >>
> >> I thought so too originally. Unfortunately I now think that no, legacy is not
> >> going to be a byproduct of transport virtqueue for modern -
> >> it is different enough that it needs dedicated commands.
> >
> > If you mean the transport virtqueue, I think some dedicated commands
> > for legacy are needed. Then it would be a transport that supports
> > transitional devices. It would be much better than having commands for
> > a partial transport like this patch did.
> >
> >> Consider simplest case, multibyte fields. Legacy needs multibyte write,
> >> modern does not even need multibyte read.
> >
> > I'm not sure I will get here, since we can't expose admin vq to
> > guests, it means we need some software mediation. So if we just
> > implement what PCI allows us, then everything would be fine (even if
> > some method is not used).
> >
> > Thanks
>
> The fundamental reason for not accessing the 1.x VF and SIOV device
> registers, config space, feature bits through PF is: it requires PF
> device mediation. VF and SIOV devices are first class citizen in PCIe
> spec and deserve direct interaction with the device.
Unless I miss something obvious, SIOV requires mediation (or
composition) for sure. Otherwise you break the compatibility.
>
> Hence, the transport we built is to consider this in mind for the coming
> future.
For transport virtqueue, it's not specific to PCI. It could be used in
a much broader use case.
> So if each VF has its own configq, or cmdq, it totally make sense to me
> which is bootstrap interface to transport existing config space interface.
> The problem is: it is not backward compatible;
> Hence a device has no way of when to support both or only new configq.
Providing compatibility in software is much more simpler than
inventing new hardware interfaces. Isn't it? (e.g if we want to
provide compatibility for VF on a SIOV device). And inventing a new
hardware interface for compatibility might not always work, it may
break the advantages of the new hardware (like scalability).
>
> So eve growing these fields and optionally placement on configq doesn't
> really help and device builder to build it efficiently (without much
> predictability).
Config queue is not the only choice, we have a lot of other choices
(for example PASID may help to reduce the on-chip resources).
>
> Instead of we say, that what exists today in config space stays in
> config space, anything additional on new q, than its deterministic
> behavior to size up the scale.
Just to be clear, if we have PCI over adminq, VF's config space could
be the minimal one for PCI spec complaint. The real config space is
accessed via the admin virtqueue.
>
> For example, a PCI device who wants to support 100 VFs, can easily size
> its memory to 30 bytes * 100 reserved for supporting config space.
Those capabilities (30 bytes) can be accessed via admin virtqueue. So
we don't need to place them in the config space.
> And new 40 bytes * 100 fields doesn't have to be in the resident memory.
>
> If we have optional configq/cmdq for transport, than 30*100 bytes are
> used (reserved) as 3000/(30+40) = 42 VFs.
>
> Only if some VFs use configq, more VFs can be deployed.
I don't understand here.
> It is hard to
> build scale this way. Therefore suggestion is to place new attributes on
> new config/cmd/transport q, and old to stay as-is.
Just to be sure we're on the same page. The proposal of both you and
mine are based on the adminq for PF not VF. The reason is obvious:
adminq per VF won't work without PASID, since it would have security
issues.
>
> The legacy infra is unfortunately is for the exception path due to the
> history; hence they are different commands as Michael suggests.
>
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-09 3:44 ` [virtio-dev] " Jason Wang
@ 2023-05-09 3:56 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-09 3:56 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 8, 2023 11:45 PM
> > The fundamental reason for not accessing the 1.x VF and SIOV device
> > registers, config space, feature bits through PF is: it requires PF
> > device mediation. VF and SIOV devices are first class citizen in PCIe
> > spec and deserve direct interaction with the device.
>
> Unless I miss something obvious, SIOV requires mediation (or
> composition) for sure. Otherwise you break the compatibility.
>
SIOV does not require mediation.
Composition is optional like VFs.
> >
> > Hence, the transport we built is to consider this in mind for the
> > coming future.
>
> For transport virtqueue, it's not specific to PCI. It could be used in a much
> broader use case.
>
May be.
More below.
> > So if each VF has its own configq, or cmdq, it totally make sense to
> > me which is bootstrap interface to transport existing config space interface.
> > The problem is: it is not backward compatible; Hence a device has no
> > way of when to support both or only new configq.
>
> Providing compatibility in software is much more simpler than inventing new
> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> a SIOV device). And inventing a new hardware interface for compatibility might
> not always work, it may break the advantages of the new hardware (like
> scalability).
>
What I proposed below is new hardware interface for new features.
Not for compatibility.
Compatibility is coming at a cost which is not scaling.
If you attempt to scale, it breaks the future path forward to go without mediation.
> >
> > So eve growing these fields and optionally placement on configq
> > doesn't really help and device builder to build it efficiently
> > (without much predictability).
>
> Config queue is not the only choice, we have a lot of other choices (for example
> PASID may help to reduce the on-chip resources).
>
PASID is for process isolation security construct, it is not for transportation.
> >
> > Instead of we say, that what exists today in config space stays in
> > config space, anything additional on new q, than its deterministic
> > behavior to size up the scale.
>
> Just to be clear, if we have PCI over adminq, VF's config space could be the
> minimal one for PCI spec complaint. The real config space is accessed via the
> admin virtqueue.
>
Yeah I understood what you are saying. We don’t see a path forward with such mediation.
VF and PF to have same level of direct access to the device.
> >
> > For example, a PCI device who wants to support 100 VFs, can easily
> > size its memory to 30 bytes * 100 reserved for supporting config space.
>
> Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> need to place them in the config space.
>
Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
> > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> >
> > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > used (reserved) as 3000/(30+40) = 42 VFs.
> >
> > Only if some VFs use configq, more VFs can be deployed.
>
> I don't understand here.
>
Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
> > It is hard to
> > build scale this way. Therefore suggestion is to place new attributes
> > on new config/cmd/transport q, and old to stay as-is.
>
> Just to be sure we're on the same page. The proposal of both you and mine are
> based on the adminq for PF not VF. The reason is obvious:
> adminq per VF won't work without PASID, since it would have security issues.
>
The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
(And so additional configq follows the natural model for config space access).
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-09 3:56 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-09 3:56 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 8, 2023 11:45 PM
> > The fundamental reason for not accessing the 1.x VF and SIOV device
> > registers, config space, feature bits through PF is: it requires PF
> > device mediation. VF and SIOV devices are first class citizen in PCIe
> > spec and deserve direct interaction with the device.
>
> Unless I miss something obvious, SIOV requires mediation (or
> composition) for sure. Otherwise you break the compatibility.
>
SIOV does not require mediation.
Composition is optional like VFs.
> >
> > Hence, the transport we built is to consider this in mind for the
> > coming future.
>
> For transport virtqueue, it's not specific to PCI. It could be used in a much
> broader use case.
>
May be.
More below.
> > So if each VF has its own configq, or cmdq, it totally make sense to
> > me which is bootstrap interface to transport existing config space interface.
> > The problem is: it is not backward compatible; Hence a device has no
> > way of when to support both or only new configq.
>
> Providing compatibility in software is much more simpler than inventing new
> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> a SIOV device). And inventing a new hardware interface for compatibility might
> not always work, it may break the advantages of the new hardware (like
> scalability).
>
What I proposed below is new hardware interface for new features.
Not for compatibility.
Compatibility is coming at a cost which is not scaling.
If you attempt to scale, it breaks the future path forward to go without mediation.
> >
> > So eve growing these fields and optionally placement on configq
> > doesn't really help and device builder to build it efficiently
> > (without much predictability).
>
> Config queue is not the only choice, we have a lot of other choices (for example
> PASID may help to reduce the on-chip resources).
>
PASID is for process isolation security construct, it is not for transportation.
> >
> > Instead of we say, that what exists today in config space stays in
> > config space, anything additional on new q, than its deterministic
> > behavior to size up the scale.
>
> Just to be clear, if we have PCI over adminq, VF's config space could be the
> minimal one for PCI spec complaint. The real config space is accessed via the
> admin virtqueue.
>
Yeah I understood what you are saying. We don’t see a path forward with such mediation.
VF and PF to have same level of direct access to the device.
> >
> > For example, a PCI device who wants to support 100 VFs, can easily
> > size its memory to 30 bytes * 100 reserved for supporting config space.
>
> Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> need to place them in the config space.
>
Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
> > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> >
> > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > used (reserved) as 3000/(30+40) = 42 VFs.
> >
> > Only if some VFs use configq, more VFs can be deployed.
>
> I don't understand here.
>
Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
> > It is hard to
> > build scale this way. Therefore suggestion is to place new attributes
> > on new config/cmd/transport q, and old to stay as-is.
>
> Just to be sure we're on the same page. The proposal of both you and mine are
> based on the adminq for PF not VF. The reason is obvious:
> adminq per VF won't work without PASID, since it would have security issues.
>
The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
(And so additional configq follows the natural model for config space access).
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-09 3:56 ` [virtio-comment] " Parav Pandit
@ 2023-05-10 3:51 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 3:51 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 8, 2023 11:45 PM
>
> > > The fundamental reason for not accessing the 1.x VF and SIOV device
> > > registers, config space, feature bits through PF is: it requires PF
> > > device mediation. VF and SIOV devices are first class citizen in PCIe
> > > spec and deserve direct interaction with the device.
> >
> > Unless I miss something obvious, SIOV requires mediation (or
> > composition) for sure. Otherwise you break the compatibility.
> >
> SIOV does not require mediation.
> Composition is optional like VFs.
This is not what I read from SIOV spec.
>
> > >
> > > Hence, the transport we built is to consider this in mind for the
> > > coming future.
> >
> > For transport virtqueue, it's not specific to PCI. It could be used in a much
> > broader use case.
> >
> May be.
> More below.
>
> > > So if each VF has its own configq, or cmdq, it totally make sense to
> > > me which is bootstrap interface to transport existing config space interface.
> > > The problem is: it is not backward compatible; Hence a device has no
> > > way of when to support both or only new configq.
> >
> > Providing compatibility in software is much more simpler than inventing new
> > hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> > a SIOV device). And inventing a new hardware interface for compatibility might
> > not always work, it may break the advantages of the new hardware (like
> > scalability).
> >
> What I proposed below is new hardware interface for new features.
> Not for compatibility.
I'm confused, the proposal is for legacy drives so it's for
compatibility for sure. No?
> Compatibility is coming at a cost which is not scaling.
> If you attempt to scale, it breaks the future path forward to go without mediation.
>
> > >
> > > So eve growing these fields and optionally placement on configq
> > > doesn't really help and device builder to build it efficiently
> > > (without much predictability).
> >
> > Config queue is not the only choice, we have a lot of other choices (for example
> > PASID may help to reduce the on-chip resources).
> >
> PASID is for process isolation security construct, it is not for transportation.
I meant with PASID you don't even need a BAR.
>
> > >
> > > Instead of we say, that what exists today in config space stays in
> > > config space, anything additional on new q, than its deterministic
> > > behavior to size up the scale.
> >
> > Just to be clear, if we have PCI over adminq, VF's config space could be the
> > minimal one for PCI spec complaint. The real config space is accessed via the
> > admin virtqueue.
> >
> Yeah I understood what you are saying. We don’t see a path forward with such mediation.
> VF and PF to have same level of direct access to the device.
>
> > >
> > > For example, a PCI device who wants to support 100 VFs, can easily
> > > size its memory to 30 bytes * 100 reserved for supporting config space.
> >
> > Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> > need to place them in the config space.
> >
> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
Unless you don't want to provide VF compatibility for SIOV.
>
> > > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> > >
> > > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > > used (reserved) as 3000/(30+40) = 42 VFs.
> > >
> > > Only if some VFs use configq, more VFs can be deployed.
> >
> > I don't understand here.
> >
> Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
>
> > > It is hard to
> > > build scale this way. Therefore suggestion is to place new attributes
> > > on new config/cmd/transport q, and old to stay as-is.
> >
> > Just to be sure we're on the same page. The proposal of both you and mine are
> > based on the adminq for PF not VF. The reason is obvious:
> > adminq per VF won't work without PASID, since it would have security issues.
> >
> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
>
> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> (And so additional configq follows the natural model for config space access).
This is only true if:
1) you don't want to provide any backward compatibility for current
PCI transport, this means you need to use new drivers in the guest
2) you don't want to do live migration
If you need one of the above, PASID is a must.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 3:51 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 3:51 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 8, 2023 11:45 PM
>
> > > The fundamental reason for not accessing the 1.x VF and SIOV device
> > > registers, config space, feature bits through PF is: it requires PF
> > > device mediation. VF and SIOV devices are first class citizen in PCIe
> > > spec and deserve direct interaction with the device.
> >
> > Unless I miss something obvious, SIOV requires mediation (or
> > composition) for sure. Otherwise you break the compatibility.
> >
> SIOV does not require mediation.
> Composition is optional like VFs.
This is not what I read from SIOV spec.
>
> > >
> > > Hence, the transport we built is to consider this in mind for the
> > > coming future.
> >
> > For transport virtqueue, it's not specific to PCI. It could be used in a much
> > broader use case.
> >
> May be.
> More below.
>
> > > So if each VF has its own configq, or cmdq, it totally make sense to
> > > me which is bootstrap interface to transport existing config space interface.
> > > The problem is: it is not backward compatible; Hence a device has no
> > > way of when to support both or only new configq.
> >
> > Providing compatibility in software is much more simpler than inventing new
> > hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> > a SIOV device). And inventing a new hardware interface for compatibility might
> > not always work, it may break the advantages of the new hardware (like
> > scalability).
> >
> What I proposed below is new hardware interface for new features.
> Not for compatibility.
I'm confused, the proposal is for legacy drives so it's for
compatibility for sure. No?
> Compatibility is coming at a cost which is not scaling.
> If you attempt to scale, it breaks the future path forward to go without mediation.
>
> > >
> > > So eve growing these fields and optionally placement on configq
> > > doesn't really help and device builder to build it efficiently
> > > (without much predictability).
> >
> > Config queue is not the only choice, we have a lot of other choices (for example
> > PASID may help to reduce the on-chip resources).
> >
> PASID is for process isolation security construct, it is not for transportation.
I meant with PASID you don't even need a BAR.
>
> > >
> > > Instead of we say, that what exists today in config space stays in
> > > config space, anything additional on new q, than its deterministic
> > > behavior to size up the scale.
> >
> > Just to be clear, if we have PCI over adminq, VF's config space could be the
> > minimal one for PCI spec complaint. The real config space is accessed via the
> > admin virtqueue.
> >
> Yeah I understood what you are saying. We don’t see a path forward with such mediation.
> VF and PF to have same level of direct access to the device.
>
> > >
> > > For example, a PCI device who wants to support 100 VFs, can easily
> > > size its memory to 30 bytes * 100 reserved for supporting config space.
> >
> > Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> > need to place them in the config space.
> >
> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
Unless you don't want to provide VF compatibility for SIOV.
>
> > > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> > >
> > > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > > used (reserved) as 3000/(30+40) = 42 VFs.
> > >
> > > Only if some VFs use configq, more VFs can be deployed.
> >
> > I don't understand here.
> >
> Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
>
> > > It is hard to
> > > build scale this way. Therefore suggestion is to place new attributes
> > > on new config/cmd/transport q, and old to stay as-is.
> >
> > Just to be sure we're on the same page. The proposal of both you and mine are
> > based on the adminq for PF not VF. The reason is obvious:
> > adminq per VF won't work without PASID, since it would have security issues.
> >
> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
>
> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> (And so additional configq follows the natural model for config space access).
This is only true if:
1) you don't want to provide any backward compatibility for current
PCI transport, this means you need to use new drivers in the guest
2) you don't want to do live migration
If you need one of the above, PASID is a must.
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 3:51 ` [virtio-comment] " Jason Wang
@ 2023-05-10 4:22 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 4:22 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Monday, May 8, 2023 11:45 PM
> >
> > > > The fundamental reason for not accessing the 1.x VF and SIOV device
> > > > registers, config space, feature bits through PF is: it requires PF
> > > > device mediation. VF and SIOV devices are first class citizen in PCIe
> > > > spec and deserve direct interaction with the device.
> > >
> > > Unless I miss something obvious, SIOV requires mediation (or
> > > composition) for sure. Otherwise you break the compatibility.
> > >
> > SIOV does not require mediation.
> > Composition is optional like VFs.
>
> This is not what I read from SIOV spec.
>
> >
> > > >
> > > > Hence, the transport we built is to consider this in mind for the
> > > > coming future.
> > >
> > > For transport virtqueue, it's not specific to PCI. It could be used in a much
> > > broader use case.
> > >
> > May be.
> > More below.
> >
> > > > So if each VF has its own configq, or cmdq, it totally make sense to
> > > > me which is bootstrap interface to transport existing config space interface.
> > > > The problem is: it is not backward compatible; Hence a device has no
> > > > way of when to support both or only new configq.
> > >
> > > Providing compatibility in software is much more simpler than inventing new
> > > hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> > > a SIOV device). And inventing a new hardware interface for compatibility might
> > > not always work, it may break the advantages of the new hardware (like
> > > scalability).
> > >
> > What I proposed below is new hardware interface for new features.
> > Not for compatibility.
>
> I'm confused, the proposal is for legacy drives so it's for
> compatibility for sure. No?
>
> > Compatibility is coming at a cost which is not scaling.
> > If you attempt to scale, it breaks the future path forward to go without mediation.
> >
> > > >
> > > > So eve growing these fields and optionally placement on configq
> > > > doesn't really help and device builder to build it efficiently
> > > > (without much predictability).
> > >
> > > Config queue is not the only choice, we have a lot of other choices (for example
> > > PASID may help to reduce the on-chip resources).
> > >
> > PASID is for process isolation security construct, it is not for transportation.
>
> I meant with PASID you don't even need a BAR.
>
> >
> > > >
> > > > Instead of we say, that what exists today in config space stays in
> > > > config space, anything additional on new q, than its deterministic
> > > > behavior to size up the scale.
> > >
> > > Just to be clear, if we have PCI over adminq, VF's config space could be the
> > > minimal one for PCI spec complaint. The real config space is accessed via the
> > > admin virtqueue.
> > >
> > Yeah I understood what you are saying. We don’t see a path forward with such mediation.
> > VF and PF to have same level of direct access to the device.
> >
> > > >
> > > > For example, a PCI device who wants to support 100 VFs, can easily
> > > > size its memory to 30 bytes * 100 reserved for supporting config space.
> > >
> > > Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> > > need to place them in the config space.
> > >
> > Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
>
> Unless you don't want to provide VF compatibility for SIOV.
>
> >
> > > > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> > > >
> > > > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > > > used (reserved) as 3000/(30+40) = 42 VFs.
> > > >
> > > > Only if some VFs use configq, more VFs can be deployed.
> > >
> > > I don't understand here.
> > >
> > Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
> >
> > > > It is hard to
> > > > build scale this way. Therefore suggestion is to place new attributes
> > > > on new config/cmd/transport q, and old to stay as-is.
> > >
> > > Just to be sure we're on the same page. The proposal of both you and mine are
> > > based on the adminq for PF not VF. The reason is obvious:
> > > adminq per VF won't work without PASID, since it would have security issues.
> > >
> > The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
> >
> > adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> > Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> > (And so additional configq follows the natural model for config space access).
>
> This is only true if:
>
> 1) you don't want to provide any backward compatibility for current
> PCI transport, this means you need to use new drivers in the guest
> 2) you don't want to do live migration
>
> If you need one of the above, PASID is a must.
Just to make sure we are at the same page.
1) if the hardware has configq and we need to make it work for current
virtio-pci driver, hypervisor needs to trap guest PCI access and
translate it to configq command. This means the hypervisor needs to
hide the configq from guests. In this case the configq needs a
dedicated DMA address which is what PASID can help.
2) if the hardware can report the device states, unless we want to
migrate L2 guest, hypervisor should hide it from L1, so PASID is
required to isolate the DMA for guest traffic and device state.
Thanks
>
> Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 4:22 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 4:22 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Monday, May 8, 2023 11:45 PM
> >
> > > > The fundamental reason for not accessing the 1.x VF and SIOV device
> > > > registers, config space, feature bits through PF is: it requires PF
> > > > device mediation. VF and SIOV devices are first class citizen in PCIe
> > > > spec and deserve direct interaction with the device.
> > >
> > > Unless I miss something obvious, SIOV requires mediation (or
> > > composition) for sure. Otherwise you break the compatibility.
> > >
> > SIOV does not require mediation.
> > Composition is optional like VFs.
>
> This is not what I read from SIOV spec.
>
> >
> > > >
> > > > Hence, the transport we built is to consider this in mind for the
> > > > coming future.
> > >
> > > For transport virtqueue, it's not specific to PCI. It could be used in a much
> > > broader use case.
> > >
> > May be.
> > More below.
> >
> > > > So if each VF has its own configq, or cmdq, it totally make sense to
> > > > me which is bootstrap interface to transport existing config space interface.
> > > > The problem is: it is not backward compatible; Hence a device has no
> > > > way of when to support both or only new configq.
> > >
> > > Providing compatibility in software is much more simpler than inventing new
> > > hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> > > a SIOV device). And inventing a new hardware interface for compatibility might
> > > not always work, it may break the advantages of the new hardware (like
> > > scalability).
> > >
> > What I proposed below is new hardware interface for new features.
> > Not for compatibility.
>
> I'm confused, the proposal is for legacy drives so it's for
> compatibility for sure. No?
>
> > Compatibility is coming at a cost which is not scaling.
> > If you attempt to scale, it breaks the future path forward to go without mediation.
> >
> > > >
> > > > So eve growing these fields and optionally placement on configq
> > > > doesn't really help and device builder to build it efficiently
> > > > (without much predictability).
> > >
> > > Config queue is not the only choice, we have a lot of other choices (for example
> > > PASID may help to reduce the on-chip resources).
> > >
> > PASID is for process isolation security construct, it is not for transportation.
>
> I meant with PASID you don't even need a BAR.
>
> >
> > > >
> > > > Instead of we say, that what exists today in config space stays in
> > > > config space, anything additional on new q, than its deterministic
> > > > behavior to size up the scale.
> > >
> > > Just to be clear, if we have PCI over adminq, VF's config space could be the
> > > minimal one for PCI spec complaint. The real config space is accessed via the
> > > admin virtqueue.
> > >
> > Yeah I understood what you are saying. We don’t see a path forward with such mediation.
> > VF and PF to have same level of direct access to the device.
> >
> > > >
> > > > For example, a PCI device who wants to support 100 VFs, can easily
> > > > size its memory to 30 bytes * 100 reserved for supporting config space.
> > >
> > > Those capabilities (30 bytes) can be accessed via admin virtqueue. So we don't
> > > need to place them in the config space.
> > >
> > Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
>
> Unless you don't want to provide VF compatibility for SIOV.
>
> >
> > > > And new 40 bytes * 100 fields doesn't have to be in the resident memory.
> > > >
> > > > If we have optional configq/cmdq for transport, than 30*100 bytes are
> > > > used (reserved) as 3000/(30+40) = 42 VFs.
> > > >
> > > > Only if some VFs use configq, more VFs can be deployed.
> > >
> > > I don't understand here.
> > >
> > Lets first clarify the non-mediated passthrough model of the VF first than come to above scale example.
> >
> > > > It is hard to
> > > > build scale this way. Therefore suggestion is to place new attributes
> > > > on new config/cmd/transport q, and old to stay as-is.
> > >
> > > Just to be sure we're on the same page. The proposal of both you and mine are
> > > based on the adminq for PF not VF. The reason is obvious:
> > > adminq per VF won't work without PASID, since it would have security issues.
> > >
> > The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
> >
> > adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> > Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> > (And so additional configq follows the natural model for config space access).
>
> This is only true if:
>
> 1) you don't want to provide any backward compatibility for current
> PCI transport, this means you need to use new drivers in the guest
> 2) you don't want to do live migration
>
> If you need one of the above, PASID is a must.
Just to make sure we are at the same page.
1) if the hardware has configq and we need to make it work for current
virtio-pci driver, hypervisor needs to trap guest PCI access and
translate it to configq command. This means the hypervisor needs to
hide the configq from guests. In this case the configq needs a
dedicated DMA address which is what PASID can help.
2) if the hardware can report the device states, unless we want to
migrate L2 guest, hypervisor should hide it from L1, so PASID is
required to isolate the DMA for guest traffic and device state.
Thanks
>
> Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-08 2:23 ` [virtio-comment] " Jason Wang
@ 2023-05-10 6:04 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 6:04 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > I thought so too originally. Unfortunately I now think that no, legacy is not
> > going to be a byproduct of transport virtqueue for modern -
> > it is different enough that it needs dedicated commands.
>
> If you mean the transport virtqueue, I think some dedicated commands
> for legacy are needed. Then it would be a transport that supports
> transitional devices. It would be much better than having commands for
> a partial transport like this patch did.
OK I am beginning to get what you are saying. So your criticism is
this: what if device supports vq transport for modern, and we want to
build a transitional device on top. how will that look. yes?
A reasonable thing to include at least in the commit log. Parav?
You are also asking what if the device uses transport vq,
and we want transitional on top of that.
It's a fair question but I don't exactly get why would
this legacy support feature be wanted for the vq transport
and not for other transports.
> > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > modern does not even need multibyte read.
>
> I'm not sure I will get here,
What does this mean?
> since we can't expose admin vq to
> guests, it means we need some software mediation. So if we just
> implement what PCI allows us, then everything would be fine (even if
> some method is not used).
>
> Thanks
To repeat discussion on one of the previous versions, no it will not be
fine because legacy virtio abuses pci in fundamentally broken ways.
So yes you need a mediator on the host but even giving this
mediator a chance to be robust on top of hardware
means the hardware interface can not simply mirror legacy
to hardware.
For example, host mediator needs to trap writes into mac,
buffer them and then send a 6 byte write.
Now, pci actually does allow 6 byte writes, but legacy
guests instead to 6 single byte writes for portability reasons.
--
MSr
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 6:04 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 6:04 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > I thought so too originally. Unfortunately I now think that no, legacy is not
> > going to be a byproduct of transport virtqueue for modern -
> > it is different enough that it needs dedicated commands.
>
> If you mean the transport virtqueue, I think some dedicated commands
> for legacy are needed. Then it would be a transport that supports
> transitional devices. It would be much better than having commands for
> a partial transport like this patch did.
OK I am beginning to get what you are saying. So your criticism is
this: what if device supports vq transport for modern, and we want to
build a transitional device on top. how will that look. yes?
A reasonable thing to include at least in the commit log. Parav?
You are also asking what if the device uses transport vq,
and we want transitional on top of that.
It's a fair question but I don't exactly get why would
this legacy support feature be wanted for the vq transport
and not for other transports.
> > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > modern does not even need multibyte read.
>
> I'm not sure I will get here,
What does this mean?
> since we can't expose admin vq to
> guests, it means we need some software mediation. So if we just
> implement what PCI allows us, then everything would be fine (even if
> some method is not used).
>
> Thanks
To repeat discussion on one of the previous versions, no it will not be
fine because legacy virtio abuses pci in fundamentally broken ways.
So yes you need a mediator on the host but even giving this
mediator a chance to be robust on top of hardware
means the hardware interface can not simply mirror legacy
to hardware.
For example, host mediator needs to trap writes into mac,
buffer them and then send a 6 byte write.
Now, pci actually does allow 6 byte writes, but legacy
guests instead to 6 single byte writes for portability reasons.
--
MSr
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 6:04 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-10 7:01 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 7:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > going to be a byproduct of transport virtqueue for modern -
> > > it is different enough that it needs dedicated commands.
> >
> > If you mean the transport virtqueue, I think some dedicated commands
> > for legacy are needed. Then it would be a transport that supports
> > transitional devices. It would be much better than having commands for
> > a partial transport like this patch did.
>
> OK I am beginning to get what you are saying. So your criticism is
> this: what if device supports vq transport for modern, and we want to
> build a transitional device on top. how will that look. yes?
Yes. I think it needs to be done through the transport virtqueue
otherwise the transport is not self-contained.
> A reasonable thing to include at least in the commit log. Parav?
>
> You are also asking what if the device uses transport vq,
> and we want transitional on top of that.
> It's a fair question but I don't exactly get why would
> this legacy support feature be wanted for the vq transport
> and not for other transports.
Not sure I get the question, but all the existing transport support
legacy, if we want to have another, should the legacy support be a
must or not?
>
>
>
>
> > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > modern does not even need multibyte read.
> >
> > I'm not sure I will get here,
>
> What does this mean?
I meant I don't get what the issue if "modern does not even need
multibyte read".
>
> > since we can't expose admin vq to
> > guests, it means we need some software mediation. So if we just
> > implement what PCI allows us, then everything would be fine (even if
> > some method is not used).
> >
> > Thanks
>
> To repeat discussion on one of the previous versions, no it will not be
> fine because legacy virtio abuses pci in fundamentally broken ways.
> So yes you need a mediator on the host but even giving this
> mediator a chance to be robust on top of hardware
> means the hardware interface can not simply mirror legacy
> to hardware.
>
> For example, host mediator needs to trap writes into mac,
> buffer them and then send a 6 byte write.
> Now, pci actually does allow 6 byte writes, but legacy
> guests instead to 6 single byte writes for portability reasons.
It's a known race condition, so PCI over adminq doesn't make it worse.
The mediator can just mirror what guests write over the admin
commands.
Thanks
> --
> MSr
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 7:01 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-10 7:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > going to be a byproduct of transport virtqueue for modern -
> > > it is different enough that it needs dedicated commands.
> >
> > If you mean the transport virtqueue, I think some dedicated commands
> > for legacy are needed. Then it would be a transport that supports
> > transitional devices. It would be much better than having commands for
> > a partial transport like this patch did.
>
> OK I am beginning to get what you are saying. So your criticism is
> this: what if device supports vq transport for modern, and we want to
> build a transitional device on top. how will that look. yes?
Yes. I think it needs to be done through the transport virtqueue
otherwise the transport is not self-contained.
> A reasonable thing to include at least in the commit log. Parav?
>
> You are also asking what if the device uses transport vq,
> and we want transitional on top of that.
> It's a fair question but I don't exactly get why would
> this legacy support feature be wanted for the vq transport
> and not for other transports.
Not sure I get the question, but all the existing transport support
legacy, if we want to have another, should the legacy support be a
must or not?
>
>
>
>
> > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > modern does not even need multibyte read.
> >
> > I'm not sure I will get here,
>
> What does this mean?
I meant I don't get what the issue if "modern does not even need
multibyte read".
>
> > since we can't expose admin vq to
> > guests, it means we need some software mediation. So if we just
> > implement what PCI allows us, then everything would be fine (even if
> > some method is not used).
> >
> > Thanks
>
> To repeat discussion on one of the previous versions, no it will not be
> fine because legacy virtio abuses pci in fundamentally broken ways.
> So yes you need a mediator on the host but even giving this
> mediator a chance to be robust on top of hardware
> means the hardware interface can not simply mirror legacy
> to hardware.
>
> For example, host mediator needs to trap writes into mac,
> buffer them and then send a 6 byte write.
> Now, pci actually does allow 6 byte writes, but legacy
> guests instead to 6 single byte writes for portability reasons.
It's a known race condition, so PCI over adminq doesn't make it worse.
The mediator can just mirror what guests write over the admin
commands.
Thanks
> --
> MSr
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 7:01 ` [virtio-comment] " Jason Wang
@ 2023-05-10 7:43 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 7:43 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > going to be a byproduct of transport virtqueue for modern -
> > > > it is different enough that it needs dedicated commands.
> > >
> > > If you mean the transport virtqueue, I think some dedicated commands
> > > for legacy are needed. Then it would be a transport that supports
> > > transitional devices. It would be much better than having commands for
> > > a partial transport like this patch did.
> >
> > OK I am beginning to get what you are saying. So your criticism is
> > this: what if device supports vq transport for modern, and we want to
> > build a transitional device on top. how will that look. yes?
>
> Yes. I think it needs to be done through the transport virtqueue
> otherwise the transport is not self-contained.
I mean, any feature can be done over transport vq.
But there is value in adding legacy commands to an otherwise
modern device without reworking it completely to
switch to a different transport.
> > A reasonable thing to include at least in the commit log. Parav?
> >
> > You are also asking what if the device uses transport vq,
> > and we want transitional on top of that.
> > It's a fair question but I don't exactly get why would
> > this legacy support feature be wanted for the vq transport
> > and not for other transports.
>
> Not sure I get the question, but all the existing transport support
> legacy, if we want to have another, should the legacy support be a
> must or not?
This specific proposal is for tunneling legacy over admin vq.
It can live alongside a normal modern VF, with hypervisor
combining these to create a transitional device.
> >
> >
> >
> >
> > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > modern does not even need multibyte read.
> > >
> > > I'm not sure I will get here,
> >
> > What does this mean?
>
> I meant I don't get what the issue if "modern does not even need
> multibyte read".
parse error again. reword?
> >
> > > since we can't expose admin vq to
> > > guests, it means we need some software mediation. So if we just
> > > implement what PCI allows us, then everything would be fine (even if
> > > some method is not used).
> > >
> > > Thanks
> >
> > To repeat discussion on one of the previous versions, no it will not be
> > fine because legacy virtio abuses pci in fundamentally broken ways.
> > So yes you need a mediator on the host but even giving this
> > mediator a chance to be robust on top of hardware
> > means the hardware interface can not simply mirror legacy
> > to hardware.
> >
> > For example, host mediator needs to trap writes into mac,
> > buffer them and then send a 6 byte write.
> > Now, pci actually does allow 6 byte writes, but legacy
> > guests instead to 6 single byte writes for portability reasons.
>
> It's a known race condition, so PCI over adminq doesn't make it worse.
it can however make it better - you can do a single 6 byte write command.
> The mediator can just mirror what guests write over the admin
> commands.
and this "just" just isn't good enough, or we end up with hacks
in hardware.
> Thanks
>
> > --
> > MSr
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 7:43 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 7:43 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > going to be a byproduct of transport virtqueue for modern -
> > > > it is different enough that it needs dedicated commands.
> > >
> > > If you mean the transport virtqueue, I think some dedicated commands
> > > for legacy are needed. Then it would be a transport that supports
> > > transitional devices. It would be much better than having commands for
> > > a partial transport like this patch did.
> >
> > OK I am beginning to get what you are saying. So your criticism is
> > this: what if device supports vq transport for modern, and we want to
> > build a transitional device on top. how will that look. yes?
>
> Yes. I think it needs to be done through the transport virtqueue
> otherwise the transport is not self-contained.
I mean, any feature can be done over transport vq.
But there is value in adding legacy commands to an otherwise
modern device without reworking it completely to
switch to a different transport.
> > A reasonable thing to include at least in the commit log. Parav?
> >
> > You are also asking what if the device uses transport vq,
> > and we want transitional on top of that.
> > It's a fair question but I don't exactly get why would
> > this legacy support feature be wanted for the vq transport
> > and not for other transports.
>
> Not sure I get the question, but all the existing transport support
> legacy, if we want to have another, should the legacy support be a
> must or not?
This specific proposal is for tunneling legacy over admin vq.
It can live alongside a normal modern VF, with hypervisor
combining these to create a transitional device.
> >
> >
> >
> >
> > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > modern does not even need multibyte read.
> > >
> > > I'm not sure I will get here,
> >
> > What does this mean?
>
> I meant I don't get what the issue if "modern does not even need
> multibyte read".
parse error again. reword?
> >
> > > since we can't expose admin vq to
> > > guests, it means we need some software mediation. So if we just
> > > implement what PCI allows us, then everything would be fine (even if
> > > some method is not used).
> > >
> > > Thanks
> >
> > To repeat discussion on one of the previous versions, no it will not be
> > fine because legacy virtio abuses pci in fundamentally broken ways.
> > So yes you need a mediator on the host but even giving this
> > mediator a chance to be robust on top of hardware
> > means the hardware interface can not simply mirror legacy
> > to hardware.
> >
> > For example, host mediator needs to trap writes into mac,
> > buffer them and then send a 6 byte write.
> > Now, pci actually does allow 6 byte writes, but legacy
> > guests instead to 6 single byte writes for portability reasons.
>
> It's a known race condition, so PCI over adminq doesn't make it worse.
it can however make it better - you can do a single 6 byte write command.
> The mediator can just mirror what guests write over the admin
> commands.
and this "just" just isn't good enough, or we end up with hacks
in hardware.
> Thanks
>
> > --
> > MSr
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 3:51 ` [virtio-comment] " Jason Wang
@ 2023-05-10 16:04 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:04 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On 5/9/2023 11:51 PM, Jason Wang wrote:
> On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
>>
>> SIOV does not require mediation.
>> Composition is optional like VFs.
>
> This is not what I read from SIOV spec.
>
SIOV R1 spec was defined by _a_ vendor and it is undergoing upgrade to
more practical use now. So we should let them finish the work.
It is hard to comment in this forum about it.
>>
>>>>
>>>> Hence, the transport we built is to consider this in mind for the
>>>> coming future.
>>>
>>> For transport virtqueue, it's not specific to PCI. It could be used in a much
>>> broader use case.
>>>
>> May be.
>> More below.
>>
>>>> So if each VF has its own configq, or cmdq, it totally make sense to
>>>> me which is bootstrap interface to transport existing config space interface.
>>>> The problem is: it is not backward compatible; Hence a device has no
>>>> way of when to support both or only new configq.
>>>
>>> Providing compatibility in software is much more simpler than inventing new
>>> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
>>> a SIOV device). And inventing a new hardware interface for compatibility might
>>> not always work, it may break the advantages of the new hardware (like
>>> scalability).
>>>
>> What I proposed below is new hardware interface for new features.
>> Not for compatibility.
>
> I'm confused, the proposal is for legacy drives so it's for
> compatibility for sure. No?
This proposal is to get legacy devices to work over a PCI VF.
a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
devices to work in _non_ backward compatible way, because for new
features there is no such thing as backward compatibility between guest
driver and the device.
>
>> Compatibility is coming at a cost which is not scaling.
>> If you attempt to scale, it breaks the future path forward to go without mediation.
>>
>>>>
>>>> So eve growing these fields and optionally placement on configq
>>>> doesn't really help and device builder to build it efficiently
>>>> (without much predictability).
>>>
>>> Config queue is not the only choice, we have a lot of other choices (for example
>>> PASID may help to reduce the on-chip resources).
>>>
>> PASID is for process isolation security construct, it is not for transportation.
>
> I meant with PASID you don't even need a BAR.
>
Not sure I fully understand, but my guess is, this is coming from
mediation thought process.
>> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
>
> Unless you don't want to provide VF compatibility for SIOV.
I do not understand what compatibility is this between which two elements?
VF has its identify currently and SIOV will have its own identity in future.
>>>
>>> Just to be sure we're on the same page. The proposal of both you and mine are
>>> based on the adminq for PF not VF. The reason is obvious:
>>> adminq per VF won't work without PASID, since it would have security issues.
>>>
>> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
>>
>> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
>> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
>> (And so additional configq follows the natural model for config space access).
>
> This is only true if:
>
> 1) you don't want to provide any backward compatibility for current
> PCI transport,
There is no concept of backward compatibility for new features.
There will be new driver anyway in the guest.
yes, I don't see a point in mediating 1.x config space for existing
drivers as it is requires mediation.
> this means you need to use new drivers in the guest
> 2) you don't want to do live migration
>
Not sure how you assert this.
Live migration at VF level for the passthrough device can be just fine
without any mediation as long as all the parameters on src and dst side
match.
We already discussed in v0, so I prefer to avoid this again and focus on
the patch in discussion.
> If you need one of the above, PASID is a must.
A PCI VF passthrough device LM without mediation is achievable without
PASID for all the existing defined features using config space.
and also possible when new features are communicated over configq/cmdq.
Transporting existing config space using some additional queue is _not_
the objective, because building a scalable system where device has zero
knowledge about needed backward compatibility is hard.
I provided the example of 100 Vfs with 30 and (30+40) bytes config
layout previously.
Yet we are repeating and diverging from the discussion, that we should
not intermix:
1. How to access legacy registers of the VF
vs
2. How to access existing or new configuration of the VF
Because both have very different requirements.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 16:04 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:04 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On 5/9/2023 11:51 PM, Jason Wang wrote:
> On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
>>
>> SIOV does not require mediation.
>> Composition is optional like VFs.
>
> This is not what I read from SIOV spec.
>
SIOV R1 spec was defined by _a_ vendor and it is undergoing upgrade to
more practical use now. So we should let them finish the work.
It is hard to comment in this forum about it.
>>
>>>>
>>>> Hence, the transport we built is to consider this in mind for the
>>>> coming future.
>>>
>>> For transport virtqueue, it's not specific to PCI. It could be used in a much
>>> broader use case.
>>>
>> May be.
>> More below.
>>
>>>> So if each VF has its own configq, or cmdq, it totally make sense to
>>>> me which is bootstrap interface to transport existing config space interface.
>>>> The problem is: it is not backward compatible; Hence a device has no
>>>> way of when to support both or only new configq.
>>>
>>> Providing compatibility in software is much more simpler than inventing new
>>> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
>>> a SIOV device). And inventing a new hardware interface for compatibility might
>>> not always work, it may break the advantages of the new hardware (like
>>> scalability).
>>>
>> What I proposed below is new hardware interface for new features.
>> Not for compatibility.
>
> I'm confused, the proposal is for legacy drives so it's for
> compatibility for sure. No?
This proposal is to get legacy devices to work over a PCI VF.
a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
devices to work in _non_ backward compatible way, because for new
features there is no such thing as backward compatibility between guest
driver and the device.
>
>> Compatibility is coming at a cost which is not scaling.
>> If you attempt to scale, it breaks the future path forward to go without mediation.
>>
>>>>
>>>> So eve growing these fields and optionally placement on configq
>>>> doesn't really help and device builder to build it efficiently
>>>> (without much predictability).
>>>
>>> Config queue is not the only choice, we have a lot of other choices (for example
>>> PASID may help to reduce the on-chip resources).
>>>
>> PASID is for process isolation security construct, it is not for transportation.
>
> I meant with PASID you don't even need a BAR.
>
Not sure I fully understand, but my guess is, this is coming from
mediation thought process.
>> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
>
> Unless you don't want to provide VF compatibility for SIOV.
I do not understand what compatibility is this between which two elements?
VF has its identify currently and SIOV will have its own identity in future.
>>>
>>> Just to be sure we're on the same page. The proposal of both you and mine are
>>> based on the adminq for PF not VF. The reason is obvious:
>>> adminq per VF won't work without PASID, since it would have security issues.
>>>
>> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
>>
>> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
>> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
>> (And so additional configq follows the natural model for config space access).
>
> This is only true if:
>
> 1) you don't want to provide any backward compatibility for current
> PCI transport,
There is no concept of backward compatibility for new features.
There will be new driver anyway in the guest.
yes, I don't see a point in mediating 1.x config space for existing
drivers as it is requires mediation.
> this means you need to use new drivers in the guest
> 2) you don't want to do live migration
>
Not sure how you assert this.
Live migration at VF level for the passthrough device can be just fine
without any mediation as long as all the parameters on src and dst side
match.
We already discussed in v0, so I prefer to avoid this again and focus on
the patch in discussion.
> If you need one of the above, PASID is a must.
A PCI VF passthrough device LM without mediation is achievable without
PASID for all the existing defined features using config space.
and also possible when new features are communicated over configq/cmdq.
Transporting existing config space using some additional queue is _not_
the objective, because building a scalable system where device has zero
knowledge about needed backward compatibility is hard.
I provided the example of 100 Vfs with 30 and (30+40) bytes config
layout previously.
Yet we are repeating and diverging from the discussion, that we should
not intermix:
1. How to access legacy registers of the VF
vs
2. How to access existing or new configuration of the VF
Because both have very different requirements.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 4:22 ` [virtio-comment] " Jason Wang
@ 2023-05-10 16:07 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:07 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On 5/10/2023 12:22 AM, Jason Wang wrote:
> On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Just to make sure we are at the same page.
>
> 1) if the hardware has configq and we need to make it work for current
> virtio-pci driver, hypervisor needs to trap guest PCI access and
> translate it to configq command. This means the hypervisor needs to
> hide the configq from guests. In this case the configq needs a
> dedicated DMA address which is what PASID can help.
> 2) if the hardware can report the device states, unless we want to
> migrate L2 guest, hypervisor should hide it from L1, so PASID is
> required to isolate the DMA for guest traffic and device state.
A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
devices over PCIe or other transports.
So no need to hide/mediate for hw based devices, like cvq and like data vqs.
For vdpa kind of use case it can work like cvq mediation, which I
believe is happening without PASID today.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 16:07 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:07 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On 5/10/2023 12:22 AM, Jason Wang wrote:
> On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> Just to make sure we are at the same page.
>
> 1) if the hardware has configq and we need to make it work for current
> virtio-pci driver, hypervisor needs to trap guest PCI access and
> translate it to configq command. This means the hypervisor needs to
> hide the configq from guests. In this case the configq needs a
> dedicated DMA address which is what PASID can help.
> 2) if the hardware can report the device states, unless we want to
> migrate L2 guest, hypervisor should hide it from L1, so PASID is
> required to isolate the DMA for guest traffic and device state.
A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
devices over PCIe or other transports.
So no need to hide/mediate for hw based devices, like cvq and like data vqs.
For vdpa kind of use case it can work like cvq mediation, which I
believe is happening without PASID today.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 6:04 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-10 16:11 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:11 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>> going to be a byproduct of transport virtqueue for modern -
>>> it is different enough that it needs dedicated commands.
>>
>> If you mean the transport virtqueue, I think some dedicated commands
>> for legacy are needed. Then it would be a transport that supports
>> transitional devices. It would be much better than having commands for
>> a partial transport like this patch did.
>
> OK I am beginning to get what you are saying. So your criticism is
> this: what if device supports vq transport for modern, and we want to
> build a transitional device on top. how will that look. yes?
> A reasonable thing to include at least in the commit log. Parav?
>
I am still trying to understand what is "vq transport for modern"?
Do you mean transporting currently defined config space access over vq?
If so, is this VQ belong to the guest or hypervisor?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 16:11 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:11 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>> going to be a byproduct of transport virtqueue for modern -
>>> it is different enough that it needs dedicated commands.
>>
>> If you mean the transport virtqueue, I think some dedicated commands
>> for legacy are needed. Then it would be a transport that supports
>> transitional devices. It would be much better than having commands for
>> a partial transport like this patch did.
>
> OK I am beginning to get what you are saying. So your criticism is
> this: what if device supports vq transport for modern, and we want to
> build a transitional device on top. how will that look. yes?
> A reasonable thing to include at least in the commit log. Parav?
>
I am still trying to understand what is "vq transport for modern"?
Do you mean transporting currently defined config space access over vq?
If so, is this VQ belong to the guest or hypervisor?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 7:43 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-10 16:13 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:13 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/10/2023 3:43 AM, Michael S. Tsirkin wrote:
> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
>> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>
>>> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>>>> going to be a byproduct of transport virtqueue for modern -
>>>>> it is different enough that it needs dedicated commands.
>>>>
>>>> If you mean the transport virtqueue, I think some dedicated commands
>>>> for legacy are needed. Then it would be a transport that supports
>>>> transitional devices. It would be much better than having commands for
>>>> a partial transport like this patch did.
>>>
>>> OK I am beginning to get what you are saying. So your criticism is
>>> this: what if device supports vq transport for modern, and we want to
>>> build a transitional device on top. how will that look. yes?
>>
>> Yes. I think it needs to be done through the transport virtqueue
>> otherwise the transport is not self-contained.
>
> I mean, any feature can be done over transport vq.
>
> But there is value in adding legacy commands to an otherwise
> modern device without reworking it completely to
> switch to a different transport.
>
Yes.
>
>>> A reasonable thing to include at least in the commit log. Parav?
>>>
>>> You are also asking what if the device uses transport vq,
>>> and we want transitional on top of that.
>>> It's a fair question but I don't exactly get why would
>>> this legacy support feature be wanted for the vq transport
>>> and not for other transports.
>>
>> Not sure I get the question, but all the existing transport support
>> legacy, if we want to have another, should the legacy support be a
>> must or not?
>
> This specific proposal is for tunneling legacy over admin vq.
> It can live alongside a normal modern VF, with hypervisor
> combining these to create a transitional device.
>
True.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 16:13 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 16:13 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, yishaih, maorg,
virtio-comment, shahafs
On 5/10/2023 3:43 AM, Michael S. Tsirkin wrote:
> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
>> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>
>>> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>>>> going to be a byproduct of transport virtqueue for modern -
>>>>> it is different enough that it needs dedicated commands.
>>>>
>>>> If you mean the transport virtqueue, I think some dedicated commands
>>>> for legacy are needed. Then it would be a transport that supports
>>>> transitional devices. It would be much better than having commands for
>>>> a partial transport like this patch did.
>>>
>>> OK I am beginning to get what you are saying. So your criticism is
>>> this: what if device supports vq transport for modern, and we want to
>>> build a transitional device on top. how will that look. yes?
>>
>> Yes. I think it needs to be done through the transport virtqueue
>> otherwise the transport is not self-contained.
>
> I mean, any feature can be done over transport vq.
>
> But there is value in adding legacy commands to an otherwise
> modern device without reworking it completely to
> switch to a different transport.
>
Yes.
>
>>> A reasonable thing to include at least in the commit log. Parav?
>>>
>>> You are also asking what if the device uses transport vq,
>>> and we want transitional on top of that.
>>> It's a fair question but I don't exactly get why would
>>> this legacy support feature be wanted for the vq transport
>>> and not for other transports.
>>
>> Not sure I get the question, but all the existing transport support
>> legacy, if we want to have another, should the legacy support be a
>> must or not?
>
> This specific proposal is for tunneling legacy over admin vq.
> It can live alongside a normal modern VF, with hypervisor
> combining these to create a transitional device.
>
True.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 16:11 ` [virtio-dev] " Parav Pandit
@ 2023-05-10 16:16 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 16:16 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla, yishaih,
maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
>
>
> On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > going to be a byproduct of transport virtqueue for modern -
> > > > it is different enough that it needs dedicated commands.
> > >
> > > If you mean the transport virtqueue, I think some dedicated commands
> > > for legacy are needed. Then it would be a transport that supports
> > > transitional devices. It would be much better than having commands for
> > > a partial transport like this patch did.
> >
> > OK I am beginning to get what you are saying. So your criticism is
> > this: what if device supports vq transport for modern, and we want to
> > build a transitional device on top. how will that look. yes?
> > A reasonable thing to include at least in the commit log. Parav?
> >
> I am still trying to understand what is "vq transport for modern"?
> Do you mean transporting currently defined config space access over vq?
> If so, is this VQ belong to the guest or hypervisor?
https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu%40intel.com/t.mbox.gz
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 16:16 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 16:16 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla, yishaih,
maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
>
>
> On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > going to be a byproduct of transport virtqueue for modern -
> > > > it is different enough that it needs dedicated commands.
> > >
> > > If you mean the transport virtqueue, I think some dedicated commands
> > > for legacy are needed. Then it would be a transport that supports
> > > transitional devices. It would be much better than having commands for
> > > a partial transport like this patch did.
> >
> > OK I am beginning to get what you are saying. So your criticism is
> > this: what if device supports vq transport for modern, and we want to
> > build a transitional device on top. how will that look. yes?
> > A reasonable thing to include at least in the commit log. Parav?
> >
> I am still trying to understand what is "vq transport for modern"?
> Do you mean transporting currently defined config space access over vq?
> If so, is this VQ belong to the guest or hypervisor?
https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu%40intel.com/t.mbox.gz
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 16:16 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-10 17:33 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 17:33 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 10, 2023 12:16 PM
>
> On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> >
> >
> > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > I thought so too originally. Unfortunately I now think that no,
> > > > > legacy is not going to be a byproduct of transport virtqueue for
> > > > > modern - it is different enough that it needs dedicated commands.
> > > >
> > > > If you mean the transport virtqueue, I think some dedicated
> > > > commands for legacy are needed. Then it would be a transport that
> > > > supports transitional devices. It would be much better than having
> > > > commands for a partial transport like this patch did.
> > >
> > > OK I am beginning to get what you are saying. So your criticism is
> > > this: what if device supports vq transport for modern, and we want
> > > to build a transitional device on top. how will that look. yes?
> > > A reasonable thing to include at least in the commit log. Parav?
> > >
> > I am still trying to understand what is "vq transport for modern"?
> > Do you mean transporting currently defined config space access over vq?
> > If so, is this VQ belong to the guest or hypervisor?
>
> https://lore.kernel.org/all/20220826100034.200432-2-
> lingshan.zhu%40intel.com/t.mbox.gz
The gz link is not accessible.
But I got the right link [1].
[1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
1. Above patch cover letter [1] is missing the basic objective/problem statement.
i.e. why a transport virtqueue is needed?
But I probably get the idea of [1] as we did the AQ.
2. Commit log says about
a. querying resource of management device (aka group owner in AQ now)
b. creating and destroying the managed device (aka group owner creating group member devices)
c. configure the managed device (aka group owner configuring/composing group member devices such as VFs, SFs, SIOV).
So, all above 2.a to 2.c belongs to the admin group owner and group management commands like how it is defined in the AQ proposal.
So, 3 out of the 4 motivations are achieved by AQ proposal.
This AQ belongs to the hypervisor. I am clear on this part.
4th point in cover letter is: "config virtqueues of the managed device".
This work belongs to the driver -> device direct communication using a queue from driver to device.
So, I imagine this work can be done using a queue by the guest driver and serviced by the device like how a guest driver configures the queue today without any mediation.
For PCI, MMIO transport, surely this can be done by the PCI device directly being is PF, VF or SIOV.
(Instead of config register, using a new queue interface). Looks fine to me.
Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
Should it be always mediated when it is of VF, SIOV Device? Mostly no because it is yet another VQ for PF, VF, SIOV.
I am yet to parse rest of the 4 patches, please give me some time to review it.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 17:33 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 17:33 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 10, 2023 12:16 PM
>
> On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> >
> >
> > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > I thought so too originally. Unfortunately I now think that no,
> > > > > legacy is not going to be a byproduct of transport virtqueue for
> > > > > modern - it is different enough that it needs dedicated commands.
> > > >
> > > > If you mean the transport virtqueue, I think some dedicated
> > > > commands for legacy are needed. Then it would be a transport that
> > > > supports transitional devices. It would be much better than having
> > > > commands for a partial transport like this patch did.
> > >
> > > OK I am beginning to get what you are saying. So your criticism is
> > > this: what if device supports vq transport for modern, and we want
> > > to build a transitional device on top. how will that look. yes?
> > > A reasonable thing to include at least in the commit log. Parav?
> > >
> > I am still trying to understand what is "vq transport for modern"?
> > Do you mean transporting currently defined config space access over vq?
> > If so, is this VQ belong to the guest or hypervisor?
>
> https://lore.kernel.org/all/20220826100034.200432-2-
> lingshan.zhu%40intel.com/t.mbox.gz
The gz link is not accessible.
But I got the right link [1].
[1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
1. Above patch cover letter [1] is missing the basic objective/problem statement.
i.e. why a transport virtqueue is needed?
But I probably get the idea of [1] as we did the AQ.
2. Commit log says about
a. querying resource of management device (aka group owner in AQ now)
b. creating and destroying the managed device (aka group owner creating group member devices)
c. configure the managed device (aka group owner configuring/composing group member devices such as VFs, SFs, SIOV).
So, all above 2.a to 2.c belongs to the admin group owner and group management commands like how it is defined in the AQ proposal.
So, 3 out of the 4 motivations are achieved by AQ proposal.
This AQ belongs to the hypervisor. I am clear on this part.
4th point in cover letter is: "config virtqueues of the managed device".
This work belongs to the driver -> device direct communication using a queue from driver to device.
So, I imagine this work can be done using a queue by the guest driver and serviced by the device like how a guest driver configures the queue today without any mediation.
For PCI, MMIO transport, surely this can be done by the PCI device directly being is PF, VF or SIOV.
(Instead of config register, using a new queue interface). Looks fine to me.
Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
Should it be always mediated when it is of VF, SIOV Device? Mostly no because it is yet another VQ for PF, VF, SIOV.
I am yet to parse rest of the 4 patches, please give me some time to review it.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 17:33 ` [virtio-comment] " Parav Pandit
@ 2023-05-10 21:08 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 21:08 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Jason, Michel,
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Parav Pandit
> Sent: Wednesday, May 10, 2023 1:34 PM
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 10, 2023 12:16 PM
> >
> > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > >
> > >
> > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > virtqueue for modern - it is different enough that it needs dedicated
> commands.
> > > > >
> > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > commands for legacy are needed. Then it would be a transport
> > > > > that supports transitional devices. It would be much better than
> > > > > having commands for a partial transport like this patch did.
> > > >
> > > > OK I am beginning to get what you are saying. So your criticism
> > > > is
> > > > this: what if device supports vq transport for modern, and we want
> > > > to build a transitional device on top. how will that look. yes?
> > > > A reasonable thing to include at least in the commit log. Parav?
> > > >
> > > I am still trying to understand what is "vq transport for modern"?
> > > Do you mean transporting currently defined config space access over vq?
> > > If so, is this VQ belong to the guest or hypervisor?
> >
> > https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu%40intel.com/t.mbox.gz
>
> The gz link is not accessible.
> But I got the right link [1].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-
> lingshan.zhu@intel.com/
>
> 1. Above patch cover letter [1] is missing the basic objective/problem
> statement.
> i.e. why a transport virtqueue is needed?
> But I probably get the idea of [1] as we did the AQ.
>
> 2. Commit log says about
> a. querying resource of management device (aka group owner in AQ now) b.
> creating and destroying the managed device (aka group owner creating group
> member devices) c. configure the managed device (aka group owner
> configuring/composing group member devices such as VFs, SFs, SIOV).
>
> So, all above 2.a to 2.c belongs to the admin group owner and group
> management commands like how it is defined in the AQ proposal.
>
> So, 3 out of the 4 motivations are achieved by AQ proposal.
> This AQ belongs to the hypervisor. I am clear on this part.
>
> 4th point in cover letter is: "config virtqueues of the managed device".
>
> This work belongs to the driver -> device direct communication using a queue
> from driver to device.
> So, I imagine this work can be done using a queue by the guest driver and
> serviced by the device like how a guest driver configures the queue today
> without any mediation.
> For PCI, MMIO transport, surely this can be done by the PCI device directly
> being is PF, VF or SIOV.
> (Instead of config register, using a new queue interface). Looks fine to me.
>
> Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> it is yet another VQ for PF, VF, SIOV.
>
> I am yet to parse rest of the 4 patches, please give me some time to review it.
I went over the past work of [1], [2].
[1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
[2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
It also scales device register memory also very well in predictable way.
The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
Both legacy and non-legacy command have different semantics as Michael mentioned.
The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
With that I don't see legacy 3 commands anyway conflict with [1].
Some commands functionality is common between [1] and this proposal.
But that's how the legacy is. It is confined to legacy emulation.
So [1] can be done as follow_on to AQ and these series.
A small note about [2].
virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
So that VF and SF/SIOV can both utilize it.
SF/SIOV_R1 can use the creation and config part.
VFs will use only the device features + config space.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 21:08 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 21:08 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Jason, Michel,
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Parav Pandit
> Sent: Wednesday, May 10, 2023 1:34 PM
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 10, 2023 12:16 PM
> >
> > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > >
> > >
> > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > virtqueue for modern - it is different enough that it needs dedicated
> commands.
> > > > >
> > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > commands for legacy are needed. Then it would be a transport
> > > > > that supports transitional devices. It would be much better than
> > > > > having commands for a partial transport like this patch did.
> > > >
> > > > OK I am beginning to get what you are saying. So your criticism
> > > > is
> > > > this: what if device supports vq transport for modern, and we want
> > > > to build a transitional device on top. how will that look. yes?
> > > > A reasonable thing to include at least in the commit log. Parav?
> > > >
> > > I am still trying to understand what is "vq transport for modern"?
> > > Do you mean transporting currently defined config space access over vq?
> > > If so, is this VQ belong to the guest or hypervisor?
> >
> > https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu%40intel.com/t.mbox.gz
>
> The gz link is not accessible.
> But I got the right link [1].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-
> lingshan.zhu@intel.com/
>
> 1. Above patch cover letter [1] is missing the basic objective/problem
> statement.
> i.e. why a transport virtqueue is needed?
> But I probably get the idea of [1] as we did the AQ.
>
> 2. Commit log says about
> a. querying resource of management device (aka group owner in AQ now) b.
> creating and destroying the managed device (aka group owner creating group
> member devices) c. configure the managed device (aka group owner
> configuring/composing group member devices such as VFs, SFs, SIOV).
>
> So, all above 2.a to 2.c belongs to the admin group owner and group
> management commands like how it is defined in the AQ proposal.
>
> So, 3 out of the 4 motivations are achieved by AQ proposal.
> This AQ belongs to the hypervisor. I am clear on this part.
>
> 4th point in cover letter is: "config virtqueues of the managed device".
>
> This work belongs to the driver -> device direct communication using a queue
> from driver to device.
> So, I imagine this work can be done using a queue by the guest driver and
> serviced by the device like how a guest driver configures the queue today
> without any mediation.
> For PCI, MMIO transport, surely this can be done by the PCI device directly
> being is PF, VF or SIOV.
> (Instead of config register, using a new queue interface). Looks fine to me.
>
> Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> it is yet another VQ for PF, VF, SIOV.
>
> I am yet to parse rest of the 4 patches, please give me some time to review it.
I went over the past work of [1], [2].
[1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
[2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
It also scales device register memory also very well in predictable way.
The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
Both legacy and non-legacy command have different semantics as Michael mentioned.
The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
With that I don't see legacy 3 commands anyway conflict with [1].
Some commands functionality is common between [1] and this proposal.
But that's how the legacy is. It is confined to legacy emulation.
So [1] can be done as follow_on to AQ and these series.
A small note about [2].
virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
So that VF and SF/SIOV can both utilize it.
SF/SIOV_R1 can use the creation and config part.
VFs will use only the device features + config space.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 21:08 ` [virtio-comment] " Parav Pandit
@ 2023-05-10 21:33 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 21:33 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 10, 2023 at 09:08:44PM +0000, Parav Pandit wrote:
> Hi Jason, Michel,
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Parav Pandit
> > Sent: Wednesday, May 10, 2023 1:34 PM
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 10, 2023 12:16 PM
> > >
> > > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > > >
> > > >
> > > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > > virtqueue for modern - it is different enough that it needs dedicated
> > commands.
> > > > > >
> > > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > > commands for legacy are needed. Then it would be a transport
> > > > > > that supports transitional devices. It would be much better than
> > > > > > having commands for a partial transport like this patch did.
> > > > >
> > > > > OK I am beginning to get what you are saying. So your criticism
> > > > > is
> > > > > this: what if device supports vq transport for modern, and we want
> > > > > to build a transitional device on top. how will that look. yes?
> > > > > A reasonable thing to include at least in the commit log. Parav?
> > > > >
> > > > I am still trying to understand what is "vq transport for modern"?
> > > > Do you mean transporting currently defined config space access over vq?
> > > > If so, is this VQ belong to the guest or hypervisor?
> > >
> > > https://lore.kernel.org/all/20220826100034.200432-2-
> > > lingshan.zhu%40intel.com/t.mbox.gz
> >
> > The gz link is not accessible.
> > But I got the right link [1].
> >
> > [1] https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu@intel.com/
> >
> > 1. Above patch cover letter [1] is missing the basic objective/problem
> > statement.
> > i.e. why a transport virtqueue is needed?
> > But I probably get the idea of [1] as we did the AQ.
> >
> > 2. Commit log says about
> > a. querying resource of management device (aka group owner in AQ now) b.
> > creating and destroying the managed device (aka group owner creating group
> > member devices) c. configure the managed device (aka group owner
> > configuring/composing group member devices such as VFs, SFs, SIOV).
> >
> > So, all above 2.a to 2.c belongs to the admin group owner and group
> > management commands like how it is defined in the AQ proposal.
> >
> > So, 3 out of the 4 motivations are achieved by AQ proposal.
> > This AQ belongs to the hypervisor. I am clear on this part.
> >
> > 4th point in cover letter is: "config virtqueues of the managed device".
> >
> > This work belongs to the driver -> device direct communication using a queue
> > from driver to device.
> > So, I imagine this work can be done using a queue by the guest driver and
> > serviced by the device like how a guest driver configures the queue today
> > without any mediation.
> > For PCI, MMIO transport, surely this can be done by the PCI device directly
> > being is PF, VF or SIOV.
> > (Instead of config register, using a new queue interface). Looks fine to me.
> >
> > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> > it is yet another VQ for PF, VF, SIOV.
> >
> > I am yet to parse rest of the 4 patches, please give me some time to review it.
>
> I went over the past work of [1], [2].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
> [2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
>
> The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
> Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
>
> And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
>
> It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
>
> It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
> It also scales device register memory also very well in predictable way.
>
> The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
> Both legacy and non-legacy command have different semantics as Michael mentioned.
>
> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
>
> There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
> Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
>
> With that I don't see legacy 3 commands anyway conflict with [1].
> Some commands functionality is common between [1] and this proposal.
> But that's how the legacy is. It is confined to legacy emulation.
> So [1] can be done as follow_on to AQ and these series.
>
> A small note about [2].
> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> So that VF and SF/SIOV can both utilize it.
> SF/SIOV_R1 can use the creation and config part.
> VFs will use only the device features + config space.
I think some kind of "enable" command for VFs might have value. No?
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 21:33 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-10 21:33 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 10, 2023 at 09:08:44PM +0000, Parav Pandit wrote:
> Hi Jason, Michel,
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Parav Pandit
> > Sent: Wednesday, May 10, 2023 1:34 PM
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 10, 2023 12:16 PM
> > >
> > > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > > >
> > > >
> > > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > > virtqueue for modern - it is different enough that it needs dedicated
> > commands.
> > > > > >
> > > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > > commands for legacy are needed. Then it would be a transport
> > > > > > that supports transitional devices. It would be much better than
> > > > > > having commands for a partial transport like this patch did.
> > > > >
> > > > > OK I am beginning to get what you are saying. So your criticism
> > > > > is
> > > > > this: what if device supports vq transport for modern, and we want
> > > > > to build a transitional device on top. how will that look. yes?
> > > > > A reasonable thing to include at least in the commit log. Parav?
> > > > >
> > > > I am still trying to understand what is "vq transport for modern"?
> > > > Do you mean transporting currently defined config space access over vq?
> > > > If so, is this VQ belong to the guest or hypervisor?
> > >
> > > https://lore.kernel.org/all/20220826100034.200432-2-
> > > lingshan.zhu%40intel.com/t.mbox.gz
> >
> > The gz link is not accessible.
> > But I got the right link [1].
> >
> > [1] https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu@intel.com/
> >
> > 1. Above patch cover letter [1] is missing the basic objective/problem
> > statement.
> > i.e. why a transport virtqueue is needed?
> > But I probably get the idea of [1] as we did the AQ.
> >
> > 2. Commit log says about
> > a. querying resource of management device (aka group owner in AQ now) b.
> > creating and destroying the managed device (aka group owner creating group
> > member devices) c. configure the managed device (aka group owner
> > configuring/composing group member devices such as VFs, SFs, SIOV).
> >
> > So, all above 2.a to 2.c belongs to the admin group owner and group
> > management commands like how it is defined in the AQ proposal.
> >
> > So, 3 out of the 4 motivations are achieved by AQ proposal.
> > This AQ belongs to the hypervisor. I am clear on this part.
> >
> > 4th point in cover letter is: "config virtqueues of the managed device".
> >
> > This work belongs to the driver -> device direct communication using a queue
> > from driver to device.
> > So, I imagine this work can be done using a queue by the guest driver and
> > serviced by the device like how a guest driver configures the queue today
> > without any mediation.
> > For PCI, MMIO transport, surely this can be done by the PCI device directly
> > being is PF, VF or SIOV.
> > (Instead of config register, using a new queue interface). Looks fine to me.
> >
> > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> > it is yet another VQ for PF, VF, SIOV.
> >
> > I am yet to parse rest of the 4 patches, please give me some time to review it.
>
> I went over the past work of [1], [2].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
> [2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
>
> The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
> Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
>
> And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
>
> It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
>
> It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
> It also scales device register memory also very well in predictable way.
>
> The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
> Both legacy and non-legacy command have different semantics as Michael mentioned.
>
> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
>
> There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
> Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
>
> With that I don't see legacy 3 commands anyway conflict with [1].
> Some commands functionality is common between [1] and this proposal.
> But that's how the legacy is. It is confined to legacy emulation.
> So [1] can be done as follow_on to AQ and these series.
>
> A small note about [2].
> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> So that VF and SF/SIOV can both utilize it.
> SF/SIOV_R1 can use the creation and config part.
> VFs will use only the device features + config space.
I think some kind of "enable" command for VFs might have value. No?
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 21:33 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-10 21:48 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 21:48 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 10, 2023 5:33 PM
>
> I think some kind of "enable" command for VFs might have value. No?
Unlike the SIOV devices, VFs life cycle is under PF control, not sure how the enable command can help.
After SR-IOV is enabled, VF is short of enabled, it may not be ready, but that is guarded by the device reset flow.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-10 21:48 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-10 21:48 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 10, 2023 5:33 PM
>
> I think some kind of "enable" command for VFs might have value. No?
Unlike the SIOV devices, VFs life cycle is under PF control, not sure how the enable command can help.
After SR-IOV is enabled, VF is short of enabled, it may not be ready, but that is guarded by the device reset flow.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 7:43 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-11 7:04 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:04 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> > On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > > going to be a byproduct of transport virtqueue for modern -
> > > > > it is different enough that it needs dedicated commands.
> > > >
> > > > If you mean the transport virtqueue, I think some dedicated commands
> > > > for legacy are needed. Then it would be a transport that supports
> > > > transitional devices. It would be much better than having commands for
> > > > a partial transport like this patch did.
> > >
> > > OK I am beginning to get what you are saying. So your criticism is
> > > this: what if device supports vq transport for modern, and we want to
> > > build a transitional device on top. how will that look. yes?
> >
> > Yes. I think it needs to be done through the transport virtqueue
> > otherwise the transport is not self-contained.
>
> I mean, any feature can be done over transport vq.
>
> But there is value in adding legacy commands to an otherwise
> modern device without reworking it completely to
> switch to a different transport.
There's probably no need for a rework since legacy is not complicated.
More below.
>
>
> > > A reasonable thing to include at least in the commit log. Parav?
> > >
> > > You are also asking what if the device uses transport vq,
> > > and we want transitional on top of that.
> > > It's a fair question but I don't exactly get why would
> > > this legacy support feature be wanted for the vq transport
> > > and not for other transports.
> >
> > Not sure I get the question, but all the existing transport support
> > legacy, if we want to have another, should the legacy support be a
> > must or not?
>
> This specific proposal is for tunneling legacy over admin vq.
> It can live alongside a normal modern VF, with hypervisor
> combining these to create a transitional device.
Exactly, but what I meant here is
If we decide to use the admin vq, is there any good reason to tie it
to PCI if we don't want to tunneling PCI over adminq?
Why not simply invent individual commands to access legacy facilities
like commands to access like what transport virtqueue did for modern
device?:
1) device features
2) driver features
3) queue address
4) queue size
5) queue select
6) queue notify
7) device status
8) ISR status
9) config msix
10) queue msix
11) device configuration space
It focuses on the facilities instead of transport specific details
like registers (we don't even need legacy registers in this case), I
gives more deterministic behavior so we don't need to care about the
cross registers read/write.
>
> > >
> > >
> > >
> > >
> > > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > > modern does not even need multibyte read.
> > > >
> > > > I'm not sure I will get here,
> > >
> > > What does this mean?
> >
> > I meant I don't get what the issue if "modern does not even need
> > multibyte read".
>
> parse error again. reword?
I meant we need two sets of command for legacy and modern. We can
choose not to expose multibyte reads for modern commands.
>
> > >
> > > > since we can't expose admin vq to
> > > > guests, it means we need some software mediation. So if we just
> > > > implement what PCI allows us, then everything would be fine (even if
> > > > some method is not used).
> > > >
> > > > Thanks
> > >
> > > To repeat discussion on one of the previous versions, no it will not be
> > > fine because legacy virtio abuses pci in fundamentally broken ways.
> > > So yes you need a mediator on the host but even giving this
> > > mediator a chance to be robust on top of hardware
> > > means the hardware interface can not simply mirror legacy
> > > to hardware.
> > >
> > > For example, host mediator needs to trap writes into mac,
> > > buffer them and then send a 6 byte write.
> > > Now, pci actually does allow 6 byte writes, but legacy
> > > guests instead to 6 single byte writes for portability reasons.
> >
> > It's a known race condition, so PCI over adminq doesn't make it worse.
>
> it can however make it better - you can do a single 6 byte write command.
>
It would be tricky to detect when a legacy guest has finished writing
to the mac.
> > The mediator can just mirror what guests write over the admin
> > commands.
>
> and this "just" just isn't good enough, or we end up with hacks
> in hardware.
Yes but this "issue" exists in this proposal as well.
Thanks
>
> > Thanks
> >
> > > --
> > > MSr
> > >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 7:04 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:04 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> > On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > > going to be a byproduct of transport virtqueue for modern -
> > > > > it is different enough that it needs dedicated commands.
> > > >
> > > > If you mean the transport virtqueue, I think some dedicated commands
> > > > for legacy are needed. Then it would be a transport that supports
> > > > transitional devices. It would be much better than having commands for
> > > > a partial transport like this patch did.
> > >
> > > OK I am beginning to get what you are saying. So your criticism is
> > > this: what if device supports vq transport for modern, and we want to
> > > build a transitional device on top. how will that look. yes?
> >
> > Yes. I think it needs to be done through the transport virtqueue
> > otherwise the transport is not self-contained.
>
> I mean, any feature can be done over transport vq.
>
> But there is value in adding legacy commands to an otherwise
> modern device without reworking it completely to
> switch to a different transport.
There's probably no need for a rework since legacy is not complicated.
More below.
>
>
> > > A reasonable thing to include at least in the commit log. Parav?
> > >
> > > You are also asking what if the device uses transport vq,
> > > and we want transitional on top of that.
> > > It's a fair question but I don't exactly get why would
> > > this legacy support feature be wanted for the vq transport
> > > and not for other transports.
> >
> > Not sure I get the question, but all the existing transport support
> > legacy, if we want to have another, should the legacy support be a
> > must or not?
>
> This specific proposal is for tunneling legacy over admin vq.
> It can live alongside a normal modern VF, with hypervisor
> combining these to create a transitional device.
Exactly, but what I meant here is
If we decide to use the admin vq, is there any good reason to tie it
to PCI if we don't want to tunneling PCI over adminq?
Why not simply invent individual commands to access legacy facilities
like commands to access like what transport virtqueue did for modern
device?:
1) device features
2) driver features
3) queue address
4) queue size
5) queue select
6) queue notify
7) device status
8) ISR status
9) config msix
10) queue msix
11) device configuration space
It focuses on the facilities instead of transport specific details
like registers (we don't even need legacy registers in this case), I
gives more deterministic behavior so we don't need to care about the
cross registers read/write.
>
> > >
> > >
> > >
> > >
> > > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > > modern does not even need multibyte read.
> > > >
> > > > I'm not sure I will get here,
> > >
> > > What does this mean?
> >
> > I meant I don't get what the issue if "modern does not even need
> > multibyte read".
>
> parse error again. reword?
I meant we need two sets of command for legacy and modern. We can
choose not to expose multibyte reads for modern commands.
>
> > >
> > > > since we can't expose admin vq to
> > > > guests, it means we need some software mediation. So if we just
> > > > implement what PCI allows us, then everything would be fine (even if
> > > > some method is not used).
> > > >
> > > > Thanks
> > >
> > > To repeat discussion on one of the previous versions, no it will not be
> > > fine because legacy virtio abuses pci in fundamentally broken ways.
> > > So yes you need a mediator on the host but even giving this
> > > mediator a chance to be robust on top of hardware
> > > means the hardware interface can not simply mirror legacy
> > > to hardware.
> > >
> > > For example, host mediator needs to trap writes into mac,
> > > buffer them and then send a 6 byte write.
> > > Now, pci actually does allow 6 byte writes, but legacy
> > > guests instead to 6 single byte writes for portability reasons.
> >
> > It's a known race condition, so PCI over adminq doesn't make it worse.
>
> it can however make it better - you can do a single 6 byte write command.
>
It would be tricky to detect when a legacy guest has finished writing
to the mac.
> > The mediator can just mirror what guests write over the admin
> > commands.
>
> and this "just" just isn't good enough, or we end up with hacks
> in hardware.
Yes but this "issue" exists in this proposal as well.
Thanks
>
> > Thanks
> >
> > > --
> > > MSr
> > >
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 21:08 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 7:06 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:06 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 5:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
> Hi Jason, Michel,
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Parav Pandit
> > Sent: Wednesday, May 10, 2023 1:34 PM
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 10, 2023 12:16 PM
> > >
> > > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > > >
> > > >
> > > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > > virtqueue for modern - it is different enough that it needs dedicated
> > commands.
> > > > > >
> > > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > > commands for legacy are needed. Then it would be a transport
> > > > > > that supports transitional devices. It would be much better than
> > > > > > having commands for a partial transport like this patch did.
> > > > >
> > > > > OK I am beginning to get what you are saying. So your criticism
> > > > > is
> > > > > this: what if device supports vq transport for modern, and we want
> > > > > to build a transitional device on top. how will that look. yes?
> > > > > A reasonable thing to include at least in the commit log. Parav?
> > > > >
> > > > I am still trying to understand what is "vq transport for modern"?
> > > > Do you mean transporting currently defined config space access over vq?
> > > > If so, is this VQ belong to the guest or hypervisor?
> > >
> > > https://lore.kernel.org/all/20220826100034.200432-2-
> > > lingshan.zhu%40intel.com/t.mbox.gz
> >
> > The gz link is not accessible.
> > But I got the right link [1].
> >
> > [1] https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu@intel.com/
> >
> > 1. Above patch cover letter [1] is missing the basic objective/problem
> > statement.
> > i.e. why a transport virtqueue is needed?
> > But I probably get the idea of [1] as we did the AQ.
> >
> > 2. Commit log says about
> > a. querying resource of management device (aka group owner in AQ now) b.
> > creating and destroying the managed device (aka group owner creating group
> > member devices) c. configure the managed device (aka group owner
> > configuring/composing group member devices such as VFs, SFs, SIOV).
> >
> > So, all above 2.a to 2.c belongs to the admin group owner and group
> > management commands like how it is defined in the AQ proposal.
Yes, the plan is to rebase on top of AQ. Lingshan may add more details here.
> >
> > So, 3 out of the 4 motivations are achieved by AQ proposal.
> > This AQ belongs to the hypervisor. I am clear on this part.
> >
> > 4th point in cover letter is: "config virtqueues of the managed device".
> >
> > This work belongs to the driver -> device direct communication using a queue
> > from driver to device.
> > So, I imagine this work can be done using a queue by the guest driver and
> > serviced by the device like how a guest driver configures the queue today
> > without any mediation.
> > For PCI, MMIO transport, surely this can be done by the PCI device directly
> > being is PF, VF or SIOV.
> > (Instead of config register, using a new queue interface). Looks fine to me.
Great.
> >
> > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> > it is yet another VQ for PF, VF, SIOV.
Yes, the mediation is always conditional but not mandated when doing the design.
> >
> > I am yet to parse rest of the 4 patches, please give me some time to review it.
>
> I went over the past work of [1], [2].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
> [2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
>
> The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
> Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
I think this can be tweaked for static provisioning setups like
SR-IOV. E.g group owner can choose to not advertise the life cycle
management commands. Then the hypervisor may know the provisioning is
transport specific.
>
> And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
>
> It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
>
> It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
My understanding is that the cfgq/cmdq could be done on top, since the
commands are part of transport unless I miss something.
> It also scales device register memory also very well in predictable way.
>
> The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
> Both legacy and non-legacy command have different semantics as Michael mentioned.
>
> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
>
> There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
I think you meant IMS, actually, it's not hard to add new commands to
support a general IMS. We can start from a virtio specific one (as it
looks a common requirement not only for PCI/SIOV but also for
transport like MMIO)
> Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
Maybe we can say, one of the implementations is siov_r1, since it can
be used to implement devices in other transport. One of the main
motivations for transport virtqueue is to reduce the transport
specific resources to ease the virtio device implementation.
>
> With that I don't see legacy 3 commands anyway conflict with [1].
It doesn't, but it's not elegant since:
1) Modern transport, admin virtqueue with well defined transport commands
2) Legacy transport, works moe like a PCI transport on top of the
admin virtqueue
Transport virtqueue tend to be transport independent, but 2) tie it
somehow to the PCI. That's why I suggest to
1) introduce legacy transport commands
or
2) tweak your proposal to become a PCI over admin virtqueue
> Some commands functionality is common between [1] and this proposal.
> But that's how the legacy is. It is confined to legacy emulation.
> So [1] can be done as follow_on to AQ and these series.
>
> A small note about [2].
> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> So that VF and SF/SIOV can both utilize it.
> SF/SIOV_R1 can use the creation and config part.
> VFs will use only the device features + config space.
Good point. This could be fixed.
Thanks
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 7:06 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:06 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 5:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
> Hi Jason, Michel,
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Parav Pandit
> > Sent: Wednesday, May 10, 2023 1:34 PM
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 10, 2023 12:16 PM
> > >
> > > On Wed, May 10, 2023 at 12:11:50PM -0400, Parav Pandit wrote:
> > > >
> > > >
> > > > On 5/10/2023 2:04 AM, Michael S. Tsirkin wrote:
> > > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > > I thought so too originally. Unfortunately I now think that
> > > > > > > no, legacy is not going to be a byproduct of transport
> > > > > > > virtqueue for modern - it is different enough that it needs dedicated
> > commands.
> > > > > >
> > > > > > If you mean the transport virtqueue, I think some dedicated
> > > > > > commands for legacy are needed. Then it would be a transport
> > > > > > that supports transitional devices. It would be much better than
> > > > > > having commands for a partial transport like this patch did.
> > > > >
> > > > > OK I am beginning to get what you are saying. So your criticism
> > > > > is
> > > > > this: what if device supports vq transport for modern, and we want
> > > > > to build a transitional device on top. how will that look. yes?
> > > > > A reasonable thing to include at least in the commit log. Parav?
> > > > >
> > > > I am still trying to understand what is "vq transport for modern"?
> > > > Do you mean transporting currently defined config space access over vq?
> > > > If so, is this VQ belong to the guest or hypervisor?
> > >
> > > https://lore.kernel.org/all/20220826100034.200432-2-
> > > lingshan.zhu%40intel.com/t.mbox.gz
> >
> > The gz link is not accessible.
> > But I got the right link [1].
> >
> > [1] https://lore.kernel.org/all/20220826100034.200432-2-
> > lingshan.zhu@intel.com/
> >
> > 1. Above patch cover letter [1] is missing the basic objective/problem
> > statement.
> > i.e. why a transport virtqueue is needed?
> > But I probably get the idea of [1] as we did the AQ.
> >
> > 2. Commit log says about
> > a. querying resource of management device (aka group owner in AQ now) b.
> > creating and destroying the managed device (aka group owner creating group
> > member devices) c. configure the managed device (aka group owner
> > configuring/composing group member devices such as VFs, SFs, SIOV).
> >
> > So, all above 2.a to 2.c belongs to the admin group owner and group
> > management commands like how it is defined in the AQ proposal.
Yes, the plan is to rebase on top of AQ. Lingshan may add more details here.
> >
> > So, 3 out of the 4 motivations are achieved by AQ proposal.
> > This AQ belongs to the hypervisor. I am clear on this part.
> >
> > 4th point in cover letter is: "config virtqueues of the managed device".
> >
> > This work belongs to the driver -> device direct communication using a queue
> > from driver to device.
> > So, I imagine this work can be done using a queue by the guest driver and
> > serviced by the device like how a guest driver configures the queue today
> > without any mediation.
> > For PCI, MMIO transport, surely this can be done by the PCI device directly
> > being is PF, VF or SIOV.
> > (Instead of config register, using a new queue interface). Looks fine to me.
Great.
> >
> > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > Should it be always mediated when it is of VF, SIOV Device? Mostly no because
> > it is yet another VQ for PF, VF, SIOV.
Yes, the mediation is always conditional but not mandated when doing the design.
> >
> > I am yet to parse rest of the 4 patches, please give me some time to review it.
>
> I went over the past work of [1], [2].
>
> [1] https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel.com/
> [2] https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.html
>
> The "virtio q as transport" in [2] is bit misleading as its only role is to transport the _registers_ of the SIOV_R1 device through its parent PF.
> Rest of the work is the pure management work to manage the life cycle of SIOV devices (create/delete/configure/compose).
I think this can be tweaked for static provisioning setups like
SR-IOV. E.g group owner can choose to not advertise the life cycle
management commands. Then the hypervisor may know the provisioning is
transport specific.
>
> And the motivation is also clear is to provide composing a virtio device for the guest VM for the backward compatibility for 1.x part of the specification.
>
> It seems fine and indeed orthogonal to me that: it is for backward compatibility for already defined config fields for existing guest VM driver.
>
> It does not conflict with the cfgq/cmdq idea whose main purpose is for the new config fields, new use cases that doesn't require any mediation.
> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without mediation.
My understanding is that the cfgq/cmdq could be done on top, since the
commands are part of transport unless I miss something.
> It also scales device register memory also very well in predictable way.
>
> The registers and feature bits access described in [2], can certainly be done using new non_legacy new AQ commands.
> Both legacy and non-legacy command have different semantics as Michael mentioned.
>
> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits well for the use case described in [1].
>
> There are changes on going in MSI-X area and SIOV, so you might want to wait for it.
I think you meant IMS, actually, it's not hard to add new commands to
support a general IMS. We can start from a virtio specific one (as it
looks a common requirement not only for PCI/SIOV but also for
transport like MMIO)
> Or proposed command in [1] should be tagged as siov_r1, then things will be cleaner.
Maybe we can say, one of the implementations is siov_r1, since it can
be used to implement devices in other transport. One of the main
motivations for transport virtqueue is to reduce the transport
specific resources to ease the virtio device implementation.
>
> With that I don't see legacy 3 commands anyway conflict with [1].
It doesn't, but it's not elegant since:
1) Modern transport, admin virtqueue with well defined transport commands
2) Legacy transport, works moe like a PCI transport on top of the
admin virtqueue
Transport virtqueue tend to be transport independent, but 2) tie it
somehow to the PCI. That's why I suggest to
1) introduce legacy transport commands
or
2) tweak your proposal to become a PCI over admin virtqueue
> Some commands functionality is common between [1] and this proposal.
> But that's how the legacy is. It is confined to legacy emulation.
> So [1] can be done as follow_on to AQ and these series.
>
> A small note about [2].
> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> So that VF and SF/SIOV can both utilize it.
> SF/SIOV_R1 can use the creation and config part.
> VFs will use only the device features + config space.
Good point. This could be fixed.
Thanks
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 16:04 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 7:17 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:17 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 12:04 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/9/2023 11:51 PM, Jason Wang wrote:
> > On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
> >>
> >> SIOV does not require mediation.
> >> Composition is optional like VFs.
> >
> > This is not what I read from SIOV spec.
> >
> SIOV R1 spec was defined by _a_ vendor and it is undergoing upgrade to
> more practical use now. So we should let them finish the work.
>
That's fine.
> It is hard to comment in this forum about it.
Yes.
>
> >>
> >>>>
> >>>> Hence, the transport we built is to consider this in mind for the
> >>>> coming future.
> >>>
> >>> For transport virtqueue, it's not specific to PCI. It could be used in a much
> >>> broader use case.
> >>>
> >> May be.
> >> More below.
> >>
> >>>> So if each VF has its own configq, or cmdq, it totally make sense to
> >>>> me which is bootstrap interface to transport existing config space interface.
> >>>> The problem is: it is not backward compatible; Hence a device has no
> >>>> way of when to support both or only new configq.
> >>>
> >>> Providing compatibility in software is much more simpler than inventing new
> >>> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> >>> a SIOV device). And inventing a new hardware interface for compatibility might
> >>> not always work, it may break the advantages of the new hardware (like
> >>> scalability).
> >>>
> >> What I proposed below is new hardware interface for new features.
> >> Not for compatibility.
> >
> > I'm confused, the proposal is for legacy drives so it's for
> > compatibility for sure. No?
> This proposal is to get legacy devices to work over a PCI VF.
>
> a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
> devices to work in _non_ backward compatible way,
> because for new
> features there is no such thing as backward compatibility between guest
> driver and the device.
Ok, so what command did configq/cmdq have? I thought it was similar to
transport virtqueue, but it looks more like a control virtqueue here.
>
> >
> >> Compatibility is coming at a cost which is not scaling.
> >> If you attempt to scale, it breaks the future path forward to go without mediation.
> >>
> >>>>
> >>>> So eve growing these fields and optionally placement on configq
> >>>> doesn't really help and device builder to build it efficiently
> >>>> (without much predictability).
> >>>
> >>> Config queue is not the only choice, we have a lot of other choices (for example
> >>> PASID may help to reduce the on-chip resources).
> >>>
> >> PASID is for process isolation security construct, it is not for transportation.
> >
> > I meant with PASID you don't even need a BAR.
> >
> Not sure I fully understand, but my guess is, this is coming from
> mediation thought process.
Not actually. I meant, for example without PASID, for modern devices,
the common cfg structure must be placed on a BAR through MMIO.
But with the help of PASID, it could stay in the memory and the device
can fetch it via DMA. The help to reduce the resources for registers.
>
> >> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
> >
> > Unless you don't want to provide VF compatibility for SIOV.
> I do not understand what compatibility is this between which two elements?
> VF has its identify currently and SIOV will have its own identity in future.
Ok, so it really depends on how to implement the transport for SIOV. I
guess it won't use registers/BAR any more. If this is true, if we want
to run a virtio-pci driver, it needs some mediation.
It works like what you proposed here. Someday, SR-IOV or BAR transport
will become "legacy", but we need a way to make the "legacy virtio-pci
driver" work.
>
> >>>
> >>> Just to be sure we're on the same page. The proposal of both you and mine are
> >>> based on the adminq for PF not VF. The reason is obvious:
> >>> adminq per VF won't work without PASID, since it would have security issues.
> >>>
> >> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
> >>
> >> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> >> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> >> (And so additional configq follows the natural model for config space access).
> >
> > This is only true if:
> >
> > 1) you don't want to provide any backward compatibility for current
> > PCI transport,
> There is no concept of backward compatibility for new features.
> There will be new driver anyway in the guest.
>
> yes, I don't see a point in mediating 1.x config space for existing
> drivers as it is requires mediation.
>
> > this means you need to use new drivers in the guest
> > 2) you don't want to do live migration
> >
> Not sure how you assert this.
>
> Live migration at VF level for the passthrough device can be just fine
> without any mediation as long as all the parameters on src and dst side
> match.
> We already discussed in v0, so I prefer to avoid this again and focus on
> the patch in discussion.
I agree we are far from the topic of legacy support. But my point
stands still. We can discuss when live migration patches were posted
by somebody.
>
> > If you need one of the above, PASID is a must.
>
> A PCI VF passthrough device LM without mediation is achievable without
> PASID for all the existing defined features using config space.
>
> and also possible when new features are communicated over configq/cmdq.
>
> Transporting existing config space using some additional queue is _not_
> the objective, because building a scalable system where device has zero
> knowledge about needed backward compatibility is hard.
> I provided the example of 100 Vfs with 30 and (30+40) bytes config
> layout previously.
>
> Yet we are repeating and diverging from the discussion, that we should
> not intermix:
> 1. How to access legacy registers of the VF
With what you proposed, we don't even need legacy registers for VF. We
just need to keep its semantic via the admin virtqueue commands.
> vs
> 2. How to access existing or new configuration of the VF
>
> Because both have very different requirements.
>
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 7:17 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:17 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 12:04 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/9/2023 11:51 PM, Jason Wang wrote:
> > On Tue, May 9, 2023 at 11:56 AM Parav Pandit <parav@nvidia.com> wrote:
> >>
> >> SIOV does not require mediation.
> >> Composition is optional like VFs.
> >
> > This is not what I read from SIOV spec.
> >
> SIOV R1 spec was defined by _a_ vendor and it is undergoing upgrade to
> more practical use now. So we should let them finish the work.
>
That's fine.
> It is hard to comment in this forum about it.
Yes.
>
> >>
> >>>>
> >>>> Hence, the transport we built is to consider this in mind for the
> >>>> coming future.
> >>>
> >>> For transport virtqueue, it's not specific to PCI. It could be used in a much
> >>> broader use case.
> >>>
> >> May be.
> >> More below.
> >>
> >>>> So if each VF has its own configq, or cmdq, it totally make sense to
> >>>> me which is bootstrap interface to transport existing config space interface.
> >>>> The problem is: it is not backward compatible; Hence a device has no
> >>>> way of when to support both or only new configq.
> >>>
> >>> Providing compatibility in software is much more simpler than inventing new
> >>> hardware interfaces. Isn't it? (e.g if we want to provide compatibility for VF on
> >>> a SIOV device). And inventing a new hardware interface for compatibility might
> >>> not always work, it may break the advantages of the new hardware (like
> >>> scalability).
> >>>
> >> What I proposed below is new hardware interface for new features.
> >> Not for compatibility.
> >
> > I'm confused, the proposal is for legacy drives so it's for
> > compatibility for sure. No?
> This proposal is to get legacy devices to work over a PCI VF.
>
> a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
> devices to work in _non_ backward compatible way,
> because for new
> features there is no such thing as backward compatibility between guest
> driver and the device.
Ok, so what command did configq/cmdq have? I thought it was similar to
transport virtqueue, but it looks more like a control virtqueue here.
>
> >
> >> Compatibility is coming at a cost which is not scaling.
> >> If you attempt to scale, it breaks the future path forward to go without mediation.
> >>
> >>>>
> >>>> So eve growing these fields and optionally placement on configq
> >>>> doesn't really help and device builder to build it efficiently
> >>>> (without much predictability).
> >>>
> >>> Config queue is not the only choice, we have a lot of other choices (for example
> >>> PASID may help to reduce the on-chip resources).
> >>>
> >> PASID is for process isolation security construct, it is not for transportation.
> >
> > I meant with PASID you don't even need a BAR.
> >
> Not sure I fully understand, but my guess is, this is coming from
> mediation thought process.
Not actually. I meant, for example without PASID, for modern devices,
the common cfg structure must be placed on a BAR through MMIO.
But with the help of PASID, it could stay in the memory and the device
can fetch it via DMA. The help to reduce the resources for registers.
>
> >> Same, SIOV and VFs do not prefer mediation going forward in the use cases we come across.
> >
> > Unless you don't want to provide VF compatibility for SIOV.
> I do not understand what compatibility is this between which two elements?
> VF has its identify currently and SIOV will have its own identity in future.
Ok, so it really depends on how to implement the transport for SIOV. I
guess it won't use registers/BAR any more. If this is true, if we want
to run a virtio-pci driver, it needs some mediation.
It works like what you proposed here. Someday, SR-IOV or BAR transport
will become "legacy", but we need a way to make the "legacy virtio-pci
driver" work.
>
> >>>
> >>> Just to be sure we're on the same page. The proposal of both you and mine are
> >>> based on the adminq for PF not VF. The reason is obvious:
> >>> adminq per VF won't work without PASID, since it would have security issues.
> >>>
> >> The current proposal for legacy should be seen as separate and not to intermix with per VF based configuration queue.
> >>
> >> adminvq/config/control vq per VF and SIOV both devices will work without PASID (unrelated to legacy).
> >> Because they are like any other queue, such as txq, rxq, cvq. All of these queues are non-mediated today for PF, VF devices.
> >> (And so additional configq follows the natural model for config space access).
> >
> > This is only true if:
> >
> > 1) you don't want to provide any backward compatibility for current
> > PCI transport,
> There is no concept of backward compatibility for new features.
> There will be new driver anyway in the guest.
>
> yes, I don't see a point in mediating 1.x config space for existing
> drivers as it is requires mediation.
>
> > this means you need to use new drivers in the guest
> > 2) you don't want to do live migration
> >
> Not sure how you assert this.
>
> Live migration at VF level for the passthrough device can be just fine
> without any mediation as long as all the parameters on src and dst side
> match.
> We already discussed in v0, so I prefer to avoid this again and focus on
> the patch in discussion.
I agree we are far from the topic of legacy support. But my point
stands still. We can discuss when live migration patches were posted
by somebody.
>
> > If you need one of the above, PASID is a must.
>
> A PCI VF passthrough device LM without mediation is achievable without
> PASID for all the existing defined features using config space.
>
> and also possible when new features are communicated over configq/cmdq.
>
> Transporting existing config space using some additional queue is _not_
> the objective, because building a scalable system where device has zero
> knowledge about needed backward compatibility is hard.
> I provided the example of 100 Vfs with 30 and (30+40) bytes config
> layout previously.
>
> Yet we are repeating and diverging from the discussion, that we should
> not intermix:
> 1. How to access legacy registers of the VF
With what you proposed, we don't even need legacy registers for VF. We
just need to keep its semantic via the admin virtqueue commands.
> vs
> 2. How to access existing or new configuration of the VF
>
> Because both have very different requirements.
>
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-10 16:07 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 7:20 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:20 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/10/2023 12:22 AM, Jason Wang wrote:
> > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> >
> > Just to make sure we are at the same page.
> >
> > 1) if the hardware has configq and we need to make it work for current
> > virtio-pci driver, hypervisor needs to trap guest PCI access and
> > translate it to configq command. This means the hypervisor needs to
> > hide the configq from guests. In this case the configq needs a
> > dedicated DMA address which is what PASID can help.
>
> > 2) if the hardware can report the device states, unless we want to
> > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > required to isolate the DMA for guest traffic and device state.
>
> A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
> devices over PCIe or other transports.
> So no need to hide/mediate for hw based devices, like cvq and like data vqs.
As discussed, I may misunderstand the function of the configq. If it
works like cvq, then yes. We don't need PASID.
But if we want to report device state via let's say migration q, it
requires PASID since it is used by the hypervisor.
Thanks
>
> For vdpa kind of use case it can work like cvq mediation, which I
> believe is happening without PASID today.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 7:20 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-11 7:20 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> On 5/10/2023 12:22 AM, Jason Wang wrote:
> > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>
> >
> > Just to make sure we are at the same page.
> >
> > 1) if the hardware has configq and we need to make it work for current
> > virtio-pci driver, hypervisor needs to trap guest PCI access and
> > translate it to configq command. This means the hypervisor needs to
> > hide the configq from guests. In this case the configq needs a
> > dedicated DMA address which is what PASID can help.
>
> > 2) if the hardware can report the device states, unless we want to
> > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > required to isolate the DMA for guest traffic and device state.
>
> A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
> devices over PCIe or other transports.
> So no need to hide/mediate for hw based devices, like cvq and like data vqs.
As discussed, I may misunderstand the function of the configq. If it
works like cvq, then yes. We don't need PASID.
But if we want to report device state via let's say migration q, it
requires PASID since it is used by the hypervisor.
Thanks
>
> For vdpa kind of use case it can work like cvq mediation, which I
> believe is happening without PASID today.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:20 ` Jason Wang
@ 2023-05-11 11:35 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 11:35 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 03:20:52PM +0800, Jason Wang wrote:
> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > On 5/10/2023 12:22 AM, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > >
> > > Just to make sure we are at the same page.
> > >
> > > 1) if the hardware has configq and we need to make it work for current
> > > virtio-pci driver, hypervisor needs to trap guest PCI access and
> > > translate it to configq command. This means the hypervisor needs to
> > > hide the configq from guests. In this case the configq needs a
> > > dedicated DMA address which is what PASID can help.
> >
> > > 2) if the hardware can report the device states, unless we want to
> > > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > > required to isolate the DMA for guest traffic and device state.
> >
> > A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
> > devices over PCIe or other transports.
> > So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>
> As discussed, I may misunderstand the function of the configq. If it
> works like cvq, then yes. We don't need PASID.
>
> But if we want to report device state via let's say migration q, it
> requires PASID since it is used by the hypervisor.
>
> Thanks
I don't exactly get this part. With SRIOV migration q normally would be
part of the PF, accessing hypervisor memory.
> >
> > For vdpa kind of use case it can work like cvq mediation, which I
> > believe is happening without PASID today.
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 11:35 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 11:35 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 03:20:52PM +0800, Jason Wang wrote:
> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > On 5/10/2023 12:22 AM, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > >
> > > Just to make sure we are at the same page.
> > >
> > > 1) if the hardware has configq and we need to make it work for current
> > > virtio-pci driver, hypervisor needs to trap guest PCI access and
> > > translate it to configq command. This means the hypervisor needs to
> > > hide the configq from guests. In this case the configq needs a
> > > dedicated DMA address which is what PASID can help.
> >
> > > 2) if the hardware can report the device states, unless we want to
> > > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > > required to isolate the DMA for guest traffic and device state.
> >
> > A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
> > devices over PCIe or other transports.
> > So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>
> As discussed, I may misunderstand the function of the configq. If it
> works like cvq, then yes. We don't need PASID.
>
> But if we want to report device state via let's say migration q, it
> requires PASID since it is used by the hypervisor.
>
> Thanks
I don't exactly get this part. With SRIOV migration q normally would be
part of the PF, accessing hypervisor memory.
> >
> > For vdpa kind of use case it can work like cvq mediation, which I
> > believe is happening without PASID today.
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:04 ` [virtio-comment] " Jason Wang
@ 2023-05-11 12:54 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 12:54 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Thu, May 11, 2023 at 03:04:40PM +0800, Jason Wang wrote:
> On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > > > going to be a byproduct of transport virtqueue for modern -
> > > > > > it is different enough that it needs dedicated commands.
> > > > >
> > > > > If you mean the transport virtqueue, I think some dedicated commands
> > > > > for legacy are needed. Then it would be a transport that supports
> > > > > transitional devices. It would be much better than having commands for
> > > > > a partial transport like this patch did.
> > > >
> > > > OK I am beginning to get what you are saying. So your criticism is
> > > > this: what if device supports vq transport for modern, and we want to
> > > > build a transitional device on top. how will that look. yes?
> > >
> > > Yes. I think it needs to be done through the transport virtqueue
> > > otherwise the transport is not self-contained.
> >
> > I mean, any feature can be done over transport vq.
> >
> > But there is value in adding legacy commands to an otherwise
> > modern device without reworking it completely to
> > switch to a different transport.
>
> There's probably no need for a rework since legacy is not complicated.
> More below.
>
> >
> >
> > > > A reasonable thing to include at least in the commit log. Parav?
> > > >
> > > > You are also asking what if the device uses transport vq,
> > > > and we want transitional on top of that.
> > > > It's a fair question but I don't exactly get why would
> > > > this legacy support feature be wanted for the vq transport
> > > > and not for other transports.
> > >
> > > Not sure I get the question, but all the existing transport support
> > > legacy, if we want to have another, should the legacy support be a
> > > must or not?
> >
> > This specific proposal is for tunneling legacy over admin vq.
> > It can live alongside a normal modern VF, with hypervisor
> > combining these to create a transitional device.
>
> Exactly, but what I meant here is
>
> If we decide to use the admin vq, is there any good reason to tie it
> to PCI if we don't want to tunneling PCI over adminq?
>
> Why not simply invent individual commands to access legacy facilities
> like commands to access like what transport virtqueue did for modern
> device?:
>
> 1) device features
> 2) driver features
> 3) queue address
> 4) queue size
> 5) queue select
> 6) queue notify
> 7) device status
> 8) ISR status
> 9) config msix
> 10) queue msix
> 11) device configuration space
>
> It focuses on the facilities instead of transport specific details
> like registers (we don't even need legacy registers in this case), I
> gives more deterministic behavior so we don't need to care about the
> cross registers read/write.
This needs thought, it is definitely more work. Effort that could be
maybe spent on new features. What is the motivation
here? supporting legacy mmio guests?
> >
> > > >
> > > >
> > > >
> > > >
> > > > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > > > modern does not even need multibyte read.
> > > > >
> > > > > I'm not sure I will get here,
> > > >
> > > > What does this mean?
> > >
> > > I meant I don't get what the issue if "modern does not even need
> > > multibyte read".
> >
> > parse error again. reword?
>
> I meant we need two sets of command for legacy and modern. We can
> choose not to expose multibyte reads for modern commands.
>
> >
> > > >
> > > > > since we can't expose admin vq to
> > > > > guests, it means we need some software mediation. So if we just
> > > > > implement what PCI allows us, then everything would be fine (even if
> > > > > some method is not used).
> > > > >
> > > > > Thanks
> > > >
> > > > To repeat discussion on one of the previous versions, no it will not be
> > > > fine because legacy virtio abuses pci in fundamentally broken ways.
> > > > So yes you need a mediator on the host but even giving this
> > > > mediator a chance to be robust on top of hardware
> > > > means the hardware interface can not simply mirror legacy
> > > > to hardware.
> > > >
> > > > For example, host mediator needs to trap writes into mac,
> > > > buffer them and then send a 6 byte write.
> > > > Now, pci actually does allow 6 byte writes, but legacy
> > > > guests instead to 6 single byte writes for portability reasons.
> > >
> > > It's a known race condition, so PCI over adminq doesn't make it worse.
> >
> > it can however make it better - you can do a single 6 byte write command.
> >
>
> It would be tricky to detect when a legacy guest has finished writing
> to the mac.
>
> > > The mediator can just mirror what guests write over the admin
> > > commands.
> >
> > and this "just" just isn't good enough, or we end up with hacks
> > in hardware.
>
> Yes but this "issue" exists in this proposal as well.
>
> Thanks
>
> >
> > > Thanks
> > >
> > > > --
> > > > MSr
> > > >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 12:54 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 12:54 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
On Thu, May 11, 2023 at 03:04:40PM +0800, Jason Wang wrote:
> On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
> > > > > > I thought so too originally. Unfortunately I now think that no, legacy is not
> > > > > > going to be a byproduct of transport virtqueue for modern -
> > > > > > it is different enough that it needs dedicated commands.
> > > > >
> > > > > If you mean the transport virtqueue, I think some dedicated commands
> > > > > for legacy are needed. Then it would be a transport that supports
> > > > > transitional devices. It would be much better than having commands for
> > > > > a partial transport like this patch did.
> > > >
> > > > OK I am beginning to get what you are saying. So your criticism is
> > > > this: what if device supports vq transport for modern, and we want to
> > > > build a transitional device on top. how will that look. yes?
> > >
> > > Yes. I think it needs to be done through the transport virtqueue
> > > otherwise the transport is not self-contained.
> >
> > I mean, any feature can be done over transport vq.
> >
> > But there is value in adding legacy commands to an otherwise
> > modern device without reworking it completely to
> > switch to a different transport.
>
> There's probably no need for a rework since legacy is not complicated.
> More below.
>
> >
> >
> > > > A reasonable thing to include at least in the commit log. Parav?
> > > >
> > > > You are also asking what if the device uses transport vq,
> > > > and we want transitional on top of that.
> > > > It's a fair question but I don't exactly get why would
> > > > this legacy support feature be wanted for the vq transport
> > > > and not for other transports.
> > >
> > > Not sure I get the question, but all the existing transport support
> > > legacy, if we want to have another, should the legacy support be a
> > > must or not?
> >
> > This specific proposal is for tunneling legacy over admin vq.
> > It can live alongside a normal modern VF, with hypervisor
> > combining these to create a transitional device.
>
> Exactly, but what I meant here is
>
> If we decide to use the admin vq, is there any good reason to tie it
> to PCI if we don't want to tunneling PCI over adminq?
>
> Why not simply invent individual commands to access legacy facilities
> like commands to access like what transport virtqueue did for modern
> device?:
>
> 1) device features
> 2) driver features
> 3) queue address
> 4) queue size
> 5) queue select
> 6) queue notify
> 7) device status
> 8) ISR status
> 9) config msix
> 10) queue msix
> 11) device configuration space
>
> It focuses on the facilities instead of transport specific details
> like registers (we don't even need legacy registers in this case), I
> gives more deterministic behavior so we don't need to care about the
> cross registers read/write.
This needs thought, it is definitely more work. Effort that could be
maybe spent on new features. What is the motivation
here? supporting legacy mmio guests?
> >
> > > >
> > > >
> > > >
> > > >
> > > > > > Consider simplest case, multibyte fields. Legacy needs multibyte write,
> > > > > > modern does not even need multibyte read.
> > > > >
> > > > > I'm not sure I will get here,
> > > >
> > > > What does this mean?
> > >
> > > I meant I don't get what the issue if "modern does not even need
> > > multibyte read".
> >
> > parse error again. reword?
>
> I meant we need two sets of command for legacy and modern. We can
> choose not to expose multibyte reads for modern commands.
>
> >
> > > >
> > > > > since we can't expose admin vq to
> > > > > guests, it means we need some software mediation. So if we just
> > > > > implement what PCI allows us, then everything would be fine (even if
> > > > > some method is not used).
> > > > >
> > > > > Thanks
> > > >
> > > > To repeat discussion on one of the previous versions, no it will not be
> > > > fine because legacy virtio abuses pci in fundamentally broken ways.
> > > > So yes you need a mediator on the host but even giving this
> > > > mediator a chance to be robust on top of hardware
> > > > means the hardware interface can not simply mirror legacy
> > > > to hardware.
> > > >
> > > > For example, host mediator needs to trap writes into mac,
> > > > buffer them and then send a 6 byte write.
> > > > Now, pci actually does allow 6 byte writes, but legacy
> > > > guests instead to 6 single byte writes for portability reasons.
> > >
> > > It's a known race condition, so PCI over adminq doesn't make it worse.
> >
> > it can however make it better - you can do a single 6 byte write command.
> >
>
> It would be tricky to detect when a legacy guest has finished writing
> to the mac.
>
> > > The mediator can just mirror what guests write over the admin
> > > commands.
> >
> > and this "just" just isn't good enough, or we end up with hacks
> > in hardware.
>
> Yes but this "issue" exists in this proposal as well.
>
> Thanks
>
> >
> > > Thanks
> > >
> > > > --
> > > > MSr
> > > >
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 12:54 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-11 13:02 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:02 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 8:55 AM
> > 1) device features
> > 2) driver features
> > 3) queue address
> > 4) queue size
> > 5) queue select
> > 6) queue notify
> > 7) device status
> > 8) ISR status
> > 9) config msix
> > 10) queue msix
> > 11) device configuration space
> >
#9 may not even go to the group owner device.
What do we gain from bisecting it?
Every new additional needs involvement of the hypervisor as Michael noted below on "effort" point.
The register offset and read/write is far simpler interface for hypervisor.
Not to do cross register access is what we need to define in the spec for 1.x or future things.
> > It focuses on the facilities instead of transport specific details
> > like registers (we don't even need legacy registers in this case), I
> > gives more deterministic behavior so we don't need to care about the
> > cross registers read/write.
>
> This needs thought, it is definitely more work. Effort that could be maybe spent
> on new features. What is the motivation here? supporting legacy mmio guests?
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:02 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:02 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 8:55 AM
> > 1) device features
> > 2) driver features
> > 3) queue address
> > 4) queue size
> > 5) queue select
> > 6) queue notify
> > 7) device status
> > 8) ISR status
> > 9) config msix
> > 10) queue msix
> > 11) device configuration space
> >
#9 may not even go to the group owner device.
What do we gain from bisecting it?
Every new additional needs involvement of the hypervisor as Michael noted below on "effort" point.
The register offset and read/write is far simpler interface for hypervisor.
Not to do cross register access is what we need to define in the spec for 1.x or future things.
> > It focuses on the facilities instead of transport specific details
> > like registers (we don't even need legacy registers in this case), I
> > gives more deterministic behavior so we don't need to care about the
> > cross registers read/write.
>
> This needs thought, it is definitely more work. Effort that could be maybe spent
> on new features. What is the motivation here? supporting legacy mmio guests?
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:06 ` [virtio-comment] " Jason Wang
@ 2023-05-11 13:04 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:04 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 03:06:48PM +0800, Jason Wang wrote:
> > With that I don't see legacy 3 commands anyway conflict with [1].
>
> It doesn't, but it's not elegant since:
>
> 1) Modern transport, admin virtqueue with well defined transport commands
> 2) Legacy transport, works moe like a PCI transport on top of the
> admin virtqueue
>
> Transport virtqueue tend to be transport independent, but 2) tie it
> somehow to the PCI.
This "somehow" matters though. The connection is really just the
common config layout.
> That's why I suggest to
>
> 1) introduce legacy transport commands
>
> or
>
> 2) tweak your proposal to become a PCI over admin virtqueue
So you are saying it's really not generally "legacy over
admin queue" it is more like "legacy pci over admin queue"?
Good point.
And I feel it's a good point that this chooses legacy pci but it could
really be mmio or ccw or the upcoming vq transport just as well. I suspect
mmio or ccw won't work well for a physical pci card though.
transport vq might, but I am worried we'll have to spend extra
work clarifying it's a legacy interface. For example we will
have to explain that only 32 bits of features must be used.
Focusing more effort on transport vq is attractive though ...
It is worth examining this option in more depth.
Did you? Want to share how is transport vq closer than
legacy pci to what we might want?
> > Some commands functionality is common between [1] and this proposal.
> > But that's how the legacy is. It is confined to legacy emulation.
> > So [1] can be done as follow_on to AQ and these series.
> >
> > A small note about [2].
> > virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> > So that VF and SF/SIOV can both utilize it.
> > SF/SIOV_R1 can use the creation and config part.
> > VFs will use only the device features + config space.
>
> Good point. This could be fixed.
>
> Thanks
>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:04 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:04 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 03:06:48PM +0800, Jason Wang wrote:
> > With that I don't see legacy 3 commands anyway conflict with [1].
>
> It doesn't, but it's not elegant since:
>
> 1) Modern transport, admin virtqueue with well defined transport commands
> 2) Legacy transport, works moe like a PCI transport on top of the
> admin virtqueue
>
> Transport virtqueue tend to be transport independent, but 2) tie it
> somehow to the PCI.
This "somehow" matters though. The connection is really just the
common config layout.
> That's why I suggest to
>
> 1) introduce legacy transport commands
>
> or
>
> 2) tweak your proposal to become a PCI over admin virtqueue
So you are saying it's really not generally "legacy over
admin queue" it is more like "legacy pci over admin queue"?
Good point.
And I feel it's a good point that this chooses legacy pci but it could
really be mmio or ccw or the upcoming vq transport just as well. I suspect
mmio or ccw won't work well for a physical pci card though.
transport vq might, but I am worried we'll have to spend extra
work clarifying it's a legacy interface. For example we will
have to explain that only 32 bits of features must be used.
Focusing more effort on transport vq is attractive though ...
It is worth examining this option in more depth.
Did you? Want to share how is transport vq closer than
legacy pci to what we might want?
> > Some commands functionality is common between [1] and this proposal.
> > But that's how the legacy is. It is confined to legacy emulation.
> > So [1] can be done as follow_on to AQ and these series.
> >
> > A small note about [2].
> > virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
> > So that VF and SF/SIOV can both utilize it.
> > SF/SIOV_R1 can use the creation and config part.
> > VFs will use only the device features + config space.
>
> Good point. This could be fixed.
>
> Thanks
>
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:04 ` [virtio-comment] " Jason Wang
@ 2023-05-11 13:15 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:15 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:05 AM
> If we decide to use the admin vq, is there any good reason to tie it to PCI if we
> don't want to tunneling PCI over adminq?
To truly tunnel PCI over adminq, please tunnel _all_ pci things over AQ, not just below 1 to 11.
Doorbells, capabilities and more. Then it is qualifying as a transport.
Otherwise, it is not a "virtio pci transport over aq".
I neither see why one would tunnel whole PCI over some queue, which was not your intention either from the patches I see.
VQ over fabrics is much cleaner design to transport VQ over non-PCI.
People have tried to tunnel pci over some other transports and it only fine for non-production toys.
>
> Why not simply invent individual commands to access legacy facilities like
> commands to access like what transport virtqueue did for modern
> device?:
>
I don’t see how this is being any different than register-offset interface.
It bisects more things at hypervisor level that makes things hard to add #12th entry.
> 1) device features
> 2) driver features
> 3) queue address
> 4) queue size
> 5) queue select
> 6) queue notify
> 7) device status
> 8) ISR status
> 9) config msix
> 10) queue msix
> 11) device configuration space
>
> It focuses on the facilities instead of transport specific details like registers (we
> don't even need legacy registers in this case), I gives more deterministic
> behavior so we don't need to care about the cross registers read/write.
>
1.x has these registers at raw level and that seems fine.
Anything new will come on the cfgq and hence no bisection or registers needed.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:15 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:15 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:05 AM
> If we decide to use the admin vq, is there any good reason to tie it to PCI if we
> don't want to tunneling PCI over adminq?
To truly tunnel PCI over adminq, please tunnel _all_ pci things over AQ, not just below 1 to 11.
Doorbells, capabilities and more. Then it is qualifying as a transport.
Otherwise, it is not a "virtio pci transport over aq".
I neither see why one would tunnel whole PCI over some queue, which was not your intention either from the patches I see.
VQ over fabrics is much cleaner design to transport VQ over non-PCI.
People have tried to tunnel pci over some other transports and it only fine for non-production toys.
>
> Why not simply invent individual commands to access legacy facilities like
> commands to access like what transport virtqueue did for modern
> device?:
>
I don’t see how this is being any different than register-offset interface.
It bisects more things at hypervisor level that makes things hard to add #12th entry.
> 1) device features
> 2) driver features
> 3) queue address
> 4) queue size
> 5) queue select
> 6) queue notify
> 7) device status
> 8) ISR status
> 9) config msix
> 10) queue msix
> 11) device configuration space
>
> It focuses on the facilities instead of transport specific details like registers (we
> don't even need legacy registers in this case), I gives more deterministic
> behavior so we don't need to care about the cross registers read/write.
>
1.x has these registers at raw level and that seems fine.
Anything new will come on the cfgq and hence no bisection or registers needed.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:06 ` [virtio-comment] " Jason Wang
@ 2023-05-11 13:28 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:28 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:07 AM
> > > 4th point in cover letter is: "config virtqueues of the managed device".
> > >
> > > This work belongs to the driver -> device direct communication using
> > > a queue from driver to device.
> > > So, I imagine this work can be done using a queue by the guest
> > > driver and serviced by the device like how a guest driver configures
> > > the queue today without any mediation.
> > > For PCI, MMIO transport, surely this can be done by the PCI device
> > > directly being is PF, VF or SIOV.
> > > (Instead of config register, using a new queue interface). Looks fine to me.
>
> Great.
>
> > >
> > > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > > Should it be always mediated when it is of VF, SIOV Device? Mostly
> > > no because it is yet another VQ for PF, VF, SIOV.
>
> Yes, the mediation is always conditional but not mandated when doing the
> design.
>
> > >
> > > I am yet to parse rest of the 4 patches, please give me some time to review
> it.
> >
> > I went over the past work of [1], [2].
> >
> > [1]
> > https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel
> > .com/ [2]
> > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.h
> > tml
> >
> > The "virtio q as transport" in [2] is bit misleading as its only role is to
> transport the _registers_ of the SIOV_R1 device through its parent PF.
> > Rest of the work is the pure management work to manage the life cycle of
> SIOV devices (create/delete/configure/compose).
>
> I think this can be tweaked for static provisioning setups like SR-IOV. E.g group
> owner can choose to not advertise the life cycle management commands. Then
> the hypervisor may know the provisioning is transport specific.
>
It is just bunch of admin commands in the new SIOV or virtual virtio device chapter.
It is not a transport chapter.
> >
> > And the motivation is also clear is to provide composing a virtio device for the
> guest VM for the backward compatibility for 1.x part of the specification.
> >
> > It seems fine and indeed orthogonal to me that: it is for backward
> compatibility for already defined config fields for existing guest VM driver.
> >
> > It does not conflict with the cfgq/cmdq idea whose main purpose is for the
> new config fields, new use cases that doesn't require any mediation.
> > Such cfgq works across PF, VF, SF/SIOV devices in uniform way without
> mediation.
>
> My understanding is that the cfgq/cmdq could be done on top, since the
> commands are part of transport unless I miss something.
>
On top of what?
cfgq/cmdq is just another queue like any other VQ.
it is orthogonal to accessing 1.x registers using group owner's queue.
> > It also scales device register memory also very well in predictable way.
> >
> > The registers and feature bits access described in [2], can certainly be done
> using new non_legacy new AQ commands.
> > Both legacy and non-legacy command have different semantics as Michael
> mentioned.
> >
> > The AQ semantics that Michael did as opposed to "virtqueue as transport" fits
> well for the use case described in [1].
> >
> > There are changes on going in MSI-X area and SIOV, so you might want to
> wait for it.
>
> I think you meant IMS, actually, it's not hard to add new commands to support
> a general IMS. We can start from a virtio specific one (as it looks a common
> requirement not only for PCI/SIOV but also for transport like MMIO)
>
IMS configuration usually supposed to go directly to the device. Long discussion with Thomas on IMS topic.
I will avoid diverting to unrelated discussion for now.
> > Or proposed command in [1] should be tagged as siov_r1, then things will be
> cleaner.
>
> Maybe we can say, one of the implementations is siov_r1, since it can be used
> to implement devices in other transport. One of the main motivations for
> transport virtqueue is to reduce the transport specific resources to ease the
> virtio device implementation.
>
Yes, the main motivation as for backward compatibility for the currently defined features.
> >
> > With that I don't see legacy 3 commands anyway conflict with [1].
>
> It doesn't, but it's not elegant since:
>
I don’t see how making 3 commands to 9 commands makes it elegant by doing bisecting of registers offset in hypervisor.
Its same thing using more opcodes.
> 1) Modern transport, admin virtqueue with well defined transport commands
> 2) Legacy transport, works moe like a PCI transport on top of the admin
> virtqueue
>
> Transport virtqueue tend to be transport independent, but 2) tie it somehow to
> the PCI. That's why I suggest to
>
The 4 patches are not transporting all the PCI things over transport VQ. it is not a transport.
It is mediation done for SIOV like how it's done for 2 patches.
> 1) introduce legacy transport commands
>
> or
>
> 2) tweak your proposal to become a PCI over admin virtqueue
>
This is not the objective to transport all PCI virtio fields over AQ.
Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:28 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 13:28 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:07 AM
> > > 4th point in cover letter is: "config virtqueues of the managed device".
> > >
> > > This work belongs to the driver -> device direct communication using
> > > a queue from driver to device.
> > > So, I imagine this work can be done using a queue by the guest
> > > driver and serviced by the device like how a guest driver configures
> > > the queue today without any mediation.
> > > For PCI, MMIO transport, surely this can be done by the PCI device
> > > directly being is PF, VF or SIOV.
> > > (Instead of config register, using a new queue interface). Looks fine to me.
>
> Great.
>
> > >
> > > Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
> > > Should it be always mediated when it is of VF, SIOV Device? Mostly
> > > no because it is yet another VQ for PF, VF, SIOV.
>
> Yes, the mediation is always conditional but not mandated when doing the
> design.
>
> > >
> > > I am yet to parse rest of the 4 patches, please give me some time to review
> it.
> >
> > I went over the past work of [1], [2].
> >
> > [1]
> > https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel
> > .com/ [2]
> > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.h
> > tml
> >
> > The "virtio q as transport" in [2] is bit misleading as its only role is to
> transport the _registers_ of the SIOV_R1 device through its parent PF.
> > Rest of the work is the pure management work to manage the life cycle of
> SIOV devices (create/delete/configure/compose).
>
> I think this can be tweaked for static provisioning setups like SR-IOV. E.g group
> owner can choose to not advertise the life cycle management commands. Then
> the hypervisor may know the provisioning is transport specific.
>
It is just bunch of admin commands in the new SIOV or virtual virtio device chapter.
It is not a transport chapter.
> >
> > And the motivation is also clear is to provide composing a virtio device for the
> guest VM for the backward compatibility for 1.x part of the specification.
> >
> > It seems fine and indeed orthogonal to me that: it is for backward
> compatibility for already defined config fields for existing guest VM driver.
> >
> > It does not conflict with the cfgq/cmdq idea whose main purpose is for the
> new config fields, new use cases that doesn't require any mediation.
> > Such cfgq works across PF, VF, SF/SIOV devices in uniform way without
> mediation.
>
> My understanding is that the cfgq/cmdq could be done on top, since the
> commands are part of transport unless I miss something.
>
On top of what?
cfgq/cmdq is just another queue like any other VQ.
it is orthogonal to accessing 1.x registers using group owner's queue.
> > It also scales device register memory also very well in predictable way.
> >
> > The registers and feature bits access described in [2], can certainly be done
> using new non_legacy new AQ commands.
> > Both legacy and non-legacy command have different semantics as Michael
> mentioned.
> >
> > The AQ semantics that Michael did as opposed to "virtqueue as transport" fits
> well for the use case described in [1].
> >
> > There are changes on going in MSI-X area and SIOV, so you might want to
> wait for it.
>
> I think you meant IMS, actually, it's not hard to add new commands to support
> a general IMS. We can start from a virtio specific one (as it looks a common
> requirement not only for PCI/SIOV but also for transport like MMIO)
>
IMS configuration usually supposed to go directly to the device. Long discussion with Thomas on IMS topic.
I will avoid diverting to unrelated discussion for now.
> > Or proposed command in [1] should be tagged as siov_r1, then things will be
> cleaner.
>
> Maybe we can say, one of the implementations is siov_r1, since it can be used
> to implement devices in other transport. One of the main motivations for
> transport virtqueue is to reduce the transport specific resources to ease the
> virtio device implementation.
>
Yes, the main motivation as for backward compatibility for the currently defined features.
> >
> > With that I don't see legacy 3 commands anyway conflict with [1].
>
> It doesn't, but it's not elegant since:
>
I don’t see how making 3 commands to 9 commands makes it elegant by doing bisecting of registers offset in hypervisor.
Its same thing using more opcodes.
> 1) Modern transport, admin virtqueue with well defined transport commands
> 2) Legacy transport, works moe like a PCI transport on top of the admin
> virtqueue
>
> Transport virtqueue tend to be transport independent, but 2) tie it somehow to
> the PCI. That's why I suggest to
>
The 4 patches are not transporting all the PCI things over transport VQ. it is not a transport.
It is mediation done for SIOV like how it's done for 2 patches.
> 1) introduce legacy transport commands
>
> or
>
> 2) tweak your proposal to become a PCI over admin virtqueue
>
This is not the objective to transport all PCI virtio fields over AQ.
Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:28 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 13:38 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:38 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> This is not the objective to transport all PCI virtio fields over AQ.
> Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
I mean, it's a transport in a sense of where it will reside in e.g.
linux virtio stack - below virtio core.
We don't save much code there by reusing the same register
layout of legacy pci.
But let's at least make this feature complete, and then we'll
see whether it is cleaner to reuse legacy layout or build
a new one.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:38 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:38 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> This is not the objective to transport all PCI virtio fields over AQ.
> Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
I mean, it's a transport in a sense of where it will reside in e.g.
linux virtio stack - below virtio core.
We don't save much code there by reusing the same register
layout of legacy pci.
But let's at least make this feature complete, and then we'll
see whether it is cleaner to reuse legacy layout or build
a new one.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:15 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 13:45 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:45 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 01:15:15PM +0000, Parav Pandit wrote:
> > Why not simply invent individual commands to access legacy facilities like
> > commands to access like what transport virtqueue did for modern
> > device?:
> >
> I don’t see how this is being any different than register-offset interface.
For example legacy device config's offset moves depending
on MSI enable. Which is a missing piece in the current
proposal btw as I keep saying.
Setting a flag or using a separate command for device config
does indeed sound cleaner.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 13:45 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 13:45 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 01:15:15PM +0000, Parav Pandit wrote:
> > Why not simply invent individual commands to access legacy facilities like
> > commands to access like what transport virtqueue did for modern
> > device?:
> >
> I don’t see how this is being any different than register-offset interface.
For example legacy device config's offset moves depending
on MSI enable. Which is a missing piece in the current
proposal btw as I keep saying.
Setting a flag or using a separate command for device config
does indeed sound cleaner.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:17 ` [virtio-comment] " Jason Wang
@ 2023-05-11 14:31 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 14:31 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:17 AM
> > a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
> > devices to work in _non_ backward compatible way, because for new
> > features there is no such thing as backward compatibility between
> > guest driver and the device.
>
> Ok, so what command did configq/cmdq have?
> I thought it was similar to transport virtqueue, but it looks more like a control virtqueue here.
Yes, it is more like a cvq to do device attributes discovery and configuration.
Just that its generic enough.
For example
creating VQ that spans multiple physical addresses.
Discovery and enabling new device attributes which are rarely accessed or accessed during device init/cleanup time.
The fundamental difference between cfgvq to tvq is:
Cfgvq is between the member virtio device and its driver.
Tvq=aq is on group owner of the virtio device.
>
> >
> > >
> > >> Compatibility is coming at a cost which is not scaling.
> > >> If you attempt to scale, it breaks the future path forward to go without
> mediation.
> > >>
> > >>>>
> > >>>> So eve growing these fields and optionally placement on configq
> > >>>> doesn't really help and device builder to build it efficiently
> > >>>> (without much predictability).
> > >>>
> > >>> Config queue is not the only choice, we have a lot of other
> > >>> choices (for example PASID may help to reduce the on-chip resources).
> > >>>
> > >> PASID is for process isolation security construct, it is not for transportation.
> > >
> > > I meant with PASID you don't even need a BAR.
> > >
> > Not sure I fully understand, but my guess is, this is coming from
> > mediation thought process.
>
> Not actually. I meant, for example without PASID, for modern devices, the
> common cfg structure must be placed on a BAR through MMIO.
>
> But with the help of PASID, it could stay in the memory and the device can fetch
> it via DMA. The help to reduce the resources for registers.
>
Yes, it can stay in memory and device can fetch via DMA using a VQ using its own q from guest driver.
Why does it need PASID?
> >
> > >> Same, SIOV and VFs do not prefer mediation going forward in the use
> cases we come across.
> > >
> > > Unless you don't want to provide VF compatibility for SIOV.
> > I do not understand what compatibility is this between which two elements?
> > VF has its identify currently and SIOV will have its own identity in future.
>
> Ok, so it really depends on how to implement the transport for SIOV. I guess it
> won't use registers/BAR any more. If this is true, if we want to run a virtio-pci
> driver, it needs some mediation.
>
> It works like what you proposed here. Someday, SR-IOV or BAR transport will
> become "legacy", but we need a way to make the "legacy virtio-pci driver"
> work.
> >
> > Live migration at VF level for the passthrough device can be just fine
> > without any mediation as long as all the parameters on src and dst
> > side match.
> > We already discussed in v0, so I prefer to avoid this again and focus
> > on the patch in discussion.
>
> I agree we are far from the topic of legacy support. But my point stands still.
> We can discuss when live migration patches were posted by somebody.
>
Ok.
> > Yet we are repeating and diverging from the discussion, that we should
> > not intermix:
> > 1. How to access legacy registers of the VF
>
> With what you proposed, we don't even need legacy registers for VF. We just
> need to keep its semantic via the admin virtqueue commands.
Yes, we don’t have IOBAR registers for the PCI VF. It is the 1.x VF so there are no legacy registers on it.
As proposed it is accessed are via aq commands of PF.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 14:31 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 14:31 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:17 AM
> > a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
> > devices to work in _non_ backward compatible way, because for new
> > features there is no such thing as backward compatibility between
> > guest driver and the device.
>
> Ok, so what command did configq/cmdq have?
> I thought it was similar to transport virtqueue, but it looks more like a control virtqueue here.
Yes, it is more like a cvq to do device attributes discovery and configuration.
Just that its generic enough.
For example
creating VQ that spans multiple physical addresses.
Discovery and enabling new device attributes which are rarely accessed or accessed during device init/cleanup time.
The fundamental difference between cfgvq to tvq is:
Cfgvq is between the member virtio device and its driver.
Tvq=aq is on group owner of the virtio device.
>
> >
> > >
> > >> Compatibility is coming at a cost which is not scaling.
> > >> If you attempt to scale, it breaks the future path forward to go without
> mediation.
> > >>
> > >>>>
> > >>>> So eve growing these fields and optionally placement on configq
> > >>>> doesn't really help and device builder to build it efficiently
> > >>>> (without much predictability).
> > >>>
> > >>> Config queue is not the only choice, we have a lot of other
> > >>> choices (for example PASID may help to reduce the on-chip resources).
> > >>>
> > >> PASID is for process isolation security construct, it is not for transportation.
> > >
> > > I meant with PASID you don't even need a BAR.
> > >
> > Not sure I fully understand, but my guess is, this is coming from
> > mediation thought process.
>
> Not actually. I meant, for example without PASID, for modern devices, the
> common cfg structure must be placed on a BAR through MMIO.
>
> But with the help of PASID, it could stay in the memory and the device can fetch
> it via DMA. The help to reduce the resources for registers.
>
Yes, it can stay in memory and device can fetch via DMA using a VQ using its own q from guest driver.
Why does it need PASID?
> >
> > >> Same, SIOV and VFs do not prefer mediation going forward in the use
> cases we come across.
> > >
> > > Unless you don't want to provide VF compatibility for SIOV.
> > I do not understand what compatibility is this between which two elements?
> > VF has its identify currently and SIOV will have its own identity in future.
>
> Ok, so it really depends on how to implement the transport for SIOV. I guess it
> won't use registers/BAR any more. If this is true, if we want to run a virtio-pci
> driver, it needs some mediation.
>
> It works like what you proposed here. Someday, SR-IOV or BAR transport will
> become "legacy", but we need a way to make the "legacy virtio-pci driver"
> work.
> >
> > Live migration at VF level for the passthrough device can be just fine
> > without any mediation as long as all the parameters on src and dst
> > side match.
> > We already discussed in v0, so I prefer to avoid this again and focus
> > on the patch in discussion.
>
> I agree we are far from the topic of legacy support. But my point stands still.
> We can discuss when live migration patches were posted by somebody.
>
Ok.
> > Yet we are repeating and diverging from the discussion, that we should
> > not intermix:
> > 1. How to access legacy registers of the VF
>
> With what you proposed, we don't even need legacy registers for VF. We just
> need to keep its semantic via the admin virtqueue commands.
Yes, we don’t have IOBAR registers for the PCI VF. It is the 1.x VF so there are no legacy registers on it.
As proposed it is accessed are via aq commands of PF.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:38 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-11 16:00 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 16:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 9:39 AM
>
> On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > This is not the objective to transport all PCI virtio fields over AQ.
> > Transport VQ commands proposal commands and commands in this proposal
> are just fine the way there without quoting it as "transport".
>
> I mean, it's a transport in a sense of where it will reside in e.g.
> linux virtio stack - below virtio core.
Not sure I understand, if you are asking for SF/SIOV virtio device, it should have its own communication path from driver to device without going through the PF AQ.
Because it has no backward compatibility baggage.
There is no need for PF based AQ for it.
PF based AQ is only needed for exposing a backward compatible guest VM facing PCI or MMIO virtio device whose backend is either PCI VF or SF/SIOV virtio passthrough device.
Such backward compatible composition requirement and above SIOV device are two very different requirements.
> We don't save much code there by reusing the same register layout of legacy
> pci.
> But let's at least make this feature complete, and then we'll see whether it is
> cleaner to reuse legacy layout or build a new one.
>
> --
> MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 16:00 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 16:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 9:39 AM
>
> On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > This is not the objective to transport all PCI virtio fields over AQ.
> > Transport VQ commands proposal commands and commands in this proposal
> are just fine the way there without quoting it as "transport".
>
> I mean, it's a transport in a sense of where it will reside in e.g.
> linux virtio stack - below virtio core.
Not sure I understand, if you are asking for SF/SIOV virtio device, it should have its own communication path from driver to device without going through the PF AQ.
Because it has no backward compatibility baggage.
There is no need for PF based AQ for it.
PF based AQ is only needed for exposing a backward compatible guest VM facing PCI or MMIO virtio device whose backend is either PCI VF or SF/SIOV virtio passthrough device.
Such backward compatible composition requirement and above SIOV device are two very different requirements.
> We don't save much code there by reusing the same register layout of legacy
> pci.
> But let's at least make this feature complete, and then we'll see whether it is
> cleaner to reuse legacy layout or build a new one.
>
> --
> MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:38 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-11 20:47 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 20:47 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 9:39 AM
>
> On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > This is not the objective to transport all PCI virtio fields over AQ.
> > Transport VQ commands proposal commands and commands in this proposal
> are just fine the way there without quoting it as "transport".
>
> I mean, it's a transport in a sense of where it will reside in e.g.
> linux virtio stack - below virtio core.
> We don't save much code there by reusing the same register layout of legacy
> pci.
> But let's at least make this feature complete, and then we'll see whether it is
> cleaner to reuse legacy layout or build a new one.
I don't follow your comment about "this feature complete".
When you say _this_, it means the feature proposed in this patch of supporting legacy over PCI VFs?
If so, anything missing in this proposal?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 20:47 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 20:47 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 9:39 AM
>
> On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > This is not the objective to transport all PCI virtio fields over AQ.
> > Transport VQ commands proposal commands and commands in this proposal
> are just fine the way there without quoting it as "transport".
>
> I mean, it's a transport in a sense of where it will reside in e.g.
> linux virtio stack - below virtio core.
> We don't save much code there by reusing the same register layout of legacy
> pci.
> But let's at least make this feature complete, and then we'll see whether it is
> cleaner to reuse legacy layout or build a new one.
I don't follow your comment about "this feature complete".
When you say _this_, it means the feature proposed in this patch of supporting legacy over PCI VFs?
If so, anything missing in this proposal?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 20:47 ` [virtio-comment] " Parav Pandit
@ 2023-05-11 20:58 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 20:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, May 11, 2023 9:39 AM
> >
> > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > This is not the objective to transport all PCI virtio fields over AQ.
> > > Transport VQ commands proposal commands and commands in this proposal
> > are just fine the way there without quoting it as "transport".
> >
> > I mean, it's a transport in a sense of where it will reside in e.g.
> > linux virtio stack - below virtio core.
> > We don't save much code there by reusing the same register layout of legacy
> > pci.
> > But let's at least make this feature complete, and then we'll see whether it is
> > cleaner to reuse legacy layout or build a new one.
>
> I don't follow your comment about "this feature complete".
> When you say _this_, it means the feature proposed in this patch of supporting legacy over PCI VFs?
>
> If so, anything missing in this proposal?
I think I commented at least on what I see as missing is first of all
a need to support INT#x emulation since level interrupts are I think
not supported for VFs. right?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 20:58 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-11 20:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, May 11, 2023 9:39 AM
> >
> > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > This is not the objective to transport all PCI virtio fields over AQ.
> > > Transport VQ commands proposal commands and commands in this proposal
> > are just fine the way there without quoting it as "transport".
> >
> > I mean, it's a transport in a sense of where it will reside in e.g.
> > linux virtio stack - below virtio core.
> > We don't save much code there by reusing the same register layout of legacy
> > pci.
> > But let's at least make this feature complete, and then we'll see whether it is
> > cleaner to reuse legacy layout or build a new one.
>
> I don't follow your comment about "this feature complete".
> When you say _this_, it means the feature proposed in this patch of supporting legacy over PCI VFs?
>
> If so, anything missing in this proposal?
I think I commented at least on what I see as missing is first of all
a need to support INT#x emulation since level interrupts are I think
not supported for VFs. right?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 20:58 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-11 21:03 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 21:03 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 4:58 PM
>
> On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, May 11, 2023 9:39 AM
> > >
> > > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > > This is not the objective to transport all PCI virtio fields over AQ.
> > > > Transport VQ commands proposal commands and commands in this
> > > > proposal
> > > are just fine the way there without quoting it as "transport".
> > >
> > > I mean, it's a transport in a sense of where it will reside in e.g.
> > > linux virtio stack - below virtio core.
> > > We don't save much code there by reusing the same register layout of
> > > legacy pci.
> > > But let's at least make this feature complete, and then we'll see
> > > whether it is cleaner to reuse legacy layout or build a new one.
> >
> > I don't follow your comment about "this feature complete".
> > When you say _this_, it means the feature proposed in this patch of
> supporting legacy over PCI VFs?
> >
> > If so, anything missing in this proposal?
>
> I think I commented at least on what I see as missing is first of all a need to
> support INT#x emulation since level interrupts are I think not supported for VFs.
> right?
I don't see it mandatory as I responded in [1].
I didn't see your response on [1].
[1] https://lore.kernel.org/virtio-comment/71d65eb3-c025-9287-0157-81e1d05574d1@nvidia.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-11 21:03 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-11 21:03 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 11, 2023 4:58 PM
>
> On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, May 11, 2023 9:39 AM
> > >
> > > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > > This is not the objective to transport all PCI virtio fields over AQ.
> > > > Transport VQ commands proposal commands and commands in this
> > > > proposal
> > > are just fine the way there without quoting it as "transport".
> > >
> > > I mean, it's a transport in a sense of where it will reside in e.g.
> > > linux virtio stack - below virtio core.
> > > We don't save much code there by reusing the same register layout of
> > > legacy pci.
> > > But let's at least make this feature complete, and then we'll see
> > > whether it is cleaner to reuse legacy layout or build a new one.
> >
> > I don't follow your comment about "this feature complete".
> > When you say _this_, it means the feature proposed in this patch of
> supporting legacy over PCI VFs?
> >
> > If so, anything missing in this proposal?
>
> I think I commented at least on what I see as missing is first of all a need to
> support INT#x emulation since level interrupts are I think not supported for VFs.
> right?
I don't see it mandatory as I responded in [1].
I didn't see your response on [1].
[1] https://lore.kernel.org/virtio-comment/71d65eb3-c025-9287-0157-81e1d05574d1@nvidia.com/
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:45 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-12 14:03 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-12 14:03 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
> Behalf Of Michael S. Tsirkin
> Sent: Thursday, May 11, 2023 9:45 AM
>
> On Thu, May 11, 2023 at 01:15:15PM +0000, Parav Pandit wrote:
> > > Why not simply invent individual commands to access legacy
> > > facilities like commands to access like what transport virtqueue did
> > > for modern
> > > device?:
> > >
> > I don’t see how this is being any different than register-offset interface.
>
> For example legacy device config's offset moves depending on MSI enable.
> Which is a missing piece in the current proposal btw as I keep saying.
How is this missing?
The offset is decided by the driver when msi enable.
Device also know that msi is enable/disabled.
So device responds based on what is done with msi, like how a device responds today on iobar.
I responded in this previously, don’t have the link, not sure if you got chance to read through.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-12 14:03 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-12 14:03 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
> Behalf Of Michael S. Tsirkin
> Sent: Thursday, May 11, 2023 9:45 AM
>
> On Thu, May 11, 2023 at 01:15:15PM +0000, Parav Pandit wrote:
> > > Why not simply invent individual commands to access legacy
> > > facilities like commands to access like what transport virtqueue did
> > > for modern
> > > device?:
> > >
> > I don’t see how this is being any different than register-offset interface.
>
> For example legacy device config's offset moves depending on MSI enable.
> Which is a missing piece in the current proposal btw as I keep saying.
How is this missing?
The offset is decided by the driver when msi enable.
Device also know that msi is enable/disabled.
So device responds based on what is done with msi, like how a device responds today on iobar.
I responded in this previously, don’t have the link, not sure if you got chance to read through.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 11:35 ` Michael S. Tsirkin
@ 2023-05-15 5:08 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:08 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 19:35, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:20:52PM +0800, Jason Wang wrote:
>> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
>>>
>>>
>>> On 5/10/2023 12:22 AM, Jason Wang wrote:
>>>> On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>>>> Just to make sure we are at the same page.
>>>>
>>>> 1) if the hardware has configq and we need to make it work for current
>>>> virtio-pci driver, hypervisor needs to trap guest PCI access and
>>>> translate it to configq command. This means the hypervisor needs to
>>>> hide the configq from guests. In this case the configq needs a
>>>> dedicated DMA address which is what PASID can help.
>>>> 2) if the hardware can report the device states, unless we want to
>>>> migrate L2 guest, hypervisor should hide it from L1, so PASID is
>>>> required to isolate the DMA for guest traffic and device state.
>>> A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
>>> devices over PCIe or other transports.
>>> So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>> As discussed, I may misunderstand the function of the configq. If it
>> works like cvq, then yes. We don't need PASID.
>>
>> But if we want to report device state via let's say migration q, it
>> requires PASID since it is used by the hypervisor.
>>
>> Thanks
> I don't exactly get this part. With SRIOV migration q normally would be
> part of the PF, accessing hypervisor memory.
Yes, I meant if we have a per VF adminq.
Thanks
>
>
>
>
>
>>> For vdpa kind of use case it can work like cvq mediation, which I
>>> believe is happening without PASID today.
>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 5:08 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:08 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 19:35, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:20:52PM +0800, Jason Wang wrote:
>> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
>>>
>>>
>>> On 5/10/2023 12:22 AM, Jason Wang wrote:
>>>> On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com> wrote:
>>>> Just to make sure we are at the same page.
>>>>
>>>> 1) if the hardware has configq and we need to make it work for current
>>>> virtio-pci driver, hypervisor needs to trap guest PCI access and
>>>> translate it to configq command. This means the hypervisor needs to
>>>> hide the configq from guests. In this case the configq needs a
>>>> dedicated DMA address which is what PASID can help.
>>>> 2) if the hardware can report the device states, unless we want to
>>>> migrate L2 guest, hypervisor should hide it from L1, so PASID is
>>>> required to isolate the DMA for guest traffic and device state.
>>> A configq negotiates + discovers new fields for the PCI PF, VF, SF/SIOV
>>> devices over PCIe or other transports.
>>> So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>> As discussed, I may misunderstand the function of the configq. If it
>> works like cvq, then yes. We don't need PASID.
>>
>> But if we want to report device state via let's say migration q, it
>> requires PASID since it is used by the hypervisor.
>>
>> Thanks
> I don't exactly get this part. With SRIOV migration q normally would be
> part of the PF, accessing hypervisor memory.
Yes, I meant if we have a per VF adminq.
Thanks
>
>
>
>
>
>>> For vdpa kind of use case it can work like cvq mediation, which I
>>> believe is happening without PASID today.
>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 14:31 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 5:12 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:12 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 22:31, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:17 AM
>>> a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
>>> devices to work in _non_ backward compatible way, because for new
>>> features there is no such thing as backward compatibility between
>>> guest driver and the device.
>> Ok, so what command did configq/cmdq have?
>> I thought it was similar to transport virtqueue, but it looks more like a control virtqueue here.
> Yes, it is more like a cvq to do device attributes discovery and configuration.
> Just that its generic enough.
> For example
> creating VQ that spans multiple physical addresses.
> Discovery and enabling new device attributes which are rarely accessed or accessed during device init/cleanup time.
>
> The fundamental difference between cfgvq to tvq is:
> Cfgvq is between the member virtio device and its driver.
> Tvq=aq is on group owner of the virtio device.
>
>>>>> Compatibility is coming at a cost which is not scaling.
>>>>> If you attempt to scale, it breaks the future path forward to go without
>> mediation.
>>>>>>> So eve growing these fields and optionally placement on configq
>>>>>>> doesn't really help and device builder to build it efficiently
>>>>>>> (without much predictability).
>>>>>> Config queue is not the only choice, we have a lot of other
>>>>>> choices (for example PASID may help to reduce the on-chip resources).
>>>>>>
>>>>> PASID is for process isolation security construct, it is not for transportation.
>>>> I meant with PASID you don't even need a BAR.
>>>>
>>> Not sure I fully understand, but my guess is, this is coming from
>>> mediation thought process.
>> Not actually. I meant, for example without PASID, for modern devices, the
>> common cfg structure must be placed on a BAR through MMIO.
>>
>> But with the help of PASID, it could stay in the memory and the device can fetch
>> it via DMA. The help to reduce the resources for registers.
>>
> Yes, it can stay in memory and device can fetch via DMA using a VQ using its own q from guest driver.
> Why does it need PASID?
Let me clarify my understanding.
If we want a per VF virtqueue that is used by the hypervisor. We need
PASID. Otherwise not.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 5:12 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:12 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 22:31, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:17 AM
>>> a configq/cmdq discussion is for new features for PF, VF, and SF/SIOV
>>> devices to work in _non_ backward compatible way, because for new
>>> features there is no such thing as backward compatibility between
>>> guest driver and the device.
>> Ok, so what command did configq/cmdq have?
>> I thought it was similar to transport virtqueue, but it looks more like a control virtqueue here.
> Yes, it is more like a cvq to do device attributes discovery and configuration.
> Just that its generic enough.
> For example
> creating VQ that spans multiple physical addresses.
> Discovery and enabling new device attributes which are rarely accessed or accessed during device init/cleanup time.
>
> The fundamental difference between cfgvq to tvq is:
> Cfgvq is between the member virtio device and its driver.
> Tvq=aq is on group owner of the virtio device.
>
>>>>> Compatibility is coming at a cost which is not scaling.
>>>>> If you attempt to scale, it breaks the future path forward to go without
>> mediation.
>>>>>>> So eve growing these fields and optionally placement on configq
>>>>>>> doesn't really help and device builder to build it efficiently
>>>>>>> (without much predictability).
>>>>>> Config queue is not the only choice, we have a lot of other
>>>>>> choices (for example PASID may help to reduce the on-chip resources).
>>>>>>
>>>>> PASID is for process isolation security construct, it is not for transportation.
>>>> I meant with PASID you don't even need a BAR.
>>>>
>>> Not sure I fully understand, but my guess is, this is coming from
>>> mediation thought process.
>> Not actually. I meant, for example without PASID, for modern devices, the
>> common cfg structure must be placed on a BAR through MMIO.
>>
>> But with the help of PASID, it could stay in the memory and the device can fetch
>> it via DMA. The help to reduce the resources for registers.
>>
> Yes, it can stay in memory and device can fetch via DMA using a VQ using its own q from guest driver.
> Why does it need PASID?
Let me clarify my understanding.
If we want a per VF virtqueue that is used by the hypervisor. We need
PASID. Otherwise not.
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:04 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 5:19 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:19 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:04, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:06:48PM +0800, Jason Wang wrote:
>>> With that I don't see legacy 3 commands anyway conflict with [1].
>> It doesn't, but it's not elegant since:
>>
>> 1) Modern transport, admin virtqueue with well defined transport commands
>> 2) Legacy transport, works moe like a PCI transport on top of the
>> admin virtqueue
>>
>> Transport virtqueue tend to be transport independent, but 2) tie it
>> somehow to the PCI.
> This "somehow" matters though. The connection is really just the
> common config layout.
>
>> That's why I suggest to
>>
>> 1) introduce legacy transport commands
>>
>> or
>>
>> 2) tweak your proposal to become a PCI over admin virtqueue
> So you are saying it's really not generally "legacy over
> admin queue" it is more like "legacy pci over admin queue"?
Yes.
> Good point.
>
> And I feel it's a good point that this chooses legacy pci but it could
> really be mmio or ccw or the upcoming vq transport just as well. I suspect
> mmio or ccw won't work well for a physical pci card though.
Right, but as discussed, a possible user is the SIOV transitional device
via transport virtqueues.
> transport vq might, but I am worried we'll have to spend extra
> work clarifying it's a legacy interface. For example we will
> have to explain that only 32 bits of features must be used.
> Focusing more effort on transport vq is attractive though ...
> It is worth examining this option in more depth.
> Did you?
My understanding is, for any case, we need such clarifications. What
Parav proposed here is not the access to the actual registers but stick
to the semantic of those registers. I think we must say high 32bits of
the features must be assumed to be 0.
> Want to share how is transport vq closer than
> legacy pci to what we might want?
I'm not sure I get the question, but in another thread I show 10+
commands to access the legacy facilities which is really very close to
the transport virtqueue.
Thanks
>
>
>>> Some commands functionality is common between [1] and this proposal.
>>> But that's how the legacy is. It is confined to legacy emulation.
>>> So [1] can be done as follow_on to AQ and these series.
>>>
>>> A small note about [2].
>>> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
>>> So that VF and SF/SIOV can both utilize it.
>>> SF/SIOV_R1 can use the creation and config part.
>>> VFs will use only the device features + config space.
>> Good point. This could be fixed.
>>
>> Thanks
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 5:19 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 5:19 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:04, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:06:48PM +0800, Jason Wang wrote:
>>> With that I don't see legacy 3 commands anyway conflict with [1].
>> It doesn't, but it's not elegant since:
>>
>> 1) Modern transport, admin virtqueue with well defined transport commands
>> 2) Legacy transport, works moe like a PCI transport on top of the
>> admin virtqueue
>>
>> Transport virtqueue tend to be transport independent, but 2) tie it
>> somehow to the PCI.
> This "somehow" matters though. The connection is really just the
> common config layout.
>
>> That's why I suggest to
>>
>> 1) introduce legacy transport commands
>>
>> or
>>
>> 2) tweak your proposal to become a PCI over admin virtqueue
> So you are saying it's really not generally "legacy over
> admin queue" it is more like "legacy pci over admin queue"?
Yes.
> Good point.
>
> And I feel it's a good point that this chooses legacy pci but it could
> really be mmio or ccw or the upcoming vq transport just as well. I suspect
> mmio or ccw won't work well for a physical pci card though.
Right, but as discussed, a possible user is the SIOV transitional device
via transport virtqueues.
> transport vq might, but I am worried we'll have to spend extra
> work clarifying it's a legacy interface. For example we will
> have to explain that only 32 bits of features must be used.
> Focusing more effort on transport vq is attractive though ...
> It is worth examining this option in more depth.
> Did you?
My understanding is, for any case, we need such clarifications. What
Parav proposed here is not the access to the actual registers but stick
to the semantic of those registers. I think we must say high 32bits of
the features must be assumed to be 0.
> Want to share how is transport vq closer than
> legacy pci to what we might want?
I'm not sure I get the question, but in another thread I show 10+
commands to access the legacy facilities which is really very close to
the transport virtqueue.
Thanks
>
>
>>> Some commands functionality is common between [1] and this proposal.
>>> But that's how the legacy is. It is confined to legacy emulation.
>>> So [1] can be done as follow_on to AQ and these series.
>>>
>>> A small note about [2].
>>> virtio_transportq_ctrl_dev_attribute should be detached from CREATE call and split to two commands.
>>> So that VF and SF/SIOV can both utilize it.
>>> SF/SIOV_R1 can use the creation and config part.
>>> VFs will use only the device features + config space.
>> Good point. This could be fixed.
>>
>> Thanks
>>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:28 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 7:10 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:10 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:28, Parav Pandit 写道:
>
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:07 AM
>>>> 4th point in cover letter is: "config virtqueues of the managed device".
>>>>
>>>> This work belongs to the driver -> device direct communication using
>>>> a queue from driver to device.
>>>> So, I imagine this work can be done using a queue by the guest
>>>> driver and serviced by the device like how a guest driver configures
>>>> the queue today without any mediation.
>>>> For PCI, MMIO transport, surely this can be done by the PCI device
>>>> directly being is PF, VF or SIOV.
>>>> (Instead of config register, using a new queue interface). Looks fine to me.
>> Great.
>>
>>>> Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
>>>> Should it be always mediated when it is of VF, SIOV Device? Mostly
>>>> no because it is yet another VQ for PF, VF, SIOV.
>> Yes, the mediation is always conditional but not mandated when doing the
>> design.
>>
>>>> I am yet to parse rest of the 4 patches, please give me some time to review
>> it.
>>> I went over the past work of [1], [2].
>>>
>>> [1]
>>> https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel
>>> .com/ [2]
>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.h
>>> tml
>>>
>>> The "virtio q as transport" in [2] is bit misleading as its only role is to
>> transport the _registers_ of the SIOV_R1 device through its parent PF.
>>> Rest of the work is the pure management work to manage the life cycle of
>> SIOV devices (create/delete/configure/compose).
>>
>> I think this can be tweaked for static provisioning setups like SR-IOV. E.g group
>> owner can choose to not advertise the life cycle management commands. Then
>> the hypervisor may know the provisioning is transport specific.
>>
> It is just bunch of admin commands in the new SIOV or virtual virtio device chapter.
> It is not a transport chapter.
Provisioning could be a part of the transport. For example SR-IOV allows
static and dynamic provisioning. If a virtio device needs to be
implemented in VF, it needs to be provisioned first.
>
>>> And the motivation is also clear is to provide composing a virtio device for the
>> guest VM for the backward compatibility for 1.x part of the specification.
>>> It seems fine and indeed orthogonal to me that: it is for backward
>> compatibility for already defined config fields for existing guest VM driver.
>>> It does not conflict with the cfgq/cmdq idea whose main purpose is for the
>> new config fields, new use cases that doesn't require any mediation.
>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without
>> mediation.
>>
>> My understanding is that the cfgq/cmdq could be done on top, since the
>> commands are part of transport unless I miss something.
>>
> On top of what?
> cfgq/cmdq is just another queue like any other VQ.
> it is orthogonal to accessing 1.x registers using group owner's queue.
I think there hasn't been any proposal that call any virtqueue like
cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
the context here is to provide or extend transport facilities via a
virtqueue not something like the control virtqueue.
>
>>> It also scales device register memory also very well in predictable way.
>>>
>>> The registers and feature bits access described in [2], can certainly be done
>> using new non_legacy new AQ commands.
>>> Both legacy and non-legacy command have different semantics as Michael
>> mentioned.
>>> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits
>> well for the use case described in [1].
>>> There are changes on going in MSI-X area and SIOV, so you might want to
>> wait for it.
>>
>> I think you meant IMS, actually, it's not hard to add new commands to support
>> a general IMS. We can start from a virtio specific one (as it looks a common
>> requirement not only for PCI/SIOV but also for transport like MMIO)
>>
> IMS configuration usually supposed to go directly to the device.
Admin virtqueue doesn't preventing anything from letting IMS go directly
to the device.
> Long discussion with Thomas on IMS topic.
> I will avoid diverting to unrelated discussion for now.
>
>>> Or proposed command in [1] should be tagged as siov_r1, then things will be
>> cleaner.
>>
>> Maybe we can say, one of the implementations is siov_r1, since it can be used
>> to implement devices in other transport. One of the main motivations for
>> transport virtqueue is to reduce the transport specific resources to ease the
>> virtio device implementation.
>>
> Yes, the main motivation as for backward compatibility for the currently defined features.
>
>>> With that I don't see legacy 3 commands anyway conflict with [1].
>> It doesn't, but it's not elegant since:
>>
> I don’t see how making 3 commands to 9 commands makes it elegant by doing bisecting of registers offset in hypervisor.
Or it needs to be done by the hardware and cross register write/read
needs to be handled there as well.
> Its same thing using more opcodes.
Do we really care about the number of opcodes? I think we've reserved
sufficient spaces.
>
>> 1) Modern transport, admin virtqueue with well defined transport commands
>> 2) Legacy transport, works moe like a PCI transport on top of the admin
>> virtqueue
>>
>> Transport virtqueue tend to be transport independent, but 2) tie it somehow to
>> the PCI. That's why I suggest to
>>
> The 4 patches are not transporting all the PCI things over transport VQ. it is not a transport.
It really depends on the different viewpoint. To me, adminq is not
specific to PCI, so this proposal looks more like a partial transport
for legacy.
So we have
1) all commands via PCI
2) all commands via adminq
3) part of the commands via PCI and the rest via adminq
This proposal goes to for 3 with transport specific commands while I
think we need to try 1 and 2 then it is more general and it helps to
avoid duplication with future features.
Thanks
> It is mediation done for SIOV like how it's done for 2 patches.
>
>> 1) introduce legacy transport commands
>>
>> or
>>
>> 2) tweak your proposal to become a PCI over admin virtqueue
>>
> This is not the objective to transport all PCI virtio fields over AQ.
> Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 7:10 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:10 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:28, Parav Pandit 写道:
>
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:07 AM
>>>> 4th point in cover letter is: "config virtqueues of the managed device".
>>>>
>>>> This work belongs to the driver -> device direct communication using
>>>> a queue from driver to device.
>>>> So, I imagine this work can be done using a queue by the guest
>>>> driver and serviced by the device like how a guest driver configures
>>>> the queue today without any mediation.
>>>> For PCI, MMIO transport, surely this can be done by the PCI device
>>>> directly being is PF, VF or SIOV.
>>>> (Instead of config register, using a new queue interface). Looks fine to me.
>> Great.
>>
>>>> Can this new cfg queue mediated like CVQ that is done in a sw? May be yes.
>>>> Should it be always mediated when it is of VF, SIOV Device? Mostly
>>>> no because it is yet another VQ for PF, VF, SIOV.
>> Yes, the mediation is always conditional but not mandated when doing the
>> design.
>>
>>>> I am yet to parse rest of the 4 patches, please give me some time to review
>> it.
>>> I went over the past work of [1], [2].
>>>
>>> [1]
>>> https://lore.kernel.org/all/20220826100034.200432-2-lingshan.zhu@intel
>>> .com/ [2]
>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00141.h
>>> tml
>>>
>>> The "virtio q as transport" in [2] is bit misleading as its only role is to
>> transport the _registers_ of the SIOV_R1 device through its parent PF.
>>> Rest of the work is the pure management work to manage the life cycle of
>> SIOV devices (create/delete/configure/compose).
>>
>> I think this can be tweaked for static provisioning setups like SR-IOV. E.g group
>> owner can choose to not advertise the life cycle management commands. Then
>> the hypervisor may know the provisioning is transport specific.
>>
> It is just bunch of admin commands in the new SIOV or virtual virtio device chapter.
> It is not a transport chapter.
Provisioning could be a part of the transport. For example SR-IOV allows
static and dynamic provisioning. If a virtio device needs to be
implemented in VF, it needs to be provisioned first.
>
>>> And the motivation is also clear is to provide composing a virtio device for the
>> guest VM for the backward compatibility for 1.x part of the specification.
>>> It seems fine and indeed orthogonal to me that: it is for backward
>> compatibility for already defined config fields for existing guest VM driver.
>>> It does not conflict with the cfgq/cmdq idea whose main purpose is for the
>> new config fields, new use cases that doesn't require any mediation.
>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way without
>> mediation.
>>
>> My understanding is that the cfgq/cmdq could be done on top, since the
>> commands are part of transport unless I miss something.
>>
> On top of what?
> cfgq/cmdq is just another queue like any other VQ.
> it is orthogonal to accessing 1.x registers using group owner's queue.
I think there hasn't been any proposal that call any virtqueue like
cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
the context here is to provide or extend transport facilities via a
virtqueue not something like the control virtqueue.
>
>>> It also scales device register memory also very well in predictable way.
>>>
>>> The registers and feature bits access described in [2], can certainly be done
>> using new non_legacy new AQ commands.
>>> Both legacy and non-legacy command have different semantics as Michael
>> mentioned.
>>> The AQ semantics that Michael did as opposed to "virtqueue as transport" fits
>> well for the use case described in [1].
>>> There are changes on going in MSI-X area and SIOV, so you might want to
>> wait for it.
>>
>> I think you meant IMS, actually, it's not hard to add new commands to support
>> a general IMS. We can start from a virtio specific one (as it looks a common
>> requirement not only for PCI/SIOV but also for transport like MMIO)
>>
> IMS configuration usually supposed to go directly to the device.
Admin virtqueue doesn't preventing anything from letting IMS go directly
to the device.
> Long discussion with Thomas on IMS topic.
> I will avoid diverting to unrelated discussion for now.
>
>>> Or proposed command in [1] should be tagged as siov_r1, then things will be
>> cleaner.
>>
>> Maybe we can say, one of the implementations is siov_r1, since it can be used
>> to implement devices in other transport. One of the main motivations for
>> transport virtqueue is to reduce the transport specific resources to ease the
>> virtio device implementation.
>>
> Yes, the main motivation as for backward compatibility for the currently defined features.
>
>>> With that I don't see legacy 3 commands anyway conflict with [1].
>> It doesn't, but it's not elegant since:
>>
> I don’t see how making 3 commands to 9 commands makes it elegant by doing bisecting of registers offset in hypervisor.
Or it needs to be done by the hardware and cross register write/read
needs to be handled there as well.
> Its same thing using more opcodes.
Do we really care about the number of opcodes? I think we've reserved
sufficient spaces.
>
>> 1) Modern transport, admin virtqueue with well defined transport commands
>> 2) Legacy transport, works moe like a PCI transport on top of the admin
>> virtqueue
>>
>> Transport virtqueue tend to be transport independent, but 2) tie it somehow to
>> the PCI. That's why I suggest to
>>
> The 4 patches are not transporting all the PCI things over transport VQ. it is not a transport.
It really depends on the different viewpoint. To me, adminq is not
specific to PCI, so this proposal looks more like a partial transport
for legacy.
So we have
1) all commands via PCI
2) all commands via adminq
3) part of the commands via PCI and the rest via adminq
This proposal goes to for 3 with transport specific commands while I
think we need to try 1 and 2 then it is more general and it helps to
avoid duplication with future features.
Thanks
> It is mediation done for SIOV like how it's done for 2 patches.
>
>> 1) introduce legacy transport commands
>>
>> or
>>
>> 2) tweak your proposal to become a PCI over admin virtqueue
>>
> This is not the objective to transport all PCI virtio fields over AQ.
> Transport VQ commands proposal commands and commands in this proposal are just fine the way there without quoting it as "transport".
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 12:54 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 7:13 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
在 2023/5/11 20:54, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:04:40PM +0800, Jason Wang wrote:
>> On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
>>>> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>>>>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>>>>>> going to be a byproduct of transport virtqueue for modern -
>>>>>>> it is different enough that it needs dedicated commands.
>>>>>> If you mean the transport virtqueue, I think some dedicated commands
>>>>>> for legacy are needed. Then it would be a transport that supports
>>>>>> transitional devices. It would be much better than having commands for
>>>>>> a partial transport like this patch did.
>>>>> OK I am beginning to get what you are saying. So your criticism is
>>>>> this: what if device supports vq transport for modern, and we want to
>>>>> build a transitional device on top. how will that look. yes?
>>>> Yes. I think it needs to be done through the transport virtqueue
>>>> otherwise the transport is not self-contained.
>>> I mean, any feature can be done over transport vq.
>>>
>>> But there is value in adding legacy commands to an otherwise
>>> modern device without reworking it completely to
>>> switch to a different transport.
>> There's probably no need for a rework since legacy is not complicated.
>> More below.
>>
>>>
>>>>> A reasonable thing to include at least in the commit log. Parav?
>>>>>
>>>>> You are also asking what if the device uses transport vq,
>>>>> and we want transitional on top of that.
>>>>> It's a fair question but I don't exactly get why would
>>>>> this legacy support feature be wanted for the vq transport
>>>>> and not for other transports.
>>>> Not sure I get the question, but all the existing transport support
>>>> legacy, if we want to have another, should the legacy support be a
>>>> must or not?
>>> This specific proposal is for tunneling legacy over admin vq.
>>> It can live alongside a normal modern VF, with hypervisor
>>> combining these to create a transitional device.
>> Exactly, but what I meant here is
>>
>> If we decide to use the admin vq, is there any good reason to tie it
>> to PCI if we don't want to tunneling PCI over adminq?
>>
>> Why not simply invent individual commands to access legacy facilities
>> like commands to access like what transport virtqueue did for modern
>> device?:
>>
>> 1) device features
>> 2) driver features
>> 3) queue address
>> 4) queue size
>> 5) queue select
>> 6) queue notify
>> 7) device status
>> 8) ISR status
>> 9) config msix
>> 10) queue msix
>> 11) device configuration space
>>
>> It focuses on the facilities instead of transport specific details
>> like registers (we don't even need legacy registers in this case), I
>> gives more deterministic behavior so we don't need to care about the
>> cross registers read/write.
> This needs thought, it is definitely more work. Effort that could be
> maybe spent on new features. What is the motivation
> here? supporting legacy mmio guests?
Probably. It tries to make legacy transport independent, and it's the
way that how transport virtqueue want to be.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 7:13 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
yishaih, maorg, virtio-comment, shahafs
在 2023/5/11 20:54, Michael S. Tsirkin 写道:
> On Thu, May 11, 2023 at 03:04:40PM +0800, Jason Wang wrote:
>> On Wed, May 10, 2023 at 3:43 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>> On Wed, May 10, 2023 at 03:01:25PM +0800, Jason Wang wrote:
>>>> On Wed, May 10, 2023 at 2:05 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>>>>> On Mon, May 08, 2023 at 10:23:39AM +0800, Jason Wang wrote:
>>>>>>> I thought so too originally. Unfortunately I now think that no, legacy is not
>>>>>>> going to be a byproduct of transport virtqueue for modern -
>>>>>>> it is different enough that it needs dedicated commands.
>>>>>> If you mean the transport virtqueue, I think some dedicated commands
>>>>>> for legacy are needed. Then it would be a transport that supports
>>>>>> transitional devices. It would be much better than having commands for
>>>>>> a partial transport like this patch did.
>>>>> OK I am beginning to get what you are saying. So your criticism is
>>>>> this: what if device supports vq transport for modern, and we want to
>>>>> build a transitional device on top. how will that look. yes?
>>>> Yes. I think it needs to be done through the transport virtqueue
>>>> otherwise the transport is not self-contained.
>>> I mean, any feature can be done over transport vq.
>>>
>>> But there is value in adding legacy commands to an otherwise
>>> modern device without reworking it completely to
>>> switch to a different transport.
>> There's probably no need for a rework since legacy is not complicated.
>> More below.
>>
>>>
>>>>> A reasonable thing to include at least in the commit log. Parav?
>>>>>
>>>>> You are also asking what if the device uses transport vq,
>>>>> and we want transitional on top of that.
>>>>> It's a fair question but I don't exactly get why would
>>>>> this legacy support feature be wanted for the vq transport
>>>>> and not for other transports.
>>>> Not sure I get the question, but all the existing transport support
>>>> legacy, if we want to have another, should the legacy support be a
>>>> must or not?
>>> This specific proposal is for tunneling legacy over admin vq.
>>> It can live alongside a normal modern VF, with hypervisor
>>> combining these to create a transitional device.
>> Exactly, but what I meant here is
>>
>> If we decide to use the admin vq, is there any good reason to tie it
>> to PCI if we don't want to tunneling PCI over adminq?
>>
>> Why not simply invent individual commands to access legacy facilities
>> like commands to access like what transport virtqueue did for modern
>> device?:
>>
>> 1) device features
>> 2) driver features
>> 3) queue address
>> 4) queue size
>> 5) queue select
>> 6) queue notify
>> 7) device status
>> 8) ISR status
>> 9) config msix
>> 10) queue msix
>> 11) device configuration space
>>
>> It focuses on the facilities instead of transport specific details
>> like registers (we don't even need legacy registers in this case), I
>> gives more deterministic behavior so we don't need to care about the
>> cross registers read/write.
> This needs thought, it is definitely more work. Effort that could be
> maybe spent on new features. What is the motivation
> here? supporting legacy mmio guests?
Probably. It tries to make legacy transport independent, and it's the
way that how transport virtqueue want to be.
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:02 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 7:30 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:30 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:02, Parav Pandit 写道:
>
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Thursday, May 11, 2023 8:55 AM
>>> 1) device features
>>> 2) driver features
>>> 3) queue address
>>> 4) queue size
>>> 5) queue select
>>> 6) queue notify
>>> 7) device status
>>> 8) ISR status
>>> 9) config msix
>>> 10) queue msix
>>> 11) device configuration space
>>>
> #9 may not even go to the group owner device.
It the config_msix_vector register so I don't understand why it can't
work but can work with your proposal.
> What do we gain from bisecting it?
1) No need to deal with the offset in the hardware
2) transport independent
3) avoid duplication with the (legacy support of) transport virtqueue
> Every new additional needs involvement of the hypervisor as Michael noted below on "effort" point.
I'm not sure I get your point.
If additional effort is not wanted, we need a full PCI over adminq,
that's what I suggested, then we don't need any extension for any new
features probably.
If we choose to invent dedicated adminq commands:
For legacy, we don't need any new additional effort since the above 11
maps to the legacy virtio-pci facilities.
For modern, if the feature is something related to the transport, we
need new transport interface for sure.
>
> The register offset and read/write is far simpler interface for hypervisor.
If this is true, why not go full PCI over adminq? Hypervisor in this
case even don't need to deal with config space.
>
> Not to do cross register access is what we need to define in the spec for 1.x or future things.
You can't assume the hypervisor behaviors, so the device or at least the
spec should be ready for that.
Thanks
>
>>> It focuses on the facilities instead of transport specific details
>>> like registers (we don't even need legacy registers in this case), I
>>> gives more deterministic behavior so we don't need to care about the
>>> cross registers read/write.
>> This needs thought, it is definitely more work. Effort that could be maybe spent
>> on new features. What is the motivation here? supporting legacy mmio guests?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 7:30 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-15 7:30 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:02, Parav Pandit 写道:
>
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Thursday, May 11, 2023 8:55 AM
>>> 1) device features
>>> 2) driver features
>>> 3) queue address
>>> 4) queue size
>>> 5) queue select
>>> 6) queue notify
>>> 7) device status
>>> 8) ISR status
>>> 9) config msix
>>> 10) queue msix
>>> 11) device configuration space
>>>
> #9 may not even go to the group owner device.
It the config_msix_vector register so I don't understand why it can't
work but can work with your proposal.
> What do we gain from bisecting it?
1) No need to deal with the offset in the hardware
2) transport independent
3) avoid duplication with the (legacy support of) transport virtqueue
> Every new additional needs involvement of the hypervisor as Michael noted below on "effort" point.
I'm not sure I get your point.
If additional effort is not wanted, we need a full PCI over adminq,
that's what I suggested, then we don't need any extension for any new
features probably.
If we choose to invent dedicated adminq commands:
For legacy, we don't need any new additional effort since the above 11
maps to the legacy virtio-pci facilities.
For modern, if the feature is something related to the transport, we
need new transport interface for sure.
>
> The register offset and read/write is far simpler interface for hypervisor.
If this is true, why not go full PCI over adminq? Hypervisor in this
case even don't need to deal with config space.
>
> Not to do cross register access is what we need to define in the spec for 1.x or future things.
You can't assume the hypervisor behaviors, so the device or at least the
spec should be ready for that.
Thanks
>
>>> It focuses on the facilities instead of transport specific details
>>> like registers (we don't even need legacy registers in this case), I
>>> gives more deterministic behavior so we don't need to care about the
>>> cross registers read/write.
>> This needs thought, it is definitely more work. Effort that could be maybe spent
>> on new features. What is the motivation here? supporting legacy mmio guests?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 7:30 ` [virtio-comment] " Jason Wang
@ 2023-05-15 10:08 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 10:08 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:30:47PM +0800, Jason Wang wrote:
>
> 在 2023/5/11 21:02, Parav Pandit 写道:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, May 11, 2023 8:55 AM
> > > > 1) device features
> > > > 2) driver features
> > > > 3) queue address
> > > > 4) queue size
> > > > 5) queue select
> > > > 6) queue notify
> > > > 7) device status
> > > > 8) ISR status
> > > > 9) config msix
> > > > 10) queue msix
> > > > 11) device configuration space
> > > >
> > #9 may not even go to the group owner device.
>
>
> It the config_msix_vector register so I don't understand why it can't work
> but can work with your proposal.
>
>
> > What do we gain from bisecting it?
>
>
> 1) No need to deal with the offset in the hardware
>
> 2) transport independent
>
> 3) avoid duplication with the (legacy support of) transport virtqueue
To be more specific, device config often has the same layout under
legacy and modern. Thus if getting an offset within device config,
same decoding logic should be reusable between legacy and modern.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 10:08 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 10:08 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:30:47PM +0800, Jason Wang wrote:
>
> 在 2023/5/11 21:02, Parav Pandit 写道:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, May 11, 2023 8:55 AM
> > > > 1) device features
> > > > 2) driver features
> > > > 3) queue address
> > > > 4) queue size
> > > > 5) queue select
> > > > 6) queue notify
> > > > 7) device status
> > > > 8) ISR status
> > > > 9) config msix
> > > > 10) queue msix
> > > > 11) device configuration space
> > > >
> > #9 may not even go to the group owner device.
>
>
> It the config_msix_vector register so I don't understand why it can't work
> but can work with your proposal.
>
>
> > What do we gain from bisecting it?
>
>
> 1) No need to deal with the offset in the hardware
>
> 2) transport independent
>
> 3) avoid duplication with the (legacy support of) transport virtqueue
To be more specific, device config often has the same layout under
legacy and modern. Thus if getting an offset within device config,
same decoding logic should be reusable between legacy and modern.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 10:08 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 14:30 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 14:30 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 6:09 AM
> To be more specific, device config often has the same layout under legacy and
> modern. Thus if getting an offset within device config, same decoding logic
> should be reusable between legacy and modern.
Modern device registers are directly accessible from the guest driver without mediation for VF,SF,SIOV.
Hence, there is no need to have commands for modern register access over some VQ.
Will reply to rest of the Jason's comments in some time.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 14:30 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 14:30 UTC (permalink / raw)
To: Michael S. Tsirkin, Jason Wang
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 6:09 AM
> To be more specific, device config often has the same layout under legacy and
> modern. Thus if getting an offset within device config, same decoding logic
> should be reusable between legacy and modern.
Modern device registers are directly accessible from the guest driver without mediation for VF,SF,SIOV.
Hence, there is no need to have commands for modern register access over some VQ.
Will reply to rest of the Jason's comments in some time.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 7:20 ` Jason Wang
@ 2023-05-15 15:25 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:25 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:21 AM
>
> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > On 5/10/2023 12:22 AM, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com>
> wrote:
> >
> > >
> > > Just to make sure we are at the same page.
> > >
> > > 1) if the hardware has configq and we need to make it work for
> > > current virtio-pci driver, hypervisor needs to trap guest PCI access
> > > and translate it to configq command. This means the hypervisor needs
> > > to hide the configq from guests. In this case the configq needs a
> > > dedicated DMA address which is what PASID can help.
> >
> > > 2) if the hardware can report the device states, unless we want to
> > > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > > required to isolate the DMA for guest traffic and device state.
> >
> > A configq negotiates + discovers new fields for the PCI PF, VF,
> > SF/SIOV devices over PCIe or other transports.
> > So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>
> As discussed, I may misunderstand the function of the configq. If it works like
> cvq, then yes. We don't need PASID.
>
Right.
> But if we want to report device state via let's say migration q, it requires PASID
> since it is used by the hypervisor.
When a device is used as passthrough, usually the owner group member would perform the device state management.
Hence passthrough mode will not use pasid.
When device is mediated in HV, yes PASID may be needed to isolate the DMA of hypervisor from the guest.
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 15:25 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:25 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 11, 2023 3:21 AM
>
> On Thu, May 11, 2023 at 12:08 AM Parav Pandit <parav@nvidia.com> wrote:
> >
> >
> >
> > On 5/10/2023 12:22 AM, Jason Wang wrote:
> > > On Wed, May 10, 2023 at 11:51 AM Jason Wang <jasowang@redhat.com>
> wrote:
> >
> > >
> > > Just to make sure we are at the same page.
> > >
> > > 1) if the hardware has configq and we need to make it work for
> > > current virtio-pci driver, hypervisor needs to trap guest PCI access
> > > and translate it to configq command. This means the hypervisor needs
> > > to hide the configq from guests. In this case the configq needs a
> > > dedicated DMA address which is what PASID can help.
> >
> > > 2) if the hardware can report the device states, unless we want to
> > > migrate L2 guest, hypervisor should hide it from L1, so PASID is
> > > required to isolate the DMA for guest traffic and device state.
> >
> > A configq negotiates + discovers new fields for the PCI PF, VF,
> > SF/SIOV devices over PCIe or other transports.
> > So no need to hide/mediate for hw based devices, like cvq and like data vqs.
>
> As discussed, I may misunderstand the function of the configq. If it works like
> cvq, then yes. We don't need PASID.
>
Right.
> But if we want to report device state via let's say migration q, it requires PASID
> since it is used by the hypervisor.
When a device is used as passthrough, usually the owner group member would perform the device state management.
Hence passthrough mode will not use pasid.
When device is mediated in HV, yes PASID may be needed to isolate the DMA of hypervisor from the guest.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 5:12 ` Jason Wang
@ 2023-05-15 15:26 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:26 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 1:12 AM
>
> 在 2023/5/11 22:31, Parav Pandit 写道:
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Thursday, May 11, 2023 3:17 AM
> >>> a configq/cmdq discussion is for new features for PF, VF, and
> >>> SF/SIOV devices to work in _non_ backward compatible way, because
> >>> for new features there is no such thing as backward compatibility
> >>> between guest driver and the device.
> >> Ok, so what command did configq/cmdq have?
> >> I thought it was similar to transport virtqueue, but it looks more like a
> control virtqueue here.
> > Yes, it is more like a cvq to do device attributes discovery and configuration.
> > Just that its generic enough.
> > For example
> > creating VQ that spans multiple physical addresses.
> > Discovery and enabling new device attributes which are rarely accessed or
> accessed during device init/cleanup time.
> >
> > The fundamental difference between cfgvq to tvq is:
> > Cfgvq is between the member virtio device and its driver.
> > Tvq=aq is on group owner of the virtio device.
> >
> >>>>> Compatibility is coming at a cost which is not scaling.
> >>>>> If you attempt to scale, it breaks the future path forward to go
> >>>>> without
> >> mediation.
> >>>>>>> So eve growing these fields and optionally placement on configq
> >>>>>>> doesn't really help and device builder to build it efficiently
> >>>>>>> (without much predictability).
> >>>>>> Config queue is not the only choice, we have a lot of other
> >>>>>> choices (for example PASID may help to reduce the on-chip resources).
> >>>>>>
> >>>>> PASID is for process isolation security construct, it is not for
> transportation.
> >>>> I meant with PASID you don't even need a BAR.
> >>>>
> >>> Not sure I fully understand, but my guess is, this is coming from
> >>> mediation thought process.
> >> Not actually. I meant, for example without PASID, for modern devices,
> >> the common cfg structure must be placed on a BAR through MMIO.
> >>
> >> But with the help of PASID, it could stay in the memory and the
> >> device can fetch it via DMA. The help to reduce the resources for registers.
> >>
> > Yes, it can stay in memory and device can fetch via DMA using a VQ using its
> own q from guest driver.
> > Why does it need PASID?
>
>
> Let me clarify my understanding.
>
> If we want a per VF virtqueue that is used by the hypervisor. We need PASID.
> Otherwise not.
Yes, we are on same page, only when such device is mediated by hypervisor.
Without mediation PASID is not required.
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 15:26 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:26 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 1:12 AM
>
> 在 2023/5/11 22:31, Parav Pandit 写道:
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Thursday, May 11, 2023 3:17 AM
> >>> a configq/cmdq discussion is for new features for PF, VF, and
> >>> SF/SIOV devices to work in _non_ backward compatible way, because
> >>> for new features there is no such thing as backward compatibility
> >>> between guest driver and the device.
> >> Ok, so what command did configq/cmdq have?
> >> I thought it was similar to transport virtqueue, but it looks more like a
> control virtqueue here.
> > Yes, it is more like a cvq to do device attributes discovery and configuration.
> > Just that its generic enough.
> > For example
> > creating VQ that spans multiple physical addresses.
> > Discovery and enabling new device attributes which are rarely accessed or
> accessed during device init/cleanup time.
> >
> > The fundamental difference between cfgvq to tvq is:
> > Cfgvq is between the member virtio device and its driver.
> > Tvq=aq is on group owner of the virtio device.
> >
> >>>>> Compatibility is coming at a cost which is not scaling.
> >>>>> If you attempt to scale, it breaks the future path forward to go
> >>>>> without
> >> mediation.
> >>>>>>> So eve growing these fields and optionally placement on configq
> >>>>>>> doesn't really help and device builder to build it efficiently
> >>>>>>> (without much predictability).
> >>>>>> Config queue is not the only choice, we have a lot of other
> >>>>>> choices (for example PASID may help to reduce the on-chip resources).
> >>>>>>
> >>>>> PASID is for process isolation security construct, it is not for
> transportation.
> >>>> I meant with PASID you don't even need a BAR.
> >>>>
> >>> Not sure I fully understand, but my guess is, this is coming from
> >>> mediation thought process.
> >> Not actually. I meant, for example without PASID, for modern devices,
> >> the common cfg structure must be placed on a BAR through MMIO.
> >>
> >> But with the help of PASID, it could stay in the memory and the
> >> device can fetch it via DMA. The help to reduce the resources for registers.
> >>
> > Yes, it can stay in memory and device can fetch via DMA using a VQ using its
> own q from guest driver.
> > Why does it need PASID?
>
>
> Let me clarify my understanding.
>
> If we want a per VF virtqueue that is used by the hypervisor. We need PASID.
> Otherwise not.
Yes, we are on same page, only when such device is mediated by hypervisor.
Without mediation PASID is not required.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 5:19 ` [virtio-comment] " Jason Wang
@ 2023-05-15 15:31 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:31 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 1:20 AM
>
> Right, but as discussed, a possible user is the SIOV transitional device via
> transport virtqueues.
>
>
> > transport vq might, but I am worried we'll have to spend extra
> > work clarifying it's a legacy interface. For example we will
> > have to explain that only 32 bits of features must be used.
> > Focusing more effort on transport vq is attractive though ...
> > It is worth examining this option in more depth.
> > Did you?
>
>
> My understanding is, for any case, we need such clarifications. What
> Parav proposed here is not the access to the actual registers but stick
> to the semantic of those registers. I think we must say high 32bits of
> the features must be assumed to be 0.
>
I am avoided duplicating anything that is already written for the PCI interface.
In legacy interface section of PCI, we already have:
"Note that only Feature Bits 0 to 31 are accessible through the Legacy Interface".
If MMIO is missing such line in existing interface, it is better to be added there.
It is not scope of this patch.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 15:31 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:31 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 1:20 AM
>
> Right, but as discussed, a possible user is the SIOV transitional device via
> transport virtqueues.
>
>
> > transport vq might, but I am worried we'll have to spend extra
> > work clarifying it's a legacy interface. For example we will
> > have to explain that only 32 bits of features must be used.
> > Focusing more effort on transport vq is attractive though ...
> > It is worth examining this option in more depth.
> > Did you?
>
>
> My understanding is, for any case, we need such clarifications. What
> Parav proposed here is not the access to the actual registers but stick
> to the semantic of those registers. I think we must say high 32bits of
> the features must be assumed to be 0.
>
I am avoided duplicating anything that is already written for the PCI interface.
In legacy interface section of PCI, we already have:
"Note that only Feature Bits 0 to 31 are accessible through the Legacy Interface".
If MMIO is missing such line in existing interface, it is better to be added there.
It is not scope of this patch.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 7:10 ` [virtio-comment] " Jason Wang
@ 2023-05-15 15:49 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:49 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 3:11 AM
> > It is just bunch of admin commands in the new SIOV or virtual virtio device
> chapter.
> > It is not a transport chapter.
>
>
> Provisioning could be a part of the transport. For example SR-IOV allows static
> and dynamic provisioning. If a virtio device needs to be implemented in VF, it
> needs to be provisioned first.
>
Provisioning a device by admin != transport guest driver <-> device communication.
Two very different things.
>
> >
> >>> And the motivation is also clear is to provide composing a virtio
> >>> device for the
> >> guest VM for the backward compatibility for 1.x part of the specification.
> >>> It seems fine and indeed orthogonal to me that: it is for backward
> >> compatibility for already defined config fields for existing guest VM driver.
> >>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> >>> for the
> >> new config fields, new use cases that doesn't require any mediation.
> >>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> >>> without
> >> mediation.
> >>
> >> My understanding is that the cfgq/cmdq could be done on top, since
> >> the commands are part of transport unless I miss something.
> >>
> > On top of what?
> > cfgq/cmdq is just another queue like any other VQ.
> > it is orthogonal to accessing 1.x registers using group owner's queue.
>
>
> I think there hasn't been any proposal that call any virtqueue like cfgq/cmdq. So
> using that may cause a lot of misunderstanding. I think the context here is to
> provide or extend transport facilities via a virtqueue not something like the
> control virtqueue.
The context is to compose a PCI device for a guest which for the currently defined features.
Once a new attribute is done using cfgq/cmdq there is no problem of transporting it via its group member.
And therefore, saying any new attribute also to ride on same tranportq/aq via is not appropriate.
>
> Admin virtqueue doesn't preventing anything from letting IMS go directly to the
> device.
But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
>
>
> > Long discussion with Thomas on IMS topic.
> > I will avoid diverting to unrelated discussion for now.
> >
> >>> Or proposed command in [1] should be tagged as siov_r1, then things will
> be
> >> cleaner.
> >>
> >> Maybe we can say, one of the implementations is siov_r1, since it can be
> used
> >> to implement devices in other transport. One of the main motivations for
> >> transport virtqueue is to reduce the transport specific resources to ease the
> >> virtio device implementation.
> >>
> > Yes, the main motivation as for backward compatibility for the currently
> defined features.
> >
> >>> With that I don't see legacy 3 commands anyway conflict with [1].
> >> It doesn't, but it's not elegant since:
> >>
> > I don’t see how making 3 commands to 9 commands makes it elegant by
> doing bisecting of registers offset in hypervisor.
>
>
> Or it needs to be done by the hardware and cross register write/read
> needs to be handled there as well.
>
That is the limitation of legacy and device can always _decide_ when to apply the parameter later when enough bits are written.
Given its control path, it is not an overhead.
>
> > Its same thing using more opcodes.
>
>
> Do we really care about the number of opcodes? I think we've reserved
> sufficient spaces.
>
I don’t see opcode as problem, we have plenty.
And hence, when there is a device that _does not implement config space for SIOV/SF, it should be introduced.
It is the introduction and bisection at the hypervisor level unnecessary.
And I don’t have object to it either, the main reason is: I don’t see it being useful for 1.x access going forward.
>
> >
> >> 1) Modern transport, admin virtqueue with well defined transport
> commands
> >> 2) Legacy transport, works moe like a PCI transport on top of the admin
> >> virtqueue
> >>
> >> Transport virtqueue tend to be transport independent, but 2) tie it somehow
> to
> >> the PCI. That's why I suggest to
> >>
> > The 4 patches are not transporting all the PCI things over transport VQ. it is
> not a transport.
>
>
> It really depends on the different viewpoint.
You wrote/asked for "PCI transport over admin q".
That means PCI capabilities, PCI config registers and all things defined in the PCI transport section to go over the AQ.
This is clearly not the objective of these 2 patches.
Nor it is your proposal of transport VQ.
You only want to exchange currently defined config registers, interrupts and notification configuration using AQ.
Transport VQ is NOT transporting actual data, it is not transporting notifications etc.
It is just a config channel.
Hence, they are just commands not a "full transport".
> To me, adminq is not
> specific to PCI, so this proposal looks more like a partial transport
> for legacy.
It is not a partial transport for legacy.
A future transport may be able to do for SIOV/MMIO by adding those group member identifier.
Hence, I agreed that commands in the tranportvq proposal is fine, as commands, not as new transport.
>
> So we have
>
> 1) all commands via PCI
> 2) all commands via adminq
> 3) part of the commands via PCI and the rest via adminq
>
> This proposal goes to for 3 with transport specific commands while I
> think we need to try 1 and 2 then it is more general and it helps to
> avoid duplication with future features.
>
All legacy interface via AQ.
All modern interface access via PCI or its own transport between driver and device.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 15:49 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:49 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 3:11 AM
> > It is just bunch of admin commands in the new SIOV or virtual virtio device
> chapter.
> > It is not a transport chapter.
>
>
> Provisioning could be a part of the transport. For example SR-IOV allows static
> and dynamic provisioning. If a virtio device needs to be implemented in VF, it
> needs to be provisioned first.
>
Provisioning a device by admin != transport guest driver <-> device communication.
Two very different things.
>
> >
> >>> And the motivation is also clear is to provide composing a virtio
> >>> device for the
> >> guest VM for the backward compatibility for 1.x part of the specification.
> >>> It seems fine and indeed orthogonal to me that: it is for backward
> >> compatibility for already defined config fields for existing guest VM driver.
> >>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> >>> for the
> >> new config fields, new use cases that doesn't require any mediation.
> >>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> >>> without
> >> mediation.
> >>
> >> My understanding is that the cfgq/cmdq could be done on top, since
> >> the commands are part of transport unless I miss something.
> >>
> > On top of what?
> > cfgq/cmdq is just another queue like any other VQ.
> > it is orthogonal to accessing 1.x registers using group owner's queue.
>
>
> I think there hasn't been any proposal that call any virtqueue like cfgq/cmdq. So
> using that may cause a lot of misunderstanding. I think the context here is to
> provide or extend transport facilities via a virtqueue not something like the
> control virtqueue.
The context is to compose a PCI device for a guest which for the currently defined features.
Once a new attribute is done using cfgq/cmdq there is no problem of transporting it via its group member.
And therefore, saying any new attribute also to ride on same tranportq/aq via is not appropriate.
>
> Admin virtqueue doesn't preventing anything from letting IMS go directly to the
> device.
But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
>
>
> > Long discussion with Thomas on IMS topic.
> > I will avoid diverting to unrelated discussion for now.
> >
> >>> Or proposed command in [1] should be tagged as siov_r1, then things will
> be
> >> cleaner.
> >>
> >> Maybe we can say, one of the implementations is siov_r1, since it can be
> used
> >> to implement devices in other transport. One of the main motivations for
> >> transport virtqueue is to reduce the transport specific resources to ease the
> >> virtio device implementation.
> >>
> > Yes, the main motivation as for backward compatibility for the currently
> defined features.
> >
> >>> With that I don't see legacy 3 commands anyway conflict with [1].
> >> It doesn't, but it's not elegant since:
> >>
> > I don’t see how making 3 commands to 9 commands makes it elegant by
> doing bisecting of registers offset in hypervisor.
>
>
> Or it needs to be done by the hardware and cross register write/read
> needs to be handled there as well.
>
That is the limitation of legacy and device can always _decide_ when to apply the parameter later when enough bits are written.
Given its control path, it is not an overhead.
>
> > Its same thing using more opcodes.
>
>
> Do we really care about the number of opcodes? I think we've reserved
> sufficient spaces.
>
I don’t see opcode as problem, we have plenty.
And hence, when there is a device that _does not implement config space for SIOV/SF, it should be introduced.
It is the introduction and bisection at the hypervisor level unnecessary.
And I don’t have object to it either, the main reason is: I don’t see it being useful for 1.x access going forward.
>
> >
> >> 1) Modern transport, admin virtqueue with well defined transport
> commands
> >> 2) Legacy transport, works moe like a PCI transport on top of the admin
> >> virtqueue
> >>
> >> Transport virtqueue tend to be transport independent, but 2) tie it somehow
> to
> >> the PCI. That's why I suggest to
> >>
> > The 4 patches are not transporting all the PCI things over transport VQ. it is
> not a transport.
>
>
> It really depends on the different viewpoint.
You wrote/asked for "PCI transport over admin q".
That means PCI capabilities, PCI config registers and all things defined in the PCI transport section to go over the AQ.
This is clearly not the objective of these 2 patches.
Nor it is your proposal of transport VQ.
You only want to exchange currently defined config registers, interrupts and notification configuration using AQ.
Transport VQ is NOT transporting actual data, it is not transporting notifications etc.
It is just a config channel.
Hence, they are just commands not a "full transport".
> To me, adminq is not
> specific to PCI, so this proposal looks more like a partial transport
> for legacy.
It is not a partial transport for legacy.
A future transport may be able to do for SIOV/MMIO by adding those group member identifier.
Hence, I agreed that commands in the tranportvq proposal is fine, as commands, not as new transport.
>
> So we have
>
> 1) all commands via PCI
> 2) all commands via adminq
> 3) part of the commands via PCI and the rest via adminq
>
> This proposal goes to for 3 with transport specific commands while I
> think we need to try 1 and 2 then it is more general and it helps to
> avoid duplication with future features.
>
All legacy interface via AQ.
All modern interface access via PCI or its own transport between driver and device.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 7:30 ` [virtio-comment] " Jason Wang
@ 2023-05-15 15:59 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:59 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 3:31 AM
> > What do we gain from bisecting it?
>
>
> 1) No need to deal with the offset in the hardware
>
In context of legacy is doesn’t matter much given that its over AQ.
> 2) transport independent
>
AQ commands are transport independent so, it is covered.
> 3) avoid duplication with the (legacy support of) transport virtqueue
>
Modern interface does not need mediation; hence it is not duplication with legacy.
>
> > Every new additional needs involvement of the hypervisor as Michael noted
> below on "effort" point.
>
>
> I'm not sure I get your point.
>
> If additional effort is not wanted, we need a full PCI over adminq,
> that's what I suggested, then we don't need any extension for any new
> features probably.
>
When there is already a PCI VF, I don’t see why one would implement "PCI over adminq".
It is impractical.
And SF/SIOV is bit still far in future.
When "PCI over adminq" happen, it is really a very different set of commands that emulated the "PCI virtio" device.
I don’t know when/why.
> If we choose to invent dedicated adminq commands:
>
> For legacy, we don't need any new additional effort since the above 11
> maps to the legacy virtio-pci facilities.
>
Only few commands are needed.
Modern interface config doesn’t travel through mediation.
> For modern, if the feature is something related to the transport, we
> need new transport interface for sure.
>
Everything is transport as it flows to/from driver to device covered in the respective transport chapter. 😊
>
> >
> > The register offset and read/write is far simpler interface for hypervisor.
>
>
> If this is true, why not go full PCI over adminq? Hypervisor in this
> case even don't need to deal with config space.
>
I don’t see a use case now and its extremely heavy to do so and it doesn’t apply to PCI VFs at all.
>
> >
> > Not to do cross register access is what we need to define in the spec for 1.x or
> future things.
>
>
> You can't assume the hypervisor behaviors, so the device or at least the
> spec should be ready for that.
>
My above comment is for the driver to device interface to access registers on its natural boundary.
Which part of the proposal assumes a hypervisor behavior?
Nothing to do with hypervisor.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 15:59 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 15:59 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 3:31 AM
> > What do we gain from bisecting it?
>
>
> 1) No need to deal with the offset in the hardware
>
In context of legacy is doesn’t matter much given that its over AQ.
> 2) transport independent
>
AQ commands are transport independent so, it is covered.
> 3) avoid duplication with the (legacy support of) transport virtqueue
>
Modern interface does not need mediation; hence it is not duplication with legacy.
>
> > Every new additional needs involvement of the hypervisor as Michael noted
> below on "effort" point.
>
>
> I'm not sure I get your point.
>
> If additional effort is not wanted, we need a full PCI over adminq,
> that's what I suggested, then we don't need any extension for any new
> features probably.
>
When there is already a PCI VF, I don’t see why one would implement "PCI over adminq".
It is impractical.
And SF/SIOV is bit still far in future.
When "PCI over adminq" happen, it is really a very different set of commands that emulated the "PCI virtio" device.
I don’t know when/why.
> If we choose to invent dedicated adminq commands:
>
> For legacy, we don't need any new additional effort since the above 11
> maps to the legacy virtio-pci facilities.
>
Only few commands are needed.
Modern interface config doesn’t travel through mediation.
> For modern, if the feature is something related to the transport, we
> need new transport interface for sure.
>
Everything is transport as it flows to/from driver to device covered in the respective transport chapter. 😊
>
> >
> > The register offset and read/write is far simpler interface for hypervisor.
>
>
> If this is true, why not go full PCI over adminq? Hypervisor in this
> case even don't need to deal with config space.
>
I don’t see a use case now and its extremely heavy to do so and it doesn’t apply to PCI VFs at all.
>
> >
> > Not to do cross register access is what we need to define in the spec for 1.x or
> future things.
>
>
> You can't assume the hypervisor behaviors, so the device or at least the
> spec should be ready for that.
>
My above comment is for the driver to device interface to access registers on its natural boundary.
Which part of the proposal assumes a hypervisor behavior?
Nothing to do with hypervisor.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 21:03 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 16:55 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 16:55 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Michael,
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Parav Pandit
> Sent: Thursday, May 11, 2023 5:03 PM
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, May 11, 2023 4:58 PM
> >
> > On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, May 11, 2023 9:39 AM
> > > >
> > > > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > > > This is not the objective to transport all PCI virtio fields over AQ.
> > > > > Transport VQ commands proposal commands and commands in this
> > > > > proposal
> > > > are just fine the way there without quoting it as "transport".
> > > >
> > > > I mean, it's a transport in a sense of where it will reside in e.g.
> > > > linux virtio stack - below virtio core.
> > > > We don't save much code there by reusing the same register layout
> > > > of legacy pci.
> > > > But let's at least make this feature complete, and then we'll see
> > > > whether it is cleaner to reuse legacy layout or build a new one.
> > >
> > > I don't follow your comment about "this feature complete".
> > > When you say _this_, it means the feature proposed in this patch of
> > supporting legacy over PCI VFs?
> > >
> > > If so, anything missing in this proposal?
> >
> > I think I commented at least on what I see as missing is first of all
> > a need to support INT#x emulation since level interrupts are I think not
> supported for VFs.
> > right?
>
> I don't see it mandatory as I responded in [1].
> I didn't see your response on [1].
>
> [1] https://lore.kernel.org/virtio-comment/71d65eb3-c025-9287-0157-
> 81e1d05574d1@nvidia.com/
Do you have any further suggestion, or we can differ the eventq for future when/if there is any real problem?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 16:55 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 16:55 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Michael,
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Parav Pandit
> Sent: Thursday, May 11, 2023 5:03 PM
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, May 11, 2023 4:58 PM
> >
> > On Thu, May 11, 2023 at 08:47:04PM +0000, Parav Pandit wrote:
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, May 11, 2023 9:39 AM
> > > >
> > > > On Thu, May 11, 2023 at 01:28:40PM +0000, Parav Pandit wrote:
> > > > > This is not the objective to transport all PCI virtio fields over AQ.
> > > > > Transport VQ commands proposal commands and commands in this
> > > > > proposal
> > > > are just fine the way there without quoting it as "transport".
> > > >
> > > > I mean, it's a transport in a sense of where it will reside in e.g.
> > > > linux virtio stack - below virtio core.
> > > > We don't save much code there by reusing the same register layout
> > > > of legacy pci.
> > > > But let's at least make this feature complete, and then we'll see
> > > > whether it is cleaner to reuse legacy layout or build a new one.
> > >
> > > I don't follow your comment about "this feature complete".
> > > When you say _this_, it means the feature proposed in this patch of
> > supporting legacy over PCI VFs?
> > >
> > > If so, anything missing in this proposal?
> >
> > I think I commented at least on what I see as missing is first of all
> > a need to support INT#x emulation since level interrupts are I think not
> supported for VFs.
> > right?
>
> I don't see it mandatory as I responded in [1].
> I didn't see your response on [1].
>
> [1] https://lore.kernel.org/virtio-comment/71d65eb3-c025-9287-0157-
> 81e1d05574d1@nvidia.com/
Do you have any further suggestion, or we can differ the eventq for future when/if there is any real problem?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 15:49 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 17:44 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 17:44 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> All legacy interface via AQ.
> All modern interface access via PCI or its own transport between driver and device.
I am wondering however about the hypervisor notifications. Generally
these are the most problematic aspect here I feel. For example, how
does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having
both guest and host be allowed to access device's BAR seems problematic.
Can we reserve a PF BAR region for these things or is that too
expensive?
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 17:44 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 17:44 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> All legacy interface via AQ.
> All modern interface access via PCI or its own transport between driver and device.
I am wondering however about the hypervisor notifications. Generally
these are the most problematic aspect here I feel. For example, how
does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having
both guest and host be allowed to access device's BAR seems problematic.
Can we reserve a PF BAR region for these things or is that too
expensive?
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 17:44 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 17:51 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 17:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 1:45 PM
>
> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > All legacy interface via AQ.
> > All modern interface access via PCI or its own transport between driver and
> device.
>
> I am wondering however about the hypervisor notifications. Generally these
> are the most problematic aspect here I feel. For example, how does it interact
> with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having both guest and
> host be allowed to access device's BAR seems problematic.
> Can we reserve a PF BAR region for these things or is that too expensive?
VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
For modern device, guest driver will access the device own BAR directly with its own config_data anyway.
Should we reserve a region in the PF BAR for SF/SIOV device?, should device report which BAR/area to use for the given SF/SIOV device?
May be yes, those are different discussions with tradeoff to consider during SIOV discussions. It is not related to VFs.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 17:51 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 17:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 1:45 PM
>
> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > All legacy interface via AQ.
> > All modern interface access via PCI or its own transport between driver and
> device.
>
> I am wondering however about the hypervisor notifications. Generally these
> are the most problematic aspect here I feel. For example, how does it interact
> with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having both guest and
> host be allowed to access device's BAR seems problematic.
> Can we reserve a PF BAR region for these things or is that too expensive?
VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
For modern device, guest driver will access the device own BAR directly with its own config_data anyway.
Should we reserve a region in the PF BAR for SF/SIOV device?, should device report which BAR/area to use for the given SF/SIOV device?
May be yes, those are different discussions with tradeoff to consider during SIOV discussions. It is not related to VFs.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 17:51 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 17:56 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 17:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 1:45 PM
> >
> > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > All legacy interface via AQ.
> > > All modern interface access via PCI or its own transport between driver and
> > device.
> >
> > I am wondering however about the hypervisor notifications. Generally these
> > are the most problematic aspect here I feel. For example, how does it interact
> > with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having both guest and
> > host be allowed to access device's BAR seems problematic.
> > Can we reserve a PF BAR region for these things or is that too expensive?
>
> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
it is not but it just might be required, or it won't be there.
> For modern device, guest driver will access the device own BAR directly with its own config_data anyway.
> Should we reserve a region in the PF BAR for SF/SIOV device?, should device report which BAR/area to use for the given SF/SIOV device?
> May be yes, those are different discussions with tradeoff to consider during SIOV discussions. It is not related to VFs.
For SIOV it's a given. But we can do this for SRIOV: reserve a PF region
per VF.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 17:56 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 17:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 1:45 PM
> >
> > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > All legacy interface via AQ.
> > > All modern interface access via PCI or its own transport between driver and
> > device.
> >
> > I am wondering however about the hypervisor notifications. Generally these
> > are the most problematic aspect here I feel. For example, how does it interact
> > with VIRTIO_F_NOTIF_CONFIG_DATA? And generally, having both guest and
> > host be allowed to access device's BAR seems problematic.
> > Can we reserve a PF BAR region for these things or is that too expensive?
>
> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
it is not but it just might be required, or it won't be there.
> For modern device, guest driver will access the device own BAR directly with its own config_data anyway.
> Should we reserve a region in the PF BAR for SF/SIOV device?, should device report which BAR/area to use for the given SF/SIOV device?
> May be yes, those are different discussions with tradeoff to consider during SIOV discussions. It is not related to VFs.
For SIOV it's a given. But we can do this for SRIOV: reserve a PF region
per VF.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 17:56 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 18:00 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 18:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 1:56 PM
>
> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Monday, May 15, 2023 1:45 PM
> > >
> > > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > > All legacy interface via AQ.
> > > > All modern interface access via PCI or its own transport between
> > > > driver and
> > > device.
> > >
> > > I am wondering however about the hypervisor notifications.
> > > Generally these are the most problematic aspect here I feel. For
> > > example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > > generally, having both guest and host be allowed to access device's BAR
> seems problematic.
> > > Can we reserve a PF BAR region for these things or is that too expensive?
> >
> > VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
>
> it is not but it just might be required, or it won't be there.
>
How can it be required if it not part of it?
I likely didn't follow your question/comment, can you please explain.
> > For modern device, guest driver will access the device own BAR directly with
> its own config_data anyway.
> > Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> report which BAR/area to use for the given SF/SIOV device?
> > May be yes, those are different discussions with tradeoff to consider during
> SIOV discussions. It is not related to VFs.
>
> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
Each VF has its own BAR area for driver notifications.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 18:00 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 18:00 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 1:56 PM
>
> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Monday, May 15, 2023 1:45 PM
> > >
> > > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > > All legacy interface via AQ.
> > > > All modern interface access via PCI or its own transport between
> > > > driver and
> > > device.
> > >
> > > I am wondering however about the hypervisor notifications.
> > > Generally these are the most problematic aspect here I feel. For
> > > example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > > generally, having both guest and host be allowed to access device's BAR
> seems problematic.
> > > Can we reserve a PF BAR region for these things or is that too expensive?
> >
> > VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
>
> it is not but it just might be required, or it won't be there.
>
How can it be required if it not part of it?
I likely didn't follow your question/comment, can you please explain.
> > For modern device, guest driver will access the device own BAR directly with
> its own config_data anyway.
> > Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> report which BAR/area to use for the given SF/SIOV device?
> > May be yes, those are different discussions with tradeoff to consider during
> SIOV discussions. It is not related to VFs.
>
> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
Each VF has its own BAR area for driver notifications.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 18:00 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 18:01 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 18:01 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 1:56 PM
> >
> > On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Monday, May 15, 2023 1:45 PM
> > > >
> > > > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > > > All legacy interface via AQ.
> > > > > All modern interface access via PCI or its own transport between
> > > > > driver and
> > > > device.
> > > >
> > > > I am wondering however about the hypervisor notifications.
> > > > Generally these are the most problematic aspect here I feel. For
> > > > example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > > > generally, having both guest and host be allowed to access device's BAR
> > seems problematic.
> > > > Can we reserve a PF BAR region for these things or is that too expensive?
> > >
> > > VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> >
> > it is not but it just might be required, or it won't be there.
> >
> How can it be required if it not part of it?
> I likely didn't follow your question/comment, can you please explain.
VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> > > For modern device, guest driver will access the device own BAR directly with
> > its own config_data anyway.
> > > Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> > report which BAR/area to use for the given SF/SIOV device?
> > > May be yes, those are different discussions with tradeoff to consider during
> > SIOV discussions. It is not related to VFs.
> >
> > For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> Each VF has its own BAR area for driver notifications.
But it is passed through to guest presumably.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 18:01 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 18:01 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 1:56 PM
> >
> > On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Monday, May 15, 2023 1:45 PM
> > > >
> > > > On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > > > > All legacy interface via AQ.
> > > > > All modern interface access via PCI or its own transport between
> > > > > driver and
> > > > device.
> > > >
> > > > I am wondering however about the hypervisor notifications.
> > > > Generally these are the most problematic aspect here I feel. For
> > > > example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > > > generally, having both guest and host be allowed to access device's BAR
> > seems problematic.
> > > > Can we reserve a PF BAR region for these things or is that too expensive?
> > >
> > > VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> >
> > it is not but it just might be required, or it won't be there.
> >
> How can it be required if it not part of it?
> I likely didn't follow your question/comment, can you please explain.
VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> > > For modern device, guest driver will access the device own BAR directly with
> > its own config_data anyway.
> > > Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> > report which BAR/area to use for the given SF/SIOV device?
> > > May be yes, those are different discussions with tradeoff to consider during
> > SIOV discussions. It is not related to VFs.
> >
> > For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> Each VF has its own BAR area for driver notifications.
But it is passed through to guest presumably.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 18:01 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-15 18:05 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 18:05 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 2:02 PM
> > >
> > How can it be required if it not part of it?
> > I likely didn't follow your question/comment, can you please explain.
>
> VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
The discussion is for the legacy interface for which CONFIG_DATA doesn't exist.
>
> > > > For modern device, guest driver will access the device own BAR
> > > > directly with
> > > its own config_data anyway.
> > > > Should we reserve a region in the PF BAR for SF/SIOV device?,
> > > > should device
> > > report which BAR/area to use for the given SF/SIOV device?
> > > > May be yes, those are different discussions with tradeoff to
> > > > consider during
> > > SIOV discussions. It is not related to VFs.
> > >
> > > For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per
> VF.
> > Each VF has its own BAR area for driver notifications.
>
> But it is passed through to guest presumably.
Yes, so?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 18:05 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 18:05 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 2:02 PM
> > >
> > How can it be required if it not part of it?
> > I likely didn't follow your question/comment, can you please explain.
>
> VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
The discussion is for the legacy interface for which CONFIG_DATA doesn't exist.
>
> > > > For modern device, guest driver will access the device own BAR
> > > > directly with
> > > its own config_data anyway.
> > > > Should we reserve a region in the PF BAR for SF/SIOV device?,
> > > > should device
> > > report which BAR/area to use for the given SF/SIOV device?
> > > > May be yes, those are different discussions with tradeoff to
> > > > consider during
> > > SIOV discussions. It is not related to VFs.
> > >
> > > For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per
> VF.
> > Each VF has its own BAR area for driver notifications.
>
> But it is passed through to guest presumably.
Yes, so?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-08 16:54 ` [virtio-comment] " Parav Pandit
@ 2023-05-15 20:29 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 20:29 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Mon, May 08, 2023 at 12:54:55PM -0400, Parav Pandit wrote:
>
> On 5/7/2023 5:04 AM, Michael S. Tsirkin wrote:
> >
> > one things I still don't see addressed here is support for
> > legacy interrupts. legacy driver can disable msix and
> > interrupts will be then sent.
> > how about a special command that is used when device would
> > normally send INT#x? it can also return ISR to reduce latency.
>
> I am not sure if this is a real issue. Because even the legacy guests have
> msix enabled by default. In theory yes, it can fall back to intx.
Well. I feel we should be closer to being sure it's not
an issue if we are going to ignore it.
some actual data here:
Even linux only enabled MSI-X in 2009.
Of course, other guests took longer. E.g.
a quick google search gave me this for some bsd variant (2017):
https://twitter.com/dragonflybsd/status/834494984229421057
Many guests have tunables to disable msix. Why?
E.g. BSD keeps maintaining it at
hw.virtio.pci.disable_msix
not a real use-case and you know 100% no guests have set this
to work around some bug e.g. in bsd MSI-X core? How can you be sure?
intx is used when guests run out of legacy interrupts,
these setups are not hard to create at all: just constrain the
number of vCPUs while creating lots of devices.
I could go on.
> There are few options.
> 1. A hypervisor driver can be conservative and steal an msix of the VF for
> transporting intx.
> Pros: Does not need special things in device
> Cons:
> a. Fairly intrusive in hypervisor vf driver.
> b. May not be ever used as guest is unlikely to fail on msix
Yea I do not like this since we are burning up msix vectors.
More reasons: this "pass through" msix has no chance to
set ISR properly since msix does not set ISR.
> 2. Since multiple VFs intx to be serviced, one command per VF in AQ is too
> much overhead that device needs to map a request to,
>
> A better way is to have an eventq of depth = num_vfs, like many other virtio
> devices have it.
>
> An eventq can hold per VF interrupt entry including the isr value that you
> suggest above.
>
> Something like,
>
> union eventq_entry {
> u8 raw_data[16];
> struct intx_entry {
> u8 event_opcode;
> u8 group_type;
> u8 reserved[6];
> le64 group_identifier;
> u8 isr_status;
> };
> };
>
> This eventq resides on the owner parent PF.
> isr_status is read on clear like today.
This is what I wrote no?
lore.kernel.org/all/20230507050146-mutt-send-email-mst%40kernel.org/t.mbox.gz
how about a special command that is used when device would
normally send INT#x? it can also return ISR to reduce latency.
> May be such eventq can be useful in future for wider case.
There's no maybe here is there? Things like live migration
need events for sure.
> We may have to find a different name for it as other devices has device
> specific eventq.
We don't need a special name for it. Just use an adminq
with a special command that is only consumed when there is an event.
Note you only need to queue a command if MSI is disabled.
Which is nice.
> I am inclined to differ this to a later point if one can identify the real
> failure with msix for the guest VM.
> So far we don't see this ever happening.
What is the question exactly?
Just have more devices than vectors,
an intel CPU only has ~200 of these, and current drivers
want to use 2 vectors and then fall back on INTx since that is shared.
Extremely easy to create - do you want a qemu command line to try?
Do specific customers event use guests with msi-x disabled? Maybe no.
Does anyone use virtio with msi-x disabled? Most likely yes.
So if we are going for legacy pci emulation let's have
a comprehensive legacy pci emulation please where
host can either enable it for a guest or deny completely,
not kind of start running then fail mysteriously.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 20:29 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-15 20:29 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Mon, May 08, 2023 at 12:54:55PM -0400, Parav Pandit wrote:
>
> On 5/7/2023 5:04 AM, Michael S. Tsirkin wrote:
> >
> > one things I still don't see addressed here is support for
> > legacy interrupts. legacy driver can disable msix and
> > interrupts will be then sent.
> > how about a special command that is used when device would
> > normally send INT#x? it can also return ISR to reduce latency.
>
> I am not sure if this is a real issue. Because even the legacy guests have
> msix enabled by default. In theory yes, it can fall back to intx.
Well. I feel we should be closer to being sure it's not
an issue if we are going to ignore it.
some actual data here:
Even linux only enabled MSI-X in 2009.
Of course, other guests took longer. E.g.
a quick google search gave me this for some bsd variant (2017):
https://twitter.com/dragonflybsd/status/834494984229421057
Many guests have tunables to disable msix. Why?
E.g. BSD keeps maintaining it at
hw.virtio.pci.disable_msix
not a real use-case and you know 100% no guests have set this
to work around some bug e.g. in bsd MSI-X core? How can you be sure?
intx is used when guests run out of legacy interrupts,
these setups are not hard to create at all: just constrain the
number of vCPUs while creating lots of devices.
I could go on.
> There are few options.
> 1. A hypervisor driver can be conservative and steal an msix of the VF for
> transporting intx.
> Pros: Does not need special things in device
> Cons:
> a. Fairly intrusive in hypervisor vf driver.
> b. May not be ever used as guest is unlikely to fail on msix
Yea I do not like this since we are burning up msix vectors.
More reasons: this "pass through" msix has no chance to
set ISR properly since msix does not set ISR.
> 2. Since multiple VFs intx to be serviced, one command per VF in AQ is too
> much overhead that device needs to map a request to,
>
> A better way is to have an eventq of depth = num_vfs, like many other virtio
> devices have it.
>
> An eventq can hold per VF interrupt entry including the isr value that you
> suggest above.
>
> Something like,
>
> union eventq_entry {
> u8 raw_data[16];
> struct intx_entry {
> u8 event_opcode;
> u8 group_type;
> u8 reserved[6];
> le64 group_identifier;
> u8 isr_status;
> };
> };
>
> This eventq resides on the owner parent PF.
> isr_status is read on clear like today.
This is what I wrote no?
lore.kernel.org/all/20230507050146-mutt-send-email-mst%40kernel.org/t.mbox.gz
how about a special command that is used when device would
normally send INT#x? it can also return ISR to reduce latency.
> May be such eventq can be useful in future for wider case.
There's no maybe here is there? Things like live migration
need events for sure.
> We may have to find a different name for it as other devices has device
> specific eventq.
We don't need a special name for it. Just use an adminq
with a special command that is only consumed when there is an event.
Note you only need to queue a command if MSI is disabled.
Which is nice.
> I am inclined to differ this to a later point if one can identify the real
> failure with msix for the guest VM.
> So far we don't see this ever happening.
What is the question exactly?
Just have more devices than vectors,
an intel CPU only has ~200 of these, and current drivers
want to use 2 vectors and then fall back on INTx since that is shared.
Extremely easy to create - do you want a qemu command line to try?
Do specific customers event use guests with msi-x disabled? Maybe no.
Does anyone use virtio with msi-x disabled? Most likely yes.
So if we are going for legacy pci emulation let's have
a comprehensive legacy pci emulation please where
host can either enable it for a guest or deny completely,
not kind of start running then fail mysteriously.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 20:29 ` Michael S. Tsirkin
@ 2023-05-15 20:56 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 20:56 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 4:30 PM
> >
> > I am not sure if this is a real issue. Because even the legacy guests
> > have msix enabled by default. In theory yes, it can fall back to intx.
>
> Well. I feel we should be closer to being sure it's not an issue if we are going to
> ignore it.
> some actual data here:
>
> Even linux only enabled MSI-X in 2009.
> Of course, other guests took longer. E.g.
> a quick google search gave me this for some bsd variant (2017):
> https://twitter.com/dragonflybsd/status/834494984229421057
>
> Many guests have tunables to disable msix. Why?
> E.g. BSD keeps maintaining it at
> hw.virtio.pci.disable_msix
> not a real use-case and you know 100% no guests have set this to work around
> some bug e.g. in bsd MSI-X core? How can you be sure?
>
>
>
> intx is used when guests run out of legacy interrupts, these setups are not hard
> to create at all: just constrain the number of vCPUs while creating lots of
> devices.
>
>
> I could go on.
>
>
>
> > There are few options.
> > 1. A hypervisor driver can be conservative and steal an msix of the VF
> > for transporting intx.
> > Pros: Does not need special things in device
> > Cons:
> > a. Fairly intrusive in hypervisor vf driver.
> > b. May not be ever used as guest is unlikely to fail on msix
>
> Yea I do not like this since we are burning up msix vectors.
> More reasons: this "pass through" msix has no chance to set ISR properly since
> msix does not set ISR.
>
>
> > 2. Since multiple VFs intx to be serviced, one command per VF in AQ is
> > too much overhead that device needs to map a request to,
> >
> > A better way is to have an eventq of depth = num_vfs, like many other
> > virtio devices have it.
> >
> > An eventq can hold per VF interrupt entry including the isr value that
> > you suggest above.
> >
> > Something like,
> >
> > union eventq_entry {
> > u8 raw_data[16];
> > struct intx_entry {
> > u8 event_opcode;
> > u8 group_type;
> > u8 reserved[6];
> > le64 group_identifier;
> > u8 isr_status;
> > };
> > };
> >
> > This eventq resides on the owner parent PF.
> > isr_status is read on clear like today.
>
> This is what I wrote no?
> lore.kernel.org/all/20230507050146-mutt-send-email-
> mst%40kernel.org/t.mbox.gz
>
> how about a special command that is used when device would
> normally send INT#x? it can also return ISR to reduce latency.
>
In response to your above suggestion of AQ command,
I suggested the eventq that contains the isr_status that reduces latency as you suggest.
> > May be such eventq can be useful in future for wider case.
>
> There's no maybe here is there? Things like live migration need events for sure.
>
> > We may have to find a different name for it as other devices has
> > device specific eventq.
>
> We don't need a special name for it. Just use an adminq with a special
> command that is only consumed when there is an event.
This requires too many commands to be issued on the PF device.
Potentially one per VF. And device needs to keep track of command to VF mapping.
> Note you only need to queue a command if MSI is disabled.
> Which is nice.
Yes, it is nice.
An eventq is a variation of it, where device can keep reporting the events without doing the extra mapping and without too many commands.
Additionally, eventq also works for 1.x device which will read the ISR status registers directly from the device.
>
> > I am inclined to differ this to a later point if one can identify the
> > real failure with msix for the guest VM.
> > So far we don't see this ever happening.
>
> What is the question exactly?
>
> Just have more devices than vectors,
> an intel CPU only has ~200 of these, and current drivers want to use 2 vectors
> and then fall back on INTx since that is shared.
> Extremely easy to create - do you want a qemu command line to try?
>
Intel CPU has 256 per core (per vcpu). So they are really a lot.
One needs to connect lot more devices to the cpu to run out of it.
So yes, I would like to try the command to fail.
> Do specific customers event use guests with msi-x disabled? Maybe no.
> Does anyone use virtio with msi-x disabled? Most likely yes.
I just feel that INTx emulation is extremely rare/narrow case of some applications that may never find its use on hw based devices.
> So if we are going for legacy pci emulation let's have a comprehensive legacy
> pci emulation please where host can either enable it for a guest or deny
> completely, not kind of start running then fail mysteriously.
A driver will be easily able to fail the call on INTx configuration failing the guest.
But lets see if can align to eventq/aq scheme to make it work.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-15 20:56 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-15 20:56 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 15, 2023 4:30 PM
> >
> > I am not sure if this is a real issue. Because even the legacy guests
> > have msix enabled by default. In theory yes, it can fall back to intx.
>
> Well. I feel we should be closer to being sure it's not an issue if we are going to
> ignore it.
> some actual data here:
>
> Even linux only enabled MSI-X in 2009.
> Of course, other guests took longer. E.g.
> a quick google search gave me this for some bsd variant (2017):
> https://twitter.com/dragonflybsd/status/834494984229421057
>
> Many guests have tunables to disable msix. Why?
> E.g. BSD keeps maintaining it at
> hw.virtio.pci.disable_msix
> not a real use-case and you know 100% no guests have set this to work around
> some bug e.g. in bsd MSI-X core? How can you be sure?
>
>
>
> intx is used when guests run out of legacy interrupts, these setups are not hard
> to create at all: just constrain the number of vCPUs while creating lots of
> devices.
>
>
> I could go on.
>
>
>
> > There are few options.
> > 1. A hypervisor driver can be conservative and steal an msix of the VF
> > for transporting intx.
> > Pros: Does not need special things in device
> > Cons:
> > a. Fairly intrusive in hypervisor vf driver.
> > b. May not be ever used as guest is unlikely to fail on msix
>
> Yea I do not like this since we are burning up msix vectors.
> More reasons: this "pass through" msix has no chance to set ISR properly since
> msix does not set ISR.
>
>
> > 2. Since multiple VFs intx to be serviced, one command per VF in AQ is
> > too much overhead that device needs to map a request to,
> >
> > A better way is to have an eventq of depth = num_vfs, like many other
> > virtio devices have it.
> >
> > An eventq can hold per VF interrupt entry including the isr value that
> > you suggest above.
> >
> > Something like,
> >
> > union eventq_entry {
> > u8 raw_data[16];
> > struct intx_entry {
> > u8 event_opcode;
> > u8 group_type;
> > u8 reserved[6];
> > le64 group_identifier;
> > u8 isr_status;
> > };
> > };
> >
> > This eventq resides on the owner parent PF.
> > isr_status is read on clear like today.
>
> This is what I wrote no?
> lore.kernel.org/all/20230507050146-mutt-send-email-
> mst%40kernel.org/t.mbox.gz
>
> how about a special command that is used when device would
> normally send INT#x? it can also return ISR to reduce latency.
>
In response to your above suggestion of AQ command,
I suggested the eventq that contains the isr_status that reduces latency as you suggest.
> > May be such eventq can be useful in future for wider case.
>
> There's no maybe here is there? Things like live migration need events for sure.
>
> > We may have to find a different name for it as other devices has
> > device specific eventq.
>
> We don't need a special name for it. Just use an adminq with a special
> command that is only consumed when there is an event.
This requires too many commands to be issued on the PF device.
Potentially one per VF. And device needs to keep track of command to VF mapping.
> Note you only need to queue a command if MSI is disabled.
> Which is nice.
Yes, it is nice.
An eventq is a variation of it, where device can keep reporting the events without doing the extra mapping and without too many commands.
Additionally, eventq also works for 1.x device which will read the ISR status registers directly from the device.
>
> > I am inclined to differ this to a later point if one can identify the
> > real failure with msix for the guest VM.
> > So far we don't see this ever happening.
>
> What is the question exactly?
>
> Just have more devices than vectors,
> an intel CPU only has ~200 of these, and current drivers want to use 2 vectors
> and then fall back on INTx since that is shared.
> Extremely easy to create - do you want a qemu command line to try?
>
Intel CPU has 256 per core (per vcpu). So they are really a lot.
One needs to connect lot more devices to the cpu to run out of it.
So yes, I would like to try the command to fail.
> Do specific customers event use guests with msi-x disabled? Maybe no.
> Does anyone use virtio with msi-x disabled? Most likely yes.
I just feel that INTx emulation is extremely rare/narrow case of some applications that may never find its use on hw based devices.
> So if we are going for legacy pci emulation let's have a comprehensive legacy
> pci emulation please where host can either enable it for a guest or deny
> completely, not kind of start running then fail mysteriously.
A driver will be easily able to fail the call on INTx configuration failing the guest.
But lets see if can align to eventq/aq scheme to make it work.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 15:49 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 3:28 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:28 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Parav:
在 2023/5/15 23:49, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Monday, May 15, 2023 3:11 AM
>>> It is just bunch of admin commands in the new SIOV or virtual virtio device
>> chapter.
>>> It is not a transport chapter.
>>
>> Provisioning could be a part of the transport. For example SR-IOV allows static
>> and dynamic provisioning. If a virtio device needs to be implemented in VF, it
>> needs to be provisioned first.
>>
> Provisioning a device by admin != transport guest driver <-> device communication.
>
> Two very different things.
When developing a new transport, we need to consider at least how to
probe the device. Before probe the device, it needs to be provisioned.
If we want a new transport, it could implement the provisioning
especially if it want the design to be bus independent. For the proposal
like transport virtqueue, it's also fine to use bus specific facility to
provision and probe the device.
That is what I want to say.
>
>>>>> And the motivation is also clear is to provide composing a virtio
>>>>> device for the
>>>> guest VM for the backward compatibility for 1.x part of the specification.
>>>>> It seems fine and indeed orthogonal to me that: it is for backward
>>>> compatibility for already defined config fields for existing guest VM driver.
>>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
>>>>> for the
>>>> new config fields, new use cases that doesn't require any mediation.
>>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
>>>>> without
>>>> mediation.
>>>>
>>>> My understanding is that the cfgq/cmdq could be done on top, since
>>>> the commands are part of transport unless I miss something.
>>>>
>>> On top of what?
>>> cfgq/cmdq is just another queue like any other VQ.
>>> it is orthogonal to accessing 1.x registers using group owner's queue.
>>
>> I think there hasn't been any proposal that call any virtqueue like cfgq/cmdq. So
>> using that may cause a lot of misunderstanding. I think the context here is to
>> provide or extend transport facilities via a virtqueue not something like the
>> control virtqueue.
> The context is to compose a PCI device for a guest which for the currently defined features.
> Once a new attribute is done using cfgq/cmdq there is no problem of transporting it via its group member.
>
> And therefore, saying any new attribute also to ride on same tranportq/aq via is not appropriate.
The point is to try to find if there's a better way, more below.
>
>> Admin virtqueue doesn't preventing anything from letting IMS go directly to the
>> device.
> But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
It's a choice of the implementation but not spec. Spec doesn't say
adminq can't be used for guest.
>
>>
>>> Long discussion with Thomas on IMS topic.
>>> I will avoid diverting to unrelated discussion for now.
>>>
>>>>> Or proposed command in [1] should be tagged as siov_r1, then things will
>> be
>>>> cleaner.
>>>>
>>>> Maybe we can say, one of the implementations is siov_r1, since it can be
>> used
>>>> to implement devices in other transport. One of the main motivations for
>>>> transport virtqueue is to reduce the transport specific resources to ease the
>>>> virtio device implementation.
>>>>
>>> Yes, the main motivation as for backward compatibility for the currently
>> defined features.
>>>>> With that I don't see legacy 3 commands anyway conflict with [1].
>>>> It doesn't, but it's not elegant since:
>>>>
>>> I don’t see how making 3 commands to 9 commands makes it elegant by
>> doing bisecting of registers offset in hypervisor.
>>
>>
>> Or it needs to be done by the hardware and cross register write/read
>> needs to be handled there as well.
>>
> That is the limitation of legacy and device can always _decide_ when to apply the parameter later when enough bits are written.
> Given its control path, it is not an overhead.
I think we've discussed many times that legacy is tricky for hardware.
And your proposal proves this further by introducing modern alike
notification areas. We probably need to deal with others like endianess.
For any case, having a simple and deterministic device design is always
better assuming we've agreed that mediation is a must for legacy
drivers. Using dedicated commands can make sure the implementation won't
need to go with corner cases of legacy.
>
>>> Its same thing using more opcodes.
>>
>> Do we really care about the number of opcodes? I think we've reserved
>> sufficient spaces.
>>
> I don’t see opcode as problem, we have plenty.
> And hence, when there is a device that _does not implement config space for SIOV/SF, it should be introduced.
> It is the introduction and bisection at the hypervisor level unnecessary.
>
> And I don’t have object to it either, the main reason is: I don’t see it being useful for 1.x access going forward.
As you said for future SIOV/SF? For example, it can have better
scalability in IMS since it doesn't require any real capabilities or
registers.
>
>>>> 1) Modern transport, admin virtqueue with well defined transport
>> commands
>>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
>>>> virtqueue
>>>>
>>>> Transport virtqueue tend to be transport independent, but 2) tie it somehow
>> to
>>>> the PCI. That's why I suggest to
>>>>
>>> The 4 patches are not transporting all the PCI things over transport VQ. it is
>> not a transport.
>>
>>
>> It really depends on the different viewpoint.
> You wrote/asked for "PCI transport over admin q".
> That means PCI capabilities, PCI config registers and all things defined in the PCI transport section to go over the AQ.
> This is clearly not the objective of these 2 patches.
It's not but it can achieve the same function as this series with much
broader use cases.
> Nor it is your proposal of transport VQ.
Same as the above.
I think we know your proposal might but it's worthwhile to figure out
whether or not there could be other better ways to achieve the same
thing. That's why the discussion happens on the cover letter.
Both transport virtqueue and "PCI over adminq" can support legacy guest
with a lot of other use cases.
>
> You only want to exchange currently defined config registers, interrupts and notification configuration using AQ.
>
> Transport VQ is NOT transporting actual data, it is not transporting notifications etc.
> It is just a config channel.
> Hence, they are just commands not a "full transport".
For any case, a virtqueue could not be a full transport. It needs bus
facilities to do bootstrap/DMA/notifications at least. The idea is to
save for transport specific resources and use a more scalable way to
config devices.
>
>> To me, adminq is not
>> specific to PCI, so this proposal looks more like a partial transport
>> for legacy.
> It is not a partial transport for legacy.
> A future transport may be able to do for SIOV/MMIO by adding those group member identifier.
>
> Hence, I agreed that commands in the tranportvq proposal is fine, as commands, not as new transport.
>
>> So we have
>>
>> 1) all commands via PCI
>> 2) all commands via adminq
>> 3) part of the commands via PCI and the rest via adminq
>>
>> This proposal goes to for 3 with transport specific commands while I
>> think we need to try 1 and 2 then it is more general and it helps to
>> avoid duplication with future features.
>>
> All legacy interface via AQ.
That's pretty fine. So we have already had modern transport via a
virtqueue, why not unify the design to avoid duplication? Otherwise we
may end up with
1) legacy PCI alike commands via admin virtqueue with modern alike
notification areas
2) modern device commands via admin virtqueue
> All modern interface access via PCI or its own transport between driver and device.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 3:28 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:28 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Parav:
在 2023/5/15 23:49, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Monday, May 15, 2023 3:11 AM
>>> It is just bunch of admin commands in the new SIOV or virtual virtio device
>> chapter.
>>> It is not a transport chapter.
>>
>> Provisioning could be a part of the transport. For example SR-IOV allows static
>> and dynamic provisioning. If a virtio device needs to be implemented in VF, it
>> needs to be provisioned first.
>>
> Provisioning a device by admin != transport guest driver <-> device communication.
>
> Two very different things.
When developing a new transport, we need to consider at least how to
probe the device. Before probe the device, it needs to be provisioned.
If we want a new transport, it could implement the provisioning
especially if it want the design to be bus independent. For the proposal
like transport virtqueue, it's also fine to use bus specific facility to
provision and probe the device.
That is what I want to say.
>
>>>>> And the motivation is also clear is to provide composing a virtio
>>>>> device for the
>>>> guest VM for the backward compatibility for 1.x part of the specification.
>>>>> It seems fine and indeed orthogonal to me that: it is for backward
>>>> compatibility for already defined config fields for existing guest VM driver.
>>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
>>>>> for the
>>>> new config fields, new use cases that doesn't require any mediation.
>>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
>>>>> without
>>>> mediation.
>>>>
>>>> My understanding is that the cfgq/cmdq could be done on top, since
>>>> the commands are part of transport unless I miss something.
>>>>
>>> On top of what?
>>> cfgq/cmdq is just another queue like any other VQ.
>>> it is orthogonal to accessing 1.x registers using group owner's queue.
>>
>> I think there hasn't been any proposal that call any virtqueue like cfgq/cmdq. So
>> using that may cause a lot of misunderstanding. I think the context here is to
>> provide or extend transport facilities via a virtqueue not something like the
>> control virtqueue.
> The context is to compose a PCI device for a guest which for the currently defined features.
> Once a new attribute is done using cfgq/cmdq there is no problem of transporting it via its group member.
>
> And therefore, saying any new attribute also to ride on same tranportq/aq via is not appropriate.
The point is to try to find if there's a better way, more below.
>
>> Admin virtqueue doesn't preventing anything from letting IMS go directly to the
>> device.
> But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
It's a choice of the implementation but not spec. Spec doesn't say
adminq can't be used for guest.
>
>>
>>> Long discussion with Thomas on IMS topic.
>>> I will avoid diverting to unrelated discussion for now.
>>>
>>>>> Or proposed command in [1] should be tagged as siov_r1, then things will
>> be
>>>> cleaner.
>>>>
>>>> Maybe we can say, one of the implementations is siov_r1, since it can be
>> used
>>>> to implement devices in other transport. One of the main motivations for
>>>> transport virtqueue is to reduce the transport specific resources to ease the
>>>> virtio device implementation.
>>>>
>>> Yes, the main motivation as for backward compatibility for the currently
>> defined features.
>>>>> With that I don't see legacy 3 commands anyway conflict with [1].
>>>> It doesn't, but it's not elegant since:
>>>>
>>> I don’t see how making 3 commands to 9 commands makes it elegant by
>> doing bisecting of registers offset in hypervisor.
>>
>>
>> Or it needs to be done by the hardware and cross register write/read
>> needs to be handled there as well.
>>
> That is the limitation of legacy and device can always _decide_ when to apply the parameter later when enough bits are written.
> Given its control path, it is not an overhead.
I think we've discussed many times that legacy is tricky for hardware.
And your proposal proves this further by introducing modern alike
notification areas. We probably need to deal with others like endianess.
For any case, having a simple and deterministic device design is always
better assuming we've agreed that mediation is a must for legacy
drivers. Using dedicated commands can make sure the implementation won't
need to go with corner cases of legacy.
>
>>> Its same thing using more opcodes.
>>
>> Do we really care about the number of opcodes? I think we've reserved
>> sufficient spaces.
>>
> I don’t see opcode as problem, we have plenty.
> And hence, when there is a device that _does not implement config space for SIOV/SF, it should be introduced.
> It is the introduction and bisection at the hypervisor level unnecessary.
>
> And I don’t have object to it either, the main reason is: I don’t see it being useful for 1.x access going forward.
As you said for future SIOV/SF? For example, it can have better
scalability in IMS since it doesn't require any real capabilities or
registers.
>
>>>> 1) Modern transport, admin virtqueue with well defined transport
>> commands
>>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
>>>> virtqueue
>>>>
>>>> Transport virtqueue tend to be transport independent, but 2) tie it somehow
>> to
>>>> the PCI. That's why I suggest to
>>>>
>>> The 4 patches are not transporting all the PCI things over transport VQ. it is
>> not a transport.
>>
>>
>> It really depends on the different viewpoint.
> You wrote/asked for "PCI transport over admin q".
> That means PCI capabilities, PCI config registers and all things defined in the PCI transport section to go over the AQ.
> This is clearly not the objective of these 2 patches.
It's not but it can achieve the same function as this series with much
broader use cases.
> Nor it is your proposal of transport VQ.
Same as the above.
I think we know your proposal might but it's worthwhile to figure out
whether or not there could be other better ways to achieve the same
thing. That's why the discussion happens on the cover letter.
Both transport virtqueue and "PCI over adminq" can support legacy guest
with a lot of other use cases.
>
> You only want to exchange currently defined config registers, interrupts and notification configuration using AQ.
>
> Transport VQ is NOT transporting actual data, it is not transporting notifications etc.
> It is just a config channel.
> Hence, they are just commands not a "full transport".
For any case, a virtqueue could not be a full transport. It needs bus
facilities to do bootstrap/DMA/notifications at least. The idea is to
save for transport specific resources and use a more scalable way to
config devices.
>
>> To me, adminq is not
>> specific to PCI, so this proposal looks more like a partial transport
>> for legacy.
> It is not a partial transport for legacy.
> A future transport may be able to do for SIOV/MMIO by adding those group member identifier.
>
> Hence, I agreed that commands in the tranportvq proposal is fine, as commands, not as new transport.
>
>> So we have
>>
>> 1) all commands via PCI
>> 2) all commands via adminq
>> 3) part of the commands via PCI and the rest via adminq
>>
>> This proposal goes to for 3 with transport specific commands while I
>> think we need to try 1 and 2 then it is more general and it helps to
>> avoid duplication with future features.
>>
> All legacy interface via AQ.
That's pretty fine. So we have already had modern transport via a
virtqueue, why not unify the design to avoid duplication? Otherwise we
may end up with
1) legacy PCI alike commands via admin virtqueue with modern alike
notification areas
2) modern device commands via admin virtqueue
> All modern interface access via PCI or its own transport between driver and device.
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 18:01 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-16 3:37 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:37 UTC (permalink / raw)
To: Michael S. Tsirkin, Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
>>> From: Michael S. Tsirkin <mst@redhat.com>
>>> Sent: Monday, May 15, 2023 1:56 PM
>>>
>>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
>>>>> From: Michael S. Tsirkin <mst@redhat.com>
>>>>> Sent: Monday, May 15, 2023 1:45 PM
>>>>>
>>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
>>>>>> All legacy interface via AQ.
>>>>>> All modern interface access via PCI or its own transport between
>>>>>> driver and
>>>>> device.
>>>>>
>>>>> I am wondering however about the hypervisor notifications.
>>>>> Generally these are the most problematic aspect here I feel. For
>>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
>>>>> generally, having both guest and host be allowed to access device's BAR
>>> seems problematic.
>>>>> Can we reserve a PF BAR region for these things or is that too expensive?
>>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
>>> it is not but it just might be required, or it won't be there.
>>>
>> How can it be required if it not part of it?
>> I likely didn't follow your question/comment, can you please explain.
> VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
>
>>>> For modern device, guest driver will access the device own BAR directly with
>>> its own config_data anyway.
>>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
>>> report which BAR/area to use for the given SF/SIOV device?
>>>> May be yes, those are different discussions with tradeoff to consider during
>>> SIOV discussions. It is not related to VFs.
>>>
>>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
>> Each VF has its own BAR area for driver notifications.
> But it is passed through to guest presumably.
Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
problem if we do mediation here except for some performance penalty?
Thanks
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 3:37 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:37 UTC (permalink / raw)
To: Michael S. Tsirkin, Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
>>> From: Michael S. Tsirkin <mst@redhat.com>
>>> Sent: Monday, May 15, 2023 1:56 PM
>>>
>>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
>>>>> From: Michael S. Tsirkin <mst@redhat.com>
>>>>> Sent: Monday, May 15, 2023 1:45 PM
>>>>>
>>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
>>>>>> All legacy interface via AQ.
>>>>>> All modern interface access via PCI or its own transport between
>>>>>> driver and
>>>>> device.
>>>>>
>>>>> I am wondering however about the hypervisor notifications.
>>>>> Generally these are the most problematic aspect here I feel. For
>>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
>>>>> generally, having both guest and host be allowed to access device's BAR
>>> seems problematic.
>>>>> Can we reserve a PF BAR region for these things or is that too expensive?
>>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
>>> it is not but it just might be required, or it won't be there.
>>>
>> How can it be required if it not part of it?
>> I likely didn't follow your question/comment, can you please explain.
> VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
>
>>>> For modern device, guest driver will access the device own BAR directly with
>>> its own config_data anyway.
>>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
>>> report which BAR/area to use for the given SF/SIOV device?
>>>> May be yes, those are different discussions with tradeoff to consider during
>>> SIOV discussions. It is not related to VFs.
>>>
>>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
>> Each VF has its own BAR area for driver notifications.
> But it is passed through to guest presumably.
Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
problem if we do mediation here except for some performance penalty?
Thanks
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:37 ` [virtio-comment] " Jason Wang
@ 2023-05-16 3:43 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:43 UTC (permalink / raw)
To: Michael S. Tsirkin, Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:37 AM Jason Wang <jasowang@redhat.com> wrote:
>
>
> 在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> > On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> >>> From: Michael S. Tsirkin <mst@redhat.com>
> >>> Sent: Monday, May 15, 2023 1:56 PM
> >>>
> >>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> >>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>> Sent: Monday, May 15, 2023 1:45 PM
> >>>>>
> >>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> >>>>>> All legacy interface via AQ.
> >>>>>> All modern interface access via PCI or its own transport between
> >>>>>> driver and
> >>>>> device.
> >>>>>
> >>>>> I am wondering however about the hypervisor notifications.
> >>>>> Generally these are the most problematic aspect here I feel. For
> >>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> >>>>> generally, having both guest and host be allowed to access device's BAR
> >>> seems problematic.
> >>>>> Can we reserve a PF BAR region for these things or is that too expensive?
> >>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> >>> it is not but it just might be required, or it won't be there.
> >>>
> >> How can it be required if it not part of it?
> >> I likely didn't follow your question/comment, can you please explain.
> > VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> >
> >>>> For modern device, guest driver will access the device own BAR directly with
> >>> its own config_data anyway.
> >>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> >>> report which BAR/area to use for the given SF/SIOV device?
> >>>> May be yes, those are different discussions with tradeoff to consider during
> >>> SIOV discussions. It is not related to VFs.
> >>>
> >>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> >> Each VF has its own BAR area for driver notifications.
> > But it is passed through to guest presumably.
>
>
> Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
> problem if we do mediation here except for some performance penalty?
Ok, rethink of this, when using legacy, VIRTIO_F_NOTIF_CONFIG_DATA
should be assumed to be not negotiated. So I think I agree with Parav,
it should not be a problem.
Or do you mean for the devices that have some mandated features?
Thanks
>
> Thanks
>
>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 3:43 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:43 UTC (permalink / raw)
To: Michael S. Tsirkin, Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:37 AM Jason Wang <jasowang@redhat.com> wrote:
>
>
> 在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> > On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> >>> From: Michael S. Tsirkin <mst@redhat.com>
> >>> Sent: Monday, May 15, 2023 1:56 PM
> >>>
> >>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> >>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>> Sent: Monday, May 15, 2023 1:45 PM
> >>>>>
> >>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> >>>>>> All legacy interface via AQ.
> >>>>>> All modern interface access via PCI or its own transport between
> >>>>>> driver and
> >>>>> device.
> >>>>>
> >>>>> I am wondering however about the hypervisor notifications.
> >>>>> Generally these are the most problematic aspect here I feel. For
> >>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> >>>>> generally, having both guest and host be allowed to access device's BAR
> >>> seems problematic.
> >>>>> Can we reserve a PF BAR region for these things or is that too expensive?
> >>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> >>> it is not but it just might be required, or it won't be there.
> >>>
> >> How can it be required if it not part of it?
> >> I likely didn't follow your question/comment, can you please explain.
> > VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> >
> >>>> For modern device, guest driver will access the device own BAR directly with
> >>> its own config_data anyway.
> >>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> >>> report which BAR/area to use for the given SF/SIOV device?
> >>>> May be yes, those are different discussions with tradeoff to consider during
> >>> SIOV discussions. It is not related to VFs.
> >>>
> >>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> >> Each VF has its own BAR area for driver notifications.
> > But it is passed through to guest presumably.
>
>
> Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
> problem if we do mediation here except for some performance penalty?
Ok, rethink of this, when using legacy, VIRTIO_F_NOTIF_CONFIG_DATA
should be assumed to be not negotiated. So I think I agree with Parav,
it should not be a problem.
Or do you mean for the devices that have some mandated features?
Thanks
>
> Thanks
>
>
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:28 ` [virtio-comment] " Jason Wang
@ 2023-05-16 3:45 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 3:45 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 11:28 PM
>
> Hi Parav:
>
> 在 2023/5/15 23:49, Parav Pandit 写道:
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Monday, May 15, 2023 3:11 AM
> >>> It is just bunch of admin commands in the new SIOV or virtual virtio
> >>> device
> >> chapter.
> >>> It is not a transport chapter.
> >>
> >> Provisioning could be a part of the transport. For example SR-IOV
> >> allows static and dynamic provisioning. If a virtio device needs to
> >> be implemented in VF, it needs to be provisioned first.
> >>
> > Provisioning a device by admin != transport guest driver <-> device
> communication.
> >
> > Two very different things.
>
>
> When developing a new transport, we need to consider at least how to probe
> the device. Before probe the device, it needs to be provisioned.
Yes, and yes to both. I agree.
> If we want a new transport, it could implement the provisioning especially if it
> want the design to be bus independent. For the proposal like transport
> virtqueue, it's also fine to use bus specific facility to provision and probe the
> device.
>
Provisioning is for defining the transport specific attributes.
Each transport has few command things and few unique things.
So provisioning commands can define transport specific attributes and few generic things.
This is fine.
>
> That is what I want to say.
>
>
> >
> >>>>> And the motivation is also clear is to provide composing a virtio
> >>>>> device for the
> >>>> guest VM for the backward compatibility for 1.x part of the specification.
> >>>>> It seems fine and indeed orthogonal to me that: it is for backward
> >>>> compatibility for already defined config fields for existing guest VM driver.
> >>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> >>>>> for the
> >>>> new config fields, new use cases that doesn't require any mediation.
> >>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> >>>>> without
> >>>> mediation.
> >>>>
> >>>> My understanding is that the cfgq/cmdq could be done on top, since
> >>>> the commands are part of transport unless I miss something.
> >>>>
> >>> On top of what?
> >>> cfgq/cmdq is just another queue like any other VQ.
> >>> it is orthogonal to accessing 1.x registers using group owner's queue.
> >>
> >> I think there hasn't been any proposal that call any virtqueue like
> >> cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
> >> the context here is to provide or extend transport facilities via a
> >> virtqueue not something like the control virtqueue.
> > The context is to compose a PCI device for a guest which for the currently
> defined features.
> > Once a new attribute is done using cfgq/cmdq there is no problem of
> transporting it via its group member.
> >
> > And therefore, saying any new attribute also to ride on same tranportq/aq via
> is not appropriate.
>
>
> The point is to try to find if there's a better way, more below.
>
>
> >
> >> Admin virtqueue doesn't preventing anything from letting IMS go
> >> directly to the device.
> > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it
> is prevented.
>
>
> It's a choice of the implementation but not spec. Spec doesn't say adminq can't
> be used for guest.
>
For guest and by guest are two different things.
I am proposing that a cfgq/cmdq should be used directly by the guest driver as base line like any other VQ.
And if there is a need for mediation, it should be possible.
> I think we've discussed many times that legacy is tricky for hardware.
And we want to proceed for the described use case and requirements.
> And your proposal proves this further by introducing modern alike
> notification areas.
Not really. A modern device is offering the notification area like legacy to the legacy interface.
So this is just fine.
> We probably need to deal with others like endianess.
>
No point in discussing this again. One size does not fit all.
It solves large part of the use cases so it is valuable and will be supported.
> For any case, having a simple and deterministic device design is always
> better assuming we've agreed that mediation is a must for legacy
> drivers. Using dedicated commands can make sure the implementation won't
> need to go with corner cases of legacy.
>
Corner case handling just moved from the device to hypervisor.
>
> >
> >>> Its same thing using more opcodes.
> >>
> >> Do we really care about the number of opcodes? I think we've reserved
> >> sufficient spaces.
> >>
> > I don’t see opcode as problem, we have plenty.
> > And hence, when there is a device that _does not implement config space for
> SIOV/SF, it should be introduced.
> > It is the introduction and bisection at the hypervisor level unnecessary.
> >
> > And I don’t have object to it either, the main reason is: I don’t see it being
> useful for 1.x access going forward.
>
>
> As you said for future SIOV/SF? For example, it can have better
> scalability in IMS since it doesn't require any real capabilities or
> registers.
>
For SIOV there is no backward compatibility requirement. Hence, it can start with fresh infra for real capabilities, registers and IMS.
>
> >
> >>>> 1) Modern transport, admin virtqueue with well defined transport
> >> commands
> >>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
> >>>> virtqueue
> >>>>
> >>>> Transport virtqueue tend to be transport independent, but 2) tie it
> somehow
> >> to
> >>>> the PCI. That's why I suggest to
> >>>>
> >>> The 4 patches are not transporting all the PCI things over transport VQ. it is
> >> not a transport.
> >>
> >>
> >> It really depends on the different viewpoint.
> > You wrote/asked for "PCI transport over admin q".
> > That means PCI capabilities, PCI config registers and all things defined in the
> PCI transport section to go over the AQ.
> > This is clearly not the objective of these 2 patches.
>
>
> It's not but it can achieve the same function as this series with much
> broader use cases.
>
>
> > Nor it is your proposal of transport VQ.
>
>
> Same as the above.
>
> I think we know your proposal might but it's worthwhile to figure out
> whether or not there could be other better ways to achieve the same
> thing. That's why the discussion happens on the cover letter.
>
> Both transport virtqueue and "PCI over adminq" can support legacy guest
> with a lot of other use cases.
>
PCI over admin queue just doesn’t work at performance scale to transport doorbells over a queue interface.
So, things just stop there.
>
> >
> > You only want to exchange currently defined config registers, interrupts and
> notification configuration using AQ.
> >
> > Transport VQ is NOT transporting actual data, it is not transporting
> notifications etc.
> > It is just a config channel.
> > Hence, they are just commands not a "full transport".
>
>
> For any case, a virtqueue could not be a full transport. It needs bus
> facilities to do bootstrap/DMA/notifications at least. The idea is to
> save for transport specific resources and use a more scalable way to
> config devices.
>
Hence it needs only needs a cfgq, not the full transport. :)
>
> >
> >> To me, adminq is not
> >> specific to PCI, so this proposal looks more like a partial transport
> >> for legacy.
> > It is not a partial transport for legacy.
> > A future transport may be able to do for SIOV/MMIO by adding those group
> member identifier.
> >
> > Hence, I agreed that commands in the tranportvq proposal is fine, as
> commands, not as new transport.
> >
> >> So we have
> >>
> >> 1) all commands via PCI
> >> 2) all commands via adminq
> >> 3) part of the commands via PCI and the rest via adminq
> >>
> >> This proposal goes to for 3 with transport specific commands while I
> >> think we need to try 1 and 2 then it is more general and it helps to
> >> avoid duplication with future features.
> >>
> > All legacy interface via AQ.
>
>
> That's pretty fine. So we have already had modern transport via a
> virtqueue, why not unify the design to avoid duplication? Otherwise we
> may end up with
>
> 1) legacy PCI alike commands via admin virtqueue with modern alike
> notification areas
Modern commands on SIOV device for some backward compat is needed.
A brand new SIOV device doesn’t need it.
> 2) modern device commands via admin virtqueue
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 3:45 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 3:45 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, May 15, 2023 11:28 PM
>
> Hi Parav:
>
> 在 2023/5/15 23:49, Parav Pandit 写道:
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Monday, May 15, 2023 3:11 AM
> >>> It is just bunch of admin commands in the new SIOV or virtual virtio
> >>> device
> >> chapter.
> >>> It is not a transport chapter.
> >>
> >> Provisioning could be a part of the transport. For example SR-IOV
> >> allows static and dynamic provisioning. If a virtio device needs to
> >> be implemented in VF, it needs to be provisioned first.
> >>
> > Provisioning a device by admin != transport guest driver <-> device
> communication.
> >
> > Two very different things.
>
>
> When developing a new transport, we need to consider at least how to probe
> the device. Before probe the device, it needs to be provisioned.
Yes, and yes to both. I agree.
> If we want a new transport, it could implement the provisioning especially if it
> want the design to be bus independent. For the proposal like transport
> virtqueue, it's also fine to use bus specific facility to provision and probe the
> device.
>
Provisioning is for defining the transport specific attributes.
Each transport has few command things and few unique things.
So provisioning commands can define transport specific attributes and few generic things.
This is fine.
>
> That is what I want to say.
>
>
> >
> >>>>> And the motivation is also clear is to provide composing a virtio
> >>>>> device for the
> >>>> guest VM for the backward compatibility for 1.x part of the specification.
> >>>>> It seems fine and indeed orthogonal to me that: it is for backward
> >>>> compatibility for already defined config fields for existing guest VM driver.
> >>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> >>>>> for the
> >>>> new config fields, new use cases that doesn't require any mediation.
> >>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> >>>>> without
> >>>> mediation.
> >>>>
> >>>> My understanding is that the cfgq/cmdq could be done on top, since
> >>>> the commands are part of transport unless I miss something.
> >>>>
> >>> On top of what?
> >>> cfgq/cmdq is just another queue like any other VQ.
> >>> it is orthogonal to accessing 1.x registers using group owner's queue.
> >>
> >> I think there hasn't been any proposal that call any virtqueue like
> >> cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
> >> the context here is to provide or extend transport facilities via a
> >> virtqueue not something like the control virtqueue.
> > The context is to compose a PCI device for a guest which for the currently
> defined features.
> > Once a new attribute is done using cfgq/cmdq there is no problem of
> transporting it via its group member.
> >
> > And therefore, saying any new attribute also to ride on same tranportq/aq via
> is not appropriate.
>
>
> The point is to try to find if there's a better way, more below.
>
>
> >
> >> Admin virtqueue doesn't preventing anything from letting IMS go
> >> directly to the device.
> > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it
> is prevented.
>
>
> It's a choice of the implementation but not spec. Spec doesn't say adminq can't
> be used for guest.
>
For guest and by guest are two different things.
I am proposing that a cfgq/cmdq should be used directly by the guest driver as base line like any other VQ.
And if there is a need for mediation, it should be possible.
> I think we've discussed many times that legacy is tricky for hardware.
And we want to proceed for the described use case and requirements.
> And your proposal proves this further by introducing modern alike
> notification areas.
Not really. A modern device is offering the notification area like legacy to the legacy interface.
So this is just fine.
> We probably need to deal with others like endianess.
>
No point in discussing this again. One size does not fit all.
It solves large part of the use cases so it is valuable and will be supported.
> For any case, having a simple and deterministic device design is always
> better assuming we've agreed that mediation is a must for legacy
> drivers. Using dedicated commands can make sure the implementation won't
> need to go with corner cases of legacy.
>
Corner case handling just moved from the device to hypervisor.
>
> >
> >>> Its same thing using more opcodes.
> >>
> >> Do we really care about the number of opcodes? I think we've reserved
> >> sufficient spaces.
> >>
> > I don’t see opcode as problem, we have plenty.
> > And hence, when there is a device that _does not implement config space for
> SIOV/SF, it should be introduced.
> > It is the introduction and bisection at the hypervisor level unnecessary.
> >
> > And I don’t have object to it either, the main reason is: I don’t see it being
> useful for 1.x access going forward.
>
>
> As you said for future SIOV/SF? For example, it can have better
> scalability in IMS since it doesn't require any real capabilities or
> registers.
>
For SIOV there is no backward compatibility requirement. Hence, it can start with fresh infra for real capabilities, registers and IMS.
>
> >
> >>>> 1) Modern transport, admin virtqueue with well defined transport
> >> commands
> >>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
> >>>> virtqueue
> >>>>
> >>>> Transport virtqueue tend to be transport independent, but 2) tie it
> somehow
> >> to
> >>>> the PCI. That's why I suggest to
> >>>>
> >>> The 4 patches are not transporting all the PCI things over transport VQ. it is
> >> not a transport.
> >>
> >>
> >> It really depends on the different viewpoint.
> > You wrote/asked for "PCI transport over admin q".
> > That means PCI capabilities, PCI config registers and all things defined in the
> PCI transport section to go over the AQ.
> > This is clearly not the objective of these 2 patches.
>
>
> It's not but it can achieve the same function as this series with much
> broader use cases.
>
>
> > Nor it is your proposal of transport VQ.
>
>
> Same as the above.
>
> I think we know your proposal might but it's worthwhile to figure out
> whether or not there could be other better ways to achieve the same
> thing. That's why the discussion happens on the cover letter.
>
> Both transport virtqueue and "PCI over adminq" can support legacy guest
> with a lot of other use cases.
>
PCI over admin queue just doesn’t work at performance scale to transport doorbells over a queue interface.
So, things just stop there.
>
> >
> > You only want to exchange currently defined config registers, interrupts and
> notification configuration using AQ.
> >
> > Transport VQ is NOT transporting actual data, it is not transporting
> notifications etc.
> > It is just a config channel.
> > Hence, they are just commands not a "full transport".
>
>
> For any case, a virtqueue could not be a full transport. It needs bus
> facilities to do bootstrap/DMA/notifications at least. The idea is to
> save for transport specific resources and use a more scalable way to
> config devices.
>
Hence it needs only needs a cfgq, not the full transport. :)
>
> >
> >> To me, adminq is not
> >> specific to PCI, so this proposal looks more like a partial transport
> >> for legacy.
> > It is not a partial transport for legacy.
> > A future transport may be able to do for SIOV/MMIO by adding those group
> member identifier.
> >
> > Hence, I agreed that commands in the tranportvq proposal is fine, as
> commands, not as new transport.
> >
> >> So we have
> >>
> >> 1) all commands via PCI
> >> 2) all commands via adminq
> >> 3) part of the commands via PCI and the rest via adminq
> >>
> >> This proposal goes to for 3 with transport specific commands while I
> >> think we need to try 1 and 2 then it is more general and it helps to
> >> avoid duplication with future features.
> >>
> > All legacy interface via AQ.
>
>
> That's pretty fine. So we have already had modern transport via a
> virtqueue, why not unify the design to avoid duplication? Otherwise we
> may end up with
>
> 1) legacy PCI alike commands via admin virtqueue with modern alike
> notification areas
Modern commands on SIOV device for some backward compat is needed.
A brand new SIOV device doesn’t need it.
> 2) modern device commands via admin virtqueue
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-11 13:15 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 3:54 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:54 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:15, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:05 AM
>> If we decide to use the admin vq, is there any good reason to tie it to PCI if we
>> don't want to tunneling PCI over adminq?
> To truly tunnel PCI over adminq, please tunnel _all_ pci things over AQ, not just below 1 to 11.
Just to clarify, below 1 to 11 is not tunnel PCI over adminq, it's the
idea of transport virtqueue which is not specific to PCI.
If we want to tunnel PCI over adminq, it needs be accessed via your BAR
access commands.
> Doorbells, capabilities and more. Then it is qualifying as a transport.
> Otherwise, it is not a "virtio pci transport over aq".
Yes, it's basically just have some extension on your proposal, adding a
config space access method that should be fine.
>
> I neither see why one would tunnel whole PCI over some queue, which was not your intention either from the patches I see.
> VQ over fabrics is much cleaner design to transport VQ over non-PCI.
That's a different topic, and we don't need to think of the legacy
support there.
> People have tried to tunnel pci over some other transports and it only fine for non-production toys.
I've replied this in another thread.
>
>> Why not simply invent individual commands to access legacy facilities like
>> commands to access like what transport virtqueue did for modern
>> device?:
>>
> I don’t see how this is being any different than register-offset interface.
> It bisects more things at hypervisor level that makes things hard to add #12th entry.
>
>> 1) device features
>> 2) driver features
>> 3) queue address
>> 4) queue size
>> 5) queue select
>> 6) queue notify
>> 7) device status
>> 8) ISR status
>> 9) config msix
>> 10) queue msix
>> 11) device configuration space
>>
>> It focuses on the facilities instead of transport specific details like registers (we
>> don't even need legacy registers in this case), I gives more deterministic
>> behavior so we don't need to care about the cross registers read/write.
>>
> 1.x has these registers at raw level and that seems fine.
Note that 1.x has more, the above is dumped from the section of "Legacy
Interfaces: A Note on PCI Device Layout".
Thanks
> Anything new will come on the cfgq and hence no bisection or registers needed.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 3:54 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 3:54 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
在 2023/5/11 21:15, Parav Pandit 写道:
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, May 11, 2023 3:05 AM
>> If we decide to use the admin vq, is there any good reason to tie it to PCI if we
>> don't want to tunneling PCI over adminq?
> To truly tunnel PCI over adminq, please tunnel _all_ pci things over AQ, not just below 1 to 11.
Just to clarify, below 1 to 11 is not tunnel PCI over adminq, it's the
idea of transport virtqueue which is not specific to PCI.
If we want to tunnel PCI over adminq, it needs be accessed via your BAR
access commands.
> Doorbells, capabilities and more. Then it is qualifying as a transport.
> Otherwise, it is not a "virtio pci transport over aq".
Yes, it's basically just have some extension on your proposal, adding a
config space access method that should be fine.
>
> I neither see why one would tunnel whole PCI over some queue, which was not your intention either from the patches I see.
> VQ over fabrics is much cleaner design to transport VQ over non-PCI.
That's a different topic, and we don't need to think of the legacy
support there.
> People have tried to tunnel pci over some other transports and it only fine for non-production toys.
I've replied this in another thread.
>
>> Why not simply invent individual commands to access legacy facilities like
>> commands to access like what transport virtqueue did for modern
>> device?:
>>
> I don’t see how this is being any different than register-offset interface.
> It bisects more things at hypervisor level that makes things hard to add #12th entry.
>
>> 1) device features
>> 2) driver features
>> 3) queue address
>> 4) queue size
>> 5) queue select
>> 6) queue notify
>> 7) device status
>> 8) ISR status
>> 9) config msix
>> 10) queue msix
>> 11) device configuration space
>>
>> It focuses on the facilities instead of transport specific details like registers (we
>> don't even need legacy registers in this case), I gives more deterministic
>> behavior so we don't need to care about the cross registers read/write.
>>
> 1.x has these registers at raw level and that seems fine.
Note that 1.x has more, the above is dumped from the section of "Legacy
Interfaces: A Note on PCI Device Layout".
Thanks
> Anything new will come on the cfgq and hence no bisection or registers needed.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:45 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 4:08 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 4:08 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:45 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 15, 2023 11:28 PM
> >
> > Hi Parav:
> >
> > 在 2023/5/15 23:49, Parav Pandit 写道:
> > >> From: Jason Wang <jasowang@redhat.com>
> > >> Sent: Monday, May 15, 2023 3:11 AM
> > >>> It is just bunch of admin commands in the new SIOV or virtual virtio
> > >>> device
> > >> chapter.
> > >>> It is not a transport chapter.
> > >>
> > >> Provisioning could be a part of the transport. For example SR-IOV
> > >> allows static and dynamic provisioning. If a virtio device needs to
> > >> be implemented in VF, it needs to be provisioned first.
> > >>
> > > Provisioning a device by admin != transport guest driver <-> device
> > communication.
> > >
> > > Two very different things.
> >
> >
> > When developing a new transport, we need to consider at least how to probe
> > the device. Before probe the device, it needs to be provisioned.
> Yes, and yes to both. I agree.
>
> > If we want a new transport, it could implement the provisioning especially if it
> > want the design to be bus independent. For the proposal like transport
> > virtqueue, it's also fine to use bus specific facility to provision and probe the
> > device.
> >
> Provisioning is for defining the transport specific attributes.
> Each transport has few command things and few unique things.
> So provisioning commands can define transport specific attributes and few generic things.
> This is fine.
>
> >
> > That is what I want to say.
> >
> >
> > >
> > >>>>> And the motivation is also clear is to provide composing a virtio
> > >>>>> device for the
> > >>>> guest VM for the backward compatibility for 1.x part of the specification.
> > >>>>> It seems fine and indeed orthogonal to me that: it is for backward
> > >>>> compatibility for already defined config fields for existing guest VM driver.
> > >>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> > >>>>> for the
> > >>>> new config fields, new use cases that doesn't require any mediation.
> > >>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> > >>>>> without
> > >>>> mediation.
> > >>>>
> > >>>> My understanding is that the cfgq/cmdq could be done on top, since
> > >>>> the commands are part of transport unless I miss something.
> > >>>>
> > >>> On top of what?
> > >>> cfgq/cmdq is just another queue like any other VQ.
> > >>> it is orthogonal to accessing 1.x registers using group owner's queue.
> > >>
> > >> I think there hasn't been any proposal that call any virtqueue like
> > >> cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
> > >> the context here is to provide or extend transport facilities via a
> > >> virtqueue not something like the control virtqueue.
> > > The context is to compose a PCI device for a guest which for the currently
> > defined features.
> > > Once a new attribute is done using cfgq/cmdq there is no problem of
> > transporting it via its group member.
> > >
> > > And therefore, saying any new attribute also to ride on same tranportq/aq via
> > is not appropriate.
> >
> >
> > The point is to try to find if there's a better way, more below.
> >
> >
> > >
> > >> Admin virtqueue doesn't preventing anything from letting IMS go
> > >> directly to the device.
> > > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it
> > is prevented.
> >
> >
> > It's a choice of the implementation but not spec. Spec doesn't say adminq can't
> > be used for guest.
> >
> For guest and by guest are two different things.
> I am proposing that a cfgq/cmdq should be used directly by the guest driver as base line like any other VQ.
I'm fine with this, but if I understand correctly, but I don't see any
proposal like this that is posted to the list?
> And if there is a need for mediation, it should be possible.
>
> > I think we've discussed many times that legacy is tricky for hardware.
> And we want to proceed for the described use case and requirements.
>
> > And your proposal proves this further by introducing modern alike
> > notification areas.
> Not really. A modern device is offering the notification area like legacy to the legacy interface.
> So this is just fine.
I don't say it's not fine, it's pretty fine and transport virtqueue
have similar commands. And this also answer the question that you
think tunnel PCI over adminq is not scalable. We can keep this command
in that case as well, or invent new capabilities.
>
> > We probably need to deal with others like endianess.
> >
> No point in discussing this again. One size does not fit all.
> It solves large part of the use cases so it is valuable and will be supported.
This is doable, we need to leave some space for future development. Or
is it just complicated to have an endian command?
>
> > For any case, having a simple and deterministic device design is always
> > better assuming we've agreed that mediation is a must for legacy
> > drivers. Using dedicated commands can make sure the implementation won't
> > need to go with corner cases of legacy.
> >
> Corner case handling just moved from the device to hypervisor.
That's not safe, the device should be ready for malicious inputs.
>
> >
> > >
> > >>> Its same thing using more opcodes.
> > >>
> > >> Do we really care about the number of opcodes? I think we've reserved
> > >> sufficient spaces.
> > >>
> > > I don’t see opcode as problem, we have plenty.
> > > And hence, when there is a device that _does not implement config space for
> > SIOV/SF, it should be introduced.
> > > It is the introduction and bisection at the hypervisor level unnecessary.
> > >
> > > And I don’t have object to it either, the main reason is: I don’t see it being
> > useful for 1.x access going forward.
> >
> >
> > As you said for future SIOV/SF? For example, it can have better
> > scalability in IMS since it doesn't require any real capabilities or
> > registers.
> >
> For SIOV there is no backward compatibility requirement.
It has several ways to support current virtio-pci drivers in the
guest, and such compatibility is almost a must for the public cloud.
We can't enforce the user to reinstall/recompile their
kernels/application in order to work for SIOV.
> Hence, it can start with fresh infra for real capabilities, registers and IMS.
>
> >
> > >
> > >>>> 1) Modern transport, admin virtqueue with well defined transport
> > >> commands
> > >>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
> > >>>> virtqueue
> > >>>>
> > >>>> Transport virtqueue tend to be transport independent, but 2) tie it
> > somehow
> > >> to
> > >>>> the PCI. That's why I suggest to
> > >>>>
> > >>> The 4 patches are not transporting all the PCI things over transport VQ. it is
> > >> not a transport.
> > >>
> > >>
> > >> It really depends on the different viewpoint.
> > > You wrote/asked for "PCI transport over admin q".
> > > That means PCI capabilities, PCI config registers and all things defined in the
> > PCI transport section to go over the AQ.
> > > This is clearly not the objective of these 2 patches.
> >
> >
> > It's not but it can achieve the same function as this series with much
> > broader use cases.
> >
> >
> > > Nor it is your proposal of transport VQ.
> >
> >
> > Same as the above.
> >
> > I think we know your proposal might but it's worthwhile to figure out
> > whether or not there could be other better ways to achieve the same
> > thing. That's why the discussion happens on the cover letter.
> >
>
> > Both transport virtqueue and "PCI over adminq" can support legacy guest
> > with a lot of other use cases.
> >
> PCI over admin queue just doesn’t work at performance scale to transport doorbells over a queue interface.
> So, things just stop there.
I think we are discussing what it can be, so see the above reply for
the notification areas, we can stick
virtio_admin_cmd_lq_notify_query_result command for PCI over adminq,
or invent new capabilities to make it fully self contained.
>
> >
> > >
> > > You only want to exchange currently defined config registers, interrupts and
> > notification configuration using AQ.
> > >
> > > Transport VQ is NOT transporting actual data, it is not transporting
> > notifications etc.
> > > It is just a config channel.
> > > Hence, they are just commands not a "full transport".
> >
> >
> > For any case, a virtqueue could not be a full transport. It needs bus
> > facilities to do bootstrap/DMA/notifications at least. The idea is to
> > save for transport specific resources and use a more scalable way to
> > config devices.
> >
> Hence it needs only needs a cfgq, not the full transport. :)
>
Well, not a native speaker but I'm fine if you want to call it configq.
> >
> > >
> > >> To me, adminq is not
> > >> specific to PCI, so this proposal looks more like a partial transport
> > >> for legacy.
> > > It is not a partial transport for legacy.
> > > A future transport may be able to do for SIOV/MMIO by adding those group
> > member identifier.
> > >
> > > Hence, I agreed that commands in the tranportvq proposal is fine, as
> > commands, not as new transport.
> > >
> > >> So we have
> > >>
> > >> 1) all commands via PCI
> > >> 2) all commands via adminq
> > >> 3) part of the commands via PCI and the rest via adminq
> > >>
> > >> This proposal goes to for 3 with transport specific commands while I
> > >> think we need to try 1 and 2 then it is more general and it helps to
> > >> avoid duplication with future features.
> > >>
> > > All legacy interface via AQ.
> >
> >
> > That's pretty fine. So we have already had modern transport via a
> > virtqueue, why not unify the design to avoid duplication? Otherwise we
> > may end up with
> >
> > 1) legacy PCI alike commands via admin virtqueue with modern alike
> > notification areas
> Modern commands on SIOV device for some backward compat is needed.
> A brand new SIOV device doesn’t need it.
So I meant, if we invent legacy commands for transport/admin/config
virtqueue, we don't need those PCI alike commands.
Thanks
>
> > 2) modern device commands via admin virtqueue
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 4:08 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-16 4:08 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:45 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 15, 2023 11:28 PM
> >
> > Hi Parav:
> >
> > 在 2023/5/15 23:49, Parav Pandit 写道:
> > >> From: Jason Wang <jasowang@redhat.com>
> > >> Sent: Monday, May 15, 2023 3:11 AM
> > >>> It is just bunch of admin commands in the new SIOV or virtual virtio
> > >>> device
> > >> chapter.
> > >>> It is not a transport chapter.
> > >>
> > >> Provisioning could be a part of the transport. For example SR-IOV
> > >> allows static and dynamic provisioning. If a virtio device needs to
> > >> be implemented in VF, it needs to be provisioned first.
> > >>
> > > Provisioning a device by admin != transport guest driver <-> device
> > communication.
> > >
> > > Two very different things.
> >
> >
> > When developing a new transport, we need to consider at least how to probe
> > the device. Before probe the device, it needs to be provisioned.
> Yes, and yes to both. I agree.
>
> > If we want a new transport, it could implement the provisioning especially if it
> > want the design to be bus independent. For the proposal like transport
> > virtqueue, it's also fine to use bus specific facility to provision and probe the
> > device.
> >
> Provisioning is for defining the transport specific attributes.
> Each transport has few command things and few unique things.
> So provisioning commands can define transport specific attributes and few generic things.
> This is fine.
>
> >
> > That is what I want to say.
> >
> >
> > >
> > >>>>> And the motivation is also clear is to provide composing a virtio
> > >>>>> device for the
> > >>>> guest VM for the backward compatibility for 1.x part of the specification.
> > >>>>> It seems fine and indeed orthogonal to me that: it is for backward
> > >>>> compatibility for already defined config fields for existing guest VM driver.
> > >>>>> It does not conflict with the cfgq/cmdq idea whose main purpose is
> > >>>>> for the
> > >>>> new config fields, new use cases that doesn't require any mediation.
> > >>>>> Such cfgq works across PF, VF, SF/SIOV devices in uniform way
> > >>>>> without
> > >>>> mediation.
> > >>>>
> > >>>> My understanding is that the cfgq/cmdq could be done on top, since
> > >>>> the commands are part of transport unless I miss something.
> > >>>>
> > >>> On top of what?
> > >>> cfgq/cmdq is just another queue like any other VQ.
> > >>> it is orthogonal to accessing 1.x registers using group owner's queue.
> > >>
> > >> I think there hasn't been any proposal that call any virtqueue like
> > >> cfgq/cmdq. So using that may cause a lot of misunderstanding. I think
> > >> the context here is to provide or extend transport facilities via a
> > >> virtqueue not something like the control virtqueue.
> > > The context is to compose a PCI device for a guest which for the currently
> > defined features.
> > > Once a new attribute is done using cfgq/cmdq there is no problem of
> > transporting it via its group member.
> > >
> > > And therefore, saying any new attribute also to ride on same tranportq/aq via
> > is not appropriate.
> >
> >
> > The point is to try to find if there's a better way, more below.
> >
> >
> > >
> > >> Admin virtqueue doesn't preventing anything from letting IMS go
> > >> directly to the device.
> > > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it
> > is prevented.
> >
> >
> > It's a choice of the implementation but not spec. Spec doesn't say adminq can't
> > be used for guest.
> >
> For guest and by guest are two different things.
> I am proposing that a cfgq/cmdq should be used directly by the guest driver as base line like any other VQ.
I'm fine with this, but if I understand correctly, but I don't see any
proposal like this that is posted to the list?
> And if there is a need for mediation, it should be possible.
>
> > I think we've discussed many times that legacy is tricky for hardware.
> And we want to proceed for the described use case and requirements.
>
> > And your proposal proves this further by introducing modern alike
> > notification areas.
> Not really. A modern device is offering the notification area like legacy to the legacy interface.
> So this is just fine.
I don't say it's not fine, it's pretty fine and transport virtqueue
have similar commands. And this also answer the question that you
think tunnel PCI over adminq is not scalable. We can keep this command
in that case as well, or invent new capabilities.
>
> > We probably need to deal with others like endianess.
> >
> No point in discussing this again. One size does not fit all.
> It solves large part of the use cases so it is valuable and will be supported.
This is doable, we need to leave some space for future development. Or
is it just complicated to have an endian command?
>
> > For any case, having a simple and deterministic device design is always
> > better assuming we've agreed that mediation is a must for legacy
> > drivers. Using dedicated commands can make sure the implementation won't
> > need to go with corner cases of legacy.
> >
> Corner case handling just moved from the device to hypervisor.
That's not safe, the device should be ready for malicious inputs.
>
> >
> > >
> > >>> Its same thing using more opcodes.
> > >>
> > >> Do we really care about the number of opcodes? I think we've reserved
> > >> sufficient spaces.
> > >>
> > > I don’t see opcode as problem, we have plenty.
> > > And hence, when there is a device that _does not implement config space for
> > SIOV/SF, it should be introduced.
> > > It is the introduction and bisection at the hypervisor level unnecessary.
> > >
> > > And I don’t have object to it either, the main reason is: I don’t see it being
> > useful for 1.x access going forward.
> >
> >
> > As you said for future SIOV/SF? For example, it can have better
> > scalability in IMS since it doesn't require any real capabilities or
> > registers.
> >
> For SIOV there is no backward compatibility requirement.
It has several ways to support current virtio-pci drivers in the
guest, and such compatibility is almost a must for the public cloud.
We can't enforce the user to reinstall/recompile their
kernels/application in order to work for SIOV.
> Hence, it can start with fresh infra for real capabilities, registers and IMS.
>
> >
> > >
> > >>>> 1) Modern transport, admin virtqueue with well defined transport
> > >> commands
> > >>>> 2) Legacy transport, works moe like a PCI transport on top of the admin
> > >>>> virtqueue
> > >>>>
> > >>>> Transport virtqueue tend to be transport independent, but 2) tie it
> > somehow
> > >> to
> > >>>> the PCI. That's why I suggest to
> > >>>>
> > >>> The 4 patches are not transporting all the PCI things over transport VQ. it is
> > >> not a transport.
> > >>
> > >>
> > >> It really depends on the different viewpoint.
> > > You wrote/asked for "PCI transport over admin q".
> > > That means PCI capabilities, PCI config registers and all things defined in the
> > PCI transport section to go over the AQ.
> > > This is clearly not the objective of these 2 patches.
> >
> >
> > It's not but it can achieve the same function as this series with much
> > broader use cases.
> >
> >
> > > Nor it is your proposal of transport VQ.
> >
> >
> > Same as the above.
> >
> > I think we know your proposal might but it's worthwhile to figure out
> > whether or not there could be other better ways to achieve the same
> > thing. That's why the discussion happens on the cover letter.
> >
>
> > Both transport virtqueue and "PCI over adminq" can support legacy guest
> > with a lot of other use cases.
> >
> PCI over admin queue just doesn’t work at performance scale to transport doorbells over a queue interface.
> So, things just stop there.
I think we are discussing what it can be, so see the above reply for
the notification areas, we can stick
virtio_admin_cmd_lq_notify_query_result command for PCI over adminq,
or invent new capabilities to make it fully self contained.
>
> >
> > >
> > > You only want to exchange currently defined config registers, interrupts and
> > notification configuration using AQ.
> > >
> > > Transport VQ is NOT transporting actual data, it is not transporting
> > notifications etc.
> > > It is just a config channel.
> > > Hence, they are just commands not a "full transport".
> >
> >
> > For any case, a virtqueue could not be a full transport. It needs bus
> > facilities to do bootstrap/DMA/notifications at least. The idea is to
> > save for transport specific resources and use a more scalable way to
> > config devices.
> >
> Hence it needs only needs a cfgq, not the full transport. :)
>
Well, not a native speaker but I'm fine if you want to call it configq.
> >
> > >
> > >> To me, adminq is not
> > >> specific to PCI, so this proposal looks more like a partial transport
> > >> for legacy.
> > > It is not a partial transport for legacy.
> > > A future transport may be able to do for SIOV/MMIO by adding those group
> > member identifier.
> > >
> > > Hence, I agreed that commands in the tranportvq proposal is fine, as
> > commands, not as new transport.
> > >
> > >> So we have
> > >>
> > >> 1) all commands via PCI
> > >> 2) all commands via adminq
> > >> 3) part of the commands via PCI and the rest via adminq
> > >>
> > >> This proposal goes to for 3 with transport specific commands while I
> > >> think we need to try 1 and 2 then it is more general and it helps to
> > >> avoid duplication with future features.
> > >>
> > > All legacy interface via AQ.
> >
> >
> > That's pretty fine. So we have already had modern transport via a
> > virtqueue, why not unify the design to avoid duplication? Otherwise we
> > may end up with
> >
> > 1) legacy PCI alike commands via admin virtqueue with modern alike
> > notification areas
> Modern commands on SIOV device for some backward compat is needed.
> A brand new SIOV device doesn’t need it.
So I meant, if we invent legacy commands for transport/admin/config
virtqueue, we don't need those PCI alike commands.
Thanks
>
> > 2) modern device commands via admin virtqueue
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:28 ` [virtio-comment] " Jason Wang
@ 2023-05-16 4:18 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 4:18 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:28:04AM +0800, Jason Wang wrote:
> >
> > > Admin virtqueue doesn't preventing anything from letting IMS go directly to the
> > > device.
> > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
>
>
> It's a choice of the implementation but not spec. Spec doesn't say adminq
> can't be used for guest.
It actually kind of does. adminq controls group members through the
group owner. owner is not itself a member. For a guest controlling a
member to use adminq you have to also pass the owner to guest.
Possible but not a standard architecture.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 4:18 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 4:18 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:28:04AM +0800, Jason Wang wrote:
> >
> > > Admin virtqueue doesn't preventing anything from letting IMS go directly to the
> > > device.
> > But there is no aq/cfgq/cmdq defined for the guest, so with current proposal it is prevented.
>
>
> It's a choice of the implementation but not spec. Spec doesn't say adminq
> can't be used for guest.
It actually kind of does. adminq controls group members through the
group owner. owner is not itself a member. For a guest controlling a
member to use adminq you have to also pass the owner to guest.
Possible but not a standard architecture.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 20:56 ` Parav Pandit
@ 2023-05-16 4:32 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 4:32 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 08:56:42PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 4:30 PM
> > >
> > > I am not sure if this is a real issue. Because even the legacy guests
> > > have msix enabled by default. In theory yes, it can fall back to intx.
> >
> > Well. I feel we should be closer to being sure it's not an issue if we are going to
> > ignore it.
> > some actual data here:
> >
> > Even linux only enabled MSI-X in 2009.
> > Of course, other guests took longer. E.g.
> > a quick google search gave me this for some bsd variant (2017):
> > https://twitter.com/dragonflybsd/status/834494984229421057
> >
> > Many guests have tunables to disable msix. Why?
> > E.g. BSD keeps maintaining it at
> > hw.virtio.pci.disable_msix
> > not a real use-case and you know 100% no guests have set this to work around
> > some bug e.g. in bsd MSI-X core? How can you be sure?
> >
> >
> >
> > intx is used when guests run out of legacy interrupts, these setups are not hard
> > to create at all: just constrain the number of vCPUs while creating lots of
> > devices.
> >
> >
> > I could go on.
> >
> >
> >
> > > There are few options.
> > > 1. A hypervisor driver can be conservative and steal an msix of the VF
> > > for transporting intx.
> > > Pros: Does not need special things in device
> > > Cons:
> > > a. Fairly intrusive in hypervisor vf driver.
> > > b. May not be ever used as guest is unlikely to fail on msix
> >
> > Yea I do not like this since we are burning up msix vectors.
> > More reasons: this "pass through" msix has no chance to set ISR properly since
> > msix does not set ISR.
> >
> >
> > > 2. Since multiple VFs intx to be serviced, one command per VF in AQ is
> > > too much overhead that device needs to map a request to,
> > >
> > > A better way is to have an eventq of depth = num_vfs, like many other
> > > virtio devices have it.
> > >
> > > An eventq can hold per VF interrupt entry including the isr value that
> > > you suggest above.
> > >
> > > Something like,
> > >
> > > union eventq_entry {
> > > u8 raw_data[16];
> > > struct intx_entry {
> > > u8 event_opcode;
> > > u8 group_type;
> > > u8 reserved[6];
> > > le64 group_identifier;
> > > u8 isr_status;
> > > };
> > > };
> > >
> > > This eventq resides on the owner parent PF.
> > > isr_status is read on clear like today.
> >
> > This is what I wrote no?
> > lore.kernel.org/all/20230507050146-mutt-send-email-
> > mst%40kernel.org/t.mbox.gz
> >
> > how about a special command that is used when device would
> > normally send INT#x? it can also return ISR to reduce latency.
> >
> In response to your above suggestion of AQ command,
> I suggested the eventq that contains the isr_status that reduces latency as you suggest.
I don't see why we need to keep adding queues though.
Just use one of admin queues.
> > > May be such eventq can be useful in future for wider case.
> >
> > There's no maybe here is there? Things like live migration need events for sure.
> >
> > > We may have to find a different name for it as other devices has
> > > device specific eventq.
> >
> > We don't need a special name for it. Just use an adminq with a special
> > command that is only consumed when there is an event.
> This requires too many commands to be issued on the PF device.
> Potentially one per VF. And device needs to keep track of command to VF mapping.
>
> > Note you only need to queue a command if MSI is disabled.
> > Which is nice.
> Yes, it is nice.
> An eventq is a variation of it, where device can keep reporting the events without doing the extra mapping and without too many commands.
I don't get the difference then. The format you showed seems very close
to admin command. What is the difference? How do you avoid the need
to add a command per VF using INTx#?
> Additionally, eventq also works for 1.x device which will read the ISR status registers directly from the device.
>
> >
> > > I am inclined to differ this to a later point if one can identify the
> > > real failure with msix for the guest VM.
> > > So far we don't see this ever happening.
> >
> > What is the question exactly?
> >
> > Just have more devices than vectors,
> > an intel CPU only has ~200 of these, and current drivers want to use 2 vectors
> > and then fall back on INTx since that is shared.
> > Extremely easy to create - do you want a qemu command line to try?
> >
> Intel CPU has 256 per core (per vcpu). So they are really a lot.
> One needs to connect lot more devices to the cpu to run out of it.
> So yes, I would like to try the command to fail.
order of 128 functions then for a 1vcpu VM. You were previously talking
about tends of 1000s of functions as justification for avoiding config
space.
> > Do specific customers event use guests with msi-x disabled? Maybe no.
> > Does anyone use virtio with msi-x disabled? Most likely yes.
> I just feel that INTx emulation is extremely rare/narrow case of some applications that may never find its use on hw based devices.
If we use a dedicated command for this, I guess devices can just
avoid implementing the command if they do not feel like it?
> > So if we are going for legacy pci emulation let's have a comprehensive legacy
> > pci emulation please where host can either enable it for a guest or deny
> > completely, not kind of start running then fail mysteriously.
> A driver will be easily able to fail the call on INTx configuration failing the guest.
There's no configuration - INTx is the default - and no way to fail gracefully
for legacy. That is one of the things we should fix, at least hypervisor
should be able to detect failures.
> But lets see if can align to eventq/aq scheme to make it work.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 4:32 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 4:32 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 08:56:42PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 4:30 PM
> > >
> > > I am not sure if this is a real issue. Because even the legacy guests
> > > have msix enabled by default. In theory yes, it can fall back to intx.
> >
> > Well. I feel we should be closer to being sure it's not an issue if we are going to
> > ignore it.
> > some actual data here:
> >
> > Even linux only enabled MSI-X in 2009.
> > Of course, other guests took longer. E.g.
> > a quick google search gave me this for some bsd variant (2017):
> > https://twitter.com/dragonflybsd/status/834494984229421057
> >
> > Many guests have tunables to disable msix. Why?
> > E.g. BSD keeps maintaining it at
> > hw.virtio.pci.disable_msix
> > not a real use-case and you know 100% no guests have set this to work around
> > some bug e.g. in bsd MSI-X core? How can you be sure?
> >
> >
> >
> > intx is used when guests run out of legacy interrupts, these setups are not hard
> > to create at all: just constrain the number of vCPUs while creating lots of
> > devices.
> >
> >
> > I could go on.
> >
> >
> >
> > > There are few options.
> > > 1. A hypervisor driver can be conservative and steal an msix of the VF
> > > for transporting intx.
> > > Pros: Does not need special things in device
> > > Cons:
> > > a. Fairly intrusive in hypervisor vf driver.
> > > b. May not be ever used as guest is unlikely to fail on msix
> >
> > Yea I do not like this since we are burning up msix vectors.
> > More reasons: this "pass through" msix has no chance to set ISR properly since
> > msix does not set ISR.
> >
> >
> > > 2. Since multiple VFs intx to be serviced, one command per VF in AQ is
> > > too much overhead that device needs to map a request to,
> > >
> > > A better way is to have an eventq of depth = num_vfs, like many other
> > > virtio devices have it.
> > >
> > > An eventq can hold per VF interrupt entry including the isr value that
> > > you suggest above.
> > >
> > > Something like,
> > >
> > > union eventq_entry {
> > > u8 raw_data[16];
> > > struct intx_entry {
> > > u8 event_opcode;
> > > u8 group_type;
> > > u8 reserved[6];
> > > le64 group_identifier;
> > > u8 isr_status;
> > > };
> > > };
> > >
> > > This eventq resides on the owner parent PF.
> > > isr_status is read on clear like today.
> >
> > This is what I wrote no?
> > lore.kernel.org/all/20230507050146-mutt-send-email-
> > mst%40kernel.org/t.mbox.gz
> >
> > how about a special command that is used when device would
> > normally send INT#x? it can also return ISR to reduce latency.
> >
> In response to your above suggestion of AQ command,
> I suggested the eventq that contains the isr_status that reduces latency as you suggest.
I don't see why we need to keep adding queues though.
Just use one of admin queues.
> > > May be such eventq can be useful in future for wider case.
> >
> > There's no maybe here is there? Things like live migration need events for sure.
> >
> > > We may have to find a different name for it as other devices has
> > > device specific eventq.
> >
> > We don't need a special name for it. Just use an adminq with a special
> > command that is only consumed when there is an event.
> This requires too many commands to be issued on the PF device.
> Potentially one per VF. And device needs to keep track of command to VF mapping.
>
> > Note you only need to queue a command if MSI is disabled.
> > Which is nice.
> Yes, it is nice.
> An eventq is a variation of it, where device can keep reporting the events without doing the extra mapping and without too many commands.
I don't get the difference then. The format you showed seems very close
to admin command. What is the difference? How do you avoid the need
to add a command per VF using INTx#?
> Additionally, eventq also works for 1.x device which will read the ISR status registers directly from the device.
>
> >
> > > I am inclined to differ this to a later point if one can identify the
> > > real failure with msix for the guest VM.
> > > So far we don't see this ever happening.
> >
> > What is the question exactly?
> >
> > Just have more devices than vectors,
> > an intel CPU only has ~200 of these, and current drivers want to use 2 vectors
> > and then fall back on INTx since that is shared.
> > Extremely easy to create - do you want a qemu command line to try?
> >
> Intel CPU has 256 per core (per vcpu). So they are really a lot.
> One needs to connect lot more devices to the cpu to run out of it.
> So yes, I would like to try the command to fail.
order of 128 functions then for a 1vcpu VM. You were previously talking
about tends of 1000s of functions as justification for avoiding config
space.
> > Do specific customers event use guests with msi-x disabled? Maybe no.
> > Does anyone use virtio with msi-x disabled? Most likely yes.
> I just feel that INTx emulation is extremely rare/narrow case of some applications that may never find its use on hw based devices.
If we use a dedicated command for this, I guess devices can just
avoid implementing the command if they do not feel like it?
> > So if we are going for legacy pci emulation let's have a comprehensive legacy
> > pci emulation please where host can either enable it for a guest or deny
> > completely, not kind of start running then fail mysteriously.
> A driver will be easily able to fail the call on INTx configuration failing the guest.
There's no configuration - INTx is the default - and no way to fail gracefully
for legacy. That is one of the things we should fix, at least hypervisor
should be able to detect failures.
> But lets see if can align to eventq/aq scheme to make it work.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:43 ` [virtio-comment] " Jason Wang
@ 2023-05-16 5:38 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 5:38 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:43:09AM +0800, Jason Wang wrote:
> On Tue, May 16, 2023 at 11:37 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> >
> > 在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> > > On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> > >>> From: Michael S. Tsirkin <mst@redhat.com>
> > >>> Sent: Monday, May 15, 2023 1:56 PM
> > >>>
> > >>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> > >>>>> From: Michael S. Tsirkin <mst@redhat.com>
> > >>>>> Sent: Monday, May 15, 2023 1:45 PM
> > >>>>>
> > >>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > >>>>>> All legacy interface via AQ.
> > >>>>>> All modern interface access via PCI or its own transport between
> > >>>>>> driver and
> > >>>>> device.
> > >>>>>
> > >>>>> I am wondering however about the hypervisor notifications.
> > >>>>> Generally these are the most problematic aspect here I feel. For
> > >>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > >>>>> generally, having both guest and host be allowed to access device's BAR
> > >>> seems problematic.
> > >>>>> Can we reserve a PF BAR region for these things or is that too expensive?
> > >>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> > >>> it is not but it just might be required, or it won't be there.
> > >>>
> > >> How can it be required if it not part of it?
> > >> I likely didn't follow your question/comment, can you please explain.
> > > VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> > >
> > >>>> For modern device, guest driver will access the device own BAR directly with
> > >>> its own config_data anyway.
> > >>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> > >>> report which BAR/area to use for the given SF/SIOV device?
> > >>>> May be yes, those are different discussions with tradeoff to consider during
> > >>> SIOV discussions. It is not related to VFs.
> > >>>
> > >>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> > >> Each VF has its own BAR area for driver notifications.
> > > But it is passed through to guest presumably.
> >
> >
> > Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
> > problem if we do mediation here except for some performance penalty?
>
> Ok, rethink of this, when using legacy, VIRTIO_F_NOTIF_CONFIG_DATA
> should be assumed to be not negotiated. So I think I agree with Parav,
> it should not be a problem.
>
> Or do you mean for the devices that have some mandated features?
>
> Thanks
Exactly. VIRTIO_F_NOTIF_CONFIG_DATA if there is likely to be
required for device to work as opposed to being an optimization.
> >
> > Thanks
> >
> >
> > >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 5:38 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 5:38 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 11:43:09AM +0800, Jason Wang wrote:
> On Tue, May 16, 2023 at 11:37 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> >
> > 在 2023/5/16 02:01, Michael S. Tsirkin 写道:
> > > On Mon, May 15, 2023 at 06:00:02PM +0000, Parav Pandit wrote:
> > >>> From: Michael S. Tsirkin <mst@redhat.com>
> > >>> Sent: Monday, May 15, 2023 1:56 PM
> > >>>
> > >>> On Mon, May 15, 2023 at 05:51:05PM +0000, Parav Pandit wrote:
> > >>>>> From: Michael S. Tsirkin <mst@redhat.com>
> > >>>>> Sent: Monday, May 15, 2023 1:45 PM
> > >>>>>
> > >>>>> On Mon, May 15, 2023 at 03:49:44PM +0000, Parav Pandit wrote:
> > >>>>>> All legacy interface via AQ.
> > >>>>>> All modern interface access via PCI or its own transport between
> > >>>>>> driver and
> > >>>>> device.
> > >>>>>
> > >>>>> I am wondering however about the hypervisor notifications.
> > >>>>> Generally these are the most problematic aspect here I feel. For
> > >>>>> example, how does it interact with VIRTIO_F_NOTIF_CONFIG_DATA? And
> > >>>>> generally, having both guest and host be allowed to access device's BAR
> > >>> seems problematic.
> > >>>>> Can we reserve a PF BAR region for these things or is that too expensive?
> > >>>> VIRTIO_F_NOTIF_CONFIG_DATA is not present for the legacy.
> > >>> it is not but it just might be required, or it won't be there.
> > >>>
> > >> How can it be required if it not part of it?
> > >> I likely didn't follow your question/comment, can you please explain.
> > > VIRTIO_F_NOTIF_CONFIG_DATA is there presumably for a reason.
> > >
> > >>>> For modern device, guest driver will access the device own BAR directly with
> > >>> its own config_data anyway.
> > >>>> Should we reserve a region in the PF BAR for SF/SIOV device?, should device
> > >>> report which BAR/area to use for the given SF/SIOV device?
> > >>>> May be yes, those are different discussions with tradeoff to consider during
> > >>> SIOV discussions. It is not related to VFs.
> > >>>
> > >>> For SIOV it's a given. But we can do this for SRIOV: reserve a PF region per VF.
> > >> Each VF has its own BAR area for driver notifications.
> > > But it is passed through to guest presumably.
> >
> >
> > Probably not in the case of VIRTIO_F_NOTIF_CONFIG_DATA. Did you see any
> > problem if we do mediation here except for some performance penalty?
>
> Ok, rethink of this, when using legacy, VIRTIO_F_NOTIF_CONFIG_DATA
> should be assumed to be not negotiated. So I think I agree with Parav,
> it should not be a problem.
>
> Or do you mean for the devices that have some mandated features?
>
> Thanks
Exactly. VIRTIO_F_NOTIF_CONFIG_DATA if there is likely to be
required for device to work as opposed to being an optimization.
> >
> > Thanks
> >
> >
> > >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 15:59 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 6:21 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 6:21 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 15, 2023 3:31 AM
>
> > > What do we gain from bisecting it?
> >
> >
> > 1) No need to deal with the offset in the hardware
> >
> In context of legacy is doesn’t matter much given that its over AQ.
Let me give you an example. I feel that device specific
config should allow arbitrary length accesses so that e.g.
mac can be read and written atomically.
On the other hand, this is not the case for common config.
I feel we do not need to allow e.g. access to both common config
and device specific one in a single operation, that is
just messy. Now sure, we can add text with an explanation,
but if instead there's a flag saying "this is common cfg"
or "this is device cfg" then it fall out automatically.
Is this a deal breaker? No, but note how neither I nor Jason did not
think about this ahead of the time and the correct usage just falls out
of the interface automatically. This is a mark of a good interface
design. Lots of corner cases that one has to be careful around is not.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 6:21 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 6:21 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Monday, May 15, 2023 3:31 AM
>
> > > What do we gain from bisecting it?
> >
> >
> > 1) No need to deal with the offset in the hardware
> >
> In context of legacy is doesn’t matter much given that its over AQ.
Let me give you an example. I feel that device specific
config should allow arbitrary length accesses so that e.g.
mac can be read and written atomically.
On the other hand, this is not the case for common config.
I feel we do not need to allow e.g. access to both common config
and device specific one in a single operation, that is
just messy. Now sure, we can add text with an explanation,
but if instead there's a flag saying "this is common cfg"
or "this is device cfg" then it fall out automatically.
Is this a deal breaker? No, but note how neither I nor Jason did not
think about this ahead of the time and the correct usage just falls out
of the interface automatically. This is a mark of a good interface
design. Lots of corner cases that one has to be careful around is not.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 4:32 ` Michael S. Tsirkin
@ 2023-05-16 18:45 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 18:45 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 12:33 AM
[..]
> > > > A better way is to have an eventq of depth = num_vfs, like many
> > > > other virtio devices have it.
> > > >
> > > > An eventq can hold per VF interrupt entry including the isr value
> > > > that you suggest above.
> > > >
> > > > Something like,
> > > >
> > > > union eventq_entry {
> > > > u8 raw_data[16];
> > > > struct intx_entry {
> > > > u8 event_opcode;
> > > > u8 group_type;
> > > > u8 reserved[6];
> > > > le64 group_identifier;
> > > > u8 isr_status;
> > > > };
> > > > };
> > > >
> > > > This eventq resides on the owner parent PF.
> > > > isr_status is read on clear like today.
> > >
> > > This is what I wrote no?
> > > lore.kernel.org/all/20230507050146-mutt-send-email-
> > > mst%40kernel.org/t.mbox.gz
> > >
> > > how about a special command that is used when device would
> > > normally send INT#x? it can also return ISR to reduce latency.
> > >
> > In response to your above suggestion of AQ command, I suggested the
> > eventq that contains the isr_status that reduces latency as you suggest.
>
> I don't see why we need to keep adding queues though.
> Just use one of admin queues.
>
Admin queue is cmd->rsp mode.
The purpose is different here. A device wants to async, notify the interrupt for each VF.
Like a nic rq.
Filling up the AQ full of these commands is sub-optimal for those VFs who may generate interrupt.
> > An eventq is a variation of it, where device can keep reporting the events
> without doing the extra mapping and without too many commands.
>
> I don't get the difference then. The format you showed seems very close to
> admin command. What is the difference? How do you avoid the need to add a
> command per VF using INTx#?
>
The main difference is it behaves like nic rq.
Driver posts empty descriptors, device reports the consumed entries.
Specific entry is not attached to any specific VF.
The eventq_entry structure is written by the device for a VF there fired the interrupt fired.
> How do you avoid the need to add a command per VF using INTx#?
The empty buffer is posted pointing to eventq_entry.
Total number of buffers posted = number_of_vfs in intx mode.
So entry is not attached to a specific VF, entries are in a pool (like nic rq).
> > Intel CPU has 256 per core (per vcpu). So they are really a lot.
> > One needs to connect lot more devices to the cpu to run out of it.
> > So yes, I would like to try the command to fail.
>
> order of 128 functions then for a 1vcpu VM. You were previously talking about
> tends of 1000s of functions as justification for avoiding config space.
>
Will try.
>
> > > Do specific customers event use guests with msi-x disabled? Maybe no.
> > > Does anyone use virtio with msi-x disabled? Most likely yes.
> > I just feel that INTx emulation is extremely rare/narrow case of some
> applications that may never find its use on hw based devices.
>
> If we use a dedicated command for this, I guess devices can just avoid
> implementing the command if they do not feel like it?
>
I didn't follow. The guest device?
Do you mean guest driver?
> > A driver will be easily able to fail the call on INTx configuration failing the
> guest.
>
> There's no configuration - INTx is the default - and no way to fail gracefully for
> legacy. That is one of the things we should fix, at least hypervisor should be able
> to detect failures.
For sure hypervisor can detect the failure on INTx configuration done by the guest and fail the call for this config.
What we are discussing is not anything legacy specific, it applies to 1.x as well sadly.
So, we don't need to mix INTx support with register emulation proposal, better to handle in the follow up proposal itself.
I also have some patches to improve driver resiliency when guest is not involved to work without INTx.
And INTx can still work from system and device point of view, because INTx generation does not have function id in it.
Existing PCIe INTx can work if PCI PF device can trigger INTx for the VFs.
So eventq scheme is still optional and orthogonal to legacy.
I would like to discuss it separately outside of legacy proposal.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 18:45 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 18:45 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 12:33 AM
[..]
> > > > A better way is to have an eventq of depth = num_vfs, like many
> > > > other virtio devices have it.
> > > >
> > > > An eventq can hold per VF interrupt entry including the isr value
> > > > that you suggest above.
> > > >
> > > > Something like,
> > > >
> > > > union eventq_entry {
> > > > u8 raw_data[16];
> > > > struct intx_entry {
> > > > u8 event_opcode;
> > > > u8 group_type;
> > > > u8 reserved[6];
> > > > le64 group_identifier;
> > > > u8 isr_status;
> > > > };
> > > > };
> > > >
> > > > This eventq resides on the owner parent PF.
> > > > isr_status is read on clear like today.
> > >
> > > This is what I wrote no?
> > > lore.kernel.org/all/20230507050146-mutt-send-email-
> > > mst%40kernel.org/t.mbox.gz
> > >
> > > how about a special command that is used when device would
> > > normally send INT#x? it can also return ISR to reduce latency.
> > >
> > In response to your above suggestion of AQ command, I suggested the
> > eventq that contains the isr_status that reduces latency as you suggest.
>
> I don't see why we need to keep adding queues though.
> Just use one of admin queues.
>
Admin queue is cmd->rsp mode.
The purpose is different here. A device wants to async, notify the interrupt for each VF.
Like a nic rq.
Filling up the AQ full of these commands is sub-optimal for those VFs who may generate interrupt.
> > An eventq is a variation of it, where device can keep reporting the events
> without doing the extra mapping and without too many commands.
>
> I don't get the difference then. The format you showed seems very close to
> admin command. What is the difference? How do you avoid the need to add a
> command per VF using INTx#?
>
The main difference is it behaves like nic rq.
Driver posts empty descriptors, device reports the consumed entries.
Specific entry is not attached to any specific VF.
The eventq_entry structure is written by the device for a VF there fired the interrupt fired.
> How do you avoid the need to add a command per VF using INTx#?
The empty buffer is posted pointing to eventq_entry.
Total number of buffers posted = number_of_vfs in intx mode.
So entry is not attached to a specific VF, entries are in a pool (like nic rq).
> > Intel CPU has 256 per core (per vcpu). So they are really a lot.
> > One needs to connect lot more devices to the cpu to run out of it.
> > So yes, I would like to try the command to fail.
>
> order of 128 functions then for a 1vcpu VM. You were previously talking about
> tends of 1000s of functions as justification for avoiding config space.
>
Will try.
>
> > > Do specific customers event use guests with msi-x disabled? Maybe no.
> > > Does anyone use virtio with msi-x disabled? Most likely yes.
> > I just feel that INTx emulation is extremely rare/narrow case of some
> applications that may never find its use on hw based devices.
>
> If we use a dedicated command for this, I guess devices can just avoid
> implementing the command if they do not feel like it?
>
I didn't follow. The guest device?
Do you mean guest driver?
> > A driver will be easily able to fail the call on INTx configuration failing the
> guest.
>
> There's no configuration - INTx is the default - and no way to fail gracefully for
> legacy. That is one of the things we should fix, at least hypervisor should be able
> to detect failures.
For sure hypervisor can detect the failure on INTx configuration done by the guest and fail the call for this config.
What we are discussing is not anything legacy specific, it applies to 1.x as well sadly.
So, we don't need to mix INTx support with register emulation proposal, better to handle in the follow up proposal itself.
I also have some patches to improve driver resiliency when guest is not involved to work without INTx.
And INTx can still work from system and device point of view, because INTx generation does not have function id in it.
Existing PCIe INTx can work if PCI PF device can trigger INTx for the VFs.
So eventq scheme is still optional and orthogonal to legacy.
I would like to discuss it separately outside of legacy proposal.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 6:21 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-16 19:11 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:11 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 2:22 AM
>
> On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Monday, May 15, 2023 3:31 AM
> >
> > > > What do we gain from bisecting it?
> > >
> > >
> > > 1) No need to deal with the offset in the hardware
> > >
> > In context of legacy is doesn’t matter much given that its over AQ.
>
> Let me give you an example. I feel that device specific config should allow
> arbitrary length accesses so that e.g.
> mac can be read and written atomically.
>
> On the other hand, this is not the case for common config.
>
Why is this not the case with common config?
Spec snippet: "When using the legacy interface the driver MAY access the device-specific configuration region using any
width accesses"
> I feel we do not need to allow e.g. access to both common config and device
> specific one in a single operation, that is just messy.
It is just an offset and value.
What part bothers you?
> Now sure, we can add text
> with an explanation, but if instead there's a flag saying "this is common cfg"
> or "this is device cfg" then it fall out automatically.
>
> Is this a deal breaker? No, but note how neither I nor Jason did not think about
> this ahead of the time and the correct usage just falls out of the interface
> automatically. This is a mark of a good interface design. Lots of corner cases
> that one has to be careful around is not.
It just moves corner case one layer above in multiple software hypervisors and hypervisor is not failing it either on cfg read/writes as expected by the guest.
So bisecting is not adding anything extra. Those failure checks are going to be ignored in hypervisor anyway.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 19:11 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:11 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 2:22 AM
>
> On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
> >
> > > From: Jason Wang <jasowang@redhat.com>
> > > Sent: Monday, May 15, 2023 3:31 AM
> >
> > > > What do we gain from bisecting it?
> > >
> > >
> > > 1) No need to deal with the offset in the hardware
> > >
> > In context of legacy is doesn’t matter much given that its over AQ.
>
> Let me give you an example. I feel that device specific config should allow
> arbitrary length accesses so that e.g.
> mac can be read and written atomically.
>
> On the other hand, this is not the case for common config.
>
Why is this not the case with common config?
Spec snippet: "When using the legacy interface the driver MAY access the device-specific configuration region using any
width accesses"
> I feel we do not need to allow e.g. access to both common config and device
> specific one in a single operation, that is just messy.
It is just an offset and value.
What part bothers you?
> Now sure, we can add text
> with an explanation, but if instead there's a flag saying "this is common cfg"
> or "this is device cfg" then it fall out automatically.
>
> Is this a deal breaker? No, but note how neither I nor Jason did not think about
> this ahead of the time and the correct usage just falls out of the interface
> automatically. This is a mark of a good interface design. Lots of corner cases
> that one has to be careful around is not.
It just moves corner case one layer above in multiple software hypervisors and hypervisor is not failing it either on cfg read/writes as expected by the guest.
So bisecting is not adding anything extra. Those failure checks are going to be ignored in hypervisor anyway.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 4:08 ` [virtio-comment] " Jason Wang
@ 2023-05-16 19:29 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:29 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Tuesday, May 16, 2023 12:08 AM
> > For guest and by guest are two different things.
> > I am proposing that a cfgq/cmdq should be used directly by the guest driver as
> base line like any other VQ.
>
> I'm fine with this, but if I understand correctly, but I don't see any proposal like
> this that is posted to the list?
>
Because it is orthogonal to the work here, it's hard to make progress by mixing everything.
> > And if there is a need for mediation, it should be possible.
> >
> > > I think we've discussed many times that legacy is tricky for hardware.
> > And we want to proceed for the described use case and requirements.
> >
> > > And your proposal proves this further by introducing modern alike
> > > notification areas.
> > Not really. A modern device is offering the notification area like legacy to the
> legacy interface.
> > So this is just fine.
>
> I don't say it's not fine, it's pretty fine and transport virtqueue have similar
> commands. And this also answer the question that you think tunnel PCI over
> adminq is not scalable. We can keep this command in that case as well, or
> invent new capabilities.
>
Ok. so virtio_admin_q_cmd_lq_notify_result is fine.
> >
> > > We probably need to deal with others like endianess.
> > >
> > No point in discussing this again. One size does not fit all.
> > It solves large part of the use cases so it is valuable and will be supported.
>
> This is doable, we need to leave some space for future development. Or is it
> just complicated to have an endian command?
>
Possibly one can set. I don’t know if any actual device really supported endianness.
No users have asked for it, even asking explicitly to those non_little_endian users.
So may be when there is/if a real user, it can be done in future.
> >
> > > For any case, having a simple and deterministic device design is
> > > always better assuming we've agreed that mediation is a must for
> > > legacy drivers. Using dedicated commands can make sure the
> > > implementation won't need to go with corner cases of legacy.
> > >
> > Corner case handling just moved from the device to hypervisor.
>
> That's not safe, the device should be ready for malicious inputs.
>
With current offset, register, device will be just fine as most implementations have to program control path etc on these registers write/read etc.
So, device will be able to handle them with plain 2 commands.
> > For SIOV there is no backward compatibility requirement.
>
> It has several ways to support current virtio-pci drivers in the guest, and such
> compatibility is almost a must for the public cloud.
> We can't enforce the user to reinstall/recompile their kernels/application in
> order to work for SIOV.
Sure, an SIOV device emulating PCI device for guest is one requirement.
And SIOV device working natively spawned from the parent PF and used without guest is another requirement.
And second can be done bit elegantly in self-contained way without depending on the parent PF.
And requires more work unrelated to this one.
> I think we are discussing what it can be, so see the above reply for the
> notification areas, we can stick virtio_admin_cmd_lq_notify_query_result
> command for PCI over adminq, or invent new capabilities to make it fully self
> contained.
>
As we both understand well that a queue is not going to transport a doorbell and actual IMS interrupts, we will mostly have commands to service some needs for backward compat.
It is a separate work when there are no PCI VF devices.
> >
> > >
> > > >
> > > > You only want to exchange currently defined config registers,
> > > > interrupts and
> > > notification configuration using AQ.
> > > >
> > > > Transport VQ is NOT transporting actual data, it is not
> > > > transporting
> > > notifications etc.
> > > > It is just a config channel.
> > > > Hence, they are just commands not a "full transport".
> > >
> > >
> > > For any case, a virtqueue could not be a full transport. It needs
> > > bus facilities to do bootstrap/DMA/notifications at least. The idea
> > > is to save for transport specific resources and use a more scalable
> > > way to config devices.
> > >
> > Hence it needs only needs a cfgq, not the full transport. :)
> >
>
> Well, not a native speaker but I'm fine if you want to call it configq.
Michael called it admin vq, so will stick to it for now. :)
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 19:29 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:29 UTC (permalink / raw)
To: Jason Wang
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Tuesday, May 16, 2023 12:08 AM
> > For guest and by guest are two different things.
> > I am proposing that a cfgq/cmdq should be used directly by the guest driver as
> base line like any other VQ.
>
> I'm fine with this, but if I understand correctly, but I don't see any proposal like
> this that is posted to the list?
>
Because it is orthogonal to the work here, it's hard to make progress by mixing everything.
> > And if there is a need for mediation, it should be possible.
> >
> > > I think we've discussed many times that legacy is tricky for hardware.
> > And we want to proceed for the described use case and requirements.
> >
> > > And your proposal proves this further by introducing modern alike
> > > notification areas.
> > Not really. A modern device is offering the notification area like legacy to the
> legacy interface.
> > So this is just fine.
>
> I don't say it's not fine, it's pretty fine and transport virtqueue have similar
> commands. And this also answer the question that you think tunnel PCI over
> adminq is not scalable. We can keep this command in that case as well, or
> invent new capabilities.
>
Ok. so virtio_admin_q_cmd_lq_notify_result is fine.
> >
> > > We probably need to deal with others like endianess.
> > >
> > No point in discussing this again. One size does not fit all.
> > It solves large part of the use cases so it is valuable and will be supported.
>
> This is doable, we need to leave some space for future development. Or is it
> just complicated to have an endian command?
>
Possibly one can set. I don’t know if any actual device really supported endianness.
No users have asked for it, even asking explicitly to those non_little_endian users.
So may be when there is/if a real user, it can be done in future.
> >
> > > For any case, having a simple and deterministic device design is
> > > always better assuming we've agreed that mediation is a must for
> > > legacy drivers. Using dedicated commands can make sure the
> > > implementation won't need to go with corner cases of legacy.
> > >
> > Corner case handling just moved from the device to hypervisor.
>
> That's not safe, the device should be ready for malicious inputs.
>
With current offset, register, device will be just fine as most implementations have to program control path etc on these registers write/read etc.
So, device will be able to handle them with plain 2 commands.
> > For SIOV there is no backward compatibility requirement.
>
> It has several ways to support current virtio-pci drivers in the guest, and such
> compatibility is almost a must for the public cloud.
> We can't enforce the user to reinstall/recompile their kernels/application in
> order to work for SIOV.
Sure, an SIOV device emulating PCI device for guest is one requirement.
And SIOV device working natively spawned from the parent PF and used without guest is another requirement.
And second can be done bit elegantly in self-contained way without depending on the parent PF.
And requires more work unrelated to this one.
> I think we are discussing what it can be, so see the above reply for the
> notification areas, we can stick virtio_admin_cmd_lq_notify_query_result
> command for PCI over adminq, or invent new capabilities to make it fully self
> contained.
>
As we both understand well that a queue is not going to transport a doorbell and actual IMS interrupts, we will mostly have commands to service some needs for backward compat.
It is a separate work when there are no PCI VF devices.
> >
> > >
> > > >
> > > > You only want to exchange currently defined config registers,
> > > > interrupts and
> > > notification configuration using AQ.
> > > >
> > > > Transport VQ is NOT transporting actual data, it is not
> > > > transporting
> > > notifications etc.
> > > > It is just a config channel.
> > > > Hence, they are just commands not a "full transport".
> > >
> > >
> > > For any case, a virtqueue could not be a full transport. It needs
> > > bus facilities to do bootstrap/DMA/notifications at least. The idea
> > > is to save for transport specific resources and use a more scalable
> > > way to config devices.
> > >
> > Hence it needs only needs a cfgq, not the full transport. :)
> >
>
> Well, not a native speaker but I'm fine if you want to call it configq.
Michael called it admin vq, so will stick to it for now. :)
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 3:54 ` [virtio-comment] " Jason Wang
@ 2023-05-16 19:35 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:35 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Jason Wang
> Sent: Monday, May 15, 2023 11:55 PM
> > I don’t see how this is being any different than register-offset interface.
> > It bisects more things at hypervisor level that makes things hard to add #12th
> entry.
> >
> >> 1) device features
> >> 2) driver features
> >> 3) queue address
> >> 4) queue size
> >> 5) queue select
> >> 6) queue notify
> >> 7) device status
> >> 8) ISR status
> >> 9) config msix
> >> 10) queue msix
> >> 11) device configuration space
> >>
> >> It focuses on the facilities instead of transport specific details like registers
> (we
> >> don't even need legacy registers in this case), I gives more deterministic
> >> behavior so we don't need to care about the cross registers read/write.
> >>
> > 1.x has these registers at raw level and that seems fine.
>
>
> Note that 1.x has more, the above is dumped from the section of "Legacy
> Interfaces: A Note on PCI Device Layout".
Yeah.
Current two commands proposal has one box to transport legacy registers that follows the legacy semantics.
Above commands in future with legacy commands of this work will be able to work together in future anyway.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 19:35 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 19:35 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Jason Wang
> Sent: Monday, May 15, 2023 11:55 PM
> > I don’t see how this is being any different than register-offset interface.
> > It bisects more things at hypervisor level that makes things hard to add #12th
> entry.
> >
> >> 1) device features
> >> 2) driver features
> >> 3) queue address
> >> 4) queue size
> >> 5) queue select
> >> 6) queue notify
> >> 7) device status
> >> 8) ISR status
> >> 9) config msix
> >> 10) queue msix
> >> 11) device configuration space
> >>
> >> It focuses on the facilities instead of transport specific details like registers
> (we
> >> don't even need legacy registers in this case), I gives more deterministic
> >> behavior so we don't need to care about the cross registers read/write.
> >>
> > 1.x has these registers at raw level and that seems fine.
>
>
> Note that 1.x has more, the above is dumped from the section of "Legacy
> Interfaces: A Note on PCI Device Layout".
Yeah.
Current two commands proposal has one box to transport legacy registers that follows the legacy semantics.
Above commands in future with legacy commands of this work will be able to work together in future anyway.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 18:45 ` Parav Pandit
@ 2023-05-16 20:42 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 20:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 06:45:58PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 12:33 AM
>
> [..]
> > > > > A better way is to have an eventq of depth = num_vfs, like many
> > > > > other virtio devices have it.
> > > > >
> > > > > An eventq can hold per VF interrupt entry including the isr value
> > > > > that you suggest above.
> > > > >
> > > > > Something like,
> > > > >
> > > > > union eventq_entry {
> > > > > u8 raw_data[16];
> > > > > struct intx_entry {
> > > > > u8 event_opcode;
> > > > > u8 group_type;
> > > > > u8 reserved[6];
> > > > > le64 group_identifier;
> > > > > u8 isr_status;
> > > > > };
> > > > > };
> > > > >
> > > > > This eventq resides on the owner parent PF.
> > > > > isr_status is read on clear like today.
> > > >
> > > > This is what I wrote no?
> > > > lore.kernel.org/all/20230507050146-mutt-send-email-
> > > > mst%40kernel.org/t.mbox.gz
> > > >
> > > > how about a special command that is used when device would
> > > > normally send INT#x? it can also return ISR to reduce latency.
> > > >
> > > In response to your above suggestion of AQ command, I suggested the
> > > eventq that contains the isr_status that reduces latency as you suggest.
> >
> > I don't see why we need to keep adding queues though.
> > Just use one of admin queues.
> >
> Admin queue is cmd->rsp mode.
> The purpose is different here. A device wants to async, notify the interrupt for each VF.
> Like a nic rq.
> Filling up the AQ full of these commands is sub-optimal for those VFs who may generate interrupt.
I don't get the difference sorry. what you do with rq
is exactly fill it up with buffers. here you fill adminq with
commands.
> > > An eventq is a variation of it, where device can keep reporting the events
> > without doing the extra mapping and without too many commands.
> >
> > I don't get the difference then. The format you showed seems very close to
> > admin command. What is the difference? How do you avoid the need to add a
> > command per VF using INTx#?
> >
> The main difference is it behaves like nic rq.
> Driver posts empty descriptors, device reports the consumed entries.
> Specific entry is not attached to any specific VF.
> The eventq_entry structure is written by the device for a VF there fired the interrupt fired.
If this is what you want, we already have commands that do not refer to
a specific VF, specifically VIRTIO_ADMIN_CMD_LIST_QUERY.
But I think this is misguided see below.
> > How do you avoid the need to add a command per VF using INTx#?
> The empty buffer is posted pointing to eventq_entry.
> Total number of buffers posted = number_of_vfs in intx mode.
> So entry is not attached to a specific VF, entries are in a pool (like nic rq).
There is a very simple problem with your approach:
you need to report to VF read of ISR since that
clears the interrupt line.
This command has to be per VF.
As I tried to explain before, I proposed combining the two:
same command clears ISR for a VF and if ISR is not 0
returns immediately. If it is 0 the buffer is queued and
used later for interrupt. I prefer the approach
where the buffer is reused by the same VF,
relying on out of order use.
But yes, if you want use in-order, this does not work.
Write the VF that triggered interrupt in command specific
data maybe?
> > > Intel CPU has 256 per core (per vcpu). So they are really a lot.
> > > One needs to connect lot more devices to the cpu to run out of it.
> > > So yes, I would like to try the command to fail.
> >
> > order of 128 functions then for a 1vcpu VM. You were previously talking about
> > tends of 1000s of functions as justification for avoiding config space.
> >
> Will try.
>
> >
> > > > Do specific customers event use guests with msi-x disabled? Maybe no.
> > > > Does anyone use virtio with msi-x disabled? Most likely yes.
> > > I just feel that INTx emulation is extremely rare/narrow case of some
> > applications that may never find its use on hw based devices.
> >
> > If we use a dedicated command for this, I guess devices can just avoid
> > implementing the command if they do not feel like it?
> >
> I didn't follow. The guest device?
> Do you mean guest driver?
Device. Clear the bit in VIRTIO_ADMIN_CMD_LIST_QUERY and
then hypervisor knows it can't support INTx.
> > > A driver will be easily able to fail the call on INTx configuration failing the
> > guest.
> >
> > There's no configuration - INTx is the default - and no way to fail gracefully for
> > legacy. That is one of the things we should fix, at least hypervisor should be able
> > to detect failures.
> For sure hypervisor can detect the failure on INTx configuration done by the guest and fail the call for this config.
> What we are discussing is not anything legacy specific, it applies to 1.x as well sadly.
Problem is, enabling MSIX changes layout for legacy.
This is my concern really we need to make the decision whether the
reserved msi vector hack is used or eventq will work. Because
reserved msi vector hack changes legacy layout.
Fundamentally, that only matters because you insist on
reusing the same command for common and device config space
access with difference in the offset. Split them out as Jason proposed
and the issue goes away.
> So, we don't need to mix INTx support with register emulation proposal, better to handle in the follow up proposal itself.
> I also have some patches to improve driver resiliency when guest is not involved to work without INTx.
>
> And INTx can still work from system and device point of view, because INTx generation does not have function id in it.
> Existing PCIe INTx can work if PCI PF device can trigger INTx for the VFs.
> So eventq scheme is still optional and orthogonal to legacy.
>
> I would like to discuss it separately outside of legacy proposal.
Hmm this sounds convincing. I think if we split device and common
config then we can defer INTx issue.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 20:42 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 20:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 06:45:58PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 12:33 AM
>
> [..]
> > > > > A better way is to have an eventq of depth = num_vfs, like many
> > > > > other virtio devices have it.
> > > > >
> > > > > An eventq can hold per VF interrupt entry including the isr value
> > > > > that you suggest above.
> > > > >
> > > > > Something like,
> > > > >
> > > > > union eventq_entry {
> > > > > u8 raw_data[16];
> > > > > struct intx_entry {
> > > > > u8 event_opcode;
> > > > > u8 group_type;
> > > > > u8 reserved[6];
> > > > > le64 group_identifier;
> > > > > u8 isr_status;
> > > > > };
> > > > > };
> > > > >
> > > > > This eventq resides on the owner parent PF.
> > > > > isr_status is read on clear like today.
> > > >
> > > > This is what I wrote no?
> > > > lore.kernel.org/all/20230507050146-mutt-send-email-
> > > > mst%40kernel.org/t.mbox.gz
> > > >
> > > > how about a special command that is used when device would
> > > > normally send INT#x? it can also return ISR to reduce latency.
> > > >
> > > In response to your above suggestion of AQ command, I suggested the
> > > eventq that contains the isr_status that reduces latency as you suggest.
> >
> > I don't see why we need to keep adding queues though.
> > Just use one of admin queues.
> >
> Admin queue is cmd->rsp mode.
> The purpose is different here. A device wants to async, notify the interrupt for each VF.
> Like a nic rq.
> Filling up the AQ full of these commands is sub-optimal for those VFs who may generate interrupt.
I don't get the difference sorry. what you do with rq
is exactly fill it up with buffers. here you fill adminq with
commands.
> > > An eventq is a variation of it, where device can keep reporting the events
> > without doing the extra mapping and without too many commands.
> >
> > I don't get the difference then. The format you showed seems very close to
> > admin command. What is the difference? How do you avoid the need to add a
> > command per VF using INTx#?
> >
> The main difference is it behaves like nic rq.
> Driver posts empty descriptors, device reports the consumed entries.
> Specific entry is not attached to any specific VF.
> The eventq_entry structure is written by the device for a VF there fired the interrupt fired.
If this is what you want, we already have commands that do not refer to
a specific VF, specifically VIRTIO_ADMIN_CMD_LIST_QUERY.
But I think this is misguided see below.
> > How do you avoid the need to add a command per VF using INTx#?
> The empty buffer is posted pointing to eventq_entry.
> Total number of buffers posted = number_of_vfs in intx mode.
> So entry is not attached to a specific VF, entries are in a pool (like nic rq).
There is a very simple problem with your approach:
you need to report to VF read of ISR since that
clears the interrupt line.
This command has to be per VF.
As I tried to explain before, I proposed combining the two:
same command clears ISR for a VF and if ISR is not 0
returns immediately. If it is 0 the buffer is queued and
used later for interrupt. I prefer the approach
where the buffer is reused by the same VF,
relying on out of order use.
But yes, if you want use in-order, this does not work.
Write the VF that triggered interrupt in command specific
data maybe?
> > > Intel CPU has 256 per core (per vcpu). So they are really a lot.
> > > One needs to connect lot more devices to the cpu to run out of it.
> > > So yes, I would like to try the command to fail.
> >
> > order of 128 functions then for a 1vcpu VM. You were previously talking about
> > tends of 1000s of functions as justification for avoiding config space.
> >
> Will try.
>
> >
> > > > Do specific customers event use guests with msi-x disabled? Maybe no.
> > > > Does anyone use virtio with msi-x disabled? Most likely yes.
> > > I just feel that INTx emulation is extremely rare/narrow case of some
> > applications that may never find its use on hw based devices.
> >
> > If we use a dedicated command for this, I guess devices can just avoid
> > implementing the command if they do not feel like it?
> >
> I didn't follow. The guest device?
> Do you mean guest driver?
Device. Clear the bit in VIRTIO_ADMIN_CMD_LIST_QUERY and
then hypervisor knows it can't support INTx.
> > > A driver will be easily able to fail the call on INTx configuration failing the
> > guest.
> >
> > There's no configuration - INTx is the default - and no way to fail gracefully for
> > legacy. That is one of the things we should fix, at least hypervisor should be able
> > to detect failures.
> For sure hypervisor can detect the failure on INTx configuration done by the guest and fail the call for this config.
> What we are discussing is not anything legacy specific, it applies to 1.x as well sadly.
Problem is, enabling MSIX changes layout for legacy.
This is my concern really we need to make the decision whether the
reserved msi vector hack is used or eventq will work. Because
reserved msi vector hack changes legacy layout.
Fundamentally, that only matters because you insist on
reusing the same command for common and device config space
access with difference in the offset. Split them out as Jason proposed
and the issue goes away.
> So, we don't need to mix INTx support with register emulation proposal, better to handle in the follow up proposal itself.
> I also have some patches to improve driver resiliency when guest is not involved to work without INTx.
>
> And INTx can still work from system and device point of view, because INTx generation does not have function id in it.
> Existing PCIe INTx can work if PCI PF device can trigger INTx for the VFs.
> So eventq scheme is still optional and orthogonal to legacy.
>
> I would like to discuss it separately outside of legacy proposal.
Hmm this sounds convincing. I think if we split device and common
config then we can defer INTx issue.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 19:11 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 20:58 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 20:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:11:36PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 2:22 AM
> >
> > On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Monday, May 15, 2023 3:31 AM
> > >
> > > > > What do we gain from bisecting it?
> > > >
> > > >
> > > > 1) No need to deal with the offset in the hardware
> > > >
> > > In context of legacy is doesn’t matter much given that its over AQ.
> >
> > Let me give you an example. I feel that device specific config should allow
> > arbitrary length accesses so that e.g.
> > mac can be read and written atomically.
> >
> > On the other hand, this is not the case for common config.
> >
> Why is this not the case with common config?
> Spec snippet: "When using the legacy interface the driver MAY access the device-specific configuration region using any
> width accesses"
I mean, what you said above exactly?
device-specific configuration excludes common config.
> > I feel we do not need to allow e.g. access to both common config and device
> > specific one in a single operation, that is just messy.
> It is just an offset and value.
> What part bothers you?
E.g. that it can cross the coundary between common and device config.
> > Now sure, we can add text
> > with an explanation, but if instead there's a flag saying "this is common cfg"
> > or "this is device cfg" then it fall out automatically.
> >
> > Is this a deal breaker? No, but note how neither I nor Jason did not think about
> > this ahead of the time and the correct usage just falls out of the interface
> > automatically. This is a mark of a good interface design. Lots of corner cases
> > that one has to be careful around is not.
> It just moves corner case one layer above in multiple software hypervisors and hypervisor is not failing it either on cfg read/writes as expected by the guest.
> So bisecting is not adding anything extra. Those failure checks are going to be ignored in hypervisor anyway.
Look we didn't build modern because we wanted to make work, we did
because legacy is broken. So either let legacy die already or let's
build a sane interface to emulate it please.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 20:58 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 20:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:11:36PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 2:22 AM
> >
> > On Mon, May 15, 2023 at 03:59:41PM +0000, Parav Pandit wrote:
> > >
> > > > From: Jason Wang <jasowang@redhat.com>
> > > > Sent: Monday, May 15, 2023 3:31 AM
> > >
> > > > > What do we gain from bisecting it?
> > > >
> > > >
> > > > 1) No need to deal with the offset in the hardware
> > > >
> > > In context of legacy is doesn’t matter much given that its over AQ.
> >
> > Let me give you an example. I feel that device specific config should allow
> > arbitrary length accesses so that e.g.
> > mac can be read and written atomically.
> >
> > On the other hand, this is not the case for common config.
> >
> Why is this not the case with common config?
> Spec snippet: "When using the legacy interface the driver MAY access the device-specific configuration region using any
> width accesses"
I mean, what you said above exactly?
device-specific configuration excludes common config.
> > I feel we do not need to allow e.g. access to both common config and device
> > specific one in a single operation, that is just messy.
> It is just an offset and value.
> What part bothers you?
E.g. that it can cross the coundary between common and device config.
> > Now sure, we can add text
> > with an explanation, but if instead there's a flag saying "this is common cfg"
> > or "this is device cfg" then it fall out automatically.
> >
> > Is this a deal breaker? No, but note how neither I nor Jason did not think about
> > this ahead of the time and the correct usage just falls out of the interface
> > automatically. This is a mark of a good interface design. Lots of corner cases
> > that one has to be careful around is not.
> It just moves corner case one layer above in multiple software hypervisors and hypervisor is not failing it either on cfg read/writes as expected by the guest.
> So bisecting is not adding anything extra. Those failure checks are going to be ignored in hypervisor anyway.
Look we didn't build modern because we wanted to make work, we did
because legacy is broken. So either let legacy die already or let's
build a sane interface to emulate it please.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 19:29 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 21:09 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:09 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:29:44PM +0000, Parav Pandit wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, May 16, 2023 12:08 AM
>
> > > For guest and by guest are two different things.
> > > I am proposing that a cfgq/cmdq should be used directly by the guest driver as
> > base line like any other VQ.
> >
> > I'm fine with this, but if I understand correctly, but I don't see any proposal like
> > this that is posted to the list?
> >
> Because it is orthogonal to the work here, it's hard to make progress by mixing everything.
>
> > > And if there is a need for mediation, it should be possible.
> > >
> > > > I think we've discussed many times that legacy is tricky for hardware.
> > > And we want to proceed for the described use case and requirements.
> > >
> > > > And your proposal proves this further by introducing modern alike
> > > > notification areas.
> > > Not really. A modern device is offering the notification area like legacy to the
> > legacy interface.
> > > So this is just fine.
> >
> > I don't say it's not fine, it's pretty fine and transport virtqueue have similar
> > commands. And this also answer the question that you think tunnel PCI over
> > adminq is not scalable. We can keep this command in that case as well, or
> > invent new capabilities.
> >
> Ok. so virtio_admin_q_cmd_lq_notify_result is fine.
>
> > >
> > > > We probably need to deal with others like endianess.
> > > >
> > > No point in discussing this again. One size does not fit all.
> > > It solves large part of the use cases so it is valuable and will be supported.
> >
> > This is doable, we need to leave some space for future development. Or is it
> > just complicated to have an endian command?
> >
> Possibly one can set. I don’t know if any actual device really supported endianness.
> No users have asked for it, even asking explicitly to those non_little_endian users.
For sure, ARM BE does exist and Red Hat supports it because yes,
it was requested. You can not just go by what marketing tells you
today we either try to build future proof interfaces or we don't
bother at all - by the time these things are in the field everything
shifted.
> So may be when there is/if a real user, it can be done in future.
The concern is if you assume LE is default, no way to say
"I do not support LE". Something like an explicit
"SET LE" and "SET BE" commands will automatically allow
discovering which endian-ness is supported.
> > >
> > > > For any case, having a simple and deterministic device design is
> > > > always better assuming we've agreed that mediation is a must for
> > > > legacy drivers. Using dedicated commands can make sure the
> > > > implementation won't need to go with corner cases of legacy.
> > > >
> > > Corner case handling just moved from the device to hypervisor.
> >
> > That's not safe, the device should be ready for malicious inputs.
> >
> With current offset, register, device will be just fine as most implementations have to program control path etc on these registers write/read etc.
> So, device will be able to handle them with plain 2 commands.
Except legacy is broken broken broken. It just does not work for
physical devices in a crazy number of ways. How about the weird 48 bit
limitation on PAs? Because no one ever will need any more.
I have 0 sympathy to this idea of reviving all these bugs then circling
back and coming up with weird work arounds. Please, just build a sane
transport and have software deal with bugs as best it can.
> > > For SIOV there is no backward compatibility requirement.
> >
> > It has several ways to support current virtio-pci drivers in the guest, and such
> > compatibility is almost a must for the public cloud.
> > We can't enforce the user to reinstall/recompile their kernels/application in
> > order to work for SIOV.
> Sure, an SIOV device emulating PCI device for guest is one requirement.
>
> And SIOV device working natively spawned from the parent PF and used without guest is another requirement.
>
> And second can be done bit elegantly in self-contained way without depending on the parent PF.
> And requires more work unrelated to this one.
>
> > I think we are discussing what it can be, so see the above reply for the
> > notification areas, we can stick virtio_admin_cmd_lq_notify_query_result
> > command for PCI over adminq, or invent new capabilities to make it fully self
> > contained.
> >
> As we both understand well that a queue is not going to transport a doorbell and actual IMS interrupts, we will mostly have commands to service some needs for backward compat.
> It is a separate work when there are no PCI VF devices.
>
> > >
> > > >
> > > > >
> > > > > You only want to exchange currently defined config registers,
> > > > > interrupts and
> > > > notification configuration using AQ.
> > > > >
> > > > > Transport VQ is NOT transporting actual data, it is not
> > > > > transporting
> > > > notifications etc.
> > > > > It is just a config channel.
> > > > > Hence, they are just commands not a "full transport".
> > > >
> > > >
> > > > For any case, a virtqueue could not be a full transport. It needs
> > > > bus facilities to do bootstrap/DMA/notifications at least. The idea
> > > > is to save for transport specific resources and use a more scalable
> > > > way to config devices.
> > > >
> > > Hence it needs only needs a cfgq, not the full transport. :)
> > >
> >
> > Well, not a native speaker but I'm fine if you want to call it configq.
> Michael called it admin vq, so will stick to it for now. :)
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:09 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:09 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:29:44PM +0000, Parav Pandit wrote:
>
> > From: Jason Wang <jasowang@redhat.com>
> > Sent: Tuesday, May 16, 2023 12:08 AM
>
> > > For guest and by guest are two different things.
> > > I am proposing that a cfgq/cmdq should be used directly by the guest driver as
> > base line like any other VQ.
> >
> > I'm fine with this, but if I understand correctly, but I don't see any proposal like
> > this that is posted to the list?
> >
> Because it is orthogonal to the work here, it's hard to make progress by mixing everything.
>
> > > And if there is a need for mediation, it should be possible.
> > >
> > > > I think we've discussed many times that legacy is tricky for hardware.
> > > And we want to proceed for the described use case and requirements.
> > >
> > > > And your proposal proves this further by introducing modern alike
> > > > notification areas.
> > > Not really. A modern device is offering the notification area like legacy to the
> > legacy interface.
> > > So this is just fine.
> >
> > I don't say it's not fine, it's pretty fine and transport virtqueue have similar
> > commands. And this also answer the question that you think tunnel PCI over
> > adminq is not scalable. We can keep this command in that case as well, or
> > invent new capabilities.
> >
> Ok. so virtio_admin_q_cmd_lq_notify_result is fine.
>
> > >
> > > > We probably need to deal with others like endianess.
> > > >
> > > No point in discussing this again. One size does not fit all.
> > > It solves large part of the use cases so it is valuable and will be supported.
> >
> > This is doable, we need to leave some space for future development. Or is it
> > just complicated to have an endian command?
> >
> Possibly one can set. I don’t know if any actual device really supported endianness.
> No users have asked for it, even asking explicitly to those non_little_endian users.
For sure, ARM BE does exist and Red Hat supports it because yes,
it was requested. You can not just go by what marketing tells you
today we either try to build future proof interfaces or we don't
bother at all - by the time these things are in the field everything
shifted.
> So may be when there is/if a real user, it can be done in future.
The concern is if you assume LE is default, no way to say
"I do not support LE". Something like an explicit
"SET LE" and "SET BE" commands will automatically allow
discovering which endian-ness is supported.
> > >
> > > > For any case, having a simple and deterministic device design is
> > > > always better assuming we've agreed that mediation is a must for
> > > > legacy drivers. Using dedicated commands can make sure the
> > > > implementation won't need to go with corner cases of legacy.
> > > >
> > > Corner case handling just moved from the device to hypervisor.
> >
> > That's not safe, the device should be ready for malicious inputs.
> >
> With current offset, register, device will be just fine as most implementations have to program control path etc on these registers write/read etc.
> So, device will be able to handle them with plain 2 commands.
Except legacy is broken broken broken. It just does not work for
physical devices in a crazy number of ways. How about the weird 48 bit
limitation on PAs? Because no one ever will need any more.
I have 0 sympathy to this idea of reviving all these bugs then circling
back and coming up with weird work arounds. Please, just build a sane
transport and have software deal with bugs as best it can.
> > > For SIOV there is no backward compatibility requirement.
> >
> > It has several ways to support current virtio-pci drivers in the guest, and such
> > compatibility is almost a must for the public cloud.
> > We can't enforce the user to reinstall/recompile their kernels/application in
> > order to work for SIOV.
> Sure, an SIOV device emulating PCI device for guest is one requirement.
>
> And SIOV device working natively spawned from the parent PF and used without guest is another requirement.
>
> And second can be done bit elegantly in self-contained way without depending on the parent PF.
> And requires more work unrelated to this one.
>
> > I think we are discussing what it can be, so see the above reply for the
> > notification areas, we can stick virtio_admin_cmd_lq_notify_query_result
> > command for PCI over adminq, or invent new capabilities to make it fully self
> > contained.
> >
> As we both understand well that a queue is not going to transport a doorbell and actual IMS interrupts, we will mostly have commands to service some needs for backward compat.
> It is a separate work when there are no PCI VF devices.
>
> > >
> > > >
> > > > >
> > > > > You only want to exchange currently defined config registers,
> > > > > interrupts and
> > > > notification configuration using AQ.
> > > > >
> > > > > Transport VQ is NOT transporting actual data, it is not
> > > > > transporting
> > > > notifications etc.
> > > > > It is just a config channel.
> > > > > Hence, they are just commands not a "full transport".
> > > >
> > > >
> > > > For any case, a virtqueue could not be a full transport. It needs
> > > > bus facilities to do bootstrap/DMA/notifications at least. The idea
> > > > is to save for transport specific resources and use a more scalable
> > > > way to config devices.
> > > >
> > > Hence it needs only needs a cfgq, not the full transport. :)
> > >
> >
> > Well, not a native speaker but I'm fine if you want to call it configq.
> Michael called it admin vq, so will stick to it for now. :)
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 19:35 ` [virtio-dev] " Parav Pandit
@ 2023-05-16 21:11 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:11 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:35:20PM +0000, Parav Pandit wrote:
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Jason Wang
> > Sent: Monday, May 15, 2023 11:55 PM
> > > I don’t see how this is being any different than register-offset interface.
>
> > > It bisects more things at hypervisor level that makes things hard to add #12th
> > entry.
> > >
> > >> 1) device features
> > >> 2) driver features
> > >> 3) queue address
> > >> 4) queue size
> > >> 5) queue select
> > >> 6) queue notify
> > >> 7) device status
> > >> 8) ISR status
> > >> 9) config msix
> > >> 10) queue msix
> > >> 11) device configuration space
> > >>
> > >> It focuses on the facilities instead of transport specific details like registers
> > (we
> > >> don't even need legacy registers in this case), I gives more deterministic
> > >> behavior so we don't need to care about the cross registers read/write.
> > >>
> > > 1.x has these registers at raw level and that seems fine.
> >
> >
> > Note that 1.x has more, the above is dumped from the section of "Legacy
> > Interfaces: A Note on PCI Device Layout".
>
> Yeah.
> Current two commands proposal has one box to transport legacy registers that follows the legacy semantics.
> Above commands in future with legacy commands of this work will be able to work together in future anyway.
Nah, we don't need a "break randomly unless it's a full moon and you
cross your heart three times" mode. If you are going to implement
support for legacy emulation implement it in a way that either
predictably works or predictably refuses to load.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:11 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:11 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 07:35:20PM +0000, Parav Pandit wrote:
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Jason Wang
> > Sent: Monday, May 15, 2023 11:55 PM
> > > I don’t see how this is being any different than register-offset interface.
>
> > > It bisects more things at hypervisor level that makes things hard to add #12th
> > entry.
> > >
> > >> 1) device features
> > >> 2) driver features
> > >> 3) queue address
> > >> 4) queue size
> > >> 5) queue select
> > >> 6) queue notify
> > >> 7) device status
> > >> 8) ISR status
> > >> 9) config msix
> > >> 10) queue msix
> > >> 11) device configuration space
> > >>
> > >> It focuses on the facilities instead of transport specific details like registers
> > (we
> > >> don't even need legacy registers in this case), I gives more deterministic
> > >> behavior so we don't need to care about the cross registers read/write.
> > >>
> > > 1.x has these registers at raw level and that seems fine.
> >
> >
> > Note that 1.x has more, the above is dumped from the section of "Legacy
> > Interfaces: A Note on PCI Device Layout".
>
> Yeah.
> Current two commands proposal has one box to transport legacy registers that follows the legacy semantics.
> Above commands in future with legacy commands of this work will be able to work together in future anyway.
Nah, we don't need a "break randomly unless it's a full moon and you
cross your heart three times" mode. If you are going to implement
support for legacy emulation implement it in a way that either
predictably works or predictably refuses to load.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 20:58 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-16 21:19 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:19 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 4:58 PM
> > > Let me give you an example. I feel that device specific config
> > > should allow arbitrary length accesses so that e.g.
> > > mac can be read and written atomically.
> > >
> > > On the other hand, this is not the case for common config.
> > >
> > Why is this not the case with common config?
> > Spec snippet: "When using the legacy interface the driver MAY access
> > the device-specific configuration region using any width accesses"
>
> I mean, what you said above exactly?
> device-specific configuration excludes common config.
>
>
So, what prevents the device to fail AQ command with the error code when alignment/width/size is not right?
> > > I feel we do not need to allow e.g. access to both common config and
> > > device specific one in a single operation, that is just messy.
> > It is just an offset and value.
> > What part bothers you?
>
> E.g. that it can cross the coundary between common and device config.
>
This is the legacy transport.
> Look we didn't build modern because we wanted to make work, we did because
> legacy is broken. So either let legacy die already or let's build a sane interface
> to emulate it please.
Almost everyone building virtio device would prefer for legacy to die.
I am still trying to understand that why bisecting register definition of legacy makes it sane.
Device behaves based on the register offset, is fine for the legacy transport, guest driver will make the register accesses anyway it prefers.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:19 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:19 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 4:58 PM
> > > Let me give you an example. I feel that device specific config
> > > should allow arbitrary length accesses so that e.g.
> > > mac can be read and written atomically.
> > >
> > > On the other hand, this is not the case for common config.
> > >
> > Why is this not the case with common config?
> > Spec snippet: "When using the legacy interface the driver MAY access
> > the device-specific configuration region using any width accesses"
>
> I mean, what you said above exactly?
> device-specific configuration excludes common config.
>
>
So, what prevents the device to fail AQ command with the error code when alignment/width/size is not right?
> > > I feel we do not need to allow e.g. access to both common config and
> > > device specific one in a single operation, that is just messy.
> > It is just an offset and value.
> > What part bothers you?
>
> E.g. that it can cross the coundary between common and device config.
>
This is the legacy transport.
> Look we didn't build modern because we wanted to make work, we did because
> legacy is broken. So either let legacy die already or let's build a sane interface
> to emulate it please.
Almost everyone building virtio device would prefer for legacy to die.
I am still trying to understand that why bisecting register definition of legacy makes it sane.
Device behaves based on the register offset, is fine for the legacy transport, guest driver will make the register accesses anyway it prefers.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:19 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 21:23 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:23 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:19:02PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 4:58 PM
>
> > > > Let me give you an example. I feel that device specific config
> > > > should allow arbitrary length accesses so that e.g.
> > > > mac can be read and written atomically.
> > > >
> > > > On the other hand, this is not the case for common config.
> > > >
> > > Why is this not the case with common config?
> > > Spec snippet: "When using the legacy interface the driver MAY access
> > > the device-specific configuration region using any width accesses"
> >
> > I mean, what you said above exactly?
> > device-specific configuration excludes common config.
> >
> >
> So, what prevents the device to fail AQ command with the error code when alignment/width/size is not right?
Nothing but you then need to specify what is and what isn't right.
> > > > I feel we do not need to allow e.g. access to both common config and
> > > > device specific one in a single operation, that is just messy.
> > > It is just an offset and value.
> > > What part bothers you?
> >
> > E.g. that it can cross the coundary between common and device config.
> >
> This is the legacy transport.
>
> > Look we didn't build modern because we wanted to make work, we did because
> > legacy is broken. So either let legacy die already or let's build a sane interface
> > to emulate it please.
> Almost everyone building virtio device would prefer for legacy to die.
> I am still trying to understand that why bisecting register definition of legacy makes it sane.
Because legacy common config is very evil, do not copy it.
Legacy device config is only bad in minor tolerable ways.
> Device behaves based on the register offset, is fine for the legacy transport, guest driver will make the register accesses anyway it prefers.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:23 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:23 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:19:02PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 4:58 PM
>
> > > > Let me give you an example. I feel that device specific config
> > > > should allow arbitrary length accesses so that e.g.
> > > > mac can be read and written atomically.
> > > >
> > > > On the other hand, this is not the case for common config.
> > > >
> > > Why is this not the case with common config?
> > > Spec snippet: "When using the legacy interface the driver MAY access
> > > the device-specific configuration region using any width accesses"
> >
> > I mean, what you said above exactly?
> > device-specific configuration excludes common config.
> >
> >
> So, what prevents the device to fail AQ command with the error code when alignment/width/size is not right?
Nothing but you then need to specify what is and what isn't right.
> > > > I feel we do not need to allow e.g. access to both common config and
> > > > device specific one in a single operation, that is just messy.
> > > It is just an offset and value.
> > > What part bothers you?
> >
> > E.g. that it can cross the coundary between common and device config.
> >
> This is the legacy transport.
>
> > Look we didn't build modern because we wanted to make work, we did because
> > legacy is broken. So either let legacy die already or let's build a sane interface
> > to emulate it please.
> Almost everyone building virtio device would prefer for legacy to die.
> I am still trying to understand that why bisecting register definition of legacy makes it sane.
Because legacy common config is very evil, do not copy it.
Legacy device config is only bad in minor tolerable ways.
> Device behaves based on the register offset, is fine for the legacy transport, guest driver will make the register accesses anyway it prefers.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:23 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-16 21:30 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:30 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:23 PM
>
> On Tue, May 16, 2023 at 09:19:02PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, May 16, 2023 4:58 PM
> >
> > > > > Let me give you an example. I feel that device specific config
> > > > > should allow arbitrary length accesses so that e.g.
> > > > > mac can be read and written atomically.
> > > > >
> > > > > On the other hand, this is not the case for common config.
> > > > >
> > > > Why is this not the case with common config?
> > > > Spec snippet: "When using the legacy interface the driver MAY
> > > > access the device-specific configuration region using any width accesses"
> > >
> > > I mean, what you said above exactly?
> > > device-specific configuration excludes common config.
> > >
> > >
> > So, what prevents the device to fail AQ command with the error code when
> alignment/width/size is not right?
>
> Nothing but you then need to specify what is and what isn't right.
>
It follows the existing legacy interface section, you asked to not duplicate it.
>
> > > > > I feel we do not need to allow e.g. access to both common config
> > > > > and device specific one in a single operation, that is just messy.
> > > > It is just an offset and value.
> > > > What part bothers you?
> > >
> > > E.g. that it can cross the coundary between common and device config.
> > >
> > This is the legacy transport.
> >
> > > Look we didn't build modern because we wanted to make work, we did
> > > because legacy is broken. So either let legacy die already or let's
> > > build a sane interface to emulate it please.
> > Almost everyone building virtio device would prefer for legacy to die.
> > I am still trying to understand that why bisecting register definition of legacy
> makes it sane.
>
> Because legacy common config is very evil, do not copy it.
> Legacy device config is only bad in minor tolerable ways.
>
It is not used outside of the legacy interface.
The point is guest driver will follow certain config write/read pattern.
Hypervisor has no way to fail it.
So what do you expect hypervisor to do on undesired sequence?
Do you propose to correct it and send it to device?
And if so, the intention is building sane device for legacy interface? If that is the intention, sound fine.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:30 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:30 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:23 PM
>
> On Tue, May 16, 2023 at 09:19:02PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, May 16, 2023 4:58 PM
> >
> > > > > Let me give you an example. I feel that device specific config
> > > > > should allow arbitrary length accesses so that e.g.
> > > > > mac can be read and written atomically.
> > > > >
> > > > > On the other hand, this is not the case for common config.
> > > > >
> > > > Why is this not the case with common config?
> > > > Spec snippet: "When using the legacy interface the driver MAY
> > > > access the device-specific configuration region using any width accesses"
> > >
> > > I mean, what you said above exactly?
> > > device-specific configuration excludes common config.
> > >
> > >
> > So, what prevents the device to fail AQ command with the error code when
> alignment/width/size is not right?
>
> Nothing but you then need to specify what is and what isn't right.
>
It follows the existing legacy interface section, you asked to not duplicate it.
>
> > > > > I feel we do not need to allow e.g. access to both common config
> > > > > and device specific one in a single operation, that is just messy.
> > > > It is just an offset and value.
> > > > What part bothers you?
> > >
> > > E.g. that it can cross the coundary between common and device config.
> > >
> > This is the legacy transport.
> >
> > > Look we didn't build modern because we wanted to make work, we did
> > > because legacy is broken. So either let legacy die already or let's
> > > build a sane interface to emulate it please.
> > Almost everyone building virtio device would prefer for legacy to die.
> > I am still trying to understand that why bisecting register definition of legacy
> makes it sane.
>
> Because legacy common config is very evil, do not copy it.
> Legacy device config is only bad in minor tolerable ways.
>
It is not used outside of the legacy interface.
The point is guest driver will follow certain config write/read pattern.
Hypervisor has no way to fail it.
So what do you expect hypervisor to do on undesired sequence?
Do you propose to correct it and send it to device?
And if so, the intention is building sane device for legacy interface? If that is the intention, sound fine.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:09 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-16 21:41 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:41 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:09 PM
> > Possibly one can set. I don’t know if any actual device really supported
> endianness.
> > No users have asked for it, even asking explicitly to those non_little_endian
> users.
>
> For sure, ARM BE does exist and Red Hat supports it because yes, it was
> requested.
Interesting.
Any vdpa device support this? How is the guest endianness conveyed to this vdpa device?
And which barrier this ARM uses before issuing notification on legacy?
> You can not just go by what marketing tells you today we either try
> to build future proof interfaces or we don't bother at all - by the time these
> things are in the field everything shifted.
>
Huh.
One can add the BE support incrementally when the device plans to support it.
1.x is already future proof from endianness, so no need to bring endianness for it.
Only legacy can work in the single platform with hw in wider deployment as David acked.
So over engineering is avoided by not introducing BE support.
> > So may be when there is/if a real user, it can be done in future.
>
> The concern is if you assume LE is default, no way to say "I do not support LE".
> Something like an explicit "SET LE" and "SET BE" commands will automatically
> allow discovering which endian-ness is supported.
>
This is good suggestion.
The LE for legacy is assumed because barrier etc cannot work.
So, avoiding theoretical spec commands here that may not find any user.
> > > >
> > > > > For any case, having a simple and deterministic device design is
> > > > > always better assuming we've agreed that mediation is a must for
> > > > > legacy drivers. Using dedicated commands can make sure the
> > > > > implementation won't need to go with corner cases of legacy.
> > > > >
> > > > Corner case handling just moved from the device to hypervisor.
> > >
> > > That's not safe, the device should be ready for malicious inputs.
> > >
> > With current offset, register, device will be just fine as most implementations
> have to program control path etc on these registers write/read etc.
> > So, device will be able to handle them with plain 2 commands.
>
> Except legacy is broken broken broken. It just does not work for physical
> devices in a crazy number of ways. How about the weird 48 bit limitation on
> PAs? Because no one ever will need any more.
>
Legacy is broken and works only in small but widely used platform.
Hence, it attempts to support it.
48-bit PA limitation in virtio?
> I have 0 sympathy to this idea of reviving all these bugs then circling back and
> coming up with weird work arounds. Please, just build a sane transport and
> have software deal with bugs as best it can.
>
When I proposed the register interface that adheres to the alignment boundary, you suggested to follow legacy semantics.
Now when legacy semantics is followed, you go opposite direction of let sw decide it.
Can we agree that alignment, width and offset to be decided by the sw?
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:41 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:41 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:09 PM
> > Possibly one can set. I don’t know if any actual device really supported
> endianness.
> > No users have asked for it, even asking explicitly to those non_little_endian
> users.
>
> For sure, ARM BE does exist and Red Hat supports it because yes, it was
> requested.
Interesting.
Any vdpa device support this? How is the guest endianness conveyed to this vdpa device?
And which barrier this ARM uses before issuing notification on legacy?
> You can not just go by what marketing tells you today we either try
> to build future proof interfaces or we don't bother at all - by the time these
> things are in the field everything shifted.
>
Huh.
One can add the BE support incrementally when the device plans to support it.
1.x is already future proof from endianness, so no need to bring endianness for it.
Only legacy can work in the single platform with hw in wider deployment as David acked.
So over engineering is avoided by not introducing BE support.
> > So may be when there is/if a real user, it can be done in future.
>
> The concern is if you assume LE is default, no way to say "I do not support LE".
> Something like an explicit "SET LE" and "SET BE" commands will automatically
> allow discovering which endian-ness is supported.
>
This is good suggestion.
The LE for legacy is assumed because barrier etc cannot work.
So, avoiding theoretical spec commands here that may not find any user.
> > > >
> > > > > For any case, having a simple and deterministic device design is
> > > > > always better assuming we've agreed that mediation is a must for
> > > > > legacy drivers. Using dedicated commands can make sure the
> > > > > implementation won't need to go with corner cases of legacy.
> > > > >
> > > > Corner case handling just moved from the device to hypervisor.
> > >
> > > That's not safe, the device should be ready for malicious inputs.
> > >
> > With current offset, register, device will be just fine as most implementations
> have to program control path etc on these registers write/read etc.
> > So, device will be able to handle them with plain 2 commands.
>
> Except legacy is broken broken broken. It just does not work for physical
> devices in a crazy number of ways. How about the weird 48 bit limitation on
> PAs? Because no one ever will need any more.
>
Legacy is broken and works only in small but widely used platform.
Hence, it attempts to support it.
48-bit PA limitation in virtio?
> I have 0 sympathy to this idea of reviving all these bugs then circling back and
> coming up with weird work arounds. Please, just build a sane transport and
> have software deal with bugs as best it can.
>
When I proposed the register interface that adheres to the alignment boundary, you suggested to follow legacy semantics.
Now when legacy semantics is followed, you go opposite direction of let sw decide it.
Can we agree that alignment, width and offset to be decided by the sw?
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:11 ` Michael S. Tsirkin
@ 2023-05-16 21:49 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:12 PM
>
> Nah, we don't need a "break randomly unless it's a full moon and you cross your
> heart three times" mode. If you are going to implement support for legacy
> emulation implement it in a way that either predictably works or predictably
> refuses to load.
It works with same level of predictability as current legacy interface definition for x86_64 platform when msix in guest is supported.
Supporting intx we discussed to deal incrementally after this.
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:49 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-16 21:49 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 16, 2023 5:12 PM
>
> Nah, we don't need a "break randomly unless it's a full moon and you cross your
> heart three times" mode. If you are going to implement support for legacy
> emulation implement it in a way that either predictably works or predictably
> refuses to load.
It works with same level of predictability as current legacy interface definition for x86_64 platform when msix in guest is supported.
Supporting intx we discussed to deal incrementally after this.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:41 ` [virtio-comment] " Parav Pandit
@ 2023-05-16 21:54 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:54 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:41:07PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 5:09 PM
>
> > > Possibly one can set. I don’t know if any actual device really supported
> > endianness.
> > > No users have asked for it, even asking explicitly to those non_little_endian
> > users.
> >
> > For sure, ARM BE does exist and Red Hat supports it because yes, it was
> > requested.
> Interesting.
> Any vdpa device support this? How is the guest endianness conveyed to this vdpa device?
I don't think, so far.
> And which barrier this ARM uses before issuing notification on legacy?
Would be challenging with hardware, but can work for software virtio
for nested virt I guess?
> > You can not just go by what marketing tells you today we either try
> > to build future proof interfaces or we don't bother at all - by the time these
> > things are in the field everything shifted.
> >
> Huh.
> One can add the BE support incrementally when the device plans to support it.
>
> 1.x is already future proof from endianness, so no need to bring endianness for it.
> Only legacy can work in the single platform with hw in wider deployment as David acked.
> So over engineering is avoided by not introducing BE support.
>
> > > So may be when there is/if a real user, it can be done in future.
> >
> > The concern is if you assume LE is default, no way to say "I do not support LE".
> > Something like an explicit "SET LE" and "SET BE" commands will automatically
> > allow discovering which endian-ness is supported.
> >
> This is good suggestion.
> The LE for legacy is assumed because barrier etc cannot work.
> So, avoiding theoretical spec commands here that may not find any user.
maybe at least outlike what happens if some device needs to add BE and
drop LE?
> > > > >
> > > > > > For any case, having a simple and deterministic device design is
> > > > > > always better assuming we've agreed that mediation is a must for
> > > > > > legacy drivers. Using dedicated commands can make sure the
> > > > > > implementation won't need to go with corner cases of legacy.
> > > > > >
> > > > > Corner case handling just moved from the device to hypervisor.
> > > >
> > > > That's not safe, the device should be ready for malicious inputs.
> > > >
> > > With current offset, register, device will be just fine as most implementations
> > have to program control path etc on these registers write/read etc.
> > > So, device will be able to handle them with plain 2 commands.
> >
> > Except legacy is broken broken broken. It just does not work for physical
> > devices in a crazy number of ways. How about the weird 48 bit limitation on
> > PAs? Because no one ever will need any more.
> >
> Legacy is broken and works only in small but widely used platform.
> Hence, it attempts to support it.
>
> 48-bit PA limitation in virtio?
For queue base, yes. One of the crazy things in virtio ...
> > I have 0 sympathy to this idea of reviving all these bugs then circling back and
> > coming up with weird work arounds. Please, just build a sane transport and
> > have software deal with bugs as best it can.
> >
> When I proposed the register interface that adheres to the alignment boundary, you suggested to follow legacy semantics.
> Now when legacy semantics is followed, you go opposite direction of let sw decide it.
Sorry if I was unclear.
I was making a distinction between device specific with arbitrary
alignment, where we should just follow legacy semantics, and
common config where I think sw should decide it.
> Can we agree that alignment, width and offset to be decided by the sw?
For device type specific we can't since spec said we can't :(
For common sure.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:54 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:54 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:41:07PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 5:09 PM
>
> > > Possibly one can set. I don’t know if any actual device really supported
> > endianness.
> > > No users have asked for it, even asking explicitly to those non_little_endian
> > users.
> >
> > For sure, ARM BE does exist and Red Hat supports it because yes, it was
> > requested.
> Interesting.
> Any vdpa device support this? How is the guest endianness conveyed to this vdpa device?
I don't think, so far.
> And which barrier this ARM uses before issuing notification on legacy?
Would be challenging with hardware, but can work for software virtio
for nested virt I guess?
> > You can not just go by what marketing tells you today we either try
> > to build future proof interfaces or we don't bother at all - by the time these
> > things are in the field everything shifted.
> >
> Huh.
> One can add the BE support incrementally when the device plans to support it.
>
> 1.x is already future proof from endianness, so no need to bring endianness for it.
> Only legacy can work in the single platform with hw in wider deployment as David acked.
> So over engineering is avoided by not introducing BE support.
>
> > > So may be when there is/if a real user, it can be done in future.
> >
> > The concern is if you assume LE is default, no way to say "I do not support LE".
> > Something like an explicit "SET LE" and "SET BE" commands will automatically
> > allow discovering which endian-ness is supported.
> >
> This is good suggestion.
> The LE for legacy is assumed because barrier etc cannot work.
> So, avoiding theoretical spec commands here that may not find any user.
maybe at least outlike what happens if some device needs to add BE and
drop LE?
> > > > >
> > > > > > For any case, having a simple and deterministic device design is
> > > > > > always better assuming we've agreed that mediation is a must for
> > > > > > legacy drivers. Using dedicated commands can make sure the
> > > > > > implementation won't need to go with corner cases of legacy.
> > > > > >
> > > > > Corner case handling just moved from the device to hypervisor.
> > > >
> > > > That's not safe, the device should be ready for malicious inputs.
> > > >
> > > With current offset, register, device will be just fine as most implementations
> > have to program control path etc on these registers write/read etc.
> > > So, device will be able to handle them with plain 2 commands.
> >
> > Except legacy is broken broken broken. It just does not work for physical
> > devices in a crazy number of ways. How about the weird 48 bit limitation on
> > PAs? Because no one ever will need any more.
> >
> Legacy is broken and works only in small but widely used platform.
> Hence, it attempts to support it.
>
> 48-bit PA limitation in virtio?
For queue base, yes. One of the crazy things in virtio ...
> > I have 0 sympathy to this idea of reviving all these bugs then circling back and
> > coming up with weird work arounds. Please, just build a sane transport and
> > have software deal with bugs as best it can.
> >
> When I proposed the register interface that adheres to the alignment boundary, you suggested to follow legacy semantics.
> Now when legacy semantics is followed, you go opposite direction of let sw decide it.
Sorry if I was unclear.
I was making a distinction between device specific with arbitrary
alignment, where we should just follow legacy semantics, and
common config where I think sw should decide it.
> Can we agree that alignment, width and offset to be decided by the sw?
For device type specific we can't since spec said we can't :(
For common sure.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-16 21:49 ` Parav Pandit
@ 2023-05-16 21:56 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:49:19PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 5:12 PM
>
> >
> > Nah, we don't need a "break randomly unless it's a full moon and you cross your
> > heart three times" mode. If you are going to implement support for legacy
> > emulation implement it in a way that either predictably works or predictably
> > refuses to load.
>
> It works with same level of predictability as current legacy interface definition for x86_64 platform when msix in guest is supported.
>
> Supporting intx we discussed to deal incrementally after this.
Only thing I am worried is that we do not decide to go back
to msix vector after all because of the shifting offset
of device config. As I said, split device and transport
configs to separate commands rather than just using offset
and then I think it's ok to defer INTx work a bit.
Though I feel it's valuable and will help us move towards
more important goals like LM.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-16 21:56 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-16 21:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 16, 2023 at 09:49:19PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 16, 2023 5:12 PM
>
> >
> > Nah, we don't need a "break randomly unless it's a full moon and you cross your
> > heart three times" mode. If you are going to implement support for legacy
> > emulation implement it in a way that either predictably works or predictably
> > refuses to load.
>
> It works with same level of predictability as current legacy interface definition for x86_64 platform when msix in guest is supported.
>
> Supporting intx we discussed to deal incrementally after this.
Only thing I am worried is that we do not decide to go back
to msix vector after all because of the shifting offset
of device config. As I said, split device and transport
configs to separate commands rather than just using offset
and then I think it's ok to defer INTx work a bit.
Though I feel it's valuable and will help us move towards
more important goals like LM.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-17 5:44 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-17 5:44 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> +\begin{lstlisting}
> +struct virtio_admin_cmd_lreg_wr_data {
> + u8 offset; /* Starting byte offset of the register(s) to write */
> + u8 register[];
> +};
> +\end{lstlisting}
So to summarize, I think the main comment here was to have separate
commands for access to common versus device config. Just today I thought
some more and came up with another reason why this is a good idea: if we
ever want legacy MMIO emulation, we can reuse the device config command.
Jason also feels the common config command can be shared
with vq transport effort. We can discuss this, for sure,
I guess its putting code in hardware versus in hypervisor,
but if there are hardware vendors who strictly want
this code in hardware I don't have a big problem with this
even if I don't exactly get why.
With this idea all hypervisor has to do is subtract the offset from
device config access, sounds like a small price to pay. Does this sound
like a reasonable compromize to you?
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-17 5:44 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-17 5:44 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> +\begin{lstlisting}
> +struct virtio_admin_cmd_lreg_wr_data {
> + u8 offset; /* Starting byte offset of the register(s) to write */
> + u8 register[];
> +};
> +\end{lstlisting}
So to summarize, I think the main comment here was to have separate
commands for access to common versus device config. Just today I thought
some more and came up with another reason why this is a good idea: if we
ever want legacy MMIO emulation, we can reuse the device config command.
Jason also feels the common config command can be shared
with vq transport effort. We can discuss this, for sure,
I guess its putting code in hardware versus in hypervisor,
but if there are hardware vendors who strictly want
this code in hardware I don't have a big problem with this
even if I don't exactly get why.
With this idea all hypervisor has to do is subtract the offset from
device config access, sounds like a small price to pay. Does this sound
like a reasonable compromize to you?
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-17 5:44 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-17 19:32 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-17 19:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 17, 2023 1:45 AM
>
> On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_lreg_wr_data {
> > + u8 offset; /* Starting byte offset of the register(s) to write */
> > + u8 register[];
> > +};
> > +\end{lstlisting}
>
> So to summarize, I think the main comment here was to have separate
> commands for access to common versus device config. Just today I thought
> some more and came up with another reason why this is a good idea: if we ever
> want legacy MMIO emulation, we can reuse the device config command.
>
MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
So single command with offset can service just fine.
More below.
> Jason also feels the common config command can be shared with vq transport
> effort. We can discuss this, for sure, I guess its putting code in hardware versus
> in hypervisor, but if there are hardware vendors who strictly want this code in
> hardware I don't have a big problem with this even if I don't exactly get why.
>
> With this idea all hypervisor has to do is subtract the offset from device config
> access, sounds like a small price to pay.
It is more than the subtraction cost.
Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
> Does this sound like a reasonable
> compromize to you?
Splitting proposed one command to two commands,
1. one for accessing legacy common config
2. second for accessing legacy device specific config
seems fine to me as below.
So we will have total 5 commands (instead of 3).
1. legacy common config read
2. legacy common config write
3. legacy device config read
4. legacy device config write
5. query device notification area
#1 and #3 same cmd signature but different opcode.
#2 and #4 same cmd signature but different opcode.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-17 19:32 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-17 19:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 17, 2023 1:45 AM
>
> On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_lreg_wr_data {
> > + u8 offset; /* Starting byte offset of the register(s) to write */
> > + u8 register[];
> > +};
> > +\end{lstlisting}
>
> So to summarize, I think the main comment here was to have separate
> commands for access to common versus device config. Just today I thought
> some more and came up with another reason why this is a good idea: if we ever
> want legacy MMIO emulation, we can reuse the device config command.
>
MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
So single command with offset can service just fine.
More below.
> Jason also feels the common config command can be shared with vq transport
> effort. We can discuss this, for sure, I guess its putting code in hardware versus
> in hypervisor, but if there are hardware vendors who strictly want this code in
> hardware I don't have a big problem with this even if I don't exactly get why.
>
> With this idea all hypervisor has to do is subtract the offset from device config
> access, sounds like a small price to pay.
It is more than the subtraction cost.
Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
> Does this sound like a reasonable
> compromize to you?
Splitting proposed one command to two commands,
1. one for accessing legacy common config
2. second for accessing legacy device specific config
seems fine to me as below.
So we will have total 5 commands (instead of 3).
1. legacy common config read
2. legacy common config write
3. legacy device config read
4. legacy device config write
5. query device notification area
#1 and #3 same cmd signature but different opcode.
#2 and #4 same cmd signature but different opcode.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-17 19:32 ` [virtio-comment] " Parav Pandit
@ 2023-05-18 19:42 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-18 19:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 17, 2023 1:45 AM
> >
> > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_lreg_wr_data {
> > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > + u8 register[];
> > > +};
> > > +\end{lstlisting}
> >
> > So to summarize, I think the main comment here was to have separate
> > commands for access to common versus device config. Just today I thought
> > some more and came up with another reason why this is a good idea: if we ever
> > want legacy MMIO emulation, we can reuse the device config command.
> >
> MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
> So single command with offset can service just fine.
yes but then you need to implement both MMIO and PCI in hardware.
with a flag you can implement legacy MMIO in software on top of
legacy PCI in hardware.
> More below.
>
> > Jason also feels the common config command can be shared with vq transport
> > effort. We can discuss this, for sure, I guess its putting code in hardware versus
> > in hypervisor, but if there are hardware vendors who strictly want this code in
> > hardware I don't have a big problem with this even if I don't exactly get why.
> >
> > With this idea all hypervisor has to do is subtract the offset from device config
> > access, sounds like a small price to pay.
> It is more than the subtraction cost.
>
> Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
I thought that for now you wanted to ignore MSI-X disable case?
Because if you do care then we need INTx emulation.
> > Does this sound like a reasonable
> > compromize to you?
>
> Splitting proposed one command to two commands,
> 1. one for accessing legacy common config
> 2. second for accessing legacy device specific config
>
> seems fine to me as below.
>
> So we will have total 5 commands (instead of 3).
>
> 1. legacy common config read
> 2. legacy common config write
>
> 3. legacy device config read
> 4. legacy device config write
> 5. query device notification area
>
> #1 and #3 same cmd signature but different opcode.
> #2 and #4 same cmd signature but different opcode.
>
Sounds reasonable. Jason?
notification thing needs more thought I feel though.
It feels weirdly bolted on, but I can't put my finger
on what's wrong exactly yet. Will think it over.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-18 19:42 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-18 19:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 17, 2023 1:45 AM
> >
> > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_lreg_wr_data {
> > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > + u8 register[];
> > > +};
> > > +\end{lstlisting}
> >
> > So to summarize, I think the main comment here was to have separate
> > commands for access to common versus device config. Just today I thought
> > some more and came up with another reason why this is a good idea: if we ever
> > want legacy MMIO emulation, we can reuse the device config command.
> >
> MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
> So single command with offset can service just fine.
yes but then you need to implement both MMIO and PCI in hardware.
with a flag you can implement legacy MMIO in software on top of
legacy PCI in hardware.
> More below.
>
> > Jason also feels the common config command can be shared with vq transport
> > effort. We can discuss this, for sure, I guess its putting code in hardware versus
> > in hypervisor, but if there are hardware vendors who strictly want this code in
> > hardware I don't have a big problem with this even if I don't exactly get why.
> >
> > With this idea all hypervisor has to do is subtract the offset from device config
> > access, sounds like a small price to pay.
> It is more than the subtraction cost.
>
> Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
I thought that for now you wanted to ignore MSI-X disable case?
Because if you do care then we need INTx emulation.
> > Does this sound like a reasonable
> > compromize to you?
>
> Splitting proposed one command to two commands,
> 1. one for accessing legacy common config
> 2. second for accessing legacy device specific config
>
> seems fine to me as below.
>
> So we will have total 5 commands (instead of 3).
>
> 1. legacy common config read
> 2. legacy common config write
>
> 3. legacy device config read
> 4. legacy device config write
> 5. query device notification area
>
> #1 and #3 same cmd signature but different opcode.
> #2 and #4 same cmd signature but different opcode.
>
Sounds reasonable. Jason?
notification thing needs more thought I feel though.
It feels weirdly bolted on, but I can't put my finger
on what's wrong exactly yet. Will think it over.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-18 19:42 ` Michael S. Tsirkin
@ 2023-05-18 20:51 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-18 20:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 18, 2023 3:42 PM
>
> On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 17, 2023 1:45 AM
> > >
> > > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_lreg_wr_data {
> > > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > > + u8 register[];
> > > > +};
> > > > +\end{lstlisting}
> > >
> > > So to summarize, I think the main comment here was to have separate
> > > commands for access to common versus device config. Just today I
> > > thought some more and came up with another reason why this is a good
> > > idea: if we ever want legacy MMIO emulation, we can reuse the device
> config command.
> > >
> > MMIO is actually simpler than PCI because all the registers are contiguous
> regardless of msix.
> > So single command with offset can service just fine.
>
> yes but then you need to implement both MMIO and PCI in hardware.
> with a flag you can implement legacy MMIO in software on top of legacy PCI in
> hardware.
>
> > More below.
> >
> > > Jason also feels the common config command can be shared with vq
> > > transport effort. We can discuss this, for sure, I guess its putting
> > > code in hardware versus in hypervisor, but if there are hardware
> > > vendors who strictly want this code in hardware I don't have a big problem
> with this even if I don't exactly get why.
> > >
> > > With this idea all hypervisor has to do is subtract the offset from
> > > device config access, sounds like a small price to pay.
> > It is more than the subtraction cost.
> >
> > Hypervisor needs to inspect MSI-X enable/disable settings to decide when to
> issue command #1 vs #3 for the requested offset.
>
> I thought that for now you wanted to ignore MSI-X disable case?
> Because if you do care then we need INTx emulation.
>
Right. But the ignorance falls under should category.
So when INTX is enabled/disabled, AQ commands etc can be anchored there.
As opposed to that, config register access needs to decide what to do based on MSIX state.
So hypervisor now must trap MSIX capability access.
With single command, it is not the case in hypervisor.
But it is likely not a big problem given both are in the control path.
>
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands, 1. one for accessing
> > legacy common config 2. second for accessing legacy device specific
> > config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger on what's wrong exactly yet.
> Will think it over.
>
Ok. I considered alternative to get that from the VF itself, but it was not elegant with little value in future.
Will wait if you have better idea, else will proceed to v3.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-18 20:51 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-18 20:51 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, May 18, 2023 3:42 PM
>
> On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 17, 2023 1:45 AM
> > >
> > > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_lreg_wr_data {
> > > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > > + u8 register[];
> > > > +};
> > > > +\end{lstlisting}
> > >
> > > So to summarize, I think the main comment here was to have separate
> > > commands for access to common versus device config. Just today I
> > > thought some more and came up with another reason why this is a good
> > > idea: if we ever want legacy MMIO emulation, we can reuse the device
> config command.
> > >
> > MMIO is actually simpler than PCI because all the registers are contiguous
> regardless of msix.
> > So single command with offset can service just fine.
>
> yes but then you need to implement both MMIO and PCI in hardware.
> with a flag you can implement legacy MMIO in software on top of legacy PCI in
> hardware.
>
> > More below.
> >
> > > Jason also feels the common config command can be shared with vq
> > > transport effort. We can discuss this, for sure, I guess its putting
> > > code in hardware versus in hypervisor, but if there are hardware
> > > vendors who strictly want this code in hardware I don't have a big problem
> with this even if I don't exactly get why.
> > >
> > > With this idea all hypervisor has to do is subtract the offset from
> > > device config access, sounds like a small price to pay.
> > It is more than the subtraction cost.
> >
> > Hypervisor needs to inspect MSI-X enable/disable settings to decide when to
> issue command #1 vs #3 for the requested offset.
>
> I thought that for now you wanted to ignore MSI-X disable case?
> Because if you do care then we need INTx emulation.
>
Right. But the ignorance falls under should category.
So when INTX is enabled/disabled, AQ commands etc can be anchored there.
As opposed to that, config register access needs to decide what to do based on MSIX state.
So hypervisor now must trap MSIX capability access.
With single command, it is not the case in hypervisor.
But it is likely not a big problem given both are in the control path.
>
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands, 1. one for accessing
> > legacy common config 2. second for accessing legacy device specific
> > config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger on what's wrong exactly yet.
> Will think it over.
>
Ok. I considered alternative to get that from the VF itself, but it was not elegant with little value in future.
Will wait if you have better idea, else will proceed to v3.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-18 19:42 ` Michael S. Tsirkin
@ 2023-05-19 1:54 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-19 1:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Fri, May 19, 2023 at 3:42 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 17, 2023 1:45 AM
> > >
> > > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_lreg_wr_data {
> > > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > > + u8 register[];
> > > > +};
> > > > +\end{lstlisting}
> > >
> > > So to summarize, I think the main comment here was to have separate
> > > commands for access to common versus device config. Just today I thought
> > > some more and came up with another reason why this is a good idea: if we ever
> > > want legacy MMIO emulation, we can reuse the device config command.
> > >
> > MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
> > So single command with offset can service just fine.
>
> yes but then you need to implement both MMIO and PCI in hardware.
> with a flag you can implement legacy MMIO in software on top of
> legacy PCI in hardware.
>
> > More below.
> >
> > > Jason also feels the common config command can be shared with vq transport
> > > effort. We can discuss this, for sure, I guess its putting code in hardware versus
> > > in hypervisor, but if there are hardware vendors who strictly want this code in
> > > hardware I don't have a big problem with this even if I don't exactly get why.
> > >
> > > With this idea all hypervisor has to do is subtract the offset from device config
> > > access, sounds like a small price to pay.
> > It is more than the subtraction cost.
> >
> > Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
>
I don't know why it is a problem, for any hypervisor that supports
legacy, the code (MSI-X enablement detection) should already be there.
And it is just something like what Qemu had:
/* The remaining space is defined by each driver as the per-driver
* configuration space */
#define VIRTIO_PCI_CONFIG_SIZE(dev) VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
Is this really expensive?
> I thought that for now you wanted to ignore MSI-X disable case?
> Because if you do care then we need INTx emulation.
>
>
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands,
> > 1. one for accessing legacy common config
> > 2. second for accessing legacy device specific config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
Better, but for transport virtqueue/adminq we won't go this way but
using individual commands for common and other configurations for a
more deterministic behaviour with a simpler device.
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger
> on what's wrong exactly yet. Will think it over.
Two things in my mind:
1) There's probably no need for a BAR and it should be sufficient for
an address, the device just needs to make sure to report a correct
address if BAR is remapped. It helps to save BAR for VF as well.
2) We need a size, otherwise we don't know if the notification area
could be mapped safely to the guest. And in the case of
virtio_pci_notify_cap, though it could be implied by the offset but it
deserves a formal clarification as well there.
Thanks
>
>
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-19 1:54 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-19 1:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Fri, May 19, 2023 at 3:42 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, May 17, 2023 at 07:32:33PM +0000, Parav Pandit wrote:
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Wednesday, May 17, 2023 1:45 AM
> > >
> > > On Sat, May 06, 2023 at 03:01:34AM +0300, Parav Pandit wrote:
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_lreg_wr_data {
> > > > + u8 offset; /* Starting byte offset of the register(s) to write */
> > > > + u8 register[];
> > > > +};
> > > > +\end{lstlisting}
> > >
> > > So to summarize, I think the main comment here was to have separate
> > > commands for access to common versus device config. Just today I thought
> > > some more and came up with another reason why this is a good idea: if we ever
> > > want legacy MMIO emulation, we can reuse the device config command.
> > >
> > MMIO is actually simpler than PCI because all the registers are contiguous regardless of msix.
> > So single command with offset can service just fine.
>
> yes but then you need to implement both MMIO and PCI in hardware.
> with a flag you can implement legacy MMIO in software on top of
> legacy PCI in hardware.
>
> > More below.
> >
> > > Jason also feels the common config command can be shared with vq transport
> > > effort. We can discuss this, for sure, I guess its putting code in hardware versus
> > > in hypervisor, but if there are hardware vendors who strictly want this code in
> > > hardware I don't have a big problem with this even if I don't exactly get why.
> > >
> > > With this idea all hypervisor has to do is subtract the offset from device config
> > > access, sounds like a small price to pay.
> > It is more than the subtraction cost.
> >
> > Hypervisor needs to inspect MSI-X enable/disable settings to decide when to issue command #1 vs #3 for the requested offset.
>
I don't know why it is a problem, for any hypervisor that supports
legacy, the code (MSI-X enablement detection) should already be there.
And it is just something like what Qemu had:
/* The remaining space is defined by each driver as the per-driver
* configuration space */
#define VIRTIO_PCI_CONFIG_SIZE(dev) VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
Is this really expensive?
> I thought that for now you wanted to ignore MSI-X disable case?
> Because if you do care then we need INTx emulation.
>
>
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands,
> > 1. one for accessing legacy common config
> > 2. second for accessing legacy device specific config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
Better, but for transport virtqueue/adminq we won't go this way but
using individual commands for common and other configurations for a
more deterministic behaviour with a simpler device.
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger
> on what's wrong exactly yet. Will think it over.
Two things in my mind:
1) There's probably no need for a BAR and it should be sufficient for
an address, the device just needs to make sure to report a correct
address if BAR is remapped. It helps to save BAR for VF as well.
2) We need a size, otherwise we don't know if the notification area
could be mapped safely to the guest. And in the case of
virtio_pci_notify_cap, though it could be implied by the offset but it
deserves a formal clarification as well there.
Thanks
>
>
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-19 1:54 ` Jason Wang
@ 2023-05-19 2:04 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-19 2:04 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 18, 2023 9:54 PM
> > > Hypervisor needs to inspect MSI-X enable/disable settings to decide when
> to issue command #1 vs #3 for the requested offset.
> >
>
> I don't know why it is a problem, for any hypervisor that supports legacy, the
> code (MSI-X enablement detection) should already be there.
>
> And it is just something like what Qemu had:
>
> /* The remaining space is defined by each driver as the per-driver
> * configuration space */
> #define VIRTIO_PCI_CONFIG_SIZE(dev)
> VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
>
> Is this really expensive?
>
Not really, its just an additional query on the config access.
> > > So we will have total 5 commands (instead of 3).
> > >
> > > 1. legacy common config read
> > > 2. legacy common config write
> > >
> > > 3. legacy device config read
> > > 4. legacy device config write
> > > 5. query device notification area
> > >
> > > #1 and #3 same cmd signature but different opcode.
> > > #2 and #4 same cmd signature but different opcode.
> > >
> >
> > Sounds reasonable. Jason?
>
> Better, but for transport virtqueue/adminq we won't go this way but using
> individual commands for common and other configurations for a more
> deterministic behaviour with a simpler device.
>
> Two things in my mind:
>
> 1) There's probably no need for a BAR and it should be sufficient for an address,
> the device just needs to make sure to report a correct address if BAR is
> remapped. It helps to save BAR for VF as well.
> 2) We need a size, otherwise we don't know if the notification area could be
> mapped safely to the guest. And in the case of virtio_pci_notify_cap, though it
> could be implied by the offset but it deserves a formal clarification as well there.
>
A dedicated command that returns BAR, offset and size is elegant to avoid mixing pci capability and register linking.
Sort of self-contained command.
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-19 2:04 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-19 2:04 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, May 18, 2023 9:54 PM
> > > Hypervisor needs to inspect MSI-X enable/disable settings to decide when
> to issue command #1 vs #3 for the requested offset.
> >
>
> I don't know why it is a problem, for any hypervisor that supports legacy, the
> code (MSI-X enablement detection) should already be there.
>
> And it is just something like what Qemu had:
>
> /* The remaining space is defined by each driver as the per-driver
> * configuration space */
> #define VIRTIO_PCI_CONFIG_SIZE(dev)
> VIRTIO_PCI_CONFIG_OFF(msix_enabled(dev))
>
> Is this really expensive?
>
Not really, its just an additional query on the config access.
> > > So we will have total 5 commands (instead of 3).
> > >
> > > 1. legacy common config read
> > > 2. legacy common config write
> > >
> > > 3. legacy device config read
> > > 4. legacy device config write
> > > 5. query device notification area
> > >
> > > #1 and #3 same cmd signature but different opcode.
> > > #2 and #4 same cmd signature but different opcode.
> > >
> >
> > Sounds reasonable. Jason?
>
> Better, but for transport virtqueue/adminq we won't go this way but using
> individual commands for common and other configurations for a more
> deterministic behaviour with a simpler device.
>
> Two things in my mind:
>
> 1) There's probably no need for a BAR and it should be sufficient for an address,
> the device just needs to make sure to report a correct address if BAR is
> remapped. It helps to save BAR for VF as well.
> 2) We need a size, otherwise we don't know if the notification area could be
> mapped safely to the guest. And in the case of virtio_pci_notify_cap, though it
> could be implied by the offset but it deserves a formal clarification as well there.
>
A dedicated command that returns BAR, offset and size is elegant to avoid mixing pci capability and register linking.
Sort of self-contained command.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-18 19:42 ` Michael S. Tsirkin
@ 2023-05-19 6:06 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-19 6:06 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands,
> > 1. one for accessing legacy common config
> > 2. second for accessing legacy device specific config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger
> on what's wrong exactly yet. Will think it over.
So with a fresh mind, at least three things:
1. given driver attaches to the PF, it should be possible to forward
notifications there, as opposed to individual VFs. NumVFs is 16 bit so
it will fit in a 32 bit write together with VQ index.
2. It should be possible to send notifications through an admin command too,
otherwise admin commands are an incomplete set of functionality.
3. I feel using a capability to describe legacy notification
area would be better just because we already have a
structure for this. make it an express capability if you like.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-19 6:06 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-19 6:06 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > Does this sound like a reasonable
> > > compromize to you?
> >
> > Splitting proposed one command to two commands,
> > 1. one for accessing legacy common config
> > 2. second for accessing legacy device specific config
> >
> > seems fine to me as below.
> >
> > So we will have total 5 commands (instead of 3).
> >
> > 1. legacy common config read
> > 2. legacy common config write
> >
> > 3. legacy device config read
> > 4. legacy device config write
> > 5. query device notification area
> >
> > #1 and #3 same cmd signature but different opcode.
> > #2 and #4 same cmd signature but different opcode.
> >
>
> Sounds reasonable. Jason?
>
> notification thing needs more thought I feel though.
> It feels weirdly bolted on, but I can't put my finger
> on what's wrong exactly yet. Will think it over.
So with a fresh mind, at least three things:
1. given driver attaches to the PF, it should be possible to forward
notifications there, as opposed to individual VFs. NumVFs is 16 bit so
it will fit in a 32 bit write together with VQ index.
2. It should be possible to send notifications through an admin command too,
otherwise admin commands are an incomplete set of functionality.
3. I feel using a capability to describe legacy notification
area would be better just because we already have a
structure for this. make it an express capability if you like.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-19 6:06 ` Michael S. Tsirkin
@ 2023-05-19 16:37 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-19 16:37 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, May 19, 2023 2:07 AM
>
> On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > > Does this sound like a reasonable
> > > > compromize to you?
> > >
> > > Splitting proposed one command to two commands, 1. one for accessing
> > > legacy common config 2. second for accessing legacy device specific
> > > config
> > >
> > > seems fine to me as below.
> > >
> > > So we will have total 5 commands (instead of 3).
> > >
> > > 1. legacy common config read
> > > 2. legacy common config write
> > >
> > > 3. legacy device config read
> > > 4. legacy device config write
> > > 5. query device notification area
> > >
> > > #1 and #3 same cmd signature but different opcode.
> > > #2 and #4 same cmd signature but different opcode.
> > >
> >
> > Sounds reasonable. Jason?
> >
> > notification thing needs more thought I feel though.
> > It feels weirdly bolted on, but I can't put my finger on what's wrong
> > exactly yet. Will think it over.
>
>
> So with a fresh mind, at least three things:
>
> 1. given driver attaches to the PF, it should be possible to forward
> notifications there, as opposed to individual VFs. NumVFs is 16 bit so
> it will fit in a 32 bit write together with VQ index.
>
The Notification of the VFs are on the VF BAR for modern or legacy.
One needs to build additional cross forwarding hardware from PF to VF for the doorbells.
And it cannot utilize what already exists for 1.x VF.
> 2. It should be possible to send notifications through an admin command too,
> otherwise admin commands are an incomplete set of functionality.
>
Yes. it is only for the functionality. As we discussed in past already, this will not get any performance.
> 3. I feel using a capability to describe legacy notification
> area would be better just because we already have a
> structure for this. make it an express capability if you like.
The AQ command interface is far more programable object than PCI capability to return this admin info.
Hence I prefer AQ cmd.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-19 16:37 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-19 16:37 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, May 19, 2023 2:07 AM
>
> On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > > Does this sound like a reasonable
> > > > compromize to you?
> > >
> > > Splitting proposed one command to two commands, 1. one for accessing
> > > legacy common config 2. second for accessing legacy device specific
> > > config
> > >
> > > seems fine to me as below.
> > >
> > > So we will have total 5 commands (instead of 3).
> > >
> > > 1. legacy common config read
> > > 2. legacy common config write
> > >
> > > 3. legacy device config read
> > > 4. legacy device config write
> > > 5. query device notification area
> > >
> > > #1 and #3 same cmd signature but different opcode.
> > > #2 and #4 same cmd signature but different opcode.
> > >
> >
> > Sounds reasonable. Jason?
> >
> > notification thing needs more thought I feel though.
> > It feels weirdly bolted on, but I can't put my finger on what's wrong
> > exactly yet. Will think it over.
>
>
> So with a fresh mind, at least three things:
>
> 1. given driver attaches to the PF, it should be possible to forward
> notifications there, as opposed to individual VFs. NumVFs is 16 bit so
> it will fit in a 32 bit write together with VQ index.
>
The Notification of the VFs are on the VF BAR for modern or legacy.
One needs to build additional cross forwarding hardware from PF to VF for the doorbells.
And it cannot utilize what already exists for 1.x VF.
> 2. It should be possible to send notifications through an admin command too,
> otherwise admin commands are an incomplete set of functionality.
>
Yes. it is only for the functionality. As we discussed in past already, this will not get any performance.
> 3. I feel using a capability to describe legacy notification
> area would be better just because we already have a
> structure for this. make it an express capability if you like.
The AQ command interface is far more programable object than PCI capability to return this admin info.
Hence I prefer AQ cmd.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-19 16:37 ` Parav Pandit
@ 2023-05-21 9:16 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-21 9:16 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Fri, May 19, 2023 at 04:37:16PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, May 19, 2023 2:07 AM
> >
> > On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > > > Does this sound like a reasonable
> > > > > compromize to you?
> > > >
> > > > Splitting proposed one command to two commands, 1. one for accessing
> > > > legacy common config 2. second for accessing legacy device specific
> > > > config
> > > >
> > > > seems fine to me as below.
> > > >
> > > > So we will have total 5 commands (instead of 3).
> > > >
> > > > 1. legacy common config read
> > > > 2. legacy common config write
> > > >
> > > > 3. legacy device config read
> > > > 4. legacy device config write
> > > > 5. query device notification area
> > > >
> > > > #1 and #3 same cmd signature but different opcode.
> > > > #2 and #4 same cmd signature but different opcode.
> > > >
> > >
> > > Sounds reasonable. Jason?
> > >
> > > notification thing needs more thought I feel though.
> > > It feels weirdly bolted on, but I can't put my finger on what's wrong
> > > exactly yet. Will think it over.
> >
> >
> > So with a fresh mind, at least three things:
> >
> > 1. given driver attaches to the PF, it should be possible to forward
> > notifications there, as opposed to individual VFs. NumVFs is 16 bit so
> > it will fit in a 32 bit write together with VQ index.
> >
> The Notification of the VFs are on the VF BAR for modern or legacy.
> One needs to build additional cross forwarding hardware from PF to VF for the doorbells.
I think doorbells are driver notifications (linux driver calls them
kicks)?
I don't understand what you are saying above really.
what can and what can't be done?
Again all this idea (as opposed to Jason's transport vq)
is to have a simple software model. Attaching a driver to two
devices at the same time is hard to achive e.g. under windows.
> And it cannot utilize what already exists for 1.x VF.
>
> > 2. It should be possible to send notifications through an admin command too,
> > otherwise admin commands are an incomplete set of functionality.
> >
> Yes. it is only for the functionality. As we discussed in past already, this will not get any performance.
Performance might be ok if hardware disables kicks most of the time.
> > 3. I feel using a capability to describe legacy notification
> > area would be better just because we already have a
> > structure for this. make it an express capability if you like.
> The AQ command interface is far more programable object than PCI capability to return this admin info.
> Hence I prefer AQ cmd.
I feel your preferences for 1 and 3 conflict. If you really insist on
kicks being on VFs then at least let us make VF driver in the
host simple. If it has to talk to PF driver things are really
complex. At this point we are very very far from VFIO model,
and then what exactly have we gained by implementing
legacy control path in hardware? Let's do software with
maybe a couple of features such as VIRTIO_NET_F_LEGACY_HEADER.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-21 9:16 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-21 9:16 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Fri, May 19, 2023 at 04:37:16PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, May 19, 2023 2:07 AM
> >
> > On Thu, May 18, 2023 at 03:42:19PM -0400, Michael S. Tsirkin wrote:
> > > > > Does this sound like a reasonable
> > > > > compromize to you?
> > > >
> > > > Splitting proposed one command to two commands, 1. one for accessing
> > > > legacy common config 2. second for accessing legacy device specific
> > > > config
> > > >
> > > > seems fine to me as below.
> > > >
> > > > So we will have total 5 commands (instead of 3).
> > > >
> > > > 1. legacy common config read
> > > > 2. legacy common config write
> > > >
> > > > 3. legacy device config read
> > > > 4. legacy device config write
> > > > 5. query device notification area
> > > >
> > > > #1 and #3 same cmd signature but different opcode.
> > > > #2 and #4 same cmd signature but different opcode.
> > > >
> > >
> > > Sounds reasonable. Jason?
> > >
> > > notification thing needs more thought I feel though.
> > > It feels weirdly bolted on, but I can't put my finger on what's wrong
> > > exactly yet. Will think it over.
> >
> >
> > So with a fresh mind, at least three things:
> >
> > 1. given driver attaches to the PF, it should be possible to forward
> > notifications there, as opposed to individual VFs. NumVFs is 16 bit so
> > it will fit in a 32 bit write together with VQ index.
> >
> The Notification of the VFs are on the VF BAR for modern or legacy.
> One needs to build additional cross forwarding hardware from PF to VF for the doorbells.
I think doorbells are driver notifications (linux driver calls them
kicks)?
I don't understand what you are saying above really.
what can and what can't be done?
Again all this idea (as opposed to Jason's transport vq)
is to have a simple software model. Attaching a driver to two
devices at the same time is hard to achive e.g. under windows.
> And it cannot utilize what already exists for 1.x VF.
>
> > 2. It should be possible to send notifications through an admin command too,
> > otherwise admin commands are an incomplete set of functionality.
> >
> Yes. it is only for the functionality. As we discussed in past already, this will not get any performance.
Performance might be ok if hardware disables kicks most of the time.
> > 3. I feel using a capability to describe legacy notification
> > area would be better just because we already have a
> > structure for this. make it an express capability if you like.
> The AQ command interface is far more programable object than PCI capability to return this admin info.
> Hence I prefer AQ cmd.
I feel your preferences for 1 and 3 conflict. If you really insist on
kicks being on VFs then at least let us make VF driver in the
host simple. If it has to talk to PF driver things are really
complex. At this point we are very very far from VFIO model,
and then what exactly have we gained by implementing
legacy control path in hardware? Let's do software with
maybe a couple of features such as VIRTIO_NET_F_LEGACY_HEADER.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-21 9:16 ` Michael S. Tsirkin
@ 2023-05-21 13:21 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-21 13:21 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Michael S. Tsirkin
> Sent: Sunday, May 21, 2023 5:17 AM
> > The Notification of the VFs are on the VF BAR for modern or legacy.
> > One needs to build additional cross forwarding hardware from PF to VF for the
> doorbells.
>
> I think doorbells are driver notifications (linux driver calls them kicks)?
> I don't understand what you are saying above really.
> what can and what can't be done?
>
VF has the notification BAR region.
All 1.x and legacy such notification lands on the VF BAR.
> Again all this idea (as opposed to Jason's transport vq) is to have a simple
> software model. Attaching a driver to two devices at the same time is hard to
> achive e.g. under windows.
>
Yet you initiate same discussion point that we already discussed again after summarizing.
A driver is not attached to two devices.
A driver is attached to a single device.
A device uses a parent/owner group device to access legacy.
Software model may evolve over time.
>
> > And it cannot utilize what already exists for 1.x VF.
> >
> > > 2. It should be possible to send notifications through an admin
> > command too,
> > > otherwise admin commands are an incomplete set of functionality.
> > >
> > Yes. it is only for the functionality. As we discussed in past already, this will not
> get any performance.
>
> Performance might be ok if hardware disables kicks most of the time.
>
Even for the first kick it is order of magnitude higher.
Kicks is the natural tool for the low latency.
>
> > > 3. I feel using a capability to describe legacy notification
> > > area would be better just because we already have a
> > > structure for this. make it an express capability if you like.
> > The AQ command interface is far more programable object than PCI capability
> to return this admin info.
> > Hence I prefer AQ cmd.
>
> I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> VFs then at least let us make VF driver in the host simple.
It is straight forward.
If you prefer the "offset" example of past,
If (legacy_offset == queue_notify_offset)
*db = guest_supplied_q_notify_content;
else
virtio_net_send_aq_cmd();
"simple" is really a subjective term in this context.
> If it has to talk to PF
> driver things are really complex. At this point we are very very far from VFIO
> model, and then what exactly have we gained by implementing legacy control
> path in hardware?
It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> Let's do software with maybe a couple of features such as
> VIRTIO_NET_F_LEGACY_HEADER.
>
We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
We cannot iterate exact 100% same points again in this summary discussion.
If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
We again past that discussion.
This is different requirement, than
"There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
This is it.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-21 13:21 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-21 13:21 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Michael S. Tsirkin
> Sent: Sunday, May 21, 2023 5:17 AM
> > The Notification of the VFs are on the VF BAR for modern or legacy.
> > One needs to build additional cross forwarding hardware from PF to VF for the
> doorbells.
>
> I think doorbells are driver notifications (linux driver calls them kicks)?
> I don't understand what you are saying above really.
> what can and what can't be done?
>
VF has the notification BAR region.
All 1.x and legacy such notification lands on the VF BAR.
> Again all this idea (as opposed to Jason's transport vq) is to have a simple
> software model. Attaching a driver to two devices at the same time is hard to
> achive e.g. under windows.
>
Yet you initiate same discussion point that we already discussed again after summarizing.
A driver is not attached to two devices.
A driver is attached to a single device.
A device uses a parent/owner group device to access legacy.
Software model may evolve over time.
>
> > And it cannot utilize what already exists for 1.x VF.
> >
> > > 2. It should be possible to send notifications through an admin
> > command too,
> > > otherwise admin commands are an incomplete set of functionality.
> > >
> > Yes. it is only for the functionality. As we discussed in past already, this will not
> get any performance.
>
> Performance might be ok if hardware disables kicks most of the time.
>
Even for the first kick it is order of magnitude higher.
Kicks is the natural tool for the low latency.
>
> > > 3. I feel using a capability to describe legacy notification
> > > area would be better just because we already have a
> > > structure for this. make it an express capability if you like.
> > The AQ command interface is far more programable object than PCI capability
> to return this admin info.
> > Hence I prefer AQ cmd.
>
> I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> VFs then at least let us make VF driver in the host simple.
It is straight forward.
If you prefer the "offset" example of past,
If (legacy_offset == queue_notify_offset)
*db = guest_supplied_q_notify_content;
else
virtio_net_send_aq_cmd();
"simple" is really a subjective term in this context.
> If it has to talk to PF
> driver things are really complex. At this point we are very very far from VFIO
> model, and then what exactly have we gained by implementing legacy control
> path in hardware?
It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> Let's do software with maybe a couple of features such as
> VIRTIO_NET_F_LEGACY_HEADER.
>
We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
We cannot iterate exact 100% same points again in this summary discussion.
If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
We again past that discussion.
This is different requirement, than
"There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
This is it.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-21 13:21 ` Parav Pandit
@ 2023-05-21 14:33 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-21 14:33 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Michael S. Tsirkin
> > Sent: Sunday, May 21, 2023 5:17 AM
>
> > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > One needs to build additional cross forwarding hardware from PF to VF for the
> > doorbells.
> >
> > I think doorbells are driver notifications (linux driver calls them kicks)?
> > I don't understand what you are saying above really.
> > what can and what can't be done?
> >
> VF has the notification BAR region.
> All 1.x and legacy such notification lands on the VF BAR.
>
> > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > software model. Attaching a driver to two devices at the same time is hard to
> > achive e.g. under windows.
> >
> Yet you initiate same discussion point that we already discussed again after summarizing.
> A driver is not attached to two devices.
> A driver is attached to a single device.
And that device is the owner no? to send commands?
> A device uses a parent/owner group device to access legacy.
>
> Software model may evolve over time.
>
> >
> > > And it cannot utilize what already exists for 1.x VF.
> > >
> > > > 2. It should be possible to send notifications through an admin
> > > command too,
> > > > otherwise admin commands are an incomplete set of functionality.
> > > >
> > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > get any performance.
> >
> > Performance might be ok if hardware disables kicks most of the time.
> >
> Even for the first kick it is order of magnitude higher.
> Kicks is the natural tool for the low latency.
>
> >
> > > > 3. I feel using a capability to describe legacy notification
> > > > area would be better just because we already have a
> > > > structure for this. make it an express capability if you like.
> > > The AQ command interface is far more programable object than PCI capability
> > to return this admin info.
> > > Hence I prefer AQ cmd.
> >
> > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > VFs then at least let us make VF driver in the host simple.
> It is straight forward.
>
> If you prefer the "offset" example of past,
>
> If (legacy_offset == queue_notify_offset)
> *db = guest_supplied_q_notify_content;
> else
> virtio_net_send_aq_cmd();
>
> "simple" is really a subjective term in this context.
yes this is qemu. sure.
So we have legacy emulation send commands to VF or to PF. Okay. But let
us avoid the need for VF driver to send commands to PF to initialize.
Just get all information it needs from VF itself.
Maybe it's a good idea to reuse existing notification capability,
or maybe a new one, but let's avoid making VF driver depend on PF
commands.
> > If it has to talk to PF
> > driver things are really complex. At this point we are very very far from VFIO
> > model, and then what exactly have we gained by implementing legacy control
> > path in hardware?
> It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
>
> > Let's do software with maybe a couple of features such as
> > VIRTIO_NET_F_LEGACY_HEADER.
> >
> We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
BTW not for this project but generally this is why I thought it is not
at all a bad idea to have a requirements text document e.g. under
todo/
discuss it as normally maybe even vote on including features in a
plan for a release.
> We cannot iterate exact 100% same points again in this summary discussion.
>
> If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
>
> What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> We again past that discussion.
>
> This is different requirement, than
> "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
>
> This is it.
So, I am saying one model is small driver for VF and a big one for PF.
And to keep the VF driver simple, it should get information simply from
config space capability.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-21 14:33 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-21 14:33 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
>
> > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > open.org> On Behalf Of Michael S. Tsirkin
> > Sent: Sunday, May 21, 2023 5:17 AM
>
> > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > One needs to build additional cross forwarding hardware from PF to VF for the
> > doorbells.
> >
> > I think doorbells are driver notifications (linux driver calls them kicks)?
> > I don't understand what you are saying above really.
> > what can and what can't be done?
> >
> VF has the notification BAR region.
> All 1.x and legacy such notification lands on the VF BAR.
>
> > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > software model. Attaching a driver to two devices at the same time is hard to
> > achive e.g. under windows.
> >
> Yet you initiate same discussion point that we already discussed again after summarizing.
> A driver is not attached to two devices.
> A driver is attached to a single device.
And that device is the owner no? to send commands?
> A device uses a parent/owner group device to access legacy.
>
> Software model may evolve over time.
>
> >
> > > And it cannot utilize what already exists for 1.x VF.
> > >
> > > > 2. It should be possible to send notifications through an admin
> > > command too,
> > > > otherwise admin commands are an incomplete set of functionality.
> > > >
> > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > get any performance.
> >
> > Performance might be ok if hardware disables kicks most of the time.
> >
> Even for the first kick it is order of magnitude higher.
> Kicks is the natural tool for the low latency.
>
> >
> > > > 3. I feel using a capability to describe legacy notification
> > > > area would be better just because we already have a
> > > > structure for this. make it an express capability if you like.
> > > The AQ command interface is far more programable object than PCI capability
> > to return this admin info.
> > > Hence I prefer AQ cmd.
> >
> > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > VFs then at least let us make VF driver in the host simple.
> It is straight forward.
>
> If you prefer the "offset" example of past,
>
> If (legacy_offset == queue_notify_offset)
> *db = guest_supplied_q_notify_content;
> else
> virtio_net_send_aq_cmd();
>
> "simple" is really a subjective term in this context.
yes this is qemu. sure.
So we have legacy emulation send commands to VF or to PF. Okay. But let
us avoid the need for VF driver to send commands to PF to initialize.
Just get all information it needs from VF itself.
Maybe it's a good idea to reuse existing notification capability,
or maybe a new one, but let's avoid making VF driver depend on PF
commands.
> > If it has to talk to PF
> > driver things are really complex. At this point we are very very far from VFIO
> > model, and then what exactly have we gained by implementing legacy control
> > path in hardware?
> It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
>
> > Let's do software with maybe a couple of features such as
> > VIRTIO_NET_F_LEGACY_HEADER.
> >
> We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
BTW not for this project but generally this is why I thought it is not
at all a bad idea to have a requirements text document e.g. under
todo/
discuss it as normally maybe even vote on including features in a
plan for a release.
> We cannot iterate exact 100% same points again in this summary discussion.
>
> If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
>
> What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> We again past that discussion.
>
> This is different requirement, than
> "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
>
> This is it.
So, I am saying one model is small driver for VF and a big one for PF.
And to keep the VF driver simple, it should get information simply from
config space capability.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-21 14:33 ` Michael S. Tsirkin
@ 2023-05-21 14:44 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-21 14:44 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Sunday, May 21, 2023 10:33 AM
> > Yet you initiate same discussion point that we already discussed again after
> summarizing.
> > A driver is not attached to two devices.
> > A driver is attached to a single device.
>
> And that device is the owner no? to send commands?
>
Not for legacy registers access as discussed before.
> > If (legacy_offset == queue_notify_offset)
> > *db = guest_supplied_q_notify_content; else
> > virtio_net_send_aq_cmd();
> >
> > "simple" is really a subjective term in this context.
>
> yes this is qemu. sure.
>
Not limited to QEMU.
A driver will be able to do this as well.
> So we have legacy emulation send commands to VF or to PF. Okay. But let us
> avoid the need for VF driver to send commands to PF to initialize.
> Just get all information it needs from VF itself.
>
>
> Maybe it's a good idea to reuse existing notification capability, or maybe a new
> one, but let's avoid making VF driver depend on PF commands.
>
We agreed in v1 on Jason's suggestion to have the AQ command and yet you and Jason hinder this in v2 with this exact repeated question.
Lets please avoid this and move forward.
> > We have already these design choices and tradeoff in v0 and v1, it doesn't fit
> the requirements.
>
> So, I am saying one model is small driver for VF and a big one for PF.
> And to keep the VF driver simple, it should get information simply from config
> space capability.
VF driver is small that does usual vfio passthrough work.
PF driver implement AQ for variety of use cases that we listed in the AQ cover letter.
VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-21 14:44 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-21 14:44 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Sunday, May 21, 2023 10:33 AM
> > Yet you initiate same discussion point that we already discussed again after
> summarizing.
> > A driver is not attached to two devices.
> > A driver is attached to a single device.
>
> And that device is the owner no? to send commands?
>
Not for legacy registers access as discussed before.
> > If (legacy_offset == queue_notify_offset)
> > *db = guest_supplied_q_notify_content; else
> > virtio_net_send_aq_cmd();
> >
> > "simple" is really a subjective term in this context.
>
> yes this is qemu. sure.
>
Not limited to QEMU.
A driver will be able to do this as well.
> So we have legacy emulation send commands to VF or to PF. Okay. But let us
> avoid the need for VF driver to send commands to PF to initialize.
> Just get all information it needs from VF itself.
>
>
> Maybe it's a good idea to reuse existing notification capability, or maybe a new
> one, but let's avoid making VF driver depend on PF commands.
>
We agreed in v1 on Jason's suggestion to have the AQ command and yet you and Jason hinder this in v2 with this exact repeated question.
Lets please avoid this and move forward.
> > We have already these design choices and tradeoff in v0 and v1, it doesn't fit
> the requirements.
>
> So, I am saying one model is small driver for VF and a big one for PF.
> And to keep the VF driver simple, it should get information simply from config
> space capability.
VF driver is small that does usual vfio passthrough work.
PF driver implement AQ for variety of use cases that we listed in the AQ cover letter.
VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-21 14:33 ` Michael S. Tsirkin
@ 2023-05-22 0:54 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-22 0:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 10:33 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
> >
> > > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > open.org> On Behalf Of Michael S. Tsirkin
> > > Sent: Sunday, May 21, 2023 5:17 AM
> >
> > > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > > One needs to build additional cross forwarding hardware from PF to VF for the
> > > doorbells.
> > >
> > > I think doorbells are driver notifications (linux driver calls them kicks)?
> > > I don't understand what you are saying above really.
> > > what can and what can't be done?
> > >
> > VF has the notification BAR region.
> > All 1.x and legacy such notification lands on the VF BAR.
> >
> > > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > > software model. Attaching a driver to two devices at the same time is hard to
> > > achive e.g. under windows.
> > >
> > Yet you initiate same discussion point that we already discussed again after summarizing.
> > A driver is not attached to two devices.
> > A driver is attached to a single device.
>
> And that device is the owner no? to send commands?
>
> > A device uses a parent/owner group device to access legacy.
> >
> > Software model may evolve over time.
> >
> > >
> > > > And it cannot utilize what already exists for 1.x VF.
> > > >
> > > > > 2. It should be possible to send notifications through an admin
> > > > command too,
> > > > > otherwise admin commands are an incomplete set of functionality.
> > > > >
> > > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > > get any performance.
> > >
> > > Performance might be ok if hardware disables kicks most of the time.
> > >
> > Even for the first kick it is order of magnitude higher.
> > Kicks is the natural tool for the low latency.
> >
> > >
> > > > > 3. I feel using a capability to describe legacy notification
> > > > > area would be better just because we already have a
> > > > > structure for this. make it an express capability if you like.
> > > > The AQ command interface is far more programable object than PCI capability
> > > to return this admin info.
> > > > Hence I prefer AQ cmd.
> > >
> > > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > > VFs then at least let us make VF driver in the host simple.
> > It is straight forward.
> >
> > If you prefer the "offset" example of past,
> >
> > If (legacy_offset == queue_notify_offset)
> > *db = guest_supplied_q_notify_content;
> > else
> > virtio_net_send_aq_cmd();
> >
> > "simple" is really a subjective term in this context.
>
> yes this is qemu. sure.
>
> So we have legacy emulation send commands to VF or to PF. Okay. But let
> us avoid the need for VF driver to send commands to PF to initialize.
> Just get all information it needs from VF itself.
>
>
> Maybe it's a good idea to reuse existing notification capability,
> or maybe a new one, but let's avoid making VF driver depend on PF
> commands.
>
>
>
> > > If it has to talk to PF
> > > driver things are really complex. At this point we are very very far from VFIO
> > > model, and then what exactly have we gained by implementing legacy control
> > > path in hardware?
> > It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> >
> > > Let's do software with maybe a couple of features such as
> > > VIRTIO_NET_F_LEGACY_HEADER.
> > >
> > We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
>
> BTW not for this project but generally this is why I thought it is not
> at all a bad idea to have a requirements text document e.g. under
> todo/
> discuss it as normally maybe even vote on including features in a
> plan for a release.
>
> > We cannot iterate exact 100% same points again in this summary discussion.
> >
> > If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
> >
> > What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> > We again past that discussion.
> >
> > This is different requirement, than
> > "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> > Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> > With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
> >
> > This is it.
>
>
> So, I am saying one model is small driver for VF and a big one for PF.
> And to keep the VF driver simple, it should get information simply from
> config space capability.
Think in a different way, if we don't care about the speed of those
slow registers. For the method that accesses the configuration
structure, could we simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
Thanks
>
>
>
> --
> MST
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 0:54 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-22 0:54 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 10:33 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
> >
> > > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > open.org> On Behalf Of Michael S. Tsirkin
> > > Sent: Sunday, May 21, 2023 5:17 AM
> >
> > > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > > One needs to build additional cross forwarding hardware from PF to VF for the
> > > doorbells.
> > >
> > > I think doorbells are driver notifications (linux driver calls them kicks)?
> > > I don't understand what you are saying above really.
> > > what can and what can't be done?
> > >
> > VF has the notification BAR region.
> > All 1.x and legacy such notification lands on the VF BAR.
> >
> > > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > > software model. Attaching a driver to two devices at the same time is hard to
> > > achive e.g. under windows.
> > >
> > Yet you initiate same discussion point that we already discussed again after summarizing.
> > A driver is not attached to two devices.
> > A driver is attached to a single device.
>
> And that device is the owner no? to send commands?
>
> > A device uses a parent/owner group device to access legacy.
> >
> > Software model may evolve over time.
> >
> > >
> > > > And it cannot utilize what already exists for 1.x VF.
> > > >
> > > > > 2. It should be possible to send notifications through an admin
> > > > command too,
> > > > > otherwise admin commands are an incomplete set of functionality.
> > > > >
> > > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > > get any performance.
> > >
> > > Performance might be ok if hardware disables kicks most of the time.
> > >
> > Even for the first kick it is order of magnitude higher.
> > Kicks is the natural tool for the low latency.
> >
> > >
> > > > > 3. I feel using a capability to describe legacy notification
> > > > > area would be better just because we already have a
> > > > > structure for this. make it an express capability if you like.
> > > > The AQ command interface is far more programable object than PCI capability
> > > to return this admin info.
> > > > Hence I prefer AQ cmd.
> > >
> > > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > > VFs then at least let us make VF driver in the host simple.
> > It is straight forward.
> >
> > If you prefer the "offset" example of past,
> >
> > If (legacy_offset == queue_notify_offset)
> > *db = guest_supplied_q_notify_content;
> > else
> > virtio_net_send_aq_cmd();
> >
> > "simple" is really a subjective term in this context.
>
> yes this is qemu. sure.
>
> So we have legacy emulation send commands to VF or to PF. Okay. But let
> us avoid the need for VF driver to send commands to PF to initialize.
> Just get all information it needs from VF itself.
>
>
> Maybe it's a good idea to reuse existing notification capability,
> or maybe a new one, but let's avoid making VF driver depend on PF
> commands.
>
>
>
> > > If it has to talk to PF
> > > driver things are really complex. At this point we are very very far from VFIO
> > > model, and then what exactly have we gained by implementing legacy control
> > > path in hardware?
> > It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> >
> > > Let's do software with maybe a couple of features such as
> > > VIRTIO_NET_F_LEGACY_HEADER.
> > >
> > We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
>
> BTW not for this project but generally this is why I thought it is not
> at all a bad idea to have a requirements text document e.g. under
> todo/
> discuss it as normally maybe even vote on including features in a
> plan for a release.
>
> > We cannot iterate exact 100% same points again in this summary discussion.
> >
> > If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
> >
> > What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> > We again past that discussion.
> >
> > This is different requirement, than
> > "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> > Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> > With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
> >
> > This is it.
>
>
> So, I am saying one model is small driver for VF and a big one for PF.
> And to keep the VF driver simple, it should get information simply from
> config space capability.
Think in a different way, if we don't care about the speed of those
slow registers. For the method that accesses the configuration
structure, could we simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
Thanks
>
>
>
> --
> MST
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 0:54 ` Jason Wang
@ 2023-05-22 2:46 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-22 2:46 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Sunday, May 21, 2023 8:55 PM
>
> Think in a different way, if we don't care about the speed of those slow
> registers. For the method that accesses the configuration structure, could we
> simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
I am familiar with this method which is what we had in v0.
It was a register interface as MMIO area which was just fine.
It has problem with the reset, which also can work if we make the reset register to behave in sane way in this MMIO region.
(slight different than what 0.9.5 had).
Sane way = device must tell that reset is complete (rather than driver assuming)
This was burdensome for the device, specially when the AQ exist to access these slow registers.
This is light weight mechanism for device to implement compared to MMIO or config cap registers.
So we need to decide and agree one way or other, to progress.
This has been lingering for a while now. Let's close on it.
1. legacy register access via AQ (v1)
Pros:
a. Light weight for hypervisor and devices (mainly PCI) to implement.
b. Enables sw to coalesce some device specific registers.
Cons:
a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
2. Legacy registers access via new MMIO region (v0 + sane reset)
Pros:
a. smaller code in slow register access than AQ
b. Sw cannot coalesce some device specific registers.
c. Self-contained in the VF,
Cons:
a. Relatively burdensome for the device as it requires more RW registers at scale.
3. legacy registers tunneling with additional indirection via PCI capability
Cons:
a. transport specific (but not big concern to me)
b. Twice the slow as it requires RW cap.
c. Inferior than #2, still requires sane reset as #2
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 2:46 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-22 2:46 UTC (permalink / raw)
To: Jason Wang, Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, Yishai Hadas,
Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Jason Wang <jasowang@redhat.com>
> Sent: Sunday, May 21, 2023 8:55 PM
>
> Think in a different way, if we don't care about the speed of those slow
> registers. For the method that accesses the configuration structure, could we
> simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
I am familiar with this method which is what we had in v0.
It was a register interface as MMIO area which was just fine.
It has problem with the reset, which also can work if we make the reset register to behave in sane way in this MMIO region.
(slight different than what 0.9.5 had).
Sane way = device must tell that reset is complete (rather than driver assuming)
This was burdensome for the device, specially when the AQ exist to access these slow registers.
This is light weight mechanism for device to implement compared to MMIO or config cap registers.
So we need to decide and agree one way or other, to progress.
This has been lingering for a while now. Let's close on it.
1. legacy register access via AQ (v1)
Pros:
a. Light weight for hypervisor and devices (mainly PCI) to implement.
b. Enables sw to coalesce some device specific registers.
Cons:
a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
2. Legacy registers access via new MMIO region (v0 + sane reset)
Pros:
a. smaller code in slow register access than AQ
b. Sw cannot coalesce some device specific registers.
c. Self-contained in the VF,
Cons:
a. Relatively burdensome for the device as it requires more RW registers at scale.
3. legacy registers tunneling with additional indirection via PCI capability
Cons:
a. transport specific (but not big concern to me)
b. Twice the slow as it requires RW cap.
c. Inferior than #2, still requires sane reset as #2
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 0:54 ` Jason Wang
@ 2023-05-22 19:35 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 19:35 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 08:54:44AM +0800, Jason Wang wrote:
> On Sun, May 21, 2023 at 10:33 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
> > >
> > > > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > open.org> On Behalf Of Michael S. Tsirkin
> > > > Sent: Sunday, May 21, 2023 5:17 AM
> > >
> > > > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > > > One needs to build additional cross forwarding hardware from PF to VF for the
> > > > doorbells.
> > > >
> > > > I think doorbells are driver notifications (linux driver calls them kicks)?
> > > > I don't understand what you are saying above really.
> > > > what can and what can't be done?
> > > >
> > > VF has the notification BAR region.
> > > All 1.x and legacy such notification lands on the VF BAR.
> > >
> > > > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > > > software model. Attaching a driver to two devices at the same time is hard to
> > > > achive e.g. under windows.
> > > >
> > > Yet you initiate same discussion point that we already discussed again after summarizing.
> > > A driver is not attached to two devices.
> > > A driver is attached to a single device.
> >
> > And that device is the owner no? to send commands?
> >
> > > A device uses a parent/owner group device to access legacy.
> > >
> > > Software model may evolve over time.
> > >
> > > >
> > > > > And it cannot utilize what already exists for 1.x VF.
> > > > >
> > > > > > 2. It should be possible to send notifications through an admin
> > > > > command too,
> > > > > > otherwise admin commands are an incomplete set of functionality.
> > > > > >
> > > > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > > > get any performance.
> > > >
> > > > Performance might be ok if hardware disables kicks most of the time.
> > > >
> > > Even for the first kick it is order of magnitude higher.
> > > Kicks is the natural tool for the low latency.
> > >
> > > >
> > > > > > 3. I feel using a capability to describe legacy notification
> > > > > > area would be better just because we already have a
> > > > > > structure for this. make it an express capability if you like.
> > > > > The AQ command interface is far more programable object than PCI capability
> > > > to return this admin info.
> > > > > Hence I prefer AQ cmd.
> > > >
> > > > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > > > VFs then at least let us make VF driver in the host simple.
> > > It is straight forward.
> > >
> > > If you prefer the "offset" example of past,
> > >
> > > If (legacy_offset == queue_notify_offset)
> > > *db = guest_supplied_q_notify_content;
> > > else
> > > virtio_net_send_aq_cmd();
> > >
> > > "simple" is really a subjective term in this context.
> >
> > yes this is qemu. sure.
> >
> > So we have legacy emulation send commands to VF or to PF. Okay. But let
> > us avoid the need for VF driver to send commands to PF to initialize.
> > Just get all information it needs from VF itself.
> >
> >
> > Maybe it's a good idea to reuse existing notification capability,
> > or maybe a new one, but let's avoid making VF driver depend on PF
> > commands.
> >
> >
> >
> > > > If it has to talk to PF
> > > > driver things are really complex. At this point we are very very far from VFIO
> > > > model, and then what exactly have we gained by implementing legacy control
> > > > path in hardware?
> > > It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> > >
> > > > Let's do software with maybe a couple of features such as
> > > > VIRTIO_NET_F_LEGACY_HEADER.
> > > >
> > > We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
> >
> > BTW not for this project but generally this is why I thought it is not
> > at all a bad idea to have a requirements text document e.g. under
> > todo/
> > discuss it as normally maybe even vote on including features in a
> > plan for a release.
> >
> > > We cannot iterate exact 100% same points again in this summary discussion.
> > >
> > > If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
> > >
> > > What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> > > We again past that discussion.
> > >
> > > This is different requirement, than
> > > "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> > > Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> > > With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
> > >
> > > This is it.
> >
> >
> > So, I am saying one model is small driver for VF and a big one for PF.
> > And to keep the VF driver simple, it should get information simply from
> > config space capability.
>
> Think in a different way, if we don't care about the speed of those
> slow registers. For the method that accesses the configuration
> structure, could we simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
>
> Thanks
Yes this is what I suggested.
> >
> >
> >
> > --
> > MST
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 19:35 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 19:35 UTC (permalink / raw)
To: Jason Wang
Cc: Parav Pandit, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 08:54:44AM +0800, Jason Wang wrote:
> On Sun, May 21, 2023 at 10:33 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, May 21, 2023 at 01:21:17PM +0000, Parav Pandit wrote:
> > >
> > > > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > open.org> On Behalf Of Michael S. Tsirkin
> > > > Sent: Sunday, May 21, 2023 5:17 AM
> > >
> > > > > The Notification of the VFs are on the VF BAR for modern or legacy.
> > > > > One needs to build additional cross forwarding hardware from PF to VF for the
> > > > doorbells.
> > > >
> > > > I think doorbells are driver notifications (linux driver calls them kicks)?
> > > > I don't understand what you are saying above really.
> > > > what can and what can't be done?
> > > >
> > > VF has the notification BAR region.
> > > All 1.x and legacy such notification lands on the VF BAR.
> > >
> > > > Again all this idea (as opposed to Jason's transport vq) is to have a simple
> > > > software model. Attaching a driver to two devices at the same time is hard to
> > > > achive e.g. under windows.
> > > >
> > > Yet you initiate same discussion point that we already discussed again after summarizing.
> > > A driver is not attached to two devices.
> > > A driver is attached to a single device.
> >
> > And that device is the owner no? to send commands?
> >
> > > A device uses a parent/owner group device to access legacy.
> > >
> > > Software model may evolve over time.
> > >
> > > >
> > > > > And it cannot utilize what already exists for 1.x VF.
> > > > >
> > > > > > 2. It should be possible to send notifications through an admin
> > > > > command too,
> > > > > > otherwise admin commands are an incomplete set of functionality.
> > > > > >
> > > > > Yes. it is only for the functionality. As we discussed in past already, this will not
> > > > get any performance.
> > > >
> > > > Performance might be ok if hardware disables kicks most of the time.
> > > >
> > > Even for the first kick it is order of magnitude higher.
> > > Kicks is the natural tool for the low latency.
> > >
> > > >
> > > > > > 3. I feel using a capability to describe legacy notification
> > > > > > area would be better just because we already have a
> > > > > > structure for this. make it an express capability if you like.
> > > > > The AQ command interface is far more programable object than PCI capability
> > > > to return this admin info.
> > > > > Hence I prefer AQ cmd.
> > > >
> > > > I feel your preferences for 1 and 3 conflict. If you really insist on kicks being on
> > > > VFs then at least let us make VF driver in the host simple.
> > > It is straight forward.
> > >
> > > If you prefer the "offset" example of past,
> > >
> > > If (legacy_offset == queue_notify_offset)
> > > *db = guest_supplied_q_notify_content;
> > > else
> > > virtio_net_send_aq_cmd();
> > >
> > > "simple" is really a subjective term in this context.
> >
> > yes this is qemu. sure.
> >
> > So we have legacy emulation send commands to VF or to PF. Okay. But let
> > us avoid the need for VF driver to send commands to PF to initialize.
> > Just get all information it needs from VF itself.
> >
> >
> > Maybe it's a good idea to reuse existing notification capability,
> > or maybe a new one, but let's avoid making VF driver depend on PF
> > commands.
> >
> >
> >
> > > > If it has to talk to PF
> > > > driver things are really complex. At this point we are very very far from VFIO
> > > > model, and then what exactly have we gained by implementing legacy control
> > > > path in hardware?
> > > It is in the VFIO model, legacy largely falls in the special path for backward compat for the hw devices.
> > >
> > > > Let's do software with maybe a couple of features such as
> > > > VIRTIO_NET_F_LEGACY_HEADER.
> > > >
> > > We have already these design choices and tradeoff in v0 and v1, it doesn't fit the requirements.
> >
> > BTW not for this project but generally this is why I thought it is not
> > at all a bad idea to have a requirements text document e.g. under
> > todo/
> > discuss it as normally maybe even vote on including features in a
> > plan for a release.
> >
> > > We cannot iterate exact 100% same points again in this summary discussion.
> > >
> > > If you have any different comments from v0 and v1, please share, otherwise these commands should proceed.
> > >
> > > What you hint is saying hey, "transport everything through the PF, then why do you have a VF?"
> > > We again past that discussion.
> > >
> > > This is different requirement, than
> > > "There is a VF passthrough accessible directly from the guest without a VI, some of them need to have legacy support also".
> > > Hence how do one make that legacy guest utilize it the VF in passthrough manner for 1.x and 0.9.5.
> > > With this proposal 0.9.5 slow registers are accessed over the owner group admin PF keeping the whole model and functionality intact.
> > >
> > > This is it.
> >
> >
> > So, I am saying one model is small driver for VF and a big one for PF.
> > And to keep the VF driver simple, it should get information simply from
> > config space capability.
>
> Think in a different way, if we don't care about the speed of those
> slow registers. For the method that accesses the configuration
> structure, could we simply extend VIRTIO_PCI_CAP_PCI_CFG capability?
>
> Thanks
Yes this is what I suggested.
> >
> >
> >
> > --
> > MST
> >
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-21 14:44 ` [virtio-dev] " Parav Pandit
@ 2023-05-22 20:07 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 20:07 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Sunday, May 21, 2023 10:33 AM
>
> > > Yet you initiate same discussion point that we already discussed again after
> > summarizing.
> > > A driver is not attached to two devices.
> > > A driver is attached to a single device.
> >
> > And that device is the owner no? to send commands?
> >
> Not for legacy registers access as discussed before.
Now I am lost. legacy register access is on top of aq which
sends commands through pf. how are you going to send
commands without attaching a driver, I don't know.
> > > If (legacy_offset == queue_notify_offset)
> > > *db = guest_supplied_q_notify_content; else
> > > virtio_net_send_aq_cmd();
> > >
> > > "simple" is really a subjective term in this context.
> >
> > yes this is qemu. sure.
> >
> Not limited to QEMU.
> A driver will be able to do this as well.
No, on some OS-es (windows, iokit, more) a single driver can't bind to
many devices without a lot of pain. If you are mapping VF BAR, this has
to be done by VF driver, and please just avoid pain for everyone by
exposing the necessary info through the VF itself.
A capability seems perfect for this.
An alternative is not to map VF BAR at all, steal space from PF BAR instead.
Guest is not accessing this anyway. So I really do not see why not
from software point of view. There's a hardware reason? Could you
talk to hardware guys about this? You objected to AQ too then
hardware guys told you it is not a big deal.
> > So we have legacy emulation send commands to VF or to PF. Okay. But let us
> > avoid the need for VF driver to send commands to PF to initialize.
> > Just get all information it needs from VF itself.
> >
> >
> > Maybe it's a good idea to reuse existing notification capability, or maybe a new
> > one, but let's avoid making VF driver depend on PF commands.
> >
> We agreed in v1 on Jason's suggestion to have the AQ command and yet
> you and Jason hinder this in v2 with this exact repeated question.
> Lets please avoid this and move forward.
How was I supposed to raise this issue on v1 if it was not there?
Stop arguing in circles and things will move forward.
If we have trouble converging on the notification thing, how about we
make progress without it for now? Merge a slow version that sends kicks
through aq, then work on the best way to make notifications faster,
separately.
> > > We have already these design choices and tradeoff in v0 and v1, it doesn't fit
> > the requirements.
> >
>
> > So, I am saying one model is small driver for VF and a big one for PF.
> > And to keep the VF driver simple, it should get information simply from config
> > space capability.
>
> VF driver is small that does usual vfio passthrough work.
> PF driver implement AQ for variety of use cases that we listed in the AQ cover letter.
> VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
VF(member) driver can't implement AQ commands at all. They are sent to
PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
legacy access and talk to owner driver to send them. It can also talk
to member driver to send notifications. You can I guess have VMM get
member BAR offsets from owner and pass them to member for mapping. This
raises all kind of questions of trust.
If you can map BAR from PF that would be simplest, VMM then only pokes
at PF and not VF. If not then we really should expose this info
n the VF if at all possible.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 20:07 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 20:07 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Sunday, May 21, 2023 10:33 AM
>
> > > Yet you initiate same discussion point that we already discussed again after
> > summarizing.
> > > A driver is not attached to two devices.
> > > A driver is attached to a single device.
> >
> > And that device is the owner no? to send commands?
> >
> Not for legacy registers access as discussed before.
Now I am lost. legacy register access is on top of aq which
sends commands through pf. how are you going to send
commands without attaching a driver, I don't know.
> > > If (legacy_offset == queue_notify_offset)
> > > *db = guest_supplied_q_notify_content; else
> > > virtio_net_send_aq_cmd();
> > >
> > > "simple" is really a subjective term in this context.
> >
> > yes this is qemu. sure.
> >
> Not limited to QEMU.
> A driver will be able to do this as well.
No, on some OS-es (windows, iokit, more) a single driver can't bind to
many devices without a lot of pain. If you are mapping VF BAR, this has
to be done by VF driver, and please just avoid pain for everyone by
exposing the necessary info through the VF itself.
A capability seems perfect for this.
An alternative is not to map VF BAR at all, steal space from PF BAR instead.
Guest is not accessing this anyway. So I really do not see why not
from software point of view. There's a hardware reason? Could you
talk to hardware guys about this? You objected to AQ too then
hardware guys told you it is not a big deal.
> > So we have legacy emulation send commands to VF or to PF. Okay. But let us
> > avoid the need for VF driver to send commands to PF to initialize.
> > Just get all information it needs from VF itself.
> >
> >
> > Maybe it's a good idea to reuse existing notification capability, or maybe a new
> > one, but let's avoid making VF driver depend on PF commands.
> >
> We agreed in v1 on Jason's suggestion to have the AQ command and yet
> you and Jason hinder this in v2 with this exact repeated question.
> Lets please avoid this and move forward.
How was I supposed to raise this issue on v1 if it was not there?
Stop arguing in circles and things will move forward.
If we have trouble converging on the notification thing, how about we
make progress without it for now? Merge a slow version that sends kicks
through aq, then work on the best way to make notifications faster,
separately.
> > > We have already these design choices and tradeoff in v0 and v1, it doesn't fit
> > the requirements.
> >
>
> > So, I am saying one model is small driver for VF and a big one for PF.
> > And to keep the VF driver simple, it should get information simply from config
> > space capability.
>
> VF driver is small that does usual vfio passthrough work.
> PF driver implement AQ for variety of use cases that we listed in the AQ cover letter.
> VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
VF(member) driver can't implement AQ commands at all. They are sent to
PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
legacy access and talk to owner driver to send them. It can also talk
to member driver to send notifications. You can I guess have VMM get
member BAR offsets from owner and pass them to member for mapping. This
raises all kind of questions of trust.
If you can map BAR from PF that would be simplest, VMM then only pokes
at PF and not VF. If not then we really should expose this info
n the VF if at all possible.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 20:07 ` Michael S. Tsirkin
@ 2023-05-22 21:05 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-22 21:05 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 22, 2023 4:07 PM
>
> On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
> >
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Sunday, May 21, 2023 10:33 AM
> >
> > > > Yet you initiate same discussion point that we already discussed
> > > > again after
> > > summarizing.
> > > > A driver is not attached to two devices.
> > > > A driver is attached to a single device.
> > >
> > > And that device is the owner no? to send commands?
> > >
> > Not for legacy registers access as discussed before.
>
> Now I am lost. legacy register access is on top of aq which sends commands
> through pf.
> how are you going to send commands without attaching a driver, I
> don't know.
A VF driver can send command to its parent PF driver using well defined kernel API.
There is no need to attach PF driver.
Upper and lower devices are common concept.
> > Not limited to QEMU.
> > A driver will be able to do this as well.
>
> No, on some OS-es (windows, iokit, more) a single driver can't bind to many
> devices without a lot of pain.
There is no need to bind. A well-defined kernel API is enough.
It's been while I worked on iokit, but iokit has similar kernel APIs.
It is very exciting to think that one has built iokit based hypervisor and hosting PCI accelerated device and VMs in it.
Last time when I checked, I was told that virtio driver is not present in the latest OS, not sure how true it is.
> If you are mapping VF BAR, this has to be done by
> VF driver, and please just avoid pain for everyone by exposing the necessary
> info through the VF itself.
> A capability seems perfect for this.
>
Why not the MMIO region as proposed in v0?
Capability based read/write requires extra redirection, it is less optimal than MMIO.
> An alternative is not to map VF BAR at all, steal space from PF BAR instead.
Come an iokit will be able to access this PF BAR without binding to it?
> Guest is not accessing this anyway. So I really do not see why not from software
> point of view. There's a hardware reason? Could you talk to hardware guys
> about this? You objected to AQ too then hardware guys told you it is not a big
> deal.
I didn't object to AQ on hw reason.
I explained AQ is not the best compared to MMIO because it is slow compared to MMIO by law of physics as it needs to move more bits than MMIO.
And so, the capability based tunneling too is slow to MMIO.
>
> > > So we have legacy emulation send commands to VF or to PF. Okay. But
> > > let us avoid the need for VF driver to send commands to PF to initialize.
> > > Just get all information it needs from VF itself.
> > >
> > >
> > > Maybe it's a good idea to reuse existing notification capability, or
> > > maybe a new one, but let's avoid making VF driver depend on PF
> commands.
> > >
> > We agreed in v1 on Jason's suggestion to have the AQ command and yet
> > you and Jason hinder this in v2 with this exact repeated question.
> > Lets please avoid this and move forward.
>
> How was I supposed to raise this issue on v1 if it was not there?
> Stop arguing in circles and things will move forward.
v1 version clearly had AQ commands.
Please revisit your comments.
v0 -> you suggested:
- "why not AQ, like transport vq, it can coalesce partial mac writes".
v1-> over AQ, you suggested:
- to "drop the size"
- summarized to split 2 commands to 4 commands to split device and config area.
After that summary, you propose, why not self-contained in VF itself? -> circle back to v0 method.
(Instead of mmio, suggesting capability, but not explaining why mmio is not fine. Which is better than cap).
Then now you propose "have BAR in the PF which is owned by other PF driver", with least reasoning.
And it convolutes with your suggestion of PF driver access...
So, I am bit lost on your circular proposal, or it was just a probe question...
Not sure.
>
> If we have trouble converging on the notification thing, how about we make
> progress without it for now? Merge a slow version that sends kicks through aq,
> then work on the best way to make notifications faster, separately.
>
Sure, we can differ, but the VF has the notification BAR to utilize.
So why not use it?
>
> > > > We have already these design choices and tradeoff in v0 and v1, it
> > > > doesn't fit
> > > the requirements.
> > >
> >
> > > So, I am saying one model is small driver for VF and a big one for PF.
> > > And to keep the VF driver simple, it should get information simply
> > > from config space capability.
> >
> > VF driver is small that does usual vfio passthrough work.
> > PF driver implement AQ for variety of use cases that we listed in the AQ
> cover letter.
> > VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
>
> VF(member) driver can't implement AQ commands at all. They are sent to
> PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap legacy access
> and talk to owner driver to send them. It can also talk to member driver to
> send notifications. You can I guess have VMM get member BAR offsets from
> owner and pass them to member for mapping. This raises all kind of questions
> of trust.
Only the member driver has access to the owner driver via kernel API.
Part of the VF BAR information at the PCI level is also on the PF.
There is no trust break here. A kernel exposes single self-contained object to user space VMM.
> If you can map BAR from PF that would be simplest, VMM then only pokes at PF
> and not VF. If not then we really should expose this info n the VF if at all
> possible.
>
There is clearly no need to use PF for notification when the notification method already exists on the VF.
In fact, it is yet extra steering entry for the hw to perform such mapping.
To iterate what presented yday in [1] to Jason.
[1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.html
1. legacy register access via AQ (v1)
Pros:
a. Light weight for hypervisor and devices (mainly PCI) to implement.
b. Enables sw to coalesce some device specific registers, if needed.
Cons:
a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
2. Legacy registers access via new MMIO region (v0 + sane reset)
Pros:
a. smaller code in slow register access than AQ
b. Sw cannot coalesce some device specific registers.
c. Self-contained in the VF
Cons:
a. Relatively burdensome for the device as it requires more RW registers at scale, but it can be indirect register.
3. legacy registers tunneling with additional indirection via PCI capability
Cons:
a. transport specific (but not big concern to me)
b. Twice the slow as it requires RW cap.
c. Inferior to #2, still requires sane reset as #2
d. MMIO indirect registers are better than capabilities because PCI caps are largely for read only purpose.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 21:05 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-22 21:05 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 22, 2023 4:07 PM
>
> On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
> >
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Sunday, May 21, 2023 10:33 AM
> >
> > > > Yet you initiate same discussion point that we already discussed
> > > > again after
> > > summarizing.
> > > > A driver is not attached to two devices.
> > > > A driver is attached to a single device.
> > >
> > > And that device is the owner no? to send commands?
> > >
> > Not for legacy registers access as discussed before.
>
> Now I am lost. legacy register access is on top of aq which sends commands
> through pf.
> how are you going to send commands without attaching a driver, I
> don't know.
A VF driver can send command to its parent PF driver using well defined kernel API.
There is no need to attach PF driver.
Upper and lower devices are common concept.
> > Not limited to QEMU.
> > A driver will be able to do this as well.
>
> No, on some OS-es (windows, iokit, more) a single driver can't bind to many
> devices without a lot of pain.
There is no need to bind. A well-defined kernel API is enough.
It's been while I worked on iokit, but iokit has similar kernel APIs.
It is very exciting to think that one has built iokit based hypervisor and hosting PCI accelerated device and VMs in it.
Last time when I checked, I was told that virtio driver is not present in the latest OS, not sure how true it is.
> If you are mapping VF BAR, this has to be done by
> VF driver, and please just avoid pain for everyone by exposing the necessary
> info through the VF itself.
> A capability seems perfect for this.
>
Why not the MMIO region as proposed in v0?
Capability based read/write requires extra redirection, it is less optimal than MMIO.
> An alternative is not to map VF BAR at all, steal space from PF BAR instead.
Come an iokit will be able to access this PF BAR without binding to it?
> Guest is not accessing this anyway. So I really do not see why not from software
> point of view. There's a hardware reason? Could you talk to hardware guys
> about this? You objected to AQ too then hardware guys told you it is not a big
> deal.
I didn't object to AQ on hw reason.
I explained AQ is not the best compared to MMIO because it is slow compared to MMIO by law of physics as it needs to move more bits than MMIO.
And so, the capability based tunneling too is slow to MMIO.
>
> > > So we have legacy emulation send commands to VF or to PF. Okay. But
> > > let us avoid the need for VF driver to send commands to PF to initialize.
> > > Just get all information it needs from VF itself.
> > >
> > >
> > > Maybe it's a good idea to reuse existing notification capability, or
> > > maybe a new one, but let's avoid making VF driver depend on PF
> commands.
> > >
> > We agreed in v1 on Jason's suggestion to have the AQ command and yet
> > you and Jason hinder this in v2 with this exact repeated question.
> > Lets please avoid this and move forward.
>
> How was I supposed to raise this issue on v1 if it was not there?
> Stop arguing in circles and things will move forward.
v1 version clearly had AQ commands.
Please revisit your comments.
v0 -> you suggested:
- "why not AQ, like transport vq, it can coalesce partial mac writes".
v1-> over AQ, you suggested:
- to "drop the size"
- summarized to split 2 commands to 4 commands to split device and config area.
After that summary, you propose, why not self-contained in VF itself? -> circle back to v0 method.
(Instead of mmio, suggesting capability, but not explaining why mmio is not fine. Which is better than cap).
Then now you propose "have BAR in the PF which is owned by other PF driver", with least reasoning.
And it convolutes with your suggestion of PF driver access...
So, I am bit lost on your circular proposal, or it was just a probe question...
Not sure.
>
> If we have trouble converging on the notification thing, how about we make
> progress without it for now? Merge a slow version that sends kicks through aq,
> then work on the best way to make notifications faster, separately.
>
Sure, we can differ, but the VF has the notification BAR to utilize.
So why not use it?
>
> > > > We have already these design choices and tradeoff in v0 and v1, it
> > > > doesn't fit
> > > the requirements.
> > >
> >
> > > So, I am saying one model is small driver for VF and a big one for PF.
> > > And to keep the VF driver simple, it should get information simply
> > > from config space capability.
> >
> > VF driver is small that does usual vfio passthrough work.
> > PF driver implement AQ for variety of use cases that we listed in the AQ
> cover letter.
> > VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
>
> VF(member) driver can't implement AQ commands at all. They are sent to
> PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap legacy access
> and talk to owner driver to send them. It can also talk to member driver to
> send notifications. You can I guess have VMM get member BAR offsets from
> owner and pass them to member for mapping. This raises all kind of questions
> of trust.
Only the member driver has access to the owner driver via kernel API.
Part of the VF BAR information at the PCI level is also on the PF.
There is no trust break here. A kernel exposes single self-contained object to user space VMM.
> If you can map BAR from PF that would be simplest, VMM then only pokes at PF
> and not VF. If not then we really should expose this info n the VF if at all
> possible.
>
There is clearly no need to use PF for notification when the notification method already exists on the VF.
In fact, it is yet extra steering entry for the hw to perform such mapping.
To iterate what presented yday in [1] to Jason.
[1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.html
1. legacy register access via AQ (v1)
Pros:
a. Light weight for hypervisor and devices (mainly PCI) to implement.
b. Enables sw to coalesce some device specific registers, if needed.
Cons:
a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
2. Legacy registers access via new MMIO region (v0 + sane reset)
Pros:
a. smaller code in slow register access than AQ
b. Sw cannot coalesce some device specific registers.
c. Self-contained in the VF
Cons:
a. Relatively burdensome for the device as it requires more RW registers at scale, but it can be indirect register.
3. legacy registers tunneling with additional indirection via PCI capability
Cons:
a. transport specific (but not big concern to me)
b. Twice the slow as it requires RW cap.
c. Inferior to #2, still requires sane reset as #2
d. MMIO indirect registers are better than capabilities because PCI caps are largely for read only purpose.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 21:05 ` Parav Pandit
@ 2023-05-22 21:34 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 21:34 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 09:05:13PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 22, 2023 4:07 PM
> >
> > On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
> > >
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Sunday, May 21, 2023 10:33 AM
> > >
> > > > > Yet you initiate same discussion point that we already discussed
> > > > > again after
> > > > summarizing.
> > > > > A driver is not attached to two devices.
> > > > > A driver is attached to a single device.
> > > >
> > > > And that device is the owner no? to send commands?
> > > >
> > > Not for legacy registers access as discussed before.
> >
> > Now I am lost. legacy register access is on top of aq which sends commands
> > through pf.
>
> > how are you going to send commands without attaching a driver, I
> > don't know.
>
> A VF driver can send command to its parent PF driver using well defined kernel API.
> There is no need to attach PF driver.
> Upper and lower devices are common concept.
> > > Not limited to QEMU.
> > > A driver will be able to do this as well.
> >
> > No, on some OS-es (windows, iokit, more) a single driver can't bind to many
> > devices without a lot of pain.
> There is no need to bind. A well-defined kernel API is enough.
>
> It's been while I worked on iokit, but iokit has similar kernel APIs.
>
> It is very exciting to think that one has built iokit based hypervisor and hosting PCI accelerated device and VMs in it.
> Last time when I checked, I was told that virtio driver is not present in the latest OS, not sure how true it is.
iokit is from my previous work experience not virtio.
For virtio we had a lot of trouble with hacks like this on windows.
That's an important platform.
> > If you are mapping VF BAR, this has to be done by
> > VF driver, and please just avoid pain for everyone by exposing the necessary
> > info through the VF itself.
> > A capability seems perfect for this.
> >
> Why not the MMIO region as proposed in v0?
Are you asking what were issues with v0? The issue with v0 was not MMIO
as such at least for me. The issue was that it was not giving us any
way to work around bugs in the legacy interface. AQ was a fix for that.
> Capability based read/write requires extra redirection, it is less optimal than MMIO.
>
> > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> Come an iokit will be able to access this PF BAR without binding to it?
I don't get what you are saying here. You have to bind a PF driver.
Otherwise you can not send AQ command. That driver can map PF BAR into
applications.
> > Guest is not accessing this anyway. So I really do not see why not from software
> > point of view. There's a hardware reason? Could you talk to hardware guys
> > about this? You objected to AQ too then hardware guys told you it is not a big
> > deal.
> I didn't object to AQ on hw reason.
> I explained AQ is not the best compared to MMIO because it is slow compared to MMIO by law of physics as it needs to move more bits than MMIO.
> And so, the capability based tunneling too is slow to MMIO.
> >
> > > > So we have legacy emulation send commands to VF or to PF. Okay. But
> > > > let us avoid the need for VF driver to send commands to PF to initialize.
> > > > Just get all information it needs from VF itself.
> > > >
> > > >
> > > > Maybe it's a good idea to reuse existing notification capability, or
> > > > maybe a new one, but let's avoid making VF driver depend on PF
> > commands.
> > > >
> > > We agreed in v1 on Jason's suggestion to have the AQ command and yet
> > > you and Jason hinder this in v2 with this exact repeated question.
> > > Lets please avoid this and move forward.
> >
> > How was I supposed to raise this issue on v1 if it was not there?
> > Stop arguing in circles and things will move forward.
> v1 version clearly had AQ commands.
>
> Please revisit your comments.
> v0 -> you suggested:
> - "why not AQ, like transport vq, it can coalesce partial mac writes".
>
> v1-> over AQ, you suggested:
> - to "drop the size"
> - summarized to split 2 commands to 4 commands to split device and config area.
>
> After that summary, you propose, why not self-contained in VF itself? -> circle back to v0 method.
> (Instead of mmio, suggesting capability, but not explaining why mmio is not fine. Which is better than cap).
>
> Then now you propose "have BAR in the PF which is owned by other PF driver", with least reasoning.
> And it convolutes with your suggestion of PF driver access...
>
> So, I am bit lost on your circular proposal, or it was just a probe question...
> Not sure.
You did move things to AQ however not completely. You kept notification
in MMIO. If everything was in AQ we would be done as far as I am
concerned (though Jason still does not like duplication of functionality
with transport vq, but that is a separate issue).
As notification is in VF MMIO I am trying to help you by suggesting
other ways to keep things consistent. If it's in VF I think it follows
that where it is has to be in VF as well. Not in AQ.
And look if that just serves to annoy I can just point out issues and
complain without suggestions. Was trying to be helpful here really.
>
> >
> > If we have trouble converging on the notification thing, how about we make
> > progress without it for now? Merge a slow version that sends kicks through aq,
> > then work on the best way to make notifications faster, separately.
> >
> Sure, we can differ, but the VF has the notification BAR to utilize.
> So why not use it?
Are you asking why is merging a smaller partial patch then adding
functionality a good idea?
> >
> > > > > We have already these design choices and tradeoff in v0 and v1, it
> > > > > doesn't fit
> > > > the requirements.
> > > >
> > >
> > > > So, I am saying one model is small driver for VF and a big one for PF.
> > > > And to keep the VF driver simple, it should get information simply
> > > > from config space capability.
> > >
> > > VF driver is small that does usual vfio passthrough work.
> > > PF driver implement AQ for variety of use cases that we listed in the AQ
> > cover letter.
> > > VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
> >
> > VF(member) driver can't implement AQ commands at all. They are sent to
> > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap legacy access
> > and talk to owner driver to send them. It can also talk to member driver to
> > send notifications. You can I guess have VMM get member BAR offsets from
> > owner and pass them to member for mapping. This raises all kind of questions
> > of trust.
> Only the member driver has access to the owner driver via kernel API.
Assuming you can predict existance of specific kernel APIs on all OSes
is IMO a fool's errand. One of the things virtio is supposed to be about
is precisely simple straight forward design for drivers. Which this is
not.
> Part of the VF BAR information at the PCI level is also on the PF.
> There is no trust break here. A kernel exposes single self-contained object to user space VMM.
That is ok. The trust break is if PF driver is in userspace. You then
are trusting userspace to pass offsets back to kernel VF driver.
> > If you can map BAR from PF that would be simplest, VMM then only pokes at PF
> > and not VF. If not then we really should expose this info n the VF if at all
> > possible.
> >
> There is clearly no need to use PF for notification when the notification method already exists on the VF.
All this looks backwards with mixed up functionality. What VF has is
modern BAR. If you insist on reusing modern functionality for legacy
when it suits you then maybe we should reuse it for everything, that
would be extending transport vq to support legacy somehow.
> In fact, it is yet extra steering entry for the hw to perform such mapping.
>
> To iterate what presented yday in [1] to Jason.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.html
>
> 1. legacy register access via AQ (v1)
> Pros:
> a. Light weight for hypervisor and devices (mainly PCI) to implement.
> b. Enables sw to coalesce some device specific registers, if needed.
> Cons:
> a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
>
> 2. Legacy registers access via new MMIO region (v0 + sane reset)
I think I would buy this one if it was actually in BAR0 exactly legacy
with no changes, no reset no nothing. a hack like Jason described that
at least makes legacy linux drivers work with no software glue. That is
horrible, fragile but engineering history is full of hacks like this.
If you need software in the middle then let's please make it robust.
> Pros:
> a. smaller code in slow register access than AQ
> b. Sw cannot coalesce some device specific registers.
> c. Self-contained in the VF
>
> Cons:
> a. Relatively burdensome for the device as it requires more RW registers at scale, but it can be indirect register.
what I keep saying
b. a ton of legacy bugs which hypervisor can't detect or work around
(reset is just one issue)
> 3. legacy registers tunneling with additional indirection via PCI capability
I don't know what's this one.
> Cons:
> a. transport specific (but not big concern to me)
> b. Twice the slow as it requires RW cap.
> c. Inferior to #2, still requires sane reset as #2
> d. MMIO indirect registers are better than capabilities because PCI caps are largely for read only purpose.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 21:34 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 21:34 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 09:05:13PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 22, 2023 4:07 PM
> >
> > On Sun, May 21, 2023 at 02:44:03PM +0000, Parav Pandit wrote:
> > >
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Sunday, May 21, 2023 10:33 AM
> > >
> > > > > Yet you initiate same discussion point that we already discussed
> > > > > again after
> > > > summarizing.
> > > > > A driver is not attached to two devices.
> > > > > A driver is attached to a single device.
> > > >
> > > > And that device is the owner no? to send commands?
> > > >
> > > Not for legacy registers access as discussed before.
> >
> > Now I am lost. legacy register access is on top of aq which sends commands
> > through pf.
>
> > how are you going to send commands without attaching a driver, I
> > don't know.
>
> A VF driver can send command to its parent PF driver using well defined kernel API.
> There is no need to attach PF driver.
> Upper and lower devices are common concept.
> > > Not limited to QEMU.
> > > A driver will be able to do this as well.
> >
> > No, on some OS-es (windows, iokit, more) a single driver can't bind to many
> > devices without a lot of pain.
> There is no need to bind. A well-defined kernel API is enough.
>
> It's been while I worked on iokit, but iokit has similar kernel APIs.
>
> It is very exciting to think that one has built iokit based hypervisor and hosting PCI accelerated device and VMs in it.
> Last time when I checked, I was told that virtio driver is not present in the latest OS, not sure how true it is.
iokit is from my previous work experience not virtio.
For virtio we had a lot of trouble with hacks like this on windows.
That's an important platform.
> > If you are mapping VF BAR, this has to be done by
> > VF driver, and please just avoid pain for everyone by exposing the necessary
> > info through the VF itself.
> > A capability seems perfect for this.
> >
> Why not the MMIO region as proposed in v0?
Are you asking what were issues with v0? The issue with v0 was not MMIO
as such at least for me. The issue was that it was not giving us any
way to work around bugs in the legacy interface. AQ was a fix for that.
> Capability based read/write requires extra redirection, it is less optimal than MMIO.
>
> > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> Come an iokit will be able to access this PF BAR without binding to it?
I don't get what you are saying here. You have to bind a PF driver.
Otherwise you can not send AQ command. That driver can map PF BAR into
applications.
> > Guest is not accessing this anyway. So I really do not see why not from software
> > point of view. There's a hardware reason? Could you talk to hardware guys
> > about this? You objected to AQ too then hardware guys told you it is not a big
> > deal.
> I didn't object to AQ on hw reason.
> I explained AQ is not the best compared to MMIO because it is slow compared to MMIO by law of physics as it needs to move more bits than MMIO.
> And so, the capability based tunneling too is slow to MMIO.
> >
> > > > So we have legacy emulation send commands to VF or to PF. Okay. But
> > > > let us avoid the need for VF driver to send commands to PF to initialize.
> > > > Just get all information it needs from VF itself.
> > > >
> > > >
> > > > Maybe it's a good idea to reuse existing notification capability, or
> > > > maybe a new one, but let's avoid making VF driver depend on PF
> > commands.
> > > >
> > > We agreed in v1 on Jason's suggestion to have the AQ command and yet
> > > you and Jason hinder this in v2 with this exact repeated question.
> > > Lets please avoid this and move forward.
> >
> > How was I supposed to raise this issue on v1 if it was not there?
> > Stop arguing in circles and things will move forward.
> v1 version clearly had AQ commands.
>
> Please revisit your comments.
> v0 -> you suggested:
> - "why not AQ, like transport vq, it can coalesce partial mac writes".
>
> v1-> over AQ, you suggested:
> - to "drop the size"
> - summarized to split 2 commands to 4 commands to split device and config area.
>
> After that summary, you propose, why not self-contained in VF itself? -> circle back to v0 method.
> (Instead of mmio, suggesting capability, but not explaining why mmio is not fine. Which is better than cap).
>
> Then now you propose "have BAR in the PF which is owned by other PF driver", with least reasoning.
> And it convolutes with your suggestion of PF driver access...
>
> So, I am bit lost on your circular proposal, or it was just a probe question...
> Not sure.
You did move things to AQ however not completely. You kept notification
in MMIO. If everything was in AQ we would be done as far as I am
concerned (though Jason still does not like duplication of functionality
with transport vq, but that is a separate issue).
As notification is in VF MMIO I am trying to help you by suggesting
other ways to keep things consistent. If it's in VF I think it follows
that where it is has to be in VF as well. Not in AQ.
And look if that just serves to annoy I can just point out issues and
complain without suggestions. Was trying to be helpful here really.
>
> >
> > If we have trouble converging on the notification thing, how about we make
> > progress without it for now? Merge a slow version that sends kicks through aq,
> > then work on the best way to make notifications faster, separately.
> >
> Sure, we can differ, but the VF has the notification BAR to utilize.
> So why not use it?
Are you asking why is merging a smaller partial patch then adding
functionality a good idea?
> >
> > > > > We have already these design choices and tradeoff in v0 and v1, it
> > > > > doesn't fit
> > > > the requirements.
> > > >
> > >
> > > > So, I am saying one model is small driver for VF and a big one for PF.
> > > > And to keep the VF driver simple, it should get information simply
> > > > from config space capability.
> > >
> > > VF driver is small that does usual vfio passthrough work.
> > > PF driver implement AQ for variety of use cases that we listed in the AQ
> > cover letter.
> > > VF driver implements 5 AQ commands that you suggested to split from 2 to 4.
> >
> > VF(member) driver can't implement AQ commands at all. They are sent to
> > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap legacy access
> > and talk to owner driver to send them. It can also talk to member driver to
> > send notifications. You can I guess have VMM get member BAR offsets from
> > owner and pass them to member for mapping. This raises all kind of questions
> > of trust.
> Only the member driver has access to the owner driver via kernel API.
Assuming you can predict existance of specific kernel APIs on all OSes
is IMO a fool's errand. One of the things virtio is supposed to be about
is precisely simple straight forward design for drivers. Which this is
not.
> Part of the VF BAR information at the PCI level is also on the PF.
> There is no trust break here. A kernel exposes single self-contained object to user space VMM.
That is ok. The trust break is if PF driver is in userspace. You then
are trusting userspace to pass offsets back to kernel VF driver.
> > If you can map BAR from PF that would be simplest, VMM then only pokes at PF
> > and not VF. If not then we really should expose this info n the VF if at all
> > possible.
> >
> There is clearly no need to use PF for notification when the notification method already exists on the VF.
All this looks backwards with mixed up functionality. What VF has is
modern BAR. If you insist on reusing modern functionality for legacy
when it suits you then maybe we should reuse it for everything, that
would be extending transport vq to support legacy somehow.
> In fact, it is yet extra steering entry for the hw to perform such mapping.
>
> To iterate what presented yday in [1] to Jason.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.html
>
> 1. legacy register access via AQ (v1)
> Pros:
> a. Light weight for hypervisor and devices (mainly PCI) to implement.
> b. Enables sw to coalesce some device specific registers, if needed.
> Cons:
> a. Not self-contained, requires PF's AQ which is anyway designed for such purpose.
>
> 2. Legacy registers access via new MMIO region (v0 + sane reset)
I think I would buy this one if it was actually in BAR0 exactly legacy
with no changes, no reset no nothing. a hack like Jason described that
at least makes legacy linux drivers work with no software glue. That is
horrible, fragile but engineering history is full of hacks like this.
If you need software in the middle then let's please make it robust.
> Pros:
> a. smaller code in slow register access than AQ
> b. Sw cannot coalesce some device specific registers.
> c. Self-contained in the VF
>
> Cons:
> a. Relatively burdensome for the device as it requires more RW registers at scale, but it can be indirect register.
what I keep saying
b. a ton of legacy bugs which hypervisor can't detect or work around
(reset is just one issue)
> 3. legacy registers tunneling with additional indirection via PCI capability
I don't know what's this one.
> Cons:
> a. transport specific (but not big concern to me)
> b. Twice the slow as it requires RW cap.
> c. Inferior to #2, still requires sane reset as #2
> d. MMIO indirect registers are better than capabilities because PCI caps are largely for read only purpose.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 21:05 ` Parav Pandit
@ 2023-05-22 21:42 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 21:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 09:05:13PM +0000, Parav Pandit wrote:
> > How was I supposed to raise this issue on v1 if it was not there?
> > Stop arguing in circles and things will move forward.
> v1 version clearly had AQ commands.
>
> Please revisit your comments.
Just for fun, I did look it up, and there I asked:
What is this notify offset? It's never explained.
so yes, it was there but without explanation, only technically.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-22 21:42 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-22 21:42 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 22, 2023 at 09:05:13PM +0000, Parav Pandit wrote:
> > How was I supposed to raise this issue on v1 if it was not there?
> > Stop arguing in circles and things will move forward.
> v1 version clearly had AQ commands.
>
> Please revisit your comments.
Just for fun, I did look it up, and there I asked:
What is this notify offset? It's never explained.
so yes, it was there but without explanation, only technically.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-23 6:38 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 6:38 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:33AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
Just thought of an issue: one use-case this can't address is SVQ.
Specifically, legacy has:
/* A 32-bit r/w PFN for the currently selected queue */
#define VIRTIO_PCI_QUEUE_PFN 8
this can only address up to 40 bits which might be ok for legacy guests,
but if instead we want the queue to live in host memory (this is what
SVQ does) then this does not work as host is likely to have more than
2^40 byte memory.
Further, one of advantages of modern is that used ring can live
in host memory with aliasing when available is passed through directly
(only with split, not packed). Again, does not work if we mirror
legacy.
All this does not mean this specific effort has to die, but I would
like us to better document how and when does device switch to
and from legacy mode, and for the switch to be somehow
reusable for solutions that are closer to vdpa.
Does enabling legacy commands cause the switch? Or should we
add a LEGACY_PCI command to enable/disable? I kind of like
the second option ...
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-23 6:38 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 6:38 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang, yishaih,
maorg, virtio-comment, shahafs
On Sat, May 06, 2023 at 03:01:33AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member PCI PF to access the legacy registers of the member VFs.
>
> If in future any SIOV devices to support legacy registers, they
> can be easily supported using same commands by using the group
> member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
>
> Patch summary:
> --------------
> patch-1 adds administrative virtuqueue commands
> patch-2 adds its conformance section
>
> This short series is on top of latest work [1] from Michael.
> It uses the newly introduced administrative virtqueue facility with 3 new
> commands which uses the existing virtio_admin_cmd.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202305/msg00112.html
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements can be solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> The third command suggested by Jason queries the VF device's driver
> notification region.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it as native device in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
Just thought of an issue: one use-case this can't address is SVQ.
Specifically, legacy has:
/* A 32-bit r/w PFN for the currently selected queue */
#define VIRTIO_PCI_QUEUE_PFN 8
this can only address up to 40 bits which might be ok for legacy guests,
but if instead we want the queue to live in host memory (this is what
SVQ does) then this does not work as host is likely to have more than
2^40 byte memory.
Further, one of advantages of modern is that used ring can live
in host memory with aliasing when available is passed through directly
(only with split, not packed). Again, does not work if we mirror
legacy.
All this does not mean this specific effort has to die, but I would
like us to better document how and when does device switch to
and from legacy mode, and for the switch to be somehow
reusable for solutions that are closer to vdpa.
Does enabling legacy commands cause the switch? Or should we
add a LEGACY_PCI command to enable/disable? I kind of like
the second option ...
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-22 21:34 ` Michael S. Tsirkin
@ 2023-05-23 17:13 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 17:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 22, 2023 5:35 PM
>
> iokit is from my previous work experience not virtio.
> For virtio we had a lot of trouble with hacks like this on windows.
I don't know why windows driver need to talk to PF driver until now when the AQ doesn't exists.
> That's an important platform.
>
I am not sure how important is it.
What I learnt that a decade has passed and even after that virtio drivers are not part of the OS distro.
> > > If you are mapping VF BAR, this has to be done by VF driver, and
> > > please just avoid pain for everyone by exposing the necessary info
> > > through the VF itself.
> > > A capability seems perfect for this.
> > >
> > Why not the MMIO region as proposed in v0?
>
> Are you asking what were issues with v0? The issue with v0 was not MMIO as
> such at least for me. The issue was that it was not giving us any way to work
> around bugs in the legacy interface. AQ was a fix for that.
>
It doesn't need to because guest is anyway going to do whatever it does.
The clear requirement is: to not fix legacy but to support legacy.
Fixing legacy is role of 1.x. Not the role of legacy interface itself using AQ/MMIO.
And even if one desire to coalesce MMIO writes, it can still do it for mac address.
We discussed this.
We also discussed that kernel as old as 2.6.32 uses CVQ for mac address.
And Even device can apply mac address when it reaches the count of byte writes to 6, if wishes to.
>
> > Capability based read/write requires extra redirection, it is less optimal than
> MMIO.
> >
> > > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> > Come an iokit will be able to access this PF BAR without binding to it?
>
> I don't get what you are saying here. You have to bind a PF driver.
> Otherwise you can not send AQ command. That driver can map PF BAR into
> applications.
>
This is usual model that we follow. When PF driver is serving and emulating it provides the shared platform for other devices for legacy functionality.
It is for legacy not for any other newer interface.
And for purpose of BAR mapping, such BAR in the VF can be mapped too.
You didn't answer why VF BAR is a problem and suggest PF BAR.
Such VF BAR already exists and asking for new resource creation doesn't fit in your "simple" scheme suggestion either.
>
> > > Guest is not accessing this anyway. So I really do not see why not
> > > from software point of view. There's a hardware reason? Could you
> > > talk to hardware guys about this? You objected to AQ too then
> > > hardware guys told you it is not a big deal.
> > I didn't object to AQ on hw reason.
> > I explained AQ is not the best compared to MMIO because it is slow
> compared to MMIO by law of physics as it needs to move more bits than
> MMIO.
> > And so, the capability based tunneling too is slow to MMIO.
> > >
> > > > > So we have legacy emulation send commands to VF or to PF. Okay.
> > > > > But let us avoid the need for VF driver to send commands to PF to
> initialize.
> > > > > Just get all information it needs from VF itself.
> > > > >
> > > > >
> > > > > Maybe it's a good idea to reuse existing notification
> > > > > capability, or maybe a new one, but let's avoid making VF driver
> > > > > depend on PF
> > > commands.
> > > > >
> > > > We agreed in v1 on Jason's suggestion to have the AQ command and
> > > > yet you and Jason hinder this in v2 with this exact repeated question.
> > > > Lets please avoid this and move forward.
> > >
> > > How was I supposed to raise this issue on v1 if it was not there?
> > > Stop arguing in circles and things will move forward.
> > v1 version clearly had AQ commands.
> >
> > Please revisit your comments.
> > v0 -> you suggested:
> > - "why not AQ, like transport vq, it can coalesce partial mac writes".
> >
> > v1-> over AQ, you suggested:
> > - to "drop the size"
> > - summarized to split 2 commands to 4 commands to split device and config
> area.
> >
> > After that summary, you propose, why not self-contained in VF itself? -> circle
> back to v0 method.
> > (Instead of mmio, suggesting capability, but not explaining why mmio is not
> fine. Which is better than cap).
> >
> > Then now you propose "have BAR in the PF which is owned by other PF
> driver", with least reasoning.
> > And it convolutes with your suggestion of PF driver access...
> >
> > So, I am bit lost on your circular proposal, or it was just a probe question...
> > Not sure.
>
> You did move things to AQ however not completely. You kept notification in
> MMIO. If everything was in AQ we would be done as far as I am concerned
> (though Jason still does not like duplication of functionality with transport vq,
> but that is a separate issue).
>
It is a wrong design point to ask to "move everything to AQ", because AQ cannot be the data link layer for everything.
One can ask in the, why don't I put my 1GB of data inside the AQ itself? AQ should do everything...
Why not device build MB of shared memory for MMIO, queue data ... one shared memory should do everything.
And list like that can continue with no end.
We have seen the perf numbers with iobar and memory bar both for PCI devices in lab.
For the PCI transport, perf wise, memory bar for notification, second is iobar, 3rd is AQ.
They are order of magnitude apart for latency.
So AQ to send DB is only looks good in the PDF like how INTX looks good in PDF with SR-IOV but doesn't work well on perf wise.
> As notification is in VF MMIO I am trying to help you by suggesting other ways
> to keep things consistent. If it's in VF I think it follows that where it is has to be
> in VF as well. Not in AQ.
Huh.
If notification for VF is in VF -> hence other config registers also in VF, such blanket statements does not help.
I explained in past that notifications are latency sensitive, and they are best served using MMIO.
Can one implement legacy registers as MMIO as proposed in v0?
Yes, comes at high cost to build such hw.
Is there a better optimal option for legacy?
Yes, i.e. use AQ which is what we wrote in cover letter of AQ.
b. Use MMIO registers pair on the VF itself for example,
legacy_register_addr = offset + size + operation + completion
legacy_register_data = register content
an example sequence is:
hv sw writes
legacy_register_addr = offset=0x2, size=0x4, operation=read,
Poll for
legacy_register_addr = offset=0x2, size=0x4, operation=read, cmpl=1
Read the register data from legacy_register_data.
This way a small register footprint on device. Self-contained. Works for the device reset too.
Less memory footprint on the device too.
Does this solve the legacy interface problem? No. Because that is not the objective.
Is this small enough to carry in spec without disrupting the spec too much -> yes.
Is this lighter for sw and device to do? -> yes.
Does it work at scale of VFs? -> yes.
Does it work as_is with legacy sw? -> No.
>
> And look if that just serves to annoy I can just point out issues and complain
> without suggestions. Was trying to be helpful here really.
>
Your comments and suggestions are helpful.
Some of them follow the loop that doesn't help.
Again, thanks a lot for the inputs and reviews.
The design discussion helped to produce two solutions here than what was started in v0.
> > >
> > > If we have trouble converging on the notification thing, how about
> > > we make progress without it for now? Merge a slow version that
> > > sends kicks through aq, then work on the best way to make notifications
> faster, separately.
> > >
> > Sure, we can differ, but the VF has the notification BAR to utilize.
> > So why not use it?
>
> Are you asking why is merging a smaller partial patch then adding functionality
> a good idea?
>
I am asking a 1.x PCI device has the VF notification BAR already, why we don't use it, why to build an additional one in device?
> > > VF(member) driver can't implement AQ commands at all. They are sent
> > > to
> > > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
> > > legacy access and talk to owner driver to send them. It can also
> > > talk to member driver to send notifications. You can I guess have
> > > VMM get member BAR offsets from owner and pass them to member for
> > > mapping. This raises all kind of questions of trust.
> > Only the member driver has access to the owner driver via kernel API.
>
> Assuming you can predict existance of specific kernel APIs on all OSes is IMO a
> fool's errand.
A specific kernel API is not expected.
A framework is expected in OS.
And sw improvise as use case arise.
Driver to driver communication may be possible in WDM using [1].
Outside of virtio, I can imagine for PRI level functionality two drivers interaction would be needed too.
[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/different-ways-of-handling-irps-cheat-sheet#scenario-3-forward-with-a-completion-routine
> > Part of the VF BAR information at the PCI level is also on the PF.
> > There is no trust break here. A kernel exposes single self-contained object to
> user space VMM.
>
> That is ok. The trust break is if PF driver is in userspace. You then are trusting
> userspace to pass offsets back to kernel VF driver.
>
>
I don't see how a MMIO over PF is somehow more secure than AQ over PF regardless of in user or kernel.
I don't see how this a problem is.
Even today when/if IOBAR is trapped by the user space forwarding to kernel can modify the offset and value without any AQ etc.
I don't even see how one OS can spawn SR-IOV VFs from the user space driver in the kernel.
Look one can even take the PF from user space to yet another guest VM and add ten other complexities to put oneself to a dead end.
>
> > > If you can map BAR from PF that would be simplest, VMM then only
> > > pokes at PF and not VF. If not then we really should expose this
> > > info n the VF if at all possible.
> > >
> > There is clearly no need to use PF for notification when the notification
> method already exists on the VF.
>
> All this looks backwards with mixed up functionality. What VF has is modern
> BAR. If you insist on reusing modern functionality for legacy when it suits you
> then maybe we should reuse it for everything, that would be extending
> transport vq to support legacy somehow.
>
You bring back same point that we already closed in discussion.
It is not about suiting me.
It is what is there in the spec and in device for transitional device. (not specific to 1.x).
The design criteria is:
a. use as much as possible from 1.x and transitional device for the legacy support so that spec changes, device side, sw side changes are minimal.
AQ fits very well to this criteria.
b. Avoid inventing new interfaces between driver and device, specifically at PCI device level for things that are often implemented in hw.
Here also AQ scores well.
MMIO two register example also works well.
And notification also works well that reuses transitional device's notification.
3 out of 4 functionalities are done on AQ of the TVQ.
So, it is time to stop asking to replicate the AQ functionality on some new TVQ.
This was discussed in depth.
>
> > In fact, it is yet extra steering entry for the hw to perform such mapping.
> >
> > To iterate what presented yday in [1] to Jason.
> >
> > [1]
> > https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.h
> > tml
> >
> > 1. legacy register access via AQ (v1)
> > Pros:
> > a. Light weight for hypervisor and devices (mainly PCI) to implement.
> > b. Enables sw to coalesce some device specific registers, if needed.
> > Cons:
> > a. Not self-contained, requires PF's AQ which is anyway designed for such
> purpose.
> >
> > 2. Legacy registers access via new MMIO region (v0 + sane reset)
>
> I think I would buy this one if it was actually in BAR0 exactly legacy with no
> changes, no reset no nothing. a hack like Jason described that at least makes
> legacy linux drivers work with no software glue. That is horrible, fragile but
> engineering history is full of hacks like this.
> If you need software in the middle then let's please make it robust.
>
Your ignorance and inclination to see only from the sw pov is not helpful here.
Feedback from the PCI device side given here to be considered too.
This approach doesn't work well for device reset at scale which is a one way signal.
Hence, I proposed the either the AQ over PF or the two-register scheme of above over MMIO VF.
Does that work for you?
>
> > Pros:
> > a. smaller code in slow register access than AQ b. Sw cannot coalesce
> > some device specific registers.
> > c. Self-contained in the VF
> >
> > Cons:
> > a. Relatively burdensome for the device as it requires more RW registers at
> scale, but it can be indirect register.
>
> what I keep saying
>
> b. a ton of legacy bugs which hypervisor can't detect or work around
> (reset is just one issue)
>
>
> > 3. legacy registers tunneling with additional indirection via PCI
> > capability
>
>
> I don't know what's this one.
Above example of legacy_register_addr and legacy register_data on the VF.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-23 17:13 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 17:13 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, May 22, 2023 5:35 PM
>
> iokit is from my previous work experience not virtio.
> For virtio we had a lot of trouble with hacks like this on windows.
I don't know why windows driver need to talk to PF driver until now when the AQ doesn't exists.
> That's an important platform.
>
I am not sure how important is it.
What I learnt that a decade has passed and even after that virtio drivers are not part of the OS distro.
> > > If you are mapping VF BAR, this has to be done by VF driver, and
> > > please just avoid pain for everyone by exposing the necessary info
> > > through the VF itself.
> > > A capability seems perfect for this.
> > >
> > Why not the MMIO region as proposed in v0?
>
> Are you asking what were issues with v0? The issue with v0 was not MMIO as
> such at least for me. The issue was that it was not giving us any way to work
> around bugs in the legacy interface. AQ was a fix for that.
>
It doesn't need to because guest is anyway going to do whatever it does.
The clear requirement is: to not fix legacy but to support legacy.
Fixing legacy is role of 1.x. Not the role of legacy interface itself using AQ/MMIO.
And even if one desire to coalesce MMIO writes, it can still do it for mac address.
We discussed this.
We also discussed that kernel as old as 2.6.32 uses CVQ for mac address.
And Even device can apply mac address when it reaches the count of byte writes to 6, if wishes to.
>
> > Capability based read/write requires extra redirection, it is less optimal than
> MMIO.
> >
> > > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> > Come an iokit will be able to access this PF BAR without binding to it?
>
> I don't get what you are saying here. You have to bind a PF driver.
> Otherwise you can not send AQ command. That driver can map PF BAR into
> applications.
>
This is usual model that we follow. When PF driver is serving and emulating it provides the shared platform for other devices for legacy functionality.
It is for legacy not for any other newer interface.
And for purpose of BAR mapping, such BAR in the VF can be mapped too.
You didn't answer why VF BAR is a problem and suggest PF BAR.
Such VF BAR already exists and asking for new resource creation doesn't fit in your "simple" scheme suggestion either.
>
> > > Guest is not accessing this anyway. So I really do not see why not
> > > from software point of view. There's a hardware reason? Could you
> > > talk to hardware guys about this? You objected to AQ too then
> > > hardware guys told you it is not a big deal.
> > I didn't object to AQ on hw reason.
> > I explained AQ is not the best compared to MMIO because it is slow
> compared to MMIO by law of physics as it needs to move more bits than
> MMIO.
> > And so, the capability based tunneling too is slow to MMIO.
> > >
> > > > > So we have legacy emulation send commands to VF or to PF. Okay.
> > > > > But let us avoid the need for VF driver to send commands to PF to
> initialize.
> > > > > Just get all information it needs from VF itself.
> > > > >
> > > > >
> > > > > Maybe it's a good idea to reuse existing notification
> > > > > capability, or maybe a new one, but let's avoid making VF driver
> > > > > depend on PF
> > > commands.
> > > > >
> > > > We agreed in v1 on Jason's suggestion to have the AQ command and
> > > > yet you and Jason hinder this in v2 with this exact repeated question.
> > > > Lets please avoid this and move forward.
> > >
> > > How was I supposed to raise this issue on v1 if it was not there?
> > > Stop arguing in circles and things will move forward.
> > v1 version clearly had AQ commands.
> >
> > Please revisit your comments.
> > v0 -> you suggested:
> > - "why not AQ, like transport vq, it can coalesce partial mac writes".
> >
> > v1-> over AQ, you suggested:
> > - to "drop the size"
> > - summarized to split 2 commands to 4 commands to split device and config
> area.
> >
> > After that summary, you propose, why not self-contained in VF itself? -> circle
> back to v0 method.
> > (Instead of mmio, suggesting capability, but not explaining why mmio is not
> fine. Which is better than cap).
> >
> > Then now you propose "have BAR in the PF which is owned by other PF
> driver", with least reasoning.
> > And it convolutes with your suggestion of PF driver access...
> >
> > So, I am bit lost on your circular proposal, or it was just a probe question...
> > Not sure.
>
> You did move things to AQ however not completely. You kept notification in
> MMIO. If everything was in AQ we would be done as far as I am concerned
> (though Jason still does not like duplication of functionality with transport vq,
> but that is a separate issue).
>
It is a wrong design point to ask to "move everything to AQ", because AQ cannot be the data link layer for everything.
One can ask in the, why don't I put my 1GB of data inside the AQ itself? AQ should do everything...
Why not device build MB of shared memory for MMIO, queue data ... one shared memory should do everything.
And list like that can continue with no end.
We have seen the perf numbers with iobar and memory bar both for PCI devices in lab.
For the PCI transport, perf wise, memory bar for notification, second is iobar, 3rd is AQ.
They are order of magnitude apart for latency.
So AQ to send DB is only looks good in the PDF like how INTX looks good in PDF with SR-IOV but doesn't work well on perf wise.
> As notification is in VF MMIO I am trying to help you by suggesting other ways
> to keep things consistent. If it's in VF I think it follows that where it is has to be
> in VF as well. Not in AQ.
Huh.
If notification for VF is in VF -> hence other config registers also in VF, such blanket statements does not help.
I explained in past that notifications are latency sensitive, and they are best served using MMIO.
Can one implement legacy registers as MMIO as proposed in v0?
Yes, comes at high cost to build such hw.
Is there a better optimal option for legacy?
Yes, i.e. use AQ which is what we wrote in cover letter of AQ.
b. Use MMIO registers pair on the VF itself for example,
legacy_register_addr = offset + size + operation + completion
legacy_register_data = register content
an example sequence is:
hv sw writes
legacy_register_addr = offset=0x2, size=0x4, operation=read,
Poll for
legacy_register_addr = offset=0x2, size=0x4, operation=read, cmpl=1
Read the register data from legacy_register_data.
This way a small register footprint on device. Self-contained. Works for the device reset too.
Less memory footprint on the device too.
Does this solve the legacy interface problem? No. Because that is not the objective.
Is this small enough to carry in spec without disrupting the spec too much -> yes.
Is this lighter for sw and device to do? -> yes.
Does it work at scale of VFs? -> yes.
Does it work as_is with legacy sw? -> No.
>
> And look if that just serves to annoy I can just point out issues and complain
> without suggestions. Was trying to be helpful here really.
>
Your comments and suggestions are helpful.
Some of them follow the loop that doesn't help.
Again, thanks a lot for the inputs and reviews.
The design discussion helped to produce two solutions here than what was started in v0.
> > >
> > > If we have trouble converging on the notification thing, how about
> > > we make progress without it for now? Merge a slow version that
> > > sends kicks through aq, then work on the best way to make notifications
> faster, separately.
> > >
> > Sure, we can differ, but the VF has the notification BAR to utilize.
> > So why not use it?
>
> Are you asking why is merging a smaller partial patch then adding functionality
> a good idea?
>
I am asking a 1.x PCI device has the VF notification BAR already, why we don't use it, why to build an additional one in device?
> > > VF(member) driver can't implement AQ commands at all. They are sent
> > > to
> > > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
> > > legacy access and talk to owner driver to send them. It can also
> > > talk to member driver to send notifications. You can I guess have
> > > VMM get member BAR offsets from owner and pass them to member for
> > > mapping. This raises all kind of questions of trust.
> > Only the member driver has access to the owner driver via kernel API.
>
> Assuming you can predict existance of specific kernel APIs on all OSes is IMO a
> fool's errand.
A specific kernel API is not expected.
A framework is expected in OS.
And sw improvise as use case arise.
Driver to driver communication may be possible in WDM using [1].
Outside of virtio, I can imagine for PRI level functionality two drivers interaction would be needed too.
[1] https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/different-ways-of-handling-irps-cheat-sheet#scenario-3-forward-with-a-completion-routine
> > Part of the VF BAR information at the PCI level is also on the PF.
> > There is no trust break here. A kernel exposes single self-contained object to
> user space VMM.
>
> That is ok. The trust break is if PF driver is in userspace. You then are trusting
> userspace to pass offsets back to kernel VF driver.
>
>
I don't see how a MMIO over PF is somehow more secure than AQ over PF regardless of in user or kernel.
I don't see how this a problem is.
Even today when/if IOBAR is trapped by the user space forwarding to kernel can modify the offset and value without any AQ etc.
I don't even see how one OS can spawn SR-IOV VFs from the user space driver in the kernel.
Look one can even take the PF from user space to yet another guest VM and add ten other complexities to put oneself to a dead end.
>
> > > If you can map BAR from PF that would be simplest, VMM then only
> > > pokes at PF and not VF. If not then we really should expose this
> > > info n the VF if at all possible.
> > >
> > There is clearly no need to use PF for notification when the notification
> method already exists on the VF.
>
> All this looks backwards with mixed up functionality. What VF has is modern
> BAR. If you insist on reusing modern functionality for legacy when it suits you
> then maybe we should reuse it for everything, that would be extending
> transport vq to support legacy somehow.
>
You bring back same point that we already closed in discussion.
It is not about suiting me.
It is what is there in the spec and in device for transitional device. (not specific to 1.x).
The design criteria is:
a. use as much as possible from 1.x and transitional device for the legacy support so that spec changes, device side, sw side changes are minimal.
AQ fits very well to this criteria.
b. Avoid inventing new interfaces between driver and device, specifically at PCI device level for things that are often implemented in hw.
Here also AQ scores well.
MMIO two register example also works well.
And notification also works well that reuses transitional device's notification.
3 out of 4 functionalities are done on AQ of the TVQ.
So, it is time to stop asking to replicate the AQ functionality on some new TVQ.
This was discussed in depth.
>
> > In fact, it is yet extra steering entry for the hw to perform such mapping.
> >
> > To iterate what presented yday in [1] to Jason.
> >
> > [1]
> > https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.h
> > tml
> >
> > 1. legacy register access via AQ (v1)
> > Pros:
> > a. Light weight for hypervisor and devices (mainly PCI) to implement.
> > b. Enables sw to coalesce some device specific registers, if needed.
> > Cons:
> > a. Not self-contained, requires PF's AQ which is anyway designed for such
> purpose.
> >
> > 2. Legacy registers access via new MMIO region (v0 + sane reset)
>
> I think I would buy this one if it was actually in BAR0 exactly legacy with no
> changes, no reset no nothing. a hack like Jason described that at least makes
> legacy linux drivers work with no software glue. That is horrible, fragile but
> engineering history is full of hacks like this.
> If you need software in the middle then let's please make it robust.
>
Your ignorance and inclination to see only from the sw pov is not helpful here.
Feedback from the PCI device side given here to be considered too.
This approach doesn't work well for device reset at scale which is a one way signal.
Hence, I proposed the either the AQ over PF or the two-register scheme of above over MMIO VF.
Does that work for you?
>
> > Pros:
> > a. smaller code in slow register access than AQ b. Sw cannot coalesce
> > some device specific registers.
> > c. Self-contained in the VF
> >
> > Cons:
> > a. Relatively burdensome for the device as it requires more RW registers at
> scale, but it can be indirect register.
>
> what I keep saying
>
> b. a ton of legacy bugs which hypervisor can't detect or work around
> (reset is just one issue)
>
>
> > 3. legacy registers tunneling with additional indirection via PCI
> > capability
>
>
> I don't know what's this one.
Above example of legacy_register_addr and legacy register_data on the VF.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-23 6:38 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-23 17:28 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 17:28 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:39 AM
> but I would like us to better
> document how and when does device switch to and from legacy mode, and for
> the switch to be somehow reusable for solutions that are closer to vdpa.
>
To my knowledge vdpa do not have such switch, nor such switch in the spec, nor in the vdpa or PCI device.
> Does enabling legacy commands cause the switch? Or should we add a
> LEGACY_PCI command to enable/disable? I kind of like the second option ...
Some devices may find it helpful to issue this command on the AQ, to say enable your legacy emulation and allocate needed resources.
I guess before getting distracted on this side topic, lets close on the AQ or two register MMIO approach in other email thread.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-23 17:28 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 17:28 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:39 AM
> but I would like us to better
> document how and when does device switch to and from legacy mode, and for
> the switch to be somehow reusable for solutions that are closer to vdpa.
>
To my knowledge vdpa do not have such switch, nor such switch in the spec, nor in the vdpa or PCI device.
> Does enabling legacy commands cause the switch? Or should we add a
> LEGACY_PCI command to enable/disable? I kind of like the second option ...
Some devices may find it helpful to issue this command on the AQ, to say enable your legacy emulation and allocate needed resources.
I guess before getting distracted on this side topic, lets close on the AQ or two register MMIO approach in other email thread.
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-15 14:30 ` [virtio-comment] " Parav Pandit
@ 2023-05-23 18:16 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 18:16 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 6:09 AM
> > To be more specific, device config often has the same layout under legacy and
> > modern. Thus if getting an offset within device config, same decoding logic
> > should be reusable between legacy and modern.
> Modern device registers are directly accessible from the guest driver without mediation for VF,SF,SIOV.
> Hence, there is no need to have commands for modern register access over some VQ.
Yes, there's need, and some of it you were pushing yourself.
AQ will allow reduced memory with SRIOV, and support for SIOV.
> Will reply to rest of the Jason's comments in some time.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-23 18:16 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 18:16 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 15, 2023 6:09 AM
> > To be more specific, device config often has the same layout under legacy and
> > modern. Thus if getting an offset within device config, same decoding logic
> > should be reusable between legacy and modern.
> Modern device registers are directly accessible from the guest driver without mediation for VF,SF,SIOV.
> Hence, there is no need to have commands for modern register access over some VQ.
Yes, there's need, and some of it you were pushing yourself.
AQ will allow reduced memory with SRIOV, and support for SIOV.
> Will reply to rest of the Jason's comments in some time.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-23 17:13 ` Parav Pandit
@ 2023-05-23 18:48 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 18:48 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 05:13:54PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 22, 2023 5:35 PM
>
> >
> > iokit is from my previous work experience not virtio.
> > For virtio we had a lot of trouble with hacks like this on windows.
> I don't know why windows driver need to talk to PF driver until now when the AQ doesn't exists.
What is it you want to know?
> > That's an important platform.
> >
> I am not sure how important is it.
> What I learnt that a decade has passed and even after that virtio drivers are not part of the OS distro.
Because with QEMU they are so easy to side-load. So we never bothered.
> > > > If you are mapping VF BAR, this has to be done by VF driver, and
> > > > please just avoid pain for everyone by exposing the necessary info
> > > > through the VF itself.
> > > > A capability seems perfect for this.
> > > >
> > > Why not the MMIO region as proposed in v0?
> >
> > Are you asking what were issues with v0? The issue with v0 was not MMIO as
> > such at least for me. The issue was that it was not giving us any way to work
> > around bugs in the legacy interface. AQ was a fix for that.
> >
> It doesn't need to because guest is anyway going to do whatever it does.
yes. But hypervisor can do work-arounds or detect broken functionality
and stop cleanly.
> The clear requirement is: to not fix legacy but to support legacy.
> Fixing legacy is role of 1.x. Not the role of legacy interface itself using AQ/MMIO.
>
> And even if one desire to coalesce MMIO writes, it can still do it for mac address.
> We discussed this.
> We also discussed that kernel as old as 2.6.32 uses CVQ for mac address.
>
> And Even device can apply mac address when it reaches the count of byte writes to 6, if wishes to.
> >
> > > Capability based read/write requires extra redirection, it is less optimal than
> > MMIO.
> > >
> > > > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> > > Come an iokit will be able to access this PF BAR without binding to it?
> >
> > I don't get what you are saying here. You have to bind a PF driver.
> > Otherwise you can not send AQ command. That driver can map PF BAR into
> > applications.
> >
> This is usual model that we follow. When PF driver is serving and emulating it provides the shared platform for other devices for legacy functionality.
> It is for legacy not for any other newer interface.
>
> And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> You didn't answer why VF BAR is a problem and suggest PF BAR.
VF BAR by itself is not a problem. Discovering VF BAR mapping offset
through PF is.
> Such VF BAR already exists and asking for new resource creation doesn't fit in your "simple" scheme suggestion either.
>
> >
> > > > Guest is not accessing this anyway. So I really do not see why not
> > > > from software point of view. There's a hardware reason? Could you
> > > > talk to hardware guys about this? You objected to AQ too then
> > > > hardware guys told you it is not a big deal.
> > > I didn't object to AQ on hw reason.
> > > I explained AQ is not the best compared to MMIO because it is slow
> > compared to MMIO by law of physics as it needs to move more bits than
> > MMIO.
> > > And so, the capability based tunneling too is slow to MMIO.
> > > >
> > > > > > So we have legacy emulation send commands to VF or to PF. Okay.
> > > > > > But let us avoid the need for VF driver to send commands to PF to
> > initialize.
> > > > > > Just get all information it needs from VF itself.
> > > > > >
> > > > > >
> > > > > > Maybe it's a good idea to reuse existing notification
> > > > > > capability, or maybe a new one, but let's avoid making VF driver
> > > > > > depend on PF
> > > > commands.
> > > > > >
> > > > > We agreed in v1 on Jason's suggestion to have the AQ command and
> > > > > yet you and Jason hinder this in v2 with this exact repeated question.
> > > > > Lets please avoid this and move forward.
> > > >
> > > > How was I supposed to raise this issue on v1 if it was not there?
> > > > Stop arguing in circles and things will move forward.
> > > v1 version clearly had AQ commands.
> > >
> > > Please revisit your comments.
> > > v0 -> you suggested:
> > > - "why not AQ, like transport vq, it can coalesce partial mac writes".
> > >
> > > v1-> over AQ, you suggested:
> > > - to "drop the size"
> > > - summarized to split 2 commands to 4 commands to split device and config
> > area.
> > >
> > > After that summary, you propose, why not self-contained in VF itself? -> circle
> > back to v0 method.
> > > (Instead of mmio, suggesting capability, but not explaining why mmio is not
> > fine. Which is better than cap).
> > >
> > > Then now you propose "have BAR in the PF which is owned by other PF
> > driver", with least reasoning.
> > > And it convolutes with your suggestion of PF driver access...
> > >
> > > So, I am bit lost on your circular proposal, or it was just a probe question...
> > > Not sure.
> >
> > You did move things to AQ however not completely. You kept notification in
> > MMIO. If everything was in AQ we would be done as far as I am concerned
> > (though Jason still does not like duplication of functionality with transport vq,
> > but that is a separate issue).
> >
> It is a wrong design point to ask to "move everything to AQ", because AQ cannot be the data link layer for everything.
> One can ask in the, why don't I put my 1GB of data inside the AQ itself? AQ should do everything...
> Why not device build MB of shared memory for MMIO, queue data ... one shared memory should do everything.
>
> And list like that can continue with no end.
>
> We have seen the perf numbers with iobar and memory bar both for PCI devices in lab.
> For the PCI transport, perf wise, memory bar for notification, second is iobar, 3rd is AQ.
> They are order of magnitude apart for latency.
> So AQ to send DB is only looks good in the PDF like how INTX looks good in PDF with SR-IOV but doesn't work well on perf wise.
>
> > As notification is in VF MMIO I am trying to help you by suggesting other ways
> > to keep things consistent. If it's in VF I think it follows that where it is has to be
> > in VF as well. Not in AQ.
> Huh.
> If notification for VF is in VF -> hence other config registers also in VF, such blanket statements does not help.
not other config registers. what's needed in VF driver should be
retrievable through VF.
> I explained in past that notifications are latency sensitive, and they are best served using MMIO.
>
> Can one implement legacy registers as MMIO as proposed in v0?
> Yes, comes at high cost to build such hw.
> Is there a better optimal option for legacy?
> Yes, i.e. use AQ which is what we wrote in cover letter of AQ.
>
> b. Use MMIO registers pair on the VF itself for example,
> legacy_register_addr = offset + size + operation + completion
> legacy_register_data = register content
>
> an example sequence is:
> hv sw writes
> legacy_register_addr = offset=0x2, size=0x4, operation=read,
> Poll for
> legacy_register_addr = offset=0x2, size=0x4, operation=read, cmpl=1
> Read the register data from legacy_register_data.
>
> This way a small register footprint on device. Self-contained. Works for the device reset too.
> Less memory footprint on the device too.
>
> Does this solve the legacy interface problem? No. Because that is not the objective.
>
> Is this small enough to carry in spec without disrupting the spec too much -> yes.
> Is this lighter for sw and device to do? -> yes.
> Does it work at scale of VFs? -> yes.
> Does it work as_is with legacy sw? -> No.
>
> >
> > And look if that just serves to annoy I can just point out issues and complain
> > without suggestions. Was trying to be helpful here really.
> >
> Your comments and suggestions are helpful.
> Some of them follow the loop that doesn't help.
> Again, thanks a lot for the inputs and reviews.
>
> The design discussion helped to produce two solutions here than what was started in v0.
OK good. Now I suggest you make an effort to come up with some
compromise that will make this thread stop. I really can't say
I even remember what we said in the beginning.
You had this idea of starting with a problem statement document
how about doing that? Because I feel every time people bring
up some idea and a problem this idea would solve, you go
"this is out of scope". Let's decide what the scope is?
> > > >
> > > > If we have trouble converging on the notification thing, how about
> > > > we make progress without it for now? Merge a slow version that
> > > > sends kicks through aq, then work on the best way to make notifications
> > faster, separately.
> > > >
> > > Sure, we can differ, but the VF has the notification BAR to utilize.
> > > So why not use it?
> >
> > Are you asking why is merging a smaller partial patch then adding functionality
> > a good idea?
> >
> I am asking a 1.x PCI device has the VF notification BAR already, why we don't use it, why to build an additional one in device?
Following this logic, a 1.x PCI device also has modern common config.
All you really truly need is a flag to make it behave like legacy,
from POV of device config and VQs.
> > > > VF(member) driver can't implement AQ commands at all. They are sent
> > > > to
> > > > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
> > > > legacy access and talk to owner driver to send them. It can also
> > > > talk to member driver to send notifications. You can I guess have
> > > > VMM get member BAR offsets from owner and pass them to member for
> > > > mapping. This raises all kind of questions of trust.
> > > Only the member driver has access to the owner driver via kernel API.
> >
> > Assuming you can predict existance of specific kernel APIs on all OSes is IMO a
> > fool's errand.
> A specific kernel API is not expected.
> A framework is expected in OS.
> And sw improvise as use case arise.
>
> Driver to driver communication may be possible in WDM using [1].
> Outside of virtio, I can imagine for PRI level functionality two drivers interaction would be needed too.
>
> [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/different-ways-of-handling-irps-cheat-sheet#scenario-3-forward-with-a-completion-routine
>
> > > Part of the VF BAR information at the PCI level is also on the PF.
> > > There is no trust break here. A kernel exposes single self-contained object to
> > user space VMM.
> >
> > That is ok. The trust break is if PF driver is in userspace. You then are trusting
> > userspace to pass offsets back to kernel VF driver.
> >
> >
> I don't see how a MMIO over PF is somehow more secure than AQ over PF regardless of in user or kernel.
> I don't see how this a problem is.
> Even today when/if IOBAR is trapped by the user space forwarding to kernel can modify the offset and value without any AQ etc.
>
> I don't even see how one OS can spawn SR-IOV VFs from the user space driver in the kernel.
>
> Look one can even take the PF from user space to yet another guest VM and add ten other complexities to put oneself to a dead end.
So let's try to keep drivers self-contained as much as possible.
> >
> > > > If you can map BAR from PF that would be simplest, VMM then only
> > > > pokes at PF and not VF. If not then we really should expose this
> > > > info n the VF if at all possible.
> > > >
> > > There is clearly no need to use PF for notification when the notification
> > method already exists on the VF.
> >
> > All this looks backwards with mixed up functionality. What VF has is modern
> > BAR. If you insist on reusing modern functionality for legacy when it suits you
> > then maybe we should reuse it for everything, that would be extending
> > transport vq to support legacy somehow.
> >
> You bring back same point that we already closed in discussion.
> It is not about suiting me.
> It is what is there in the spec and in device for transitional device. (not specific to 1.x).
> The design criteria is:
> a. use as much as possible from 1.x and transitional device for the legacy support so that spec changes, device side, sw side changes are minimal.
> AQ fits very well to this criteria.
>
> b. Avoid inventing new interfaces between driver and device, specifically at PCI device level for things that are often implemented in hw.
> Here also AQ scores well.
> MMIO two register example also works well.
>
> And notification also works well that reuses transitional device's notification.
I don't know what these are. You had some weird
+struct virtio_admin_cmd_lq_notify_query_result {
+ u8 bar; /* PCI BAR number of the member VF */
+ u8 reserved[7];
+ le64 offset; /* Byte offset within the BAR */
+};
if you want to re-use the normal notifications for VFs we already
describe them using capabilities.
However, this creates a weird mix where there's this command that mostly
follows legacy pci config layout except notifications which are there in
the legacy pci config are actually somewhere else. And this is just
inconsistent and confusing, and the reason for this is implementation
defined.
> 3 out of 4 functionalities are done on AQ of the TVQ.
> So, it is time to stop asking to replicate the AQ functionality on some new TVQ.
> This was discussed in depth.
I don't think I ever did that.
> >
> > > In fact, it is yet extra steering entry for the hw to perform such mapping.
> > >
> > > To iterate what presented yday in [1] to Jason.
> > >
> > > [1]
> > > https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.h
> > > tml
> > >
> > > 1. legacy register access via AQ (v1)
> > > Pros:
> > > a. Light weight for hypervisor and devices (mainly PCI) to implement.
> > > b. Enables sw to coalesce some device specific registers, if needed.
> > > Cons:
> > > a. Not self-contained, requires PF's AQ which is anyway designed for such
> > purpose.
> > >
> > > 2. Legacy registers access via new MMIO region (v0 + sane reset)
> >
> > I think I would buy this one if it was actually in BAR0 exactly legacy with no
> > changes, no reset no nothing. a hack like Jason described that at least makes
> > legacy linux drivers work with no software glue. That is horrible, fragile but
> > engineering history is full of hacks like this.
> > If you need software in the middle then let's please make it robust.
> >
> Your ignorance and inclination to see only from the sw pov is not helpful here.
> Feedback from the PCI device side given here to be considered too.
After hearing this for the 10th time I don't think I'm not aware, thanks
for bringing this to my attention for the 11th time.
My point however is the last sentence *If you need software in the
middle then let's please make it robust.* If BAR0 hack does not work
then let's use sane interfaces as much as possible.
> This approach doesn't work well for device reset at scale which is a one way signal.
>
> Hence, I proposed the either the AQ over PF or the two-register scheme of above over MMIO VF.
>
> Does that work for you?
AQ works fine I think. This new one I haven't seen yet hard to say.
Probably has its own set of problems ...
> >
> > > Pros:
> > > a. smaller code in slow register access than AQ b. Sw cannot coalesce
> > > some device specific registers.
> > > c. Self-contained in the VF
> > >
> > > Cons:
> > > a. Relatively burdensome for the device as it requires more RW registers at
> > scale, but it can be indirect register.
> >
> > what I keep saying
> >
> > b. a ton of legacy bugs which hypervisor can't detect or work around
> > (reset is just one issue)
> >
> >
> > > 3. legacy registers tunneling with additional indirection via PCI
> > > capability
> >
> >
> > I don't know what's this one.
>
> Above example of legacy_register_addr and legacy register_data on the VF.
Talking about the two register thing. We actually have this already:
VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
But really since apparently you want to focus on how we solve
notifications let's focus on that. I feel the main issue is
poking at 1.x registers and at the same time at legacy registers.
The spec was written with a strong assumption that it is either or,
never a mix. Changing that will be painful.
This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER -
legacy can live in software then and we do not need to worry
about it in the spec. I got it that there's too much software
here for your taste, but the problem with legacy is really
that it was always ad hoc and implementation derived.
So we need to keep it siloed.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-23 18:48 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-23 18:48 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 05:13:54PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, May 22, 2023 5:35 PM
>
> >
> > iokit is from my previous work experience not virtio.
> > For virtio we had a lot of trouble with hacks like this on windows.
> I don't know why windows driver need to talk to PF driver until now when the AQ doesn't exists.
What is it you want to know?
> > That's an important platform.
> >
> I am not sure how important is it.
> What I learnt that a decade has passed and even after that virtio drivers are not part of the OS distro.
Because with QEMU they are so easy to side-load. So we never bothered.
> > > > If you are mapping VF BAR, this has to be done by VF driver, and
> > > > please just avoid pain for everyone by exposing the necessary info
> > > > through the VF itself.
> > > > A capability seems perfect for this.
> > > >
> > > Why not the MMIO region as proposed in v0?
> >
> > Are you asking what were issues with v0? The issue with v0 was not MMIO as
> > such at least for me. The issue was that it was not giving us any way to work
> > around bugs in the legacy interface. AQ was a fix for that.
> >
> It doesn't need to because guest is anyway going to do whatever it does.
yes. But hypervisor can do work-arounds or detect broken functionality
and stop cleanly.
> The clear requirement is: to not fix legacy but to support legacy.
> Fixing legacy is role of 1.x. Not the role of legacy interface itself using AQ/MMIO.
>
> And even if one desire to coalesce MMIO writes, it can still do it for mac address.
> We discussed this.
> We also discussed that kernel as old as 2.6.32 uses CVQ for mac address.
>
> And Even device can apply mac address when it reaches the count of byte writes to 6, if wishes to.
> >
> > > Capability based read/write requires extra redirection, it is less optimal than
> > MMIO.
> > >
> > > > An alternative is not to map VF BAR at all, steal space from PF BAR instead.
> > > Come an iokit will be able to access this PF BAR without binding to it?
> >
> > I don't get what you are saying here. You have to bind a PF driver.
> > Otherwise you can not send AQ command. That driver can map PF BAR into
> > applications.
> >
> This is usual model that we follow. When PF driver is serving and emulating it provides the shared platform for other devices for legacy functionality.
> It is for legacy not for any other newer interface.
>
> And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> You didn't answer why VF BAR is a problem and suggest PF BAR.
VF BAR by itself is not a problem. Discovering VF BAR mapping offset
through PF is.
> Such VF BAR already exists and asking for new resource creation doesn't fit in your "simple" scheme suggestion either.
>
> >
> > > > Guest is not accessing this anyway. So I really do not see why not
> > > > from software point of view. There's a hardware reason? Could you
> > > > talk to hardware guys about this? You objected to AQ too then
> > > > hardware guys told you it is not a big deal.
> > > I didn't object to AQ on hw reason.
> > > I explained AQ is not the best compared to MMIO because it is slow
> > compared to MMIO by law of physics as it needs to move more bits than
> > MMIO.
> > > And so, the capability based tunneling too is slow to MMIO.
> > > >
> > > > > > So we have legacy emulation send commands to VF or to PF. Okay.
> > > > > > But let us avoid the need for VF driver to send commands to PF to
> > initialize.
> > > > > > Just get all information it needs from VF itself.
> > > > > >
> > > > > >
> > > > > > Maybe it's a good idea to reuse existing notification
> > > > > > capability, or maybe a new one, but let's avoid making VF driver
> > > > > > depend on PF
> > > > commands.
> > > > > >
> > > > > We agreed in v1 on Jason's suggestion to have the AQ command and
> > > > > yet you and Jason hinder this in v2 with this exact repeated question.
> > > > > Lets please avoid this and move forward.
> > > >
> > > > How was I supposed to raise this issue on v1 if it was not there?
> > > > Stop arguing in circles and things will move forward.
> > > v1 version clearly had AQ commands.
> > >
> > > Please revisit your comments.
> > > v0 -> you suggested:
> > > - "why not AQ, like transport vq, it can coalesce partial mac writes".
> > >
> > > v1-> over AQ, you suggested:
> > > - to "drop the size"
> > > - summarized to split 2 commands to 4 commands to split device and config
> > area.
> > >
> > > After that summary, you propose, why not self-contained in VF itself? -> circle
> > back to v0 method.
> > > (Instead of mmio, suggesting capability, but not explaining why mmio is not
> > fine. Which is better than cap).
> > >
> > > Then now you propose "have BAR in the PF which is owned by other PF
> > driver", with least reasoning.
> > > And it convolutes with your suggestion of PF driver access...
> > >
> > > So, I am bit lost on your circular proposal, or it was just a probe question...
> > > Not sure.
> >
> > You did move things to AQ however not completely. You kept notification in
> > MMIO. If everything was in AQ we would be done as far as I am concerned
> > (though Jason still does not like duplication of functionality with transport vq,
> > but that is a separate issue).
> >
> It is a wrong design point to ask to "move everything to AQ", because AQ cannot be the data link layer for everything.
> One can ask in the, why don't I put my 1GB of data inside the AQ itself? AQ should do everything...
> Why not device build MB of shared memory for MMIO, queue data ... one shared memory should do everything.
>
> And list like that can continue with no end.
>
> We have seen the perf numbers with iobar and memory bar both for PCI devices in lab.
> For the PCI transport, perf wise, memory bar for notification, second is iobar, 3rd is AQ.
> They are order of magnitude apart for latency.
> So AQ to send DB is only looks good in the PDF like how INTX looks good in PDF with SR-IOV but doesn't work well on perf wise.
>
> > As notification is in VF MMIO I am trying to help you by suggesting other ways
> > to keep things consistent. If it's in VF I think it follows that where it is has to be
> > in VF as well. Not in AQ.
> Huh.
> If notification for VF is in VF -> hence other config registers also in VF, such blanket statements does not help.
not other config registers. what's needed in VF driver should be
retrievable through VF.
> I explained in past that notifications are latency sensitive, and they are best served using MMIO.
>
> Can one implement legacy registers as MMIO as proposed in v0?
> Yes, comes at high cost to build such hw.
> Is there a better optimal option for legacy?
> Yes, i.e. use AQ which is what we wrote in cover letter of AQ.
>
> b. Use MMIO registers pair on the VF itself for example,
> legacy_register_addr = offset + size + operation + completion
> legacy_register_data = register content
>
> an example sequence is:
> hv sw writes
> legacy_register_addr = offset=0x2, size=0x4, operation=read,
> Poll for
> legacy_register_addr = offset=0x2, size=0x4, operation=read, cmpl=1
> Read the register data from legacy_register_data.
>
> This way a small register footprint on device. Self-contained. Works for the device reset too.
> Less memory footprint on the device too.
>
> Does this solve the legacy interface problem? No. Because that is not the objective.
>
> Is this small enough to carry in spec without disrupting the spec too much -> yes.
> Is this lighter for sw and device to do? -> yes.
> Does it work at scale of VFs? -> yes.
> Does it work as_is with legacy sw? -> No.
>
> >
> > And look if that just serves to annoy I can just point out issues and complain
> > without suggestions. Was trying to be helpful here really.
> >
> Your comments and suggestions are helpful.
> Some of them follow the loop that doesn't help.
> Again, thanks a lot for the inputs and reviews.
>
> The design discussion helped to produce two solutions here than what was started in v0.
OK good. Now I suggest you make an effort to come up with some
compromise that will make this thread stop. I really can't say
I even remember what we said in the beginning.
You had this idea of starting with a problem statement document
how about doing that? Because I feel every time people bring
up some idea and a problem this idea would solve, you go
"this is out of scope". Let's decide what the scope is?
> > > >
> > > > If we have trouble converging on the notification thing, how about
> > > > we make progress without it for now? Merge a slow version that
> > > > sends kicks through aq, then work on the best way to make notifications
> > faster, separately.
> > > >
> > > Sure, we can differ, but the VF has the notification BAR to utilize.
> > > So why not use it?
> >
> > Are you asking why is merging a smaller partial patch then adding functionality
> > a good idea?
> >
> I am asking a 1.x PCI device has the VF notification BAR already, why we don't use it, why to build an additional one in device?
Following this logic, a 1.x PCI device also has modern common config.
All you really truly need is a flag to make it behave like legacy,
from POV of device config and VQs.
> > > > VF(member) driver can't implement AQ commands at all. They are sent
> > > > to
> > > > PF(owner) thus by PF(owner) driver. In virt configs, VMM can trap
> > > > legacy access and talk to owner driver to send them. It can also
> > > > talk to member driver to send notifications. You can I guess have
> > > > VMM get member BAR offsets from owner and pass them to member for
> > > > mapping. This raises all kind of questions of trust.
> > > Only the member driver has access to the owner driver via kernel API.
> >
> > Assuming you can predict existance of specific kernel APIs on all OSes is IMO a
> > fool's errand.
> A specific kernel API is not expected.
> A framework is expected in OS.
> And sw improvise as use case arise.
>
> Driver to driver communication may be possible in WDM using [1].
> Outside of virtio, I can imagine for PRI level functionality two drivers interaction would be needed too.
>
> [1] https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/different-ways-of-handling-irps-cheat-sheet#scenario-3-forward-with-a-completion-routine
>
> > > Part of the VF BAR information at the PCI level is also on the PF.
> > > There is no trust break here. A kernel exposes single self-contained object to
> > user space VMM.
> >
> > That is ok. The trust break is if PF driver is in userspace. You then are trusting
> > userspace to pass offsets back to kernel VF driver.
> >
> >
> I don't see how a MMIO over PF is somehow more secure than AQ over PF regardless of in user or kernel.
> I don't see how this a problem is.
> Even today when/if IOBAR is trapped by the user space forwarding to kernel can modify the offset and value without any AQ etc.
>
> I don't even see how one OS can spawn SR-IOV VFs from the user space driver in the kernel.
>
> Look one can even take the PF from user space to yet another guest VM and add ten other complexities to put oneself to a dead end.
So let's try to keep drivers self-contained as much as possible.
> >
> > > > If you can map BAR from PF that would be simplest, VMM then only
> > > > pokes at PF and not VF. If not then we really should expose this
> > > > info n the VF if at all possible.
> > > >
> > > There is clearly no need to use PF for notification when the notification
> > method already exists on the VF.
> >
> > All this looks backwards with mixed up functionality. What VF has is modern
> > BAR. If you insist on reusing modern functionality for legacy when it suits you
> > then maybe we should reuse it for everything, that would be extending
> > transport vq to support legacy somehow.
> >
> You bring back same point that we already closed in discussion.
> It is not about suiting me.
> It is what is there in the spec and in device for transitional device. (not specific to 1.x).
> The design criteria is:
> a. use as much as possible from 1.x and transitional device for the legacy support so that spec changes, device side, sw side changes are minimal.
> AQ fits very well to this criteria.
>
> b. Avoid inventing new interfaces between driver and device, specifically at PCI device level for things that are often implemented in hw.
> Here also AQ scores well.
> MMIO two register example also works well.
>
> And notification also works well that reuses transitional device's notification.
I don't know what these are. You had some weird
+struct virtio_admin_cmd_lq_notify_query_result {
+ u8 bar; /* PCI BAR number of the member VF */
+ u8 reserved[7];
+ le64 offset; /* Byte offset within the BAR */
+};
if you want to re-use the normal notifications for VFs we already
describe them using capabilities.
However, this creates a weird mix where there's this command that mostly
follows legacy pci config layout except notifications which are there in
the legacy pci config are actually somewhere else. And this is just
inconsistent and confusing, and the reason for this is implementation
defined.
> 3 out of 4 functionalities are done on AQ of the TVQ.
> So, it is time to stop asking to replicate the AQ functionality on some new TVQ.
> This was discussed in depth.
I don't think I ever did that.
> >
> > > In fact, it is yet extra steering entry for the hw to perform such mapping.
> > >
> > > To iterate what presented yday in [1] to Jason.
> > >
> > > [1]
> > > https://lists.oasis-open.org/archives/virtio-comment/202305/msg00298.h
> > > tml
> > >
> > > 1. legacy register access via AQ (v1)
> > > Pros:
> > > a. Light weight for hypervisor and devices (mainly PCI) to implement.
> > > b. Enables sw to coalesce some device specific registers, if needed.
> > > Cons:
> > > a. Not self-contained, requires PF's AQ which is anyway designed for such
> > purpose.
> > >
> > > 2. Legacy registers access via new MMIO region (v0 + sane reset)
> >
> > I think I would buy this one if it was actually in BAR0 exactly legacy with no
> > changes, no reset no nothing. a hack like Jason described that at least makes
> > legacy linux drivers work with no software glue. That is horrible, fragile but
> > engineering history is full of hacks like this.
> > If you need software in the middle then let's please make it robust.
> >
> Your ignorance and inclination to see only from the sw pov is not helpful here.
> Feedback from the PCI device side given here to be considered too.
After hearing this for the 10th time I don't think I'm not aware, thanks
for bringing this to my attention for the 11th time.
My point however is the last sentence *If you need software in the
middle then let's please make it robust.* If BAR0 hack does not work
then let's use sane interfaces as much as possible.
> This approach doesn't work well for device reset at scale which is a one way signal.
>
> Hence, I proposed the either the AQ over PF or the two-register scheme of above over MMIO VF.
>
> Does that work for you?
AQ works fine I think. This new one I haven't seen yet hard to say.
Probably has its own set of problems ...
> >
> > > Pros:
> > > a. smaller code in slow register access than AQ b. Sw cannot coalesce
> > > some device specific registers.
> > > c. Self-contained in the VF
> > >
> > > Cons:
> > > a. Relatively burdensome for the device as it requires more RW registers at
> > scale, but it can be indirect register.
> >
> > what I keep saying
> >
> > b. a ton of legacy bugs which hypervisor can't detect or work around
> > (reset is just one issue)
> >
> >
> > > 3. legacy registers tunneling with additional indirection via PCI
> > > capability
> >
> >
> > I don't know what's this one.
>
> Above example of legacy_register_addr and legacy register_data on the VF.
Talking about the two register thing. We actually have this already:
VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
But really since apparently you want to focus on how we solve
notifications let's focus on that. I feel the main issue is
poking at 1.x registers and at the same time at legacy registers.
The spec was written with a strong assumption that it is either or,
never a mix. Changing that will be painful.
This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER -
legacy can live in software then and we do not need to worry
about it in the spec. I got it that there's too much software
here for your taste, but the problem with legacy is really
that it was always ad hoc and implementation derived.
So we need to keep it siloed.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-23 18:16 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-23 21:32 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 21:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:17 PM
>
> On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Monday, May 15, 2023 6:09 AM
> > > To be more specific, device config often has the same layout under
> > > legacy and modern. Thus if getting an offset within device config,
> > > same decoding logic should be reusable between legacy and modern.
> > Modern device registers are directly accessible from the guest driver without
> mediation for VF,SF,SIOV.
> > Hence, there is no need to have commands for modern register access over
> some VQ.
>
> Yes, there's need, and some of it you were pushing yourself.
Yes, there is need without mediation of AQ.
It was discussed in depth cfgq idea with trade off.
> AQ will allow reduced memory with SRIOV, and support for SIOV.
Already discussed, v0 and v1, mediation via AQ is not the way for SRIOV.
SIOV starting with single goal of only backward compatilbity hence, AQ is also doesn't look future forward.
And your summary email, we parked it aside for now.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-23 21:32 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 21:32 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:17 PM
>
> On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Monday, May 15, 2023 6:09 AM
> > > To be more specific, device config often has the same layout under
> > > legacy and modern. Thus if getting an offset within device config,
> > > same decoding logic should be reusable between legacy and modern.
> > Modern device registers are directly accessible from the guest driver without
> mediation for VF,SF,SIOV.
> > Hence, there is no need to have commands for modern register access over
> some VQ.
>
> Yes, there's need, and some of it you were pushing yourself.
Yes, there is need without mediation of AQ.
It was discussed in depth cfgq idea with trade off.
> AQ will allow reduced memory with SRIOV, and support for SIOV.
Already discussed, v0 and v1, mediation via AQ is not the way for SRIOV.
SIOV starting with single goal of only backward compatilbity hence, AQ is also doesn't look future forward.
And your summary email, we parked it aside for now.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-23 18:48 ` Michael S. Tsirkin
@ 2023-05-23 22:22 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 22:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:49 PM
> > > iokit is from my previous work experience not virtio.
> > > For virtio we had a lot of trouble with hacks like this on windows.
> > I don't know why windows driver need to talk to PF driver until now when the
> AQ doesn't exists.
>
> What is it you want to know?
>
Why a VF driver needs to talk to a PF driver in windows without AQ?
> > > That's an important platform.
> > >
> > I am not sure how important is it.
> > What I learnt that a decade has passed and even after that virtio drivers are
> not part of the OS distro.
>
> Because with QEMU they are so easy to side-load. So we never bothered.
>
I don't understand your comment.
I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
> > It doesn't need to because guest is anyway going to do whatever it does.
>
> yes. But hypervisor can do work-arounds or detect broken functionality and
> stop cleanly.
>
You mentioned this several times.
Below requirement still holds fine.
> > The clear requirement is: to not fix legacy but to support legacy.
> > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> AQ/MMIO.
>
> And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > You didn't answer why VF BAR is a problem and suggest PF BAR.
>
> VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> is.
>
At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
How to interpret your comment?
> > If notification for VF is in VF -> hence other config registers also in VF, such
> blanket statements does not help.
>
> not other config registers. what's needed in VF driver should be retrievable
> through VF.
>
This applies to the modern device, for legacy emulation, I don't see this a mandatory.
> > Your comments and suggestions are helpful.
> > Some of them follow the loop that doesn't help.
> > Again, thanks a lot for the inputs and reviews.
> >
> > The design discussion helped to produce two solutions here than what was
> started in v0.
>
> OK good. Now I suggest you make an effort to come up with some compromise
> that will make this thread stop. I really can't say I even remember what we said
> in the beginning.
> You had this idea of starting with a problem statement document how about
> doing that? Because I feel every time people bring up some idea and a problem
> this idea would solve, you go "this is out of scope". Let's decide what the scope
> is?
>
The requirements are simple and straightforward at start of the cover letter.
i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
I propose:
Method-1:
1. VF legacy registers access over AQ of PF
2. VF driver notifications using MMIO of VF
Method-2:
1. legacy registers access using two MMIO registers (data and addr)
2. VF driver notifications using MMIO of VF
> > I am asking a 1.x PCI device has the VF notification BAR already, why we don't
> use it, why to build an additional one in device?
>
> Following this logic, a 1.x PCI device also has modern common config.
> All you really truly need is a flag to make it behave like legacy, from POV of
> device config and VQs.
>
Dual definition to same area in device is not elegant way to achieve it.
We already discussed that some area is RW, some is not. It moves location.
And it still doesn't solve the problem of reset.
Hence, either doing via AQ or 2 registers method would work fine.
It keeps legacy also isolated from mainstream.
> So let's try to keep drivers self-contained as much as possible.
>
You said "AQ is fine" at the end of email.
Hard to understand your mixed suggestion in context of sending v3.
> > And notification also works well that reuses transitional device's notification.
>
> I don't know what these are. You had some weird
> +struct virtio_admin_cmd_lq_notify_query_result {
> + u8 bar; /* PCI BAR number of the member VF */
> + u8 reserved[7];
> + le64 offset; /* Byte offset within the BAR */ };
> if you want to re-use the normal notifications for VFs we already describe them
> using capabilities.
>
That capability is intermixed with 1.x register queue_notify_off.
So, I see following options.
a. either to create a new legacy specific extended capability, or
b. say queue_notify_off doesn't apply for legacy notifications because this register doesn't exist for legacy
c. exchange it via the AQ like above struct.
> However, this creates a weird mix where there's this command that mostly
> follows legacy pci config layout except notifications which are there in the
> legacy pci config are actually somewhere else. And this is just inconsistent and
> confusing, and the reason for this is implementation defined.
Depends on how you see it.
It is at same level of inconsistency as 1.x for a legit reason which is to split config registers from data path register.
It is not implementation defined. Notification over AQ cannot reach same perf level as MMIO.
I will avoid repeating it.
>
> > 3 out of 4 functionalities are done on AQ of the TVQ.
> > So, it is time to stop asking to replicate the AQ functionality on some new
> TVQ.
> > This was discussed in depth.
>
> I don't think I ever did that.
>
It is in Jason's reply when he was suggesting tvq in this same v2 series.
> My point however is the last sentence *If you need software in the middle then
> let's please make it robust.* If BAR0 hack does not work then let's use sane
> interfaces as much as possible.
>
> AQ works fine I think. This new one I haven't seen yet hard to say.
> Probably has its own set of problems ...
>
Ok. you review and we discuss after that.
> > Above example of legacy_register_addr and legacy register_data on the VF.
>
> Talking about the two register thing. We actually have this already:
> VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
>
To which IOBAR? The one that doesn't exist in the VF?
It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
We can have new extended capability as,
VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it also avoid intermix with 1.x using above two registers.
>
> But really since apparently you want to focus on how we solve notifications let's
> focus on that. I feel the main issue is poking at 1.x registers and at the same time
> at legacy registers.
No, it is not 1.x registers. Register and bar offset is shared via the AQ.
> The spec was written with a strong assumption that it is either or, never a mix.
Transitional device supposed to support both as written in the spec.
No different here.
Now for real device.
> Changing that will be painful.
> This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy can live
> in software then and we do not need to worry about it in the spec. I got it that
> there's too much software here for your taste, but the problem with legacy is
> really that it was always ad hoc and implementation derived.
> So we need to keep it siloed.
This is why I am trying to avoid putting in PCI registers of the VF, but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG with two registers.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-23 22:22 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-23 22:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, May 23, 2023 2:49 PM
> > > iokit is from my previous work experience not virtio.
> > > For virtio we had a lot of trouble with hacks like this on windows.
> > I don't know why windows driver need to talk to PF driver until now when the
> AQ doesn't exists.
>
> What is it you want to know?
>
Why a VF driver needs to talk to a PF driver in windows without AQ?
> > > That's an important platform.
> > >
> > I am not sure how important is it.
> > What I learnt that a decade has passed and even after that virtio drivers are
> not part of the OS distro.
>
> Because with QEMU they are so easy to side-load. So we never bothered.
>
I don't understand your comment.
I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
> > It doesn't need to because guest is anyway going to do whatever it does.
>
> yes. But hypervisor can do work-arounds or detect broken functionality and
> stop cleanly.
>
You mentioned this several times.
Below requirement still holds fine.
> > The clear requirement is: to not fix legacy but to support legacy.
> > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> AQ/MMIO.
>
> And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > You didn't answer why VF BAR is a problem and suggest PF BAR.
>
> VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> is.
>
At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
How to interpret your comment?
> > If notification for VF is in VF -> hence other config registers also in VF, such
> blanket statements does not help.
>
> not other config registers. what's needed in VF driver should be retrievable
> through VF.
>
This applies to the modern device, for legacy emulation, I don't see this a mandatory.
> > Your comments and suggestions are helpful.
> > Some of them follow the loop that doesn't help.
> > Again, thanks a lot for the inputs and reviews.
> >
> > The design discussion helped to produce two solutions here than what was
> started in v0.
>
> OK good. Now I suggest you make an effort to come up with some compromise
> that will make this thread stop. I really can't say I even remember what we said
> in the beginning.
> You had this idea of starting with a problem statement document how about
> doing that? Because I feel every time people bring up some idea and a problem
> this idea would solve, you go "this is out of scope". Let's decide what the scope
> is?
>
The requirements are simple and straightforward at start of the cover letter.
i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
I propose:
Method-1:
1. VF legacy registers access over AQ of PF
2. VF driver notifications using MMIO of VF
Method-2:
1. legacy registers access using two MMIO registers (data and addr)
2. VF driver notifications using MMIO of VF
> > I am asking a 1.x PCI device has the VF notification BAR already, why we don't
> use it, why to build an additional one in device?
>
> Following this logic, a 1.x PCI device also has modern common config.
> All you really truly need is a flag to make it behave like legacy, from POV of
> device config and VQs.
>
Dual definition to same area in device is not elegant way to achieve it.
We already discussed that some area is RW, some is not. It moves location.
And it still doesn't solve the problem of reset.
Hence, either doing via AQ or 2 registers method would work fine.
It keeps legacy also isolated from mainstream.
> So let's try to keep drivers self-contained as much as possible.
>
You said "AQ is fine" at the end of email.
Hard to understand your mixed suggestion in context of sending v3.
> > And notification also works well that reuses transitional device's notification.
>
> I don't know what these are. You had some weird
> +struct virtio_admin_cmd_lq_notify_query_result {
> + u8 bar; /* PCI BAR number of the member VF */
> + u8 reserved[7];
> + le64 offset; /* Byte offset within the BAR */ };
> if you want to re-use the normal notifications for VFs we already describe them
> using capabilities.
>
That capability is intermixed with 1.x register queue_notify_off.
So, I see following options.
a. either to create a new legacy specific extended capability, or
b. say queue_notify_off doesn't apply for legacy notifications because this register doesn't exist for legacy
c. exchange it via the AQ like above struct.
> However, this creates a weird mix where there's this command that mostly
> follows legacy pci config layout except notifications which are there in the
> legacy pci config are actually somewhere else. And this is just inconsistent and
> confusing, and the reason for this is implementation defined.
Depends on how you see it.
It is at same level of inconsistency as 1.x for a legit reason which is to split config registers from data path register.
It is not implementation defined. Notification over AQ cannot reach same perf level as MMIO.
I will avoid repeating it.
>
> > 3 out of 4 functionalities are done on AQ of the TVQ.
> > So, it is time to stop asking to replicate the AQ functionality on some new
> TVQ.
> > This was discussed in depth.
>
> I don't think I ever did that.
>
It is in Jason's reply when he was suggesting tvq in this same v2 series.
> My point however is the last sentence *If you need software in the middle then
> let's please make it robust.* If BAR0 hack does not work then let's use sane
> interfaces as much as possible.
>
> AQ works fine I think. This new one I haven't seen yet hard to say.
> Probably has its own set of problems ...
>
Ok. you review and we discuss after that.
> > Above example of legacy_register_addr and legacy register_data on the VF.
>
> Talking about the two register thing. We actually have this already:
> VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
>
To which IOBAR? The one that doesn't exist in the VF?
It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
We can have new extended capability as,
VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it also avoid intermix with 1.x using above two registers.
>
> But really since apparently you want to focus on how we solve notifications let's
> focus on that. I feel the main issue is poking at 1.x registers and at the same time
> at legacy registers.
No, it is not 1.x registers. Register and bar offset is shared via the AQ.
> The spec was written with a strong assumption that it is either or, never a mix.
Transitional device supposed to support both as written in the spec.
No different here.
Now for real device.
> Changing that will be painful.
> This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy can live
> in software then and we do not need to worry about it in the spec. I got it that
> there's too much software here for your taste, but the problem with legacy is
> really that it was always ad hoc and implementation derived.
> So we need to keep it siloed.
This is why I am trying to avoid putting in PCI registers of the VF, but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG with two registers.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-23 22:22 ` Parav Pandit
@ 2023-05-24 1:17 ` Jason Wang
-1 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-24 1:17 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Parav:
On Wed, May 24, 2023 at 6:22 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > iokit is from my previous work experience not virtio.
> > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > I don't know why windows driver need to talk to PF driver until now when the
> > AQ doesn't exists.
> >
> > What is it you want to know?
> >
> Why a VF driver needs to talk to a PF driver in windows without AQ?
>
> > > > That's an important platform.
> > > >
> > > I am not sure how important is it.
> > > What I learnt that a decade has passed and even after that virtio drivers are
> > not part of the OS distro.
> >
> > Because with QEMU they are so easy to side-load. So we never bothered.
> >
> I don't understand your comment.
> I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
>
> > > It doesn't need to because guest is anyway going to do whatever it does.
> >
> > yes. But hypervisor can do work-arounds or detect broken functionality and
> > stop cleanly.
> >
> You mentioned this several times.
> Below requirement still holds fine.
>
> > > The clear requirement is: to not fix legacy but to support legacy.
> > > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> > AQ/MMIO.
> >
> > And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > > You didn't answer why VF BAR is a problem and suggest PF BAR.
> >
> > VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> > is.
> >
> At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
> How to interpret your comment?
>
> > > If notification for VF is in VF -> hence other config registers also in VF, such
> > blanket statements does not help.
> >
> > not other config registers. what's needed in VF driver should be retrievable
> > through VF.
> >
> This applies to the modern device, for legacy emulation, I don't see this a mandatory.
>
> > > Your comments and suggestions are helpful.
> > > Some of them follow the loop that doesn't help.
> > > Again, thanks a lot for the inputs and reviews.
> > >
> > > The design discussion helped to produce two solutions here than what was
> > started in v0.
> >
> > OK good. Now I suggest you make an effort to come up with some compromise
> > that will make this thread stop. I really can't say I even remember what we said
> > in the beginning.
>
> > You had this idea of starting with a problem statement document how about
> > doing that? Because I feel every time people bring up some idea and a problem
> > this idea would solve, you go "this is out of scope". Let's decide what the scope
> > is?
> >
> The requirements are simple and straightforward at start of the cover letter.
> i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
>
> I propose:
> Method-1:
> 1. VF legacy registers access over AQ of PF
> 2. VF driver notifications using MMIO of VF
>
> Method-2:
> 1. legacy registers access using two MMIO registers (data and addr)
> 2. VF driver notifications using MMIO of VF
>
Based on your above summary (and a more detailed one in another
thread), it's not hard to see the tricky part is the try to
provide/mediate modern features for legacy. Assuming we all agree:
1) We want transitional devices but not legacy only devices
2) Mediation is a must for legacy
Then finding a way to reuse a modern interface is much better than
building a new path back to the old legacy maze. I really think we
should go with a way to meditate on top of a modern device. This
probably requires some new feature bit(s) like
VIRTIO_NET_F_LEGACY_HEADER but it has the lowest cost.
1) no new hardware interface like registers/capabilities etc
2) no datapath meditation, hypervisor will negotiate
VIRTIO_NET_F_LEGACY_HEADER for legacy guest drivers
3) endian is fine, hypervisor know it's modern so it's little
4) reset is fine, hypervisor know it's modern so it can do proper meditation
5) notification is fine, hypervisor know it's modern so it can use
modern notification areas
6) all the behaviour is deterministic, no undocumented or code defined
legacy behaviours
Anything that prevents you from adding things like VIRTIO_NET_F_LEGACY_HEADER?
You wrote the following in v1:
"""
There is virtio PCI VF.
The user who attaches this VF to the VM, it doesn’t know if its guest
is to run legacy driver or 1.x
Hence hypervisor doesn’t want to run special stack when guest may
(likely) be 1.x and device also supports 1.x.
"""
I don't understand this actually. For "special stack", are you
referring to the mediation path for legacy? I don't see how
VIRTIO_NET_F_LEGACY_HEADER differs from new hardware interfaces like
admin virtqueue in this regard. If management stack wants a
transitional device, Hypervisor just need
1) for modern BAR, assign it directly to the VM if it wishes, no
mediation at all
2) for legacy BAR, trap and mediate
The only difference happens in 2), that is which kind of interfaces
should we go, existing modern or new legacy like admin virtqueue.
Or as Michael said, it would be better to explain how the hypervisor
expects to work may help.
Thanks
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-24 1:17 ` Jason Wang
0 siblings, 0 replies; 252+ messages in thread
From: Jason Wang @ 2023-05-24 1:17 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
Hi Parav:
On Wed, May 24, 2023 at 6:22 AM Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > iokit is from my previous work experience not virtio.
> > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > I don't know why windows driver need to talk to PF driver until now when the
> > AQ doesn't exists.
> >
> > What is it you want to know?
> >
> Why a VF driver needs to talk to a PF driver in windows without AQ?
>
> > > > That's an important platform.
> > > >
> > > I am not sure how important is it.
> > > What I learnt that a decade has passed and even after that virtio drivers are
> > not part of the OS distro.
> >
> > Because with QEMU they are so easy to side-load. So we never bothered.
> >
> I don't understand your comment.
> I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
>
> > > It doesn't need to because guest is anyway going to do whatever it does.
> >
> > yes. But hypervisor can do work-arounds or detect broken functionality and
> > stop cleanly.
> >
> You mentioned this several times.
> Below requirement still holds fine.
>
> > > The clear requirement is: to not fix legacy but to support legacy.
> > > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> > AQ/MMIO.
> >
> > And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > > You didn't answer why VF BAR is a problem and suggest PF BAR.
> >
> > VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> > is.
> >
> At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
> How to interpret your comment?
>
> > > If notification for VF is in VF -> hence other config registers also in VF, such
> > blanket statements does not help.
> >
> > not other config registers. what's needed in VF driver should be retrievable
> > through VF.
> >
> This applies to the modern device, for legacy emulation, I don't see this a mandatory.
>
> > > Your comments and suggestions are helpful.
> > > Some of them follow the loop that doesn't help.
> > > Again, thanks a lot for the inputs and reviews.
> > >
> > > The design discussion helped to produce two solutions here than what was
> > started in v0.
> >
> > OK good. Now I suggest you make an effort to come up with some compromise
> > that will make this thread stop. I really can't say I even remember what we said
> > in the beginning.
>
> > You had this idea of starting with a problem statement document how about
> > doing that? Because I feel every time people bring up some idea and a problem
> > this idea would solve, you go "this is out of scope". Let's decide what the scope
> > is?
> >
> The requirements are simple and straightforward at start of the cover letter.
> i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
>
> I propose:
> Method-1:
> 1. VF legacy registers access over AQ of PF
> 2. VF driver notifications using MMIO of VF
>
> Method-2:
> 1. legacy registers access using two MMIO registers (data and addr)
> 2. VF driver notifications using MMIO of VF
>
Based on your above summary (and a more detailed one in another
thread), it's not hard to see the tricky part is the try to
provide/mediate modern features for legacy. Assuming we all agree:
1) We want transitional devices but not legacy only devices
2) Mediation is a must for legacy
Then finding a way to reuse a modern interface is much better than
building a new path back to the old legacy maze. I really think we
should go with a way to meditate on top of a modern device. This
probably requires some new feature bit(s) like
VIRTIO_NET_F_LEGACY_HEADER but it has the lowest cost.
1) no new hardware interface like registers/capabilities etc
2) no datapath meditation, hypervisor will negotiate
VIRTIO_NET_F_LEGACY_HEADER for legacy guest drivers
3) endian is fine, hypervisor know it's modern so it's little
4) reset is fine, hypervisor know it's modern so it can do proper meditation
5) notification is fine, hypervisor know it's modern so it can use
modern notification areas
6) all the behaviour is deterministic, no undocumented or code defined
legacy behaviours
Anything that prevents you from adding things like VIRTIO_NET_F_LEGACY_HEADER?
You wrote the following in v1:
"""
There is virtio PCI VF.
The user who attaches this VF to the VM, it doesn’t know if its guest
is to run legacy driver or 1.x
Hence hypervisor doesn’t want to run special stack when guest may
(likely) be 1.x and device also supports 1.x.
"""
I don't understand this actually. For "special stack", are you
referring to the mediation path for legacy? I don't see how
VIRTIO_NET_F_LEGACY_HEADER differs from new hardware interfaces like
admin virtqueue in this regard. If management stack wants a
transitional device, Hypervisor just need
1) for modern BAR, assign it directly to the VM if it wishes, no
mediation at all
2) for legacy BAR, trap and mediate
The only difference happens in 2), that is which kind of interfaces
should we go, existing modern or new legacy like admin virtqueue.
Or as Michael said, it would be better to explain how the hypervisor
expects to work may help.
Thanks
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-23 21:32 ` [virtio-comment] " Parav Pandit
@ 2023-05-24 5:56 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 5:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 09:32:41PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:17 PM
> >
> > On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Monday, May 15, 2023 6:09 AM
> > > > To be more specific, device config often has the same layout under
> > > > legacy and modern. Thus if getting an offset within device config,
> > > > same decoding logic should be reusable between legacy and modern.
> > > Modern device registers are directly accessible from the guest driver without
> > mediation for VF,SF,SIOV.
> > > Hence, there is no need to have commands for modern register access over
> > some VQ.
> >
> > Yes, there's need, and some of it you were pushing yourself.
> Yes, there is need without mediation of AQ.
> It was discussed in depth cfgq idea with trade off.
>
> > AQ will allow reduced memory with SRIOV, and support for SIOV.
> Already discussed, v0 and v1, mediation via AQ is not the way for SRIOV.
> SIOV starting with single goal of only backward compatilbity hence, AQ is also doesn't look future forward.
>
> And your summary email, we parked it aside for now.
I am frankly lost at this point, this thread is too big.
So where do you want to go for this project? I would like something
specific, email threads are growing too big.
I think the reason is that there is actually a host of small
related subprojects. So when you propose just this small
feature, everyone jumps in with their own pet project
proposing addressing that too.
For example:
- SIOV guys were adding transport over VQ. It would seem that
can include support for legacy.
- For migration we need ability to report member events to owner.
It would seem that can be used to support INT#x.
- Ability to stick capabilities in extended config space
is a good idea, for a variety of reasons.
and so on. Understandably frustrating, and it is easy to go overboard
with generality.
If you feel your path for progress is clear, that's good.
if not, here are ideas for getting this unstuck:
- start a TODO.md file with a list of things we need to address,
and for each one a list of options
- everyone will add to this file then we get a higher level
picture and can discuss "this will be addressed by A, that by B"
- have a chat at the upstream developers meeting (May 31)
so far it's about vdpa blk
- anyone coming to the kvm forum to have a chat there?
Hope this helps.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-24 5:56 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 5:56 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 09:32:41PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:17 PM
> >
> > On Mon, May 15, 2023 at 02:30:55PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Monday, May 15, 2023 6:09 AM
> > > > To be more specific, device config often has the same layout under
> > > > legacy and modern. Thus if getting an offset within device config,
> > > > same decoding logic should be reusable between legacy and modern.
> > > Modern device registers are directly accessible from the guest driver without
> > mediation for VF,SF,SIOV.
> > > Hence, there is no need to have commands for modern register access over
> > some VQ.
> >
> > Yes, there's need, and some of it you were pushing yourself.
> Yes, there is need without mediation of AQ.
> It was discussed in depth cfgq idea with trade off.
>
> > AQ will allow reduced memory with SRIOV, and support for SIOV.
> Already discussed, v0 and v1, mediation via AQ is not the way for SRIOV.
> SIOV starting with single goal of only backward compatilbity hence, AQ is also doesn't look future forward.
>
> And your summary email, we parked it aside for now.
I am frankly lost at this point, this thread is too big.
So where do you want to go for this project? I would like something
specific, email threads are growing too big.
I think the reason is that there is actually a host of small
related subprojects. So when you propose just this small
feature, everyone jumps in with their own pet project
proposing addressing that too.
For example:
- SIOV guys were adding transport over VQ. It would seem that
can include support for legacy.
- For migration we need ability to report member events to owner.
It would seem that can be used to support INT#x.
- Ability to stick capabilities in extended config space
is a good idea, for a variety of reasons.
and so on. Understandably frustrating, and it is easy to go overboard
with generality.
If you feel your path for progress is clear, that's good.
if not, here are ideas for getting this unstuck:
- start a TODO.md file with a list of things we need to address,
and for each one a list of options
- everyone will add to this file then we get a higher level
picture and can discuss "this will be addressed by A, that by B"
- have a chat at the upstream developers meeting (May 31)
so far it's about vdpa blk
- anyone coming to the kvm forum to have a chat there?
Hope this helps.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-23 22:22 ` Parav Pandit
@ 2023-05-24 10:07 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 10:07 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > iokit is from my previous work experience not virtio.
> > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > I don't know why windows driver need to talk to PF driver until now when the
> > AQ doesn't exists.
> >
> > What is it you want to know?
> >
> Why a VF driver needs to talk to a PF driver in windows without AQ?
I don't know- it doesn't? If it did, it would be hard as we learned
when trying to support pci-bridge-seat. Doable, but annoying.
> > > > That's an important platform.
> > > >
> > > I am not sure how important is it.
> > > What I learnt that a decade has passed and even after that virtio drivers are
> > not part of the OS distro.
> >
> > Because with QEMU they are so easy to side-load. So we never bothered.
> >
> I don't understand your comment.
> I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
It's important for windows to work well with virtio. And it does because
we supply an install disk with virtio drivers through QEMU
> > > It doesn't need to because guest is anyway going to do whatever it does.
> >
> > yes. But hypervisor can do work-arounds or detect broken functionality and
> > stop cleanly.
> >
> You mentioned this several times.
> Below requirement still holds fine.
>
> > > The clear requirement is: to not fix legacy but to support legacy.
> > > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> > AQ/MMIO.
> >
> > And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > > You didn't answer why VF BAR is a problem and suggest PF BAR.
> >
> > VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> > is.
> >
> At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
> How to interpret your comment?
>
> > > If notification for VF is in VF -> hence other config registers also in VF, such
> > blanket statements does not help.
> >
> > not other config registers. what's needed in VF driver should be retrievable
> > through VF.
> >
> This applies to the modern device, for legacy emulation, I don't see this a mandatory.
>
> > > Your comments and suggestions are helpful.
> > > Some of them follow the loop that doesn't help.
> > > Again, thanks a lot for the inputs and reviews.
> > >
> > > The design discussion helped to produce two solutions here than what was
> > started in v0.
> >
> > OK good. Now I suggest you make an effort to come up with some compromise
> > that will make this thread stop. I really can't say I even remember what we said
> > in the beginning.
>
> > You had this idea of starting with a problem statement document how about
> > doing that? Because I feel every time people bring up some idea and a problem
> > this idea would solve, you go "this is out of scope". Let's decide what the scope
> > is?
> >
> The requirements are simple and straightforward at start of the cover letter.
> i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
>
> I propose:
> Method-1:
> 1. VF legacy registers access over AQ of PF
> 2. VF driver notifications using MMIO of VF
>
> Method-2:
> 1. legacy registers access using two MMIO registers (data and addr)
> 2. VF driver notifications using MMIO of VF
I think I prefer Method-1, I didn't exactly see Method-2 to judge
though.
> > > I am asking a 1.x PCI device has the VF notification BAR already, why we don't
> > use it, why to build an additional one in device?
> >
> > Following this logic, a 1.x PCI device also has modern common config.
> > All you really truly need is a flag to make it behave like legacy, from POV of
> > device config and VQs.
> >
> Dual definition to same area in device is not elegant way to achieve it.
> We already discussed that some area is RW, some is not. It moves location.
>
> And it still doesn't solve the problem of reset.
> Hence, either doing via AQ or 2 registers method would work fine.
>
> It keeps legacy also isolated from mainstream.
>
> > So let's try to keep drivers self-contained as much as possible.
> >
> You said "AQ is fine" at the end of email.
> Hard to understand your mixed suggestion in context of sending v3.
I am saying a simple thing actually:
- notifications are through VF
- rest of config is through PF (AQ)
seems like a consistent model to me.
> > > And notification also works well that reuses transitional device's notification.
> >
> > I don't know what these are. You had some weird
> > +struct virtio_admin_cmd_lq_notify_query_result {
> > + u8 bar; /* PCI BAR number of the member VF */
> > + u8 reserved[7];
> > + le64 offset; /* Byte offset within the BAR */ };
> > if you want to re-use the normal notifications for VFs we already describe them
> > using capabilities.
> >
> That capability is intermixed with 1.x register queue_notify_off.
Hmm yes annoying I forgot about this.
> So, I see following options.
> a. either to create a new legacy specific extended capability, or
> b. say queue_notify_off doesn't apply for legacy notifications because this register doesn't exist for legacy
> c. exchange it via the AQ like above struct.
a feels ok to me... maybe test the waters with express capability
as well, it will come handy down the road I think.
> > However, this creates a weird mix where there's this command that mostly
> > follows legacy pci config layout except notifications which are there in the
> > legacy pci config are actually somewhere else. And this is just inconsistent and
> > confusing, and the reason for this is implementation defined.
> Depends on how you see it.
> It is at same level of inconsistency as 1.x for a legit reason which is to split config registers from data path register.
>
> It is not implementation defined. Notification over AQ cannot reach same perf level as MMIO.
> I will avoid repeating it.
yes, no need to repeat. I think what you are missing is that I am
proposing this as an intermediate step while we work on
support for notifications, or even a patch in the patchset.
> >
> > > 3 out of 4 functionalities are done on AQ of the TVQ.
> > > So, it is time to stop asking to replicate the AQ functionality on some new
> > TVQ.
> > > This was discussed in depth.
> >
> > I don't think I ever did that.
> >
> It is in Jason's reply when he was suggesting tvq in this same v2 series.
I think what he meant is that tvq project will be reworked on top
of AQ. tvq, indeed, does expose a full transport on top of AQ
and it seems logical to try and add legacy guest support to that.
> > My point however is the last sentence *If you need software in the middle then
> > let's please make it robust.* If BAR0 hack does not work then let's use sane
> > interfaces as much as possible.
> >
>
> > AQ works fine I think. This new one I haven't seen yet hard to say.
> > Probably has its own set of problems ...
> >
> Ok. you review and we discuss after that.
>
> > > Above example of legacy_register_addr and legacy register_data on the VF.
> >
> > Talking about the two register thing. We actually have this already:
> > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> >
> To which IOBAR? The one that doesn't exist in the VF?
exactly. bar value is currently 0-6 other values are reserved.
we can steal e.g. 7 and say this is legacy IOBAR.
The advantage is that we can use this IOBAR for anything at all,
e.g. VIRTIO_PCI_CAP_DEVICE_CFG too.
> It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
>
> We can have new extended capability as,
>
> VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it also avoid intermix with 1.x using above two registers.
it's not a big deal, if you prefer a new capability fine.
> >
> > But really since apparently you want to focus on how we solve notifications let's
> > focus on that. I feel the main issue is poking at 1.x registers and at the same time
> > at legacy registers.
> No, it is not 1.x registers. Register and bar offset is shared via the AQ.
That's good then. It's not always clear what you want to do, this
is the problem with high level discussion as opposed to a specific
patch.
> > The spec was written with a strong assumption that it is either or, never a mix.
> Transitional device supposed to support both as written in the spec.
> No different here.
> Now for real device.
>
> > Changing that will be painful.
> > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy can live
> > in software then and we do not need to worry about it in the spec. I got it that
> > there's too much software here for your taste, but the problem with legacy is
> > really that it was always ad hoc and implementation derived.
> > So we need to keep it siloed.
>
> This is why I am trying to avoid putting in PCI registers of the VF,
> but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> with two registers.
Oh. You are saying access to capability previously indicated modern
and now it is also done for legacy? That indeed is a good point.
Not sure what to do here. I'll ponder a bit.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-24 10:07 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 10:07 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > iokit is from my previous work experience not virtio.
> > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > I don't know why windows driver need to talk to PF driver until now when the
> > AQ doesn't exists.
> >
> > What is it you want to know?
> >
> Why a VF driver needs to talk to a PF driver in windows without AQ?
I don't know- it doesn't? If it did, it would be hard as we learned
when trying to support pci-bridge-seat. Doable, but annoying.
> > > > That's an important platform.
> > > >
> > > I am not sure how important is it.
> > > What I learnt that a decade has passed and even after that virtio drivers are
> > not part of the OS distro.
> >
> > Because with QEMU they are so easy to side-load. So we never bothered.
> >
> I don't understand your comment.
> I was saying to my knowledge a Windows OS do not have a virtio drivers in their distro which you said is important.
It's important for windows to work well with virtio. And it does because
we supply an install disk with virtio drivers through QEMU
> > > It doesn't need to because guest is anyway going to do whatever it does.
> >
> > yes. But hypervisor can do work-arounds or detect broken functionality and
> > stop cleanly.
> >
> You mentioned this several times.
> Below requirement still holds fine.
>
> > > The clear requirement is: to not fix legacy but to support legacy.
> > > Fixing legacy is role of 1.x. Not the role of legacy interface itself using
> > AQ/MMIO.
> >
> > And for purpose of BAR mapping, such BAR in the VF can be mapped too.
> > > You didn't answer why VF BAR is a problem and suggest PF BAR.
> >
> > VF BAR by itself is not a problem. Discovering VF BAR mapping offset through PF
> > is.
> >
> At the end of the email you right that "AQ is fine" but here you write discovering that through the PF AQ is.
> How to interpret your comment?
>
> > > If notification for VF is in VF -> hence other config registers also in VF, such
> > blanket statements does not help.
> >
> > not other config registers. what's needed in VF driver should be retrievable
> > through VF.
> >
> This applies to the modern device, for legacy emulation, I don't see this a mandatory.
>
> > > Your comments and suggestions are helpful.
> > > Some of them follow the loop that doesn't help.
> > > Again, thanks a lot for the inputs and reviews.
> > >
> > > The design discussion helped to produce two solutions here than what was
> > started in v0.
> >
> > OK good. Now I suggest you make an effort to come up with some compromise
> > that will make this thread stop. I really can't say I even remember what we said
> > in the beginning.
>
> > You had this idea of starting with a problem statement document how about
> > doing that? Because I feel every time people bring up some idea and a problem
> > this idea would solve, you go "this is out of scope". Let's decide what the scope
> > is?
> >
> The requirements are simple and straightforward at start of the cover letter.
> i.e. for quick reference in short: A PCI VF passthrough to guest VM to support legacy + 1.x.
>
> I propose:
> Method-1:
> 1. VF legacy registers access over AQ of PF
> 2. VF driver notifications using MMIO of VF
>
> Method-2:
> 1. legacy registers access using two MMIO registers (data and addr)
> 2. VF driver notifications using MMIO of VF
I think I prefer Method-1, I didn't exactly see Method-2 to judge
though.
> > > I am asking a 1.x PCI device has the VF notification BAR already, why we don't
> > use it, why to build an additional one in device?
> >
> > Following this logic, a 1.x PCI device also has modern common config.
> > All you really truly need is a flag to make it behave like legacy, from POV of
> > device config and VQs.
> >
> Dual definition to same area in device is not elegant way to achieve it.
> We already discussed that some area is RW, some is not. It moves location.
>
> And it still doesn't solve the problem of reset.
> Hence, either doing via AQ or 2 registers method would work fine.
>
> It keeps legacy also isolated from mainstream.
>
> > So let's try to keep drivers self-contained as much as possible.
> >
> You said "AQ is fine" at the end of email.
> Hard to understand your mixed suggestion in context of sending v3.
I am saying a simple thing actually:
- notifications are through VF
- rest of config is through PF (AQ)
seems like a consistent model to me.
> > > And notification also works well that reuses transitional device's notification.
> >
> > I don't know what these are. You had some weird
> > +struct virtio_admin_cmd_lq_notify_query_result {
> > + u8 bar; /* PCI BAR number of the member VF */
> > + u8 reserved[7];
> > + le64 offset; /* Byte offset within the BAR */ };
> > if you want to re-use the normal notifications for VFs we already describe them
> > using capabilities.
> >
> That capability is intermixed with 1.x register queue_notify_off.
Hmm yes annoying I forgot about this.
> So, I see following options.
> a. either to create a new legacy specific extended capability, or
> b. say queue_notify_off doesn't apply for legacy notifications because this register doesn't exist for legacy
> c. exchange it via the AQ like above struct.
a feels ok to me... maybe test the waters with express capability
as well, it will come handy down the road I think.
> > However, this creates a weird mix where there's this command that mostly
> > follows legacy pci config layout except notifications which are there in the
> > legacy pci config are actually somewhere else. And this is just inconsistent and
> > confusing, and the reason for this is implementation defined.
> Depends on how you see it.
> It is at same level of inconsistency as 1.x for a legit reason which is to split config registers from data path register.
>
> It is not implementation defined. Notification over AQ cannot reach same perf level as MMIO.
> I will avoid repeating it.
yes, no need to repeat. I think what you are missing is that I am
proposing this as an intermediate step while we work on
support for notifications, or even a patch in the patchset.
> >
> > > 3 out of 4 functionalities are done on AQ of the TVQ.
> > > So, it is time to stop asking to replicate the AQ functionality on some new
> > TVQ.
> > > This was discussed in depth.
> >
> > I don't think I ever did that.
> >
> It is in Jason's reply when he was suggesting tvq in this same v2 series.
I think what he meant is that tvq project will be reworked on top
of AQ. tvq, indeed, does expose a full transport on top of AQ
and it seems logical to try and add legacy guest support to that.
> > My point however is the last sentence *If you need software in the middle then
> > let's please make it robust.* If BAR0 hack does not work then let's use sane
> > interfaces as much as possible.
> >
>
> > AQ works fine I think. This new one I haven't seen yet hard to say.
> > Probably has its own set of problems ...
> >
> Ok. you review and we discuss after that.
>
> > > Above example of legacy_register_addr and legacy register_data on the VF.
> >
> > Talking about the two register thing. We actually have this already:
> > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> >
> To which IOBAR? The one that doesn't exist in the VF?
exactly. bar value is currently 0-6 other values are reserved.
we can steal e.g. 7 and say this is legacy IOBAR.
The advantage is that we can use this IOBAR for anything at all,
e.g. VIRTIO_PCI_CAP_DEVICE_CFG too.
> It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
>
> We can have new extended capability as,
>
> VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it also avoid intermix with 1.x using above two registers.
it's not a big deal, if you prefer a new capability fine.
> >
> > But really since apparently you want to focus on how we solve notifications let's
> > focus on that. I feel the main issue is poking at 1.x registers and at the same time
> > at legacy registers.
> No, it is not 1.x registers. Register and bar offset is shared via the AQ.
That's good then. It's not always clear what you want to do, this
is the problem with high level discussion as opposed to a specific
patch.
> > The spec was written with a strong assumption that it is either or, never a mix.
> Transitional device supposed to support both as written in the spec.
> No different here.
> Now for real device.
>
> > Changing that will be painful.
> > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy can live
> > in software then and we do not need to worry about it in the spec. I got it that
> > there's too much software here for your taste, but the problem with legacy is
> > really that it was always ad hoc and implementation derived.
> > So we need to keep it siloed.
>
> This is why I am trying to avoid putting in PCI registers of the VF,
> but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> with two registers.
Oh. You are saying access to capability previously indicated modern
and now it is also done for legacy? That indeed is a good point.
Not sure what to do here. I'll ponder a bit.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-24 5:56 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-24 18:57 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 18:57 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 1:57 AM
> I am frankly lost at this point, this thread is too big.
>
> So where do you want to go for this project? I would like something specific,
> email threads are growing too big.
> I think the reason is that there is actually a host of small related subprojects. So
> when you propose just this small feature, everyone jumps in with their own pet
> project proposing addressing that too.
>
> For example:
> - SIOV guys were adding transport over VQ. It would seem that
> can include support for legacy.
Yes. SIOV can support legacy and modern emulation.
We need to have the SIOV devices without this emulation support so that they natively good.
> - For migration we need ability to report member events to owner.
Which events you have in mind?
The AQ is design to carry the migration of the VFs and would be controlled via the AQ so it knows whats going on.
> It would seem that can be used to support INT#x.
> - Ability to stick capabilities in extended config space
> is a good idea, for a variety of reasons.
>
> and so on. Understandably frustrating, and it is easy to go overboard with
> generality.
>
> If you feel your path for progress is clear, that's good.
The path seems clear to me for supporting legacy registers access for the VFs and future SIOV.
Let's conclude it in other thread we are discussing of patch 1/2.
> if not, here are ideas for getting this unstuck:
> - start a TODO.md file with a list of things we need to address,
> and for each one a list of options
> - everyone will add to this file then we get a higher level
> picture and can discuss "this will be addressed by A, that by B"
This is certainly present in my personal todo list to discuss, I am happy to maintain in the public virtio doc at
https://github.com/oasis-tcs/virtio-docs/
> - have a chat at the upstream developers meeting (May 31)
> so far it's about vdpa blk
> - anyone coming to the kvm forum to have a chat there?
>
> Hope this helps.
Definitely a good suggestion.
I will send PR for virtio-docs to maintain our TODO list.
Since some of the discussions are ongoing basis if we can meet monthly on more practically on need basis, it will help to make progress faster.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-24 18:57 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 18:57 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 1:57 AM
> I am frankly lost at this point, this thread is too big.
>
> So where do you want to go for this project? I would like something specific,
> email threads are growing too big.
> I think the reason is that there is actually a host of small related subprojects. So
> when you propose just this small feature, everyone jumps in with their own pet
> project proposing addressing that too.
>
> For example:
> - SIOV guys were adding transport over VQ. It would seem that
> can include support for legacy.
Yes. SIOV can support legacy and modern emulation.
We need to have the SIOV devices without this emulation support so that they natively good.
> - For migration we need ability to report member events to owner.
Which events you have in mind?
The AQ is design to carry the migration of the VFs and would be controlled via the AQ so it knows whats going on.
> It would seem that can be used to support INT#x.
> - Ability to stick capabilities in extended config space
> is a good idea, for a variety of reasons.
>
> and so on. Understandably frustrating, and it is easy to go overboard with
> generality.
>
> If you feel your path for progress is clear, that's good.
The path seems clear to me for supporting legacy registers access for the VFs and future SIOV.
Let's conclude it in other thread we are discussing of patch 1/2.
> if not, here are ideas for getting this unstuck:
> - start a TODO.md file with a list of things we need to address,
> and for each one a list of options
> - everyone will add to this file then we get a higher level
> picture and can discuss "this will be addressed by A, that by B"
This is certainly present in my personal todo list to discuss, I am happy to maintain in the public virtio doc at
https://github.com/oasis-tcs/virtio-docs/
> - have a chat at the upstream developers meeting (May 31)
> so far it's about vdpa blk
> - anyone coming to the kvm forum to have a chat there?
>
> Hope this helps.
Definitely a good suggestion.
I will send PR for virtio-docs to maintain our TODO list.
Since some of the discussions are ongoing basis if we can meet monthly on more practically on need basis, it will help to make progress faster.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-24 10:07 ` Michael S. Tsirkin
@ 2023-05-24 19:18 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 19:18 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 6:08 AM
>
> On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > > iokit is from my previous work experience not virtio.
> > > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > > I don't know why windows driver need to talk to PF driver until
> > > > now when the
> > > AQ doesn't exists.
> > >
> > > What is it you want to know?
> > >
> > Why a VF driver needs to talk to a PF driver in windows without AQ?
>
> I don't know- it doesn't? If it did, it would be hard as we learned when trying to
> support pci-bridge-seat. Doable, but annoying.
If it does today, something likely is missing in that OS. Their model for sriov has not evolved as much as Linux from what I see.
So not to worry about it, as they may have to implement more things.
>
> > > > > That's an important platform.
> > > > >
> > > > I am not sure how important is it.
> > > > What I learnt that a decade has passed and even after that virtio
> > > > drivers are
> > > not part of the OS distro.
> > >
> > > Because with QEMU they are so easy to side-load. So we never bothered.
> > >
> > I don't understand your comment.
> > I was saying to my knowledge a Windows OS do not have a virtio drivers in
> their distro which you said is important.
>
> It's important for windows to work well with virtio. And it does because we
> supply an install disk with virtio drivers through QEMU
Ok. So WDM can use their IPR model or will be able to build infra as/if needed.
> > I propose:
> > Method-1:
> > 1. VF legacy registers access over AQ of PF 2. VF driver notifications
> > using MMIO of VF
> >
> > Method-2:
> > 1. legacy registers access using two MMIO registers (data and addr) 2.
> > VF driver notifications using MMIO of VF
>
> I think I prefer Method-1, I didn't exactly see Method-2 to judge though.
>
Ok. so lets step forward in v3 using 4 commands for registers access.
2 RW commands for common config.
2 RW commands for device config.
Notification on below.
>
> > > > I am asking a 1.x PCI device has the VF notification BAR already,
> > > > why we don't
> > > use it, why to build an additional one in device?
> > >
> > > Following this logic, a 1.x PCI device also has modern common config.
> > > All you really truly need is a flag to make it behave like legacy,
> > > from POV of device config and VQs.
> > >
> > Dual definition to same area in device is not elegant way to achieve it.
> > We already discussed that some area is RW, some is not. It moves location.
> >
> > And it still doesn't solve the problem of reset.
> > Hence, either doing via AQ or 2 registers method would work fine.
> >
> > It keeps legacy also isolated from mainstream.
> >
> > > So let's try to keep drivers self-contained as much as possible.
> > >
> > You said "AQ is fine" at the end of email.
> > Hard to understand your mixed suggestion in context of sending v3.
>
> I am saying a simple thing actually:
> - notifications are through VF
> - rest of config is through PF (AQ)
> seems like a consistent model to me.
>
Sounds good to me.
What about discovery the notification place for the VF?
We have two options.
1. discover them via AQ like config registers access
2. discover it via extended PCI capability
#1 is far more programable and does not need to bake in the PCI device layout.
So this way one day in future we can get rid of it more easily for those PCI devices who may prefer everything in the hw.
> > > if you want to re-use the normal notifications for VFs we already
> > > describe them using capabilities.
> > >
> > That capability is intermixed with 1.x register queue_notify_off.
>
> Hmm yes annoying I forgot about this.
>
> > So, I see following options.
> > a. either to create a new legacy specific extended capability, or b.
> > say queue_notify_off doesn't apply for legacy notifications because
> > this register doesn't exist for legacy c. exchange it via the AQ like above
> struct.
>
> a feels ok to me... maybe test the waters with express capability as well, it will
> come handy down the road I think.
Yes, if this is really needed for some OS, this addition will be possible in future in the spec and also of the devices in CC list here.
>
> > > However, this creates a weird mix where there's this command that
> > > mostly follows legacy pci config layout except notifications which
> > > are there in the legacy pci config are actually somewhere else. And
> > > this is just inconsistent and confusing, and the reason for this is
> implementation defined.
> > Depends on how you see it.
> > It is at same level of inconsistency as 1.x for a legit reason which is to split
> config registers from data path register.
> >
> > It is not implementation defined. Notification over AQ cannot reach same perf
> level as MMIO.
> > I will avoid repeating it.
>
> yes, no need to repeat. I think what you are missing is that I am proposing this
> as an intermediate step while we work on support for notifications, or even a
> patch in the patchset.
>
Like said above, share your thoughts on the two options and I will modify this patch or add new in v3.
>
> I think what he meant is that tvq project will be reworked on top of AQ. tvq,
> indeed, does expose a full transport on top of AQ and it seems logical to try and
> add legacy guest support to that.
Yes, I also acked that for SIOV devices to have legacy support on top of base SIOV device interface.
> > > Talking about the two register thing. We actually have this already:
> > > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> > >
> > To which IOBAR? The one that doesn't exist in the VF?
>
> exactly. bar value is currently 0-6 other values are reserved.
> we can steal e.g. 7 and say this is legacy IOBAR.
> The advantage is that we can use this IOBAR for anything at all, e.g.
> VIRTIO_PCI_CAP_DEVICE_CFG too.
>
> > It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
> >
> > We can have new extended capability as,
> >
> > VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it
> also avoid intermix with 1.x using above two registers.
>
> it's not a big deal, if you prefer a new capability fine.
>
> > >
> > > But really since apparently you want to focus on how we solve
> > > notifications let's focus on that. I feel the main issue is poking
> > > at 1.x registers and at the same time at legacy registers.
> > No, it is not 1.x registers. Register and bar offset is shared via the AQ.
>
> That's good then. It's not always clear what you want to do, this is the problem
> with high level discussion as opposed to a specific patch.
>
> > > The spec was written with a strong assumption that it is either or, never a
> mix.
> > Transitional device supposed to support both as written in the spec.
> > No different here.
> > Now for real device.
> >
> > > Changing that will be painful.
> > > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy
> > > can live in software then and we do not need to worry about it in
> > > the spec. I got it that there's too much software here for your
> > > taste, but the problem with legacy is really that it was always ad hoc and
> implementation derived.
> > > So we need to keep it siloed.
> >
> > This is why I am trying to avoid putting in PCI registers of the VF,
> > but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> > with two registers.
>
> Oh. You are saying access to capability previously indicated modern and now it
> is also done for legacy? That indeed is a good point.
> Not sure what to do here. I'll ponder a bit.
The new capability in mind is:
VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
struct virtio_pci_legacy_cfg_cap {
struct virtio_pci_cap cap;
le32 addr; /* register offset, width, op=RD/WR, status:0=pending,1=done)
le32 reg_data;
};
This will be hard to do for SIOV in future, so I prefer AQ cmd for config register access and notification discovery.
WDYT?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-24 19:18 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 19:18 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 6:08 AM
>
> On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > > iokit is from my previous work experience not virtio.
> > > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > > I don't know why windows driver need to talk to PF driver until
> > > > now when the
> > > AQ doesn't exists.
> > >
> > > What is it you want to know?
> > >
> > Why a VF driver needs to talk to a PF driver in windows without AQ?
>
> I don't know- it doesn't? If it did, it would be hard as we learned when trying to
> support pci-bridge-seat. Doable, but annoying.
If it does today, something likely is missing in that OS. Their model for sriov has not evolved as much as Linux from what I see.
So not to worry about it, as they may have to implement more things.
>
> > > > > That's an important platform.
> > > > >
> > > > I am not sure how important is it.
> > > > What I learnt that a decade has passed and even after that virtio
> > > > drivers are
> > > not part of the OS distro.
> > >
> > > Because with QEMU they are so easy to side-load. So we never bothered.
> > >
> > I don't understand your comment.
> > I was saying to my knowledge a Windows OS do not have a virtio drivers in
> their distro which you said is important.
>
> It's important for windows to work well with virtio. And it does because we
> supply an install disk with virtio drivers through QEMU
Ok. So WDM can use their IPR model or will be able to build infra as/if needed.
> > I propose:
> > Method-1:
> > 1. VF legacy registers access over AQ of PF 2. VF driver notifications
> > using MMIO of VF
> >
> > Method-2:
> > 1. legacy registers access using two MMIO registers (data and addr) 2.
> > VF driver notifications using MMIO of VF
>
> I think I prefer Method-1, I didn't exactly see Method-2 to judge though.
>
Ok. so lets step forward in v3 using 4 commands for registers access.
2 RW commands for common config.
2 RW commands for device config.
Notification on below.
>
> > > > I am asking a 1.x PCI device has the VF notification BAR already,
> > > > why we don't
> > > use it, why to build an additional one in device?
> > >
> > > Following this logic, a 1.x PCI device also has modern common config.
> > > All you really truly need is a flag to make it behave like legacy,
> > > from POV of device config and VQs.
> > >
> > Dual definition to same area in device is not elegant way to achieve it.
> > We already discussed that some area is RW, some is not. It moves location.
> >
> > And it still doesn't solve the problem of reset.
> > Hence, either doing via AQ or 2 registers method would work fine.
> >
> > It keeps legacy also isolated from mainstream.
> >
> > > So let's try to keep drivers self-contained as much as possible.
> > >
> > You said "AQ is fine" at the end of email.
> > Hard to understand your mixed suggestion in context of sending v3.
>
> I am saying a simple thing actually:
> - notifications are through VF
> - rest of config is through PF (AQ)
> seems like a consistent model to me.
>
Sounds good to me.
What about discovery the notification place for the VF?
We have two options.
1. discover them via AQ like config registers access
2. discover it via extended PCI capability
#1 is far more programable and does not need to bake in the PCI device layout.
So this way one day in future we can get rid of it more easily for those PCI devices who may prefer everything in the hw.
> > > if you want to re-use the normal notifications for VFs we already
> > > describe them using capabilities.
> > >
> > That capability is intermixed with 1.x register queue_notify_off.
>
> Hmm yes annoying I forgot about this.
>
> > So, I see following options.
> > a. either to create a new legacy specific extended capability, or b.
> > say queue_notify_off doesn't apply for legacy notifications because
> > this register doesn't exist for legacy c. exchange it via the AQ like above
> struct.
>
> a feels ok to me... maybe test the waters with express capability as well, it will
> come handy down the road I think.
Yes, if this is really needed for some OS, this addition will be possible in future in the spec and also of the devices in CC list here.
>
> > > However, this creates a weird mix where there's this command that
> > > mostly follows legacy pci config layout except notifications which
> > > are there in the legacy pci config are actually somewhere else. And
> > > this is just inconsistent and confusing, and the reason for this is
> implementation defined.
> > Depends on how you see it.
> > It is at same level of inconsistency as 1.x for a legit reason which is to split
> config registers from data path register.
> >
> > It is not implementation defined. Notification over AQ cannot reach same perf
> level as MMIO.
> > I will avoid repeating it.
>
> yes, no need to repeat. I think what you are missing is that I am proposing this
> as an intermediate step while we work on support for notifications, or even a
> patch in the patchset.
>
Like said above, share your thoughts on the two options and I will modify this patch or add new in v3.
>
> I think what he meant is that tvq project will be reworked on top of AQ. tvq,
> indeed, does expose a full transport on top of AQ and it seems logical to try and
> add legacy guest support to that.
Yes, I also acked that for SIOV devices to have legacy support on top of base SIOV device interface.
> > > Talking about the two register thing. We actually have this already:
> > > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> > >
> > To which IOBAR? The one that doesn't exist in the VF?
>
> exactly. bar value is currently 0-6 other values are reserved.
> we can steal e.g. 7 and say this is legacy IOBAR.
> The advantage is that we can use this IOBAR for anything at all, e.g.
> VIRTIO_PCI_CAP_DEVICE_CFG too.
>
> > It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
> >
> > We can have new extended capability as,
> >
> > VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it
> also avoid intermix with 1.x using above two registers.
>
> it's not a big deal, if you prefer a new capability fine.
>
> > >
> > > But really since apparently you want to focus on how we solve
> > > notifications let's focus on that. I feel the main issue is poking
> > > at 1.x registers and at the same time at legacy registers.
> > No, it is not 1.x registers. Register and bar offset is shared via the AQ.
>
> That's good then. It's not always clear what you want to do, this is the problem
> with high level discussion as opposed to a specific patch.
>
> > > The spec was written with a strong assumption that it is either or, never a
> mix.
> > Transitional device supposed to support both as written in the spec.
> > No different here.
> > Now for real device.
> >
> > > Changing that will be painful.
> > > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy
> > > can live in software then and we do not need to worry about it in
> > > the spec. I got it that there's too much software here for your
> > > taste, but the problem with legacy is really that it was always ad hoc and
> implementation derived.
> > > So we need to keep it siloed.
> >
> > This is why I am trying to avoid putting in PCI registers of the VF,
> > but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> > with two registers.
>
> Oh. You are saying access to capability previously indicated modern and now it
> is also done for legacy? That indeed is a good point.
> Not sure what to do here. I'll ponder a bit.
The new capability in mind is:
VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
struct virtio_pci_legacy_cfg_cap {
struct virtio_pci_cap cap;
le32 addr; /* register offset, width, op=RD/WR, status:0=pending,1=done)
le32 reg_data;
};
This will be hard to do for SIOV in future, so I prefer AQ cmd for config register access and notification discovery.
WDYT?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-24 18:57 ` [virtio-comment] " Parav Pandit
@ 2023-05-24 19:58 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 19:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 06:57:30PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 1:57 AM
>
>
> > I am frankly lost at this point, this thread is too big.
> >
> > So where do you want to go for this project? I would like something specific,
> > email threads are growing too big.
> > I think the reason is that there is actually a host of small related subprojects. So
> > when you propose just this small feature, everyone jumps in with their own pet
> > project proposing addressing that too.
> >
> > For example:
> > - SIOV guys were adding transport over VQ. It would seem that
> > can include support for legacy.
> Yes. SIOV can support legacy and modern emulation.
> We need to have the SIOV devices without this emulation support so that they natively good.
>
> > - For migration we need ability to report member events to owner.
> Which events you have in mind?
memory faults/dirtying for example.
> The AQ is design to carry the migration of the VFs and would be controlled via the AQ so it knows whats going on.
>
> > It would seem that can be used to support INT#x.
> > - Ability to stick capabilities in extended config space
> > is a good idea, for a variety of reasons.
> >
> > and so on. Understandably frustrating, and it is easy to go overboard with
> > generality.
> >
> > If you feel your path for progress is clear, that's good.
> The path seems clear to me for supporting legacy registers access for the VFs and future SIOV.
> Let's conclude it in other thread we are discussing of patch 1/2.
>
> > if not, here are ideas for getting this unstuck:
> > - start a TODO.md file with a list of things we need to address,
> > and for each one a list of options
> > - everyone will add to this file then we get a higher level
> > picture and can discuss "this will be addressed by A, that by B"
> This is certainly present in my personal todo list to discuss, I am happy to maintain in the public virtio doc at
> https://github.com/oasis-tcs/virtio-docs/
>
> > - have a chat at the upstream developers meeting (May 31)
> > so far it's about vdpa blk
> > - anyone coming to the kvm forum to have a chat there?
> >
> > Hope this helps.
> Definitely a good suggestion.
> I will send PR for virtio-docs to maintain our TODO list.
Hmm. This is just a dumping ground then. Fine by me, but what I had in
mind is a common list of known problems to address by TC as a whole.
If you want that we want patches on list and discussion, maybe even
voting.
> Since some of the discussions are ongoing basis if we can meet monthly
> on more practically on need basis, it will help to make progress
> faster.
That meeting is byweekly actually I think.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-24 19:58 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 19:58 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 06:57:30PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 1:57 AM
>
>
> > I am frankly lost at this point, this thread is too big.
> >
> > So where do you want to go for this project? I would like something specific,
> > email threads are growing too big.
> > I think the reason is that there is actually a host of small related subprojects. So
> > when you propose just this small feature, everyone jumps in with their own pet
> > project proposing addressing that too.
> >
> > For example:
> > - SIOV guys were adding transport over VQ. It would seem that
> > can include support for legacy.
> Yes. SIOV can support legacy and modern emulation.
> We need to have the SIOV devices without this emulation support so that they natively good.
>
> > - For migration we need ability to report member events to owner.
> Which events you have in mind?
memory faults/dirtying for example.
> The AQ is design to carry the migration of the VFs and would be controlled via the AQ so it knows whats going on.
>
> > It would seem that can be used to support INT#x.
> > - Ability to stick capabilities in extended config space
> > is a good idea, for a variety of reasons.
> >
> > and so on. Understandably frustrating, and it is easy to go overboard with
> > generality.
> >
> > If you feel your path for progress is clear, that's good.
> The path seems clear to me for supporting legacy registers access for the VFs and future SIOV.
> Let's conclude it in other thread we are discussing of patch 1/2.
>
> > if not, here are ideas for getting this unstuck:
> > - start a TODO.md file with a list of things we need to address,
> > and for each one a list of options
> > - everyone will add to this file then we get a higher level
> > picture and can discuss "this will be addressed by A, that by B"
> This is certainly present in my personal todo list to discuss, I am happy to maintain in the public virtio doc at
> https://github.com/oasis-tcs/virtio-docs/
>
> > - have a chat at the upstream developers meeting (May 31)
> > so far it's about vdpa blk
> > - anyone coming to the kvm forum to have a chat there?
> >
> > Hope this helps.
> Definitely a good suggestion.
> I will send PR for virtio-docs to maintain our TODO list.
Hmm. This is just a dumping ground then. Fine by me, but what I had in
mind is a common list of known problems to address by TC as a whole.
If you want that we want patches on list and discussion, maybe even
voting.
> Since some of the discussions are ongoing basis if we can meet monthly
> on more practically on need basis, it will help to make progress
> faster.
That meeting is byweekly actually I think.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-24 19:58 ` [virtio-comment] " Michael S. Tsirkin
@ 2023-05-24 20:01 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 20:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 3:58 PM
> > > Hope this helps.
> > Definitely a good suggestion.
> > I will send PR for virtio-docs to maintain our TODO list.
>
> Hmm. This is just a dumping ground then. Fine by me, but what I had in mind is
> a common list of known problems to address by TC as a whole.
> If you want that we want patches on list and discussion, maybe even voting.
>
It is not a dumping ground, it is the list of things to track and document without searching emails.
It is a patch on the list for the doc in virtio-docs area that if needed can be voted.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] RE: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-24 20:01 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 20:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 3:58 PM
> > > Hope this helps.
> > Definitely a good suggestion.
> > I will send PR for virtio-docs to maintain our TODO list.
>
> Hmm. This is just a dumping ground then. Fine by me, but what I had in mind is
> a common list of known problems to address by TC as a whole.
> If you want that we want patches on list and discussion, maybe even voting.
>
It is not a dumping ground, it is the list of things to track and document without searching emails.
It is a patch on the list for the doc in virtio-docs area that if needed can be voted.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-24 19:18 ` Parav Pandit
@ 2023-05-24 20:12 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 20:12 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 07:18:56PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 6:08 AM
> >
> > On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > > > iokit is from my previous work experience not virtio.
> > > > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > > > I don't know why windows driver need to talk to PF driver until
> > > > > now when the
> > > > AQ doesn't exists.
> > > >
> > > > What is it you want to know?
> > > >
> > > Why a VF driver needs to talk to a PF driver in windows without AQ?
> >
> > I don't know- it doesn't? If it did, it would be hard as we learned when trying to
> > support pci-bridge-seat. Doable, but annoying.
> If it does today, something likely is missing in that OS. Their model for sriov has not evolved as much as Linux from what I see.
> So not to worry about it, as they may have to implement more things.
>
> >
> > > > > > That's an important platform.
> > > > > >
> > > > > I am not sure how important is it.
> > > > > What I learnt that a decade has passed and even after that virtio
> > > > > drivers are
> > > > not part of the OS distro.
> > > >
> > > > Because with QEMU they are so easy to side-load. So we never bothered.
> > > >
> > > I don't understand your comment.
> > > I was saying to my knowledge a Windows OS do not have a virtio drivers in
> > their distro which you said is important.
> >
> > It's important for windows to work well with virtio. And it does because we
> > supply an install disk with virtio drivers through QEMU
>
> Ok. So WDM can use their IPR model or will be able to build infra as/if needed.
>
> > > I propose:
> > > Method-1:
> > > 1. VF legacy registers access over AQ of PF 2. VF driver notifications
> > > using MMIO of VF
> > >
> > > Method-2:
> > > 1. legacy registers access using two MMIO registers (data and addr) 2.
> > > VF driver notifications using MMIO of VF
> >
> > I think I prefer Method-1, I didn't exactly see Method-2 to judge though.
> >
> Ok. so lets step forward in v3 using 4 commands for registers access.
> 2 RW commands for common config.
> 2 RW commands for device config.
>
> Notification on below.
>
> >
> > > > > I am asking a 1.x PCI device has the VF notification BAR already,
> > > > > why we don't
> > > > use it, why to build an additional one in device?
> > > >
> > > > Following this logic, a 1.x PCI device also has modern common config.
> > > > All you really truly need is a flag to make it behave like legacy,
> > > > from POV of device config and VQs.
> > > >
> > > Dual definition to same area in device is not elegant way to achieve it.
> > > We already discussed that some area is RW, some is not. It moves location.
> > >
> > > And it still doesn't solve the problem of reset.
> > > Hence, either doing via AQ or 2 registers method would work fine.
> > >
> > > It keeps legacy also isolated from mainstream.
> > >
> > > > So let's try to keep drivers self-contained as much as possible.
> > > >
> > > You said "AQ is fine" at the end of email.
> > > Hard to understand your mixed suggestion in context of sending v3.
> >
> > I am saying a simple thing actually:
> > - notifications are through VF
> > - rest of config is through PF (AQ)
> > seems like a consistent model to me.
> >
> Sounds good to me.
> What about discovery the notification place for the VF?
> We have two options.
> 1. discover them via AQ like config registers access
> 2. discover it via extended PCI capability
>
> #1 is far more programable and does not need to bake in the PCI device layout.
I think 2 is easier for drivers to use.
> So this way one day in future we can get rid of it more easily for those PCI devices who may prefer everything in the hw.
I think these devices are better off with a new transport using
drastically less registers.
> > > > if you want to re-use the normal notifications for VFs we already
> > > > describe them using capabilities.
> > > >
> > > That capability is intermixed with 1.x register queue_notify_off.
> >
> > Hmm yes annoying I forgot about this.
> >
>
> > > So, I see following options.
> > > a. either to create a new legacy specific extended capability, or b.
> > > say queue_notify_off doesn't apply for legacy notifications because
> > > this register doesn't exist for legacy c. exchange it via the AQ like above
> > struct.
> >
> > a feels ok to me... maybe test the waters with express capability as well, it will
> > come handy down the road I think.
> Yes, if this is really needed for some OS, this addition will be possible in future in the spec and also of the devices in CC list here.
hmm not sure what do you mean. I meant to put the capabilities
with the new legacy bar in express config space.
> >
> > > > However, this creates a weird mix where there's this command that
> > > > mostly follows legacy pci config layout except notifications which
> > > > are there in the legacy pci config are actually somewhere else. And
> > > > this is just inconsistent and confusing, and the reason for this is
> > implementation defined.
> > > Depends on how you see it.
> > > It is at same level of inconsistency as 1.x for a legit reason which is to split
> > config registers from data path register.
> > >
> > > It is not implementation defined. Notification over AQ cannot reach same perf
> > level as MMIO.
> > > I will avoid repeating it.
> >
> > yes, no need to repeat. I think what you are missing is that I am proposing this
> > as an intermediate step while we work on support for notifications, or even a
> > patch in the patchset.
> >
> Like said above, share your thoughts on the two options and I will modify this patch or add new in v3.
> >
> > I think what he meant is that tvq project will be reworked on top of AQ. tvq,
> > indeed, does expose a full transport on top of AQ and it seems logical to try and
> > add legacy guest support to that.
> Yes, I also acked that for SIOV devices to have legacy support on top of base SIOV device interface.
>
> > > > Talking about the two register thing. We actually have this already:
> > > > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> > > >
> > > To which IOBAR? The one that doesn't exist in the VF?
> >
> > exactly. bar value is currently 0-6 other values are reserved.
> > we can steal e.g. 7 and say this is legacy IOBAR.
> > The advantage is that we can use this IOBAR for anything at all, e.g.
> > VIRTIO_PCI_CAP_DEVICE_CFG too.
> >
> > > It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
> > >
> > > We can have new extended capability as,
> > >
> > > VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it
> > also avoid intermix with 1.x using above two registers.
> >
> > it's not a big deal, if you prefer a new capability fine.
> >
> > > >
> > > > But really since apparently you want to focus on how we solve
> > > > notifications let's focus on that. I feel the main issue is poking
> > > > at 1.x registers and at the same time at legacy registers.
> > > No, it is not 1.x registers. Register and bar offset is shared via the AQ.
> >
> > That's good then. It's not always clear what you want to do, this is the problem
> > with high level discussion as opposed to a specific patch.
> >
> > > > The spec was written with a strong assumption that it is either or, never a
> > mix.
> > > Transitional device supposed to support both as written in the spec.
> > > No different here.
> > > Now for real device.
> > >
> > > > Changing that will be painful.
> > > > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy
> > > > can live in software then and we do not need to worry about it in
> > > > the spec. I got it that there's too much software here for your
> > > > taste, but the problem with legacy is really that it was always ad hoc and
> > implementation derived.
> > > > So we need to keep it siloed.
> > >
> > > This is why I am trying to avoid putting in PCI registers of the VF,
> > > but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> > > with two registers.
> >
> > Oh. You are saying access to capability previously indicated modern and now it
> > is also done for legacy? That indeed is a good point.
> > Not sure what to do here. I'll ponder a bit.
>
> The new capability in mind is:
> VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
>
> struct virtio_pci_legacy_cfg_cap {
> struct virtio_pci_cap cap;
> le32 addr; /* register offset, width, op=RD/WR, status:0=pending,1=done)
> le32 reg_data;
> };
I would use cap64, cap is there because we did not think about 64 bit early enough.
> This will be hard to do for SIOV in future, so I prefer AQ cmd for config register access and notification discovery.
> WDYT?
I need to go back and look at how does SIOV tvq proposal manage
notifications then. Do not remember off the top of my head.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* Re: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-24 20:12 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 20:12 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 07:18:56PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 6:08 AM
> >
> > On Tue, May 23, 2023 at 10:22:27PM +0000, Parav Pandit wrote:
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Tuesday, May 23, 2023 2:49 PM
> > > > > > iokit is from my previous work experience not virtio.
> > > > > > For virtio we had a lot of trouble with hacks like this on windows.
> > > > > I don't know why windows driver need to talk to PF driver until
> > > > > now when the
> > > > AQ doesn't exists.
> > > >
> > > > What is it you want to know?
> > > >
> > > Why a VF driver needs to talk to a PF driver in windows without AQ?
> >
> > I don't know- it doesn't? If it did, it would be hard as we learned when trying to
> > support pci-bridge-seat. Doable, but annoying.
> If it does today, something likely is missing in that OS. Their model for sriov has not evolved as much as Linux from what I see.
> So not to worry about it, as they may have to implement more things.
>
> >
> > > > > > That's an important platform.
> > > > > >
> > > > > I am not sure how important is it.
> > > > > What I learnt that a decade has passed and even after that virtio
> > > > > drivers are
> > > > not part of the OS distro.
> > > >
> > > > Because with QEMU they are so easy to side-load. So we never bothered.
> > > >
> > > I don't understand your comment.
> > > I was saying to my knowledge a Windows OS do not have a virtio drivers in
> > their distro which you said is important.
> >
> > It's important for windows to work well with virtio. And it does because we
> > supply an install disk with virtio drivers through QEMU
>
> Ok. So WDM can use their IPR model or will be able to build infra as/if needed.
>
> > > I propose:
> > > Method-1:
> > > 1. VF legacy registers access over AQ of PF 2. VF driver notifications
> > > using MMIO of VF
> > >
> > > Method-2:
> > > 1. legacy registers access using two MMIO registers (data and addr) 2.
> > > VF driver notifications using MMIO of VF
> >
> > I think I prefer Method-1, I didn't exactly see Method-2 to judge though.
> >
> Ok. so lets step forward in v3 using 4 commands for registers access.
> 2 RW commands for common config.
> 2 RW commands for device config.
>
> Notification on below.
>
> >
> > > > > I am asking a 1.x PCI device has the VF notification BAR already,
> > > > > why we don't
> > > > use it, why to build an additional one in device?
> > > >
> > > > Following this logic, a 1.x PCI device also has modern common config.
> > > > All you really truly need is a flag to make it behave like legacy,
> > > > from POV of device config and VQs.
> > > >
> > > Dual definition to same area in device is not elegant way to achieve it.
> > > We already discussed that some area is RW, some is not. It moves location.
> > >
> > > And it still doesn't solve the problem of reset.
> > > Hence, either doing via AQ or 2 registers method would work fine.
> > >
> > > It keeps legacy also isolated from mainstream.
> > >
> > > > So let's try to keep drivers self-contained as much as possible.
> > > >
> > > You said "AQ is fine" at the end of email.
> > > Hard to understand your mixed suggestion in context of sending v3.
> >
> > I am saying a simple thing actually:
> > - notifications are through VF
> > - rest of config is through PF (AQ)
> > seems like a consistent model to me.
> >
> Sounds good to me.
> What about discovery the notification place for the VF?
> We have two options.
> 1. discover them via AQ like config registers access
> 2. discover it via extended PCI capability
>
> #1 is far more programable and does not need to bake in the PCI device layout.
I think 2 is easier for drivers to use.
> So this way one day in future we can get rid of it more easily for those PCI devices who may prefer everything in the hw.
I think these devices are better off with a new transport using
drastically less registers.
> > > > if you want to re-use the normal notifications for VFs we already
> > > > describe them using capabilities.
> > > >
> > > That capability is intermixed with 1.x register queue_notify_off.
> >
> > Hmm yes annoying I forgot about this.
> >
>
> > > So, I see following options.
> > > a. either to create a new legacy specific extended capability, or b.
> > > say queue_notify_off doesn't apply for legacy notifications because
> > > this register doesn't exist for legacy c. exchange it via the AQ like above
> > struct.
> >
> > a feels ok to me... maybe test the waters with express capability as well, it will
> > come handy down the road I think.
> Yes, if this is really needed for some OS, this addition will be possible in future in the spec and also of the devices in CC list here.
hmm not sure what do you mean. I meant to put the capabilities
with the new legacy bar in express config space.
> >
> > > > However, this creates a weird mix where there's this command that
> > > > mostly follows legacy pci config layout except notifications which
> > > > are there in the legacy pci config are actually somewhere else. And
> > > > this is just inconsistent and confusing, and the reason for this is
> > implementation defined.
> > > Depends on how you see it.
> > > It is at same level of inconsistency as 1.x for a legit reason which is to split
> > config registers from data path register.
> > >
> > > It is not implementation defined. Notification over AQ cannot reach same perf
> > level as MMIO.
> > > I will avoid repeating it.
> >
> > yes, no need to repeat. I think what you are missing is that I am proposing this
> > as an intermediate step while we work on support for notifications, or even a
> > patch in the patchset.
> >
> Like said above, share your thoughts on the two options and I will modify this patch or add new in v3.
> >
> > I think what he meant is that tvq project will be reworked on top of AQ. tvq,
> > indeed, does expose a full transport on top of AQ and it seems logical to try and
> > add legacy guest support to that.
> Yes, I also acked that for SIOV devices to have legacy support on top of base SIOV device interface.
>
> > > > Talking about the two register thing. We actually have this already:
> > > > VIRTIO_PCI_CAP_PCI_CFG. We could allow access to IO BAR through it.
> > > >
> > > To which IOBAR? The one that doesn't exist in the VF?
> >
> > exactly. bar value is currently 0-6 other values are reserved.
> > we can steal e.g. 7 and say this is legacy IOBAR.
> > The advantage is that we can use this IOBAR for anything at all, e.g.
> > VIRTIO_PCI_CAP_DEVICE_CFG too.
> >
> > > It is weird to create VIRTIO_PCI_CAP_PCI_CFG and point to non-existing area.
> > >
> > > We can have new extended capability as,
> > >
> > > VIRTIO_PCI_CAP_PCI_LEGACY_REG, that has slightly better interface and it
> > also avoid intermix with 1.x using above two registers.
> >
> > it's not a big deal, if you prefer a new capability fine.
> >
> > > >
> > > > But really since apparently you want to focus on how we solve
> > > > notifications let's focus on that. I feel the main issue is poking
> > > > at 1.x registers and at the same time at legacy registers.
> > > No, it is not 1.x registers. Register and bar offset is shared via the AQ.
> >
> > That's good then. It's not always clear what you want to do, this is the problem
> > with high level discussion as opposed to a specific patch.
> >
> > > > The spec was written with a strong assumption that it is either or, never a
> > mix.
> > > Transitional device supposed to support both as written in the spec.
> > > No different here.
> > > Now for real device.
> > >
> > > > Changing that will be painful.
> > > > This is why I was asking about VIRTIO_NET_F_LEGACY_HEADER - legacy
> > > > can live in software then and we do not need to worry about it in
> > > > the spec. I got it that there's too much software here for your
> > > > taste, but the problem with legacy is really that it was always ad hoc and
> > implementation derived.
> > > > So we need to keep it siloed.
> > >
> > > This is why I am trying to avoid putting in PCI registers of the VF,
> > > but if you prefer, we can do using new VIRTIO_PCI_CAP_PCI_LEGACY_REG
> > > with two registers.
> >
> > Oh. You are saying access to capability previously indicated modern and now it
> > is also done for legacy? That indeed is a good point.
> > Not sure what to do here. I'll ponder a bit.
>
> The new capability in mind is:
> VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
>
> struct virtio_pci_legacy_cfg_cap {
> struct virtio_pci_cap cap;
> le32 addr; /* register offset, width, op=RD/WR, status:0=pending,1=done)
> le32 reg_data;
> };
I would use cap64, cap is there because we did not think about 64 bit early enough.
> This will be hard to do for SIOV in future, so I prefer AQ cmd for config register access and notification discovery.
> WDYT?
I need to go back and look at how does SIOV tvq proposal manage
notifications then. Do not remember off the top of my head.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
2023-05-24 20:01 ` [virtio-comment] " Parav Pandit
@ 2023-05-24 20:15 ` Michael S. Tsirkin
-1 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 20:15 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 08:01:18PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 3:58 PM
>
> > > > Hope this helps.
> > > Definitely a good suggestion.
> > > I will send PR for virtio-docs to maintain our TODO list.
> >
> > Hmm. This is just a dumping ground then. Fine by me, but what I had in mind is
> > a common list of known problems to address by TC as a whole.
> > If you want that we want patches on list and discussion, maybe even voting.
> >
> It is not a dumping ground, it is the list of things to track and document without searching emails.
virtio-doc is a dumping ground though :)
> It is a patch on the list for the doc in virtio-docs area that if needed can be voted.
Let's start with nvidia-todo.md in virtio-doc indeed. People can at
least read it, good starting point. Next I would like us to proceed to
a todo list under virtio-spec, so that TC at least knows what are
current problems before us.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-comment] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ
@ 2023-05-24 20:15 ` Michael S. Tsirkin
0 siblings, 0 replies; 252+ messages in thread
From: Michael S. Tsirkin @ 2023-05-24 20:15 UTC (permalink / raw)
To: Parav Pandit
Cc: Jason Wang, virtio-dev, cohuck, david.edmondson, sburla,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
On Wed, May 24, 2023 at 08:01:18PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, May 24, 2023 3:58 PM
>
> > > > Hope this helps.
> > > Definitely a good suggestion.
> > > I will send PR for virtio-docs to maintain our TODO list.
> >
> > Hmm. This is just a dumping ground then. Fine by me, but what I had in mind is
> > a common list of known problems to address by TC as a whole.
> > If you want that we want patches on list and discussion, maybe even voting.
> >
> It is not a dumping ground, it is the list of things to track and document without searching emails.
virtio-doc is a dumping ground though :)
> It is a patch on the list for the doc in virtio-docs area that if needed can be voted.
Let's start with nvidia-todo.md in virtio-doc indeed. People can at
least read it, good starting point. Next I would like us to proceed to
a todo list under virtio-spec, so that TC at least knows what are
current problems before us.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
* [virtio-dev] RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
2023-05-24 20:12 ` Michael S. Tsirkin
@ 2023-05-24 21:02 ` Parav Pandit
-1 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 21:02 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 4:13 PM
[..]
> > Sounds good to me.
> > What about discovery the notification place for the VF?
> > We have two options.
> > 1. discover them via AQ like config registers access 2. discover it
> > via extended PCI capability
> >
> > #1 is far more programable and does not need to bake in the PCI device
> layout.
>
> I think 2 is easier for drivers to use.
Yes.
Its trade off.
But not a huge difference from sw point of view as they use underlying apis.
As opposed to do that baking pci capabilities is slightly hard as it needs to intermix control path over registers..
Typically, CVQ and AQ etc a PCI device would do using some sort of firmware, so adding commands are easier than low level PCI capabilities based tunneling.
>
> > So this way one day in future we can get rid of it more easily for those PCI
> devices who may prefer everything in the hw.
>
> I think these devices are better off with a new transport using drastically less
> registers.
>
That way AQ scores more.
> > > a feels ok to me... maybe test the waters with express capability as
> > > well, it will come handy down the road I think.
> > Yes, if this is really needed for some OS, this addition will be possible in future
> in the spec and also of the devices in CC list here.
>
> hmm not sure what do you mean. I meant to put the capabilities with the new
> legacy bar in express config space.
>
I mean to say we should do AQ.
And if an OS can never implement two driver communication for legacy, may be at that future point a spec should evolve to provide this capability-based access.
> > The new capability in mind is:
> > VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
> >
> > struct virtio_pci_legacy_cfg_cap {
> > struct virtio_pci_cap cap;
> > le32 addr; /* register offset, width, op=RD/WR,
> status:0=pending,1=done)
> > le32 reg_data;
> > };
>
> I would use cap64, cap is there because we did not think about 64 bit early
> enough.
>
Yes.
> > This will be hard to do for SIOV in future, so I prefer AQ cmd for config
> register access and notification discovery.
> > WDYT?
>
> I need to go back and look at how does SIOV tvq proposal manage notifications
> then. Do not remember off the top of my head.
SIOV tvq proposal does not tunnel the driver notifications.
It exposes the address and size of db region like how I had in similar command.
This is why it is not a transport vq and I suggest to rename to aq :)
Snippet of it below.
+The virtqueue notification area information can be get through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY 11
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET 0
+
+struct virtio_transportq_ctrl_vq_notification_area {
+ u64 address;
+ u64 size;
+};
+\end{lstlisting}
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 252+ messages in thread
* RE: [virtio-comment] RE: [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands
@ 2023-05-24 21:02 ` Parav Pandit
0 siblings, 0 replies; 252+ messages in thread
From: Parav Pandit @ 2023-05-24 21:02 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-dev, cohuck, david.edmondson, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, virtio-comment, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, May 24, 2023 4:13 PM
[..]
> > Sounds good to me.
> > What about discovery the notification place for the VF?
> > We have two options.
> > 1. discover them via AQ like config registers access 2. discover it
> > via extended PCI capability
> >
> > #1 is far more programable and does not need to bake in the PCI device
> layout.
>
> I think 2 is easier for drivers to use.
Yes.
Its trade off.
But not a huge difference from sw point of view as they use underlying apis.
As opposed to do that baking pci capabilities is slightly hard as it needs to intermix control path over registers..
Typically, CVQ and AQ etc a PCI device would do using some sort of firmware, so adding commands are easier than low level PCI capabilities based tunneling.
>
> > So this way one day in future we can get rid of it more easily for those PCI
> devices who may prefer everything in the hw.
>
> I think these devices are better off with a new transport using drastically less
> registers.
>
That way AQ scores more.
> > > a feels ok to me... maybe test the waters with express capability as
> > > well, it will come handy down the road I think.
> > Yes, if this is really needed for some OS, this addition will be possible in future
> in the spec and also of the devices in CC list here.
>
> hmm not sure what do you mean. I meant to put the capabilities with the new
> legacy bar in express config space.
>
I mean to say we should do AQ.
And if an OS can never implement two driver communication for legacy, may be at that future point a spec should evolve to provide this capability-based access.
> > The new capability in mind is:
> > VIRTIO_PCI_EXT_CAP_PCI_LEGACY_REG
> >
> > struct virtio_pci_legacy_cfg_cap {
> > struct virtio_pci_cap cap;
> > le32 addr; /* register offset, width, op=RD/WR,
> status:0=pending,1=done)
> > le32 reg_data;
> > };
>
> I would use cap64, cap is there because we did not think about 64 bit early
> enough.
>
Yes.
> > This will be hard to do for SIOV in future, so I prefer AQ cmd for config
> register access and notification discovery.
> > WDYT?
>
> I need to go back and look at how does SIOV tvq proposal manage notifications
> then. Do not remember off the top of my head.
SIOV tvq proposal does not tunnel the driver notifications.
It exposes the address and size of db region like how I had in similar command.
This is why it is not a transport vq and I suggest to rename to aq :)
Snippet of it below.
+The virtqueue notification area information can be get through the following commands:
+
+\begin{lstlisting}
+#define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY 11
+ #define VIRTIO_TRANSPTQ_CTRL_VQ_NOTIFY_GET 0
+
+struct virtio_transportq_ctrl_vq_notification_area {
+ u64 address;
+ u64 size;
+};
+\end{lstlisting}
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 252+ messages in thread
end of thread, other threads:[~2023-05-24 21:02 UTC | newest]
Thread overview: 252+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-06 0:01 [virtio-dev] [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ Parav Pandit
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
2023-05-06 0:01 ` [virtio-dev] [PATCH v2 1/2] transport-pci: Introduce legacy registers access commands Parav Pandit
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
2023-05-17 5:44 ` [virtio-dev] " Michael S. Tsirkin
2023-05-17 5:44 ` [virtio-comment] " Michael S. Tsirkin
2023-05-17 19:32 ` [virtio-dev] " Parav Pandit
2023-05-17 19:32 ` [virtio-comment] " Parav Pandit
2023-05-18 19:42 ` [virtio-dev] " Michael S. Tsirkin
2023-05-18 19:42 ` Michael S. Tsirkin
2023-05-18 20:51 ` Parav Pandit
2023-05-18 20:51 ` [virtio-dev] " Parav Pandit
2023-05-19 1:54 ` [virtio-dev] " Jason Wang
2023-05-19 1:54 ` Jason Wang
2023-05-19 2:04 ` [virtio-dev] " Parav Pandit
2023-05-19 2:04 ` Parav Pandit
2023-05-19 6:06 ` [virtio-dev] " Michael S. Tsirkin
2023-05-19 6:06 ` Michael S. Tsirkin
2023-05-19 16:37 ` [virtio-dev] " Parav Pandit
2023-05-19 16:37 ` Parav Pandit
2023-05-21 9:16 ` [virtio-dev] " Michael S. Tsirkin
2023-05-21 9:16 ` Michael S. Tsirkin
2023-05-21 13:21 ` [virtio-dev] " Parav Pandit
2023-05-21 13:21 ` Parav Pandit
2023-05-21 14:33 ` [virtio-dev] " Michael S. Tsirkin
2023-05-21 14:33 ` Michael S. Tsirkin
2023-05-21 14:44 ` Parav Pandit
2023-05-21 14:44 ` [virtio-dev] " Parav Pandit
2023-05-22 20:07 ` [virtio-dev] " Michael S. Tsirkin
2023-05-22 20:07 ` Michael S. Tsirkin
2023-05-22 21:05 ` [virtio-dev] " Parav Pandit
2023-05-22 21:05 ` Parav Pandit
2023-05-22 21:34 ` [virtio-dev] " Michael S. Tsirkin
2023-05-22 21:34 ` Michael S. Tsirkin
2023-05-23 17:13 ` [virtio-dev] " Parav Pandit
2023-05-23 17:13 ` Parav Pandit
2023-05-23 18:48 ` [virtio-dev] " Michael S. Tsirkin
2023-05-23 18:48 ` Michael S. Tsirkin
2023-05-23 22:22 ` [virtio-dev] " Parav Pandit
2023-05-23 22:22 ` Parav Pandit
2023-05-24 1:17 ` [virtio-dev] " Jason Wang
2023-05-24 1:17 ` Jason Wang
2023-05-24 10:07 ` [virtio-dev] " Michael S. Tsirkin
2023-05-24 10:07 ` Michael S. Tsirkin
2023-05-24 19:18 ` [virtio-dev] " Parav Pandit
2023-05-24 19:18 ` Parav Pandit
2023-05-24 20:12 ` [virtio-dev] " Michael S. Tsirkin
2023-05-24 20:12 ` Michael S. Tsirkin
2023-05-24 21:02 ` [virtio-dev] " Parav Pandit
2023-05-24 21:02 ` Parav Pandit
2023-05-22 21:42 ` [virtio-dev] " Michael S. Tsirkin
2023-05-22 21:42 ` Michael S. Tsirkin
2023-05-22 0:54 ` [virtio-dev] " Jason Wang
2023-05-22 0:54 ` Jason Wang
2023-05-22 2:46 ` [virtio-dev] " Parav Pandit
2023-05-22 2:46 ` Parav Pandit
2023-05-22 19:35 ` [virtio-dev] " Michael S. Tsirkin
2023-05-22 19:35 ` Michael S. Tsirkin
2023-05-06 0:01 ` [virtio-dev] [PATCH v2 2/2] transport-pci: Add legacy register access conformance section Parav Pandit
2023-05-06 0:01 ` [virtio-comment] " Parav Pandit
2023-05-06 2:31 ` [virtio-dev] Re: [PATCH v2 0/2] transport-pci: Introduce legacy registers access using AQ Jason Wang
2023-05-06 2:31 ` [virtio-comment] " Jason Wang
2023-05-07 13:44 ` [virtio-dev] " Michael S. Tsirkin
2023-05-07 13:44 ` [virtio-comment] " Michael S. Tsirkin
2023-05-08 2:23 ` [virtio-dev] " Jason Wang
2023-05-08 2:23 ` [virtio-comment] " Jason Wang
2023-05-08 17:07 ` [virtio-dev] " Parav Pandit
2023-05-08 17:07 ` [virtio-comment] " Parav Pandit
2023-05-09 3:44 ` Jason Wang
2023-05-09 3:44 ` [virtio-dev] " Jason Wang
2023-05-09 3:56 ` [virtio-dev] " Parav Pandit
2023-05-09 3:56 ` [virtio-comment] " Parav Pandit
2023-05-10 3:51 ` [virtio-dev] " Jason Wang
2023-05-10 3:51 ` [virtio-comment] " Jason Wang
2023-05-10 4:22 ` [virtio-dev] " Jason Wang
2023-05-10 4:22 ` [virtio-comment] " Jason Wang
2023-05-10 16:07 ` [virtio-dev] " Parav Pandit
2023-05-10 16:07 ` [virtio-comment] " Parav Pandit
2023-05-11 7:20 ` [virtio-dev] " Jason Wang
2023-05-11 7:20 ` Jason Wang
2023-05-11 11:35 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 11:35 ` Michael S. Tsirkin
2023-05-15 5:08 ` [virtio-dev] " Jason Wang
2023-05-15 5:08 ` Jason Wang
2023-05-15 15:25 ` [virtio-dev] " Parav Pandit
2023-05-15 15:25 ` Parav Pandit
2023-05-10 16:04 ` [virtio-dev] " Parav Pandit
2023-05-10 16:04 ` [virtio-comment] " Parav Pandit
2023-05-11 7:17 ` [virtio-dev] " Jason Wang
2023-05-11 7:17 ` [virtio-comment] " Jason Wang
2023-05-11 14:31 ` [virtio-dev] " Parav Pandit
2023-05-11 14:31 ` [virtio-comment] " Parav Pandit
2023-05-15 5:12 ` [virtio-dev] " Jason Wang
2023-05-15 5:12 ` Jason Wang
2023-05-15 15:26 ` [virtio-dev] " Parav Pandit
2023-05-15 15:26 ` Parav Pandit
2023-05-10 6:04 ` [virtio-dev] " Michael S. Tsirkin
2023-05-10 6:04 ` [virtio-comment] " Michael S. Tsirkin
2023-05-10 7:01 ` [virtio-dev] " Jason Wang
2023-05-10 7:01 ` [virtio-comment] " Jason Wang
2023-05-10 7:43 ` [virtio-dev] " Michael S. Tsirkin
2023-05-10 7:43 ` [virtio-comment] " Michael S. Tsirkin
2023-05-10 16:13 ` [virtio-dev] " Parav Pandit
2023-05-10 16:13 ` [virtio-comment] " Parav Pandit
2023-05-11 7:04 ` [virtio-dev] " Jason Wang
2023-05-11 7:04 ` [virtio-comment] " Jason Wang
2023-05-11 12:54 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 12:54 ` [virtio-comment] " Michael S. Tsirkin
2023-05-11 13:02 ` [virtio-dev] " Parav Pandit
2023-05-11 13:02 ` [virtio-comment] " Parav Pandit
2023-05-15 7:30 ` [virtio-dev] " Jason Wang
2023-05-15 7:30 ` [virtio-comment] " Jason Wang
2023-05-15 10:08 ` [virtio-dev] " Michael S. Tsirkin
2023-05-15 10:08 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 14:30 ` [virtio-dev] " Parav Pandit
2023-05-15 14:30 ` [virtio-comment] " Parav Pandit
2023-05-23 18:16 ` [virtio-dev] " Michael S. Tsirkin
2023-05-23 18:16 ` [virtio-comment] " Michael S. Tsirkin
2023-05-23 21:32 ` [virtio-dev] " Parav Pandit
2023-05-23 21:32 ` [virtio-comment] " Parav Pandit
2023-05-24 5:56 ` [virtio-dev] " Michael S. Tsirkin
2023-05-24 5:56 ` [virtio-comment] " Michael S. Tsirkin
2023-05-24 18:57 ` [virtio-dev] " Parav Pandit
2023-05-24 18:57 ` [virtio-comment] " Parav Pandit
2023-05-24 19:58 ` [virtio-dev] " Michael S. Tsirkin
2023-05-24 19:58 ` [virtio-comment] " Michael S. Tsirkin
2023-05-24 20:01 ` [virtio-dev] " Parav Pandit
2023-05-24 20:01 ` [virtio-comment] " Parav Pandit
2023-05-24 20:15 ` [virtio-dev] " Michael S. Tsirkin
2023-05-24 20:15 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 15:59 ` [virtio-dev] " Parav Pandit
2023-05-15 15:59 ` [virtio-comment] " Parav Pandit
2023-05-16 6:21 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 6:21 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 19:11 ` [virtio-dev] " Parav Pandit
2023-05-16 19:11 ` [virtio-comment] " Parav Pandit
2023-05-16 20:58 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 20:58 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 21:19 ` [virtio-dev] " Parav Pandit
2023-05-16 21:19 ` [virtio-comment] " Parav Pandit
2023-05-16 21:23 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 21:23 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 21:30 ` [virtio-dev] " Parav Pandit
2023-05-16 21:30 ` [virtio-comment] " Parav Pandit
2023-05-15 7:13 ` [virtio-dev] " Jason Wang
2023-05-15 7:13 ` [virtio-comment] " Jason Wang
2023-05-11 13:15 ` [virtio-dev] " Parav Pandit
2023-05-11 13:15 ` [virtio-comment] " Parav Pandit
2023-05-11 13:45 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 13:45 ` [virtio-comment] " Michael S. Tsirkin
2023-05-12 14:03 ` [virtio-dev] " Parav Pandit
2023-05-12 14:03 ` [virtio-comment] " Parav Pandit
2023-05-16 3:54 ` Jason Wang
2023-05-16 3:54 ` [virtio-comment] " Jason Wang
2023-05-16 19:35 ` Parav Pandit
2023-05-16 19:35 ` [virtio-dev] " Parav Pandit
2023-05-16 21:11 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 21:11 ` Michael S. Tsirkin
2023-05-16 21:49 ` [virtio-dev] " Parav Pandit
2023-05-16 21:49 ` Parav Pandit
2023-05-16 21:56 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 21:56 ` Michael S. Tsirkin
2023-05-10 16:11 ` Parav Pandit
2023-05-10 16:11 ` [virtio-dev] " Parav Pandit
2023-05-10 16:16 ` Michael S. Tsirkin
2023-05-10 16:16 ` [virtio-comment] " Michael S. Tsirkin
2023-05-10 17:33 ` [virtio-dev] " Parav Pandit
2023-05-10 17:33 ` [virtio-comment] " Parav Pandit
2023-05-10 21:08 ` [virtio-dev] " Parav Pandit
2023-05-10 21:08 ` [virtio-comment] " Parav Pandit
2023-05-10 21:33 ` [virtio-dev] " Michael S. Tsirkin
2023-05-10 21:33 ` [virtio-comment] " Michael S. Tsirkin
2023-05-10 21:48 ` [virtio-dev] " Parav Pandit
2023-05-10 21:48 ` [virtio-comment] " Parav Pandit
2023-05-11 7:06 ` [virtio-dev] " Jason Wang
2023-05-11 7:06 ` [virtio-comment] " Jason Wang
2023-05-11 13:04 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 13:04 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 5:19 ` [virtio-dev] " Jason Wang
2023-05-15 5:19 ` [virtio-comment] " Jason Wang
2023-05-15 15:31 ` [virtio-dev] " Parav Pandit
2023-05-15 15:31 ` [virtio-comment] " Parav Pandit
2023-05-11 13:28 ` [virtio-dev] " Parav Pandit
2023-05-11 13:28 ` [virtio-comment] " Parav Pandit
2023-05-11 13:38 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 13:38 ` [virtio-comment] " Michael S. Tsirkin
2023-05-11 16:00 ` [virtio-dev] " Parav Pandit
2023-05-11 16:00 ` [virtio-comment] " Parav Pandit
2023-05-11 20:47 ` [virtio-dev] " Parav Pandit
2023-05-11 20:47 ` [virtio-comment] " Parav Pandit
2023-05-11 20:58 ` [virtio-dev] " Michael S. Tsirkin
2023-05-11 20:58 ` [virtio-comment] " Michael S. Tsirkin
2023-05-11 21:03 ` [virtio-dev] " Parav Pandit
2023-05-11 21:03 ` [virtio-comment] " Parav Pandit
2023-05-15 16:55 ` [virtio-dev] " Parav Pandit
2023-05-15 16:55 ` [virtio-comment] " Parav Pandit
2023-05-15 7:10 ` [virtio-dev] " Jason Wang
2023-05-15 7:10 ` [virtio-comment] " Jason Wang
2023-05-15 15:49 ` [virtio-dev] " Parav Pandit
2023-05-15 15:49 ` [virtio-comment] " Parav Pandit
2023-05-15 17:44 ` [virtio-dev] " Michael S. Tsirkin
2023-05-15 17:44 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 17:51 ` [virtio-dev] " Parav Pandit
2023-05-15 17:51 ` [virtio-comment] " Parav Pandit
2023-05-15 17:56 ` [virtio-dev] " Michael S. Tsirkin
2023-05-15 17:56 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 18:00 ` [virtio-dev] " Parav Pandit
2023-05-15 18:00 ` [virtio-comment] " Parav Pandit
2023-05-15 18:01 ` [virtio-dev] " Michael S. Tsirkin
2023-05-15 18:01 ` [virtio-comment] " Michael S. Tsirkin
2023-05-15 18:05 ` [virtio-dev] " Parav Pandit
2023-05-15 18:05 ` [virtio-comment] " Parav Pandit
2023-05-16 3:37 ` [virtio-dev] " Jason Wang
2023-05-16 3:37 ` [virtio-comment] " Jason Wang
2023-05-16 3:43 ` [virtio-dev] " Jason Wang
2023-05-16 3:43 ` [virtio-comment] " Jason Wang
2023-05-16 5:38 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 5:38 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 3:28 ` [virtio-dev] " Jason Wang
2023-05-16 3:28 ` [virtio-comment] " Jason Wang
2023-05-16 3:45 ` [virtio-dev] " Parav Pandit
2023-05-16 3:45 ` [virtio-comment] " Parav Pandit
2023-05-16 4:08 ` [virtio-dev] " Jason Wang
2023-05-16 4:08 ` [virtio-comment] " Jason Wang
2023-05-16 19:29 ` [virtio-dev] " Parav Pandit
2023-05-16 19:29 ` [virtio-comment] " Parav Pandit
2023-05-16 21:09 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 21:09 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 21:41 ` [virtio-dev] " Parav Pandit
2023-05-16 21:41 ` [virtio-comment] " Parav Pandit
2023-05-16 21:54 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 21:54 ` [virtio-comment] " Michael S. Tsirkin
2023-05-16 4:18 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 4:18 ` [virtio-comment] " Michael S. Tsirkin
2023-05-07 9:04 ` [virtio-dev] " Michael S. Tsirkin
2023-05-07 9:04 ` [virtio-comment] " Michael S. Tsirkin
2023-05-08 16:54 ` [virtio-dev] " Parav Pandit
2023-05-08 16:54 ` [virtio-comment] " Parav Pandit
2023-05-15 20:29 ` [virtio-dev] " Michael S. Tsirkin
2023-05-15 20:29 ` Michael S. Tsirkin
2023-05-15 20:56 ` [virtio-dev] " Parav Pandit
2023-05-15 20:56 ` Parav Pandit
2023-05-16 4:32 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 4:32 ` Michael S. Tsirkin
2023-05-16 18:45 ` [virtio-dev] " Parav Pandit
2023-05-16 18:45 ` Parav Pandit
2023-05-16 20:42 ` [virtio-dev] " Michael S. Tsirkin
2023-05-16 20:42 ` Michael S. Tsirkin
2023-05-23 6:38 ` [virtio-dev] " Michael S. Tsirkin
2023-05-23 6:38 ` [virtio-comment] " Michael S. Tsirkin
2023-05-23 17:28 ` [virtio-dev] " Parav Pandit
2023-05-23 17:28 ` [virtio-comment] " Parav Pandit
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.