* [virtio-dev] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
This short series introduces legacy registers access commands for the owner
group member access the legacy registers of the member VFs.
This short series introduces legacy region access commands by the group owner
device for its member devices.
Currently it is applicable to the PCI PF and VF devices. If in future any
SIOV devices to support legacy registers, they can be easily supported using
same commands by using the group member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Patch summary:
--------------
patch-1 split rows of admin opcode tables by a line
patch-2 fix section numbering
patch-3 add generic legacy region access commands
patch-4 add pci specific definition
It uses the newly introduced administration command facility with 4 new
commands and a new optional command to query the legacy notification region.
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements is solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
Legacy region Driver notification
access |
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Alternatives considered:
========================
1. Exposing BAR0 as MMIO BAR that follows legacy registers template
Pros:
a. Kind of works with legacy drivers as some of them have used API
which is agnostic to MMIO vs IOBAR.
b. Does not require hypervisor intervantion
Cons:
a. Device reset is extremely hard to implement in device at scale as
driver does not wait for device reset completion
b. Device register width related problems persist that hypervisor if
wishes, it cannot be fixed.
2. Accessing VF registers by tunneling it through new legacy PCI capability
Pros:
a. Self contained, but cannot work with future PCI SIOV devices
Cons:
a. Equally slow as AQ access
b. Still requires new capability for notification access
c. Requires hardware to build low level registers access which is not worth
for long term future
3. Accessing VF notification region using new PF BAR
Cons:
a. Requires hardware to build new PCI steering logic per PF to forward
notification from the PF to VF, requires double the amount of logic
compared to today
b. Requires very large additional PF BAR whose size must be max_Vfs * BAR size.
4. Trapping CVQ, configuration region, LEGACY_HDR
Cons:
a. This does not fullfil the very basic requirement to not trap the
1.x objects (configuration registers, vqs)
b. Requires feature negotiations mediation in hypervisor software
c. Requires constant device type specific knowledge in hypervisor driver
(Does not scale for 30+ device types)
4. F_LEACY_HDR, F_WRITE_MAC
Cons:
a. Requires device support to have read/write mac address which is
hard to implement on every member device.
b. such functionality is duplicate of existing cvq per device.
c. config space is only for the initialization specific purpose.
d. Requires mediation of 1.x objects, which is not good design.
e. Solves only for the net device.
Pros:
a. May work for nested env
conclusion for picking AQ approach:
==================================
1. Overall AQ based access is simpler to implement with combination of
best from software and device so that legacy registers do not get baked
in the device hardware
2. AQ allows hypervisor software to intercept legacy registers and make
corrections if needed
3. Provides trade-off between performance, device complexity vs spec,
while still maintaining passthrough mode for the VFs with minimal
hypervisor intercepts only for legacy registers access
4. AQ mechanism is designed for accessing other member devices registers
as noted in AQ submission, it utilizes the existing infrastructure over
other alternatives.
5. Uses existing driver notification region similar to legacy notification
saves hardware resources
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v8->v9:
- add missing articles in notify query command
- replaced 'this notification' with 'such a notification'
- addressed below comments from Michael
- dropped 'Region' from the commands
- added 7 reserved pad bytes in config write commands
- rewrote from 'use following structure' to 'field' has the following
struct..
- dropped mentioning to follow struct virtio_admin_cmd.
- added note about command limited to only sriov group type for now
- rewrote the description little differently
v7->v8:
- remove empty line at the end of file
- removed white space at the end
- addressed comments from Michael add link to pci
- renamed region to region_data
- made region_data width to be 16 bytes to cover for 8 bytes offset
- moved generic notification region related normative from pci to
generic section
- addressed comments from Michael
- made bar offset 64-bit
- prefix legacy specific structure with _legacy
- moved generic normative from pci to generic section
- added link to virtio pci capabilities when referring to bar 0
- remove 'should' from generic description
v6->v7:
- addressed several comments from Michael
- use AQ command to query legacy notify region, dropped pci capability
modifications
- moved most part of the text to the generic admin command section
- replace administrative to administration
- replace admin vq citation to admin commands
- added normatives for device and driver side
- made BAR0 to be not used at all when supporting legacy interface
- added normative around BAR0 and SR-IOV extended capability
- grammar corrections
v5->v6:
- fixed previous missed abbreviation of LCC and LD
- added text for the PCI capability for the group member device
v4->v5:
- split pci transport and generic command section to new patch
- removed multiple references to the VF
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- split from pci transport specific patch
- split conformance to transport and generic sections
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- rename fields from register to region
- avoided abbreviation for legacy, device and config
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion,
though it is fine. Instead replaced with extending virtio_pci_notify_cap
to indicate that legacy queue notifications can be done on the
notification location
- fixed spelling errors
- replaced administrative virtqueue to administration virtqueue
- moved legacy interface normative references to legacy conformance
section
v2->v3:
- added new patch to split raws of admin vq opcode table
- adddressed Jason and Michael's comment to split single register
access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as
admin command cap bit already covers it.
- added other alternative design considered and discussed in detail in v0, v1 and v2
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
v0->v1:
- addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
- far more simpler design than MMR access
- removed complexities of MMR device ids
- removed complexities of MMR registers and extended capabilities
- dropped adding new extended capabilities because if if they are
added, a pci device still needs to have existing capabilities
in the legacy configuration space and hypervisor driver do not
need to access them
Parav Pandit (4):
admin: Split opcode table rows with a line
admin: Fix section numbering
admin: Add group member legacy register access commands
transport-pci: Introduce group legacy group member config region
access
admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
admin.tex | 20 +++-
conformance.tex | 3 +
transport-pci-legacy-regs.tex | 41 +++++++
transport-pci.tex | 2 +
5 files changed, 268 insertions(+), 3 deletions(-)
create mode 100644 admin-cmds-legacy-interface.tex
create mode 100644 transport-pci-legacy-regs.tex
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
This short series introduces legacy registers access commands for the owner
group member access the legacy registers of the member VFs.
This short series introduces legacy region access commands by the group owner
device for its member devices.
Currently it is applicable to the PCI PF and VF devices. If in future any
SIOV devices to support legacy registers, they can be easily supported using
same commands by using the group member identifiers of the future SIOV devices.
More details as overview, motivation, use case are further described
below.
Patch summary:
--------------
patch-1 split rows of admin opcode tables by a line
patch-2 fix section numbering
patch-3 add generic legacy region access commands
patch-4 add pci specific definition
It uses the newly introduced administration command facility with 4 new
commands and a new optional command to query the legacy notification region.
Usecase:
--------
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
----------------------
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
---------
Above usecase requirements is solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using an admin virtqueue of
the group owner PCI PF.
Two new admin virtqueue commands are added which read/write PCI VF
registers.
Software usage example:
-----------------------
One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper | | functionalities | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
Legacy region Driver notification
access |
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Virtio pci driver to bind to the listed device id and
use it in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Please review.
Alternatives considered:
========================
1. Exposing BAR0 as MMIO BAR that follows legacy registers template
Pros:
a. Kind of works with legacy drivers as some of them have used API
which is agnostic to MMIO vs IOBAR.
b. Does not require hypervisor intervantion
Cons:
a. Device reset is extremely hard to implement in device at scale as
driver does not wait for device reset completion
b. Device register width related problems persist that hypervisor if
wishes, it cannot be fixed.
2. Accessing VF registers by tunneling it through new legacy PCI capability
Pros:
a. Self contained, but cannot work with future PCI SIOV devices
Cons:
a. Equally slow as AQ access
b. Still requires new capability for notification access
c. Requires hardware to build low level registers access which is not worth
for long term future
3. Accessing VF notification region using new PF BAR
Cons:
a. Requires hardware to build new PCI steering logic per PF to forward
notification from the PF to VF, requires double the amount of logic
compared to today
b. Requires very large additional PF BAR whose size must be max_Vfs * BAR size.
4. Trapping CVQ, configuration region, LEGACY_HDR
Cons:
a. This does not fullfil the very basic requirement to not trap the
1.x objects (configuration registers, vqs)
b. Requires feature negotiations mediation in hypervisor software
c. Requires constant device type specific knowledge in hypervisor driver
(Does not scale for 30+ device types)
4. F_LEACY_HDR, F_WRITE_MAC
Cons:
a. Requires device support to have read/write mac address which is
hard to implement on every member device.
b. such functionality is duplicate of existing cvq per device.
c. config space is only for the initialization specific purpose.
d. Requires mediation of 1.x objects, which is not good design.
e. Solves only for the net device.
Pros:
a. May work for nested env
conclusion for picking AQ approach:
==================================
1. Overall AQ based access is simpler to implement with combination of
best from software and device so that legacy registers do not get baked
in the device hardware
2. AQ allows hypervisor software to intercept legacy registers and make
corrections if needed
3. Provides trade-off between performance, device complexity vs spec,
while still maintaining passthrough mode for the VFs with minimal
hypervisor intercepts only for legacy registers access
4. AQ mechanism is designed for accessing other member devices registers
as noted in AQ submission, it utilizes the existing infrastructure over
other alternatives.
5. Uses existing driver notification region similar to legacy notification
saves hardware resources
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v8->v9:
- add missing articles in notify query command
- replaced 'this notification' with 'such a notification'
- addressed below comments from Michael
- dropped 'Region' from the commands
- added 7 reserved pad bytes in config write commands
- rewrote from 'use following structure' to 'field' has the following
struct..
- dropped mentioning to follow struct virtio_admin_cmd.
- added note about command limited to only sriov group type for now
- rewrote the description little differently
v7->v8:
- remove empty line at the end of file
- removed white space at the end
- addressed comments from Michael add link to pci
- renamed region to region_data
- made region_data width to be 16 bytes to cover for 8 bytes offset
- moved generic notification region related normative from pci to
generic section
- addressed comments from Michael
- made bar offset 64-bit
- prefix legacy specific structure with _legacy
- moved generic normative from pci to generic section
- added link to virtio pci capabilities when referring to bar 0
- remove 'should' from generic description
v6->v7:
- addressed several comments from Michael
- use AQ command to query legacy notify region, dropped pci capability
modifications
- moved most part of the text to the generic admin command section
- replace administrative to administration
- replace admin vq citation to admin commands
- added normatives for device and driver side
- made BAR0 to be not used at all when supporting legacy interface
- added normative around BAR0 and SR-IOV extended capability
- grammar corrections
v5->v6:
- fixed previous missed abbreviation of LCC and LD
- added text for the PCI capability for the group member device
v4->v5:
- split pci transport and generic command section to new patch
- removed multiple references to the VF
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- split from pci transport specific patch
- split conformance to transport and generic sections
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- rename fields from register to region
- avoided abbreviation for legacy, device and config
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion,
though it is fine. Instead replaced with extending virtio_pci_notify_cap
to indicate that legacy queue notifications can be done on the
notification location
- fixed spelling errors
- replaced administrative virtqueue to administration virtqueue
- moved legacy interface normative references to legacy conformance
section
v2->v3:
- added new patch to split raws of admin vq opcode table
- adddressed Jason and Michael's comment to split single register
access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as
admin command cap bit already covers it.
- added other alternative design considered and discussed in detail in v0, v1 and v2
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
v0->v1:
- addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
- far more simpler design than MMR access
- removed complexities of MMR device ids
- removed complexities of MMR registers and extended capabilities
- dropped adding new extended capabilities because if if they are
added, a pci device still needs to have existing capabilities
in the legacy configuration space and hypervisor driver do not
need to access them
Parav Pandit (4):
admin: Split opcode table rows with a line
admin: Fix section numbering
admin: Add group member legacy register access commands
transport-pci: Introduce group legacy group member config region
access
admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
admin.tex | 20 +++-
conformance.tex | 3 +
transport-pci-legacy-regs.tex | 41 +++++++
transport-pci.tex | 2 +
5 files changed, 268 insertions(+), 3 deletions(-)
create mode 100644 admin-cmds-legacy-interface.tex
create mode 100644 transport-pci-legacy-regs.tex
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] [PATCH v9 1/4] admin: Split opcode table rows with a line
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-04 23:37 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Currently all opcode appears to be in a single row.
Separate them with a line similar to other tables.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v2->v3:
- new patch
---
admin.tex | 2 ++
1 file changed, 2 insertions(+)
diff --git a/admin.tex b/admin.tex
index 2efd4d7..e51f9e6 100644
--- a/admin.tex
+++ b/admin.tex
@@ -114,7 +114,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
opcode & Name & Command Description \\
\hline \hline
0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\
+\hline
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
+\hline
0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-comment] [PATCH v9 1/4] admin: Split opcode table rows with a line
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Currently all opcode appears to be in a single row.
Separate them with a line similar to other tables.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v2->v3:
- new patch
---
admin.tex | 2 ++
1 file changed, 2 insertions(+)
diff --git a/admin.tex b/admin.tex
index 2efd4d7..e51f9e6 100644
--- a/admin.tex
+++ b/admin.tex
@@ -114,7 +114,9 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
opcode & Name & Command Description \\
\hline \hline
0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type \\
+\hline
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
+\hline
0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-dev] [PATCH v9 2/4] admin: Fix section numbering
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-04 23:37 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Requirements are put one additional level down. Fix it.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v4->v5:
- new patch
---
admin.tex | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/admin.tex b/admin.tex
index e51f9e6..fd3b97d 100644
--- a/admin.tex
+++ b/admin.tex
@@ -286,7 +286,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
supporting multiple group types, the list of supported commands
might differ between different group types.
-\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+\devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The device MUST validate \field{opcode}, \field{group_type} and
\field{group_member_id}, and if any of these has an invalid or
@@ -378,7 +378,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\field{VF Enable} refer to registers within the SR-IOV Extended
Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
-\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+\drivernormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The driver MAY discover whether device supports a specific group type
by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-comment] [PATCH v9 2/4] admin: Fix section numbering
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Requirements are put one additional level down. Fix it.
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v4->v5:
- new patch
---
admin.tex | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/admin.tex b/admin.tex
index e51f9e6..fd3b97d 100644
--- a/admin.tex
+++ b/admin.tex
@@ -286,7 +286,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
supporting multiple group types, the list of supported commands
might differ between different group types.
-\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+\devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The device MUST validate \field{opcode}, \field{group_type} and
\field{group_member_id}, and if any of these has an invalid or
@@ -378,7 +378,7 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\field{VF Enable} refer to registers within the SR-IOV Extended
Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
-\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
+\drivernormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The driver MAY discover whether device supports a specific group type
by issuing VIRTIO_ADMIN_CMD_LIST_QUERY with the matching
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-dev] [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-04 23:37 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Introduce group member legacy common configuration and legacy device
configuration access read/write commands.
Group member legacy configuration structure access commands enable group
owner driver software to access legacy configuration structure on behalf
of the guest virtual machine.
Usecase:
========
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
=====================
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
=========
Above usecase requirements is solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using the administration
commands of the group owner PCI PF.
Two types of administration commands are added which read/write PCI VF
registers.
Software usage example:
=======================
1. One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper& | | functionalities | |
| | forwarder | | | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
Config region |
access Driver notifications
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Continue to use the virtio pci driver to bind to the
listed device id and use it as in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v8->v9:
- add missing articles in notify query command
- replaced 'this notification' with 'such a notification'
- addressed below comments from Michael
- dropped 'Region' from the commands
- added 7 reserved pad bytes in config write commands
- rewrote from 'use following structure' to 'field' has the following
struct..
- dropped mentioning to follow struct virtio_admin_cmd.
- added note about command limited to only sriov group type for now
- rewrote the description little differently
v7->v8:
- remove empty line at the end of file
- removed white space at the end
- addressed comments from Michael add link to pci
- renamed region to region_data
- made region_data width to be 16 bytes to cover for 8 bytes offset
- moved generic notification region related normative from pci to
generic section
v6->v7:
- changed administrative to administration
- renamed admin-access.tex to admin-interface.tex
- large rewrite ad generic admin commands instead of pci
- added theory of operation section
- added driver notification region query command
v5->v6:
- fixed previous missed abbreviation of LCC and LD
v4->v5:
- split from pci transport specific patch
- split conformance to transport and generic sections
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- rename fields from register to region
- avoided abbreviation for legacy, device and config
---
admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
admin.tex | 14 ++-
conformance.tex | 2 +
3 files changed, 220 insertions(+), 1 deletion(-)
create mode 100644 admin-cmds-legacy-interface.tex
diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
new file mode 100644
index 0000000..09001d5
--- /dev/null
+++ b/admin-cmds-legacy-interface.tex
@@ -0,0 +1,205 @@
+\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Legacy Interface}
+
+In some systems, there is a need to support utilizing the legacy driver with
+the device that do not directly support the legacy interface. In such scenarios,
+a group owner device can provide the legacy interface functionality for the
+group member devices. The driver of an owner device can then access the legacy
+interface of a member device on behalf of the legacy member device driver.
+
+For example, with the PCI SR-IOV group type, group members (VFs) can not present
+the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver.
+If the legacy driver is running inside a virtual machine, the hypervisor
+executing the virtual machine can present a virtual device with an I/O BAR in
+BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and
+forwards them to the group owner device (PF) using group administration commands.
+
+The following commands support such legacy interface functionality:
+
+\begin{enumerate}
+\item Legacy Common Configuration Write Command
+\item Legacy Common Configuration Read Command
+\item Legacy Device Configuration Write Command
+\item Legacy Device Configuration Read Command
+\end{enumerate}
+
+These commands are currently only defined for the PCI SR-IOV group type and
+have, generally, the same effect as member device accesses through a legacy
+interface listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}.
+
+\paragraph{Legacy Common Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Legacy Interface / Common Configuration Write Command}
+
+This command has the same effect as writing into the virtio common configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_wr_data {
+ u8 offset; /* Starting byte offset within the common configuration structure to write */
+ u8 reserved[7];
+ u8 region_data[];
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
+The driver sets valid \field{offset} and associated \field{region_data} bytes to
+write to the common configuration region.
+
+This command does not use \field{command_specific_result}.
+
+\paragraph{Legacy Common Configuration Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Common Configuration Read Command}
+
+This command has the same effect as reading from the virtio common configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_rd_data {
+ u8 offset; /* Starting byte offset within the common configuration structure to read */
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
+
+The driver sets valid \field{offset} of the region from where to read
+\field{region_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_rd_result {
+ u8 region_data[];
+};
+\end{lstlisting}
+
+\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Write Command}
+
+This command has the same effect as writing into the virtio device configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_reg_wr_data {
+ u8 offset; /* Starting byte offset within the device configuration structure to write */
+ u8 reserved[7];
+ u8 region_data[];
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
+The driver sets valid \field{offset} and associated \field{region_data} bytes to
+the device configuration region.
+
+This command does not use \field{command_specific_result}.
+
+\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Read Command}
+
+This command has the same effect as reading from the virtio device configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
+ u8 offset; /* Starting byte offset within the device configuration structure to read */
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
+The driver sets valid \field{offset} within the device configuration structure from
+where to read \field{region_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_reg_rd_result {
+ u8 region_data[];
+};
+\end{lstlisting}
+
+\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
+
+Even though the driver notifications can be communicated through the
+administration command, if the group owner device or group member device
+supports such notifications using a memory-mapped operation or I/O operation,
+they are sent to the device by accessing such a notification region using
+a memory or an I/O operation.
+
+A group owner device optionally support querying driver notifications region
+using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
+This command does not use \field{command_specific_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_notify_query_entry {
+ u8 region_data[16];
+};
+
+struct virtio_admin_cmd_legacy_notify_query_result {
+ struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
+};
+\end{lstlisting}
+
+The driver picks the suitable entry when multiple entries are supplied
+by the device.
+
+Refer to the specific transport section for the definition of the
+\field{region_data}.
+
+This command is currently only defined for the PCI SR-IOV group type.
+
+\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+
+If the group owner device supports legacy region access for its group member
+devices, the device MUST set all corresponding bits for commands
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
+the command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
+\field{device_admin_cmd_opcodes}.
+
+The device MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
+which is outside the legacy common configuration region's address range.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
+which is outside the legacy device specific region's address range.
+
+The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for
+driver notifications. If the group owner device supports driver
+notifications region for its group member devices, the device MUST set
+VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
+VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
+
+When the driver accesses the legacy region of the member device using
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
+commands, device MUST function as if they are accessed by the legacy interface
+defined by the transport of the member device.
+
+\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+
+The driver MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
+is in the legacy common configuration region address range.
+
+The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy
+device specific configuration region address range.
+
+The group member driver SHOULD use the notification region supplied by the group
+owner device.
diff --git a/admin.tex b/admin.tex
index fd3b97d..0de26a9 100644
--- a/admin.tex
+++ b/admin.tex
@@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\hline
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
\hline
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
+0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\
+\hline
+0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\
+\hline
+0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\
+\hline
+0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\
+\hline
+0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification region for a member device \\
+\hline
+0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
\hline
@@ -286,6 +296,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
supporting multiple group types, the list of supported commands
might differ between different group types.
+\input{admin-cmds-legacy-interface.tex}
+
\devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The device MUST validate \field{opcode}, \field{group_type} and
diff --git a/conformance.tex b/conformance.tex
index 01ccd69..dc00e84 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
+\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
\item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
Introduce group member legacy common configuration and legacy device
configuration access read/write commands.
Group member legacy configuration structure access commands enable group
owner driver software to access legacy configuration structure on behalf
of the guest virtual machine.
Usecase:
========
1. A hypervisor/system needs to provide transitional
virtio devices to the guest VM at scale of thousands,
typically, one to eight devices per VM.
2. A hypervisor/system needs to provide such devices using a
vendor agnostic driver in the hypervisor system.
3. A hypervisor system prefers to have single stack regardless of
virtio device type (net/blk) and be future compatible with a
single vfio stack using SR-IOV or other scalable device
virtualization technology to map PCI devices to the guest VM.
(as transitional or otherwise)
Motivation/Background:
=====================
The existing virtio transitional PCI device is missing support for
PCI SR-IOV based devices. Currently it does not work beyond
PCI PF, or as software emulated device in reality. Currently it
has below cited system level limitations:
[a] PCIe spec citation:
VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
[b] cpu arch citiation:
Intel 64 and IA-32 Architectures Software Developer’s Manual:
The processor’s I/O address space is separate and distinct from
the physical-memory address space. The I/O address space consists
of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
[c] PCIe spec citation:
If a bridge implements an I/O address range,...I/O address range will be
aligned to a 4 KB boundary.
Overview:
=========
Above usecase requirements is solved by PCI PF group owner accessing
its group member PCI VFs legacy registers using the administration
commands of the group owner PCI PF.
Two types of administration commands are added which read/write PCI VF
registers.
Software usage example:
=======================
1. One way to use and map to the guest VM is by using vfio driver
framework in Linux kernel.
+----------------------+
|pci_dev_id = 0x100X |
+---------------|pci_rev_id = 0x0 |-----+
|vfio device |BAR0 = I/O region | |
| |Other attributes | |
| +----------------------+ |
| |
+ +--------------+ +-----------------+ |
| |I/O BAR to AQ | | Other vfio | |
| |rd/wr mapper& | | functionalities | |
| | forwarder | | | |
| +--------------+ +-----------------+ |
| |
+------+-------------------------+-----------+
| |
Config region |
access Driver notifications
| |
+----+------------+ +----+------------+
| +-----+ | | PCI VF device A |
| | AQ |-------------+---->+-------------+ |
| +-----+ | | | | legacy regs | |
| PCI PF device | | | +-------------+ |
+-----------------+ | +-----------------+
|
| +----+------------+
| | PCI VF device N |
+---->+-------------+ |
| | legacy regs | |
| +-------------+ |
+-----------------+
2. Continue to use the virtio pci driver to bind to the
listed device id and use it as in the host.
3. Use it in a light weight hypervisor to run bare-metal OS.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v8->v9:
- add missing articles in notify query command
- replaced 'this notification' with 'such a notification'
- addressed below comments from Michael
- dropped 'Region' from the commands
- added 7 reserved pad bytes in config write commands
- rewrote from 'use following structure' to 'field' has the following
struct..
- dropped mentioning to follow struct virtio_admin_cmd.
- added note about command limited to only sriov group type for now
- rewrote the description little differently
v7->v8:
- remove empty line at the end of file
- removed white space at the end
- addressed comments from Michael add link to pci
- renamed region to region_data
- made region_data width to be 16 bytes to cover for 8 bytes offset
- moved generic notification region related normative from pci to
generic section
v6->v7:
- changed administrative to administration
- renamed admin-access.tex to admin-interface.tex
- large rewrite ad generic admin commands instead of pci
- added theory of operation section
- added driver notification region query command
v5->v6:
- fixed previous missed abbreviation of LCC and LD
v4->v5:
- split from pci transport specific patch
- split conformance to transport and generic sections
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
- rename fields from register to region
- avoided abbreviation for legacy, device and config
---
admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
admin.tex | 14 ++-
conformance.tex | 2 +
3 files changed, 220 insertions(+), 1 deletion(-)
create mode 100644 admin-cmds-legacy-interface.tex
diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
new file mode 100644
index 0000000..09001d5
--- /dev/null
+++ b/admin-cmds-legacy-interface.tex
@@ -0,0 +1,205 @@
+\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Legacy Interface}
+
+In some systems, there is a need to support utilizing the legacy driver with
+the device that do not directly support the legacy interface. In such scenarios,
+a group owner device can provide the legacy interface functionality for the
+group member devices. The driver of an owner device can then access the legacy
+interface of a member device on behalf of the legacy member device driver.
+
+For example, with the PCI SR-IOV group type, group members (VFs) can not present
+the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver.
+If the legacy driver is running inside a virtual machine, the hypervisor
+executing the virtual machine can present a virtual device with an I/O BAR in
+BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and
+forwards them to the group owner device (PF) using group administration commands.
+
+The following commands support such legacy interface functionality:
+
+\begin{enumerate}
+\item Legacy Common Configuration Write Command
+\item Legacy Common Configuration Read Command
+\item Legacy Device Configuration Write Command
+\item Legacy Device Configuration Read Command
+\end{enumerate}
+
+These commands are currently only defined for the PCI SR-IOV group type and
+have, generally, the same effect as member device accesses through a legacy
+interface listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}.
+
+\paragraph{Legacy Common Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group
+administration commands / Legacy Interface / Common Configuration Write Command}
+
+This command has the same effect as writing into the virtio common configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_wr_data {
+ u8 offset; /* Starting byte offset within the common configuration structure to write */
+ u8 reserved[7];
+ u8 region_data[];
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
+The driver sets valid \field{offset} and associated \field{region_data} bytes to
+write to the common configuration region.
+
+This command does not use \field{command_specific_result}.
+
+\paragraph{Legacy Common Configuration Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Common Configuration Read Command}
+
+This command has the same effect as reading from the virtio common configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_rd_data {
+ u8 offset; /* Starting byte offset within the common configuration structure to read */
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
+
+The driver sets valid \field{offset} of the region from where to read
+\field{region_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_common_cfg_rd_result {
+ u8 region_data[];
+};
+\end{lstlisting}
+
+\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Write Command}
+
+This command has the same effect as writing into the virtio device configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_reg_wr_data {
+ u8 offset; /* Starting byte offset within the device configuration structure to write */
+ u8 reserved[7];
+ u8 region_data[];
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
+The driver sets valid \field{offset} and associated \field{region_data} bytes to
+the device configuration region.
+
+This command does not use \field{command_specific_result}.
+
+\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Read Command}
+
+This command has the same effect as reading from the virtio device configuration
+structure through the legacy interface. \field{command_specific_data} has the
+following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
+ u8 offset; /* Starting byte offset within the device configuration structure to read */
+};
+\end{lstlisting}
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
+The driver sets valid \field{offset} within the device configuration structure from
+where to read \field{region_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_dev_reg_rd_result {
+ u8 region_data[];
+};
+\end{lstlisting}
+
+\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
+
+Even though the driver notifications can be communicated through the
+administration command, if the group owner device or group member device
+supports such notifications using a memory-mapped operation or I/O operation,
+they are sent to the device by accessing such a notification region using
+a memory or an I/O operation.
+
+A group owner device optionally support querying driver notifications region
+using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
+
+The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
+This command does not use \field{command_specific_data}.
+
+When command completes successfully, \field{command_specific_result}
+uses following structure:
+
+\begin{lstlisting}
+struct virtio_admin_cmd_legacy_notify_query_entry {
+ u8 region_data[16];
+};
+
+struct virtio_admin_cmd_legacy_notify_query_result {
+ struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
+};
+\end{lstlisting}
+
+The driver picks the suitable entry when multiple entries are supplied
+by the device.
+
+Refer to the specific transport section for the definition of the
+\field{region_data}.
+
+This command is currently only defined for the PCI SR-IOV group type.
+
+\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+
+If the group owner device supports legacy region access for its group member
+devices, the device MUST set all corresponding bits for commands
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
+the command result of VIRTIO_ADMIN_CMD_LIST_QUERY in
+\field{device_admin_cmd_opcodes}.
+
+The device MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
+which is outside the legacy common configuration region's address range.
+
+The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
+which is outside the legacy device specific region's address range.
+
+The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for
+driver notifications. If the group owner device supports driver
+notifications region for its group member devices, the device MUST set
+VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
+VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
+
+When the driver accesses the legacy region of the member device using
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
+commands, device MUST function as if they are accessed by the legacy interface
+defined by the transport of the member device.
+
+\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+
+The driver MUST encode and decode legacy device specific registers using
+little-endian format.
+
+The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
+is in the legacy common configuration region address range.
+
+The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy
+device specific configuration region address range.
+
+The group member driver SHOULD use the notification region supplied by the group
+owner device.
diff --git a/admin.tex b/admin.tex
index fd3b97d..0de26a9 100644
--- a/admin.tex
+++ b/admin.tex
@@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
\hline
0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
\hline
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
+0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\
+\hline
+0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\
+\hline
+0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\
+\hline
+0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\
+\hline
+0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification region for a member device \\
+\hline
+0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
\hline
0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
\hline
@@ -286,6 +296,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
supporting multiple group types, the list of supported commands
might differ between different group types.
+\input{admin-cmds-legacy-interface.tex}
+
\devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
The device MUST validate \field{opcode}, \field{group_type} and
diff --git a/conformance.tex b/conformance.tex
index 01ccd69..dc00e84 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
\item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
+\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
+\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
\item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-dev] [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-04 23:37 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
This patch links how in a PCI transport a group owner can access group
member (PCI VFs) legacy registers using a legacy registers access
commands using administration virtqueue infrastructure.
Additionally it extend the PCI notification capability through which a
PCI VF device indicates to the driver which PCI BAR region to be used
for driver notifications.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v7->v8:
- addressed comments from Michael
- made bar offset 64-bit
- prefix legacy specific structure with _legacy
- moved generic normative from pci to generic section
- added link to virtio pci capabilities when referring to bar 0
- remove 'should' from generic description
v6->v7:
- addressed comments from Michael
- removed driver normative about I/O BAR emulation as it does not
make much sense for the spec
- removed references to administration virtqueue
- rewrote driver legacy region access without guest and hypervisor
wording
- added normative for notification region
- added normative for PCI IDs for device which support legacy commands
v5->v6:
- aligned pci capability to 4B as required by PCI spec
- added text for the PCI capability for the group member device
v4->v5:
- split pci transport and generic command section to new patch
- removed multiple references to the VF
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion,
though it is fine. Instead replaced with extending virtio_pci_notify_cap
to indicate that legacy queue notifications can be done on the
notification location.
- fixed spelling errors.
- replaced administrative virtqueue to administration virtqueue
- added queue notification capability register to indicate legacy q
notification supported
v2->v3:
- adddressed Jason and Michael's comment to split single register
access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as
admin command cap bit already covers it.
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
---
admin-cmds-legacy-interface.tex | 2 +-
conformance.tex | 1 +
transport-pci-legacy-regs.tex | 41 +++++++++++++++++++++++++++++++++
transport-pci.tex | 2 ++
4 files changed, 45 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
index 09001d5..571b256 100644
--- a/admin-cmds-legacy-interface.tex
+++ b/admin-cmds-legacy-interface.tex
@@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
by the device.
Refer to the specific transport section for the definition of the
-\field{region_data}.
+\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
This command is currently only defined for the PCI SR-IOV group type.
diff --git a/conformance.tex b/conformance.tex
index dc00e84..b3f2c92 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -267,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
+\item Section \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
\item Section \ref{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
\item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision}
\item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue}
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
new file mode 100644
index 0000000..e4f70c9
--- /dev/null
+++ b/transport-pci-legacy-regs.tex
@@ -0,0 +1,41 @@
+\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
+
+The PCI owner device or the member device or both supports driver notifications using
+a notification region defined in the \field{struct virtio_pci_notify_region}.
+
+In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
+\field{region_data} is defined as following:
+
+\begin{lstlisting}
+struct virtio_pci_legacy_notify_region {
+ u8 owner; /* When set to 1, notification region is of the owner device */
+ u8 bar; /* BAR of the member or owner device */
+ u8 padding[6];
+ le64 offset; /* Offset within bar. */
+};
+\end{lstlisting}
+
+The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
+
+The group member device does not use PCI BAR0 in various Virtio PCI capabilities
+listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
+
+\devicenormative{\subsubsection}{Legacy Interface: Group Member Device Legacy Configuration Region Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
+
+When a PCI SR-IOV group owner device supports
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE commands, the group owner device MUST
+hardwire VF BAR0 in the SR-IOV Extended capability and the group member device
+MUST NOT use BAR0 in any of the Virtio Structure PCI Capabilities.
+
+The group owner device or the group member device or both MAY support driver
+notifications region.
+
+For the SR-IOV group type, the owner device supporting
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
+commands and its member device SHOULD follow the rules for the PCI Device ID,
+Revision ID and Subsystem Device ID of the non-transitional devices documented in
+section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery}.
diff --git a/transport-pci.tex b/transport-pci.tex
index a5c6719..72c78f6 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1212,3 +1212,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
re-examine the configuration space to see what changed.
\end{itemize}
\end{itemize}
+
+\input{transport-pci-legacy-regs.tex}
--
2.26.2
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-comment] [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
@ 2023-07-04 23:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-04 23:37 UTC (permalink / raw)
To: virtio-comment, mst, cohuck, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
This patch links how in a PCI transport a group owner can access group
member (PCI VFs) legacy registers using a legacy registers access
commands using administration virtqueue infrastructure.
Additionally it extend the PCI notification capability through which a
PCI VF device indicates to the driver which PCI BAR region to be used
for driver notifications.
Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
Signed-off-by: Parav Pandit <parav@nvidia.com>
---
changelog:
v7->v8:
- addressed comments from Michael
- made bar offset 64-bit
- prefix legacy specific structure with _legacy
- moved generic normative from pci to generic section
- added link to virtio pci capabilities when referring to bar 0
- remove 'should' from generic description
v6->v7:
- addressed comments from Michael
- removed driver normative about I/O BAR emulation as it does not
make much sense for the spec
- removed references to administration virtqueue
- rewrote driver legacy region access without guest and hypervisor
wording
- added normative for notification region
- added normative for PCI IDs for device which support legacy commands
v5->v6:
- aligned pci capability to 4B as required by PCI spec
- added text for the PCI capability for the group member device
v4->v5:
- split pci transport and generic command section to new patch
- removed multiple references to the VF
- written the description of the command as generic with member
and group device terminology
- reflected many section names to remove VF
v3->v4:
- moved noted to the conformance section details in next patch
- removed queue notify address query AQ command on Michael's suggestion,
though it is fine. Instead replaced with extending virtio_pci_notify_cap
to indicate that legacy queue notifications can be done on the
notification location.
- fixed spelling errors.
- replaced administrative virtqueue to administration virtqueue
- added queue notification capability register to indicate legacy q
notification supported
v2->v3:
- adddressed Jason and Michael's comment to split single register
access command to common config and device specific commands.
- dropped the suggetion to introduce enable/disable command as
admin command cap bit already covers it.
v1->v2:
- addressed comments from Michael
- added theory of operation
- grammar corrections
- removed group fields description from individual commands as
it is already present in generic section
- added endianness normative for legacy device registers region
- renamed the file to drop vf and add legacy prefix
- added overview in commit log
- renamed subsection to reflect command
---
admin-cmds-legacy-interface.tex | 2 +-
conformance.tex | 1 +
transport-pci-legacy-regs.tex | 41 +++++++++++++++++++++++++++++++++
transport-pci.tex | 2 ++
4 files changed, 45 insertions(+), 1 deletion(-)
create mode 100644 transport-pci-legacy-regs.tex
diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
index 09001d5..571b256 100644
--- a/admin-cmds-legacy-interface.tex
+++ b/admin-cmds-legacy-interface.tex
@@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
by the device.
Refer to the specific transport section for the definition of the
-\field{region_data}.
+\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
This command is currently only defined for the PCI SR-IOV group type.
diff --git a/conformance.tex b/conformance.tex
index dc00e84..b3f2c92 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -267,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection}
\item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
+\item Section \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
\item Section \ref{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
\item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision}
\item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue}
diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
new file mode 100644
index 0000000..e4f70c9
--- /dev/null
+++ b/transport-pci-legacy-regs.tex
@@ -0,0 +1,41 @@
+\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
+
+The PCI owner device or the member device or both supports driver notifications using
+a notification region defined in the \field{struct virtio_pci_notify_region}.
+
+In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
+\field{region_data} is defined as following:
+
+\begin{lstlisting}
+struct virtio_pci_legacy_notify_region {
+ u8 owner; /* When set to 1, notification region is of the owner device */
+ u8 bar; /* BAR of the member or owner device */
+ u8 padding[6];
+ le64 offset; /* Offset within bar. */
+};
+\end{lstlisting}
+
+The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
+
+The group member device does not use PCI BAR0 in various Virtio PCI capabilities
+listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
+
+\devicenormative{\subsubsection}{Legacy Interface: Group Member Device Legacy Configuration Region Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
+
+When a PCI SR-IOV group owner device supports
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE commands, the group owner device MUST
+hardwire VF BAR0 in the SR-IOV Extended capability and the group member device
+MUST NOT use BAR0 in any of the Virtio Structure PCI Capabilities.
+
+The group owner device or the group member device or both MAY support driver
+notifications region.
+
+For the SR-IOV group type, the owner device supporting
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
+VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
+commands and its member device SHOULD follow the rules for the PCI Device ID,
+Revision ID and Subsystem Device ID of the non-transitional devices documented in
+section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery}.
diff --git a/transport-pci.tex b/transport-pci.tex
index a5c6719..72c78f6 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1212,3 +1212,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
re-examine the configuration space to see what changed.
\end{itemize}
\end{itemize}
+
+\input{transport-pci-legacy-regs.tex}
--
2.26.2
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply related [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [PATCH v9 1/4] admin: Split opcode table rows with a line
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 10:05 ` Cornelia Huck
-1 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:05 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> Currently all opcode appears to be in a single row.
> Separate them with a line similar to other tables.
>
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v2->v3:
> - new patch
> ---
> admin.tex | 2 ++
> 1 file changed, 2 insertions(+)
>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] Re: [PATCH v9 1/4] admin: Split opcode table rows with a line
@ 2023-07-05 10:05 ` Cornelia Huck
0 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:05 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> Currently all opcode appears to be in a single row.
> Separate them with a line similar to other tables.
>
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v2->v3:
> - new patch
> ---
> admin.tex | 2 ++
> 1 file changed, 2 insertions(+)
>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [PATCH v9 2/4] admin: Fix section numbering
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 10:09 ` Cornelia Huck
-1 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:09 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> Requirements are put one additional level down. Fix it.
>
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v4->v5:
> - new patch
> ---
> admin.tex | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
Same thing applies to the normative statements for the admin vq (in the
same file) -- do you want to fix that as well?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] Re: [PATCH v9 2/4] admin: Fix section numbering
@ 2023-07-05 10:09 ` Cornelia Huck
0 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:09 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> Requirements are put one additional level down. Fix it.
>
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v4->v5:
> - new patch
> ---
> admin.tex | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
Same thing applies to the normative statements for the admin vq (in the
same file) -- do you want to fix that as well?
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 10:24 ` Cornelia Huck
-1 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:24 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> new file mode 100644
> index 0000000..09001d5
> --- /dev/null
> +++ b/admin-cmds-legacy-interface.tex
> @@ -0,0 +1,205 @@
> +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface}
> +
> +In some systems, there is a need to support utilizing the legacy driver with
> +the device that do not directly support the legacy interface. In such scenarios,
Maybe better:
"In some systems, there is a need to support utilizing legacy drivers with
devices that do not directly support the legacy interface."
?
> +a group owner device can provide the legacy interface functionality for the
> +group member devices. The driver of an owner device can then access the legacy
> +interface of a member device on behalf of the legacy member device driver.
(...)
> +\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
> +
> +Even though the driver notifications can be communicated through the
> +administration command, if the group owner device or group member device
> +supports such notifications using a memory-mapped operation or I/O operation,
> +they are sent to the device by accessing such a notification region using
> +a memory or an I/O operation.
I find this sentence confusing, probably because it is too
complex... what about
"If the group owner device or the group member device support driver
notifications via a memory-mapped operation or I/O operation, these
notifications are sent to the device via accessing such a notification
region using a memory or I/O operation {in addition to|instead of}
communicating the notifications through the admistration command."
(I'm not sure whether it is either or both.)
> +
> +A group owner device optionally support querying driver notifications region
"can optionally support" ?
> +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> +This command does not use \field{command_specific_data}.
> +
> +When command completes successfully, \field{command_specific_result}
"the command"
> +uses following structure:
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] Re: [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-05 10:24 ` Cornelia Huck
0 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:24 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> new file mode 100644
> index 0000000..09001d5
> --- /dev/null
> +++ b/admin-cmds-legacy-interface.tex
> @@ -0,0 +1,205 @@
> +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface}
> +
> +In some systems, there is a need to support utilizing the legacy driver with
> +the device that do not directly support the legacy interface. In such scenarios,
Maybe better:
"In some systems, there is a need to support utilizing legacy drivers with
devices that do not directly support the legacy interface."
?
> +a group owner device can provide the legacy interface functionality for the
> +group member devices. The driver of an owner device can then access the legacy
> +interface of a member device on behalf of the legacy member device driver.
(...)
> +\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
> +
> +Even though the driver notifications can be communicated through the
> +administration command, if the group owner device or group member device
> +supports such notifications using a memory-mapped operation or I/O operation,
> +they are sent to the device by accessing such a notification region using
> +a memory or an I/O operation.
I find this sentence confusing, probably because it is too
complex... what about
"If the group owner device or the group member device support driver
notifications via a memory-mapped operation or I/O operation, these
notifications are sent to the device via accessing such a notification
region using a memory or I/O operation {in addition to|instead of}
communicating the notifications through the admistration command."
(I'm not sure whether it is either or both.)
> +
> +A group owner device optionally support querying driver notifications region
"can optionally support" ?
> +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> +This command does not use \field{command_specific_data}.
> +
> +When command completes successfully, \field{command_specific_result}
"the command"
> +uses following structure:
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 10:29 ` Cornelia Huck
-1 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:29 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> index 09001d5..571b256 100644
> --- a/admin-cmds-legacy-interface.tex
> +++ b/admin-cmds-legacy-interface.tex
> @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
> by the device.
>
> Refer to the specific transport section for the definition of the
> -\field{region_data}.
> +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
"For the PCI transport, refer to..."
>
> This command is currently only defined for the PCI SR-IOV group type.
>
(...)
> diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
> new file mode 100644
> index 0000000..e4f70c9
> --- /dev/null
> +++ b/transport-pci-legacy-regs.tex
> @@ -0,0 +1,41 @@
> +\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +The PCI owner device or the member device or both supports driver notifications using
s/supports/support/
> +a notification region defined in the \field{struct virtio_pci_notify_region}.
s/in the/in/
> +
> +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> +\field{region_data} is defined as following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_legacy_notify_region {
> + u8 owner; /* When set to 1, notification region is of the owner device */
> + u8 bar; /* BAR of the member or owner device */
> + u8 padding[6];
> + le64 offset; /* Offset within bar. */
> +};
> +\end{lstlisting}
> +
> +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
s/hardwire/hardwires/
> +
> +The group member device does not use PCI BAR0 in various Virtio PCI capabilities
"in the various" caps? Or "in some of the" caps?
> +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] Re: [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
@ 2023-07-05 10:29 ` Cornelia Huck
0 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 10:29 UTC (permalink / raw)
To: Parav Pandit, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, yishaih, maorg, shahafs, Parav Pandit
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> index 09001d5..571b256 100644
> --- a/admin-cmds-legacy-interface.tex
> +++ b/admin-cmds-legacy-interface.tex
> @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
> by the device.
>
> Refer to the specific transport section for the definition of the
> -\field{region_data}.
> +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
"For the PCI transport, refer to..."
>
> This command is currently only defined for the PCI SR-IOV group type.
>
(...)
> diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
> new file mode 100644
> index 0000000..e4f70c9
> --- /dev/null
> +++ b/transport-pci-legacy-regs.tex
> @@ -0,0 +1,41 @@
> +\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +The PCI owner device or the member device or both supports driver notifications using
s/supports/support/
> +a notification region defined in the \field{struct virtio_pci_notify_region}.
s/in the/in/
> +
> +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> +\field{region_data} is defined as following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_legacy_notify_region {
> + u8 owner; /* When set to 1, notification region is of the owner device */
> + u8 bar; /* BAR of the member or owner device */
> + u8 padding[6];
> + le64 offset; /* Offset within bar. */
> +};
> +\end{lstlisting}
> +
> +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
s/hardwire/hardwires/
> +
> +The group member device does not use PCI BAR0 in various Virtio PCI capabilities
"in the various" caps? Or "in some of the" caps?
> +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [virtio-comment] [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 10:56 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 10:56 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:24AM +0300, Parav Pandit wrote:
> This patch links how in a PCI transport a group owner can access group
> member (PCI VFs) legacy registers using a legacy registers access
> commands using administration virtqueue infrastructure.
>
> Additionally it extend the PCI notification capability through which a
> PCI VF device indicates to the driver which PCI BAR region to be used
> for driver notifications.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v7->v8:
> - addressed comments from Michael
> - made bar offset 64-bit
> - prefix legacy specific structure with _legacy
> - moved generic normative from pci to generic section
> - added link to virtio pci capabilities when referring to bar 0
> - remove 'should' from generic description
> v6->v7:
> - addressed comments from Michael
> - removed driver normative about I/O BAR emulation as it does not
> make much sense for the spec
> - removed references to administration virtqueue
> - rewrote driver legacy region access without guest and hypervisor
> wording
> - added normative for notification region
> - added normative for PCI IDs for device which support legacy commands
> v5->v6:
> - aligned pci capability to 4B as required by PCI spec
> - added text for the PCI capability for the group member device
> v4->v5:
> - split pci transport and generic command section to new patch
> - removed multiple references to the VF
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> v3->v4:
> - moved noted to the conformance section details in next patch
> - removed queue notify address query AQ command on Michael's suggestion,
> though it is fine. Instead replaced with extending virtio_pci_notify_cap
> to indicate that legacy queue notifications can be done on the
> notification location.
> - fixed spelling errors.
> - replaced administrative virtqueue to administration virtqueue
> - added queue notification capability register to indicate legacy q
> notification supported
> v2->v3:
> - adddressed Jason and Michael's comment to split single register
> access command to common config and device specific commands.
> - dropped the suggetion to introduce enable/disable command as
> admin command cap bit already covers it.
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
>
> - added overview in commit log
> - renamed subsection to reflect command
> ---
> admin-cmds-legacy-interface.tex | 2 +-
> conformance.tex | 1 +
> transport-pci-legacy-regs.tex | 41 +++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 ++
> 4 files changed, 45 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> index 09001d5..571b256 100644
> --- a/admin-cmds-legacy-interface.tex
> +++ b/admin-cmds-legacy-interface.tex
> @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
> by the device.
>
> Refer to the specific transport section for the definition of the
> -\field{region_data}.
> +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
>
> This command is currently only defined for the PCI SR-IOV group type.
>
> diff --git a/conformance.tex b/conformance.tex
> index dc00e84..b3f2c92 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -267,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
> +\item Section \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
> \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision}
> \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue}
> diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
> new file mode 100644
> index 0000000..e4f70c9
> --- /dev/null
> +++ b/transport-pci-legacy-regs.tex
> @@ -0,0 +1,41 @@
> +\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +The PCI owner device or the member device or both supports driver notifications using
> +a notification region defined in the \field{struct virtio_pci_notify_region}.
> +
> +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> +\field{region_data} is defined as following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_legacy_notify_region {
> + u8 owner; /* When set to 1, notification region is of the owner device */
> + u8 bar; /* BAR of the member or owner device */
> + u8 padding[6];
> + le64 offset; /* Offset within bar. */
> +};
> +\end{lstlisting}
> +
> +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
> +
> +The group member device does not use PCI BAR0 in various Virtio PCI capabilities
> +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
> +
> +\devicenormative{\subsubsection}{Legacy Interface: Group Member Device Legacy Configuration Region Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +When a PCI SR-IOV group owner device supports
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE commands, the group owner device MUST
> +hardwire VF BAR0 in the SR-IOV Extended capability and the group member device
> +MUST NOT use BAR0 in any of the Virtio Structure PCI Capabilities.
> +
> +The group owner device or the group member device or both MAY support driver
> +notifications region.
> +
> +For the SR-IOV group type, the owner device supporting
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> +commands and its member device SHOULD follow the rules for the PCI Device ID,
> +Revision ID and Subsystem Device ID of the non-transitional devices documented in
> +section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery}.
> diff --git a/transport-pci.tex b/transport-pci.tex
> index a5c6719..72c78f6 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -1212,3 +1212,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
> re-examine the configuration space to see what changed.
> \end{itemize}
> \end{itemize}
> +
> +\input{transport-pci-legacy-regs.tex}
This is completely apropos rien here.
Include this from admin.tex for now pls.
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [virtio-comment] [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
@ 2023-07-05 10:56 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 10:56 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:24AM +0300, Parav Pandit wrote:
> This patch links how in a PCI transport a group owner can access group
> member (PCI VFs) legacy registers using a legacy registers access
> commands using administration virtqueue infrastructure.
>
> Additionally it extend the PCI notification capability through which a
> PCI VF device indicates to the driver which PCI BAR region to be used
> for driver notifications.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v7->v8:
> - addressed comments from Michael
> - made bar offset 64-bit
> - prefix legacy specific structure with _legacy
> - moved generic normative from pci to generic section
> - added link to virtio pci capabilities when referring to bar 0
> - remove 'should' from generic description
> v6->v7:
> - addressed comments from Michael
> - removed driver normative about I/O BAR emulation as it does not
> make much sense for the spec
> - removed references to administration virtqueue
> - rewrote driver legacy region access without guest and hypervisor
> wording
> - added normative for notification region
> - added normative for PCI IDs for device which support legacy commands
> v5->v6:
> - aligned pci capability to 4B as required by PCI spec
> - added text for the PCI capability for the group member device
> v4->v5:
> - split pci transport and generic command section to new patch
> - removed multiple references to the VF
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> v3->v4:
> - moved noted to the conformance section details in next patch
> - removed queue notify address query AQ command on Michael's suggestion,
> though it is fine. Instead replaced with extending virtio_pci_notify_cap
> to indicate that legacy queue notifications can be done on the
> notification location.
> - fixed spelling errors.
> - replaced administrative virtqueue to administration virtqueue
> - added queue notification capability register to indicate legacy q
> notification supported
> v2->v3:
> - adddressed Jason and Michael's comment to split single register
> access command to common config and device specific commands.
> - dropped the suggetion to introduce enable/disable command as
> admin command cap bit already covers it.
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
>
> - added overview in commit log
> - renamed subsection to reflect command
> ---
> admin-cmds-legacy-interface.tex | 2 +-
> conformance.tex | 1 +
> transport-pci-legacy-regs.tex | 41 +++++++++++++++++++++++++++++++++
> transport-pci.tex | 2 ++
> 4 files changed, 45 insertions(+), 1 deletion(-)
> create mode 100644 transport-pci-legacy-regs.tex
>
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> index 09001d5..571b256 100644
> --- a/admin-cmds-legacy-interface.tex
> +++ b/admin-cmds-legacy-interface.tex
> @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device
> by the device.
>
> Refer to the specific transport section for the definition of the
> -\field{region_data}.
> +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}.
>
> This command is currently only defined for the PCI SR-IOV group type.
>
> diff --git a/conformance.tex b/conformance.tex
> index dc00e84..b3f2c92 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -267,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtio Device Configuration Layout Detection / Legacy Interface: A Note on Device Layout Detection}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Virtqueue Configuration / Legacy Interface: A Note on Virtqueue Configuration}
> +\item Section \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over MMIO / Legacy interface}
> \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Setting the Virtio Revision / Legacy Interfaces: A Note on Setting the Virtio Revision}
> \item Section \ref{sec:Virtio Transport Options / Virtio over channel I/O / Device Initialization / Configuring a Virtqueue / Legacy Interface: A Note on Configuring a Virtqueue}
> diff --git a/transport-pci-legacy-regs.tex b/transport-pci-legacy-regs.tex
> new file mode 100644
> index 0000000..e4f70c9
> --- /dev/null
> +++ b/transport-pci-legacy-regs.tex
> @@ -0,0 +1,41 @@
> +\subsection{Legacy Interface: Group member device Configuration Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +The PCI owner device or the member device or both supports driver notifications using
> +a notification region defined in the \field{struct virtio_pci_notify_region}.
> +
> +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> +\field{region_data} is defined as following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_legacy_notify_region {
> + u8 owner; /* When set to 1, notification region is of the owner device */
> + u8 bar; /* BAR of the member or owner device */
> + u8 padding[6];
> + le64 offset; /* Offset within bar. */
> +};
> +\end{lstlisting}
> +
> +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
> +
> +The group member device does not use PCI BAR0 in various Virtio PCI capabilities
> +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / Virtio Structure PCI Capabilities}.
> +
> +\devicenormative{\subsubsection}{Legacy Interface: Group Member Device Legacy Configuration Region Access}{Virtio Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member Device Configuration Region Access}
> +
> +When a PCI SR-IOV group owner device supports
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE commands, the group owner device MUST
> +hardwire VF BAR0 in the SR-IOV Extended capability and the group member device
> +MUST NOT use BAR0 in any of the Virtio Structure PCI Capabilities.
> +
> +The group owner device or the group member device or both MAY support driver
> +notifications region.
> +
> +For the SR-IOV group type, the owner device supporting
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> +commands and its member device SHOULD follow the rules for the PCI Device ID,
> +Revision ID and Subsystem Device ID of the non-transitional devices documented in
> +section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery}.
> diff --git a/transport-pci.tex b/transport-pci.tex
> index a5c6719..72c78f6 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -1212,3 +1212,5 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
> re-examine the configuration space to see what changed.
> \end{itemize}
> \end{itemize}
> +
> +\input{transport-pci-legacy-regs.tex}
This is completely apropos rien here.
Include this from admin.tex for now pls.
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 11:04 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 11:04 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:23AM +0300, Parav Pandit wrote:
> Introduce group member legacy common configuration and legacy device
> configuration access read/write commands.
>
> Group member legacy configuration structure access commands enable group
> owner driver software to access legacy configuration structure on behalf
> of the guest virtual machine.
>
> Usecase:
> ========
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> =====================
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> =========
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using the administration
> commands of the group owner PCI PF.
>
> Two types of administration commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> =======================
>
> 1. One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper& | | functionalities | |
> | | forwarder | | | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Config region |
> access Driver notifications
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Continue to use the virtio pci driver to bind to the
> listed device id and use it as in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v8->v9:
> - add missing articles in notify query command
> - replaced 'this notification' with 'such a notification'
> - addressed below comments from Michael
> - dropped 'Region' from the commands
> - added 7 reserved pad bytes in config write commands
> - rewrote from 'use following structure' to 'field' has the following
> struct..
> - dropped mentioning to follow struct virtio_admin_cmd.
> - added note about command limited to only sriov group type for now
> - rewrote the description little differently
Ok we are making progress.
Unfortunately in the below I'm repeating some comments from v8. take
care.
> v7->v8:
> - remove empty line at the end of file
> - removed white space at the end
> - addressed comments from Michael add link to pci
> - renamed region to region_data
> - made region_data width to be 16 bytes to cover for 8 bytes offset
> - moved generic notification region related normative from pci to
> generic section
> v6->v7:
> - changed administrative to administration
> - renamed admin-access.tex to admin-interface.tex
> - large rewrite ad generic admin commands instead of pci
> - added theory of operation section
> - added driver notification region query command
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> v4->v5:
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> ---
> admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
> admin.tex | 14 ++-
> conformance.tex | 2 +
> 3 files changed, 220 insertions(+), 1 deletion(-)
> create mode 100644 admin-cmds-legacy-interface.tex
>
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> new file mode 100644
> index 0000000..09001d5
> --- /dev/null
> +++ b/admin-cmds-legacy-interface.tex
> @@ -0,0 +1,205 @@
> +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface}
> +
> +In some systems, there is a need to support utilizing the legacy driver with
> +the device that do not directly support the legacy interface. In such scenarios,
> +a group owner device can provide the legacy interface functionality for the
> +group member devices. The driver of an owner device can then access the legacy
> +interface of a member device on behalf of the legacy member device driver.
> +
> +For example, with the PCI SR-IOV group type, group members (VFs) can not present
> +the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver.
> +If the legacy driver is running inside a virtual machine, the hypervisor
> +executing the virtual machine can present a virtual device with an I/O BAR in
> +BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and
> +forwards them to the group owner device (PF) using group administration commands.
> +
> +The following commands support such legacy interface functionality:
> +
> +\begin{enumerate}
> +\item Legacy Common Configuration Write Command
> +\item Legacy Common Configuration Read Command
> +\item Legacy Device Configuration Write Command
> +\item Legacy Device Configuration Read Command
> +\end{enumerate}
> +
> +These commands are currently only defined for the PCI SR-IOV group type and
We call it SR-IOV group type not PCI SR-IOV group type. Please be
consistent.
> +have, generally, the same effect as member device accesses through a legacy
> +interface listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}.
> +
> +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface / Common Configuration Write Command}
Fix label.
> +
> +This command has the same effect as writing into the virtio common configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> + u8 offset; /* Starting byte offset within the common configuration structure to write */
> + u8 reserved[7];
> + u8 region_data[];
I am all for something more specific than offset/data but region seems
pointless. And why apply just to data not offset?
How about legacy_offset/legacy_data?
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +write to the common configuration region.
> +
> +This command does not use \field{command_specific_result}.
> +
Let's make this consistent with other commands, even if it means we
repeat ourselves a bit. And length is not documented and has to be:
>>>
For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE \field{opcode}
is set to 0x2.
The \field{group_member_id} refers to the member device to be accessed.
The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_legacy_common_cfg_wr_data} describing the
access to be performed.
Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
The \field{offset} refers to the offset to write within the virtio common configuration
structure, and excluding the device-specific configuration. For the
SR-IOV group type, the layout used refers to the one used for the legacy interface
as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}, except that little endian format is assumed unconditionally.
The length of the data to write is simply the length of \field{region_data}.
No length or alignment restrictions are placed on the value
of the \field{offset} and the length of the \field{region_data}, except that the resulting access
refers to a single field and is completely within the virtio common configuration structure,
excluding the device-specific configuration.
This command has no command specific result.
Try to apply this to other commands please.
> +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Common Configuration Read Command}
> +
> +This command has the same effect as reading from the virtio common configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> + u8 offset; /* Starting byte offset within the common configuration structure to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> +
> +The driver sets valid \field{offset} of the region from where to read
> +\field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Write Command}
> +
> +This command has the same effect as writing into the virtio device configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> + u8 offset; /* Starting byte offset within the device configuration structure to write */
> + u8 reserved[7];
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +the device configuration region.
> +
> +This command does not use \field{command_specific_result}.
> +
> +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Read Command}
> +
> +This command has the same effect as reading from the virtio device configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> + u8 offset; /* Starting byte offset within the device configuration structure to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> +The driver sets valid \field{offset} within the device configuration structure from
> +where to read \field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
Drop "Query", this talks about notifications generally.
> +
> +Even though the driver notifications can be communicated through the
> +administration command, if the group owner device or group member device
> +supports such notifications using a memory-mapped operation or I/O operation,
> +they are sent to the device by accessing such a notification region using
> +a memory or an I/O operation.
> +
> +A group owner device optionally support querying driver notifications region
> +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
The driver of the owner device can send a driver notification to the member
device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
with the \field{offset} matching \fielf{Queue Notify} and the
\field{region_data} containing the virtqueue index to be notified.
However, as many administration commands are used for slow path
configuration, a separate fast path mechanism for such notifications is
desired. For the SR-IOV group type, the optional command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by returning to
the driver one or more addresses which to be used to send
such driver notifications.
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> +This command does not use \field{command_specific_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_notify_query_entry {
> + u8 region_data[16];
So everything is pci specific but this one we for some reason decided
to keep generic. I'd just make it pci specific, keep things simple.
We'll generalize if and when other transports want this. Which I doubt.
> +};
> +
> +struct virtio_admin_cmd_legacy_notify_query_result {
> + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> +};
> +\end{lstlisting}
here I'm not going to repeat myself. same comments as other commands.
> +
> +The driver picks the suitable entry when multiple entries are supplied
> +by the device.
How does driver know how much to allocate?
I think it allocates an extra entry, inits it to 0, then after the
command checks whether it was modified. If yes makes the buffer
larger. Makes sense? Complex though ...
Can we place an upper limit on # of entries? 4 seems sufficient no?
And what is "suitable"? I am guessing it just picks the
*first* entry that it is able to utilize.
And devices list them in the order of preference.
> +
> +Refer to the specific transport section for the definition of the
> +\field{region_data}.
Transport sections are huge. Links please do not make the reader guess.
> +
> +This command is currently only defined for the PCI SR-IOV group type.
> +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +
> +If the group owner device supports legacy region access for its group member
> +devices, the device MUST set all corresponding bits for commands
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
> +the command result
> of VIRTIO_ADMIN_CMD_LIST_QUERY in
> +\field{device_admin_cmd_opcodes}.
It's enough to just say what is supported:
A device MUST either support all of, or none of
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> +
> +The device MUST encode and decode legacy device specific registers using
> +little-endian format.
For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
the device MUST decode and encode (respectively) the value of
the \field{region_data} using the little-endian format.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
> +which is outside the legacy common configuration region's address range.
The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the value of the
\field{offset} and the length of the \field{region_data} does not refer
to a single field or is not completely within the virtio common
configuration structure, excluding the device-specific configuration.
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
> +which is outside the legacy device specific region's address range.
The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of the
\field{offset} and the length of the \field{region_data} does not refer
to a single field or is not completely within the device-specific
configuration.
> +
> +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for
> +driver notifications. If the group owner device supports driver
> +notifications region for its group member devices, the device MUST set
> +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
I don't get what does all this mean. Seems to say if you support
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
that anyway.
Should we say:
If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
also support all of
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> +
> +When the driver accesses the legacy region of the member device using
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> +commands, device MUST function as if they are accessed by the legacy interface
> +defined by the transport of the member device.
The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
have the same effect as writing into the virtio common configuration
structure through the legacy interface.
repeat for the other commands.
Also, explain about notification: is it true that writing
at this address has the same effect as writing into Queue Notify?
> +
> +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +
> +The driver MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
> +is in the legacy common configuration region address range.
> +
> +The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy
> +device specific configuration region address range.
> +
> +The group member driver SHOULD use the notification region supplied by the group
> +owner device.
Use how? More specific please. Also, only if the command is supported.
Document how is notification thingy used.
> diff --git a/admin.tex b/admin.tex
> index fd3b97d..0de26a9 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> \hline
> 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> \hline
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\
> +\hline
> +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\
> +\hline
> +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\
> +\hline
> +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\
> +\hline
> +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification region for a member device \\
> +\hline
> +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> \hline
> 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
> \hline
Grammar fix + drop "region":
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy common configuration structure
> @@ -286,6 +296,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> supporting multiple group types, the list of supported commands
> might differ between different group types.
>
> +\input{admin-cmds-legacy-interface.tex}
> +
> \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>
> The device MUST validate \field{opcode}, \field{group_type} and
> diff --git a/conformance.tex b/conformance.tex
> index 01ccd69..dc00e84 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
> +\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> \item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-05 11:04 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 11:04 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:23AM +0300, Parav Pandit wrote:
> Introduce group member legacy common configuration and legacy device
> configuration access read/write commands.
>
> Group member legacy configuration structure access commands enable group
> owner driver software to access legacy configuration structure on behalf
> of the guest virtual machine.
>
> Usecase:
> ========
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> =====================
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> =========
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using the administration
> commands of the group owner PCI PF.
>
> Two types of administration commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> =======================
>
> 1. One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper& | | functionalities | |
> | | forwarder | | | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Config region |
> access Driver notifications
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Continue to use the virtio pci driver to bind to the
> listed device id and use it as in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
> ---
> changelog:
> v8->v9:
> - add missing articles in notify query command
> - replaced 'this notification' with 'such a notification'
> - addressed below comments from Michael
> - dropped 'Region' from the commands
> - added 7 reserved pad bytes in config write commands
> - rewrote from 'use following structure' to 'field' has the following
> struct..
> - dropped mentioning to follow struct virtio_admin_cmd.
> - added note about command limited to only sriov group type for now
> - rewrote the description little differently
Ok we are making progress.
Unfortunately in the below I'm repeating some comments from v8. take
care.
> v7->v8:
> - remove empty line at the end of file
> - removed white space at the end
> - addressed comments from Michael add link to pci
> - renamed region to region_data
> - made region_data width to be 16 bytes to cover for 8 bytes offset
> - moved generic notification region related normative from pci to
> generic section
> v6->v7:
> - changed administrative to administration
> - renamed admin-access.tex to admin-interface.tex
> - large rewrite ad generic admin commands instead of pci
> - added theory of operation section
> - added driver notification region query command
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> v4->v5:
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> ---
> admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
> admin.tex | 14 ++-
> conformance.tex | 2 +
> 3 files changed, 220 insertions(+), 1 deletion(-)
> create mode 100644 admin-cmds-legacy-interface.tex
>
> diff --git a/admin-cmds-legacy-interface.tex b/admin-cmds-legacy-interface.tex
> new file mode 100644
> index 0000000..09001d5
> --- /dev/null
> +++ b/admin-cmds-legacy-interface.tex
> @@ -0,0 +1,205 @@
> +\subsubsection{Legacy Interfaces}\label{sec:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface}
> +
> +In some systems, there is a need to support utilizing the legacy driver with
> +the device that do not directly support the legacy interface. In such scenarios,
> +a group owner device can provide the legacy interface functionality for the
> +group member devices. The driver of an owner device can then access the legacy
> +interface of a member device on behalf of the legacy member device driver.
> +
> +For example, with the PCI SR-IOV group type, group members (VFs) can not present
> +the legacy interface in an I/O BAR in BAR0 as expected by the legacy pci driver.
> +If the legacy driver is running inside a virtual machine, the hypervisor
> +executing the virtual machine can present a virtual device with an I/O BAR in
> +BAR0. The hypervisor intercepts the legacy driver accesses to this I/O BAR and
> +forwards them to the group owner device (PF) using group administration commands.
> +
> +The following commands support such legacy interface functionality:
> +
> +\begin{enumerate}
> +\item Legacy Common Configuration Write Command
> +\item Legacy Common Configuration Read Command
> +\item Legacy Device Configuration Write Command
> +\item Legacy Device Configuration Read Command
> +\end{enumerate}
> +
> +These commands are currently only defined for the PCI SR-IOV group type and
We call it SR-IOV group type not PCI SR-IOV group type. Please be
consistent.
> +have, generally, the same effect as member device accesses through a legacy
> +interface listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}.
> +
> +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group
> +administration commands / Legacy Interface / Common Configuration Write Command}
Fix label.
> +
> +This command has the same effect as writing into the virtio common configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> + u8 offset; /* Starting byte offset within the common configuration structure to write */
> + u8 reserved[7];
> + u8 region_data[];
I am all for something more specific than offset/data but region seems
pointless. And why apply just to data not offset?
How about legacy_offset/legacy_data?
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +write to the common configuration region.
> +
> +This command does not use \field{command_specific_result}.
> +
Let's make this consistent with other commands, even if it means we
repeat ourselves a bit. And length is not documented and has to be:
>>>
For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE \field{opcode}
is set to 0x2.
The \field{group_member_id} refers to the member device to be accessed.
The \field{command_specific_data} is in the format
\field{struct virtio_admin_cmd_legacy_common_cfg_wr_data} describing the
access to be performed.
Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
The \field{offset} refers to the offset to write within the virtio common configuration
structure, and excluding the device-specific configuration. For the
SR-IOV group type, the layout used refers to the one used for the legacy interface
as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}, except that little endian format is assumed unconditionally.
The length of the data to write is simply the length of \field{region_data}.
No length or alignment restrictions are placed on the value
of the \field{offset} and the length of the \field{region_data}, except that the resulting access
refers to a single field and is completely within the virtio common configuration structure,
excluding the device-specific configuration.
This command has no command specific result.
Try to apply this to other commands please.
> +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Common Configuration Read Command}
> +
> +This command has the same effect as reading from the virtio common configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> + u8 offset; /* Starting byte offset within the common configuration structure to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> +
> +The driver sets valid \field{offset} of the region from where to read
> +\field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Write Command}
> +
> +This command has the same effect as writing into the virtio device configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> + u8 offset; /* Starting byte offset within the device configuration structure to write */
> + u8 reserved[7];
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> +The driver sets valid \field{offset} and associated \field{region_data} bytes to
> +the device configuration region.
> +
> +This command does not use \field{command_specific_result}.
> +
> +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Device Configuration Read Command}
> +
> +This command has the same effect as reading from the virtio device configuration
> +structure through the legacy interface. \field{command_specific_data} has the
> +following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> + u8 offset; /* Starting byte offset within the device configuration structure to read */
> +};
> +\end{lstlisting}
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> +The driver sets valid \field{offset} within the device configuration structure from
> +where to read \field{region_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> + u8 region_data[];
> +};
> +\end{lstlisting}
> +
> +\paragraph{Legacy Driver Notification Query}\label{par:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface / Legacy Driver Notifications Query}
Drop "Query", this talks about notifications generally.
> +
> +Even though the driver notifications can be communicated through the
> +administration command, if the group owner device or group member device
> +supports such notifications using a memory-mapped operation or I/O operation,
> +they are sent to the device by accessing such a notification region using
> +a memory or an I/O operation.
> +
> +A group owner device optionally support querying driver notifications region
> +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
The driver of the owner device can send a driver notification to the member
device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
with the \field{offset} matching \fielf{Queue Notify} and the
\field{region_data} containing the virtqueue index to be notified.
However, as many administration commands are used for slow path
configuration, a separate fast path mechanism for such notifications is
desired. For the SR-IOV group type, the optional command
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by returning to
the driver one or more addresses which to be used to send
such driver notifications.
> +
> +The driver sets command \field{opcode} to VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> +This command does not use \field{command_specific_data}.
> +
> +When command completes successfully, \field{command_specific_result}
> +uses following structure:
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_legacy_notify_query_entry {
> + u8 region_data[16];
So everything is pci specific but this one we for some reason decided
to keep generic. I'd just make it pci specific, keep things simple.
We'll generalize if and when other transports want this. Which I doubt.
> +};
> +
> +struct virtio_admin_cmd_legacy_notify_query_result {
> + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> +};
> +\end{lstlisting}
here I'm not going to repeat myself. same comments as other commands.
> +
> +The driver picks the suitable entry when multiple entries are supplied
> +by the device.
How does driver know how much to allocate?
I think it allocates an extra entry, inits it to 0, then after the
command checks whether it was modified. If yes makes the buffer
larger. Makes sense? Complex though ...
Can we place an upper limit on # of entries? 4 seems sufficient no?
And what is "suitable"? I am guessing it just picks the
*first* entry that it is able to utilize.
And devices list them in the order of preference.
> +
> +Refer to the specific transport section for the definition of the
> +\field{region_data}.
Transport sections are huge. Links please do not make the reader guess.
> +
> +This command is currently only defined for the PCI SR-IOV group type.
> +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +
> +If the group owner device supports legacy region access for its group member
> +devices, the device MUST set all corresponding bits for commands
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in
> +the command result
> of VIRTIO_ADMIN_CMD_LIST_QUERY in
> +\field{device_admin_cmd_opcodes}.
It's enough to just say what is supported:
A device MUST either support all of, or none of
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> +
> +The device MUST encode and decode legacy device specific registers using
> +little-endian format.
For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
the device MUST decode and encode (respectively) the value of
the \field{region_data} using the little-endian format.
> +
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the invalid offset
> +which is outside the legacy common configuration region's address range.
The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the value of the
\field{offset} and the length of the \field{region_data} does not refer
to a single field or is not completely within the virtio common
configuration structure, excluding the device-specific configuration.
> +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid offset
> +which is outside the legacy device specific region's address range.
The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of the
\field{offset} and the length of the \field{region_data} does not refer
to a single field or is not completely within the device-specific
configuration.
> +
> +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command for
> +driver notifications. If the group owner device supports driver
> +notifications region for its group member devices, the device MUST set
> +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
I don't get what does all this mean. Seems to say if you support
VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
that anyway.
Should we say:
If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
also support all of
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> +
> +When the driver accesses the legacy region of the member device using
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE, VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> +commands, device MUST function as if they are accessed by the legacy interface
> +defined by the transport of the member device.
The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
have the same effect as writing into the virtio common configuration
structure through the legacy interface.
repeat for the other commands.
Also, explain about notification: is it true that writing
at this address has the same effect as writing into Queue Notify?
> +
> +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +
> +The driver MUST encode and decode legacy device specific registers using
> +little-endian format.
> +
> +The driver SHOULD send VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a valid offset which
> +is in the legacy common configuration region address range.
> +
> +The driver SHOULD send commands VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in the legacy
> +device specific configuration region address range.
> +
> +The group member driver SHOULD use the notification region supplied by the group
> +owner device.
Use how? More specific please. Also, only if the command is supported.
Document how is notification thingy used.
> diff --git a/admin.tex b/admin.tex
> index fd3b97d..0de26a9 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -117,7 +117,17 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> \hline
> 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> \hline
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write legacy common configuration region of a member device \\
> +\hline
> +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read legacy common configuration region of a member device \\
> +\hline
> +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy device configuration region of a member device \\
> +\hline
> +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy device configuration region of a member device \\
> +\hline
> +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query notification region for a member device \\
> +\hline
> +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> \hline
> 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure) \\
> \hline
Grammar fix + drop "region":
VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy common configuration structure
> @@ -286,6 +296,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> supporting multiple group types, the list of supported commands
> might differ between different group types.
>
> +\input{admin-cmds-legacy-interface.tex}
> +
> \devicenormative{\subsubsection}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands}
>
> The device MUST validate \field{opcode}, \field{group_type} and
> diff --git a/conformance.tex b/conformance.tex
> index 01ccd69..dc00e84 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -260,6 +260,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Layout}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Legacy Interfaces: A Note on Virtqueue Endianness}
> \item Section \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Message Framing / Legacy Interface: Message Framing}
> +\item Section \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> +\item Section \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Legacy Interface}
> \item Section \ref{sec:General Initialization And Device Operation / Device Initialization / Legacy Interface: Device Initialization}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Discovery / Legacy Interfaces: A Note on PCI Device Discovery}
> \item Section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout}
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
@ 2023-07-05 11:09 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 11:09 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:20AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member access the legacy registers of the member VFs.
> This short series introduces legacy region access commands by the group owner
> device for its member devices.
> Currently it is applicable to the PCI PF and VF devices. If in future any
> SIOV devices to support legacy registers, they can be easily supported using
> same commands by using the group member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
OK I sent a bunch of comments. Pls try to address, unfortunately I
didn't go in depth over the last patch, but I feel it should be
squashed into the generic legacy command sections avoiding
need for indirection, so rewriting it now wouldn't be
efficient.
Thanks!
> Patch summary:
> --------------
> patch-1 split rows of admin opcode tables by a line
> patch-2 fix section numbering
> patch-3 add generic legacy region access commands
> patch-4 add pci specific definition
>
> It uses the newly introduced administration command facility with 4 new
> commands and a new optional command to query the legacy notification region.
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Legacy region Driver notification
> access |
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Alternatives considered:
> ========================
> 1. Exposing BAR0 as MMIO BAR that follows legacy registers template
> Pros:
> a. Kind of works with legacy drivers as some of them have used API
> which is agnostic to MMIO vs IOBAR.
> b. Does not require hypervisor intervantion
> Cons:
> a. Device reset is extremely hard to implement in device at scale as
> driver does not wait for device reset completion
> b. Device register width related problems persist that hypervisor if
> wishes, it cannot be fixed.
>
> 2. Accessing VF registers by tunneling it through new legacy PCI capability
> Pros:
> a. Self contained, but cannot work with future PCI SIOV devices
> Cons:
> a. Equally slow as AQ access
> b. Still requires new capability for notification access
> c. Requires hardware to build low level registers access which is not worth
> for long term future
>
> 3. Accessing VF notification region using new PF BAR
> Cons:
> a. Requires hardware to build new PCI steering logic per PF to forward
> notification from the PF to VF, requires double the amount of logic
> compared to today
> b. Requires very large additional PF BAR whose size must be max_Vfs * BAR size.
>
> 4. Trapping CVQ, configuration region, LEGACY_HDR
> Cons:
> a. This does not fullfil the very basic requirement to not trap the
> 1.x objects (configuration registers, vqs)
> b. Requires feature negotiations mediation in hypervisor software
> c. Requires constant device type specific knowledge in hypervisor driver
> (Does not scale for 30+ device types)
>
> 4. F_LEACY_HDR, F_WRITE_MAC
> Cons:
> a. Requires device support to have read/write mac address which is
> hard to implement on every member device.
> b. such functionality is duplicate of existing cvq per device.
> c. config space is only for the initialization specific purpose.
> d. Requires mediation of 1.x objects, which is not good design.
> e. Solves only for the net device.
> Pros:
> a. May work for nested env
>
> conclusion for picking AQ approach:
> ==================================
> 1. Overall AQ based access is simpler to implement with combination of
> best from software and device so that legacy registers do not get baked
> in the device hardware
> 2. AQ allows hypervisor software to intercept legacy registers and make
> corrections if needed
> 3. Provides trade-off between performance, device complexity vs spec,
> while still maintaining passthrough mode for the VFs with minimal
> hypervisor intercepts only for legacy registers access
> 4. AQ mechanism is designed for accessing other member devices registers
> as noted in AQ submission, it utilizes the existing infrastructure over
> other alternatives.
> 5. Uses existing driver notification region similar to legacy notification
> saves hardware resources
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v8->v9:
> - add missing articles in notify query command
> - replaced 'this notification' with 'such a notification'
> - addressed below comments from Michael
> - dropped 'Region' from the commands
> - added 7 reserved pad bytes in config write commands
> - rewrote from 'use following structure' to 'field' has the following
> struct..
> - dropped mentioning to follow struct virtio_admin_cmd.
> - added note about command limited to only sriov group type for now
> - rewrote the description little differently
> v7->v8:
> - remove empty line at the end of file
> - removed white space at the end
> - addressed comments from Michael add link to pci
> - renamed region to region_data
> - made region_data width to be 16 bytes to cover for 8 bytes offset
> - moved generic notification region related normative from pci to
> generic section
> - addressed comments from Michael
> - made bar offset 64-bit
> - prefix legacy specific structure with _legacy
> - moved generic normative from pci to generic section
> - added link to virtio pci capabilities when referring to bar 0
> - remove 'should' from generic description
> v6->v7:
> - addressed several comments from Michael
> - use AQ command to query legacy notify region, dropped pci capability
> modifications
> - moved most part of the text to the generic admin command section
> - replace administrative to administration
> - replace admin vq citation to admin commands
> - added normatives for device and driver side
> - made BAR0 to be not used at all when supporting legacy interface
> - added normative around BAR0 and SR-IOV extended capability
> - grammar corrections
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> - added text for the PCI capability for the group member device
> v4->v5:
> - split pci transport and generic command section to new patch
> - removed multiple references to the VF
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> v3->v4:
> - moved noted to the conformance section details in next patch
> - removed queue notify address query AQ command on Michael's suggestion,
> though it is fine. Instead replaced with extending virtio_pci_notify_cap
> to indicate that legacy queue notifications can be done on the
> notification location
> - fixed spelling errors
> - replaced administrative virtqueue to administration virtqueue
> - moved legacy interface normative references to legacy conformance
> section
> v2->v3:
> - added new patch to split raws of admin vq opcode table
> - adddressed Jason and Michael's comment to split single register
> access command to common config and device specific commands.
> - dropped the suggetion to introduce enable/disable command as
> admin command cap bit already covers it.
> - added other alternative design considered and discussed in detail in v0, v1 and v2
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
> Parav Pandit (4):
> admin: Split opcode table rows with a line
> admin: Fix section numbering
> admin: Add group member legacy register access commands
> transport-pci: Introduce group legacy group member config region
> access
>
> admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
> admin.tex | 20 +++-
> conformance.tex | 3 +
> transport-pci-legacy-regs.tex | 41 +++++++
> transport-pci.tex | 2 +
> 5 files changed, 268 insertions(+), 3 deletions(-)
> create mode 100644 admin-cmds-legacy-interface.tex
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-05 11:09 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-05 11:09 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, yishaih, maorg, shahafs
On Wed, Jul 05, 2023 at 02:37:20AM +0300, Parav Pandit wrote:
> This short series introduces legacy registers access commands for the owner
> group member access the legacy registers of the member VFs.
> This short series introduces legacy region access commands by the group owner
> device for its member devices.
> Currently it is applicable to the PCI PF and VF devices. If in future any
> SIOV devices to support legacy registers, they can be easily supported using
> same commands by using the group member identifiers of the future SIOV devices.
>
> More details as overview, motivation, use case are further described
> below.
OK I sent a bunch of comments. Pls try to address, unfortunately I
didn't go in depth over the last patch, but I feel it should be
squashed into the generic legacy command sections avoiding
need for indirection, so rewriting it now wouldn't be
efficient.
Thanks!
> Patch summary:
> --------------
> patch-1 split rows of admin opcode tables by a line
> patch-2 fix section numbering
> patch-3 add generic legacy region access commands
> patch-4 add pci specific definition
>
> It uses the newly introduced administration command facility with 4 new
> commands and a new optional command to query the legacy notification region.
>
> Usecase:
> --------
> 1. A hypervisor/system needs to provide transitional
> virtio devices to the guest VM at scale of thousands,
> typically, one to eight devices per VM.
>
> 2. A hypervisor/system needs to provide such devices using a
> vendor agnostic driver in the hypervisor system.
>
> 3. A hypervisor system prefers to have single stack regardless of
> virtio device type (net/blk) and be future compatible with a
> single vfio stack using SR-IOV or other scalable device
> virtualization technology to map PCI devices to the guest VM.
> (as transitional or otherwise)
>
> Motivation/Background:
> ----------------------
> The existing virtio transitional PCI device is missing support for
> PCI SR-IOV based devices. Currently it does not work beyond
> PCI PF, or as software emulated device in reality. Currently it
> has below cited system level limitations:
>
> [a] PCIe spec citation:
> VFs do not support I/O Space and thus VF BARs shall not indicate I/O Space.
>
> [b] cpu arch citiation:
> Intel 64 and IA-32 Architectures Software Developer’s Manual:
> The processor’s I/O address space is separate and distinct from
> the physical-memory address space. The I/O address space consists
> of 64K individually addressable 8-bit I/O ports, numbered 0 through FFFFH.
>
> [c] PCIe spec citation:
> If a bridge implements an I/O address range,...I/O address range will be
> aligned to a 4 KB boundary.
>
> Overview:
> ---------
> Above usecase requirements is solved by PCI PF group owner accessing
> its group member PCI VFs legacy registers using an admin virtqueue of
> the group owner PCI PF.
>
> Two new admin virtqueue commands are added which read/write PCI VF
> registers.
>
> Software usage example:
> -----------------------
> One way to use and map to the guest VM is by using vfio driver
> framework in Linux kernel.
>
> +----------------------+
> |pci_dev_id = 0x100X |
> +---------------|pci_rev_id = 0x0 |-----+
> |vfio device |BAR0 = I/O region | |
> | |Other attributes | |
> | +----------------------+ |
> | |
> + +--------------+ +-----------------+ |
> | |I/O BAR to AQ | | Other vfio | |
> | |rd/wr mapper | | functionalities | |
> | +--------------+ +-----------------+ |
> | |
> +------+-------------------------+-----------+
> | |
> Legacy region Driver notification
> access |
> | |
> +----+------------+ +----+------------+
> | +-----+ | | PCI VF device A |
> | | AQ |-------------+---->+-------------+ |
> | +-----+ | | | | legacy regs | |
> | PCI PF device | | | +-------------+ |
> +-----------------+ | +-----------------+
> |
> | +----+------------+
> | | PCI VF device N |
> +---->+-------------+ |
> | | legacy regs | |
> | +-------------+ |
> +-----------------+
>
> 2. Virtio pci driver to bind to the listed device id and
> use it in the host.
>
> 3. Use it in a light weight hypervisor to run bare-metal OS.
>
> Please review.
>
> Alternatives considered:
> ========================
> 1. Exposing BAR0 as MMIO BAR that follows legacy registers template
> Pros:
> a. Kind of works with legacy drivers as some of them have used API
> which is agnostic to MMIO vs IOBAR.
> b. Does not require hypervisor intervantion
> Cons:
> a. Device reset is extremely hard to implement in device at scale as
> driver does not wait for device reset completion
> b. Device register width related problems persist that hypervisor if
> wishes, it cannot be fixed.
>
> 2. Accessing VF registers by tunneling it through new legacy PCI capability
> Pros:
> a. Self contained, but cannot work with future PCI SIOV devices
> Cons:
> a. Equally slow as AQ access
> b. Still requires new capability for notification access
> c. Requires hardware to build low level registers access which is not worth
> for long term future
>
> 3. Accessing VF notification region using new PF BAR
> Cons:
> a. Requires hardware to build new PCI steering logic per PF to forward
> notification from the PF to VF, requires double the amount of logic
> compared to today
> b. Requires very large additional PF BAR whose size must be max_Vfs * BAR size.
>
> 4. Trapping CVQ, configuration region, LEGACY_HDR
> Cons:
> a. This does not fullfil the very basic requirement to not trap the
> 1.x objects (configuration registers, vqs)
> b. Requires feature negotiations mediation in hypervisor software
> c. Requires constant device type specific knowledge in hypervisor driver
> (Does not scale for 30+ device types)
>
> 4. F_LEACY_HDR, F_WRITE_MAC
> Cons:
> a. Requires device support to have read/write mac address which is
> hard to implement on every member device.
> b. such functionality is duplicate of existing cvq per device.
> c. config space is only for the initialization specific purpose.
> d. Requires mediation of 1.x objects, which is not good design.
> e. Solves only for the net device.
> Pros:
> a. May work for nested env
>
> conclusion for picking AQ approach:
> ==================================
> 1. Overall AQ based access is simpler to implement with combination of
> best from software and device so that legacy registers do not get baked
> in the device hardware
> 2. AQ allows hypervisor software to intercept legacy registers and make
> corrections if needed
> 3. Provides trade-off between performance, device complexity vs spec,
> while still maintaining passthrough mode for the VFs with minimal
> hypervisor intercepts only for legacy registers access
> 4. AQ mechanism is designed for accessing other member devices registers
> as noted in AQ submission, it utilizes the existing infrastructure over
> other alternatives.
> 5. Uses existing driver notification region similar to legacy notification
> saves hardware resources
>
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/167
> Signed-off-by: Parav Pandit <parav@nvidia.com>
>
> ---
> changelog:
> v8->v9:
> - add missing articles in notify query command
> - replaced 'this notification' with 'such a notification'
> - addressed below comments from Michael
> - dropped 'Region' from the commands
> - added 7 reserved pad bytes in config write commands
> - rewrote from 'use following structure' to 'field' has the following
> struct..
> - dropped mentioning to follow struct virtio_admin_cmd.
> - added note about command limited to only sriov group type for now
> - rewrote the description little differently
> v7->v8:
> - remove empty line at the end of file
> - removed white space at the end
> - addressed comments from Michael add link to pci
> - renamed region to region_data
> - made region_data width to be 16 bytes to cover for 8 bytes offset
> - moved generic notification region related normative from pci to
> generic section
> - addressed comments from Michael
> - made bar offset 64-bit
> - prefix legacy specific structure with _legacy
> - moved generic normative from pci to generic section
> - added link to virtio pci capabilities when referring to bar 0
> - remove 'should' from generic description
> v6->v7:
> - addressed several comments from Michael
> - use AQ command to query legacy notify region, dropped pci capability
> modifications
> - moved most part of the text to the generic admin command section
> - replace administrative to administration
> - replace admin vq citation to admin commands
> - added normatives for device and driver side
> - made BAR0 to be not used at all when supporting legacy interface
> - added normative around BAR0 and SR-IOV extended capability
> - grammar corrections
> v5->v6:
> - fixed previous missed abbreviation of LCC and LD
> - added text for the PCI capability for the group member device
> v4->v5:
> - split pci transport and generic command section to new patch
> - removed multiple references to the VF
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - split from pci transport specific patch
> - split conformance to transport and generic sections
> - written the description of the command as generic with member
> and group device terminology
> - reflected many section names to remove VF
> - rename fields from register to region
> - avoided abbreviation for legacy, device and config
> v3->v4:
> - moved noted to the conformance section details in next patch
> - removed queue notify address query AQ command on Michael's suggestion,
> though it is fine. Instead replaced with extending virtio_pci_notify_cap
> to indicate that legacy queue notifications can be done on the
> notification location
> - fixed spelling errors
> - replaced administrative virtqueue to administration virtqueue
> - moved legacy interface normative references to legacy conformance
> section
> v2->v3:
> - added new patch to split raws of admin vq opcode table
> - adddressed Jason and Michael's comment to split single register
> access command to common config and device specific commands.
> - dropped the suggetion to introduce enable/disable command as
> admin command cap bit already covers it.
> - added other alternative design considered and discussed in detail in v0, v1 and v2
> v1->v2:
> - addressed comments from Michael
> - added theory of operation
> - grammar corrections
> - removed group fields description from individual commands as
> it is already present in generic section
> - added endianness normative for legacy device registers region
> - renamed the file to drop vf and add legacy prefix
> - added overview in commit log
> - renamed subsection to reflect command
> v0->v1:
> - addressed comments, suggesetions and ideas from Michael Tsirkin and Jason Wang
> - far more simpler design than MMR access
> - removed complexities of MMR device ids
> - removed complexities of MMR registers and extended capabilities
> - dropped adding new extended capabilities because if if they are
> added, a pci device still needs to have existing capabilities
> in the legacy configuration space and hypervisor driver do not
> need to access them
>
> Parav Pandit (4):
> admin: Split opcode table rows with a line
> admin: Fix section numbering
> admin: Add group member legacy register access commands
> transport-pci: Introduce group legacy group member config region
> access
>
> admin-cmds-legacy-interface.tex | 205 ++++++++++++++++++++++++++++++++
> admin.tex | 20 +++-
> conformance.tex | 3 +
> transport-pci-legacy-regs.tex | 41 +++++++
> transport-pci.tex | 2 +
> 5 files changed, 268 insertions(+), 3 deletions(-)
> create mode 100644 admin-cmds-legacy-interface.tex
> create mode 100644 transport-pci-legacy-regs.tex
>
> --
> 2.26.2
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-05 11:09 ` Michael S. Tsirkin
@ 2023-07-05 13:01 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 13:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:09 AM
>
> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> depth over the last patch, but I feel it should be squashed into the generic
> legacy command sections avoiding need for indirection, so rewriting it now
> wouldn't be efficient.
I relooked at well yday.
Transport has very small section.
This aligns to how the existing spec is written, for example vq notify data is explained in generic way and defined in pci way in PCI chapter.
The indirection is fine and also self-contained similarly here.
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-05 13:01 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 13:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:09 AM
>
> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> depth over the last patch, but I feel it should be squashed into the generic
> legacy command sections avoiding need for indirection, so rewriting it now
> wouldn't be efficient.
I relooked at well yday.
Transport has very small section.
This aligns to how the existing spec is written, for example vq notify data is explained in generic way and defined in pci way in PCI chapter.
The indirection is fine and also self-contained similarly here.
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-05 13:01 ` Parav Pandit
@ 2023-07-05 13:42 ` Cornelia Huck
-1 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 13:42 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-comment, david.edmondson, virtio-dev, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Wednesday, July 5, 2023 7:09 AM
>>
>> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
>> depth over the last patch, but I feel it should be squashed into the generic
>> legacy command sections avoiding need for indirection, so rewriting it now
>> wouldn't be efficient.
>
> I relooked at well yday.
> Transport has very small section.
> This aligns to how the existing spec is written, for example vq notify data is explained in generic way and defined in pci way in PCI chapter.
> The indirection is fine and also self-contained similarly here.
Would it make sense to do this the other way around (i.e. use something
that is PCI-only from the start?)
IIUC, the "legacy" problem as it stands is something specific to PCI,
with MMIO using two distinct approaches for pre-1.0 and 1.0+, and CCW
simply using different commands (and no need for any kind of cross
access.) Not 100% sure about MMIO, but I don't think we need a version
of this for CCW.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-05 13:42 ` Cornelia Huck
0 siblings, 0 replies; 48+ messages in thread
From: Cornelia Huck @ 2023-07-05 13:42 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-comment, david.edmondson, virtio-dev, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05 2023, Parav Pandit <parav@nvidia.com> wrote:
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Wednesday, July 5, 2023 7:09 AM
>>
>> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
>> depth over the last patch, but I feel it should be squashed into the generic
>> legacy command sections avoiding need for indirection, so rewriting it now
>> wouldn't be efficient.
>
> I relooked at well yday.
> Transport has very small section.
> This aligns to how the existing spec is written, for example vq notify data is explained in generic way and defined in pci way in PCI chapter.
> The indirection is fine and also self-contained similarly here.
Would it make sense to do this the other way around (i.e. use something
that is PCI-only from the start?)
IIUC, the "legacy" problem as it stands is something specific to PCI,
with MMIO using two distinct approaches for pre-1.0 and 1.0+, and CCW
simply using different commands (and no need for any kind of cross
access.) Not 100% sure about MMIO, but I don't think we need a version
of this for CCW.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-05 13:42 ` Cornelia Huck
@ 2023-07-05 14:37 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 14:37 UTC (permalink / raw)
To: Cornelia Huck, Michael S. Tsirkin
Cc: virtio-comment, david.edmondson, virtio-dev, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 9:43 AM
>
> >> From: Michael S. Tsirkin <mst@redhat.com>
> >> Sent: Wednesday, July 5, 2023 7:09 AM
> >>
> >> OK I sent a bunch of comments. Pls try to address, unfortunately I
> >> didn't go in depth over the last patch, but I feel it should be
> >> squashed into the generic legacy command sections avoiding need for
> >> indirection, so rewriting it now wouldn't be efficient.
> >
> > I relooked at well yday.
> > Transport has very small section.
> > This aligns to how the existing spec is written, for example vq notify data is
> explained in generic way and defined in pci way in PCI chapter.
> > The indirection is fine and also self-contained similarly here.
>
> Would it make sense to do this the other way around (i.e. use something that is
> PCI-only from the start?)
>
> IIUC, the "legacy" problem as it stands is something specific to PCI, with MMIO
> using two distinct approaches for pre-1.0 and 1.0+, and CCW simply using
> different commands (and no need for any kind of cross
> access.) Not 100% sure about MMIO, but I don't think we need a version of this
> for CCW.
V4 or v5 was having everything in PCI and Michael asked to write as generic as possible.
so I moved large part as generic.
And kept pci part in pci.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-05 14:37 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 14:37 UTC (permalink / raw)
To: Cornelia Huck, Michael S. Tsirkin
Cc: virtio-comment, david.edmondson, virtio-dev, sburla, jasowang,
Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 9:43 AM
>
> >> From: Michael S. Tsirkin <mst@redhat.com>
> >> Sent: Wednesday, July 5, 2023 7:09 AM
> >>
> >> OK I sent a bunch of comments. Pls try to address, unfortunately I
> >> didn't go in depth over the last patch, but I feel it should be
> >> squashed into the generic legacy command sections avoiding need for
> >> indirection, so rewriting it now wouldn't be efficient.
> >
> > I relooked at well yday.
> > Transport has very small section.
> > This aligns to how the existing spec is written, for example vq notify data is
> explained in generic way and defined in pci way in PCI chapter.
> > The indirection is fine and also self-contained similarly here.
>
> Would it make sense to do this the other way around (i.e. use something that is
> PCI-only from the start?)
>
> IIUC, the "legacy" problem as it stands is something specific to PCI, with MMIO
> using two distinct approaches for pre-1.0 and 1.0+, and CCW simply using
> different commands (and no need for any kind of cross
> access.) Not 100% sure about MMIO, but I don't think we need a version of this
> for CCW.
V4 or v5 was having everything in PCI and Michael asked to write as generic as possible.
so I moved large part as generic.
And kept pci part in pci.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-dev] Re: [PATCH v9 2/4] admin: Fix section numbering
2023-07-05 10:09 ` [virtio-comment] " Cornelia Huck
@ 2023-07-05 16:29 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:29 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
> Behalf Of Cornelia Huck
> Sent: Wednesday, July 5, 2023 6:10 AM
>
> Same thing applies to the normative statements for the admin vq (in the same
> file) -- do you want to fix that as well?
Yes, let me fix it.
I will do this as pre-work and this patch can depend on it.
So that it can be merged quickly and I don't need to run around with more side patches for this main series.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] RE: [virtio-dev] Re: [PATCH v9 2/4] admin: Fix section numbering
@ 2023-07-05 16:29 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:29 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: virtio-dev@lists.oasis-open.org <virtio-dev@lists.oasis-open.org> On
> Behalf Of Cornelia Huck
> Sent: Wednesday, July 5, 2023 6:10 AM
>
> Same thing applies to the normative statements for the admin vq (in the same
> file) -- do you want to fix that as well?
Yes, let me fix it.
I will do this as pre-work and this patch can depend on it.
So that it can be merged quickly and I don't need to run around with more side patches for this main series.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-05 10:24 ` [virtio-comment] " Cornelia Huck
@ 2023-07-05 16:34 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:34 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 6:24 AM
> > +
> > +In some systems, there is a need to support utilizing the legacy
> > +driver with the device that do not directly support the legacy
> > +interface. In such scenarios,
>
> Maybe better:
>
> "In some systems, there is a need to support utilizing legacy drivers with devices
> that do not directly support the legacy interface."
>
> ?
>
Can you please explain how plural is better if it is?
I see this a lot intermixed in the spec.
Why cannot we describe things using single object "the driver" and "the device"?
I prefer to describe things without plurality as done in many parts of the spec.
Please let me know.
> > +a group owner device can provide the legacy interface functionality
> > +for the group member devices. The driver of an owner device can then
> > +access the legacy interface of a member device on behalf of the legacy
> member device driver.
>
> (...)
>
> > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Legacy Driver Notifications Query}
> > +
> > +Even though the driver notifications can be communicated through the
> > +administration command, if the group owner device or group member
> > +device supports such notifications using a memory-mapped operation or
> > +I/O operation, they are sent to the device by accessing such a
> > +notification region using a memory or an I/O operation.
>
> I find this sentence confusing, probably because it is too complex... what about
>
> "If the group owner device or the group member device support driver
> notifications via a memory-mapped operation or I/O operation, these
> notifications are sent to the device via accessing such a notification region using
> a memory or I/O operation {in addition to|instead of} communicating the
> notifications through the admistration command."
>
With instead of, it reads simpler. Will change it.
> (I'm not sure whether it is either or both.)
>
> > +
> > +A group owner device optionally support querying driver notifications
> > +region
>
> "can optionally support" ?
>
Ack.
> > +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > +This command does not use \field{command_specific_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
>
> "the command"
>
Ack
> > +uses following structure:
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] RE: [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-05 16:34 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:34 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 6:24 AM
> > +
> > +In some systems, there is a need to support utilizing the legacy
> > +driver with the device that do not directly support the legacy
> > +interface. In such scenarios,
>
> Maybe better:
>
> "In some systems, there is a need to support utilizing legacy drivers with devices
> that do not directly support the legacy interface."
>
> ?
>
Can you please explain how plural is better if it is?
I see this a lot intermixed in the spec.
Why cannot we describe things using single object "the driver" and "the device"?
I prefer to describe things without plurality as done in many parts of the spec.
Please let me know.
> > +a group owner device can provide the legacy interface functionality
> > +for the group member devices. The driver of an owner device can then
> > +access the legacy interface of a member device on behalf of the legacy
> member device driver.
>
> (...)
>
> > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Legacy Driver Notifications Query}
> > +
> > +Even though the driver notifications can be communicated through the
> > +administration command, if the group owner device or group member
> > +device supports such notifications using a memory-mapped operation or
> > +I/O operation, they are sent to the device by accessing such a
> > +notification region using a memory or an I/O operation.
>
> I find this sentence confusing, probably because it is too complex... what about
>
> "If the group owner device or the group member device support driver
> notifications via a memory-mapped operation or I/O operation, these
> notifications are sent to the device via accessing such a notification region using
> a memory or I/O operation {in addition to|instead of} communicating the
> notifications through the admistration command."
>
With instead of, it reads simpler. Will change it.
> (I'm not sure whether it is either or both.)
>
> > +
> > +A group owner device optionally support querying driver notifications
> > +region
>
> "can optionally support" ?
>
Ack.
> > +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > +This command does not use \field{command_specific_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
>
> "the command"
>
Ack
> > +uses following structure:
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
2023-07-05 10:29 ` [virtio-comment] " Cornelia Huck
@ 2023-07-05 16:36 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:36 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 6:29 AM
>
> > diff --git a/admin-cmds-legacy-interface.tex
> > b/admin-cmds-legacy-interface.tex index 09001d5..571b256 100644
> > --- a/admin-cmds-legacy-interface.tex
> > +++ b/admin-cmds-legacy-interface.tex
> > @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic
> > Facilities of a Virtio Device by the device.
> >
> > Refer to the specific transport section for the definition of the
> > -\field{region_data}.
> > +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio
> Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member
> Device Configuration Region Access}.
>
> "For the PCI transport, refer to..."
>
Ack.
> >
> > This command is currently only defined for the PCI SR-IOV group type.
> >
>
> (...)
>
> > diff --git a/transport-pci-legacy-regs.tex
> > b/transport-pci-legacy-regs.tex new file mode 100644 index
> > 0000000..e4f70c9
> > --- /dev/null
> > +++ b/transport-pci-legacy-regs.tex
> > @@ -0,0 +1,41 @@
> > +\subsection{Legacy Interface: Group member device Configuration
> > +Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI
> > +Bus / Legacy Interface: Group Member Device Configuration Region
> > +Access}
> > +
> > +The PCI owner device or the member device or both supports driver
> > +notifications using
>
> s/supports/support/
>
Ack
> > +a notification region defined in the \field{struct virtio_pci_notify_region}.
>
> s/in the/in/
>
Ack
> > +
> > +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> > +\field{region_data} is defined as following:
> > +
> > +\begin{lstlisting}
> > +struct virtio_pci_legacy_notify_region {
> > + u8 owner; /* When set to 1, notification region is of the owner device
> */
> > + u8 bar; /* BAR of the member or owner device */
> > + u8 padding[6];
> > + le64 offset; /* Offset within bar. */ }; \end{lstlisting}
> > +
> > +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
>
> s/hardwire/hardwires/
>
ack
> > +
> > +The group member device does not use PCI BAR0 in various Virtio PCI
> > +capabilities
>
> "in the various" caps? Or "in some of the" caps?
>
It is actually "all".
I will change it to "in all the Virtio PCI capabilities".
> > +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus /
> Virtio Structure PCI Capabilities}.
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] RE: [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access
@ 2023-07-05 16:36 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 16:36 UTC (permalink / raw)
To: Cornelia Huck, virtio-comment, mst, david.edmondson
Cc: virtio-dev, sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Cornelia Huck <cohuck@redhat.com>
> Sent: Wednesday, July 5, 2023 6:29 AM
>
> > diff --git a/admin-cmds-legacy-interface.tex
> > b/admin-cmds-legacy-interface.tex index 09001d5..571b256 100644
> > --- a/admin-cmds-legacy-interface.tex
> > +++ b/admin-cmds-legacy-interface.tex
> > @@ -150,7 +150,7 @@ \subsubsection{Legacy Interfaces}\label{sec:Basic
> > Facilities of a Virtio Device by the device.
> >
> > Refer to the specific transport section for the definition of the
> > -\field{region_data}.
> > +\field{region_data}. For PCI transport refer to section \ref{sec:Virtio
> Transport Options / Virtio Over PCI Bus / Legacy Interface: Group Member
> Device Configuration Region Access}.
>
> "For the PCI transport, refer to..."
>
Ack.
> >
> > This command is currently only defined for the PCI SR-IOV group type.
> >
>
> (...)
>
> > diff --git a/transport-pci-legacy-regs.tex
> > b/transport-pci-legacy-regs.tex new file mode 100644 index
> > 0000000..e4f70c9
> > --- /dev/null
> > +++ b/transport-pci-legacy-regs.tex
> > @@ -0,0 +1,41 @@
> > +\subsection{Legacy Interface: Group member device Configuration
> > +Region Access}\label{sec:Virtio Transport Options / Virtio Over PCI
> > +Bus / Legacy Interface: Group Member Device Configuration Region
> > +Access}
> > +
> > +The PCI owner device or the member device or both supports driver
> > +notifications using
>
> s/supports/support/
>
Ack
> > +a notification region defined in the \field{struct virtio_pci_notify_region}.
>
> s/in the/in/
>
Ack
> > +
> > +In \field{struct virtio_virtio_admin_cmd_legacy_notify_query_entry},
> > +\field{region_data} is defined as following:
> > +
> > +\begin{lstlisting}
> > +struct virtio_pci_legacy_notify_region {
> > + u8 owner; /* When set to 1, notification region is of the owner device
> */
> > + u8 bar; /* BAR of the member or owner device */
> > + u8 padding[6];
> > + le64 offset; /* Offset within bar. */ }; \end{lstlisting}
> > +
> > +The group owner device hardwire VF BAR0 in the SR-IOV Extended capability.
>
> s/hardwire/hardwires/
>
ack
> > +
> > +The group member device does not use PCI BAR0 in various Virtio PCI
> > +capabilities
>
> "in the various" caps? Or "in some of the" caps?
>
It is actually "all".
I will change it to "in all the Virtio PCI capabilities".
> > +listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI Bus /
> Virtio Structure PCI Capabilities}.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-05 11:04 ` Michael S. Tsirkin
@ 2023-07-05 18:26 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 18:26 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:05 AM
> > +These commands are currently only defined for the PCI SR-IOV group
> > +type and
>
> We call it SR-IOV group type not PCI SR-IOV group type. Please be consistent.
>
Ack.
> > +have, generally, the same effect as member device accesses through a
> > +legacy interface listed in section \ref{sec:Virtio Transport Options / Virtio
> Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device
> Layout}.
> > +
> > +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Common Configuration Write Command}
>
> Fix label.
>
It is already under the tree of Legacy interface so dropped making it further longer.
But than I kept the Legacy in the actual name...
Will add Legacy in the label.
> > +
> > +This command has the same effect as writing into the virtio common
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> > + u8 offset; /* Starting byte offset within the common configuration
> structure to write */
> > + u8 reserved[7];
> > + u8 region_data[];
>
> I am all for something more specific than offset/data but region seems pointless.
> And why apply just to data not offset?
> How about legacy_offset/legacy_data?
>
Structure name already has legacy, so no point in repeating legacy_ prefix again in all the fields inside.
I will drop region_ and just keep offset,data.
> > +};
> > +\end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> > +The driver sets valid \field{offset} and associated
> > +\field{region_data} bytes to write to the common configuration region.
> > +
> > +This command does not use \field{command_specific_result}.
> > +
>
> Let's make this consistent with other commands, even if it means we repeat
> ourselves a bit. And length is not documented and has to be:
>
Ah yes, I missed the length documentation.
Will add it.
Not sure what part you want to make consistent with other commands?
Do you mean to repeat ourselves for the opcode number? If so, than we should be avoiding it.
In fact we should remove that repetition in other commands too. We have table to describe it clearly.
>
> >>>
>
> For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> \field{opcode} is set to 0x2.
> The \field{group_member_id} refers to the member device to be accessed.
Yeah, there is certainly no point in repeating all of these again.
In previous patch when I repeated where it says it uses the struct virtio_admin_cmd, you asked to drop for the right reason.
But here you ask to repeat other things...
Not good.
All above is crystal clear in opcode table and in generic description already.
Reader has less text to read to understand it.
> The \field{command_specific_data} is in the format \field{struct
> virtio_admin_cmd_legacy_common_cfg_wr_data} describing the access to be
> performed.
>
> Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
> The \field{offset} refers to the offset to write within the virtio common
> configuration structure, and excluding the device-specific configuration.
"virtio common configuration structure" is self-explanatory that doesnt need to add the exclusion part.
Hence it was not added in v9 too.
> For the
> SR-IOV group type, the layout used refers to the one used for the legacy
> interface as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI
> Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout},
> except that little endian format is assumed unconditionally.
> The length of the data to write is simply the length of \field{region_data}.
>
> No length or alignment restrictions are placed on the value of the \field{offset}
> and the length of the \field{region_data}, except that the resulting access refers
> to a single field and is completely within the virtio common configuration
> structure, excluding the device-specific configuration.
>
> This command has no command specific result.
>
>
> Try to apply this to other commands please.
>
I will apply rest of the above text you wrote for this and rest of the commands.
> > +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Common Configuration Read Command}
> > +
> > +This command has the same effect as reading from the virtio common
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> > + u8 offset; /* Starting byte offset within the common configuration
> > +structure to read */ }; \end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> > +
> > +The driver sets valid \field{offset} of the region from where to read
> > +\field{region_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Device Configuration Write Command}
> > +
> > +This command has the same effect as writing into the virtio device
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> > + u8 offset; /* Starting byte offset within the device configuration
> structure to write */
> > + u8 reserved[7];
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> > +The driver sets valid \field{offset} and associated
> > +\field{region_data} bytes to the device configuration region.
> > +
> > +This command does not use \field{command_specific_result}.
> > +
> > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Device Configuration Read Command}
> > +
> > +This command has the same effect as reading from the virtio device
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> > + u8 offset; /* Starting byte offset within the device
> > +configuration structure to read */ }; \end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> > +The driver sets valid \field{offset} within the device configuration
> > +structure from where to read \field{region_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Legacy Driver Notifications Query}
>
> Drop "Query", this talks about notifications generally.
>
Ok.
> > +
> > +Even though the driver notifications can be communicated through the
> > +administration command, if the group owner device or group member
> > +device supports such notifications using a memory-mapped operation or
> > +I/O operation, they are sent to the device by accessing such a
> > +notification region using a memory or an I/O operation.
>
> > +
> > +A group owner device optionally support querying driver notifications
> > +region using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
>
> The driver of the owner device can send a driver notification to the member
> device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> with the \field{offset} matching \fielf{Queue Notify} and the \field{region_data}
> containing the virtqueue index to be notified.
>
> However, as many administration commands are used for slow path
> configuration, a separate fast path mechanism for such notifications is desired.
> For the SR-IOV group type, the optional command
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by
> returning to the driver one or more addresses which to be used to send such
> driver notifications.
>
Ok. will take this version.
>
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > +This command does not use \field{command_specific_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_notify_query_entry {
> > + u8 region_data[16];
>
> So everything is pci specific but this one we for some reason decided to keep
> generic. I'd just make it pci specific, keep things simple.
> We'll generalize if and when other transports want this. Which I doubt.
>
Other is kept generic.
The whole thing was documented as PCI first and ask was to make it generic for the description.
And now again you suggest making it pci specific.
What changed from PCI->GEN->PCI?
Why the whole text should reside in generic file if you want to make it PCI specific, like before?
>
> > +};
> > +
> > +struct virtio_admin_cmd_legacy_notify_query_result {
> > + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> > +}; \end{lstlisting}
>
> here I'm not going to repeat myself. same comments as other commands.
>
> > +
> > +The driver picks the suitable entry when multiple entries are
> > +supplied by the device.
>
> How does driver know how much to allocate?
> I think it allocates an extra entry, inits it to 0, then after the command checks
> whether it was modified. If yes makes the buffer larger. Makes sense? Complex
> though ...
> Can we place an upper limit on # of entries? 4 seems sufficient no?
> And what is "suitable"? I am guessing it just picks the
> *first* entry that it is able to utilize.
> And devices list them in the order of preference.
>
Driver can just allocate some random for 1, 8, 16 entries based on how it wants to do.
Suitable is whatever driver can decide, first or last or prefer PF over VF, driver's choice....
I don’t know why to complicate this a lot when most devices that I know of will do 1 entry.
May be 2nd for the PF.
And may be device can overdo it by adding one more for some large IOBAR in theory on the PF.
4 seems some arbitrary number.
> > +
> > +Refer to the specific transport section for the definition of the
> > +\field{region_data}.
>
> Transport sections are huge. Links please do not make the reader guess.
>
> > +
> > +This command is currently only defined for the PCI SR-IOV group type.
> > +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a
> > +Virtio Device / Device groups / Group administration commands /
> > +Legacy Interface}
> > +
> > +If the group owner device supports legacy region access for its group
> > +member devices, the device MUST set all corresponding bits for
> > +commands VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in the command result
> > of VIRTIO_ADMIN_CMD_LIST_QUERY in
> > +\field{device_admin_cmd_opcodes}.
>
>
> It's enough to just say what is supported:
>
> A device MUST either support all of, or none of
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
>
Ok.
> > +
> > +The device MUST encode and decode legacy device specific registers using
> > +little-endian format.
>
> For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> the device MUST decode and encode (respectively) the value of
> the \field{region_data} using the little-endian format.
>
Ack.
>
>
> > +
> > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> and
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the
> invalid offset
> > +which is outside the legacy common configuration region's address range.
>
> The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> and
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the
> value of the
> \field{offset} and the length of the \field{region_data} does not refer
> to a single field or is not completely within the virtio common
> configuration structure, excluding the device-specific configuration.
>
This is an additional limitation being added, but fine, it doesnt hurt anyone.
>
> > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid
> offset
> > +which is outside the legacy device specific region's address range.
>
> The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of
> the
> \field{offset} and the length of the \field{region_data} does not refer
> to a single field or is not completely within the device-specific
> configuration.
>
Same as above.
>
>
> > +
> > +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> command for
> > +driver notifications. If the group owner device supports driver
> > +notifications region for its group member devices, the device MUST set
> > +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> > +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
>
> I don't get what does all this mean. Seems to say if you support
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
> that anyway.
>
> Should we say:
>
> If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
> also support all of
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
>
Yes, you captured it better. Thanks.
>
> > +
> > +When the driver accesses the legacy region of the member device using
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> > +commands, device MUST function as if they are accessed by the legacy
> interface
> > +defined by the transport of the member device.
>
> The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
> have the same effect as writing into the virtio common configuration
> structure through the legacy interface.
>
ok
> repeat for the other commands.
>
o.k.
>
> Also, explain about notification: is it true that writing
> at this address has the same effect as writing into Queue Notify?
>
Yes, will add.
>
> > +
> > +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> Device / Device groups / Group administration commands / Legacy Interface}
> > +
> > +The driver MUST encode and decode legacy device specific registers using
> > +little-endian format.
> > +
> > +The driver SHOULD send
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a
> valid offset which
> > +is in the legacy common configuration region address range.
> > +
> > +The driver SHOULD send commands
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in
> the legacy
> > +device specific configuration region address range.
> > +
> > +The group member driver SHOULD use the notification region supplied by
> the group
> > +owner device.
>
> Use how? More specific please. Also, only if the command is supported.
>
>
How to use is already part of the existing spec.
I will add a line to refer to the driver notification spec section.
Ack will add about 'if command supported'.
> Document how is notification thingy used.
>
>
>
> > diff --git a/admin.tex b/admin.tex
> > index fd3b97d..0de26a9 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -117,7 +117,17 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > \hline
> > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> > \hline
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write
> legacy common configuration region of a member device \\
> > +\hline
> > +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read
> legacy common configuration region of a member device \\
> > +\hline
> > +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy
> device configuration region of a member device \\
> > +\hline
> > +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy
> device configuration region of a member device \\
> > +\hline
> > +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query
> notification region for a member device \\
> > +\hline
> > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > \hline
> > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure) \\
> > \hline
>
>
> Grammar fix + drop "region":
>
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy
> common configuration structure
>
Ack.
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-05 18:26 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 18:26 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:05 AM
> > +These commands are currently only defined for the PCI SR-IOV group
> > +type and
>
> We call it SR-IOV group type not PCI SR-IOV group type. Please be consistent.
>
Ack.
> > +have, generally, the same effect as member device accesses through a
> > +legacy interface listed in section \ref{sec:Virtio Transport Options / Virtio
> Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device
> Layout}.
> > +
> > +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Common Configuration Write Command}
>
> Fix label.
>
It is already under the tree of Legacy interface so dropped making it further longer.
But than I kept the Legacy in the actual name...
Will add Legacy in the label.
> > +
> > +This command has the same effect as writing into the virtio common
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> > + u8 offset; /* Starting byte offset within the common configuration
> structure to write */
> > + u8 reserved[7];
> > + u8 region_data[];
>
> I am all for something more specific than offset/data but region seems pointless.
> And why apply just to data not offset?
> How about legacy_offset/legacy_data?
>
Structure name already has legacy, so no point in repeating legacy_ prefix again in all the fields inside.
I will drop region_ and just keep offset,data.
> > +};
> > +\end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> > +The driver sets valid \field{offset} and associated
> > +\field{region_data} bytes to write to the common configuration region.
> > +
> > +This command does not use \field{command_specific_result}.
> > +
>
> Let's make this consistent with other commands, even if it means we repeat
> ourselves a bit. And length is not documented and has to be:
>
Ah yes, I missed the length documentation.
Will add it.
Not sure what part you want to make consistent with other commands?
Do you mean to repeat ourselves for the opcode number? If so, than we should be avoiding it.
In fact we should remove that repetition in other commands too. We have table to describe it clearly.
>
> >>>
>
> For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> \field{opcode} is set to 0x2.
> The \field{group_member_id} refers to the member device to be accessed.
Yeah, there is certainly no point in repeating all of these again.
In previous patch when I repeated where it says it uses the struct virtio_admin_cmd, you asked to drop for the right reason.
But here you ask to repeat other things...
Not good.
All above is crystal clear in opcode table and in generic description already.
Reader has less text to read to understand it.
> The \field{command_specific_data} is in the format \field{struct
> virtio_admin_cmd_legacy_common_cfg_wr_data} describing the access to be
> performed.
>
> Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
> The \field{offset} refers to the offset to write within the virtio common
> configuration structure, and excluding the device-specific configuration.
"virtio common configuration structure" is self-explanatory that doesnt need to add the exclusion part.
Hence it was not added in v9 too.
> For the
> SR-IOV group type, the layout used refers to the one used for the legacy
> interface as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI
> Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout},
> except that little endian format is assumed unconditionally.
> The length of the data to write is simply the length of \field{region_data}.
>
> No length or alignment restrictions are placed on the value of the \field{offset}
> and the length of the \field{region_data}, except that the resulting access refers
> to a single field and is completely within the virtio common configuration
> structure, excluding the device-specific configuration.
>
> This command has no command specific result.
>
>
> Try to apply this to other commands please.
>
I will apply rest of the above text you wrote for this and rest of the commands.
> > +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Common Configuration Read Command}
> > +
> > +This command has the same effect as reading from the virtio common
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> > + u8 offset; /* Starting byte offset within the common configuration
> > +structure to read */ }; \end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> > +
> > +The driver sets valid \field{offset} of the region from where to read
> > +\field{region_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Device Configuration Write Command}
> > +
> > +This command has the same effect as writing into the virtio device
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> > + u8 offset; /* Starting byte offset within the device configuration
> structure to write */
> > + u8 reserved[7];
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> > +The driver sets valid \field{offset} and associated
> > +\field{region_data} bytes to the device configuration region.
> > +
> > +This command does not use \field{command_specific_result}.
> > +
> > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Device Configuration Read Command}
> > +
> > +This command has the same effect as reading from the virtio device
> > +configuration structure through the legacy interface.
> > +\field{command_specific_data} has the following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> > + u8 offset; /* Starting byte offset within the device
> > +configuration structure to read */ }; \end{lstlisting}
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> > +The driver sets valid \field{offset} within the device configuration
> > +structure from where to read \field{region_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> > + u8 region_data[];
> > +};
> > +\end{lstlisting}
> > +
> > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Legacy Interface / Legacy Driver Notifications Query}
>
> Drop "Query", this talks about notifications generally.
>
Ok.
> > +
> > +Even though the driver notifications can be communicated through the
> > +administration command, if the group owner device or group member
> > +device supports such notifications using a memory-mapped operation or
> > +I/O operation, they are sent to the device by accessing such a
> > +notification region using a memory or an I/O operation.
>
> > +
> > +A group owner device optionally support querying driver notifications
> > +region using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
>
> The driver of the owner device can send a driver notification to the member
> device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> with the \field{offset} matching \fielf{Queue Notify} and the \field{region_data}
> containing the virtqueue index to be notified.
>
> However, as many administration commands are used for slow path
> configuration, a separate fast path mechanism for such notifications is desired.
> For the SR-IOV group type, the optional command
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by
> returning to the driver one or more addresses which to be used to send such
> driver notifications.
>
Ok. will take this version.
>
> > +
> > +The driver sets command \field{opcode} to
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > +This command does not use \field{command_specific_data}.
> > +
> > +When command completes successfully, \field{command_specific_result}
> > +uses following structure:
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_legacy_notify_query_entry {
> > + u8 region_data[16];
>
> So everything is pci specific but this one we for some reason decided to keep
> generic. I'd just make it pci specific, keep things simple.
> We'll generalize if and when other transports want this. Which I doubt.
>
Other is kept generic.
The whole thing was documented as PCI first and ask was to make it generic for the description.
And now again you suggest making it pci specific.
What changed from PCI->GEN->PCI?
Why the whole text should reside in generic file if you want to make it PCI specific, like before?
>
> > +};
> > +
> > +struct virtio_admin_cmd_legacy_notify_query_result {
> > + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> > +}; \end{lstlisting}
>
> here I'm not going to repeat myself. same comments as other commands.
>
> > +
> > +The driver picks the suitable entry when multiple entries are
> > +supplied by the device.
>
> How does driver know how much to allocate?
> I think it allocates an extra entry, inits it to 0, then after the command checks
> whether it was modified. If yes makes the buffer larger. Makes sense? Complex
> though ...
> Can we place an upper limit on # of entries? 4 seems sufficient no?
> And what is "suitable"? I am guessing it just picks the
> *first* entry that it is able to utilize.
> And devices list them in the order of preference.
>
Driver can just allocate some random for 1, 8, 16 entries based on how it wants to do.
Suitable is whatever driver can decide, first or last or prefer PF over VF, driver's choice....
I don’t know why to complicate this a lot when most devices that I know of will do 1 entry.
May be 2nd for the PF.
And may be device can overdo it by adding one more for some large IOBAR in theory on the PF.
4 seems some arbitrary number.
> > +
> > +Refer to the specific transport section for the definition of the
> > +\field{region_data}.
>
> Transport sections are huge. Links please do not make the reader guess.
>
> > +
> > +This command is currently only defined for the PCI SR-IOV group type.
> > +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a
> > +Virtio Device / Device groups / Group administration commands /
> > +Legacy Interface}
> > +
> > +If the group owner device supports legacy region access for its group
> > +member devices, the device MUST set all corresponding bits for
> > +commands VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in the command result
> > of VIRTIO_ADMIN_CMD_LIST_QUERY in
> > +\field{device_admin_cmd_opcodes}.
>
>
> It's enough to just say what is supported:
>
> A device MUST either support all of, or none of
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
>
Ok.
> > +
> > +The device MUST encode and decode legacy device specific registers using
> > +little-endian format.
>
> For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> the device MUST decode and encode (respectively) the value of
> the \field{region_data} using the little-endian format.
>
Ack.
>
>
> > +
> > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> and
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the
> invalid offset
> > +which is outside the legacy common configuration region's address range.
>
> The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> and
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the
> value of the
> \field{offset} and the length of the \field{region_data} does not refer
> to a single field or is not completely within the virtio common
> configuration structure, excluding the device-specific configuration.
>
This is an additional limitation being added, but fine, it doesnt hurt anyone.
>
> > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid
> offset
> > +which is outside the legacy device specific region's address range.
>
> The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of
> the
> \field{offset} and the length of the \field{region_data} does not refer
> to a single field or is not completely within the device-specific
> configuration.
>
Same as above.
>
>
> > +
> > +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> command for
> > +driver notifications. If the group owner device supports driver
> > +notifications region for its group member devices, the device MUST set
> > +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> > +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
>
> I don't get what does all this mean. Seems to say if you support
> VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
> that anyway.
>
> Should we say:
>
> If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
> also support all of
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
>
Yes, you captured it better. Thanks.
>
> > +
> > +When the driver accesses the legacy region of the member device using
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> > +commands, device MUST function as if they are accessed by the legacy
> interface
> > +defined by the transport of the member device.
>
> The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
> have the same effect as writing into the virtio common configuration
> structure through the legacy interface.
>
ok
> repeat for the other commands.
>
o.k.
>
> Also, explain about notification: is it true that writing
> at this address has the same effect as writing into Queue Notify?
>
Yes, will add.
>
> > +
> > +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> Device / Device groups / Group administration commands / Legacy Interface}
> > +
> > +The driver MUST encode and decode legacy device specific registers using
> > +little-endian format.
> > +
> > +The driver SHOULD send
> VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a
> valid offset which
> > +is in the legacy common configuration region address range.
> > +
> > +The driver SHOULD send commands
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in
> the legacy
> > +device specific configuration region address range.
> > +
> > +The group member driver SHOULD use the notification region supplied by
> the group
> > +owner device.
>
> Use how? More specific please. Also, only if the command is supported.
>
>
How to use is already part of the existing spec.
I will add a line to refer to the driver notification spec section.
Ack will add about 'if command supported'.
> Document how is notification thingy used.
>
>
>
> > diff --git a/admin.tex b/admin.tex
> > index fd3b97d..0de26a9 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -117,7 +117,17 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > \hline
> > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> > \hline
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write
> legacy common configuration region of a member device \\
> > +\hline
> > +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read
> legacy common configuration region of a member device \\
> > +\hline
> > +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy
> device configuration region of a member device \\
> > +\hline
> > +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy
> device configuration region of a member device \\
> > +\hline
> > +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query
> notification region for a member device \\
> > +\hline
> > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > \hline
> > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure) \\
> > \hline
>
>
> Grammar fix + drop "region":
>
> VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy
> common configuration structure
>
Ack.
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-05 11:04 ` Michael S. Tsirkin
@ 2023-07-05 22:01 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 22:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:05 AM
[..]
> > +Refer to the specific transport section for the definition of the
> > +\field{region_data}.
>
> Transport sections are huge. Links please do not make the reader guess.
Yes, it is taken care in 4th patch when the transport definition is added.
The link is added here.
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-05 22:01 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-05 22:01 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:05 AM
[..]
> > +Refer to the specific transport section for the definition of the
> > +\field{region_data}.
>
> Transport sections are huge. Links please do not make the reader guess.
Yes, it is taken care in 4th patch when the transport definition is added.
The link is added here.
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-05 11:09 ` Michael S. Tsirkin
@ 2023-07-06 4:20 ` Parav Pandit
-1 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-06 4:20 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:09 AM
> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> depth over the last patch, but I feel it should be squashed into the generic
> legacy command sections avoiding need for indirection, so rewriting it now
> wouldn't be efficient.
I address all your comments and of Cornelia.
PCI part I still prefer to be separate so kept it.
Thanks a lot.
^ permalink raw reply [flat|nested] 48+ messages in thread
* RE: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-06 4:20 ` Parav Pandit
0 siblings, 0 replies; 48+ messages in thread
From: Parav Pandit @ 2023-07-06 4:20 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Wednesday, July 5, 2023 7:09 AM
> OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> depth over the last patch, but I feel it should be squashed into the generic
> legacy command sections avoiding need for indirection, so rewriting it now
> wouldn't be efficient.
I address all your comments and of Cornelia.
PCI part I still prefer to be separate so kept it.
Thanks a lot.
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-05 16:34 ` [virtio-comment] " Parav Pandit
@ 2023-07-06 5:43 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 5:43 UTC (permalink / raw)
To: Parav Pandit
Cc: Cornelia Huck, virtio-comment, david.edmondson, virtio-dev,
sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05, 2023 at 04:34:02PM +0000, Parav Pandit wrote:
>
>
> > From: Cornelia Huck <cohuck@redhat.com>
> > Sent: Wednesday, July 5, 2023 6:24 AM
> > > +
> > > +In some systems, there is a need to support utilizing the legacy
> > > +driver with the device that do not directly support the legacy
> > > +interface. In such scenarios,
> >
> > Maybe better:
> >
> > "In some systems, there is a need to support utilizing legacy drivers with devices
> > that do not directly support the legacy interface."
> >
> > ?
> >
> Can you please explain how plural is better if it is?
> I see this a lot intermixed in the spec.
> Why cannot we describe things using single object "the driver" and "the device"?
> I prefer to describe things without plurality as done in many parts of the spec.
> Please let me know.
It's ok too, but it must be consistent: "do" goes with plural. For
singular - "does". Also this is the 1st time we mention this driver and
this device, so should be "a":
In some systems, there is a need to support utilizing a legacy driver with
a device that does not directly support the legacy interface.
> > > +a group owner device can provide the legacy interface functionality
> > > +for the group member devices. The driver of an owner device can then
> > > +access the legacy interface of a member device on behalf of the legacy
> > member device driver.
> >
> > (...)
> >
> > > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Legacy Driver Notifications Query}
> > > +
> > > +Even though the driver notifications can be communicated through the
> > > +administration command, if the group owner device or group member
> > > +device supports such notifications using a memory-mapped operation or
> > > +I/O operation, they are sent to the device by accessing such a
> > > +notification region using a memory or an I/O operation.
> >
> > I find this sentence confusing, probably because it is too complex... what about
> >
> > "If the group owner device or the group member device support driver
> > notifications via a memory-mapped operation or I/O operation, these
> > notifications are sent to the device via accessing such a notification region using
> > a memory or I/O operation {in addition to|instead of} communicating the
> > notifications through the admistration command."
> >
> With instead of, it reads simpler. Will change it.
>
> > (I'm not sure whether it is either or both.)
> >
> > > +
> > > +A group owner device optionally support querying driver notifications
> > > +region
> >
> > "can optionally support" ?
> >
> Ack.
>
> > > +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > > +This command does not use \field{command_specific_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> >
> > "the command"
> >
> Ack
>
> > > +uses following structure:
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-comment] Re: [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-06 5:43 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 5:43 UTC (permalink / raw)
To: Parav Pandit
Cc: Cornelia Huck, virtio-comment, david.edmondson, virtio-dev,
sburla, jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05, 2023 at 04:34:02PM +0000, Parav Pandit wrote:
>
>
> > From: Cornelia Huck <cohuck@redhat.com>
> > Sent: Wednesday, July 5, 2023 6:24 AM
> > > +
> > > +In some systems, there is a need to support utilizing the legacy
> > > +driver with the device that do not directly support the legacy
> > > +interface. In such scenarios,
> >
> > Maybe better:
> >
> > "In some systems, there is a need to support utilizing legacy drivers with devices
> > that do not directly support the legacy interface."
> >
> > ?
> >
> Can you please explain how plural is better if it is?
> I see this a lot intermixed in the spec.
> Why cannot we describe things using single object "the driver" and "the device"?
> I prefer to describe things without plurality as done in many parts of the spec.
> Please let me know.
It's ok too, but it must be consistent: "do" goes with plural. For
singular - "does". Also this is the 1st time we mention this driver and
this device, so should be "a":
In some systems, there is a need to support utilizing a legacy driver with
a device that does not directly support the legacy interface.
> > > +a group owner device can provide the legacy interface functionality
> > > +for the group member devices. The driver of an owner device can then
> > > +access the legacy interface of a member device on behalf of the legacy
> > member device driver.
> >
> > (...)
> >
> > > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Legacy Driver Notifications Query}
> > > +
> > > +Even though the driver notifications can be communicated through the
> > > +administration command, if the group owner device or group member
> > > +device supports such notifications using a memory-mapped operation or
> > > +I/O operation, they are sent to the device by accessing such a
> > > +notification region using a memory or an I/O operation.
> >
> > I find this sentence confusing, probably because it is too complex... what about
> >
> > "If the group owner device or the group member device support driver
> > notifications via a memory-mapped operation or I/O operation, these
> > notifications are sent to the device via accessing such a notification region using
> > a memory or I/O operation {in addition to|instead of} communicating the
> > notifications through the admistration command."
> >
> With instead of, it reads simpler. Will change it.
>
> > (I'm not sure whether it is either or both.)
> >
> > > +
> > > +A group owner device optionally support querying driver notifications
> > > +region
> >
> > "can optionally support" ?
> >
> Ack.
>
> > > +using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > > +This command does not use \field{command_specific_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> >
> > "the command"
> >
> Ack
>
> > > +uses following structure:
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
2023-07-05 18:26 ` Parav Pandit
@ 2023-07-06 6:01 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 6:01 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05, 2023 at 06:26:13PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, July 5, 2023 7:05 AM
>
>
> > > +These commands are currently only defined for the PCI SR-IOV group
> > > +type and
> >
> > We call it SR-IOV group type not PCI SR-IOV group type. Please be consistent.
> >
> Ack.
>
> > > +have, generally, the same effect as member device accesses through a
> > > +legacy interface listed in section \ref{sec:Virtio Transport Options / Virtio
> > Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device
> > Layout}.
> > > +
> > > +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Common Configuration Write Command}
> >
> > Fix label.
> >
> It is already under the tree of Legacy interface so dropped making it further longer.
> But than I kept the Legacy in the actual name...
> Will add Legacy in the label.
>
> > > +
> > > +This command has the same effect as writing into the virtio common
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> > > + u8 offset; /* Starting byte offset within the common configuration
> > structure to write */
> > > + u8 reserved[7];
> > > + u8 region_data[];
> >
> > I am all for something more specific than offset/data but region seems pointless.
> > And why apply just to data not offset?
> > How about legacy_offset/legacy_data?
> >
> Structure name already has legacy, so no point in repeating legacy_ prefix again in all the fields inside.
> I will drop region_ and just keep offset,data.
>
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> > > +The driver sets valid \field{offset} and associated
> > > +\field{region_data} bytes to write to the common configuration region.
> > > +
> > > +This command does not use \field{command_specific_result}.
> > > +
> >
> > Let's make this consistent with other commands, even if it means we repeat
> > ourselves a bit. And length is not documented and has to be:
> >
> Ah yes, I missed the length documentation.
> Will add it.
>
> Not sure what part you want to make consistent with other commands?
> Do you mean to repeat ourselves for the opcode number? If so, than we should be avoiding it.
> In fact we should remove that repetition in other commands too. We have table to describe it clearly.
Then "set opcode to XXX" would be redundant too.
> >
> > >>>
> >
> > For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > \field{opcode} is set to 0x2.
> > The \field{group_member_id} refers to the member device to be accessed.
> Yeah, there is certainly no point in repeating all of these again.
>
> In previous patch when I repeated where it says it uses the struct virtio_admin_cmd, you asked to drop for the right reason.
> But here you ask to repeat other things...
> Not good.
>
> All above is crystal clear in opcode table and in generic description already.
> Reader has less text to read to understand it.
I actually think it's helpful - whoever is writing a driver
then has a single place to read.
it does not matter if we repeat ourselves a bit - readers are
not compilers.
> > The \field{command_specific_data} is in the format \field{struct
> > virtio_admin_cmd_legacy_common_cfg_wr_data} describing the access to be
> > performed.
> >
> > Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
> > The \field{offset} refers to the offset to write within the virtio common
> > configuration structure, and excluding the device-specific configuration.
> "virtio common configuration structure" is self-explanatory that doesnt need to add the exclusion part.
> Hence it was not added in v9 too.
Again readers are not compilers. It's not 100% clear whether it's legal
to access device specific through this or not.
> > For the
> > SR-IOV group type, the layout used refers to the one used for the legacy
> > interface as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI
> > Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout},
> > except that little endian format is assumed unconditionally.
> > The length of the data to write is simply the length of \field{region_data}.
> >
> > No length or alignment restrictions are placed on the value of the \field{offset}
> > and the length of the \field{region_data}, except that the resulting access refers
> > to a single field and is completely within the virtio common configuration
> > structure, excluding the device-specific configuration.
> >
> > This command has no command specific result.
> >
> >
> > Try to apply this to other commands please.
> >
> I will apply rest of the above text you wrote for this and rest of the commands.
>
> > > +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Common Configuration Read Command}
> > > +
> > > +This command has the same effect as reading from the virtio common
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> > > + u8 offset; /* Starting byte offset within the common configuration
> > > +structure to read */ }; \end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> > > +
> > > +The driver sets valid \field{offset} of the region from where to read
> > > +\field{region_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Device Configuration Write Command}
> > > +
> > > +This command has the same effect as writing into the virtio device
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> > > + u8 offset; /* Starting byte offset within the device configuration
> > structure to write */
> > > + u8 reserved[7];
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> > > +The driver sets valid \field{offset} and associated
> > > +\field{region_data} bytes to the device configuration region.
> > > +
> > > +This command does not use \field{command_specific_result}.
> > > +
> > > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Device Configuration Read Command}
> > > +
> > > +This command has the same effect as reading from the virtio device
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> > > + u8 offset; /* Starting byte offset within the device
> > > +configuration structure to read */ }; \end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> > > +The driver sets valid \field{offset} within the device configuration
> > > +structure from where to read \field{region_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Legacy Driver Notifications Query}
> >
> > Drop "Query", this talks about notifications generally.
> >
> Ok.
>
> > > +
> > > +Even though the driver notifications can be communicated through the
> > > +administration command, if the group owner device or group member
> > > +device supports such notifications using a memory-mapped operation or
> > > +I/O operation, they are sent to the device by accessing such a
> > > +notification region using a memory or an I/O operation.
> >
> > > +
> > > +A group owner device optionally support querying driver notifications
> > > +region using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> >
> > The driver of the owner device can send a driver notification to the member
> > device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > with the \field{offset} matching \fielf{Queue Notify} and the \field{region_data}
> > containing the virtqueue index to be notified.
> >
> > However, as many administration commands are used for slow path
> > configuration, a separate fast path mechanism for such notifications is desired.
> > For the SR-IOV group type, the optional command
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by
> > returning to the driver one or more addresses which to be used to send such
> > driver notifications.
> >
> Ok. will take this version.
>
> >
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > > +This command does not use \field{command_specific_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_notify_query_entry {
> > > + u8 region_data[16];
> >
> > So everything is pci specific but this one we for some reason decided to keep
> > generic. I'd just make it pci specific, keep things simple.
> > We'll generalize if and when other transports want this. Which I doubt.
> >
> Other is kept generic.
> The whole thing was documented as PCI first and ask was to make it generic for the description.
> And now again you suggest making it pci specific.
> What changed from PCI->GEN->PCI?
> Why the whole text should reside in generic file if you want to make it PCI specific, like before?
I always said notification is the most problematic part of it.
Rest we made generic and it looks good to me.
At this point this notification part is still stated confusingly and I
think we should give up on making notification generic for now for
purpose of expediting merging this. Deleting a bunch of text is
easier than trying to restate it clearly.
>
> >
> > > +};
> > > +
> > > +struct virtio_admin_cmd_legacy_notify_query_result {
> > > + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> > > +}; \end{lstlisting}
> >
> > here I'm not going to repeat myself. same comments as other commands.
> >
> > > +
> > > +The driver picks the suitable entry when multiple entries are
> > > +supplied by the device.
> >
> > How does driver know how much to allocate?
> > I think it allocates an extra entry, inits it to 0, then after the command checks
> > whether it was modified. If yes makes the buffer larger. Makes sense? Complex
> > though ...
> > Can we place an upper limit on # of entries? 4 seems sufficient no?
> > And what is "suitable"? I am guessing it just picks the
> > *first* entry that it is able to utilize.
> > And devices list them in the order of preference.
> >
> Driver can just allocate some random for 1, 8, 16 entries based on how it wants to do.
What if e.g. it wants to access through member, allocates just 1
entry and device fills that with owner instead?
I don't think it works.
> Suitable is whatever driver can decide, first or last or prefer PF over VF, driver's choice....
>
> I don’t know why to complicate this a lot when most devices that I know of will do 1 entry.
> May be 2nd for the PF.
> And may be device can overdo it by adding one more for some large IOBAR in theory on the PF.
>
> 4 seems some arbitrary number.
So you got to 3 above, 4 just adds a spare one, and it's a power of two :)
I agree we do not need to complicate it but driver does not know
how many entries to scan. How about simply:
struct virtio_admin_cmd_legacy_notify_query_result {
struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[4];
};
and explain that device zero-initializes unused entries.
we then replace "owner" with "flags" and make the low flag "valid".
> > > +
> > > +Refer to the specific transport section for the definition of the
> > > +\field{region_data}.
> >
> > Transport sections are huge. Links please do not make the reader guess.
> >
> > > +
> > > +This command is currently only defined for the PCI SR-IOV group type.
> > > +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a
> > > +Virtio Device / Device groups / Group administration commands /
> > > +Legacy Interface}
> > > +
> > > +If the group owner device supports legacy region access for its group
> > > +member devices, the device MUST set all corresponding bits for
> > > +commands VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in the command result
> > > of VIRTIO_ADMIN_CMD_LIST_QUERY in
> > > +\field{device_admin_cmd_opcodes}.
> >
> >
> > It's enough to just say what is supported:
> >
> > A device MUST either support all of, or none of
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> >
> Ok.
> > > +
> > > +The device MUST encode and decode legacy device specific registers using
> > > +little-endian format.
> >
> > For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> > the device MUST decode and encode (respectively) the value of
> > the \field{region_data} using the little-endian format.
> >
> Ack.
>
> >
> >
> > > +
> > > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > and
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the
> > invalid offset
> > > +which is outside the legacy common configuration region's address range.
> >
> > The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > and
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the
> > value of the
> > \field{offset} and the length of the \field{region_data} does not refer
> > to a single field or is not completely within the virtio common
> > configuration structure, excluding the device-specific configuration.
> >
> This is an additional limitation being added, but fine, it doesnt hurt anyone.
Not really - your text said "offset has to be valid" but never explained
what is valid. Not how conformance statements should look.
I tried to include a definition above.
> >
> > > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid
> > offset
> > > +which is outside the legacy device specific region's address range.
> >
> > The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of
> > the
> > \field{offset} and the length of the \field{region_data} does not refer
> > to a single field or is not completely within the device-specific
> > configuration.
> >
> Same as above.
>
> >
> >
> > > +
> > > +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> > command for
> > > +driver notifications. If the group owner device supports driver
> > > +notifications region for its group member devices, the device MUST set
> > > +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> > > +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
> >
> > I don't get what does all this mean. Seems to say if you support
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
> > that anyway.
> >
> > Should we say:
> >
> > If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
> > also support all of
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> >
> Yes, you captured it better. Thanks.
>
> >
> > > +
> > > +When the driver accesses the legacy region of the member device using
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> > > +commands, device MUST function as if they are accessed by the legacy
> > interface
> > > +defined by the transport of the member device.
> >
> > The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
> > have the same effect as writing into the virtio common configuration
> > structure through the legacy interface.
> >
> ok
>
> > repeat for the other commands.
> >
> o.k.
> >
> > Also, explain about notification: is it true that writing
> > at this address has the same effect as writing into Queue Notify?
> >
> Yes, will add.
>
> >
> > > +
> > > +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> > Device / Device groups / Group administration commands / Legacy Interface}
> > > +
> > > +The driver MUST encode and decode legacy device specific registers using
> > > +little-endian format.
> > > +
> > > +The driver SHOULD send
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a
> > valid offset which
> > > +is in the legacy common configuration region address range.
> > > +
> > > +The driver SHOULD send commands
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in
> > the legacy
> > > +device specific configuration region address range.
> > > +
> > > +The group member driver SHOULD use the notification region supplied by
> > the group
> > > +owner device.
> >
> > Use how? More specific please. Also, only if the command is supported.
> >
> >
> How to use is already part of the existing spec.
No - because what is in the spec is how the legacy driver uses it.
But legacy driver also uses IO accesses for configuration
and we replace these with commands. This is a pass-through
trick. Need to document this.
> I will add a line to refer to the driver notification spec section.
>
> Ack will add about 'if command supported'.
>
>
> > Document how is notification thingy used.
> >
> >
> >
> > > diff --git a/admin.tex b/admin.tex
> > > index fd3b97d..0de26a9 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -117,7 +117,17 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > \hline
> > > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> > commands used for this group type \\
> > > \hline
> > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > > +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write
> > legacy common configuration region of a member device \\
> > > +\hline
> > > +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read
> > legacy common configuration region of a member device \\
> > > +\hline
> > > +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy
> > device configuration region of a member device \\
> > > +\hline
> > > +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy
> > device configuration region of a member device \\
> > > +\hline
> > > +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query
> > notification region for a member device \\
> > > +\hline
> > > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > > \hline
> > > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> > different structure) \\
> > > \hline
> >
> >
> > Grammar fix + drop "region":
> >
>
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy
> > common configuration structure
> >
> Ack.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [virtio-comment] [PATCH v9 3/4] admin: Add group member legacy register access commands
@ 2023-07-06 6:01 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 6:01 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Wed, Jul 05, 2023 at 06:26:13PM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, July 5, 2023 7:05 AM
>
>
> > > +These commands are currently only defined for the PCI SR-IOV group
> > > +type and
> >
> > We call it SR-IOV group type not PCI SR-IOV group type. Please be consistent.
> >
> Ack.
>
> > > +have, generally, the same effect as member device accesses through a
> > > +legacy interface listed in section \ref{sec:Virtio Transport Options / Virtio
> > Over PCI Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device
> > Layout}.
> > > +
> > > +\paragraph{Legacy Common Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Common Configuration Write Command}
> >
> > Fix label.
> >
> It is already under the tree of Legacy interface so dropped making it further longer.
> But than I kept the Legacy in the actual name...
> Will add Legacy in the label.
>
> > > +
> > > +This command has the same effect as writing into the virtio common
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_wr_data {
> > > + u8 offset; /* Starting byte offset within the common configuration
> > structure to write */
> > > + u8 reserved[7];
> > > + u8 region_data[];
> >
> > I am all for something more specific than offset/data but region seems pointless.
> > And why apply just to data not offset?
> > How about legacy_offset/legacy_data?
> >
> Structure name already has legacy, so no point in repeating legacy_ prefix again in all the fields inside.
> I will drop region_ and just keep offset,data.
>
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE.
> > > +The driver sets valid \field{offset} and associated
> > > +\field{region_data} bytes to write to the common configuration region.
> > > +
> > > +This command does not use \field{command_specific_result}.
> > > +
> >
> > Let's make this consistent with other commands, even if it means we repeat
> > ourselves a bit. And length is not documented and has to be:
> >
> Ah yes, I missed the length documentation.
> Will add it.
>
> Not sure what part you want to make consistent with other commands?
> Do you mean to repeat ourselves for the opcode number? If so, than we should be avoiding it.
> In fact we should remove that repetition in other commands too. We have table to describe it clearly.
Then "set opcode to XXX" would be redundant too.
> >
> > >>>
> >
> > For the command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > \field{opcode} is set to 0x2.
> > The \field{group_member_id} refers to the member device to be accessed.
> Yeah, there is certainly no point in repeating all of these again.
>
> In previous patch when I repeated where it says it uses the struct virtio_admin_cmd, you asked to drop for the right reason.
> But here you ask to repeat other things...
> Not good.
>
> All above is crystal clear in opcode table and in generic description already.
> Reader has less text to read to understand it.
I actually think it's helpful - whoever is writing a driver
then has a single place to read.
it does not matter if we repeat ourselves a bit - readers are
not compilers.
> > The \field{command_specific_data} is in the format \field{struct
> > virtio_admin_cmd_legacy_common_cfg_wr_data} describing the access to be
> > performed.
> >
> > Within \field{struct virtio_admin_cmd_legacy_common_cfg_wr_data}:
> > The \field{offset} refers to the offset to write within the virtio common
> > configuration structure, and excluding the device-specific configuration.
> "virtio common configuration structure" is self-explanatory that doesnt need to add the exclusion part.
> Hence it was not added in v9 too.
Again readers are not compilers. It's not 100% clear whether it's legal
to access device specific through this or not.
> > For the
> > SR-IOV group type, the layout used refers to the one used for the legacy
> > interface as listed in section \ref{sec:Virtio Transport Options / Virtio Over PCI
> > Bus / PCI Device Layout / Legacy Interfaces: A Note on PCI Device Layout},
> > except that little endian format is assumed unconditionally.
> > The length of the data to write is simply the length of \field{region_data}.
> >
> > No length or alignment restrictions are placed on the value of the \field{offset}
> > and the length of the \field{region_data}, except that the resulting access refers
> > to a single field and is completely within the virtio common configuration
> > structure, excluding the device-specific configuration.
> >
> > This command has no command specific result.
> >
> >
> > Try to apply this to other commands please.
> >
> I will apply rest of the above text you wrote for this and rest of the commands.
>
> > > +\paragraph{Legacy Common Configuration Read Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Common Configuration Read Command}
> > > +
> > > +This command has the same effect as reading from the virtio common
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_rd_data {
> > > + u8 offset; /* Starting byte offset within the common configuration
> > > +structure to read */ }; \end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ.
> > > +
> > > +The driver sets valid \field{offset} of the region from where to read
> > > +\field{region_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_common_cfg_rd_result {
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Device Configuration Write Command}
> > > +
> > > +This command has the same effect as writing into the virtio device
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_reg_wr_data {
> > > + u8 offset; /* Starting byte offset within the device configuration
> > structure to write */
> > > + u8 reserved[7];
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_WRITE.
> > > +The driver sets valid \field{offset} and associated
> > > +\field{region_data} bytes to the device configuration region.
> > > +
> > > +This command does not use \field{command_specific_result}.
> > > +
> > > +\paragraph{Legacy Device Configuration Write Command}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Device Configuration Read Command}
> > > +
> > > +This command has the same effect as reading from the virtio device
> > > +configuration structure through the legacy interface.
> > > +\field{command_specific_data} has the following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_cfg_rd_data {
> > > + u8 offset; /* Starting byte offset within the device
> > > +configuration structure to read */ }; \end{lstlisting}
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_REG_READ.
> > > +The driver sets valid \field{offset} within the device configuration
> > > +structure from where to read \field{region_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_dev_reg_rd_result {
> > > + u8 region_data[];
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +\paragraph{Legacy Driver Notification Query}\label{par:Basic
> > > +Facilities of a Virtio Device / Device groups / Group administration
> > > +commands / Legacy Interface / Legacy Driver Notifications Query}
> >
> > Drop "Query", this talks about notifications generally.
> >
> Ok.
>
> > > +
> > > +Even though the driver notifications can be communicated through the
> > > +administration command, if the group owner device or group member
> > > +device supports such notifications using a memory-mapped operation or
> > > +I/O operation, they are sent to the device by accessing such a
> > > +notification region using a memory or an I/O operation.
> >
> > > +
> > > +A group owner device optionally support querying driver notifications
> > > +region using VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY command.
> >
> > The driver of the owner device can send a driver notification to the member
> > device by executing VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > with the \field{offset} matching \fielf{Queue Notify} and the \field{region_data}
> > containing the virtqueue index to be notified.
> >
> > However, as many administration commands are used for slow path
> > configuration, a separate fast path mechanism for such notifications is desired.
> > For the SR-IOV group type, the optional command
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY addresses this need by
> > returning to the driver one or more addresses which to be used to send such
> > driver notifications.
> >
> Ok. will take this version.
>
> >
> > > +
> > > +The driver sets command \field{opcode} to
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY.
> > > +This command does not use \field{command_specific_data}.
> > > +
> > > +When command completes successfully, \field{command_specific_result}
> > > +uses following structure:
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_legacy_notify_query_entry {
> > > + u8 region_data[16];
> >
> > So everything is pci specific but this one we for some reason decided to keep
> > generic. I'd just make it pci specific, keep things simple.
> > We'll generalize if and when other transports want this. Which I doubt.
> >
> Other is kept generic.
> The whole thing was documented as PCI first and ask was to make it generic for the description.
> And now again you suggest making it pci specific.
> What changed from PCI->GEN->PCI?
> Why the whole text should reside in generic file if you want to make it PCI specific, like before?
I always said notification is the most problematic part of it.
Rest we made generic and it looks good to me.
At this point this notification part is still stated confusingly and I
think we should give up on making notification generic for now for
purpose of expediting merging this. Deleting a bunch of text is
easier than trying to restate it clearly.
>
> >
> > > +};
> > > +
> > > +struct virtio_admin_cmd_legacy_notify_query_result {
> > > + struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[];
> > > +}; \end{lstlisting}
> >
> > here I'm not going to repeat myself. same comments as other commands.
> >
> > > +
> > > +The driver picks the suitable entry when multiple entries are
> > > +supplied by the device.
> >
> > How does driver know how much to allocate?
> > I think it allocates an extra entry, inits it to 0, then after the command checks
> > whether it was modified. If yes makes the buffer larger. Makes sense? Complex
> > though ...
> > Can we place an upper limit on # of entries? 4 seems sufficient no?
> > And what is "suitable"? I am guessing it just picks the
> > *first* entry that it is able to utilize.
> > And devices list them in the order of preference.
> >
> Driver can just allocate some random for 1, 8, 16 entries based on how it wants to do.
What if e.g. it wants to access through member, allocates just 1
entry and device fills that with owner instead?
I don't think it works.
> Suitable is whatever driver can decide, first or last or prefer PF over VF, driver's choice....
>
> I don’t know why to complicate this a lot when most devices that I know of will do 1 entry.
> May be 2nd for the PF.
> And may be device can overdo it by adding one more for some large IOBAR in theory on the PF.
>
> 4 seems some arbitrary number.
So you got to 3 above, 4 just adds a spare one, and it's a power of two :)
I agree we do not need to complicate it but driver does not know
how many entries to scan. How about simply:
struct virtio_admin_cmd_legacy_notify_query_result {
struct virtio_virtio_admin_cmd_legacy_notify_query_entry entries[4];
};
and explain that device zero-initializes unused entries.
we then replace "owner" with "flags" and make the low flag "valid".
> > > +
> > > +Refer to the specific transport section for the definition of the
> > > +\field{region_data}.
> >
> > Transport sections are huge. Links please do not make the reader guess.
> >
> > > +
> > > +This command is currently only defined for the PCI SR-IOV group type.
> > > +\devicenormative{\paragraph}{Legacy Interface}{Basic Facilities of a
> > > +Virtio Device / Device groups / Group administration commands /
> > > +Legacy Interface}
> > > +
> > > +If the group owner device supports legacy region access for its group
> > > +member devices, the device MUST set all corresponding bits for
> > > +commands VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ in the command result
> > > of VIRTIO_ADMIN_CMD_LIST_QUERY in
> > > +\field{device_admin_cmd_opcodes}.
> >
> >
> > It's enough to just say what is supported:
> >
> > A device MUST either support all of, or none of
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> >
> Ok.
> > > +
> > > +The device MUST encode and decode legacy device specific registers using
> > > +little-endian format.
> >
> > For VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ,
> > the device MUST decode and encode (respectively) the value of
> > the \field{region_data} using the little-endian format.
> >
> Ack.
>
> >
> >
> > > +
> > > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > and
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands for the
> > invalid offset
> > > +which is outside the legacy common configuration region's address range.
> >
> > The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE
> > and
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands where the
> > value of the
> > \field{offset} and the length of the \field{region_data} does not refer
> > to a single field or is not completely within the virtio common
> > configuration structure, excluding the device-specific configuration.
> >
> This is an additional limitation being added, but fine, it doesnt hurt anyone.
Not really - your text said "offset has to be valid" but never explained
what is valid. Not how conformance statements should look.
I tried to include a definition above.
> >
> > > +The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands for the invalid
> > offset
> > > +which is outside the legacy device specific region's address range.
> >
> > The device MUST fail VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ commands where the value of
> > the
> > \field{offset} and the length of the \field{region_data} does not refer
> > to a single field or is not completely within the device-specific
> > configuration.
> >
> Same as above.
>
> >
> >
> > > +
> > > +The device SHOULD support VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY
> > command for
> > > +driver notifications. If the group owner device supports driver
> > > +notifications region for its group member devices, the device MUST set
> > > +VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY in the command result of
> > > +VIRTIO_ADMIN_CMD_LIST_QUERY in \field{device_admin_cmd_opcodes}.
> >
> > I don't get what does all this mean. Seems to say if you support
> > VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY then set it. Reader should know
> > that anyway.
> >
> > Should we say:
> >
> > If the device supports VIRTIO_ADMIN_CMD_LEGACY_NOTIFY_QUERY it MUST
> > also support all of
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ.
> >
> Yes, you captured it better. Thanks.
>
> >
> > > +
> > > +When the driver accesses the legacy region of the member device using
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE,
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ,
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE,
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ
> > > +commands, device MUST function as if they are accessed by the legacy
> > interface
> > > +defined by the transport of the member device.
> >
> > The command VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE MUST
> > have the same effect as writing into the virtio common configuration
> > structure through the legacy interface.
> >
> ok
>
> > repeat for the other commands.
> >
> o.k.
> >
> > Also, explain about notification: is it true that writing
> > at this address has the same effect as writing into Queue Notify?
> >
> Yes, will add.
>
> >
> > > +
> > > +\drivernormative{\paragraph}{Legacy Interface}{Basic Facilities of a Virtio
> > Device / Device groups / Group administration commands / Legacy Interface}
> > > +
> > > +The driver MUST encode and decode legacy device specific registers using
> > > +little-endian format.
> > > +
> > > +The driver SHOULD send
> > VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ commands with a
> > valid offset which
> > > +is in the legacy common configuration region address range.
> > > +
> > > +The driver SHOULD send commands
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE and
> > > +VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ with a valid offset which is in
> > the legacy
> > > +device specific configuration region address range.
> > > +
> > > +The group member driver SHOULD use the notification region supplied by
> > the group
> > > +owner device.
> >
> > Use how? More specific please. Also, only if the command is supported.
> >
> >
> How to use is already part of the existing spec.
No - because what is in the spec is how the legacy driver uses it.
But legacy driver also uses IO accesses for configuration
and we replace these with commands. This is a pass-through
trick. Need to document this.
> I will add a line to refer to the driver notification spec section.
>
> Ack will add about 'if command supported'.
>
>
> > Document how is notification thingy used.
> >
> >
> >
> > > diff --git a/admin.tex b/admin.tex
> > > index fd3b97d..0de26a9 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -117,7 +117,17 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > \hline
> > > 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> > commands used for this group type \\
> > > \hline
> > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > > +0x0002 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_WRITE & Write
> > legacy common configuration region of a member device \\
> > > +\hline
> > > +0x0003 & VIRTIO_ADMIN_CMD_LEGACY_COMMON_CFG_READ & Read
> > legacy common configuration region of a member device \\
> > > +\hline
> > > +0x0004 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_WRITE & Write legacy
> > device configuration region of a member device \\
> > > +\hline
> > > +0x0005 & VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Read legacy
> > device configuration region of a member device \\
> > > +\hline
> > > +0x0006 & VIRTIO_ADMIN_CMD_LEGACY_DEV_NOTIFY_QUERY & Query
> > notification region for a member device \\
> > > +\hline
> > > +0x0007 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd} \\
> > > \hline
> > > 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> > different structure) \\
> > > \hline
> >
> >
> > Grammar fix + drop "region":
> >
>
> > VIRTIO_ADMIN_CMD_LEGACY_DEV_CFG_READ & Writes into the legacy
> > common configuration structure
> >
> Ack.
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
* [virtio-dev] Re: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
2023-07-06 4:20 ` Parav Pandit
@ 2023-07-06 6:10 ` Michael S. Tsirkin
-1 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 6:10 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Thu, Jul 06, 2023 at 04:20:03AM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, July 5, 2023 7:09 AM
>
> > OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> > depth over the last patch, but I feel it should be squashed into the generic
> > legacy command sections avoiding need for indirection, so rewriting it now
> > wouldn't be efficient.
>
> I address all your comments and of Cornelia.
> PCI part I still prefer to be separate so kept it.
>
> Thanks a lot.
I clarified some questions you asked about. Maybe v11 is justified
in view of my answers. Let me know before I review again.
--
MST
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 48+ messages in thread
* Re: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands
@ 2023-07-06 6:10 ` Michael S. Tsirkin
0 siblings, 0 replies; 48+ messages in thread
From: Michael S. Tsirkin @ 2023-07-06 6:10 UTC (permalink / raw)
To: Parav Pandit
Cc: virtio-comment, cohuck, david.edmondson, virtio-dev, sburla,
jasowang, Yishai Hadas, Maor Gottlieb, Shahaf Shuler
On Thu, Jul 06, 2023 at 04:20:03AM +0000, Parav Pandit wrote:
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Wednesday, July 5, 2023 7:09 AM
>
> > OK I sent a bunch of comments. Pls try to address, unfortunately I didn't go in
> > depth over the last patch, but I feel it should be squashed into the generic
> > legacy command sections avoiding need for indirection, so rewriting it now
> > wouldn't be efficient.
>
> I address all your comments and of Cornelia.
> PCI part I still prefer to be separate so kept it.
>
> Thanks a lot.
I clarified some questions you asked about. Maybe v11 is justified
in view of my answers. Let me know before I review again.
--
MST
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 48+ messages in thread
end of thread, other threads:[~2023-07-06 6:10 UTC | newest]
Thread overview: 48+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-04 23:37 [virtio-dev] [PATCH v9 0/4] admin: Access legacy registers using admin commands Parav Pandit
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
2023-07-04 23:37 ` [virtio-dev] [PATCH v9 1/4] admin: Split opcode table rows with a line Parav Pandit
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
2023-07-05 10:05 ` [virtio-dev] " Cornelia Huck
2023-07-05 10:05 ` [virtio-comment] " Cornelia Huck
2023-07-04 23:37 ` [virtio-dev] [PATCH v9 2/4] admin: Fix section numbering Parav Pandit
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
2023-07-05 10:09 ` [virtio-dev] " Cornelia Huck
2023-07-05 10:09 ` [virtio-comment] " Cornelia Huck
2023-07-05 16:29 ` [virtio-dev] " Parav Pandit
2023-07-05 16:29 ` [virtio-comment] " Parav Pandit
2023-07-04 23:37 ` [virtio-dev] [PATCH v9 3/4] admin: Add group member legacy register access commands Parav Pandit
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
2023-07-05 10:24 ` [virtio-dev] " Cornelia Huck
2023-07-05 10:24 ` [virtio-comment] " Cornelia Huck
2023-07-05 16:34 ` [virtio-dev] " Parav Pandit
2023-07-05 16:34 ` [virtio-comment] " Parav Pandit
2023-07-06 5:43 ` [virtio-dev] " Michael S. Tsirkin
2023-07-06 5:43 ` [virtio-comment] " Michael S. Tsirkin
2023-07-05 11:04 ` [virtio-dev] Re: [virtio-comment] " Michael S. Tsirkin
2023-07-05 11:04 ` Michael S. Tsirkin
2023-07-05 18:26 ` [virtio-dev] " Parav Pandit
2023-07-05 18:26 ` Parav Pandit
2023-07-06 6:01 ` [virtio-dev] " Michael S. Tsirkin
2023-07-06 6:01 ` Michael S. Tsirkin
2023-07-05 22:01 ` [virtio-dev] " Parav Pandit
2023-07-05 22:01 ` Parav Pandit
2023-07-04 23:37 ` [virtio-dev] [PATCH v9 4/4] transport-pci: Introduce group legacy group member config region access Parav Pandit
2023-07-04 23:37 ` [virtio-comment] " Parav Pandit
2023-07-05 10:29 ` [virtio-dev] " Cornelia Huck
2023-07-05 10:29 ` [virtio-comment] " Cornelia Huck
2023-07-05 16:36 ` [virtio-dev] " Parav Pandit
2023-07-05 16:36 ` [virtio-comment] " Parav Pandit
2023-07-05 10:56 ` [virtio-dev] Re: [virtio-comment] " Michael S. Tsirkin
2023-07-05 10:56 ` Michael S. Tsirkin
2023-07-05 11:09 ` [virtio-dev] Re: [virtio-comment] [PATCH v9 0/4] admin: Access legacy registers using admin commands Michael S. Tsirkin
2023-07-05 11:09 ` Michael S. Tsirkin
2023-07-05 13:01 ` [virtio-dev] " Parav Pandit
2023-07-05 13:01 ` Parav Pandit
2023-07-05 13:42 ` [virtio-dev] " Cornelia Huck
2023-07-05 13:42 ` Cornelia Huck
2023-07-05 14:37 ` [virtio-dev] " Parav Pandit
2023-07-05 14:37 ` Parav Pandit
2023-07-06 4:20 ` [virtio-dev] " Parav Pandit
2023-07-06 4:20 ` Parav Pandit
2023-07-06 6:10 ` [virtio-dev] " Michael S. Tsirkin
2023-07-06 6:10 ` Michael S. Tsirkin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.