[virtio-dev] [RFC PATCH] admin-queue: bind the group member to the device

[virtio-dev] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
   and sets the ip, queue number, etc. The user obtains the identity of
   the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
   of the patch

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline \hline
 0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
 \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \field{VF Enable} refer to registers within the SR-IOV Extended
 Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
 
+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
+devices don't know about these VFs. Because each VF may be assigned a different role
+by the user, the virtio device can not choose one VF to bind random.
+So only the user knows how to bind the virtio devices to the VFs.
+On the other hand, generally the virtio devices are not created by the user
+inside the guest OS. This requires some management platform to participate.
+
+So we introduce a new admin queue command to bind the VFs and the virtio
+devices.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_bind {
+    u64 identity;
+};
+\end{lstlisting}
+
+The user got the \field{identity} from the management platform, that is not
+included by this spec.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
+
+VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
+
+The \field{identity} is passed by the user. It is the identity of the virtio
+device.
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
+
+Every device MUST have one unique \field{identity} in the host.
+
+If the PF device can not find the device by the \field{identity},
+the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
+
+If the device is found by the \field{identity}, the device MUST work as the
+device of this group member specified by the \field{group_member_id}.
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
-- 
2.32.0.3.g01195cf9f


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
devices don't know about these VFs. Because each VF may be assigned a different role
by the user, the virtio device can not choose one VF to bind random.
So only the user knows how to bind the virtio devices to the VFs.
On the other hand, generally the virtio devices are not created by the user
inside the guest OS. This requires some management platform to participate.

So the usage of this command:
1. The user purchases a virtio network card on the management platform,
   and sets the ip, queue number, etc. The user obtains the identity of
   the network card.
2. The user creates a VF with echo 8 > sriov_numvfs
3. The user binds the net crad to a VF with identity through the command
   of the patch

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 40 insertions(+), 1 deletion(-)

diff --git a/admin.tex b/admin.tex
index 2efd4d7..64d0667 100644
--- a/admin.tex
+++ b/admin.tex
@@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \hline \hline
 0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
 0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
-0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
+0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
+0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
 \hline
 0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
 \hline
@@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
 \field{VF Enable} refer to registers within the SR-IOV Extended
 Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
 
+\subsubsection{Bind the device for member}
+
+The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
+devices don't know about these VFs. Because each VF may be assigned a different role
+by the user, the virtio device can not choose one VF to bind random.
+So only the user knows how to bind the virtio devices to the VFs.
+On the other hand, generally the virtio devices are not created by the user
+inside the guest OS. This requires some management platform to participate.
+
+So we introduce a new admin queue command to bind the VFs and the virtio
+devices.
+
+\begin{lstlisting}
+struct virtio_admin_cmd_bind {
+    u64 identity;
+};
+\end{lstlisting}
+
+The user got the \field{identity} from the management platform, that is not
+included by this spec.
+
+\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
+
+VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
+
+The \field{identity} is passed by the user. It is the identity of the virtio
+device.
+
+\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
+
+Every device MUST have one unique \field{identity} in the host.
+
+If the PF device can not find the device by the \field{identity},
+the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
+
+If the device is found by the \field{identity}, the device MUST work as the
+device of this group member specified by the \field{group_member_id}.
+
 \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
 
 An administration virtqueue of an owner device is used to submit
-- 
2.32.0.3.g01195cf9f


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>     and sets the ip, queue number, etc. The user obtains the identity of
>     the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>     of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>   admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \hline \hline
>   0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>   \hline
>   0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>   \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \field{VF Enable} refer to registers within the SR-IOV Extended
>   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>   
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
If the VFs are create in a guest OS, I assume that means the user has 
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security 
issue(affects host pci).
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
I failed to understand this, once a VF is created, it has a personality, 
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device 
choose
one VF to bind?
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
Sorry, failed to process this. Maybe an orchestration sw layer can help?
Provision a device on demands and assign it to a guest?

Thanks
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>     and sets the ip, queue number, etc. The user obtains the identity of
>     the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>     of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>   admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \hline \hline
>   0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>   \hline
>   0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>   \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>   \field{VF Enable} refer to registers within the SR-IOV Extended
>   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>   
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
If the VFs are create in a guest OS, I assume that means the user has 
passthrough-ed the
PF to the guest. For nested, I am not sure whether this is a security 
issue(affects host pci).
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
I failed to understand this, once a VF is created, it has a personality, 
e.g.,
create a virtio-net VF from a virtio-net PF, and PF knows that.

I am not familiar with the background, What do you mean by virtio device 
choose
one VF to bind?
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
Sorry, failed to process this. Maybe an orchestration sw layer can help?
Provision a device on demands and assign it to a guest?

Thanks
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>   
>   An administration virtqueue of an owner device is used to submit


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >     and sets the ip, queue number, etc. The user obtains the identity of
> >     the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >     of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >   admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >   1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   \hline \hline
> >   0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >   \hline
> >   0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >   \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   \field{VF Enable} refer to registers within the SR-IOV Extended
> >   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> If the VFs are create in a guest OS, I assume that means the user has
> passthrough-ed the
> PF to the guest. For nested, I am not sure whether this is a security
> issue(affects host pci).


No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.


> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> I failed to understand this, once a VF is created, it has a personality,
> e.g.,
> create a virtio-net VF from a virtio-net PF, and PF knows that.
>
> I am not familiar with the background, What do you mean by virtio device
> choose
> one VF to bind?


On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

Thanks.


> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> Provision a device on demands and assign it to a guest?
>
> Thanks
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >   An administration virtqueue of an owner device is used to submit
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >     and sets the ip, queue number, etc. The user obtains the identity of
> >     the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >     of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >   admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >   1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   \hline \hline
> >   0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >   0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >   \hline
> >   0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >   \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >   \field{VF Enable} refer to registers within the SR-IOV Extended
> >   Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> If the VFs are create in a guest OS, I assume that means the user has
> passthrough-ed the
> PF to the guest. For nested, I am not sure whether this is a security
> issue(affects host pci).


No care about the passthrough, we always created VFs by the PF.

I should not say "inside the guest OS". I just want to say that the VF is create
by the user in the OS. The devices does not know about it.


> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> I failed to understand this, once a VF is created, it has a personality,
> e.g.,
> create a virtio-net VF from a virtio-net PF, and PF knows that.
>
> I am not familiar with the background, What do you mean by virtio device
> choose
> one VF to bind?


On the cloud, the nic is created by the management platform, the
user can not create a new nic inside the OS.

So after echo sriov_numvfs, the user just got some VFs,
there is not backend virtio-net devices.

Thanks.


> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> Provision a device on demands and assign it to a guest?
>
> Thanks
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >   \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >   An administration virtqueue of an owner device is used to submit
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>> by the user, the virtio device can not choose one VF to bind random.
>>> So only the user knows how to bind the virtio devices to the VFs.
>>> On the other hand, generally the virtio devices are not created by the user
>>> inside the guest OS. This requires some management platform to participate.
>>>
>>> So the usage of this command:
>>> 1. The user purchases a virtio network card on the management platform,
>>>      and sets the ip, queue number, etc. The user obtains the identity of
>>>      the network card.
>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>> 3. The user binds the net crad to a VF with identity through the command
>>>      of the patch
>>>
>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>> ---
>>>    admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>    1 file changed, 40 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/admin.tex b/admin.tex
>>> index 2efd4d7..64d0667 100644
>>> --- a/admin.tex
>>> +++ b/admin.tex
>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>    \hline \hline
>>>    0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>    0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>    \hline
>>>    0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>    \hline
>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>    \field{VF Enable} refer to registers within the SR-IOV Extended
>>>    Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>
>>> +\subsubsection{Bind the device for member}
>>> +
>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>> If the VFs are create in a guest OS, I assume that means the user has
>> passthrough-ed the
>> PF to the guest. For nested, I am not sure whether this is a security
>> issue(affects host pci).
>
> No care about the passthrough, we always created VFs by the PF.
>
> I should not say "inside the guest OS". I just want to say that the VF is create
> by the user in the OS. The devices does not know about it.
OK, perhaps just say create VFs from a PF in the OS?
>
>
>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>> +by the user, the virtio device can not choose one VF to bind random.
>> I failed to understand this, once a VF is created, it has a personality,
>> e.g.,
>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>
>> I am not familiar with the background, What do you mean by virtio device
>> choose
>> one VF to bind?
>
> On the cloud, the nic is created by the management platform, the
> user can not create a new nic inside the OS.
>
> So after echo sriov_numvfs, the user just got some VFs,
> there is not backend virtio-net devices.
I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a 
guest on demands.

So I am confused what the intention of this patch.
>
> Thanks.
>
>
>>> +So only the user knows how to bind the virtio devices to the VFs.
>>> +On the other hand, generally the virtio devices are not created by the user
>>> +inside the guest OS. This requires some management platform to participate.
>>> +
>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>> +devices.
>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>> Provision a device on demands and assign it to a guest?
>>
>> Thanks
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_admin_cmd_bind {
>>> +    u64 identity;
>>> +};
>>> +\end{lstlisting}
>>> +
>>> +The user got the \field{identity} from the management platform, that is not
>>> +included by this spec.
>>> +
>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>> +
>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>> +
>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>> +device.
>>> +
>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>> +
>>> +Every device MUST have one unique \field{identity} in the host.
>>> +
>>> +If the PF device can not find the device by the \field{identity},
>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>> +
>>> +If the device is found by the \field{identity}, the device MUST work as the
>>> +device of this group member specified by the \field{group_member_id}.
>>> +
>>>    \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>
>>>    An administration virtqueue of an owner device is used to submit
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>> by the user, the virtio device can not choose one VF to bind random.
>>> So only the user knows how to bind the virtio devices to the VFs.
>>> On the other hand, generally the virtio devices are not created by the user
>>> inside the guest OS. This requires some management platform to participate.
>>>
>>> So the usage of this command:
>>> 1. The user purchases a virtio network card on the management platform,
>>>      and sets the ip, queue number, etc. The user obtains the identity of
>>>      the network card.
>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>> 3. The user binds the net crad to a VF with identity through the command
>>>      of the patch
>>>
>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>> ---
>>>    admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>    1 file changed, 40 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/admin.tex b/admin.tex
>>> index 2efd4d7..64d0667 100644
>>> --- a/admin.tex
>>> +++ b/admin.tex
>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>    \hline \hline
>>>    0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>    0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>    \hline
>>>    0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>    \hline
>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>    \field{VF Enable} refer to registers within the SR-IOV Extended
>>>    Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>
>>> +\subsubsection{Bind the device for member}
>>> +
>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>> If the VFs are create in a guest OS, I assume that means the user has
>> passthrough-ed the
>> PF to the guest. For nested, I am not sure whether this is a security
>> issue(affects host pci).
>
> No care about the passthrough, we always created VFs by the PF.
>
> I should not say "inside the guest OS". I just want to say that the VF is create
> by the user in the OS. The devices does not know about it.
OK, perhaps just say create VFs from a PF in the OS?
>
>
>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>> +by the user, the virtio device can not choose one VF to bind random.
>> I failed to understand this, once a VF is created, it has a personality,
>> e.g.,
>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>
>> I am not familiar with the background, What do you mean by virtio device
>> choose
>> one VF to bind?
>
> On the cloud, the nic is created by the management platform, the
> user can not create a new nic inside the OS.
>
> So after echo sriov_numvfs, the user just got some VFs,
> there is not backend virtio-net devices.
I think it is not a "user" mange the VFs, the VFs usually provisioned by
the orchestration software and it assign properly selected a VF to a 
guest on demands.

So I am confused what the intention of this patch.
>
> Thanks.
>
>
>>> +So only the user knows how to bind the virtio devices to the VFs.
>>> +On the other hand, generally the virtio devices are not created by the user
>>> +inside the guest OS. This requires some management platform to participate.
>>> +
>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>> +devices.
>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>> Provision a device on demands and assign it to a guest?
>>
>> Thanks
>>> +
>>> +\begin{lstlisting}
>>> +struct virtio_admin_cmd_bind {
>>> +    u64 identity;
>>> +};
>>> +\end{lstlisting}
>>> +
>>> +The user got the \field{identity} from the management platform, that is not
>>> +included by this spec.
>>> +
>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>> +
>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>> +
>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>> +device.
>>> +
>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>> +
>>> +Every device MUST have one unique \field{identity} in the host.
>>> +
>>> +If the PF device can not find the device by the \field{identity},
>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>> +
>>> +If the device is found by the \field{identity}, the device MUST work as the
>>> +device of this group member specified by the \field{group_member_id}.
>>> +
>>>    \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>
>>>    An administration virtqueue of an owner device is used to submit
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>> by the user, the virtio device can not choose one VF to bind random.
> >>> So only the user knows how to bind the virtio devices to the VFs.
> >>> On the other hand, generally the virtio devices are not created by the user
> >>> inside the guest OS. This requires some management platform to participate.
> >>>
> >>> So the usage of this command:
> >>> 1. The user purchases a virtio network card on the management platform,
> >>>      and sets the ip, queue number, etc. The user obtains the identity of
> >>>      the network card.
> >>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>> 3. The user binds the net crad to a VF with identity through the command
> >>>      of the patch
> >>>
> >>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>> ---
> >>>    admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>    1 file changed, 40 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/admin.tex b/admin.tex
> >>> index 2efd4d7..64d0667 100644
> >>> --- a/admin.tex
> >>> +++ b/admin.tex
> >>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>    \hline \hline
> >>>    0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>    0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>    \hline
> >>>    0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>    \hline
> >>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>    \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>    Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>
> >>> +\subsubsection{Bind the device for member}
> >>> +
> >>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >> If the VFs are create in a guest OS, I assume that means the user has
> >> passthrough-ed the
> >> PF to the guest. For nested, I am not sure whether this is a security
> >> issue(affects host pci).
> >
> > No care about the passthrough, we always created VFs by the PF.
> >
> > I should not say "inside the guest OS". I just want to say that the VF is create
> > by the user in the OS. The devices does not know about it.
> OK, perhaps just say create VFs from a PF in the OS?

YES.


> >
> >
> >>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>> +by the user, the virtio device can not choose one VF to bind random.
> >> I failed to understand this, once a VF is created, it has a personality,
> >> e.g.,
> >> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>
> >> I am not familiar with the background, What do you mean by virtio device
> >> choose
> >> one VF to bind?
> >
> > On the cloud, the nic is created by the management platform, the
> > user can not create a new nic inside the OS.
> >
> > So after echo sriov_numvfs, the user just got some VFs,
> > there is not backend virtio-net devices.
> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> the orchestration software and it assign properly selected a VF to a
> guest on demands.


Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

Thanks.



>
> So I am confused what the intention of this patch.
> >
> > Thanks.
> >
> >
> >>> +So only the user knows how to bind the virtio devices to the VFs.
> >>> +On the other hand, generally the virtio devices are not created by the user
> >>> +inside the guest OS. This requires some management platform to participate.
> >>> +
> >>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>> +devices.
> >> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >> Provision a device on demands and assign it to a guest?
> >>
> >> Thanks
> >>> +
> >>> +\begin{lstlisting}
> >>> +struct virtio_admin_cmd_bind {
> >>> +    u64 identity;
> >>> +};
> >>> +\end{lstlisting}
> >>> +
> >>> +The user got the \field{identity} from the management platform, that is not
> >>> +included by this spec.
> >>> +
> >>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>> +
> >>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>> +
> >>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>> +device.
> >>> +
> >>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>> +
> >>> +Every device MUST have one unique \field{identity} in the host.
> >>> +
> >>> +If the PF device can not find the device by the \field{identity},
> >>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>> +
> >>> +If the device is found by the \field{identity}, the device MUST work as the
> >>> +device of this group member specified by the \field{group_member_id}.
> >>> +
> >>>    \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>
> >>>    An administration virtqueue of an owner device is used to submit
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>> by the user, the virtio device can not choose one VF to bind random.
> >>> So only the user knows how to bind the virtio devices to the VFs.
> >>> On the other hand, generally the virtio devices are not created by the user
> >>> inside the guest OS. This requires some management platform to participate.
> >>>
> >>> So the usage of this command:
> >>> 1. The user purchases a virtio network card on the management platform,
> >>>      and sets the ip, queue number, etc. The user obtains the identity of
> >>>      the network card.
> >>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>> 3. The user binds the net crad to a VF with identity through the command
> >>>      of the patch
> >>>
> >>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>> ---
> >>>    admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>    1 file changed, 40 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/admin.tex b/admin.tex
> >>> index 2efd4d7..64d0667 100644
> >>> --- a/admin.tex
> >>> +++ b/admin.tex
> >>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>    \hline \hline
> >>>    0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>    0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>    \hline
> >>>    0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>    \hline
> >>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>    \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>    Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>
> >>> +\subsubsection{Bind the device for member}
> >>> +
> >>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >> If the VFs are create in a guest OS, I assume that means the user has
> >> passthrough-ed the
> >> PF to the guest. For nested, I am not sure whether this is a security
> >> issue(affects host pci).
> >
> > No care about the passthrough, we always created VFs by the PF.
> >
> > I should not say "inside the guest OS". I just want to say that the VF is create
> > by the user in the OS. The devices does not know about it.
> OK, perhaps just say create VFs from a PF in the OS?

YES.


> >
> >
> >>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>> +by the user, the virtio device can not choose one VF to bind random.
> >> I failed to understand this, once a VF is created, it has a personality,
> >> e.g.,
> >> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>
> >> I am not familiar with the background, What do you mean by virtio device
> >> choose
> >> one VF to bind?
> >
> > On the cloud, the nic is created by the management platform, the
> > user can not create a new nic inside the OS.
> >
> > So after echo sriov_numvfs, the user just got some VFs,
> > there is not backend virtio-net devices.
> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> the orchestration software and it assign properly selected a VF to a
> guest on demands.


Yes, but we do not need to care about the guest. Because VF may only be used
in host, such as docker.

The problem is that the user (you can think of this as the orchestration
software) creates some VFs, these are only some PCI devices, which virtio
devices will work on these VFs. I think that creating a vf and creating a
virtio-net device are two different things. One is done by user in the OS, one
is done on the management platform.  So we need to bind them together.

Thanks.



>
> So I am confused what the intention of this patch.
> >
> > Thanks.
> >
> >
> >>> +So only the user knows how to bind the virtio devices to the VFs.
> >>> +On the other hand, generally the virtio devices are not created by the user
> >>> +inside the guest OS. This requires some management platform to participate.
> >>> +
> >>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>> +devices.
> >> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >> Provision a device on demands and assign it to a guest?
> >>
> >> Thanks
> >>> +
> >>> +\begin{lstlisting}
> >>> +struct virtio_admin_cmd_bind {
> >>> +    u64 identity;
> >>> +};
> >>> +\end{lstlisting}
> >>> +
> >>> +The user got the \field{identity} from the management platform, that is not
> >>> +included by this spec.
> >>> +
> >>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>> +
> >>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>> +
> >>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>> +device.
> >>> +
> >>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>> +
> >>> +Every device MUST have one unique \field{identity} in the host.
> >>> +
> >>> +If the PF device can not find the device by the \field{identity},
> >>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>> +
> >>> +If the device is found by the \field{identity}, the device MUST work as the
> >>> +device of this group member specified by the \field{group_member_id}.
> >>> +
> >>>    \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>
> >>>    An administration virtqueue of an owner device is used to submit
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>
>>>>> So the usage of this command:
>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>       and sets the ip, queue number, etc. The user obtains the identity of
>>>>>       the network card.
>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>       of the patch
>>>>>
>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>> ---
>>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/admin.tex b/admin.tex
>>>>> index 2efd4d7..64d0667 100644
>>>>> --- a/admin.tex
>>>>> +++ b/admin.tex
>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>     \hline \hline
>>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>     \hline
>>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>     \hline
>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>
>>>>> +\subsubsection{Bind the device for member}
>>>>> +
>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>> passthrough-ed the
>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>> issue(affects host pci).
>>> No care about the passthrough, we always created VFs by the PF.
>>>
>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>> by the user in the OS. The devices does not know about it.
>> OK, perhaps just say create VFs from a PF in the OS?
> YES.
>
>
>>>
>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>> I failed to understand this, once a VF is created, it has a personality,
>>>> e.g.,
>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>
>>>> I am not familiar with the background, What do you mean by virtio device
>>>> choose
>>>> one VF to bind?
>>> On the cloud, the nic is created by the management platform, the
>>> user can not create a new nic inside the OS.
>>>
>>> So after echo sriov_numvfs, the user just got some VFs,
>>> there is not backend virtio-net devices.
>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>> the orchestration software and it assign properly selected a VF to a
>> guest on demands.
>
> Yes, but we do not need to care about the guest. Because VF may only be used
> in host, such as docker.
>
> The problem is that the user (you can think of this as the orchestration
> software) creates some VFs, these are only some PCI devices, which virtio
> devices will work on these VFs. I think that creating a vf and creating a
> virtio-net device are two different things. One is done by user in the OS, one
> is done on the management platform.  So we need to bind them together.
If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF, 
(e.g., the same network
device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

Thanks
> Thanks.
>
>
>
>> So I am confused what the intention of this patch.
>>> Thanks.
>>>
>>>
>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>> +
>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>> +devices.
>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>> Provision a device on demands and assign it to a guest?
>>>>
>>>> Thanks
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_admin_cmd_bind {
>>>>> +    u64 identity;
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>> +included by this spec.
>>>>> +
>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>> +
>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>> +
>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>> +device.
>>>>> +
>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>> +
>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>> +
>>>>> +If the PF device can not find the device by the \field{identity},
>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>> +
>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>> +
>>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>
>>>>>     An administration virtqueue of an owner device is used to submit
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>
>>>>> So the usage of this command:
>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>       and sets the ip, queue number, etc. The user obtains the identity of
>>>>>       the network card.
>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>       of the patch
>>>>>
>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>> ---
>>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/admin.tex b/admin.tex
>>>>> index 2efd4d7..64d0667 100644
>>>>> --- a/admin.tex
>>>>> +++ b/admin.tex
>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>     \hline \hline
>>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>     \hline
>>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>     \hline
>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>
>>>>> +\subsubsection{Bind the device for member}
>>>>> +
>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>> passthrough-ed the
>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>> issue(affects host pci).
>>> No care about the passthrough, we always created VFs by the PF.
>>>
>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>> by the user in the OS. The devices does not know about it.
>> OK, perhaps just say create VFs from a PF in the OS?
> YES.
>
>
>>>
>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>> I failed to understand this, once a VF is created, it has a personality,
>>>> e.g.,
>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>
>>>> I am not familiar with the background, What do you mean by virtio device
>>>> choose
>>>> one VF to bind?
>>> On the cloud, the nic is created by the management platform, the
>>> user can not create a new nic inside the OS.
>>>
>>> So after echo sriov_numvfs, the user just got some VFs,
>>> there is not backend virtio-net devices.
>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>> the orchestration software and it assign properly selected a VF to a
>> guest on demands.
>
> Yes, but we do not need to care about the guest. Because VF may only be used
> in host, such as docker.
>
> The problem is that the user (you can think of this as the orchestration
> software) creates some VFs, these are only some PCI devices, which virtio
> devices will work on these VFs. I think that creating a vf and creating a
> virtio-net device are two different things. One is done by user in the OS, one
> is done on the management platform.  So we need to bind them together.
If the VFs are created through sriov_numvfs, once created, the VF device and
its personality are determined.

PCI spec says:
All VFs associated with a PF must be the same device type as the PF, 
(e.g., the same network
device type or the same storage device type.)

So how can the creating process be splitted into separated steps?

Are we discussing something beyond the spec?

Thanks
> Thanks.
>
>
>
>> So I am confused what the intention of this patch.
>>> Thanks.
>>>
>>>
>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>> +
>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>> +devices.
>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>> Provision a device on demands and assign it to a guest?
>>>>
>>>> Thanks
>>>>> +
>>>>> +\begin{lstlisting}
>>>>> +struct virtio_admin_cmd_bind {
>>>>> +    u64 identity;
>>>>> +};
>>>>> +\end{lstlisting}
>>>>> +
>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>> +included by this spec.
>>>>> +
>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>> +
>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>> +
>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>> +device.
>>>>> +
>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>> +
>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>> +
>>>>> +If the PF device can not find the device by the \field{identity},
>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>> +
>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>> +
>>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>
>>>>>     An administration virtqueue of an owner device is used to submit
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>
> >>>>> So the usage of this command:
> >>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>       and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>       the network card.
> >>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>       of the patch
> >>>>>
> >>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>> ---
> >>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/admin.tex b/admin.tex
> >>>>> index 2efd4d7..64d0667 100644
> >>>>> --- a/admin.tex
> >>>>> +++ b/admin.tex
> >>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>     \hline \hline
> >>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>     \hline
> >>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>     \hline
> >>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>
> >>>>> +\subsubsection{Bind the device for member}
> >>>>> +
> >>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>> passthrough-ed the
> >>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>> issue(affects host pci).
> >>> No care about the passthrough, we always created VFs by the PF.
> >>>
> >>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>> by the user in the OS. The devices does not know about it.
> >> OK, perhaps just say create VFs from a PF in the OS?
> > YES.
> >
> >
> >>>
> >>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>> I failed to understand this, once a VF is created, it has a personality,
> >>>> e.g.,
> >>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>
> >>>> I am not familiar with the background, What do you mean by virtio device
> >>>> choose
> >>>> one VF to bind?
> >>> On the cloud, the nic is created by the management platform, the
> >>> user can not create a new nic inside the OS.
> >>>
> >>> So after echo sriov_numvfs, the user just got some VFs,
> >>> there is not backend virtio-net devices.
> >> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >> the orchestration software and it assign properly selected a VF to a
> >> guest on demands.
> >
> > Yes, but we do not need to care about the guest. Because VF may only be used
> > in host, such as docker.
> >
> > The problem is that the user (you can think of this as the orchestration
> > software) creates some VFs, these are only some PCI devices, which virtio
> > devices will work on these VFs. I think that creating a vf and creating a
> > virtio-net device are two different things. One is done by user in the OS, one
> > is done on the management platform.  So we need to bind them together.
> If the VFs are created through sriov_numvfs, once created, the VF device and
> its personality are determined.
>
> PCI spec says:
> All VFs associated with a PF must be the same device type as the PF,
> (e.g., the same network
> device type or the same storage device type.)
>
> So how can the creating process be splitted into separated steps?
>
> Are we discussing something beyond the spec?

NO.

The device types are same.

How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
these are managed by the management platform. On the cloud, there is an abstract
object in the backend, which contains things that are generally configured on
the management platform. It is something that users purchase.
Under the virtio standard it is similar to device.

In my understanding, we just created a pci vf, and virtio works on top of pci,
so there must be two steps here (If I mistake, please point out.). When we create
a vf, it doesn't mean that the backend deivce is ready. Of course, in some
scenarios, we can immediately have a backend default device respond when the
driver probe the vf.  But in our scenario, each device is independent.

Thanks.

>
> Thanks
> > Thanks.
> >
> >
> >
> >> So I am confused what the intention of this patch.
> >>> Thanks.
> >>>
> >>>
> >>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>> +
> >>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>> +devices.
> >>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>> Provision a device on demands and assign it to a guest?
> >>>>
> >>>> Thanks
> >>>>> +
> >>>>> +\begin{lstlisting}
> >>>>> +struct virtio_admin_cmd_bind {
> >>>>> +    u64 identity;
> >>>>> +};
> >>>>> +\end{lstlisting}
> >>>>> +
> >>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>> +included by this spec.
> >>>>> +
> >>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>> +
> >>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>> +
> >>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>> +device.
> >>>>> +
> >>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>> +
> >>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>> +
> >>>>> +If the PF device can not find the device by the \field{identity},
> >>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>> +
> >>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>> +
> >>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>
> >>>>>     An administration virtqueue of an owner device is used to submit
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> >>
> >> This publicly archived list offers a means to provide input to the
> >> OASIS Virtual I/O Device (VIRTIO) TC.
> >>
> >> In order to verify user consent to the Feedback License terms and
> >> to minimize spam in the list archive, subscription is required
> >> before posting.
> >>
> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> List help: virtio-comment-help@lists.oasis-open.org
> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> Committee: https://www.oasis-open.org/committees/virtio/
> >> Join OASIS: https://www.oasis-open.org/join/
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>
> >>>>> So the usage of this command:
> >>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>       and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>       the network card.
> >>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>       of the patch
> >>>>>
> >>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>> ---
> >>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>
> >>>>> diff --git a/admin.tex b/admin.tex
> >>>>> index 2efd4d7..64d0667 100644
> >>>>> --- a/admin.tex
> >>>>> +++ b/admin.tex
> >>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>     \hline \hline
> >>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>     \hline
> >>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>     \hline
> >>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>
> >>>>> +\subsubsection{Bind the device for member}
> >>>>> +
> >>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>> passthrough-ed the
> >>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>> issue(affects host pci).
> >>> No care about the passthrough, we always created VFs by the PF.
> >>>
> >>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>> by the user in the OS. The devices does not know about it.
> >> OK, perhaps just say create VFs from a PF in the OS?
> > YES.
> >
> >
> >>>
> >>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>> I failed to understand this, once a VF is created, it has a personality,
> >>>> e.g.,
> >>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>
> >>>> I am not familiar with the background, What do you mean by virtio device
> >>>> choose
> >>>> one VF to bind?
> >>> On the cloud, the nic is created by the management platform, the
> >>> user can not create a new nic inside the OS.
> >>>
> >>> So after echo sriov_numvfs, the user just got some VFs,
> >>> there is not backend virtio-net devices.
> >> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >> the orchestration software and it assign properly selected a VF to a
> >> guest on demands.
> >
> > Yes, but we do not need to care about the guest. Because VF may only be used
> > in host, such as docker.
> >
> > The problem is that the user (you can think of this as the orchestration
> > software) creates some VFs, these are only some PCI devices, which virtio
> > devices will work on these VFs. I think that creating a vf and creating a
> > virtio-net device are two different things. One is done by user in the OS, one
> > is done on the management platform.  So we need to bind them together.
> If the VFs are created through sriov_numvfs, once created, the VF device and
> its personality are determined.
>
> PCI spec says:
> All VFs associated with a PF must be the same device type as the PF,
> (e.g., the same network
> device type or the same storage device type.)
>
> So how can the creating process be splitted into separated steps?
>
> Are we discussing something beyond the spec?

NO.

The device types are same.

How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
these are managed by the management platform. On the cloud, there is an abstract
object in the backend, which contains things that are generally configured on
the management platform. It is something that users purchase.
Under the virtio standard it is similar to device.

In my understanding, we just created a pci vf, and virtio works on top of pci,
so there must be two steps here (If I mistake, please point out.). When we create
a vf, it doesn't mean that the backend deivce is ready. Of course, in some
scenarios, we can immediately have a backend default device respond when the
driver probe the vf.  But in our scenario, each device is independent.

Thanks.

>
> Thanks
> > Thanks.
> >
> >
> >
> >> So I am confused what the intention of this patch.
> >>> Thanks.
> >>>
> >>>
> >>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>> +
> >>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>> +devices.
> >>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>> Provision a device on demands and assign it to a guest?
> >>>>
> >>>> Thanks
> >>>>> +
> >>>>> +\begin{lstlisting}
> >>>>> +struct virtio_admin_cmd_bind {
> >>>>> +    u64 identity;
> >>>>> +};
> >>>>> +\end{lstlisting}
> >>>>> +
> >>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>> +included by this spec.
> >>>>> +
> >>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>> +
> >>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>> +
> >>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>> +device.
> >>>>> +
> >>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>> +
> >>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>> +
> >>>>> +If the PF device can not find the device by the \field{identity},
> >>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>> +
> >>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>> +
> >>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>
> >>>>>     An administration virtqueue of an owner device is used to submit
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> >>
> >> This publicly archived list offers a means to provide input to the
> >> OASIS Virtual I/O Device (VIRTIO) TC.
> >>
> >> In order to verify user consent to the Feedback License terms and
> >> to minimize spam in the list archive, subscription is required
> >> before posting.
> >>
> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> List help: virtio-comment-help@lists.oasis-open.org
> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> Committee: https://www.oasis-open.org/committees/virtio/
> >> Join OASIS: https://www.oasis-open.org/join/
> >>
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 17:16:40 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>
> > >> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > >>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > >>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > >>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> > >>>>> by the user, the virtio device can not choose one VF to bind random.
> > >>>>> So only the user knows how to bind the virtio devices to the VFs.
> > >>>>> On the other hand, generally the virtio devices are not created by the user
> > >>>>> inside the guest OS. This requires some management platform to participate.
> > >>>>>
> > >>>>> So the usage of this command:
> > >>>>> 1. The user purchases a virtio network card on the management platform,
> > >>>>>       and sets the ip, queue number, etc. The user obtains the identity of
> > >>>>>       the network card.
> > >>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> > >>>>> 3. The user binds the net crad to a VF with identity through the command
> > >>>>>       of the patch
> > >>>>>
> > >>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > >>>>> ---
> > >>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
> > >>>>>
> > >>>>> diff --git a/admin.tex b/admin.tex
> > >>>>> index 2efd4d7..64d0667 100644
> > >>>>> --- a/admin.tex
> > >>>>> +++ b/admin.tex
> > >>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >>>>>     \hline \hline
> > >>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > >>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > >>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > >>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >>>>>     \hline
> > >>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > >>>>>     \hline
> > >>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
> > >>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >>>>>
> > >>>>> +\subsubsection{Bind the device for member}
> > >>>>> +
> > >>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > >>>> If the VFs are create in a guest OS, I assume that means the user has
> > >>>> passthrough-ed the
> > >>>> PF to the guest. For nested, I am not sure whether this is a security
> > >>>> issue(affects host pci).
> > >>> No care about the passthrough, we always created VFs by the PF.
> > >>>
> > >>> I should not say "inside the guest OS". I just want to say that the VF is create
> > >>> by the user in the OS. The devices does not know about it.
> > >> OK, perhaps just say create VFs from a PF in the OS?
> > > YES.
> > >
> > >
> > >>>
> > >>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> > >>>>> +by the user, the virtio device can not choose one VF to bind random.
> > >>>> I failed to understand this, once a VF is created, it has a personality,
> > >>>> e.g.,
> > >>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> > >>>>
> > >>>> I am not familiar with the background, What do you mean by virtio device
> > >>>> choose
> > >>>> one VF to bind?
> > >>> On the cloud, the nic is created by the management platform, the
> > >>> user can not create a new nic inside the OS.
> > >>>
> > >>> So after echo sriov_numvfs, the user just got some VFs,
> > >>> there is not backend virtio-net devices.
> > >> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> > >> the orchestration software and it assign properly selected a VF to a
> > >> guest on demands.
> > >
> > > Yes, but we do not need to care about the guest. Because VF may only be used
> > > in host, such as docker.
> > >
> > > The problem is that the user (you can think of this as the orchestration
> > > software) creates some VFs, these are only some PCI devices, which virtio
> > > devices will work on these VFs. I think that creating a vf and creating a
> > > virtio-net device are two different things. One is done by user in the OS, one
> > > is done on the management platform.  So we need to bind them together.
> > If the VFs are created through sriov_numvfs, once created, the VF device and
> > its personality are determined.
> >
> > PCI spec says:
> > All VFs associated with a PF must be the same device type as the PF,
> > (e.g., the same network
> > device type or the same storage device type.)
> >
> > So how can the creating process be splitted into separated steps?
> >
> > Are we discussing something beyond the spec?
>
> NO.
>
> The device types are same.
>
> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> these are managed by the management platform. On the cloud, there is an abstract
> object in the backend, which contains things that are generally configured on
> the management platform. It is something that users purchase.
> Under the virtio standard it is similar to device.

Sorry, I want to add one case.

The device may is migrated to this host. Not just purchase.

Thanks.


>
> In my understanding, we just created a pci vf, and virtio works on top of pci,
> so there must be two steps here (If I mistake, please point out.). When we create
> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> scenarios, we can immediately have a backend default device respond when the
> driver probe the vf.  But in our scenario, each device is independent.
>
> Thanks.
>
> >
> > Thanks
> > > Thanks.
> > >
> > >
> > >
> > >> So I am confused what the intention of this patch.
> > >>> Thanks.
> > >>>
> > >>>
> > >>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > >>>>> +On the other hand, generally the virtio devices are not created by the user
> > >>>>> +inside the guest OS. This requires some management platform to participate.
> > >>>>> +
> > >>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> > >>>>> +devices.
> > >>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> > >>>> Provision a device on demands and assign it to a guest?
> > >>>>
> > >>>> Thanks
> > >>>>> +
> > >>>>> +\begin{lstlisting}
> > >>>>> +struct virtio_admin_cmd_bind {
> > >>>>> +    u64 identity;
> > >>>>> +};
> > >>>>> +\end{lstlisting}
> > >>>>> +
> > >>>>> +The user got the \field{identity} from the management platform, that is not
> > >>>>> +included by this spec.
> > >>>>> +
> > >>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > >>>>> +
> > >>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > >>>>> +
> > >>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> > >>>>> +device.
> > >>>>> +
> > >>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > >>>>> +
> > >>>>> +Every device MUST have one unique \field{identity} in the host.
> > >>>>> +
> > >>>>> +If the PF device can not find the device by the \field{identity},
> > >>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > >>>>> +
> > >>>>> +If the device is found by the \field{identity}, the device MUST work as the
> > >>>>> +device of this group member specified by the \field{group_member_id}.
> > >>>>> +
> > >>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > >>>>>
> > >>>>>     An administration virtqueue of an owner device is used to submit
> > >>> This publicly archived list offers a means to provide input to the
> > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>
> > >>> In order to verify user consent to the Feedback License terms and
> > >>> to minimize spam in the list archive, subscription is required
> > >>> before posting.
> > >>>
> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>> List help: virtio-comment-help@lists.oasis-open.org
> > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>> Join OASIS: https://www.oasis-open.org/join/
> > >>>
> > >>
> > >> This publicly archived list offers a means to provide input to the
> > >> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>
> > >> In order to verify user consent to the Feedback License terms and
> > >> to minimize spam in the list archive, subscription is required
> > >> before posting.
> > >>
> > >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >> List help: virtio-comment-help@lists.oasis-open.org
> > >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >> Committee: https://www.oasis-open.org/committees/virtio/
> > >> Join OASIS: https://www.oasis-open.org/join/
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > >
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 17:16:40 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>
> > >> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > >>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > >>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > >>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> > >>>>> by the user, the virtio device can not choose one VF to bind random.
> > >>>>> So only the user knows how to bind the virtio devices to the VFs.
> > >>>>> On the other hand, generally the virtio devices are not created by the user
> > >>>>> inside the guest OS. This requires some management platform to participate.
> > >>>>>
> > >>>>> So the usage of this command:
> > >>>>> 1. The user purchases a virtio network card on the management platform,
> > >>>>>       and sets the ip, queue number, etc. The user obtains the identity of
> > >>>>>       the network card.
> > >>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> > >>>>> 3. The user binds the net crad to a VF with identity through the command
> > >>>>>       of the patch
> > >>>>>
> > >>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > >>>>> ---
> > >>>>>     admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >>>>>     1 file changed, 40 insertions(+), 1 deletion(-)
> > >>>>>
> > >>>>> diff --git a/admin.tex b/admin.tex
> > >>>>> index 2efd4d7..64d0667 100644
> > >>>>> --- a/admin.tex
> > >>>>> +++ b/admin.tex
> > >>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >>>>>     \hline \hline
> > >>>>>     0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > >>>>>     0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > >>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > >>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >>>>>     \hline
> > >>>>>     0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > >>>>>     \hline
> > >>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >>>>>     \field{VF Enable} refer to registers within the SR-IOV Extended
> > >>>>>     Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >>>>>
> > >>>>> +\subsubsection{Bind the device for member}
> > >>>>> +
> > >>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > >>>> If the VFs are create in a guest OS, I assume that means the user has
> > >>>> passthrough-ed the
> > >>>> PF to the guest. For nested, I am not sure whether this is a security
> > >>>> issue(affects host pci).
> > >>> No care about the passthrough, we always created VFs by the PF.
> > >>>
> > >>> I should not say "inside the guest OS". I just want to say that the VF is create
> > >>> by the user in the OS. The devices does not know about it.
> > >> OK, perhaps just say create VFs from a PF in the OS?
> > > YES.
> > >
> > >
> > >>>
> > >>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> > >>>>> +by the user, the virtio device can not choose one VF to bind random.
> > >>>> I failed to understand this, once a VF is created, it has a personality,
> > >>>> e.g.,
> > >>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> > >>>>
> > >>>> I am not familiar with the background, What do you mean by virtio device
> > >>>> choose
> > >>>> one VF to bind?
> > >>> On the cloud, the nic is created by the management platform, the
> > >>> user can not create a new nic inside the OS.
> > >>>
> > >>> So after echo sriov_numvfs, the user just got some VFs,
> > >>> there is not backend virtio-net devices.
> > >> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> > >> the orchestration software and it assign properly selected a VF to a
> > >> guest on demands.
> > >
> > > Yes, but we do not need to care about the guest. Because VF may only be used
> > > in host, such as docker.
> > >
> > > The problem is that the user (you can think of this as the orchestration
> > > software) creates some VFs, these are only some PCI devices, which virtio
> > > devices will work on these VFs. I think that creating a vf and creating a
> > > virtio-net device are two different things. One is done by user in the OS, one
> > > is done on the management platform.  So we need to bind them together.
> > If the VFs are created through sriov_numvfs, once created, the VF device and
> > its personality are determined.
> >
> > PCI spec says:
> > All VFs associated with a PF must be the same device type as the PF,
> > (e.g., the same network
> > device type or the same storage device type.)
> >
> > So how can the creating process be splitted into separated steps?
> >
> > Are we discussing something beyond the spec?
>
> NO.
>
> The device types are same.
>
> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> these are managed by the management platform. On the cloud, there is an abstract
> object in the backend, which contains things that are generally configured on
> the management platform. It is something that users purchase.
> Under the virtio standard it is similar to device.

Sorry, I want to add one case.

The device may is migrated to this host. Not just purchase.

Thanks.


>
> In my understanding, we just created a pci vf, and virtio works on top of pci,
> so there must be two steps here (If I mistake, please point out.). When we create
> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> scenarios, we can immediately have a backend default device respond when the
> driver probe the vf.  But in our scenario, each device is independent.
>
> Thanks.
>
> >
> > Thanks
> > > Thanks.
> > >
> > >
> > >
> > >> So I am confused what the intention of this patch.
> > >>> Thanks.
> > >>>
> > >>>
> > >>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > >>>>> +On the other hand, generally the virtio devices are not created by the user
> > >>>>> +inside the guest OS. This requires some management platform to participate.
> > >>>>> +
> > >>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> > >>>>> +devices.
> > >>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> > >>>> Provision a device on demands and assign it to a guest?
> > >>>>
> > >>>> Thanks
> > >>>>> +
> > >>>>> +\begin{lstlisting}
> > >>>>> +struct virtio_admin_cmd_bind {
> > >>>>> +    u64 identity;
> > >>>>> +};
> > >>>>> +\end{lstlisting}
> > >>>>> +
> > >>>>> +The user got the \field{identity} from the management platform, that is not
> > >>>>> +included by this spec.
> > >>>>> +
> > >>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > >>>>> +
> > >>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > >>>>> +
> > >>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> > >>>>> +device.
> > >>>>> +
> > >>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > >>>>> +
> > >>>>> +Every device MUST have one unique \field{identity} in the host.
> > >>>>> +
> > >>>>> +If the PF device can not find the device by the \field{identity},
> > >>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > >>>>> +
> > >>>>> +If the device is found by the \field{identity}, the device MUST work as the
> > >>>>> +device of this group member specified by the \field{group_member_id}.
> > >>>>> +
> > >>>>>     \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > >>>>>
> > >>>>>     An administration virtqueue of an owner device is used to submit
> > >>> This publicly archived list offers a means to provide input to the
> > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>
> > >>> In order to verify user consent to the Feedback License terms and
> > >>> to minimize spam in the list archive, subscription is required
> > >>> before posting.
> > >>>
> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>> List help: virtio-comment-help@lists.oasis-open.org
> > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>> Join OASIS: https://www.oasis-open.org/join/
> > >>>
> > >>
> > >> This publicly archived list offers a means to provide input to the
> > >> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>
> > >> In order to verify user consent to the Feedback License terms and
> > >> to minimize spam in the list archive, subscription is required
> > >> before posting.
> > >>
> > >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >> List help: virtio-comment-help@lists.oasis-open.org
> > >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >> Committee: https://www.oasis-open.org/committees/virtio/
> > >> Join OASIS: https://www.oasis-open.org/join/
> > >>
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> > >
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 02:22:10PM +0800, Xuan Zhuo wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
> 
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs

this one is unlikely, as this requires PF access and affects all VFs.

> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
> 
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>




> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
> 
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>  
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs.

I don't really get how.

> Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.

This description needs a complete rewrite, but to try to help you I'd
need to figure out what is being said. My guess is that this is an
attempt to introduce some kind of identity to VFs.  OK but what is this
identity and how is it better than using VF#?  You want it to persist
when NumVFs in the SR-IOV capability changes?

> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.

Without even understanding the above, what I don't like about this
proposal, is that it assumes some out of band interface to either
specify or discover these IDs. With admin commands we now have
ability to fully specify a management interface, and I'd rather
we had a complete proposal covering all aspects of it.
If some platform does not want to use the management aspects,
that's ok.

> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit

> -- 
> 2.32.0.3.g01195cf9f


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] Re: [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 02:22:10PM +0800, Xuan Zhuo wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
> 
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs

this one is unlikely, as this requires PF access and affects all VFs.

> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
> 
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>




> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
> 
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>  
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs.

I don't really get how.

> Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.

This description needs a complete rewrite, but to try to help you I'd
need to figure out what is being said. My guess is that this is an
attempt to introduce some kind of identity to VFs.  OK but what is this
identity and how is it better than using VF#?  You want it to persist
when NumVFs in the SR-IOV capability changes?

> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.

Without even understanding the above, what I don't like about this
proposal, is that it assumes some out of band interface to either
specify or discover these IDs. With admin commands we now have
ability to fully specify a management interface, and I'd rather
we had a complete proposal covering all aspects of it.
If some platform does not want to use the management aspects,
that's ok.

> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>  
>  An administration virtqueue of an owner device is used to submit

> -- 
> 2.32.0.3.g01195cf9f


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 05:32:47 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 02:22:10PM +0800, Xuan Zhuo wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
>
> this one is unlikely, as this requires PF access and affects all VFs.

How do you create VFs?

>
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>
>
>
>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs.
>
> I don't really get how.

Maybe there is something wrong with my expression. The backend can know these
vfs, but it doesn't know which vf should be bound to which backend.

>
> > Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
>
> This description needs a complete rewrite, but to try to help you I'd
> need to figure out what is being said. My guess is that this is an
> attempt to introduce some kind of identity to VFs.  OK but what is this

NO. An identity for the device in the backend.

> identity and how is it better than using VF#?  You want it to persist
> when NumVFs in the SR-IOV capability changes?
>
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
>
> Without even understanding the above, what I don't like about this
> proposal, is that it assumes some out of band interface to either
> specify or discover these IDs.

This is an rfc, and I actually don't like this proposal too. But I want to raise
this issue, this is a real problem, and I hope to find a standardized solution.
I don't want us to create a private solution internally.

Thanks.

> With admin commands we now have
> ability to fully specify a management interface, and I'd rather
> we had a complete proposal covering all aspects of it.
> If some platform does not want to use the management aspects,
> that's ok.
>
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
>
> > --
> > 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 05:32:47 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 02:22:10PM +0800, Xuan Zhuo wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
>
> this one is unlikely, as this requires PF access and affects all VFs.

How do you create VFs?

>
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>
>
>
>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs.
>
> I don't really get how.

Maybe there is something wrong with my expression. The backend can know these
vfs, but it doesn't know which vf should be bound to which backend.

>
> > Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
>
> This description needs a complete rewrite, but to try to help you I'd
> need to figure out what is being said. My guess is that this is an
> attempt to introduce some kind of identity to VFs.  OK but what is this

NO. An identity for the device in the backend.

> identity and how is it better than using VF#?  You want it to persist
> when NumVFs in the SR-IOV capability changes?
>
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
>
> Without even understanding the above, what I don't like about this
> proposal, is that it assumes some out of band interface to either
> specify or discover these IDs.

This is an rfc, and I actually don't like this proposal too. But I want to raise
this issue, this is a real problem, and I hope to find a standardized solution.
I don't want us to create a private solution internally.

Thanks.

> With admin commands we now have
> ability to fully specify a management interface, and I'd rather
> we had a complete proposal covering all aspects of it.
> If some platform does not want to use the management aspects,
> that's ok.
>
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
>
> > --
> > 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>
>>>>>>> So the usage of this command:
>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>        and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>        the network card.
>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>        of the patch
>>>>>>>
>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>> ---
>>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>> --- a/admin.tex
>>>>>>> +++ b/admin.tex
>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \hline \hline
>>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>      \hline
>>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>      \hline
>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>
>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>> +
>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>> passthrough-ed the
>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>> issue(affects host pci).
>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>
>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>> by the user in the OS. The devices does not know about it.
>>>> OK, perhaps just say create VFs from a PF in the OS?
>>> YES.
>>>
>>>
>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>> e.g.,
>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>
>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>> choose
>>>>>> one VF to bind?
>>>>> On the cloud, the nic is created by the management platform, the
>>>>> user can not create a new nic inside the OS.
>>>>>
>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>> there is not backend virtio-net devices.
>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>> the orchestration software and it assign properly selected a VF to a
>>>> guest on demands.
>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>> in host, such as docker.
>>>
>>> The problem is that the user (you can think of this as the orchestration
>>> software) creates some VFs, these are only some PCI devices, which virtio
>>> devices will work on these VFs. I think that creating a vf and creating a
>>> virtio-net device are two different things. One is done by user in the OS, one
>>> is done on the management platform.  So we need to bind them together.
>> If the VFs are created through sriov_numvfs, once created, the VF device and
>> its personality are determined.
>>
>> PCI spec says:
>> All VFs associated with a PF must be the same device type as the PF,
>> (e.g., the same network
>> device type or the same storage device type.)
>>
>> So how can the creating process be splitted into separated steps?
>>
>> Are we discussing something beyond the spec?
> NO.
>
> The device types are same.
>
> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> these are managed by the management platform. On the cloud, there is an abstract
> object in the backend, which contains things that are generally configured on
> the management platform. It is something that users purchase.
> Under the virtio standard it is similar to device.
>
> In my understanding, we just created a pci vf, and virtio works on top of pci,
> so there must be two steps here (If I mistake, please point out.). When we create
> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> scenarios, we can immediately have a backend default device respond when the
> driver probe the vf.  But in our scenario, each device is independent.
Once a VF is crated, there comes with some default configurations, like 
MTU and MAC.
Do you mean first step creation and second step initialize it?
If so, current spec only allow the user to config MAC through control vq.
vDPA allows to provision a device with proper configuration, maybe that 
can be the solution?

For binding, maybe the orchestration layer manages the pool and it knows 
how to initialize
the device

Thanks
>
> Thanks.
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>
>>>> So I am confused what the intention of this patch.
>>>>> Thanks.
>>>>>
>>>>>
>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>> +
>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>> +devices.
>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>
>>>>>> Thanks
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>> +    u64 identity;
>>>>>>> +};
>>>>>>> +\end{lstlisting}
>>>>>>> +
>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>> +included by this spec.
>>>>>>> +
>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>> +
>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>> +device.
>>>>>>> +
>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>> +
>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>> +
>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>> +
>>>>>>>      \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>
>>>>>>>      An administration virtqueue of an owner device is used to submit
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>>> This publicly archived list offers a means to provide input to the
>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>
>>>> In order to verify user consent to the Feedback License terms and
>>>> to minimize spam in the list archive, subscription is required
>>>> before posting.
>>>>
>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>
>>>>>>> So the usage of this command:
>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>        and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>        the network card.
>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>        of the patch
>>>>>>>
>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>> ---
>>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>> --- a/admin.tex
>>>>>>> +++ b/admin.tex
>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \hline \hline
>>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>      \hline
>>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>      \hline
>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>
>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>> +
>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>> passthrough-ed the
>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>> issue(affects host pci).
>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>
>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>> by the user in the OS. The devices does not know about it.
>>>> OK, perhaps just say create VFs from a PF in the OS?
>>> YES.
>>>
>>>
>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>> e.g.,
>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>
>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>> choose
>>>>>> one VF to bind?
>>>>> On the cloud, the nic is created by the management platform, the
>>>>> user can not create a new nic inside the OS.
>>>>>
>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>> there is not backend virtio-net devices.
>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>> the orchestration software and it assign properly selected a VF to a
>>>> guest on demands.
>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>> in host, such as docker.
>>>
>>> The problem is that the user (you can think of this as the orchestration
>>> software) creates some VFs, these are only some PCI devices, which virtio
>>> devices will work on these VFs. I think that creating a vf and creating a
>>> virtio-net device are two different things. One is done by user in the OS, one
>>> is done on the management platform.  So we need to bind them together.
>> If the VFs are created through sriov_numvfs, once created, the VF device and
>> its personality are determined.
>>
>> PCI spec says:
>> All VFs associated with a PF must be the same device type as the PF,
>> (e.g., the same network
>> device type or the same storage device type.)
>>
>> So how can the creating process be splitted into separated steps?
>>
>> Are we discussing something beyond the spec?
> NO.
>
> The device types are same.
>
> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> these are managed by the management platform. On the cloud, there is an abstract
> object in the backend, which contains things that are generally configured on
> the management platform. It is something that users purchase.
> Under the virtio standard it is similar to device.
>
> In my understanding, we just created a pci vf, and virtio works on top of pci,
> so there must be two steps here (If I mistake, please point out.). When we create
> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> scenarios, we can immediately have a backend default device respond when the
> driver probe the vf.  But in our scenario, each device is independent.
Once a VF is crated, there comes with some default configurations, like 
MTU and MAC.
Do you mean first step creation and second step initialize it?
If so, current spec only allow the user to config MAC through control vq.
vDPA allows to provision a device with proper configuration, maybe that 
can be the solution?

For binding, maybe the orchestration layer manages the pool and it knows 
how to initialize
the device

Thanks
>
> Thanks.
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>
>>>> So I am confused what the intention of this patch.
>>>>> Thanks.
>>>>>
>>>>>
>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>> +
>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>> +devices.
>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>
>>>>>> Thanks
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>> +    u64 identity;
>>>>>>> +};
>>>>>>> +\end{lstlisting}
>>>>>>> +
>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>> +included by this spec.
>>>>>>> +
>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>> +
>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>> +device.
>>>>>>> +
>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>> +
>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>> +
>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>> +
>>>>>>>      \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>
>>>>>>>      An administration virtqueue of an owner device is used to submit
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>>> This publicly archived list offers a means to provide input to the
>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>
>>>> In order to verify user consent to the Feedback License terms and
>>>> to minimize spam in the list archive, subscription is required
>>>> before posting.
>>>>
>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>>>
> >>>>>>> So the usage of this command:
> >>>>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>>>        the network card.
> >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>>>        of the patch
> >>>>>>>
> >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>>>> ---
> >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>>>
> >>>>>>> diff --git a/admin.tex b/admin.tex
> >>>>>>> index 2efd4d7..64d0667 100644
> >>>>>>> --- a/admin.tex
> >>>>>>> +++ b/admin.tex
> >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>      \hline \hline
> >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>      \hline
> >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>>>      \hline
> >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>>>
> >>>>>>> +\subsubsection{Bind the device for member}
> >>>>>>> +
> >>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>>>> passthrough-ed the
> >>>>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>>>> issue(affects host pci).
> >>>>> No care about the passthrough, we always created VFs by the PF.
> >>>>>
> >>>>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>>>> by the user in the OS. The devices does not know about it.
> >>>> OK, perhaps just say create VFs from a PF in the OS?
> >>> YES.
> >>>
> >>>
> >>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>>>> I failed to understand this, once a VF is created, it has a personality,
> >>>>>> e.g.,
> >>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>>>
> >>>>>> I am not familiar with the background, What do you mean by virtio device
> >>>>>> choose
> >>>>>> one VF to bind?
> >>>>> On the cloud, the nic is created by the management platform, the
> >>>>> user can not create a new nic inside the OS.
> >>>>>
> >>>>> So after echo sriov_numvfs, the user just got some VFs,
> >>>>> there is not backend virtio-net devices.
> >>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >>>> the orchestration software and it assign properly selected a VF to a
> >>>> guest on demands.
> >>> Yes, but we do not need to care about the guest. Because VF may only be used
> >>> in host, such as docker.
> >>>
> >>> The problem is that the user (you can think of this as the orchestration
> >>> software) creates some VFs, these are only some PCI devices, which virtio
> >>> devices will work on these VFs. I think that creating a vf and creating a
> >>> virtio-net device are two different things. One is done by user in the OS, one
> >>> is done on the management platform.  So we need to bind them together.
> >> If the VFs are created through sriov_numvfs, once created, the VF device and
> >> its personality are determined.
> >>
> >> PCI spec says:
> >> All VFs associated with a PF must be the same device type as the PF,
> >> (e.g., the same network
> >> device type or the same storage device type.)
> >>
> >> So how can the creating process be splitted into separated steps?
> >>
> >> Are we discussing something beyond the spec?
> > NO.
> >
> > The device types are same.
> >
> > How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> > these are managed by the management platform. On the cloud, there is an abstract
> > object in the backend, which contains things that are generally configured on
> > the management platform. It is something that users purchase.
> > Under the virtio standard it is similar to device.
> >
> > In my understanding, we just created a pci vf, and virtio works on top of pci,
> > so there must be two steps here (If I mistake, please point out.). When we create
> > a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> > scenarios, we can immediately have a backend default device respond when the
> > driver probe the vf.  But in our scenario, each device is independent.
> Once a VF is crated, there comes with some default configurations, like
> MTU and MAC.
> Do you mean first step creation and second step initialize it?

Not exactly correct,

The first step is just to create a vf, at this time there can be a default
virtio-net, it doesn't matter.

In the second step, we can bind a backend device to this vf.

Not just for initialization for new divice, we also want to support live
migration.

For example, on the host, we create a vf and passthrough it into a guest os,
this guest is migrated from another host, and its corresponding network card is
also migrated to this host. We need to bind this vf to the migrated network
card.

So just initialization is not enough.

Thanks

> If so, current spec only allow the user to config MAC through control vq.
> vDPA allows to provision a device with proper configuration, maybe that
> can be the solution?
>
> For binding, maybe the orchestration layer manages the pool and it knows
> how to initialize
> the device
>
> Thanks
> >
> > Thanks.
> >
> >> Thanks
> >>> Thanks.
> >>>
> >>>
> >>>
> >>>> So I am confused what the intention of this patch.
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>>>> +
> >>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>>>> +devices.
> >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>>>> Provision a device on demands and assign it to a guest?
> >>>>>>
> >>>>>> Thanks
> >>>>>>> +
> >>>>>>> +\begin{lstlisting}
> >>>>>>> +struct virtio_admin_cmd_bind {
> >>>>>>> +    u64 identity;
> >>>>>>> +};
> >>>>>>> +\end{lstlisting}
> >>>>>>> +
> >>>>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>>>> +included by this spec.
> >>>>>>> +
> >>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>> +
> >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>>>> +
> >>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>>>> +device.
> >>>>>>> +
> >>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>> +
> >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>>>> +
> >>>>>>> +If the PF device can not find the device by the \field{identity},
> >>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>>>> +
> >>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>>>> +
> >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>>>
> >>>>>>>      An administration virtqueue of an owner device is used to submit
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >>>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>>>
> >>>>>>> So the usage of this command:
> >>>>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>>>        the network card.
> >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>>>        of the patch
> >>>>>>>
> >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>>>> ---
> >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>>>
> >>>>>>> diff --git a/admin.tex b/admin.tex
> >>>>>>> index 2efd4d7..64d0667 100644
> >>>>>>> --- a/admin.tex
> >>>>>>> +++ b/admin.tex
> >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>      \hline \hline
> >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>      \hline
> >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>>>      \hline
> >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>>>
> >>>>>>> +\subsubsection{Bind the device for member}
> >>>>>>> +
> >>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>>>> passthrough-ed the
> >>>>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>>>> issue(affects host pci).
> >>>>> No care about the passthrough, we always created VFs by the PF.
> >>>>>
> >>>>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>>>> by the user in the OS. The devices does not know about it.
> >>>> OK, perhaps just say create VFs from a PF in the OS?
> >>> YES.
> >>>
> >>>
> >>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>>>> I failed to understand this, once a VF is created, it has a personality,
> >>>>>> e.g.,
> >>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>>>
> >>>>>> I am not familiar with the background, What do you mean by virtio device
> >>>>>> choose
> >>>>>> one VF to bind?
> >>>>> On the cloud, the nic is created by the management platform, the
> >>>>> user can not create a new nic inside the OS.
> >>>>>
> >>>>> So after echo sriov_numvfs, the user just got some VFs,
> >>>>> there is not backend virtio-net devices.
> >>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >>>> the orchestration software and it assign properly selected a VF to a
> >>>> guest on demands.
> >>> Yes, but we do not need to care about the guest. Because VF may only be used
> >>> in host, such as docker.
> >>>
> >>> The problem is that the user (you can think of this as the orchestration
> >>> software) creates some VFs, these are only some PCI devices, which virtio
> >>> devices will work on these VFs. I think that creating a vf and creating a
> >>> virtio-net device are two different things. One is done by user in the OS, one
> >>> is done on the management platform.  So we need to bind them together.
> >> If the VFs are created through sriov_numvfs, once created, the VF device and
> >> its personality are determined.
> >>
> >> PCI spec says:
> >> All VFs associated with a PF must be the same device type as the PF,
> >> (e.g., the same network
> >> device type or the same storage device type.)
> >>
> >> So how can the creating process be splitted into separated steps?
> >>
> >> Are we discussing something beyond the spec?
> > NO.
> >
> > The device types are same.
> >
> > How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> > these are managed by the management platform. On the cloud, there is an abstract
> > object in the backend, which contains things that are generally configured on
> > the management platform. It is something that users purchase.
> > Under the virtio standard it is similar to device.
> >
> > In my understanding, we just created a pci vf, and virtio works on top of pci,
> > so there must be two steps here (If I mistake, please point out.). When we create
> > a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> > scenarios, we can immediately have a backend default device respond when the
> > driver probe the vf.  But in our scenario, each device is independent.
> Once a VF is crated, there comes with some default configurations, like
> MTU and MAC.
> Do you mean first step creation and second step initialize it?

Not exactly correct,

The first step is just to create a vf, at this time there can be a default
virtio-net, it doesn't matter.

In the second step, we can bind a backend device to this vf.

Not just for initialization for new divice, we also want to support live
migration.

For example, on the host, we create a vf and passthrough it into a guest os,
this guest is migrated from another host, and its corresponding network card is
also migrated to this host. We need to bind this vf to the migrated network
card.

So just initialization is not enough.

Thanks

> If so, current spec only allow the user to config MAC through control vq.
> vDPA allows to provision a device with proper configuration, maybe that
> can be the solution?
>
> For binding, maybe the orchestration layer manages the pool and it knows
> how to initialize
> the device
>
> Thanks
> >
> > Thanks.
> >
> >> Thanks
> >>> Thanks.
> >>>
> >>>
> >>>
> >>>> So I am confused what the intention of this patch.
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>>>> +
> >>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>>>> +devices.
> >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>>>> Provision a device on demands and assign it to a guest?
> >>>>>>
> >>>>>> Thanks
> >>>>>>> +
> >>>>>>> +\begin{lstlisting}
> >>>>>>> +struct virtio_admin_cmd_bind {
> >>>>>>> +    u64 identity;
> >>>>>>> +};
> >>>>>>> +\end{lstlisting}
> >>>>>>> +
> >>>>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>>>> +included by this spec.
> >>>>>>> +
> >>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>> +
> >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>>>> +
> >>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>>>> +device.
> >>>>>>> +
> >>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>> +
> >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>>>> +
> >>>>>>> +If the PF device can not find the device by the \field{identity},
> >>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>>>> +
> >>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>>>> +
> >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>>>
> >>>>>>>      An administration virtqueue of an owner device is used to submit
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >>>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Monday, June 26, 2023 6:51 AM
> 
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>
> > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> can not choose one VF to bind random.
> > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > >>>>>>> On the other hand, generally the virtio devices are not
> > >>>>>>> created by the user inside the guest OS. This requires some
> management platform to participate.
> > >>>>>>>
> > >>>>>>> So the usage of this command:
> > >>>>>>> 1. The user purchases a virtio network card on the management
> platform,
> > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> of
> > >>>>>>>        the network card.
> > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > >>>>>>> user binds the net crad to a VF with identity through the command
> > >>>>>>>        of the patch
> > >>>>>>>
> > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > >>>>>>> ---
> > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > >>>>>>>
> > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > >>>>>>> 100644
> > >>>>>>> --- a/admin.tex
> > >>>>>>> +++ b/admin.tex
> > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>>>>>      \hline \hline
> > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> list of commands supported for this group type    \\
> > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> of commands used for this group type \\
> > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}    \\
> > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> one group member \\
> > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}    \\
> > >>>>>>>      \hline
> > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> using a different structure)    \\
> > >>>>>>>      \hline
> > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >>>>>>>
> > >>>>>>> +\subsubsection{Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > >>>>>>> +guest OS, so the virtio
> > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > >>>>>> not sure whether this is a security issue(affects host pci).
> > >>>>> No care about the passthrough, we always created VFs by the PF.
> > >>>>>
> > >>>>> I should not say "inside the guest OS". I just want to say that
> > >>>>> the VF is create by the user in the OS. The devices does not know
> about it.
> > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > >>> YES.
> > >>>
> > >>>
> > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > >>>>>>> +assigned a different role by the user, the virtio device can not
> choose one VF to bind random.
> > >>>>>> I failed to understand this, once a VF is created, it has a
> > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > >>>>>> and PF knows that.
> > >>>>>>
> > >>>>>> I am not familiar with the background, What do you mean by
> > >>>>>> virtio device choose one VF to bind?
> > >>>>> On the cloud, the nic is created by the management platform, the
> > >>>>> user can not create a new nic inside the OS.
> > >>>>>
> > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > >>>>> not backend virtio-net devices.
> > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > >>>> provisioned by the orchestration software and it assign properly
> > >>>> selected a VF to a guest on demands.
> > >>> Yes, but we do not need to care about the guest. Because VF may
> > >>> only be used in host, such as docker.
> > >>>
> > >>> The problem is that the user (you can think of this as the
> > >>> orchestration
> > >>> software) creates some VFs, these are only some PCI devices, which
> > >>> virtio devices will work on these VFs. I think that creating a vf
> > >>> and creating a virtio-net device are two different things. One is
> > >>> done by user in the OS, one is done on the management platform.  So we
> need to bind them together.
> > >> If the VFs are created through sriov_numvfs, once created, the VF
> > >> device and its personality are determined.
> > >>
> > >> PCI spec says:
> > >> All VFs associated with a PF must be the same device type as the
> > >> PF, (e.g., the same network device type or the same storage device
> > >> type.)
> > >>
> > >> So how can the creating process be splitted into separated steps?
> > >>
> > >> Are we discussing something beyond the spec?
> > > NO.
> > >
> > > The device types are same.
> > >
> > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > the cloud, these are managed by the management platform. On the
> > > cloud, there is an abstract object in the backend, which contains
> > > things that are generally configured on the management platform. It is
> something that users purchase.
> > > Under the virtio standard it is similar to device.
> > >
> > > In my understanding, we just created a pci vf, and virtio works on
> > > top of pci, so there must be two steps here (If I mistake, please
> > > point out.). When we create a vf, it doesn't mean that the backend
> > > deivce is ready. Of course, in some scenarios, we can immediately
> > > have a backend default device respond when the driver probe the vf.  But in
> our scenario, each device is independent.
> > Once a VF is crated, there comes with some default configurations,
> > like MTU and MAC.
> > Do you mean first step creation and second step initialize it?
> 
> Not exactly correct,
> 
> The first step is just to create a vf, at this time there can be a default virtio-net, it
> doesn't matter.
> 
> In the second step, we can bind a backend device to this vf.
> 
> Not just for initialization for new divice, we also want to support live migration.
> 
> For example, on the host, we create a vf and passthrough it into a guest os, this
> guest is migrated from another host, and its corresponding network card is also
> migrated to this host. We need to bind this vf to the migrated network card.
> 
> So just initialization is not enough.
>

Yet to catch up on the thread, so likely I am missing something.

The flow is for one OS (Linux) is:
1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
(num_vfs and vf enable bit in the PCI capability)

2. VFs are created at the PCI level in the host system and also inside the device

3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)

Between step #2 and #3, a user may configure one or multiple attributes of the VF.
This includes feature bits, config space fields, vf msix vectors and more.
This is to be using admin command.
These admin commands definition is due.


 
> Thanks
> 
> > If so, current spec only allow the user to config MAC through control vq.
> > vDPA allows to provision a device with proper configuration, maybe
> > that can be the solution?
> >
> > For binding, maybe the orchestration layer manages the pool and it
> > knows how to initialize the device
> >
> > Thanks
> > >
> > > Thanks.
> > >
> > >> Thanks
> > >>> Thanks.
> > >>>
> > >>>
> > >>>
> > >>>> So I am confused what the intention of this patch.
> > >>>>> Thanks.
> > >>>>>
> > >>>>>
> > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > >>>>>>> +On the other hand, generally the virtio devices are not
> > >>>>>>> +created by the user inside the guest OS. This requires some
> management platform to participate.
> > >>>>>>> +
> > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > >>>>>>> +the virtio devices.
> > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> help?
> > >>>>>> Provision a device on demands and assign it to a guest?
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>>> +
> > >>>>>>> +\begin{lstlisting}
> > >>>>>>> +struct virtio_admin_cmd_bind {
> > >>>>>>> +    u64 identity;
> > >>>>>>> +};
> > >>>>>>> +\end{lstlisting}
> > >>>>>>> +
> > >>>>>>> +The user got the \field{identity} from the management
> > >>>>>>> +platform, that is not included by this spec.
> > >>>>>>> +
> > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > >>>>>>> +/ Group administration commands / Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> \field{group_member_id} MUST be set.
> > >>>>>>> +
> > >>>>>>> +The \field{identity} is passed by the user. It is the
> > >>>>>>> +identity of the virtio device.
> > >>>>>>> +
> > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > >>>>>>> +/ Group administration commands / Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > >>>>>>> +
> > >>>>>>> +If the PF device can not find the device by the
> > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> VIRTIO_ADMIN_STATUS_EINVAL.
> > >>>>>>> +
> > >>>>>>> +If the device is found by the \field{identity}, the device
> > >>>>>>> +MUST work as the device of this group member specified by the
> \field{group_member_id}.
> > >>>>>>> +
> > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > >>>>>>>
> > >>>>>>>      An administration virtqueue of an owner device is used to
> > >>>>>>> submit
> > >>>>> This publicly archived list offers a means to provide input to
> > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>>
> > >>>>> In order to verify user consent to the Feedback License terms
> > >>>>> and to minimize spam in the list archive, subscription is
> > >>>>> required before posting.
> > >>>>>
> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>>> List archive:
> > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > >>>>> Feedback License:
> > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>>> List Guidelines:
> > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>>
> > >>>> This publicly archived list offers a means to provide input to
> > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>
> > >>>> In order to verify user consent to the Feedback License terms and
> > >>>> to minimize spam in the list archive, subscription is required
> > >>>> before posting.
> > >>>>
> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>> List archive:
> > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > >>>> Feedback License:
> > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>> List Guidelines:
> > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>
> > >>> ------------------------------------------------------------------
> > >>> --- To unsubscribe, e-mail:
> > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > >>> For additional commands, e-mail:
> > >>> virtio-dev-help@lists.oasis-open.org
> > >>>
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and to
> > > minimize spam in the list archive, subscription is required before
> > > posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License:
> > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines:
> > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Monday, June 26, 2023 6:51 AM
> 
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>
> > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> <lingshan.zhu@intel.com> wrote:
> > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> can not choose one VF to bind random.
> > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > >>>>>>> On the other hand, generally the virtio devices are not
> > >>>>>>> created by the user inside the guest OS. This requires some
> management platform to participate.
> > >>>>>>>
> > >>>>>>> So the usage of this command:
> > >>>>>>> 1. The user purchases a virtio network card on the management
> platform,
> > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> of
> > >>>>>>>        the network card.
> > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > >>>>>>> user binds the net crad to a VF with identity through the command
> > >>>>>>>        of the patch
> > >>>>>>>
> > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > >>>>>>> ---
> > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > >>>>>>>
> > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > >>>>>>> 100644
> > >>>>>>> --- a/admin.tex
> > >>>>>>> +++ b/admin.tex
> > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>>>>>      \hline \hline
> > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> list of commands supported for this group type    \\
> > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> of commands used for this group type \\
> > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}    \\
> > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> one group member \\
> > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> virtio_admin_cmd}    \\
> > >>>>>>>      \hline
> > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> using a different structure)    \\
> > >>>>>>>      \hline
> > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> commands}\label{sec:Basic Facilities of a Virti
> > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >>>>>>>
> > >>>>>>> +\subsubsection{Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > >>>>>>> +guest OS, so the virtio
> > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > >>>>>> not sure whether this is a security issue(affects host pci).
> > >>>>> No care about the passthrough, we always created VFs by the PF.
> > >>>>>
> > >>>>> I should not say "inside the guest OS". I just want to say that
> > >>>>> the VF is create by the user in the OS. The devices does not know
> about it.
> > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > >>> YES.
> > >>>
> > >>>
> > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > >>>>>>> +assigned a different role by the user, the virtio device can not
> choose one VF to bind random.
> > >>>>>> I failed to understand this, once a VF is created, it has a
> > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > >>>>>> and PF knows that.
> > >>>>>>
> > >>>>>> I am not familiar with the background, What do you mean by
> > >>>>>> virtio device choose one VF to bind?
> > >>>>> On the cloud, the nic is created by the management platform, the
> > >>>>> user can not create a new nic inside the OS.
> > >>>>>
> > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > >>>>> not backend virtio-net devices.
> > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > >>>> provisioned by the orchestration software and it assign properly
> > >>>> selected a VF to a guest on demands.
> > >>> Yes, but we do not need to care about the guest. Because VF may
> > >>> only be used in host, such as docker.
> > >>>
> > >>> The problem is that the user (you can think of this as the
> > >>> orchestration
> > >>> software) creates some VFs, these are only some PCI devices, which
> > >>> virtio devices will work on these VFs. I think that creating a vf
> > >>> and creating a virtio-net device are two different things. One is
> > >>> done by user in the OS, one is done on the management platform.  So we
> need to bind them together.
> > >> If the VFs are created through sriov_numvfs, once created, the VF
> > >> device and its personality are determined.
> > >>
> > >> PCI spec says:
> > >> All VFs associated with a PF must be the same device type as the
> > >> PF, (e.g., the same network device type or the same storage device
> > >> type.)
> > >>
> > >> So how can the creating process be splitted into separated steps?
> > >>
> > >> Are we discussing something beyond the spec?
> > > NO.
> > >
> > > The device types are same.
> > >
> > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > the cloud, these are managed by the management platform. On the
> > > cloud, there is an abstract object in the backend, which contains
> > > things that are generally configured on the management platform. It is
> something that users purchase.
> > > Under the virtio standard it is similar to device.
> > >
> > > In my understanding, we just created a pci vf, and virtio works on
> > > top of pci, so there must be two steps here (If I mistake, please
> > > point out.). When we create a vf, it doesn't mean that the backend
> > > deivce is ready. Of course, in some scenarios, we can immediately
> > > have a backend default device respond when the driver probe the vf.  But in
> our scenario, each device is independent.
> > Once a VF is crated, there comes with some default configurations,
> > like MTU and MAC.
> > Do you mean first step creation and second step initialize it?
> 
> Not exactly correct,
> 
> The first step is just to create a vf, at this time there can be a default virtio-net, it
> doesn't matter.
> 
> In the second step, we can bind a backend device to this vf.
> 
> Not just for initialization for new divice, we also want to support live migration.
> 
> For example, on the host, we create a vf and passthrough it into a guest os, this
> guest is migrated from another host, and its corresponding network card is also
> migrated to this host. We need to bind this vf to the migrated network card.
> 
> So just initialization is not enough.
>

Yet to catch up on the thread, so likely I am missing something.

The flow is for one OS (Linux) is:
1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
(num_vfs and vf enable bit in the PCI capability)

2. VFs are created at the PCI level in the host system and also inside the device

3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)

Between step #2 and #3, a user may configure one or multiple attributes of the VF.
This includes feature bits, config space fields, vf msix vectors and more.
This is to be using admin command.
These admin commands definition is due.


 
> Thanks
> 
> > If so, current spec only allow the user to config MAC through control vq.
> > vDPA allows to provision a device with proper configuration, maybe
> > that can be the solution?
> >
> > For binding, maybe the orchestration layer manages the pool and it
> > knows how to initialize the device
> >
> > Thanks
> > >
> > > Thanks.
> > >
> > >> Thanks
> > >>> Thanks.
> > >>>
> > >>>
> > >>>
> > >>>> So I am confused what the intention of this patch.
> > >>>>> Thanks.
> > >>>>>
> > >>>>>
> > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > >>>>>>> +On the other hand, generally the virtio devices are not
> > >>>>>>> +created by the user inside the guest OS. This requires some
> management platform to participate.
> > >>>>>>> +
> > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > >>>>>>> +the virtio devices.
> > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> help?
> > >>>>>> Provision a device on demands and assign it to a guest?
> > >>>>>>
> > >>>>>> Thanks
> > >>>>>>> +
> > >>>>>>> +\begin{lstlisting}
> > >>>>>>> +struct virtio_admin_cmd_bind {
> > >>>>>>> +    u64 identity;
> > >>>>>>> +};
> > >>>>>>> +\end{lstlisting}
> > >>>>>>> +
> > >>>>>>> +The user got the \field{identity} from the management
> > >>>>>>> +platform, that is not included by this spec.
> > >>>>>>> +
> > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > >>>>>>> +/ Group administration commands / Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> \field{group_member_id} MUST be set.
> > >>>>>>> +
> > >>>>>>> +The \field{identity} is passed by the user. It is the
> > >>>>>>> +identity of the virtio device.
> > >>>>>>> +
> > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > >>>>>>> +/ Group administration commands / Bind the device for member}
> > >>>>>>> +
> > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > >>>>>>> +
> > >>>>>>> +If the PF device can not find the device by the
> > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> VIRTIO_ADMIN_STATUS_EINVAL.
> > >>>>>>> +
> > >>>>>>> +If the device is found by the \field{identity}, the device
> > >>>>>>> +MUST work as the device of this group member specified by the
> \field{group_member_id}.
> > >>>>>>> +
> > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > >>>>>>>
> > >>>>>>>      An administration virtqueue of an owner device is used to
> > >>>>>>> submit
> > >>>>> This publicly archived list offers a means to provide input to
> > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>>
> > >>>>> In order to verify user consent to the Feedback License terms
> > >>>>> and to minimize spam in the list archive, subscription is
> > >>>>> required before posting.
> > >>>>>
> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>>> List archive:
> > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > >>>>> Feedback License:
> > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>>> List Guidelines:
> > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>>
> > >>>> This publicly archived list offers a means to provide input to
> > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>
> > >>>> In order to verify user consent to the Feedback License terms and
> > >>>> to minimize spam in the list archive, subscription is required
> > >>>> before posting.
> > >>>>
> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>> List archive:
> > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > >>>> Feedback License:
> > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>> List Guidelines:
> > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>
> > >>> ------------------------------------------------------------------
> > >>> --- To unsubscribe, e-mail:
> > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > >>> For additional commands, e-mail:
> > >>> virtio-dev-help@lists.oasis-open.org
> > >>>
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and to
> > > minimize spam in the list archive, subscription is required before
> > > posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License:
> > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines:
> > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 12:19:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Monday, June 26, 2023 6:51 AM
> >
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > can not choose one VF to bind random.
> > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> On the other hand, generally the virtio devices are not
> > > >>>>>>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>>
> > > >>>>>>> So the usage of this command:
> > > >>>>>>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > of
> > > >>>>>>>        the network card.
> > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > >>>>>>>        of the patch
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > >>>>>>> ---
> > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>>>>>
> > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>>>>>> 100644
> > > >>>>>>> --- a/admin.tex
> > > >>>>>>> +++ b/admin.tex
> > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \hline \hline
> > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type    \\
> > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>>      \hline
> > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > using a different structure)    \\
> > > >>>>>>>      \hline
> > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>>>>>
> > > >>>>>>> +\subsubsection{Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > >>>>>>> +guest OS, so the virtio
> > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > >>>>>
> > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > >>>>> the VF is create by the user in the OS. The devices does not know
> > about it.
> > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >>>>>> and PF knows that.
> > > >>>>>>
> > > >>>>>> I am not familiar with the background, What do you mean by
> > > >>>>>> virtio device choose one VF to bind?
> > > >>>>> On the cloud, the nic is created by the management platform, the
> > > >>>>> user can not create a new nic inside the OS.
> > > >>>>>
> > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > >>>>> not backend virtio-net devices.
> > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > >>>> provisioned by the orchestration software and it assign properly
> > > >>>> selected a VF to a guest on demands.
> > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > >>> only be used in host, such as docker.
> > > >>>
> > > >>> The problem is that the user (you can think of this as the
> > > >>> orchestration
> > > >>> software) creates some VFs, these are only some PCI devices, which
> > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > >>> and creating a virtio-net device are two different things. One is
> > > >>> done by user in the OS, one is done on the management platform.  So we
> > need to bind them together.
> > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > >> device and its personality are determined.
> > > >>
> > > >> PCI spec says:
> > > >> All VFs associated with a PF must be the same device type as the
> > > >> PF, (e.g., the same network device type or the same storage device
> > > >> type.)
> > > >>
> > > >> So how can the creating process be splitted into separated steps?
> > > >>
> > > >> Are we discussing something beyond the spec?
> > > > NO.
> > > >
> > > > The device types are same.
> > > >
> > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > the cloud, these are managed by the management platform. On the
> > > > cloud, there is an abstract object in the backend, which contains
> > > > things that are generally configured on the management platform. It is
> > something that users purchase.
> > > > Under the virtio standard it is similar to device.
> > > >
> > > > In my understanding, we just created a pci vf, and virtio works on
> > > > top of pci, so there must be two steps here (If I mistake, please
> > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > have a backend default device respond when the driver probe the vf.  But in
> > our scenario, each device is independent.
> > > Once a VF is crated, there comes with some default configurations,
> > > like MTU and MAC.
> > > Do you mean first step creation and second step initialize it?
> >
> > Not exactly correct,
> >
> > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > doesn't matter.
> >
> > In the second step, we can bind a backend device to this vf.
> >
> > Not just for initialization for new divice, we also want to support live migration.
> >
> > For example, on the host, we create a vf and passthrough it into a guest os, this
> > guest is migrated from another host, and its corresponding network card is also
> > migrated to this host. We need to bind this vf to the migrated network card.
> >
> > So just initialization is not enough.
> >
>
> Yet to catch up on the thread, so likely I am missing something.
>
> The flow is for one OS (Linux) is:
> 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> (num_vfs and vf enable bit in the PCI capability)
>
> 2. VFs are created at the PCI level in the host system and also inside the device
>
> 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
>
> Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> This includes feature bits, config space fields, vf msix vectors and more.
> This is to be using admin command.
> These admin commands definition is due.


YES

My problem is that the nic is created by user on the management platform.  Or
the device is migrated from other host.

So we need to bind the device(created by user) to the vf between step #2 and #3.

Thanks.


>
>
>
> > Thanks
> >
> > > If so, current spec only allow the user to config MAC through control vq.
> > > vDPA allows to provision a device with proper configuration, maybe
> > > that can be the solution?
> > >
> > > For binding, maybe the orchestration layer manages the pool and it
> > > knows how to initialize the device
> > >
> > > Thanks
> > > >
> > > > Thanks.
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> So I am confused what the intention of this patch.
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > >>>>>>> +created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>> +
> > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > >>>>>>> +the virtio devices.
> > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > help?
> > > >>>>>> Provision a device on demands and assign it to a guest?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>> +
> > > >>>>>>> +\begin{lstlisting}
> > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > >>>>>>> +    u64 identity;
> > > >>>>>>> +};
> > > >>>>>>> +\end{lstlisting}
> > > >>>>>>> +
> > > >>>>>>> +The user got the \field{identity} from the management
> > > >>>>>>> +platform, that is not included by this spec.
> > > >>>>>>> +
> > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > \field{group_member_id} MUST be set.
> > > >>>>>>> +
> > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > >>>>>>> +identity of the virtio device.
> > > >>>>>>> +
> > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > >>>>>>> +
> > > >>>>>>> +If the PF device can not find the device by the
> > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > VIRTIO_ADMIN_STATUS_EINVAL.
> > > >>>>>>> +
> > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > >>>>>>> +MUST work as the device of this group member specified by the
> > \field{group_member_id}.
> > > >>>>>>> +
> > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > >>>>>>>
> > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > >>>>>>> submit
> > > >>>>> This publicly archived list offers a means to provide input to
> > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms
> > > >>>>> and to minimize spam in the list archive, subscription is
> > > >>>>> required before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive:
> > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License:
> > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines:
> > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to
> > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive:
> > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License:
> > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines:
> > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > > >>> ------------------------------------------------------------------
> > > >>> --- To unsubscribe, e-mail:
> > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > >>> For additional commands, e-mail:
> > > >>> virtio-dev-help@lists.oasis-open.org
> > > >>>
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and to
> > > > minimize spam in the list archive, subscription is required before
> > > > posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 12:19:04 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Monday, June 26, 2023 6:51 AM
> >
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > can not choose one VF to bind random.
> > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> On the other hand, generally the virtio devices are not
> > > >>>>>>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>>
> > > >>>>>>> So the usage of this command:
> > > >>>>>>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > of
> > > >>>>>>>        the network card.
> > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > >>>>>>>        of the patch
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > >>>>>>> ---
> > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>>>>>
> > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>>>>>> 100644
> > > >>>>>>> --- a/admin.tex
> > > >>>>>>> +++ b/admin.tex
> > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \hline \hline
> > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type    \\
> > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>>      \hline
> > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > using a different structure)    \\
> > > >>>>>>>      \hline
> > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>>>>>
> > > >>>>>>> +\subsubsection{Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > >>>>>>> +guest OS, so the virtio
> > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > >>>>>
> > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > >>>>> the VF is create by the user in the OS. The devices does not know
> > about it.
> > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >>>>>> and PF knows that.
> > > >>>>>>
> > > >>>>>> I am not familiar with the background, What do you mean by
> > > >>>>>> virtio device choose one VF to bind?
> > > >>>>> On the cloud, the nic is created by the management platform, the
> > > >>>>> user can not create a new nic inside the OS.
> > > >>>>>
> > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > >>>>> not backend virtio-net devices.
> > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > >>>> provisioned by the orchestration software and it assign properly
> > > >>>> selected a VF to a guest on demands.
> > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > >>> only be used in host, such as docker.
> > > >>>
> > > >>> The problem is that the user (you can think of this as the
> > > >>> orchestration
> > > >>> software) creates some VFs, these are only some PCI devices, which
> > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > >>> and creating a virtio-net device are two different things. One is
> > > >>> done by user in the OS, one is done on the management platform.  So we
> > need to bind them together.
> > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > >> device and its personality are determined.
> > > >>
> > > >> PCI spec says:
> > > >> All VFs associated with a PF must be the same device type as the
> > > >> PF, (e.g., the same network device type or the same storage device
> > > >> type.)
> > > >>
> > > >> So how can the creating process be splitted into separated steps?
> > > >>
> > > >> Are we discussing something beyond the spec?
> > > > NO.
> > > >
> > > > The device types are same.
> > > >
> > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > the cloud, these are managed by the management platform. On the
> > > > cloud, there is an abstract object in the backend, which contains
> > > > things that are generally configured on the management platform. It is
> > something that users purchase.
> > > > Under the virtio standard it is similar to device.
> > > >
> > > > In my understanding, we just created a pci vf, and virtio works on
> > > > top of pci, so there must be two steps here (If I mistake, please
> > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > have a backend default device respond when the driver probe the vf.  But in
> > our scenario, each device is independent.
> > > Once a VF is crated, there comes with some default configurations,
> > > like MTU and MAC.
> > > Do you mean first step creation and second step initialize it?
> >
> > Not exactly correct,
> >
> > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > doesn't matter.
> >
> > In the second step, we can bind a backend device to this vf.
> >
> > Not just for initialization for new divice, we also want to support live migration.
> >
> > For example, on the host, we create a vf and passthrough it into a guest os, this
> > guest is migrated from another host, and its corresponding network card is also
> > migrated to this host. We need to bind this vf to the migrated network card.
> >
> > So just initialization is not enough.
> >
>
> Yet to catch up on the thread, so likely I am missing something.
>
> The flow is for one OS (Linux) is:
> 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> (num_vfs and vf enable bit in the PCI capability)
>
> 2. VFs are created at the PCI level in the host system and also inside the device
>
> 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
>
> Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> This includes feature bits, config space fields, vf msix vectors and more.
> This is to be using admin command.
> These admin commands definition is due.


YES

My problem is that the nic is created by user on the management platform.  Or
the device is migrated from other host.

So we need to bind the device(created by user) to the vf between step #2 and #3.

Thanks.


>
>
>
> > Thanks
> >
> > > If so, current spec only allow the user to config MAC through control vq.
> > > vDPA allows to provision a device with proper configuration, maybe
> > > that can be the solution?
> > >
> > > For binding, maybe the orchestration layer manages the pool and it
> > > knows how to initialize the device
> > >
> > > Thanks
> > > >
> > > > Thanks.
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> So I am confused what the intention of this patch.
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > >>>>>>> +created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>> +
> > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > >>>>>>> +the virtio devices.
> > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > help?
> > > >>>>>> Provision a device on demands and assign it to a guest?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>> +
> > > >>>>>>> +\begin{lstlisting}
> > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > >>>>>>> +    u64 identity;
> > > >>>>>>> +};
> > > >>>>>>> +\end{lstlisting}
> > > >>>>>>> +
> > > >>>>>>> +The user got the \field{identity} from the management
> > > >>>>>>> +platform, that is not included by this spec.
> > > >>>>>>> +
> > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > \field{group_member_id} MUST be set.
> > > >>>>>>> +
> > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > >>>>>>> +identity of the virtio device.
> > > >>>>>>> +
> > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > >>>>>>> +
> > > >>>>>>> +If the PF device can not find the device by the
> > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > VIRTIO_ADMIN_STATUS_EINVAL.
> > > >>>>>>> +
> > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > >>>>>>> +MUST work as the device of this group member specified by the
> > \field{group_member_id}.
> > > >>>>>>> +
> > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > >>>>>>>
> > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > >>>>>>> submit
> > > >>>>> This publicly archived list offers a means to provide input to
> > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms
> > > >>>>> and to minimize spam in the list archive, subscription is
> > > >>>>> required before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive:
> > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License:
> > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines:
> > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to
> > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive:
> > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License:
> > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines:
> > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > > >>> ------------------------------------------------------------------
> > > >>> --- To unsubscribe, e-mail:
> > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > >>> For additional commands, e-mail:
> > > >>> virtio-dev-help@lists.oasis-open.org
> > > >>>
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and to
> > > > minimize spam in the list archive, subscription is required before
> > > > posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 12:19:04PM +0000, Parav Pandit wrote:
> 
> 
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Monday, June 26, 2023 6:51 AM
> > 
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > can not choose one VF to bind random.
> > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> On the other hand, generally the virtio devices are not
> > > >>>>>>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>>
> > > >>>>>>> So the usage of this command:
> > > >>>>>>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > of
> > > >>>>>>>        the network card.
> > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > >>>>>>>        of the patch
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > >>>>>>> ---
> > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>>>>>
> > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>>>>>> 100644
> > > >>>>>>> --- a/admin.tex
> > > >>>>>>> +++ b/admin.tex
> > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \hline \hline
> > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type    \\
> > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>>      \hline
> > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > using a different structure)    \\
> > > >>>>>>>      \hline
> > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>>>>>
> > > >>>>>>> +\subsubsection{Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > >>>>>>> +guest OS, so the virtio
> > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > >>>>>
> > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > >>>>> the VF is create by the user in the OS. The devices does not know
> > about it.
> > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >>>>>> and PF knows that.
> > > >>>>>>
> > > >>>>>> I am not familiar with the background, What do you mean by
> > > >>>>>> virtio device choose one VF to bind?
> > > >>>>> On the cloud, the nic is created by the management platform, the
> > > >>>>> user can not create a new nic inside the OS.
> > > >>>>>
> > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > >>>>> not backend virtio-net devices.
> > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > >>>> provisioned by the orchestration software and it assign properly
> > > >>>> selected a VF to a guest on demands.
> > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > >>> only be used in host, such as docker.
> > > >>>
> > > >>> The problem is that the user (you can think of this as the
> > > >>> orchestration
> > > >>> software) creates some VFs, these are only some PCI devices, which
> > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > >>> and creating a virtio-net device are two different things. One is
> > > >>> done by user in the OS, one is done on the management platform.  So we
> > need to bind them together.
> > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > >> device and its personality are determined.
> > > >>
> > > >> PCI spec says:
> > > >> All VFs associated with a PF must be the same device type as the
> > > >> PF, (e.g., the same network device type or the same storage device
> > > >> type.)
> > > >>
> > > >> So how can the creating process be splitted into separated steps?
> > > >>
> > > >> Are we discussing something beyond the spec?
> > > > NO.
> > > >
> > > > The device types are same.
> > > >
> > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > the cloud, these are managed by the management platform. On the
> > > > cloud, there is an abstract object in the backend, which contains
> > > > things that are generally configured on the management platform. It is
> > something that users purchase.
> > > > Under the virtio standard it is similar to device.
> > > >
> > > > In my understanding, we just created a pci vf, and virtio works on
> > > > top of pci, so there must be two steps here (If I mistake, please
> > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > have a backend default device respond when the driver probe the vf.  But in
> > our scenario, each device is independent.
> > > Once a VF is crated, there comes with some default configurations,
> > > like MTU and MAC.
> > > Do you mean first step creation and second step initialize it?
> > 
> > Not exactly correct,
> > 
> > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > doesn't matter.
> > 
> > In the second step, we can bind a backend device to this vf.
> > 
> > Not just for initialization for new divice, we also want to support live migration.
> > 
> > For example, on the host, we create a vf and passthrough it into a guest os, this
> > guest is migrated from another host, and its corresponding network card is also
> > migrated to this host. We need to bind this vf to the migrated network card.
> > 
> > So just initialization is not enough.
> >
> 
> Yet to catch up on the thread, so likely I am missing something.
> 
> The flow is for one OS (Linux) is:
> 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> (num_vfs and vf enable bit in the PCI capability)
> 
> 2. VFs are created at the PCI level in the host system and also inside the device
> 
> 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
> 
> Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> This includes feature bits, config space fields, vf msix vectors and more.
> This is to be using admin command.
> These admin commands definition is due.

To be frank, I am not sure binding to an ID necessarily needs to be
gated on provisioning commands.
What was not explained at all is what purpose does this extra level
of indirection serve.


> 
>  
> > Thanks
> > 
> > > If so, current spec only allow the user to config MAC through control vq.
> > > vDPA allows to provision a device with proper configuration, maybe
> > > that can be the solution?
> > >
> > > For binding, maybe the orchestration layer manages the pool and it
> > > knows how to initialize the device
> > >
> > > Thanks
> > > >
> > > > Thanks.
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> So I am confused what the intention of this patch.
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > >>>>>>> +created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>> +
> > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > >>>>>>> +the virtio devices.
> > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > help?
> > > >>>>>> Provision a device on demands and assign it to a guest?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>> +
> > > >>>>>>> +\begin{lstlisting}
> > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > >>>>>>> +    u64 identity;
> > > >>>>>>> +};
> > > >>>>>>> +\end{lstlisting}
> > > >>>>>>> +
> > > >>>>>>> +The user got the \field{identity} from the management
> > > >>>>>>> +platform, that is not included by this spec.
> > > >>>>>>> +
> > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > \field{group_member_id} MUST be set.
> > > >>>>>>> +
> > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > >>>>>>> +identity of the virtio device.
> > > >>>>>>> +
> > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > >>>>>>> +
> > > >>>>>>> +If the PF device can not find the device by the
> > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > VIRTIO_ADMIN_STATUS_EINVAL.
> > > >>>>>>> +
> > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > >>>>>>> +MUST work as the device of this group member specified by the
> > \field{group_member_id}.
> > > >>>>>>> +
> > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > >>>>>>>
> > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > >>>>>>> submit
> > > >>>>> This publicly archived list offers a means to provide input to
> > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms
> > > >>>>> and to minimize spam in the list archive, subscription is
> > > >>>>> required before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive:
> > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License:
> > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines:
> > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to
> > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive:
> > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License:
> > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines:
> > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > > >>> ------------------------------------------------------------------
> > > >>> --- To unsubscribe, e-mail:
> > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > >>> For additional commands, e-mail:
> > > >>> virtio-dev-help@lists.oasis-open.org
> > > >>>
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and to
> > > > minimize spam in the list archive, subscription is required before
> > > > posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 12:19:04PM +0000, Parav Pandit wrote:
> 
> 
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Monday, June 26, 2023 6:51 AM
> > 
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > can not choose one VF to bind random.
> > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> On the other hand, generally the virtio devices are not
> > > >>>>>>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>>
> > > >>>>>>> So the usage of this command:
> > > >>>>>>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > of
> > > >>>>>>>        the network card.
> > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > >>>>>>>        of the patch
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > >>>>>>> ---
> > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>>>>>
> > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>>>>>> 100644
> > > >>>>>>> --- a/admin.tex
> > > >>>>>>> +++ b/admin.tex
> > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \hline \hline
> > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type    \\
> > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>>      \hline
> > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > using a different structure)    \\
> > > >>>>>>>      \hline
> > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>>>>>
> > > >>>>>>> +\subsubsection{Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > >>>>>>> +guest OS, so the virtio
> > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > >>>>>
> > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > >>>>> the VF is create by the user in the OS. The devices does not know
> > about it.
> > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >>>>>> and PF knows that.
> > > >>>>>>
> > > >>>>>> I am not familiar with the background, What do you mean by
> > > >>>>>> virtio device choose one VF to bind?
> > > >>>>> On the cloud, the nic is created by the management platform, the
> > > >>>>> user can not create a new nic inside the OS.
> > > >>>>>
> > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > >>>>> not backend virtio-net devices.
> > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > >>>> provisioned by the orchestration software and it assign properly
> > > >>>> selected a VF to a guest on demands.
> > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > >>> only be used in host, such as docker.
> > > >>>
> > > >>> The problem is that the user (you can think of this as the
> > > >>> orchestration
> > > >>> software) creates some VFs, these are only some PCI devices, which
> > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > >>> and creating a virtio-net device are two different things. One is
> > > >>> done by user in the OS, one is done on the management platform.  So we
> > need to bind them together.
> > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > >> device and its personality are determined.
> > > >>
> > > >> PCI spec says:
> > > >> All VFs associated with a PF must be the same device type as the
> > > >> PF, (e.g., the same network device type or the same storage device
> > > >> type.)
> > > >>
> > > >> So how can the creating process be splitted into separated steps?
> > > >>
> > > >> Are we discussing something beyond the spec?
> > > > NO.
> > > >
> > > > The device types are same.
> > > >
> > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > the cloud, these are managed by the management platform. On the
> > > > cloud, there is an abstract object in the backend, which contains
> > > > things that are generally configured on the management platform. It is
> > something that users purchase.
> > > > Under the virtio standard it is similar to device.
> > > >
> > > > In my understanding, we just created a pci vf, and virtio works on
> > > > top of pci, so there must be two steps here (If I mistake, please
> > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > have a backend default device respond when the driver probe the vf.  But in
> > our scenario, each device is independent.
> > > Once a VF is crated, there comes with some default configurations,
> > > like MTU and MAC.
> > > Do you mean first step creation and second step initialize it?
> > 
> > Not exactly correct,
> > 
> > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > doesn't matter.
> > 
> > In the second step, we can bind a backend device to this vf.
> > 
> > Not just for initialization for new divice, we also want to support live migration.
> > 
> > For example, on the host, we create a vf and passthrough it into a guest os, this
> > guest is migrated from another host, and its corresponding network card is also
> > migrated to this host. We need to bind this vf to the migrated network card.
> > 
> > So just initialization is not enough.
> >
> 
> Yet to catch up on the thread, so likely I am missing something.
> 
> The flow is for one OS (Linux) is:
> 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> (num_vfs and vf enable bit in the PCI capability)
> 
> 2. VFs are created at the PCI level in the host system and also inside the device
> 
> 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
> 
> Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> This includes feature bits, config space fields, vf msix vectors and more.
> This is to be using admin command.
> These admin commands definition is due.

To be frank, I am not sure binding to an ID necessarily needs to be
gated on provisioning commands.
What was not explained at all is what purpose does this extra level
of indirection serve.


> 
>  
> > Thanks
> > 
> > > If so, current spec only allow the user to config MAC through control vq.
> > > vDPA allows to provision a device with proper configuration, maybe
> > > that can be the solution?
> > >
> > > For binding, maybe the orchestration layer manages the pool and it
> > > knows how to initialize the device
> > >
> > > Thanks
> > > >
> > > > Thanks.
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> So I am confused what the intention of this patch.
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > >>>>>>> +created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>> +
> > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > >>>>>>> +the virtio devices.
> > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > help?
> > > >>>>>> Provision a device on demands and assign it to a guest?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>> +
> > > >>>>>>> +\begin{lstlisting}
> > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > >>>>>>> +    u64 identity;
> > > >>>>>>> +};
> > > >>>>>>> +\end{lstlisting}
> > > >>>>>>> +
> > > >>>>>>> +The user got the \field{identity} from the management
> > > >>>>>>> +platform, that is not included by this spec.
> > > >>>>>>> +
> > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > \field{group_member_id} MUST be set.
> > > >>>>>>> +
> > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > >>>>>>> +identity of the virtio device.
> > > >>>>>>> +
> > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > >>>>>>> +
> > > >>>>>>> +If the PF device can not find the device by the
> > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > VIRTIO_ADMIN_STATUS_EINVAL.
> > > >>>>>>> +
> > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > >>>>>>> +MUST work as the device of this group member specified by the
> > \field{group_member_id}.
> > > >>>>>>> +
> > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > >>>>>>>
> > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > >>>>>>> submit
> > > >>>>> This publicly archived list offers a means to provide input to
> > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms
> > > >>>>> and to minimize spam in the list archive, subscription is
> > > >>>>> required before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive:
> > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License:
> > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines:
> > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to
> > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive:
> > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License:
> > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines:
> > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > > >>> ------------------------------------------------------------------
> > > >>> --- To unsubscribe, e-mail:
> > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > >>> For additional commands, e-mail:
> > > >>> virtio-dev-help@lists.oasis-open.org
> > > >>>
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and to
> > > > minimize spam in the list archive, subscription is required before
> > > > posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 08:35:08 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 12:19:04PM +0000, Parav Pandit wrote:
> >
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Monday, June 26, 2023 6:51 AM
> > >
> > > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > >
> > > >
> > > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>
> > > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > > can not choose one VF to bind random.
> > > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > > >>>>>>> On the other hand, generally the virtio devices are not
> > > > >>>>>>> created by the user inside the guest OS. This requires some
> > > management platform to participate.
> > > > >>>>>>>
> > > > >>>>>>> So the usage of this command:
> > > > >>>>>>> 1. The user purchases a virtio network card on the management
> > > platform,
> > > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > > of
> > > > >>>>>>>        the network card.
> > > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > > >>>>>>>        of the patch
> > > > >>>>>>>
> > > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > >>>>>>> ---
> > > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > > >>>>>>>
> > > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > > >>>>>>> 100644
> > > > >>>>>>> --- a/admin.tex
> > > > >>>>>>> +++ b/admin.tex
> > > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > > commands}\label{sec:Basic Facilities of a Virti
> > > > >>>>>>>      \hline \hline
> > > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > > list of commands supported for this group type    \\
> > > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > > of commands used for this group type \\
> > > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > > virtio_admin_cmd}    \\
> > > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > > one group member \\
> > > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > > virtio_admin_cmd}    \\
> > > > >>>>>>>      \hline
> > > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > > using a different structure)    \\
> > > > >>>>>>>      \hline
> > > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > > commands}\label{sec:Basic Facilities of a Virti
> > > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > > >>>>>>>
> > > > >>>>>>> +\subsubsection{Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > > >>>>>>> +guest OS, so the virtio
> > > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > > >>>>>
> > > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > > >>>>> the VF is create by the user in the OS. The devices does not know
> > > about it.
> > > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > > >>> YES.
> > > > >>>
> > > > >>>
> > > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > > choose one VF to bind random.
> > > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > > >>>>>> and PF knows that.
> > > > >>>>>>
> > > > >>>>>> I am not familiar with the background, What do you mean by
> > > > >>>>>> virtio device choose one VF to bind?
> > > > >>>>> On the cloud, the nic is created by the management platform, the
> > > > >>>>> user can not create a new nic inside the OS.
> > > > >>>>>
> > > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > > >>>>> not backend virtio-net devices.
> > > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > > >>>> provisioned by the orchestration software and it assign properly
> > > > >>>> selected a VF to a guest on demands.
> > > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > > >>> only be used in host, such as docker.
> > > > >>>
> > > > >>> The problem is that the user (you can think of this as the
> > > > >>> orchestration
> > > > >>> software) creates some VFs, these are only some PCI devices, which
> > > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > > >>> and creating a virtio-net device are two different things. One is
> > > > >>> done by user in the OS, one is done on the management platform.  So we
> > > need to bind them together.
> > > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > > >> device and its personality are determined.
> > > > >>
> > > > >> PCI spec says:
> > > > >> All VFs associated with a PF must be the same device type as the
> > > > >> PF, (e.g., the same network device type or the same storage device
> > > > >> type.)
> > > > >>
> > > > >> So how can the creating process be splitted into separated steps?
> > > > >>
> > > > >> Are we discussing something beyond the spec?
> > > > > NO.
> > > > >
> > > > > The device types are same.
> > > > >
> > > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > > the cloud, these are managed by the management platform. On the
> > > > > cloud, there is an abstract object in the backend, which contains
> > > > > things that are generally configured on the management platform. It is
> > > something that users purchase.
> > > > > Under the virtio standard it is similar to device.
> > > > >
> > > > > In my understanding, we just created a pci vf, and virtio works on
> > > > > top of pci, so there must be two steps here (If I mistake, please
> > > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > > have a backend default device respond when the driver probe the vf.  But in
> > > our scenario, each device is independent.
> > > > Once a VF is crated, there comes with some default configurations,
> > > > like MTU and MAC.
> > > > Do you mean first step creation and second step initialize it?
> > >
> > > Not exactly correct,
> > >
> > > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > > doesn't matter.
> > >
> > > In the second step, we can bind a backend device to this vf.
> > >
> > > Not just for initialization for new divice, we also want to support live migration.
> > >
> > > For example, on the host, we create a vf and passthrough it into a guest os, this
> > > guest is migrated from another host, and its corresponding network card is also
> > > migrated to this host. We need to bind this vf to the migrated network card.
> > >
> > > So just initialization is not enough.
> > >
> >
> > Yet to catch up on the thread, so likely I am missing something.
> >
> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside the device
> >
> > 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
>
> To be frank, I am not sure binding to an ID necessarily needs to be
> gated on provisioning commands.
> What was not explained at all is what purpose does this extra level
> of indirection serve.


I think the first question we need to solve is how to configure the back-end
devices corresponding to all vfs. Who will configure the queue num, msix
vectors and etc of these devices.

In the cloud, all these must go through the management platform.

Thanks.

>
>
> >
> >
> > > Thanks
> > >
> > > > If so, current spec only allow the user to config MAC through control vq.
> > > > vDPA allows to provision a device with proper configuration, maybe
> > > > that can be the solution?
> > > >
> > > > For binding, maybe the orchestration layer manages the pool and it
> > > > knows how to initialize the device
> > > >
> > > > Thanks
> > > > >
> > > > > Thanks.
> > > > >
> > > > >> Thanks
> > > > >>> Thanks.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> So I am confused what the intention of this patch.
> > > > >>>>> Thanks.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > > >>>>>>> +created by the user inside the guest OS. This requires some
> > > management platform to participate.
> > > > >>>>>>> +
> > > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > > >>>>>>> +the virtio devices.
> > > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > > help?
> > > > >>>>>> Provision a device on demands and assign it to a guest?
> > > > >>>>>>
> > > > >>>>>> Thanks
> > > > >>>>>>> +
> > > > >>>>>>> +\begin{lstlisting}
> > > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > > >>>>>>> +    u64 identity;
> > > > >>>>>>> +};
> > > > >>>>>>> +\end{lstlisting}
> > > > >>>>>>> +
> > > > >>>>>>> +The user got the \field{identity} from the management
> > > > >>>>>>> +platform, that is not included by this spec.
> > > > >>>>>>> +
> > > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > > \field{group_member_id} MUST be set.
> > > > >>>>>>> +
> > > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > > >>>>>>> +identity of the virtio device.
> > > > >>>>>>> +
> > > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > > >>>>>>> +
> > > > >>>>>>> +If the PF device can not find the device by the
> > > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > > VIRTIO_ADMIN_STATUS_EINVAL.
> > > > >>>>>>> +
> > > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > > >>>>>>> +MUST work as the device of this group member specified by the
> > > \field{group_member_id}.
> > > > >>>>>>> +
> > > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > > >>>>>>>
> > > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > > >>>>>>> submit
> > > > >>>>> This publicly archived list offers a means to provide input to
> > > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>>
> > > > >>>>> In order to verify user consent to the Feedback License terms
> > > > >>>>> and to minimize spam in the list archive, subscription is
> > > > >>>>> required before posting.
> > > > >>>>>
> > > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>>> List archive:
> > > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>>> Feedback License:
> > > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>>> List Guidelines:
> > > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>>
> > > > >>>> This publicly archived list offers a means to provide input to
> > > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>
> > > > >>>> In order to verify user consent to the Feedback License terms and
> > > > >>>> to minimize spam in the list archive, subscription is required
> > > > >>>> before posting.
> > > > >>>>
> > > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>> List archive:
> > > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>> Feedback License:
> > > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>> List Guidelines:
> > > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>
> > > > >>> ------------------------------------------------------------------
> > > > >>> --- To unsubscribe, e-mail:
> > > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > > >>> For additional commands, e-mail:
> > > > >>> virtio-dev-help@lists.oasis-open.org
> > > > >>>
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and to
> > > > > minimize spam in the list archive, subscription is required before
> > > > > posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License:
> > > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines:
> > > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Mon, 26 Jun 2023 08:35:08 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 12:19:04PM +0000, Parav Pandit wrote:
> >
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Monday, June 26, 2023 6:51 AM
> > >
> > > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > >
> > > >
> > > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>
> > > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > > <lingshan.zhu@intel.com> wrote:
> > > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > > can not choose one VF to bind random.
> > > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > > >>>>>>> On the other hand, generally the virtio devices are not
> > > > >>>>>>> created by the user inside the guest OS. This requires some
> > > management platform to participate.
> > > > >>>>>>>
> > > > >>>>>>> So the usage of this command:
> > > > >>>>>>> 1. The user purchases a virtio network card on the management
> > > platform,
> > > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > > of
> > > > >>>>>>>        the network card.
> > > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > > >>>>>>>        of the patch
> > > > >>>>>>>
> > > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > >>>>>>> ---
> > > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > > >>>>>>>
> > > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > > >>>>>>> 100644
> > > > >>>>>>> --- a/admin.tex
> > > > >>>>>>> +++ b/admin.tex
> > > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > > commands}\label{sec:Basic Facilities of a Virti
> > > > >>>>>>>      \hline \hline
> > > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > > list of commands supported for this group type    \\
> > > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > > of commands used for this group type \\
> > > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > > virtio_admin_cmd}    \\
> > > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > > one group member \\
> > > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > > virtio_admin_cmd}    \\
> > > > >>>>>>>      \hline
> > > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > > using a different structure)    \\
> > > > >>>>>>>      \hline
> > > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > > commands}\label{sec:Basic Facilities of a Virti
> > > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > > >>>>>>>
> > > > >>>>>>> +\subsubsection{Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > > >>>>>>> +guest OS, so the virtio
> > > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > > >>>>>
> > > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > > >>>>> the VF is create by the user in the OS. The devices does not know
> > > about it.
> > > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > > >>> YES.
> > > > >>>
> > > > >>>
> > > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > > choose one VF to bind random.
> > > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > > >>>>>> and PF knows that.
> > > > >>>>>>
> > > > >>>>>> I am not familiar with the background, What do you mean by
> > > > >>>>>> virtio device choose one VF to bind?
> > > > >>>>> On the cloud, the nic is created by the management platform, the
> > > > >>>>> user can not create a new nic inside the OS.
> > > > >>>>>
> > > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > > >>>>> not backend virtio-net devices.
> > > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > > >>>> provisioned by the orchestration software and it assign properly
> > > > >>>> selected a VF to a guest on demands.
> > > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > > >>> only be used in host, such as docker.
> > > > >>>
> > > > >>> The problem is that the user (you can think of this as the
> > > > >>> orchestration
> > > > >>> software) creates some VFs, these are only some PCI devices, which
> > > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > > >>> and creating a virtio-net device are two different things. One is
> > > > >>> done by user in the OS, one is done on the management platform.  So we
> > > need to bind them together.
> > > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > > >> device and its personality are determined.
> > > > >>
> > > > >> PCI spec says:
> > > > >> All VFs associated with a PF must be the same device type as the
> > > > >> PF, (e.g., the same network device type or the same storage device
> > > > >> type.)
> > > > >>
> > > > >> So how can the creating process be splitted into separated steps?
> > > > >>
> > > > >> Are we discussing something beyond the spec?
> > > > > NO.
> > > > >
> > > > > The device types are same.
> > > > >
> > > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > > the cloud, these are managed by the management platform. On the
> > > > > cloud, there is an abstract object in the backend, which contains
> > > > > things that are generally configured on the management platform. It is
> > > something that users purchase.
> > > > > Under the virtio standard it is similar to device.
> > > > >
> > > > > In my understanding, we just created a pci vf, and virtio works on
> > > > > top of pci, so there must be two steps here (If I mistake, please
> > > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > > have a backend default device respond when the driver probe the vf.  But in
> > > our scenario, each device is independent.
> > > > Once a VF is crated, there comes with some default configurations,
> > > > like MTU and MAC.
> > > > Do you mean first step creation and second step initialize it?
> > >
> > > Not exactly correct,
> > >
> > > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > > doesn't matter.
> > >
> > > In the second step, we can bind a backend device to this vf.
> > >
> > > Not just for initialization for new divice, we also want to support live migration.
> > >
> > > For example, on the host, we create a vf and passthrough it into a guest os, this
> > > guest is migrated from another host, and its corresponding network card is also
> > > migrated to this host. We need to bind this vf to the migrated network card.
> > >
> > > So just initialization is not enough.
> > >
> >
> > Yet to catch up on the thread, so likely I am missing something.
> >
> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside the device
> >
> > 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
>
> To be frank, I am not sure binding to an ID necessarily needs to be
> gated on provisioning commands.
> What was not explained at all is what purpose does this extra level
> of indirection serve.


I think the first question we need to solve is how to configure the back-end
devices corresponding to all vfs. Who will configure the queue num, msix
vectors and etc of these devices.

In the cloud, all these must go through the management platform.

Thanks.

>
>
> >
> >
> > > Thanks
> > >
> > > > If so, current spec only allow the user to config MAC through control vq.
> > > > vDPA allows to provision a device with proper configuration, maybe
> > > > that can be the solution?
> > > >
> > > > For binding, maybe the orchestration layer manages the pool and it
> > > > knows how to initialize the device
> > > >
> > > > Thanks
> > > > >
> > > > > Thanks.
> > > > >
> > > > >> Thanks
> > > > >>> Thanks.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> So I am confused what the intention of this patch.
> > > > >>>>> Thanks.
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > > >>>>>>> +created by the user inside the guest OS. This requires some
> > > management platform to participate.
> > > > >>>>>>> +
> > > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > > >>>>>>> +the virtio devices.
> > > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > > help?
> > > > >>>>>> Provision a device on demands and assign it to a guest?
> > > > >>>>>>
> > > > >>>>>> Thanks
> > > > >>>>>>> +
> > > > >>>>>>> +\begin{lstlisting}
> > > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > > >>>>>>> +    u64 identity;
> > > > >>>>>>> +};
> > > > >>>>>>> +\end{lstlisting}
> > > > >>>>>>> +
> > > > >>>>>>> +The user got the \field{identity} from the management
> > > > >>>>>>> +platform, that is not included by this spec.
> > > > >>>>>>> +
> > > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > > \field{group_member_id} MUST be set.
> > > > >>>>>>> +
> > > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > > >>>>>>> +identity of the virtio device.
> > > > >>>>>>> +
> > > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > > >>>>>>> +
> > > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > > >>>>>>> +
> > > > >>>>>>> +If the PF device can not find the device by the
> > > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > > VIRTIO_ADMIN_STATUS_EINVAL.
> > > > >>>>>>> +
> > > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > > >>>>>>> +MUST work as the device of this group member specified by the
> > > \field{group_member_id}.
> > > > >>>>>>> +
> > > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > > >>>>>>>
> > > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > > >>>>>>> submit
> > > > >>>>> This publicly archived list offers a means to provide input to
> > > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>>
> > > > >>>>> In order to verify user consent to the Feedback License terms
> > > > >>>>> and to minimize spam in the list archive, subscription is
> > > > >>>>> required before posting.
> > > > >>>>>
> > > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>>> List archive:
> > > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>>> Feedback License:
> > > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>>> List Guidelines:
> > > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>>
> > > > >>>> This publicly archived list offers a means to provide input to
> > > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>
> > > > >>>> In order to verify user consent to the Feedback License terms and
> > > > >>>> to minimize spam in the list archive, subscription is required
> > > > >>>> before posting.
> > > > >>>>
> > > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>> List archive:
> > > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>> Feedback License:
> > > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>> List Guidelines:
> > > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>
> > > > >>> ------------------------------------------------------------------
> > > > >>> --- To unsubscribe, e-mail:
> > > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > > >>> For additional commands, e-mail:
> > > > >>> virtio-dev-help@lists.oasis-open.org
> > > > >>>
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and to
> > > > > minimize spam in the list archive, subscription is required before
> > > > > posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License:
> > > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines:
> > > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] RE: RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Monday, June 26, 2023 8:33 AM

[..]

> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> 
> YES
> 
> My problem is that the nic is created by user on the management platform.  Or
> the device is migrated from other host.
>
Do you mean the management platform != the owner PF device?

 
> So we need to bind the device(created by user) to the vf between step #2 and
> #3.
>
We likely have some terminology issues.
Bind the device to the vf?

At OS level, there are
a. virtio PCI PF owner device
b. virtio PCI VF member device

c. some OS level device (such as net/blk, vfio, vdpa created by attaching #b to the driver)

d. An optional representor device on the DPU side (defined and managed by the vendor, outside the virtio spec as device implemtion).
Is this #d you call management platform?
If so, yes, orchestration may need to consult management platform, but this is optional.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

RE: RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Monday, June 26, 2023 8:33 AM

[..]

> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> 
> YES
> 
> My problem is that the nic is created by user on the management platform.  Or
> the device is migrated from other host.
>
Do you mean the management platform != the owner PF device?

 
> So we need to bind the device(created by user) to the vf between step #2 and
> #3.
>
We likely have some terminology issues.
Bind the device to the vf?

At OS level, there are
a. virtio PCI PF owner device
b. virtio PCI VF member device

c. some OS level device (such as net/blk, vfio, vdpa created by attaching #b to the driver)

d. An optional representor device on the DPU side (defined and managed by the vendor, outside the virtio spec as device implemtion).
Is this #d you call management platform?
If so, yes, orchestration may need to consult management platform, but this is optional.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, June 26, 2023 8:35 AM


> > Yet to catch up on the thread, so likely I am missing something.
> >
> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> To be frank, I am not sure binding to an ID necessarily needs to be gated on
> provisioning commands.
> What was not explained at all is what purpose does this extra level of
> indirection serve.

I didn't suggest gating binding on provisioning.
As written above _may_configure_.
So provisioning is optional.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] RE: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, June 26, 2023 8:35 AM


> > Yet to catch up on the thread, so likely I am missing something.
> >
> > The flow is for one OS (Linux) is:
> > 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in
> the device.
> > (num_vfs and vf enable bit in the PCI capability)
> >
> > 2. VFs are created at the PCI level in the host system and also inside
> > the device
> >
> > 3. A user on the host may bind these VFs to the VF driver
> > (virtionet/blk or vfio or vp_vdpa or some other)
> >
> > Between step #2 and #3, a user may configure one or multiple attributes of
> the VF.
> > This includes feature bits, config space fields, vf msix vectors and more.
> > This is to be using admin command.
> > These admin commands definition is due.
> 
> To be frank, I am not sure binding to an ID necessarily needs to be gated on
> provisioning commands.
> What was not explained at all is what purpose does this extra level of
> indirection serve.

I didn't suggest gating binding on provisioning.
As written above _may_configure_.
So provisioning is optional.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>
>>>>>>>>> So the usage of this command:
>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>         the network card.
>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>         of the patch
>>>>>>>>>
>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>> ---
>>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>> --- a/admin.tex
>>>>>>>>> +++ b/admin.tex
>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \hline \hline
>>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>       \hline
>>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>       \hline
>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>
>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>> passthrough-ed the
>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>> issue(affects host pci).
>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>
>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>> YES.
>>>>>
>>>>>
>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>> e.g.,
>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>
>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>> choose
>>>>>>>> one VF to bind?
>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>> user can not create a new nic inside the OS.
>>>>>>>
>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>> there is not backend virtio-net devices.
>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>> guest on demands.
>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>> in host, such as docker.
>>>>>
>>>>> The problem is that the user (you can think of this as the orchestration
>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>> is done on the management platform.  So we need to bind them together.
>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>> its personality are determined.
>>>>
>>>> PCI spec says:
>>>> All VFs associated with a PF must be the same device type as the PF,
>>>> (e.g., the same network
>>>> device type or the same storage device type.)
>>>>
>>>> So how can the creating process be splitted into separated steps?
>>>>
>>>> Are we discussing something beyond the spec?
>>> NO.
>>>
>>> The device types are same.
>>>
>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>> these are managed by the management platform. On the cloud, there is an abstract
>>> object in the backend, which contains things that are generally configured on
>>> the management platform. It is something that users purchase.
>>> Under the virtio standard it is similar to device.
>>>
>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>> so there must be two steps here (If I mistake, please point out.). When we create
>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>> scenarios, we can immediately have a backend default device respond when the
>>> driver probe the vf.  But in our scenario, each device is independent.
>> Once a VF is crated, there comes with some default configurations, like
>> MTU and MAC.
>> Do you mean first step creation and second step initialize it?
> Not exactly correct,
>
> The first step is just to create a vf, at this time there can be a default
> virtio-net, it doesn't matter.
>
> In the second step, we can bind a backend device to this vf.
What is "bind a backend device to this vf"? The VF itself is a backend 
device, right?
>
> Not just for initialization for new divice, we also want to support live
> migration.
>
> For example, on the host, we create a vf and passthrough it into a guest os,
> this guest is migrated from another host, and its corresponding network card is
> also migrated to this host. We need to bind this vf to the migrated network
> card.
>
> So just initialization is not enough.
I think you mean to restore the source side NIC configuration?
If so, I think this should be managed in the orchestration layer and
supported by virtio/vdpa APIs, e.g., at least restore common config

Thanks
>
> Thanks
>
>> If so, current spec only allow the user to config MAC through control vq.
>> vDPA allows to provision a device with proper configuration, maybe that
>> can be the solution?
>>
>> For binding, maybe the orchestration layer manages the pool and it knows
>> how to initialize
>> the device
>>
>> Thanks
>>> Thanks.
>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>> So I am confused what the intention of this patch.
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>> +
>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>> +devices.
>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>> +
>>>>>>>>> +\begin{lstlisting}
>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>> +    u64 identity;
>>>>>>>>> +};
>>>>>>>>> +\end{lstlisting}
>>>>>>>>> +
>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>> +included by this spec.
>>>>>>>>> +
>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>> +
>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>> +device.
>>>>>>>>> +
>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>> +
>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>> +
>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>> +
>>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>
>>>>>>>>>       An administration virtqueue of an owner device is used to submit
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>
>>>>>>>>> So the usage of this command:
>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>         the network card.
>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>         of the patch
>>>>>>>>>
>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>> ---
>>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>> --- a/admin.tex
>>>>>>>>> +++ b/admin.tex
>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \hline \hline
>>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>       \hline
>>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>       \hline
>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>
>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>> passthrough-ed the
>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>> issue(affects host pci).
>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>
>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>> YES.
>>>>>
>>>>>
>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>> e.g.,
>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>
>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>> choose
>>>>>>>> one VF to bind?
>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>> user can not create a new nic inside the OS.
>>>>>>>
>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>> there is not backend virtio-net devices.
>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>> guest on demands.
>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>> in host, such as docker.
>>>>>
>>>>> The problem is that the user (you can think of this as the orchestration
>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>> is done on the management platform.  So we need to bind them together.
>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>> its personality are determined.
>>>>
>>>> PCI spec says:
>>>> All VFs associated with a PF must be the same device type as the PF,
>>>> (e.g., the same network
>>>> device type or the same storage device type.)
>>>>
>>>> So how can the creating process be splitted into separated steps?
>>>>
>>>> Are we discussing something beyond the spec?
>>> NO.
>>>
>>> The device types are same.
>>>
>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>> these are managed by the management platform. On the cloud, there is an abstract
>>> object in the backend, which contains things that are generally configured on
>>> the management platform. It is something that users purchase.
>>> Under the virtio standard it is similar to device.
>>>
>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>> so there must be two steps here (If I mistake, please point out.). When we create
>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>> scenarios, we can immediately have a backend default device respond when the
>>> driver probe the vf.  But in our scenario, each device is independent.
>> Once a VF is crated, there comes with some default configurations, like
>> MTU and MAC.
>> Do you mean first step creation and second step initialize it?
> Not exactly correct,
>
> The first step is just to create a vf, at this time there can be a default
> virtio-net, it doesn't matter.
>
> In the second step, we can bind a backend device to this vf.
What is "bind a backend device to this vf"? The VF itself is a backend 
device, right?
>
> Not just for initialization for new divice, we also want to support live
> migration.
>
> For example, on the host, we create a vf and passthrough it into a guest os,
> this guest is migrated from another host, and its corresponding network card is
> also migrated to this host. We need to bind this vf to the migrated network
> card.
>
> So just initialization is not enough.
I think you mean to restore the source side NIC configuration?
If so, I think this should be managed in the orchestration layer and
supported by virtio/vdpa APIs, e.g., at least restore common config

Thanks
>
> Thanks
>
>> If so, current spec only allow the user to config MAC through control vq.
>> vDPA allows to provision a device with proper configuration, maybe that
>> can be the solution?
>>
>> For binding, maybe the orchestration layer manages the pool and it knows
>> how to initialize
>> the device
>>
>> Thanks
>>> Thanks.
>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>> So I am confused what the intention of this patch.
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>> +
>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>> +devices.
>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>> +
>>>>>>>>> +\begin{lstlisting}
>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>> +    u64 identity;
>>>>>>>>> +};
>>>>>>>>> +\end{lstlisting}
>>>>>>>>> +
>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>> +included by this spec.
>>>>>>>>> +
>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>> +
>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>> +device.
>>>>>>>>> +
>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>> +
>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>> +
>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>> +
>>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>
>>>>>>>>>       An administration virtqueue of an owner device is used to submit
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jun 26, 2023 at 2:22 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.

Any reason we can't do this through IPC of software layers?

Thanks

> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>
>  An administration virtqueue of an owner device is used to submit
> --
> 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jun 26, 2023 at 2:22 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.

Any reason we can't do this through IPC of software layers?

Thanks

> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>
>  An administration virtqueue of an owner device is used to submit
> --
> 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> >>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>>>>>
> >>>>>>>>> So the usage of this command:
> >>>>>>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>>>>>         the network card.
> >>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>>>>>         of the patch
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>>>>>> ---
> >>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/admin.tex b/admin.tex
> >>>>>>>>> index 2efd4d7..64d0667 100644
> >>>>>>>>> --- a/admin.tex
> >>>>>>>>> +++ b/admin.tex
> >>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \hline \hline
> >>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>>       \hline
> >>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>>>>>       \hline
> >>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>>>>>
> >>>>>>>>> +\subsubsection{Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>>>>>> passthrough-ed the
> >>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>>>>>> issue(affects host pci).
> >>>>>>> No care about the passthrough, we always created VFs by the PF.
> >>>>>>>
> >>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>>>>>> by the user in the OS. The devices does not know about it.
> >>>>>> OK, perhaps just say create VFs from a PF in the OS?
> >>>>> YES.
> >>>>>
> >>>>>
> >>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>> I failed to understand this, once a VF is created, it has a personality,
> >>>>>>>> e.g.,
> >>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>>>>>
> >>>>>>>> I am not familiar with the background, What do you mean by virtio device
> >>>>>>>> choose
> >>>>>>>> one VF to bind?
> >>>>>>> On the cloud, the nic is created by the management platform, the
> >>>>>>> user can not create a new nic inside the OS.
> >>>>>>>
> >>>>>>> So after echo sriov_numvfs, the user just got some VFs,
> >>>>>>> there is not backend virtio-net devices.
> >>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >>>>>> the orchestration software and it assign properly selected a VF to a
> >>>>>> guest on demands.
> >>>>> Yes, but we do not need to care about the guest. Because VF may only be used
> >>>>> in host, such as docker.
> >>>>>
> >>>>> The problem is that the user (you can think of this as the orchestration
> >>>>> software) creates some VFs, these are only some PCI devices, which virtio
> >>>>> devices will work on these VFs. I think that creating a vf and creating a
> >>>>> virtio-net device are two different things. One is done by user in the OS, one
> >>>>> is done on the management platform.  So we need to bind them together.
> >>>> If the VFs are created through sriov_numvfs, once created, the VF device and
> >>>> its personality are determined.
> >>>>
> >>>> PCI spec says:
> >>>> All VFs associated with a PF must be the same device type as the PF,
> >>>> (e.g., the same network
> >>>> device type or the same storage device type.)
> >>>>
> >>>> So how can the creating process be splitted into separated steps?
> >>>>
> >>>> Are we discussing something beyond the spec?
> >>> NO.
> >>>
> >>> The device types are same.
> >>>
> >>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> >>> these are managed by the management platform. On the cloud, there is an abstract
> >>> object in the backend, which contains things that are generally configured on
> >>> the management platform. It is something that users purchase.
> >>> Under the virtio standard it is similar to device.
> >>>
> >>> In my understanding, we just created a pci vf, and virtio works on top of pci,
> >>> so there must be two steps here (If I mistake, please point out.). When we create
> >>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> >>> scenarios, we can immediately have a backend default device respond when the
> >>> driver probe the vf.  But in our scenario, each device is independent.
> >> Once a VF is crated, there comes with some default configurations, like
> >> MTU and MAC.
> >> Do you mean first step creation and second step initialize it?
> > Not exactly correct,
> >
> > The first step is just to create a vf, at this time there can be a default
> > virtio-net, it doesn't matter.
> >
> > In the second step, we can bind a backend device to this vf.
> What is "bind a backend device to this vf"? The VF itself is a backend
> device, right?


I think that the VF is a transport. I want to bind
the virtio backend device to this transport.


> >
> > Not just for initialization for new divice, we also want to support live
> > migration.
> >
> > For example, on the host, we create a vf and passthrough it into a guest os,
> > this guest is migrated from another host, and its corresponding network card is
> > also migrated to this host. We need to bind this vf to the migrated network
> > card.
> >
> > So just initialization is not enough.
> I think you mean to restore the source side NIC configuration?
> If so, I think this should be managed in the orchestration layer and
> supported by virtio/vdpa APIs, e.g., at least restore common config

I will post our case later.

Thanks.



>
> Thanks
> >
> > Thanks
> >
> >> If so, current spec only allow the user to config MAC through control vq.
> >> vDPA allows to provision a device with proper configuration, maybe that
> >> can be the solution?
> >>
> >> For binding, maybe the orchestration layer manages the pool and it knows
> >> how to initialize
> >> the device
> >>
> >> Thanks
> >>> Thanks.
> >>>
> >>>> Thanks
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> So I am confused what the intention of this patch.
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>>>>>> +
> >>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>>>>>> +devices.
> >>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>>>>>> Provision a device on demands and assign it to a guest?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>>> +
> >>>>>>>>> +\begin{lstlisting}
> >>>>>>>>> +struct virtio_admin_cmd_bind {
> >>>>>>>>> +    u64 identity;
> >>>>>>>>> +};
> >>>>>>>>> +\end{lstlisting}
> >>>>>>>>> +
> >>>>>>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>>>>>> +included by this spec.
> >>>>>>>>> +
> >>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>>>>>> +
> >>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>>>>>> +device.
> >>>>>>>>> +
> >>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>>>>>> +
> >>>>>>>>> +If the PF device can not find the device by the \field{identity},
> >>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>>>>>> +
> >>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>>>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>>>>>> +
> >>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>>>>>
> >>>>>>>>>       An administration virtqueue of an owner device is used to submit
> >>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>
> >>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>> before posting.
> >>>>>>>
> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>
> >>>>>> This publicly archived list offers a means to provide input to the
> >>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>
> >>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>> to minimize spam in the list archive, subscription is required
> >>>>>> before posting.
> >>>>>>
> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >>>>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> >>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>>>>>
> >>>>>>>>> So the usage of this command:
> >>>>>>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>>>>>         the network card.
> >>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>>>>>         of the patch
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>>>>>> ---
> >>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/admin.tex b/admin.tex
> >>>>>>>>> index 2efd4d7..64d0667 100644
> >>>>>>>>> --- a/admin.tex
> >>>>>>>>> +++ b/admin.tex
> >>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \hline \hline
> >>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>>       \hline
> >>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>>>>>       \hline
> >>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>>>>>
> >>>>>>>>> +\subsubsection{Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>>>>>> passthrough-ed the
> >>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>>>>>> issue(affects host pci).
> >>>>>>> No care about the passthrough, we always created VFs by the PF.
> >>>>>>>
> >>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>>>>>> by the user in the OS. The devices does not know about it.
> >>>>>> OK, perhaps just say create VFs from a PF in the OS?
> >>>>> YES.
> >>>>>
> >>>>>
> >>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>> I failed to understand this, once a VF is created, it has a personality,
> >>>>>>>> e.g.,
> >>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>>>>>
> >>>>>>>> I am not familiar with the background, What do you mean by virtio device
> >>>>>>>> choose
> >>>>>>>> one VF to bind?
> >>>>>>> On the cloud, the nic is created by the management platform, the
> >>>>>>> user can not create a new nic inside the OS.
> >>>>>>>
> >>>>>>> So after echo sriov_numvfs, the user just got some VFs,
> >>>>>>> there is not backend virtio-net devices.
> >>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >>>>>> the orchestration software and it assign properly selected a VF to a
> >>>>>> guest on demands.
> >>>>> Yes, but we do not need to care about the guest. Because VF may only be used
> >>>>> in host, such as docker.
> >>>>>
> >>>>> The problem is that the user (you can think of this as the orchestration
> >>>>> software) creates some VFs, these are only some PCI devices, which virtio
> >>>>> devices will work on these VFs. I think that creating a vf and creating a
> >>>>> virtio-net device are two different things. One is done by user in the OS, one
> >>>>> is done on the management platform.  So we need to bind them together.
> >>>> If the VFs are created through sriov_numvfs, once created, the VF device and
> >>>> its personality are determined.
> >>>>
> >>>> PCI spec says:
> >>>> All VFs associated with a PF must be the same device type as the PF,
> >>>> (e.g., the same network
> >>>> device type or the same storage device type.)
> >>>>
> >>>> So how can the creating process be splitted into separated steps?
> >>>>
> >>>> Are we discussing something beyond the spec?
> >>> NO.
> >>>
> >>> The device types are same.
> >>>
> >>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> >>> these are managed by the management platform. On the cloud, there is an abstract
> >>> object in the backend, which contains things that are generally configured on
> >>> the management platform. It is something that users purchase.
> >>> Under the virtio standard it is similar to device.
> >>>
> >>> In my understanding, we just created a pci vf, and virtio works on top of pci,
> >>> so there must be two steps here (If I mistake, please point out.). When we create
> >>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> >>> scenarios, we can immediately have a backend default device respond when the
> >>> driver probe the vf.  But in our scenario, each device is independent.
> >> Once a VF is crated, there comes with some default configurations, like
> >> MTU and MAC.
> >> Do you mean first step creation and second step initialize it?
> > Not exactly correct,
> >
> > The first step is just to create a vf, at this time there can be a default
> > virtio-net, it doesn't matter.
> >
> > In the second step, we can bind a backend device to this vf.
> What is "bind a backend device to this vf"? The VF itself is a backend
> device, right?


I think that the VF is a transport. I want to bind
the virtio backend device to this transport.


> >
> > Not just for initialization for new divice, we also want to support live
> > migration.
> >
> > For example, on the host, we create a vf and passthrough it into a guest os,
> > this guest is migrated from another host, and its corresponding network card is
> > also migrated to this host. We need to bind this vf to the migrated network
> > card.
> >
> > So just initialization is not enough.
> I think you mean to restore the source side NIC configuration?
> If so, I think this should be managed in the orchestration layer and
> supported by virtio/vdpa APIs, e.g., at least restore common config

I will post our case later.

Thanks.



>
> Thanks
> >
> > Thanks
> >
> >> If so, current spec only allow the user to config MAC through control vq.
> >> vDPA allows to provision a device with proper configuration, maybe that
> >> can be the solution?
> >>
> >> For binding, maybe the orchestration layer manages the pool and it knows
> >> how to initialize
> >> the device
> >>
> >> Thanks
> >>> Thanks.
> >>>
> >>>> Thanks
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> So I am confused what the intention of this patch.
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>>>>>> +
> >>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>>>>>> +devices.
> >>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>>>>>> Provision a device on demands and assign it to a guest?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>>> +
> >>>>>>>>> +\begin{lstlisting}
> >>>>>>>>> +struct virtio_admin_cmd_bind {
> >>>>>>>>> +    u64 identity;
> >>>>>>>>> +};
> >>>>>>>>> +\end{lstlisting}
> >>>>>>>>> +
> >>>>>>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>>>>>> +included by this spec.
> >>>>>>>>> +
> >>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>>>>>> +
> >>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>>>>>> +device.
> >>>>>>>>> +
> >>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>>>>>> +
> >>>>>>>>> +If the PF device can not find the device by the \field{identity},
> >>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>>>>>> +
> >>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>>>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>>>>>> +
> >>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>>>>>
> >>>>>>>>>       An administration virtqueue of an owner device is used to submit
> >>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>
> >>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>> before posting.
> >>>>>>>
> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>
> >>>>>> This publicly archived list offers a means to provide input to the
> >>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>
> >>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>> to minimize spam in the list archive, subscription is required
> >>>>>> before posting.
> >>>>>>
> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >>>>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 16:08:38 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 2:22 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
>
> Any reason we can't do this through IPC of software layers?


As Parav pointed out, we may have some gaps on the term. I will post our case
later.

Thanks.



>
> Thanks
>
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 16:08:38 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Jun 26, 2023 at 2:22 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
>
> Any reason we can't do this through IPC of software layers?


As Parav pointed out, we may have some gaps on the term. I will post our case
later.

Thanks.



>
> Thanks
>
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

Thanks Parav for pointing it out. We may have some gaps on the case.

Let me introduce our case, which I think it is simple and should be easy to
understand.

First, the user (customer) purchased a bare metal machine.

## Bare metal machine

Let me briefly explain the characteristics of a bare metal machine. It is not a
virtual machine, it is a physical machine, and the difference between it and a
general physical machine is that its PCI is connected to a device similar to a
DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
These devices are managed by the vendor, and must be created and purchased
on the vendor's management platform.

## DPU

There is a software implementation in the DPU, which will respond to PCI
operations. But as mentioned above, resources such as network cards must be
purchased and created before they can exist. So users can create VF, which is
just a pci-level operation, but there may not be a corresponding backend.

## Management Platform

The creation and configuration of devices is realized on the management
platform.

After the user completed the purchase on the management platform (this is an
independent platform provided by the vendor and has nothing to do with
virtio), then there will be a corresponding device implementation in the DPU.
This includes some user configurations, available bandwidth resources and other
information.

## Usage

Since the user is directly on the HOST, the user can create VMs, passthrough PF
or VF into the VM. Or users can create a large number of dockers, all of which
use a separate virtio-net device for performance.

The reason why users use vf is that we need to use a large number of virtio-net
devices. This number reaches 1k+.

Based on this scenario, we need to bind vf to the backend device. Because, we
cannot automatically complete the creation of the virtio-net backend device when
the user creates a vf.

## Migration

In addition, let's consider another scenario of migration. If a vm is migrated
from another host, of course its corresponding virtio device is also migrated to
the DPU. At this time, our newly created vf can only be used by the vm after it
is bound to the migrated device. We do not want this vf to be a brand new
device.

## Abstraction

So, this is how I understand the process of creating vf:

1. Create a PCI VF, at this time there may be no backend virtio device, or there
    is only a default backend. It does not fully meet our expectations.
2. Create device or migrate device
3. Bind the backend virtio device to the vf

In most scenarios, the first step may be enough. We can make some fine-tuning on
this default device, such as modifying its mac. In the future, we can use admin
queue to modify its msix vector and other configurations.

But we should allow, we bind a backend virtio device to a certain vf. This is
useful for live migration and virtio devices with special configurations.

The design of virtio itself is two layers, and virtio should allow switching the
transport layer by nature. This is our advantage.

## About the identity

In this patch, I used a vendor's id. The purpose of this is that I hope to
be compatible with all devices. In the network scenario, it is actually possible
to use a mac.

Perhaps, introduce a standard id for each device/driver, I think this may be
more in line with the habit of virtio spec.

Thanks.

On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>
>  An administration virtqueue of an owner device is used to submit
> --
> 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

Thanks Parav for pointing it out. We may have some gaps on the case.

Let me introduce our case, which I think it is simple and should be easy to
understand.

First, the user (customer) purchased a bare metal machine.

## Bare metal machine

Let me briefly explain the characteristics of a bare metal machine. It is not a
virtual machine, it is a physical machine, and the difference between it and a
general physical machine is that its PCI is connected to a device similar to a
DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
These devices are managed by the vendor, and must be created and purchased
on the vendor's management platform.

## DPU

There is a software implementation in the DPU, which will respond to PCI
operations. But as mentioned above, resources such as network cards must be
purchased and created before they can exist. So users can create VF, which is
just a pci-level operation, but there may not be a corresponding backend.

## Management Platform

The creation and configuration of devices is realized on the management
platform.

After the user completed the purchase on the management platform (this is an
independent platform provided by the vendor and has nothing to do with
virtio), then there will be a corresponding device implementation in the DPU.
This includes some user configurations, available bandwidth resources and other
information.

## Usage

Since the user is directly on the HOST, the user can create VMs, passthrough PF
or VF into the VM. Or users can create a large number of dockers, all of which
use a separate virtio-net device for performance.

The reason why users use vf is that we need to use a large number of virtio-net
devices. This number reaches 1k+.

Based on this scenario, we need to bind vf to the backend device. Because, we
cannot automatically complete the creation of the virtio-net backend device when
the user creates a vf.

## Migration

In addition, let's consider another scenario of migration. If a vm is migrated
from another host, of course its corresponding virtio device is also migrated to
the DPU. At this time, our newly created vf can only be used by the vm after it
is bound to the migrated device. We do not want this vf to be a brand new
device.

## Abstraction

So, this is how I understand the process of creating vf:

1. Create a PCI VF, at this time there may be no backend virtio device, or there
    is only a default backend. It does not fully meet our expectations.
2. Create device or migrate device
3. Bind the backend virtio device to the vf

In most scenarios, the first step may be enough. We can make some fine-tuning on
this default device, such as modifying its mac. In the future, we can use admin
queue to modify its msix vector and other configurations.

But we should allow, we bind a backend virtio device to a certain vf. This is
useful for live migration and virtio devices with special configurations.

The design of virtio itself is two layers, and virtio should allow switching the
transport layer by nature. This is our advantage.

## About the identity

In this patch, I used a vendor's id. The purpose of this is that I hope to
be compatible with all devices. In the network scenario, it is actually possible
to use a mac.

Perhaps, introduce a standard id for each device/driver, I think this may be
more in line with the habit of virtio spec.

Thanks.

On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> devices don't know about these VFs. Because each VF may be assigned a different role
> by the user, the virtio device can not choose one VF to bind random.
> So only the user knows how to bind the virtio devices to the VFs.
> On the other hand, generally the virtio devices are not created by the user
> inside the guest OS. This requires some management platform to participate.
>
> So the usage of this command:
> 1. The user purchases a virtio network card on the management platform,
>    and sets the ip, queue number, etc. The user obtains the identity of
>    the network card.
> 2. The user creates a VF with echo 8 > sriov_numvfs
> 3. The user binds the net crad to a VF with identity through the command
>    of the patch
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 40 insertions(+), 1 deletion(-)
>
> diff --git a/admin.tex b/admin.tex
> index 2efd4d7..64d0667 100644
> --- a/admin.tex
> +++ b/admin.tex
> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \hline \hline
>  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>  \hline
>  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>  \hline
> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>  \field{VF Enable} refer to registers within the SR-IOV Extended
>  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>
> +\subsubsection{Bind the device for member}
> +
> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> +devices don't know about these VFs. Because each VF may be assigned a different role
> +by the user, the virtio device can not choose one VF to bind random.
> +So only the user knows how to bind the virtio devices to the VFs.
> +On the other hand, generally the virtio devices are not created by the user
> +inside the guest OS. This requires some management platform to participate.
> +
> +So we introduce a new admin queue command to bind the VFs and the virtio
> +devices.
> +
> +\begin{lstlisting}
> +struct virtio_admin_cmd_bind {
> +    u64 identity;
> +};
> +\end{lstlisting}
> +
> +The user got the \field{identity} from the management platform, that is not
> +included by this spec.
> +
> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> +
> +The \field{identity} is passed by the user. It is the identity of the virtio
> +device.
> +
> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> +
> +Every device MUST have one unique \field{identity} in the host.
> +
> +If the PF device can not find the device by the \field{identity},
> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> +
> +If the device is found by the \field{identity}, the device MUST work as the
> +device of this group member specified by the \field{group_member_id}.
> +
>  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>
>  An administration virtqueue of an owner device is used to submit
> --
> 2.32.0.3.g01195cf9f
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
>
> Thanks Parav for pointing it out. We may have some gaps on the case.
>
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
>
> First, the user (customer) purchased a bare metal machine.
>
> ## Bare metal machine
>
> Let me briefly explain the characteristics of a bare metal machine. It is not a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a
> DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
>
> ## DPU
>
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
>
> ## Management Platform
>
> The creation and configuration of devices is realized on the management
> platform.
>
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and other
> information.
>
> ## Usage
>
> Since the user is directly on the HOST, the user can create VMs, passthrough PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
>
> The reason why users use vf is that we need to use a large number of virtio-net
> devices. This number reaches 1k+.
>
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device when
> the user creates a vf.
>
> ## Migration
>
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated to
> the DPU. At this time, our newly created vf can only be used by the vm after it
> is bound to the migrated device. We do not want this vf to be a brand new
> device.
>
> ## Abstraction
>
> So, this is how I understand the process of creating vf:
>
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

3) should come before 2)?

Who is going to do 3) btw, is it the user? If yes, for example, if a
user wants another 4 queue virtio-net devices, after purchase, how
does the user know its id?

>
> In most scenarios, the first step may be enough. We can make some fine-tuning on
> this default device, such as modifying its mac. In the future, we can use admin
> queue to modify its msix vector and other configurations.
>
> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.

All of these could be addressed if a dynamic provisioning model is
implemented (SIOV or transport virtqueue). Trying to have a workaround
in SR-IOV might be tricky.

>
> The design of virtio itself is two layers, and virtio should allow switching the
> transport layer by nature. This is our advantage.

Is it not switching the transport layer but about binding/unbinding
vitio devices to VF?

Is a new capability or similar admin cmd sufficient in this case?

struct virtio_pci_bind_cap {
        struct virtio_pci_cap cap;
        u16 bind; // virtio_device_id
        u16 unbind; // virtio_device_id
};

Thanks

>
> ## About the identity
>
> In this patch, I used a vendor's id. The purpose of this is that I hope to
> be compatible with all devices. In the network scenario, it is actually possible
> to use a mac.
>
> Perhaps, introduce a standard id for each device/driver, I think this may be
> more in line with the habit of virtio spec.
>
> Thanks.
>
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
>
> Thanks Parav for pointing it out. We may have some gaps on the case.
>
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
>
> First, the user (customer) purchased a bare metal machine.
>
> ## Bare metal machine
>
> Let me briefly explain the characteristics of a bare metal machine. It is not a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a
> DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
>
> ## DPU
>
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
>
> ## Management Platform
>
> The creation and configuration of devices is realized on the management
> platform.
>
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and other
> information.
>
> ## Usage
>
> Since the user is directly on the HOST, the user can create VMs, passthrough PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
>
> The reason why users use vf is that we need to use a large number of virtio-net
> devices. This number reaches 1k+.
>
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device when
> the user creates a vf.
>
> ## Migration
>
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated to
> the DPU. At this time, our newly created vf can only be used by the vm after it
> is bound to the migrated device. We do not want this vf to be a brand new
> device.
>
> ## Abstraction
>
> So, this is how I understand the process of creating vf:
>
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

3) should come before 2)?

Who is going to do 3) btw, is it the user? If yes, for example, if a
user wants another 4 queue virtio-net devices, after purchase, how
does the user know its id?

>
> In most scenarios, the first step may be enough. We can make some fine-tuning on
> this default device, such as modifying its mac. In the future, we can use admin
> queue to modify its msix vector and other configurations.
>
> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.

All of these could be addressed if a dynamic provisioning model is
implemented (SIOV or transport virtqueue). Trying to have a workaround
in SR-IOV might be tricky.

>
> The design of virtio itself is two layers, and virtio should allow switching the
> transport layer by nature. This is our advantage.

Is it not switching the transport layer but about binding/unbinding
vitio devices to VF?

Is a new capability or similar admin cmd sufficient in this case?

struct virtio_pci_bind_cap {
        struct virtio_pci_cap cap;
        u16 bind; // virtio_device_id
        u16 unbind; // virtio_device_id
};

Thanks

>
> ## About the identity
>
> In this patch, I used a vendor's id. The purpose of this is that I hope to
> be compatible with all devices. In the network scenario, it is actually possible
> to use a mac.
>
> Perhaps, introduce a standard id for each device/driver, I think this may be
> more in line with the habit of virtio spec.
>
> Thanks.
>
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > devices don't know about these VFs. Because each VF may be assigned a different role
> > by the user, the virtio device can not choose one VF to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the user
> > inside the guest OS. This requires some management platform to participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs
> > 3. The user binds the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >  \field{VF Enable} refer to registers within the SR-IOV Extended
> >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > +devices don't know about these VFs. Because each VF may be assigned a different role
> > +by the user, the virtio device can not choose one VF to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by the user
> > +inside the guest OS. This requires some management platform to participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the virtio
> > +devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that is not
> > +included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the virtio
> > +device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity},
> > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work as the
> > +device of this group member specified by the \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/27/2023 4:14 PM, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>>
>>>>>>>>>>> So the usage of this command:
>>>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>>>          and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>>>          the network card.
>>>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>>>          of the patch
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>>>> ---
>>>>>>>>>>>        admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>        1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>>>> --- a/admin.tex
>>>>>>>>>>> +++ b/admin.tex
>>>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \hline \hline
>>>>>>>>>>>        0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>>>        0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>>        0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>>>        Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>>>
>>>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>>>> passthrough-ed the
>>>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>>>> issue(affects host pci).
>>>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>>>
>>>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>>>> YES.
>>>>>>>
>>>>>>>
>>>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>>>> e.g.,
>>>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>>>
>>>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>>>> choose
>>>>>>>>>> one VF to bind?
>>>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>>>> user can not create a new nic inside the OS.
>>>>>>>>>
>>>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>>>> there is not backend virtio-net devices.
>>>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>>>> guest on demands.
>>>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>>>> in host, such as docker.
>>>>>>>
>>>>>>> The problem is that the user (you can think of this as the orchestration
>>>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>>>> is done on the management platform.  So we need to bind them together.
>>>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>>>> its personality are determined.
>>>>>>
>>>>>> PCI spec says:
>>>>>> All VFs associated with a PF must be the same device type as the PF,
>>>>>> (e.g., the same network
>>>>>> device type or the same storage device type.)
>>>>>>
>>>>>> So how can the creating process be splitted into separated steps?
>>>>>>
>>>>>> Are we discussing something beyond the spec?
>>>>> NO.
>>>>>
>>>>> The device types are same.
>>>>>
>>>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>>>> these are managed by the management platform. On the cloud, there is an abstract
>>>>> object in the backend, which contains things that are generally configured on
>>>>> the management platform. It is something that users purchase.
>>>>> Under the virtio standard it is similar to device.
>>>>>
>>>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>>>> so there must be two steps here (If I mistake, please point out.). When we create
>>>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>>>> scenarios, we can immediately have a backend default device respond when the
>>>>> driver probe the vf.  But in our scenario, each device is independent.
>>>> Once a VF is crated, there comes with some default configurations, like
>>>> MTU and MAC.
>>>> Do you mean first step creation and second step initialize it?
>>> Not exactly correct,
>>>
>>> The first step is just to create a vf, at this time there can be a default
>>> virtio-net, it doesn't matter.
>>>
>>> In the second step, we can bind a backend device to this vf.
>> What is "bind a backend device to this vf"? The VF itself is a backend
>> device, right?
>
> I think that the VF is a transport. I want to bind
> the virtio backend device to this transport.
still confused, PCIe VF is a device, transport refers to PCI since we 
are talking about VFs.
Not sure what is the binding because the VF comes with its transport, but
if you want to address a VF, shall the orchestrator simply use the num 
of a VF or its BDF?
>
>
>>> Not just for initialization for new divice, we also want to support live
>>> migration.
>>>
>>> For example, on the host, we create a vf and passthrough it into a guest os,
>>> this guest is migrated from another host, and its corresponding network card is
>>> also migrated to this host. We need to bind this vf to the migrated network
>>> card.
>>>
>>> So just initialization is not enough.
>> I think you mean to restore the source side NIC configuration?
>> If so, I think this should be managed in the orchestration layer and
>> supported by virtio/vdpa APIs, e.g., at least restore common config
> I will post our case later.
>
> Thanks.
>
>
>
>> Thanks
>>> Thanks
>>>
>>>> If so, current spec only allow the user to config MAC through control vq.
>>>> vDPA allows to provision a device with proper configuration, maybe that
>>>> can be the solution?
>>>>
>>>> For binding, maybe the orchestration layer manages the pool and it knows
>>>> how to initialize
>>>> the device
>>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>> Thanks
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> So I am confused what the intention of this patch.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>> +
>>>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>>>> +devices.
>>>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>> +
>>>>>>>>>>> +\begin{lstlisting}
>>>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>>>> +    u64 identity;
>>>>>>>>>>> +};
>>>>>>>>>>> +\end{lstlisting}
>>>>>>>>>>> +
>>>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>>>> +included by this spec.
>>>>>>>>>>> +
>>>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>>>> +
>>>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>>>> +device.
>>>>>>>>>>> +
>>>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>>>> +
>>>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>>>> +
>>>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>>>> +
>>>>>>>>>>>        \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>>>
>>>>>>>>>>>        An administration virtqueue of an owner device is used to submit
>>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>>
>>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>>> before posting.
>>>>>>>>>
>>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>>
>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>
>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>> before posting.
>>>>>>>>
>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Zhu, Lingshan]

On 6/27/2023 4:14 PM, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>>
>>>>>>>>>>> So the usage of this command:
>>>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>>>          and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>>>          the network card.
>>>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>>>          of the patch
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>>>> ---
>>>>>>>>>>>        admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>        1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>>>> --- a/admin.tex
>>>>>>>>>>> +++ b/admin.tex
>>>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \hline \hline
>>>>>>>>>>>        0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>>>        0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>>        0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>>>        Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>>>
>>>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>>>> passthrough-ed the
>>>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>>>> issue(affects host pci).
>>>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>>>
>>>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>>>> YES.
>>>>>>>
>>>>>>>
>>>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>>>> e.g.,
>>>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>>>
>>>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>>>> choose
>>>>>>>>>> one VF to bind?
>>>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>>>> user can not create a new nic inside the OS.
>>>>>>>>>
>>>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>>>> there is not backend virtio-net devices.
>>>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>>>> guest on demands.
>>>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>>>> in host, such as docker.
>>>>>>>
>>>>>>> The problem is that the user (you can think of this as the orchestration
>>>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>>>> is done on the management platform.  So we need to bind them together.
>>>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>>>> its personality are determined.
>>>>>>
>>>>>> PCI spec says:
>>>>>> All VFs associated with a PF must be the same device type as the PF,
>>>>>> (e.g., the same network
>>>>>> device type or the same storage device type.)
>>>>>>
>>>>>> So how can the creating process be splitted into separated steps?
>>>>>>
>>>>>> Are we discussing something beyond the spec?
>>>>> NO.
>>>>>
>>>>> The device types are same.
>>>>>
>>>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>>>> these are managed by the management platform. On the cloud, there is an abstract
>>>>> object in the backend, which contains things that are generally configured on
>>>>> the management platform. It is something that users purchase.
>>>>> Under the virtio standard it is similar to device.
>>>>>
>>>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>>>> so there must be two steps here (If I mistake, please point out.). When we create
>>>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>>>> scenarios, we can immediately have a backend default device respond when the
>>>>> driver probe the vf.  But in our scenario, each device is independent.
>>>> Once a VF is crated, there comes with some default configurations, like
>>>> MTU and MAC.
>>>> Do you mean first step creation and second step initialize it?
>>> Not exactly correct,
>>>
>>> The first step is just to create a vf, at this time there can be a default
>>> virtio-net, it doesn't matter.
>>>
>>> In the second step, we can bind a backend device to this vf.
>> What is "bind a backend device to this vf"? The VF itself is a backend
>> device, right?
>
> I think that the VF is a transport. I want to bind
> the virtio backend device to this transport.
still confused, PCIe VF is a device, transport refers to PCI since we 
are talking about VFs.
Not sure what is the binding because the VF comes with its transport, but
if you want to address a VF, shall the orchestrator simply use the num 
of a VF or its BDF?
>
>
>>> Not just for initialization for new divice, we also want to support live
>>> migration.
>>>
>>> For example, on the host, we create a vf and passthrough it into a guest os,
>>> this guest is migrated from another host, and its corresponding network card is
>>> also migrated to this host. We need to bind this vf to the migrated network
>>> card.
>>>
>>> So just initialization is not enough.
>> I think you mean to restore the source side NIC configuration?
>> If so, I think this should be managed in the orchestration layer and
>> supported by virtio/vdpa APIs, e.g., at least restore common config
> I will post our case later.
>
> Thanks.
>
>
>
>> Thanks
>>> Thanks
>>>
>>>> If so, current spec only allow the user to config MAC through control vq.
>>>> vDPA allows to provision a device with proper configuration, maybe that
>>>> can be the solution?
>>>>
>>>> For binding, maybe the orchestration layer manages the pool and it knows
>>>> how to initialize
>>>> the device
>>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>> Thanks
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> So I am confused what the intention of this patch.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>> +
>>>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>>>> +devices.
>>>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>> +
>>>>>>>>>>> +\begin{lstlisting}
>>>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>>>> +    u64 identity;
>>>>>>>>>>> +};
>>>>>>>>>>> +\end{lstlisting}
>>>>>>>>>>> +
>>>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>>>> +included by this spec.
>>>>>>>>>>> +
>>>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>>>> +
>>>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>>>> +device.
>>>>>>>>>>> +
>>>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>>>> +
>>>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>>>> +
>>>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>>>> +
>>>>>>>>>>>        \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>>>
>>>>>>>>>>>        An administration virtqueue of an owner device is used to submit
>>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>>
>>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>>> before posting.
>>>>>>>>>
>>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>>
>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>
>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>> before posting.
>>>>>>>>
>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> >
> > Thanks Parav for pointing it out. We may have some gaps on the case.
> >
> > Let me introduce our case, which I think it is simple and should be easy to
> > understand.
> >
> > First, the user (customer) purchased a bare metal machine.
> >
> > ## Bare metal machine
> >
> > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > virtual machine, it is a physical machine, and the difference between it and a
> > general physical machine is that its PCI is connected to a device similar to a
> > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > These devices are managed by the vendor, and must be created and purchased
> > on the vendor's management platform.
> >
> > ## DPU
> >
> > There is a software implementation in the DPU, which will respond to PCI
> > operations. But as mentioned above, resources such as network cards must be
> > purchased and created before they can exist. So users can create VF, which is
> > just a pci-level operation, but there may not be a corresponding backend.
> >
> > ## Management Platform
> >
> > The creation and configuration of devices is realized on the management
> > platform.
> >
> > After the user completed the purchase on the management platform (this is an
> > independent platform provided by the vendor and has nothing to do with
> > virtio), then there will be a corresponding device implementation in the DPU.
> > This includes some user configurations, available bandwidth resources and other
> > information.
> >
> > ## Usage
> >
> > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > or VF into the VM. Or users can create a large number of dockers, all of which
> > use a separate virtio-net device for performance.
> >
> > The reason why users use vf is that we need to use a large number of virtio-net
> > devices. This number reaches 1k+.
> >
> > Based on this scenario, we need to bind vf to the backend device. Because, we
> > cannot automatically complete the creation of the virtio-net backend device when
> > the user creates a vf.
> >
> > ## Migration
> >
> > In addition, let's consider another scenario of migration. If a vm is migrated
> > from another host, of course its corresponding virtio device is also migrated to
> > the DPU. At this time, our newly created vf can only be used by the vm after it
> > is bound to the migrated device. We do not want this vf to be a brand new
> > device.
> >
> > ## Abstraction
> >
> > So, this is how I understand the process of creating vf:
> >
> > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> >     is only a default backend. It does not fully meet our expectations.
> > 2. Create device or migrate device
> > 3. Bind the backend virtio device to the vf
>
> 3) should come before 2)?
>
> Who is going to do 3) btw, is it the user? If yes, for example, if a
> user wants another 4 queue virtio-net devices, after purchase, how
> does the user know its id?

Got the id from the management platform.

>
> >
> > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > this default device, such as modifying its mac. In the future, we can use admin
> > queue to modify its msix vector and other configurations.
> >
> > But we should allow, we bind a backend virtio device to a certain vf. This is
> > useful for live migration and virtio devices with special configurations.
>
> All of these could be addressed if a dynamic provisioning model is
> implemented (SIOV or transport virtqueue). Trying to have a workaround
> in SR-IOV might be tricky.


SR-IOV vf is native PCI device, this is the advancement.


>
> >
> > The design of virtio itself is two layers, and virtio should allow switching the
> > transport layer by nature. This is our advantage.
>
> Is it not switching the transport layer but about binding/unbinding
> vitio devices to VF?

YES.

>
> Is a new capability or similar admin cmd sufficient in this case?

All is ok.


>
> struct virtio_pci_bind_cap {
>         struct virtio_pci_cap cap;
>         u16 bind; // virtio_device_id
>         u16 unbind; // virtio_device_id
> };

You mean that the "bind" or "unbind" is writeable?

Thanks.

>
> Thanks
>
> >
> > ## About the identity
> >
> > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > be compatible with all devices. In the network scenario, it is actually possible
> > to use a mac.
> >
> > Perhaps, introduce a standard id for each device/driver, I think this may be
> > more in line with the habit of virtio spec.
> >
> > Thanks.
> >
> > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > by the user, the virtio device can not choose one VF to bind random.
> > > So only the user knows how to bind the virtio devices to the VFs.
> > > On the other hand, generally the virtio devices are not created by the user
> > > inside the guest OS. This requires some management platform to participate.
> > >
> > > So the usage of this command:
> > > 1. The user purchases a virtio network card on the management platform,
> > >    and sets the ip, queue number, etc. The user obtains the identity of
> > >    the network card.
> > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > 3. The user binds the net crad to a VF with identity through the command
> > >    of the patch
> > >
> > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > ---
> > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/admin.tex b/admin.tex
> > > index 2efd4d7..64d0667 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >  \hline \hline
> > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >  \hline
> > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > >  \hline
> > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >
> > > +\subsubsection{Bind the device for member}
> > > +
> > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > +by the user, the virtio device can not choose one VF to bind random.
> > > +So only the user knows how to bind the virtio devices to the VFs.
> > > +On the other hand, generally the virtio devices are not created by the user
> > > +inside the guest OS. This requires some management platform to participate.
> > > +
> > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > +devices.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_bind {
> > > +    u64 identity;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The user got the \field{identity} from the management platform, that is not
> > > +included by this spec.
> > > +
> > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > +
> > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > +
> > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > +device.
> > > +
> > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > +
> > > +Every device MUST have one unique \field{identity} in the host.
> > > +
> > > +If the PF device can not find the device by the \field{identity},
> > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > +
> > > +If the device is found by the \field{identity}, the device MUST work as the
> > > +device of this group member specified by the \field{group_member_id}.
> > > +
> > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > >
> > >  An administration virtqueue of an owner device is used to submit
> > > --
> > > 2.32.0.3.g01195cf9f
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> >
> > Thanks Parav for pointing it out. We may have some gaps on the case.
> >
> > Let me introduce our case, which I think it is simple and should be easy to
> > understand.
> >
> > First, the user (customer) purchased a bare metal machine.
> >
> > ## Bare metal machine
> >
> > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > virtual machine, it is a physical machine, and the difference between it and a
> > general physical machine is that its PCI is connected to a device similar to a
> > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > These devices are managed by the vendor, and must be created and purchased
> > on the vendor's management platform.
> >
> > ## DPU
> >
> > There is a software implementation in the DPU, which will respond to PCI
> > operations. But as mentioned above, resources such as network cards must be
> > purchased and created before they can exist. So users can create VF, which is
> > just a pci-level operation, but there may not be a corresponding backend.
> >
> > ## Management Platform
> >
> > The creation and configuration of devices is realized on the management
> > platform.
> >
> > After the user completed the purchase on the management platform (this is an
> > independent platform provided by the vendor and has nothing to do with
> > virtio), then there will be a corresponding device implementation in the DPU.
> > This includes some user configurations, available bandwidth resources and other
> > information.
> >
> > ## Usage
> >
> > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > or VF into the VM. Or users can create a large number of dockers, all of which
> > use a separate virtio-net device for performance.
> >
> > The reason why users use vf is that we need to use a large number of virtio-net
> > devices. This number reaches 1k+.
> >
> > Based on this scenario, we need to bind vf to the backend device. Because, we
> > cannot automatically complete the creation of the virtio-net backend device when
> > the user creates a vf.
> >
> > ## Migration
> >
> > In addition, let's consider another scenario of migration. If a vm is migrated
> > from another host, of course its corresponding virtio device is also migrated to
> > the DPU. At this time, our newly created vf can only be used by the vm after it
> > is bound to the migrated device. We do not want this vf to be a brand new
> > device.
> >
> > ## Abstraction
> >
> > So, this is how I understand the process of creating vf:
> >
> > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> >     is only a default backend. It does not fully meet our expectations.
> > 2. Create device or migrate device
> > 3. Bind the backend virtio device to the vf
>
> 3) should come before 2)?
>
> Who is going to do 3) btw, is it the user? If yes, for example, if a
> user wants another 4 queue virtio-net devices, after purchase, how
> does the user know its id?

Got the id from the management platform.

>
> >
> > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > this default device, such as modifying its mac. In the future, we can use admin
> > queue to modify its msix vector and other configurations.
> >
> > But we should allow, we bind a backend virtio device to a certain vf. This is
> > useful for live migration and virtio devices with special configurations.
>
> All of these could be addressed if a dynamic provisioning model is
> implemented (SIOV or transport virtqueue). Trying to have a workaround
> in SR-IOV might be tricky.


SR-IOV vf is native PCI device, this is the advancement.


>
> >
> > The design of virtio itself is two layers, and virtio should allow switching the
> > transport layer by nature. This is our advantage.
>
> Is it not switching the transport layer but about binding/unbinding
> vitio devices to VF?

YES.

>
> Is a new capability or similar admin cmd sufficient in this case?

All is ok.


>
> struct virtio_pci_bind_cap {
>         struct virtio_pci_cap cap;
>         u16 bind; // virtio_device_id
>         u16 unbind; // virtio_device_id
> };

You mean that the "bind" or "unbind" is writeable?

Thanks.

>
> Thanks
>
> >
> > ## About the identity
> >
> > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > be compatible with all devices. In the network scenario, it is actually possible
> > to use a mac.
> >
> > Perhaps, introduce a standard id for each device/driver, I think this may be
> > more in line with the habit of virtio spec.
> >
> > Thanks.
> >
> > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > by the user, the virtio device can not choose one VF to bind random.
> > > So only the user knows how to bind the virtio devices to the VFs.
> > > On the other hand, generally the virtio devices are not created by the user
> > > inside the guest OS. This requires some management platform to participate.
> > >
> > > So the usage of this command:
> > > 1. The user purchases a virtio network card on the management platform,
> > >    and sets the ip, queue number, etc. The user obtains the identity of
> > >    the network card.
> > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > 3. The user binds the net crad to a VF with identity through the command
> > >    of the patch
> > >
> > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > ---
> > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/admin.tex b/admin.tex
> > > index 2efd4d7..64d0667 100644
> > > --- a/admin.tex
> > > +++ b/admin.tex
> > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >  \hline \hline
> > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > >  \hline
> > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > >  \hline
> > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > >
> > > +\subsubsection{Bind the device for member}
> > > +
> > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > +by the user, the virtio device can not choose one VF to bind random.
> > > +So only the user knows how to bind the virtio devices to the VFs.
> > > +On the other hand, generally the virtio devices are not created by the user
> > > +inside the guest OS. This requires some management platform to participate.
> > > +
> > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > +devices.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_admin_cmd_bind {
> > > +    u64 identity;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +The user got the \field{identity} from the management platform, that is not
> > > +included by this spec.
> > > +
> > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > +
> > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > +
> > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > +device.
> > > +
> > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > +
> > > +Every device MUST have one unique \field{identity} in the host.
> > > +
> > > +If the PF device can not find the device by the \field{identity},
> > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > +
> > > +If the device is found by the \field{identity}, the device MUST work as the
> > > +device of this group member specified by the \field{group_member_id}.
> > > +
> > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > >
> > >  An administration virtqueue of an owner device is used to submit
> > > --
> > > 2.32.0.3.g01195cf9f
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, June 27, 2023 4:23 AM
> 
> Thanks Parav for pointing it out. We may have some gaps on the case.
> 
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
> 
> First, the user (customer) purchased a bare metal machine.
> 
> ## Bare metal machine
> 
> Let me briefly explain the characteristics of a bare metal machine. It is not a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a DPU.
> This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
> 
Each of this device can have a unique serial number or id.
Or a parent PF can have the unique serial number.

> ## DPU
> 
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
> 
> ## Management Platform
> 
> The creation and configuration of devices is realized on the management
> platform.
> 
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and
> other information.
> 
> ## Usage
> 
> Since the user is directly on the HOST, the user can create VMs, passthrough PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
> 
> The reason why users use vf is that we need to use a large number of virtio-net
> devices. This number reaches 1k+.
> 
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device
> when the user creates a vf.
> 
Yes, this is common practice for us as well.
A backend device needs to know the unique serial number/id of the virtio device.

> ## Migration
> 
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated to
> the DPU. At this time, our newly created vf can only be used by the vm after it is
> bound to the migrated device. We do not want this vf to be a brand new device.
>
Same for us too.
 
> ## Abstraction
> 
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf
> 
> In most scenarios, the first step may be enough. We can make some fine-tuning
> on this default device, such as modifying its mac. In the future, we can use
> admin queue to modify its msix vector and other configurations.
> 
Yep.

> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.
>
This is the job of non virtio layer to do the work in backend.
 
> The design of virtio itself is two layers, and virtio should allow switching the
> transport layer by nature. This is our advantage.
> 
> ## About the identity
> 
> In this patch, I used a vendor's id. The purpose of this is that I hope to be
> compatible with all devices. In the network scenario, it is actually possible to
> use a mac.
>
Mac is not good as during migration there is transient where two vfs will have same mac.

So we should use the unique device identifier.

There are few options that I am aware of.

1. Have unique serial number for the PCI PF in the PCI VPD data and VF number to uniquely identify the VF of a PF.
Management system backend will know and able to identify this.
Pros:
This does not need any virtio spec involvement. Can be done today on existing virtio device.
Cons:
There is no single device handle to work with backend, but not a big problem.
If the REST APIs etc of the VPC are crafted properly.

2. Expose unique serial number per each VF using same above PCI VPD data.
This option #2 is good in theory, not efficient to implement as PCI level.
Pros:
a. Unique per device.
b. accessible without binding to the actual driver (vfio vs virtio for different use case)

3. Expose unique serial number/identifier at virtio device level over a virtqueue interface
Pros:
a. achieves the desired scale
b. unique per device
Cons:
a. One needs to bind to the driver to discover the unique device identifier and involve the device reset flow.
Not an optimal solution when dealing with container.

In practice, we found #1 solves most issues.

 
> Perhaps, introduce a standard id for each device/driver, I think this may be more
> in line with the habit of virtio spec.
> 

One way this is solved today for non virtio already is:

when a VF is created a PCI VF device should have the VPD data to contain a unique ID.

> Thanks.
> 
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo
> <xuanzhuo@linux.alibaba.com> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so
> > the virtio devices don't know about these VFs. Because each VF may be
> > assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the
> > user inside the guest OS. This requires some management platform to
> participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs 3. The user binds
> > the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of
> commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one
> group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \field{VF Enable}
> > refer to registers within the SR-IOV Extended  Capability as specified by
> \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so
> > +the virtio devices don't know about these VFs. Because each VF may be
> > +assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by
> > +the user inside the guest OS. This requires some management platform to
> participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the
> > +virtio devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that
> > +is not included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the
> > +virtio device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity}, the
> > +\field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work
> > +as the device of this group member specified by the
> \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a
> > Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and to
> > minimize spam in the list archive, subscription is required before
> > posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License:
> > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines:
> > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, June 27, 2023 4:23 AM
> 
> Thanks Parav for pointing it out. We may have some gaps on the case.
> 
> Let me introduce our case, which I think it is simple and should be easy to
> understand.
> 
> First, the user (customer) purchased a bare metal machine.
> 
> ## Bare metal machine
> 
> Let me briefly explain the characteristics of a bare metal machine. It is not a
> virtual machine, it is a physical machine, and the difference between it and a
> general physical machine is that its PCI is connected to a device similar to a DPU.
> This DPU provides devices such as virtio-blk/net to the host through PCI.
> These devices are managed by the vendor, and must be created and purchased
> on the vendor's management platform.
> 
Each of this device can have a unique serial number or id.
Or a parent PF can have the unique serial number.

> ## DPU
> 
> There is a software implementation in the DPU, which will respond to PCI
> operations. But as mentioned above, resources such as network cards must be
> purchased and created before they can exist. So users can create VF, which is
> just a pci-level operation, but there may not be a corresponding backend.
> 
> ## Management Platform
> 
> The creation and configuration of devices is realized on the management
> platform.
> 
> After the user completed the purchase on the management platform (this is an
> independent platform provided by the vendor and has nothing to do with
> virtio), then there will be a corresponding device implementation in the DPU.
> This includes some user configurations, available bandwidth resources and
> other information.
> 
> ## Usage
> 
> Since the user is directly on the HOST, the user can create VMs, passthrough PF
> or VF into the VM. Or users can create a large number of dockers, all of which
> use a separate virtio-net device for performance.
> 
> The reason why users use vf is that we need to use a large number of virtio-net
> devices. This number reaches 1k+.
> 
> Based on this scenario, we need to bind vf to the backend device. Because, we
> cannot automatically complete the creation of the virtio-net backend device
> when the user creates a vf.
> 
Yes, this is common practice for us as well.
A backend device needs to know the unique serial number/id of the virtio device.

> ## Migration
> 
> In addition, let's consider another scenario of migration. If a vm is migrated
> from another host, of course its corresponding virtio device is also migrated to
> the DPU. At this time, our newly created vf can only be used by the vm after it is
> bound to the migrated device. We do not want this vf to be a brand new device.
>
Same for us too.
 
> ## Abstraction
> 
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf
> 
> In most scenarios, the first step may be enough. We can make some fine-tuning
> on this default device, such as modifying its mac. In the future, we can use
> admin queue to modify its msix vector and other configurations.
> 
Yep.

> But we should allow, we bind a backend virtio device to a certain vf. This is
> useful for live migration and virtio devices with special configurations.
>
This is the job of non virtio layer to do the work in backend.
 
> The design of virtio itself is two layers, and virtio should allow switching the
> transport layer by nature. This is our advantage.
> 
> ## About the identity
> 
> In this patch, I used a vendor's id. The purpose of this is that I hope to be
> compatible with all devices. In the network scenario, it is actually possible to
> use a mac.
>
Mac is not good as during migration there is transient where two vfs will have same mac.

So we should use the unique device identifier.

There are few options that I am aware of.

1. Have unique serial number for the PCI PF in the PCI VPD data and VF number to uniquely identify the VF of a PF.
Management system backend will know and able to identify this.
Pros:
This does not need any virtio spec involvement. Can be done today on existing virtio device.
Cons:
There is no single device handle to work with backend, but not a big problem.
If the REST APIs etc of the VPC are crafted properly.

2. Expose unique serial number per each VF using same above PCI VPD data.
This option #2 is good in theory, not efficient to implement as PCI level.
Pros:
a. Unique per device.
b. accessible without binding to the actual driver (vfio vs virtio for different use case)

3. Expose unique serial number/identifier at virtio device level over a virtqueue interface
Pros:
a. achieves the desired scale
b. unique per device
Cons:
a. One needs to bind to the driver to discover the unique device identifier and involve the device reset flow.
Not an optimal solution when dealing with container.

In practice, we found #1 solves most issues.

 
> Perhaps, introduce a standard id for each device/driver, I think this may be more
> in line with the habit of virtio spec.
> 

One way this is solved today for non virtio already is:

when a VF is created a PCI VF device should have the VPD data to contain a unique ID.

> Thanks.
> 
> On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo
> <xuanzhuo@linux.alibaba.com> wrote:
> > The VFs of the SR-IOV are created by the user inside the guest OS, so
> > the virtio devices don't know about these VFs. Because each VF may be
> > assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > So only the user knows how to bind the virtio devices to the VFs.
> > On the other hand, generally the virtio devices are not created by the
> > user inside the guest OS. This requires some management platform to
> participate.
> >
> > So the usage of this command:
> > 1. The user purchases a virtio network card on the management platform,
> >    and sets the ip, queue number, etc. The user obtains the identity of
> >    the network card.
> > 2. The user creates a VF with echo 8 > sriov_numvfs 3. The user binds
> > the net crad to a VF with identity through the command
> >    of the patch
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >  1 file changed, 40 insertions(+), 1 deletion(-)
> >
> > diff --git a/admin.tex b/admin.tex
> > index 2efd4d7..64d0667 100644
> > --- a/admin.tex
> > +++ b/admin.tex
> > @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \hline \hline
> >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of
> commands supported for this group type    \\
> >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of
> commands used for this group type \\
> > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one
> group member \\
> > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >  \hline
> >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a
> different structure)    \\
> >  \hline
> > @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti  \field{VF Enable}
> > refer to registers within the SR-IOV Extended  Capability as specified by
> \hyperref[intro:PCIe]{[PCIe]}.
> >
> > +\subsubsection{Bind the device for member}
> > +
> > +The VFs of the SR-IOV are created by the user inside the guest OS, so
> > +the virtio devices don't know about these VFs. Because each VF may be
> > +assigned a different role by the user, the virtio device can not choose one VF
> to bind random.
> > +So only the user knows how to bind the virtio devices to the VFs.
> > +On the other hand, generally the virtio devices are not created by
> > +the user inside the guest OS. This requires some management platform to
> participate.
> > +
> > +So we introduce a new admin queue command to bind the VFs and the
> > +virtio devices.
> > +
> > +\begin{lstlisting}
> > +struct virtio_admin_cmd_bind {
> > +    u64 identity;
> > +};
> > +\end{lstlisting}
> > +
> > +The user got the \field{identity} from the management platform, that
> > +is not included by this spec.
> > +
> > +\drivernormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> \field{group_member_id} MUST be set.
> > +
> > +The \field{identity} is passed by the user. It is the identity of the
> > +virtio device.
> > +
> > +\devicenormative{\paragraph}{Group administration commands}{Basic
> > +Facilities of a Virtio Device / Device groups / Group administration
> > +commands / Bind the device for member}
> > +
> > +Every device MUST have one unique \field{identity} in the host.
> > +
> > +If the PF device can not find the device by the \field{identity}, the
> > +\field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > +
> > +If the device is found by the \field{identity}, the device MUST work
> > +as the device of this group member specified by the
> \field{group_member_id}.
> > +
> >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a
> > Virtio Device / Administration Virtqueues}
> >
> >  An administration virtqueue of an owner device is used to submit
> > --
> > 2.32.0.3.g01195cf9f
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and to
> > minimize spam in the list archive, subscription is required before
> > posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License:
> > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines:
> > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

I can see this making sense as a feature bit that says VFs are not
initialized by default and must first be setup through an admin command.
This will likely need to be a feature bit because it's changing
behaviour outside of admin commands.

Then, we can have:

	ADMIN_SETUP VF#
	ADMIN_CLEANUP VF#

I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.


Why do we need an id as a level of indirection though? What is wrong
with just using VF# directly?



-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> So, this is how I understand the process of creating vf:
> 
> 1. Create a PCI VF, at this time there may be no backend virtio device, or there
>     is only a default backend. It does not fully meet our expectations.
> 2. Create device or migrate device
> 3. Bind the backend virtio device to the vf

I can see this making sense as a feature bit that says VFs are not
initialized by default and must first be setup through an admin command.
This will likely need to be a feature bit because it's changing
behaviour outside of admin commands.

Then, we can have:

	ADMIN_SETUP VF#
	ADMIN_CLEANUP VF#

I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.


Why do we need an id as a level of indirection though? What is wrong
with just using VF# directly?



-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > So, this is how I understand the process of creating vf:
> >
> > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> >     is only a default backend. It does not fully meet our expectations.
> > 2. Create device or migrate device
> > 3. Bind the backend virtio device to the vf
>
> I can see this making sense as a feature bit that says VFs are not
> initialized by default and must first be setup through an admin command.
> This will likely need to be a feature bit because it's changing
> behaviour outside of admin commands.
>
> Then, we can have:
>
> 	ADMIN_SETUP VF#
> 	ADMIN_CLEANUP VF#
>
> I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.

Great!!

>
>
> Why do we need an id as a level of indirection though? What is wrong
> with just using VF# directly?

I think VF# is ok. I also need to use it. But we need an ID for virtio device
not the transport(PF, VF).

What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
to connect the virtio driver and the virtio device. Right?

The virtio device does not necessarily exist depending on PCI. For example, a
virtio device is migrated from another DPU, and it is not associated with any
PCI. What I have always wanted to say is that this device(not PCI) must have its
own ID, which has nothing to do with the transport.

Now we want to use this migrated device and connect it to the corresponding
vm (migrated from the same host). We can passthrough vf to this vm. But how do I
tell our DPU to bind this migrated device with this vf?
We can specify the VF by the VF#, but how can we specify the virtio device?
Maybe there are two migrated virtio device.

Maybe we should compare it to the case of using PF directly before. The biggest
difference here is that PF is directly hot-plugged by the DPU, so when a
user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
on the host. Now the vf is created by the user, and only the user knows how to
use each one. We cannot directly bind the device purchased or migrated by the
user to a certain vf. We need the user in the host to bind the vf(transport) to
a certain virtio device.

Thanks.






>
>
>
> --
> MST
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > So, this is how I understand the process of creating vf:
> >
> > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> >     is only a default backend. It does not fully meet our expectations.
> > 2. Create device or migrate device
> > 3. Bind the backend virtio device to the vf
>
> I can see this making sense as a feature bit that says VFs are not
> initialized by default and must first be setup through an admin command.
> This will likely need to be a feature bit because it's changing
> behaviour outside of admin commands.
>
> Then, we can have:
>
> 	ADMIN_SETUP VF#
> 	ADMIN_CLEANUP VF#
>
> I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.

Great!!

>
>
> Why do we need an id as a level of indirection though? What is wrong
> with just using VF# directly?

I think VF# is ok. I also need to use it. But we need an ID for virtio device
not the transport(PF, VF).

What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
to connect the virtio driver and the virtio device. Right?

The virtio device does not necessarily exist depending on PCI. For example, a
virtio device is migrated from another DPU, and it is not associated with any
PCI. What I have always wanted to say is that this device(not PCI) must have its
own ID, which has nothing to do with the transport.

Now we want to use this migrated device and connect it to the corresponding
vm (migrated from the same host). We can passthrough vf to this vm. But how do I
tell our DPU to bind this migrated device with this vf?
We can specify the VF by the VF#, but how can we specify the virtio device?
Maybe there are two migrated virtio device.

Maybe we should compare it to the case of using PF directly before. The biggest
difference here is that PF is directly hot-plugged by the DPU, so when a
user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
on the host. Now the vf is created by the user, and only the user knows how to
use each one. We cannot directly bind the device purchased or migrated by the
user to a certain vf. We need the user in the host to bind the vf(transport) to
a certain virtio device.

Thanks.






>
>
>
> --
> MST
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > >
> > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > >
> > > Let me introduce our case, which I think it is simple and should be easy to
> > > understand.
> > >
> > > First, the user (customer) purchased a bare metal machine.
> > >
> > > ## Bare metal machine
> > >
> > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > virtual machine, it is a physical machine, and the difference between it and a
> > > general physical machine is that its PCI is connected to a device similar to a
> > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > These devices are managed by the vendor, and must be created and purchased
> > > on the vendor's management platform.
> > >
> > > ## DPU
> > >
> > > There is a software implementation in the DPU, which will respond to PCI
> > > operations. But as mentioned above, resources such as network cards must be
> > > purchased and created before they can exist. So users can create VF, which is
> > > just a pci-level operation, but there may not be a corresponding backend.
> > >
> > > ## Management Platform
> > >
> > > The creation and configuration of devices is realized on the management
> > > platform.
> > >
> > > After the user completed the purchase on the management platform (this is an
> > > independent platform provided by the vendor and has nothing to do with
> > > virtio), then there will be a corresponding device implementation in the DPU.
> > > This includes some user configurations, available bandwidth resources and other
> > > information.
> > >
> > > ## Usage
> > >
> > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > use a separate virtio-net device for performance.
> > >
> > > The reason why users use vf is that we need to use a large number of virtio-net
> > > devices. This number reaches 1k+.
> > >
> > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > cannot automatically complete the creation of the virtio-net backend device when
> > > the user creates a vf.
> > >
> > > ## Migration
> > >
> > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > from another host, of course its corresponding virtio device is also migrated to
> > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > is bound to the migrated device. We do not want this vf to be a brand new
> > > device.
> > >
> > > ## Abstraction
> > >
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > >     is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > 3) should come before 2)?
> >
> > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > user wants another 4 queue virtio-net devices, after purchase, how
> > does the user know its id?
>
> Got the id from the management platform.

So it can do the binding via that management platform which this
became a cloud vendor specific interface.

>
> >
> > >
> > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > this default device, such as modifying its mac. In the future, we can use admin
> > > queue to modify its msix vector and other configurations.
> > >
> > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > useful for live migration and virtio devices with special configurations.
> >
> > All of these could be addressed if a dynamic provisioning model is
> > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > in SR-IOV might be tricky.
>
>
> SR-IOV vf is native PCI device, this is the advancement.

The problem is that it doesn't support flexible provisioning, e.g
create and destroy a single VF.

>
>
> >
> > >
> > > The design of virtio itself is two layers, and virtio should allow switching the
> > > transport layer by nature. This is our advantage.
> >
> > Is it not switching the transport layer but about binding/unbinding
> > vitio devices to VF?
>
> YES.
>
> >
> > Is a new capability or similar admin cmd sufficient in this case?
>
> All is ok.
>
>
> >
> > struct virtio_pci_bind_cap {
> >         struct virtio_pci_cap cap;
> >         u16 bind; // virtio_device_id
> >         u16 unbind; // virtio_device_id
> > };
>
> You mean that the "bind" or "unbind" is writeable?

Yes.

Thanks

>
> Thanks.
>
> >
> > Thanks
> >
> > >
> > > ## About the identity
> > >
> > > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > > be compatible with all devices. In the network scenario, it is actually possible
> > > to use a mac.
> > >
> > > Perhaps, introduce a standard id for each device/driver, I think this may be
> > > more in line with the habit of virtio spec.
> > >
> > > Thanks.
> > >
> > > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > > by the user, the virtio device can not choose one VF to bind random.
> > > > So only the user knows how to bind the virtio devices to the VFs.
> > > > On the other hand, generally the virtio devices are not created by the user
> > > > inside the guest OS. This requires some management platform to participate.
> > > >
> > > > So the usage of this command:
> > > > 1. The user purchases a virtio network card on the management platform,
> > > >    and sets the ip, queue number, etc. The user obtains the identity of
> > > >    the network card.
> > > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > > 3. The user binds the net crad to a VF with identity through the command
> > > >    of the patch
> > > >
> > > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > ---
> > > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/admin.tex b/admin.tex
> > > > index 2efd4d7..64d0667 100644
> > > > --- a/admin.tex
> > > > +++ b/admin.tex
> > > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > >  \hline \hline
> > > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > >  \hline
> > > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > > >  \hline
> > > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >
> > > > +\subsubsection{Bind the device for member}
> > > > +
> > > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > > +by the user, the virtio device can not choose one VF to bind random.
> > > > +So only the user knows how to bind the virtio devices to the VFs.
> > > > +On the other hand, generally the virtio devices are not created by the user
> > > > +inside the guest OS. This requires some management platform to participate.
> > > > +
> > > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > > +devices.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_bind {
> > > > +    u64 identity;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +The user got the \field{identity} from the management platform, that is not
> > > > +included by this spec.
> > > > +
> > > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > +
> > > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > > +
> > > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > > +device.
> > > > +
> > > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > +
> > > > +Every device MUST have one unique \field{identity} in the host.
> > > > +
> > > > +If the PF device can not find the device by the \field{identity},
> > > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > > +
> > > > +If the device is found by the \field{identity}, the device MUST work as the
> > > > +device of this group member specified by the \field{group_member_id}.
> > > > +
> > > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > >
> > > >  An administration virtqueue of an owner device is used to submit
> > > > --
> > > > 2.32.0.3.g01195cf9f
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> >
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > >
> > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > >
> > > Let me introduce our case, which I think it is simple and should be easy to
> > > understand.
> > >
> > > First, the user (customer) purchased a bare metal machine.
> > >
> > > ## Bare metal machine
> > >
> > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > virtual machine, it is a physical machine, and the difference between it and a
> > > general physical machine is that its PCI is connected to a device similar to a
> > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > These devices are managed by the vendor, and must be created and purchased
> > > on the vendor's management platform.
> > >
> > > ## DPU
> > >
> > > There is a software implementation in the DPU, which will respond to PCI
> > > operations. But as mentioned above, resources such as network cards must be
> > > purchased and created before they can exist. So users can create VF, which is
> > > just a pci-level operation, but there may not be a corresponding backend.
> > >
> > > ## Management Platform
> > >
> > > The creation and configuration of devices is realized on the management
> > > platform.
> > >
> > > After the user completed the purchase on the management platform (this is an
> > > independent platform provided by the vendor and has nothing to do with
> > > virtio), then there will be a corresponding device implementation in the DPU.
> > > This includes some user configurations, available bandwidth resources and other
> > > information.
> > >
> > > ## Usage
> > >
> > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > use a separate virtio-net device for performance.
> > >
> > > The reason why users use vf is that we need to use a large number of virtio-net
> > > devices. This number reaches 1k+.
> > >
> > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > cannot automatically complete the creation of the virtio-net backend device when
> > > the user creates a vf.
> > >
> > > ## Migration
> > >
> > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > from another host, of course its corresponding virtio device is also migrated to
> > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > is bound to the migrated device. We do not want this vf to be a brand new
> > > device.
> > >
> > > ## Abstraction
> > >
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > >     is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > 3) should come before 2)?
> >
> > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > user wants another 4 queue virtio-net devices, after purchase, how
> > does the user know its id?
>
> Got the id from the management platform.

So it can do the binding via that management platform which this
became a cloud vendor specific interface.

>
> >
> > >
> > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > this default device, such as modifying its mac. In the future, we can use admin
> > > queue to modify its msix vector and other configurations.
> > >
> > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > useful for live migration and virtio devices with special configurations.
> >
> > All of these could be addressed if a dynamic provisioning model is
> > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > in SR-IOV might be tricky.
>
>
> SR-IOV vf is native PCI device, this is the advancement.

The problem is that it doesn't support flexible provisioning, e.g
create and destroy a single VF.

>
>
> >
> > >
> > > The design of virtio itself is two layers, and virtio should allow switching the
> > > transport layer by nature. This is our advantage.
> >
> > Is it not switching the transport layer but about binding/unbinding
> > vitio devices to VF?
>
> YES.
>
> >
> > Is a new capability or similar admin cmd sufficient in this case?
>
> All is ok.
>
>
> >
> > struct virtio_pci_bind_cap {
> >         struct virtio_pci_cap cap;
> >         u16 bind; // virtio_device_id
> >         u16 unbind; // virtio_device_id
> > };
>
> You mean that the "bind" or "unbind" is writeable?

Yes.

Thanks

>
> Thanks.
>
> >
> > Thanks
> >
> > >
> > > ## About the identity
> > >
> > > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > > be compatible with all devices. In the network scenario, it is actually possible
> > > to use a mac.
> > >
> > > Perhaps, introduce a standard id for each device/driver, I think this may be
> > > more in line with the habit of virtio spec.
> > >
> > > Thanks.
> > >
> > > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > > by the user, the virtio device can not choose one VF to bind random.
> > > > So only the user knows how to bind the virtio devices to the VFs.
> > > > On the other hand, generally the virtio devices are not created by the user
> > > > inside the guest OS. This requires some management platform to participate.
> > > >
> > > > So the usage of this command:
> > > > 1. The user purchases a virtio network card on the management platform,
> > > >    and sets the ip, queue number, etc. The user obtains the identity of
> > > >    the network card.
> > > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > > 3. The user binds the net crad to a VF with identity through the command
> > > >    of the patch
> > > >
> > > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > ---
> > > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > > >
> > > > diff --git a/admin.tex b/admin.tex
> > > > index 2efd4d7..64d0667 100644
> > > > --- a/admin.tex
> > > > +++ b/admin.tex
> > > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > >  \hline \hline
> > > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > >  \hline
> > > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > > >  \hline
> > > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >
> > > > +\subsubsection{Bind the device for member}
> > > > +
> > > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > > +by the user, the virtio device can not choose one VF to bind random.
> > > > +So only the user knows how to bind the virtio devices to the VFs.
> > > > +On the other hand, generally the virtio devices are not created by the user
> > > > +inside the guest OS. This requires some management platform to participate.
> > > > +
> > > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > > +devices.
> > > > +
> > > > +\begin{lstlisting}
> > > > +struct virtio_admin_cmd_bind {
> > > > +    u64 identity;
> > > > +};
> > > > +\end{lstlisting}
> > > > +
> > > > +The user got the \field{identity} from the management platform, that is not
> > > > +included by this spec.
> > > > +
> > > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > +
> > > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > > +
> > > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > > +device.
> > > > +
> > > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > +
> > > > +Every device MUST have one unique \field{identity} in the host.
> > > > +
> > > > +If the PF device can not find the device by the \field{identity},
> > > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > > +
> > > > +If the device is found by the \field{identity}, the device MUST work as the
> > > > +device of this group member specified by the \field{group_member_id}.
> > > > +
> > > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > >
> > > >  An administration virtqueue of an owner device is used to submit
> > > > --
> > > > 2.32.0.3.g01195cf9f
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > >
> > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > >
> > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > understand.
> > > >
> > > > First, the user (customer) purchased a bare metal machine.
> > > >
> > > > ## Bare metal machine
> > > >
> > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > general physical machine is that its PCI is connected to a device similar to a
> > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > These devices are managed by the vendor, and must be created and purchased
> > > > on the vendor's management platform.
> > > >
> > > > ## DPU
> > > >
> > > > There is a software implementation in the DPU, which will respond to PCI
> > > > operations. But as mentioned above, resources such as network cards must be
> > > > purchased and created before they can exist. So users can create VF, which is
> > > > just a pci-level operation, but there may not be a corresponding backend.
> > > >
> > > > ## Management Platform
> > > >
> > > > The creation and configuration of devices is realized on the management
> > > > platform.
> > > >
> > > > After the user completed the purchase on the management platform (this is an
> > > > independent platform provided by the vendor and has nothing to do with
> > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > This includes some user configurations, available bandwidth resources and other
> > > > information.
> > > >
> > > > ## Usage
> > > >
> > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > use a separate virtio-net device for performance.
> > > >
> > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > devices. This number reaches 1k+.
> > > >
> > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > the user creates a vf.
> > > >
> > > > ## Migration
> > > >
> > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > from another host, of course its corresponding virtio device is also migrated to
> > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > device.
> > > >
> > > > ## Abstraction
> > > >
> > > > So, this is how I understand the process of creating vf:
> > > >
> > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > >     is only a default backend. It does not fully meet our expectations.
> > > > 2. Create device or migrate device
> > > > 3. Bind the backend virtio device to the vf
> > >
> > > 3) should come before 2)?
> > >
> > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > user wants another 4 queue virtio-net devices, after purchase, how
> > > does the user know its id?
> >
> > Got the id from the management platform.
>
> So it can do the binding via that management platform which this
> became a cloud vendor specific interface.

In our scenario, this is bound by the user using this id and vf id in the os.

>
> >
> > >
> > > >
> > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > queue to modify its msix vector and other configurations.
> > > >
> > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > useful for live migration and virtio devices with special configurations.
> > >
> > > All of these could be addressed if a dynamic provisioning model is
> > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > in SR-IOV might be tricky.
> >
> >
> > SR-IOV vf is native PCI device, this is the advancement.
>
> The problem is that it doesn't support flexible provisioning, e.g
> create and destroy a single VF.

YES. ^_^!!


>
> >
> >
> > >
> > > >
> > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > transport layer by nature. This is our advantage.
> > >
> > > Is it not switching the transport layer but about binding/unbinding
> > > vitio devices to VF?
> >
> > YES.
> >
> > >
> > > Is a new capability or similar admin cmd sufficient in this case?
> >
> > All is ok.
> >
> >
> > >
> > > struct virtio_pci_bind_cap {
> > >         struct virtio_pci_cap cap;
> > >         u16 bind; // virtio_device_id
> > >         u16 unbind; // virtio_device_id
> > > };
> >
> > You mean that the "bind" or "unbind" is writeable?

This is a good idea.

Thanks.


>
> Yes.
>
> Thanks
>
> >
> > Thanks.
> >
> > >
> > > Thanks
> > >
> > > >
> > > > ## About the identity
> > > >
> > > > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > > > be compatible with all devices. In the network scenario, it is actually possible
> > > > to use a mac.
> > > >
> > > > Perhaps, introduce a standard id for each device/driver, I think this may be
> > > > more in line with the habit of virtio spec.
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > > > by the user, the virtio device can not choose one VF to bind random.
> > > > > So only the user knows how to bind the virtio devices to the VFs.
> > > > > On the other hand, generally the virtio devices are not created by the user
> > > > > inside the guest OS. This requires some management platform to participate.
> > > > >
> > > > > So the usage of this command:
> > > > > 1. The user purchases a virtio network card on the management platform,
> > > > >    and sets the ip, queue number, etc. The user obtains the identity of
> > > > >    the network card.
> > > > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > > > 3. The user binds the net crad to a VF with identity through the command
> > > > >    of the patch
> > > > >
> > > > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > > ---
> > > > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/admin.tex b/admin.tex
> > > > > index 2efd4d7..64d0667 100644
> > > > > --- a/admin.tex
> > > > > +++ b/admin.tex
> > > > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > > >  \hline \hline
> > > > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > > > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > >  \hline
> > > > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > > > >  \hline
> > > > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > > > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > > >
> > > > > +\subsubsection{Bind the device for member}
> > > > > +
> > > > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > > > +by the user, the virtio device can not choose one VF to bind random.
> > > > > +So only the user knows how to bind the virtio devices to the VFs.
> > > > > +On the other hand, generally the virtio devices are not created by the user
> > > > > +inside the guest OS. This requires some management platform to participate.
> > > > > +
> > > > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > > > +devices.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_admin_cmd_bind {
> > > > > +    u64 identity;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +The user got the \field{identity} from the management platform, that is not
> > > > > +included by this spec.
> > > > > +
> > > > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > > +
> > > > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > > > +
> > > > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > > > +device.
> > > > > +
> > > > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > > +
> > > > > +Every device MUST have one unique \field{identity} in the host.
> > > > > +
> > > > > +If the PF device can not find the device by the \field{identity},
> > > > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > > > +
> > > > > +If the device is found by the \field{identity}, the device MUST work as the
> > > > > +device of this group member specified by the \field{group_member_id}.
> > > > > +
> > > > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > > >
> > > > >  An administration virtqueue of an owner device is used to submit
> > > > > --
> > > > > 2.32.0.3.g01195cf9f
> > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
> > >
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > >
> > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > >
> > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > understand.
> > > >
> > > > First, the user (customer) purchased a bare metal machine.
> > > >
> > > > ## Bare metal machine
> > > >
> > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > general physical machine is that its PCI is connected to a device similar to a
> > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > These devices are managed by the vendor, and must be created and purchased
> > > > on the vendor's management platform.
> > > >
> > > > ## DPU
> > > >
> > > > There is a software implementation in the DPU, which will respond to PCI
> > > > operations. But as mentioned above, resources such as network cards must be
> > > > purchased and created before they can exist. So users can create VF, which is
> > > > just a pci-level operation, but there may not be a corresponding backend.
> > > >
> > > > ## Management Platform
> > > >
> > > > The creation and configuration of devices is realized on the management
> > > > platform.
> > > >
> > > > After the user completed the purchase on the management platform (this is an
> > > > independent platform provided by the vendor and has nothing to do with
> > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > This includes some user configurations, available bandwidth resources and other
> > > > information.
> > > >
> > > > ## Usage
> > > >
> > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > use a separate virtio-net device for performance.
> > > >
> > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > devices. This number reaches 1k+.
> > > >
> > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > the user creates a vf.
> > > >
> > > > ## Migration
> > > >
> > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > from another host, of course its corresponding virtio device is also migrated to
> > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > device.
> > > >
> > > > ## Abstraction
> > > >
> > > > So, this is how I understand the process of creating vf:
> > > >
> > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > >     is only a default backend. It does not fully meet our expectations.
> > > > 2. Create device or migrate device
> > > > 3. Bind the backend virtio device to the vf
> > >
> > > 3) should come before 2)?
> > >
> > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > user wants another 4 queue virtio-net devices, after purchase, how
> > > does the user know its id?
> >
> > Got the id from the management platform.
>
> So it can do the binding via that management platform which this
> became a cloud vendor specific interface.

In our scenario, this is bound by the user using this id and vf id in the os.

>
> >
> > >
> > > >
> > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > queue to modify its msix vector and other configurations.
> > > >
> > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > useful for live migration and virtio devices with special configurations.
> > >
> > > All of these could be addressed if a dynamic provisioning model is
> > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > in SR-IOV might be tricky.
> >
> >
> > SR-IOV vf is native PCI device, this is the advancement.
>
> The problem is that it doesn't support flexible provisioning, e.g
> create and destroy a single VF.

YES. ^_^!!


>
> >
> >
> > >
> > > >
> > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > transport layer by nature. This is our advantage.
> > >
> > > Is it not switching the transport layer but about binding/unbinding
> > > vitio devices to VF?
> >
> > YES.
> >
> > >
> > > Is a new capability or similar admin cmd sufficient in this case?
> >
> > All is ok.
> >
> >
> > >
> > > struct virtio_pci_bind_cap {
> > >         struct virtio_pci_cap cap;
> > >         u16 bind; // virtio_device_id
> > >         u16 unbind; // virtio_device_id
> > > };
> >
> > You mean that the "bind" or "unbind" is writeable?

This is a good idea.

Thanks.


>
> Yes.
>
> Thanks
>
> >
> > Thanks.
> >
> > >
> > > Thanks
> > >
> > > >
> > > > ## About the identity
> > > >
> > > > In this patch, I used a vendor's id. The purpose of this is that I hope to
> > > > be compatible with all devices. In the network scenario, it is actually possible
> > > > to use a mac.
> > > >
> > > > Perhaps, introduce a standard id for each device/driver, I think this may be
> > > > more in line with the habit of virtio spec.
> > > >
> > > > Thanks.
> > > >
> > > > On Mon, 26 Jun 2023 14:22:10 +0800, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > > devices don't know about these VFs. Because each VF may be assigned a different role
> > > > > by the user, the virtio device can not choose one VF to bind random.
> > > > > So only the user knows how to bind the virtio devices to the VFs.
> > > > > On the other hand, generally the virtio devices are not created by the user
> > > > > inside the guest OS. This requires some management platform to participate.
> > > > >
> > > > > So the usage of this command:
> > > > > 1. The user purchases a virtio network card on the management platform,
> > > > >    and sets the ip, queue number, etc. The user obtains the identity of
> > > > >    the network card.
> > > > > 2. The user creates a VF with echo 8 > sriov_numvfs
> > > > > 3. The user binds the net crad to a VF with identity through the command
> > > > >    of the patch
> > > > >
> > > > > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > > ---
> > > > >  admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > > >  1 file changed, 40 insertions(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/admin.tex b/admin.tex
> > > > > index 2efd4d7..64d0667 100644
> > > > > --- a/admin.tex
> > > > > +++ b/admin.tex
> > > > > @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > > >  \hline \hline
> > > > >  0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> > > > >  0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> > > > > -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > > +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> > > > > +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> > > > >  \hline
> > > > >  0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> > > > >  \hline
> > > > > @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> > > > >  \field{VF Enable} refer to registers within the SR-IOV Extended
> > > > >  Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > > >
> > > > > +\subsubsection{Bind the device for member}
> > > > > +
> > > > > +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> > > > > +devices don't know about these VFs. Because each VF may be assigned a different role
> > > > > +by the user, the virtio device can not choose one VF to bind random.
> > > > > +So only the user knows how to bind the virtio devices to the VFs.
> > > > > +On the other hand, generally the virtio devices are not created by the user
> > > > > +inside the guest OS. This requires some management platform to participate.
> > > > > +
> > > > > +So we introduce a new admin queue command to bind the VFs and the virtio
> > > > > +devices.
> > > > > +
> > > > > +\begin{lstlisting}
> > > > > +struct virtio_admin_cmd_bind {
> > > > > +    u64 identity;
> > > > > +};
> > > > > +\end{lstlisting}
> > > > > +
> > > > > +The user got the \field{identity} from the management platform, that is not
> > > > > +included by this spec.
> > > > > +
> > > > > +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > > +
> > > > > +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> > > > > +
> > > > > +The \field{identity} is passed by the user. It is the identity of the virtio
> > > > > +device.
> > > > > +
> > > > > +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> > > > > +
> > > > > +Every device MUST have one unique \field{identity} in the host.
> > > > > +
> > > > > +If the PF device can not find the device by the \field{identity},
> > > > > +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> > > > > +
> > > > > +If the device is found by the \field{identity}, the device MUST work as the
> > > > > +device of this group member specified by the \field{group_member_id}.
> > > > > +
> > > > >  \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> > > > >
> > > > >  An administration virtqueue of an owner device is used to submit
> > > > > --
> > > > > 2.32.0.3.g01195cf9f
> > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
> > >
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, June 27, 2023 10:21 PM

> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > 	ADMIN_SETUP VF#
> > 	ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 

What does this command actually do or expected do on the device?

> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device not the
> transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only used to
> connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any PCI.
> What I have always wanted to say is that this device(not PCI) must have its own
> ID, which has nothing to do with the transport.
> 
A virtio device can have the id visible to self and visible to group owner device.

> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how
> do I tell our DPU to bind this migrated device with this vf?

When a virtio device state is set on a specific VF on the compute side (not on the dpu side),
This directly indicates to the dpu side, which virtio device is attached to which VF.

> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
>
 virtio device state setting itself will contain the device identifier.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] RE: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, June 27, 2023 10:21 PM

> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > 	ADMIN_SETUP VF#
> > 	ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 

What does this command actually do or expected do on the device?

> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device not the
> transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only used to
> connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any PCI.
> What I have always wanted to say is that this device(not PCI) must have its own
> ID, which has nothing to do with the transport.
> 
A virtio device can have the id visible to self and visible to group owner device.

> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how
> do I tell our DPU to bind this migrated device with this vf?

When a virtio device state is set on a specific VF on the compute side (not on the dpu side),
This directly indicates to the dpu side, which virtio device is attached to which VF.

> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
>
 virtio device state setting itself will contain the device identifier.

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > >     is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > 	ADMIN_SETUP VF#
> > 	ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 
> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device
> not the transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> to connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any
> PCI. What I have always wanted to say is that this device(not PCI) must have its
> own ID, which has nothing to do with the transport.
> 
> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> tell our DPU to bind this migrated device with this vf?
> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
> 
> Maybe we should compare it to the case of using PF directly before. The biggest
> difference here is that PF is directly hot-plugged by the DPU, so when a
> user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> on the host. Now the vf is created by the user, and only the user knows how to
> use each one. We cannot directly bind the device purchased or migrated by the
> user to a certain vf. We need the user in the host to bind the vf(transport) to
> a certain virtio device.
> 
> Thanks.
> 
> 

Maybe I didn't have enough coffee today but I can't figure out what all
this means :( For example what does "exist depending" mean?  What does
"device is migrated" mean?  What does it mean to be "migrated from the
same host"? What is "any PCI"? E.g. I know people did vm migration
moving virtio from a hardware device to a software
implementation. Is that "not associated with any PCI" ?
What is "user(custom)"? how is "the vf is created by the user"?
what does it mean to "bind the device .. to a certain vf"?
It looks like Parav understand what's going on though, so
maybe it's my fault.

But fundamentally, the problem is that the spec patch doesn't do
anything useful by itself, it relies on some out of spec interface to
make these id values make sense. So the TC is reduced to guessing: we
could just trust you that it's useful somehow and at the same time
serves the purpose whatever it is.

But it would be better if instead of trying to explain what that does in
vague terms, you post a spec patch that allows doing whatever needs
doing for these IDs to make sense through e.g. admin commands.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > So, this is how I understand the process of creating vf:
> > >
> > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > >     is only a default backend. It does not fully meet our expectations.
> > > 2. Create device or migrate device
> > > 3. Bind the backend virtio device to the vf
> >
> > I can see this making sense as a feature bit that says VFs are not
> > initialized by default and must first be setup through an admin command.
> > This will likely need to be a feature bit because it's changing
> > behaviour outside of admin commands.
> >
> > Then, we can have:
> >
> > 	ADMIN_SETUP VF#
> > 	ADMIN_CLEANUP VF#
> >
> > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> 
> Great!!
> 
> >
> >
> > Why do we need an id as a level of indirection though? What is wrong
> > with just using VF# directly?
> 
> I think VF# is ok. I also need to use it. But we need an ID for virtio device
> not the transport(PF, VF).
> 
> What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> to connect the virtio driver and the virtio device. Right?
> 
> The virtio device does not necessarily exist depending on PCI. For example, a
> virtio device is migrated from another DPU, and it is not associated with any
> PCI. What I have always wanted to say is that this device(not PCI) must have its
> own ID, which has nothing to do with the transport.
> 
> Now we want to use this migrated device and connect it to the corresponding
> vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> tell our DPU to bind this migrated device with this vf?
> We can specify the VF by the VF#, but how can we specify the virtio device?
> Maybe there are two migrated virtio device.
> 
> Maybe we should compare it to the case of using PF directly before. The biggest
> difference here is that PF is directly hot-plugged by the DPU, so when a
> user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> on the host. Now the vf is created by the user, and only the user knows how to
> use each one. We cannot directly bind the device purchased or migrated by the
> user to a certain vf. We need the user in the host to bind the vf(transport) to
> a certain virtio device.
> 
> Thanks.
> 
> 

Maybe I didn't have enough coffee today but I can't figure out what all
this means :( For example what does "exist depending" mean?  What does
"device is migrated" mean?  What does it mean to be "migrated from the
same host"? What is "any PCI"? E.g. I know people did vm migration
moving virtio from a hardware device to a software
implementation. Is that "not associated with any PCI" ?
What is "user(custom)"? how is "the vf is created by the user"?
what does it mean to "bind the device .. to a certain vf"?
It looks like Parav understand what's going on though, so
maybe it's my fault.

But fundamentally, the problem is that the spec patch doesn't do
anything useful by itself, it relies on some out of spec interface to
make these id values make sense. So the TC is reduced to guessing: we
could just trust you that it's useful somehow and at the same time
serves the purpose whatever it is.

But it would be better if instead of trying to explain what that does in
vague terms, you post a spec patch that allows doing whatever needs
doing for these IDs to make sense through e.g. admin commands.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >
> > > > >
> > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > >
> > > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > > understand.
> > > > >
> > > > > First, the user (customer) purchased a bare metal machine.
> > > > >
> > > > > ## Bare metal machine
> > > > >
> > > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > > general physical machine is that its PCI is connected to a device similar to a
> > > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > > These devices are managed by the vendor, and must be created and purchased
> > > > > on the vendor's management platform.
> > > > >
> > > > > ## DPU
> > > > >
> > > > > There is a software implementation in the DPU, which will respond to PCI
> > > > > operations. But as mentioned above, resources such as network cards must be
> > > > > purchased and created before they can exist. So users can create VF, which is
> > > > > just a pci-level operation, but there may not be a corresponding backend.
> > > > >
> > > > > ## Management Platform
> > > > >
> > > > > The creation and configuration of devices is realized on the management
> > > > > platform.
> > > > >
> > > > > After the user completed the purchase on the management platform (this is an
> > > > > independent platform provided by the vendor and has nothing to do with
> > > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > > This includes some user configurations, available bandwidth resources and other
> > > > > information.
> > > > >
> > > > > ## Usage
> > > > >
> > > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > > use a separate virtio-net device for performance.
> > > > >
> > > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > > devices. This number reaches 1k+.
> > > > >
> > > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > > the user creates a vf.
> > > > >
> > > > > ## Migration
> > > > >
> > > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > > from another host, of course its corresponding virtio device is also migrated to
> > > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > > device.
> > > > >
> > > > > ## Abstraction
> > > > >
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > 3) should come before 2)?
> > > >
> > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > does the user know its id?
> > >
> > > Got the id from the management platform.
> >
> > So it can do the binding via that management platform which this
> > became a cloud vendor specific interface.
> 
> In our scenario, this is bound by the user using this id and vf id in the os.
> 
> >
> > >
> > > >
> > > > >
> > > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > > queue to modify its msix vector and other configurations.
> > > > >
> > > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > > useful for live migration and virtio devices with special configurations.
> > > >
> > > > All of these could be addressed if a dynamic provisioning model is
> > > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > > in SR-IOV might be tricky.
> > >
> > >
> > > SR-IOV vf is native PCI device, this is the advancement.
> >
> > The problem is that it doesn't support flexible provisioning, e.g
> > create and destroy a single VF.
> 
> YES. ^_^!!

So sure, create it. Once you have created it, you can
use the VF# to talk to it.


I *suspect* that what this ID does is replace provisioning commands.

So instead of saying "create VF#3 with MAC 0xABC and 0x1000VQs"
you would have management say "ID 0xFACE refers to MAC ABC and 1000VQs"
and later you will say "bind VF#3 to ID 0xFACE" and that will
set it up.

Is that it?

But why is it important to do it in two steps like this?
as opposed to in one step?  I have no idea.

> 
> >
> > >
> > >
> > > >
> > > > >
> > > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > > transport layer by nature. This is our advantage.
> > > >
> > > > Is it not switching the transport layer but about binding/unbinding
> > > > vitio devices to VF?
> > >
> > > YES.
> > >
> > > >
> > > > Is a new capability or similar admin cmd sufficient in this case?
> > >
> > > All is ok.
> > >
> > >
> > > >
> > > > struct virtio_pci_bind_cap {
> > > >         struct virtio_pci_cap cap;
> > > >         u16 bind; // virtio_device_id
> > > >         u16 unbind; // virtio_device_id
> > > > };
> > >
> > > You mean that the "bind" or "unbind" is writeable?
> 
> This is a good idea.
> 
> Thanks.

So stealing valuable memory from limited pci config space, no error
handling, no filtering... Ugh.  Let's not put a round peg in a square
hole.

For management I think we should use admin commands. They were built for
the management use-case.
Config space (pci and virtio) is better for driver slow path.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Michael S. Tsirkin]

On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >
> > > > >
> > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > >
> > > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > > understand.
> > > > >
> > > > > First, the user (customer) purchased a bare metal machine.
> > > > >
> > > > > ## Bare metal machine
> > > > >
> > > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > > general physical machine is that its PCI is connected to a device similar to a
> > > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > > These devices are managed by the vendor, and must be created and purchased
> > > > > on the vendor's management platform.
> > > > >
> > > > > ## DPU
> > > > >
> > > > > There is a software implementation in the DPU, which will respond to PCI
> > > > > operations. But as mentioned above, resources such as network cards must be
> > > > > purchased and created before they can exist. So users can create VF, which is
> > > > > just a pci-level operation, but there may not be a corresponding backend.
> > > > >
> > > > > ## Management Platform
> > > > >
> > > > > The creation and configuration of devices is realized on the management
> > > > > platform.
> > > > >
> > > > > After the user completed the purchase on the management platform (this is an
> > > > > independent platform provided by the vendor and has nothing to do with
> > > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > > This includes some user configurations, available bandwidth resources and other
> > > > > information.
> > > > >
> > > > > ## Usage
> > > > >
> > > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > > use a separate virtio-net device for performance.
> > > > >
> > > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > > devices. This number reaches 1k+.
> > > > >
> > > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > > the user creates a vf.
> > > > >
> > > > > ## Migration
> > > > >
> > > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > > from another host, of course its corresponding virtio device is also migrated to
> > > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > > device.
> > > > >
> > > > > ## Abstraction
> > > > >
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > 3) should come before 2)?
> > > >
> > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > does the user know its id?
> > >
> > > Got the id from the management platform.
> >
> > So it can do the binding via that management platform which this
> > became a cloud vendor specific interface.
> 
> In our scenario, this is bound by the user using this id and vf id in the os.
> 
> >
> > >
> > > >
> > > > >
> > > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > > queue to modify its msix vector and other configurations.
> > > > >
> > > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > > useful for live migration and virtio devices with special configurations.
> > > >
> > > > All of these could be addressed if a dynamic provisioning model is
> > > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > > in SR-IOV might be tricky.
> > >
> > >
> > > SR-IOV vf is native PCI device, this is the advancement.
> >
> > The problem is that it doesn't support flexible provisioning, e.g
> > create and destroy a single VF.
> 
> YES. ^_^!!

So sure, create it. Once you have created it, you can
use the VF# to talk to it.


I *suspect* that what this ID does is replace provisioning commands.

So instead of saying "create VF#3 with MAC 0xABC and 0x1000VQs"
you would have management say "ID 0xFACE refers to MAC ABC and 1000VQs"
and later you will say "bind VF#3 to ID 0xFACE" and that will
set it up.

Is that it?

But why is it important to do it in two steps like this?
as opposed to in one step?  I have no idea.

> 
> >
> > >
> > >
> > > >
> > > > >
> > > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > > transport layer by nature. This is our advantage.
> > > >
> > > > Is it not switching the transport layer but about binding/unbinding
> > > > vitio devices to VF?
> > >
> > > YES.
> > >
> > > >
> > > > Is a new capability or similar admin cmd sufficient in this case?
> > >
> > > All is ok.
> > >
> > >
> > > >
> > > > struct virtio_pci_bind_cap {
> > > >         struct virtio_pci_cap cap;
> > > >         u16 bind; // virtio_device_id
> > > >         u16 unbind; // virtio_device_id
> > > > };
> > >
> > > You mean that the "bind" or "unbind" is writeable?
> 
> This is a good idea.
> 
> Thanks.

So stealing valuable memory from limited pci config space, no error
handling, no filtering... Ugh.  Let's not put a round peg in a square
hole.

For management I think we should use admin commands. They were built for
the management use-case.
Config space (pci and virtio) is better for driver slow path.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Wed, Jun 28, 2023 at 11:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> > On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > >
> > > > > >
> > > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > > >
> > > > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > > > understand.
> > > > > >
> > > > > > First, the user (customer) purchased a bare metal machine.
> > > > > >
> > > > > > ## Bare metal machine
> > > > > >
> > > > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > > > general physical machine is that its PCI is connected to a device similar to a
> > > > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > > > These devices are managed by the vendor, and must be created and purchased
> > > > > > on the vendor's management platform.
> > > > > >
> > > > > > ## DPU
> > > > > >
> > > > > > There is a software implementation in the DPU, which will respond to PCI
> > > > > > operations. But as mentioned above, resources such as network cards must be
> > > > > > purchased and created before they can exist. So users can create VF, which is
> > > > > > just a pci-level operation, but there may not be a corresponding backend.
> > > > > >
> > > > > > ## Management Platform
> > > > > >
> > > > > > The creation and configuration of devices is realized on the management
> > > > > > platform.
> > > > > >
> > > > > > After the user completed the purchase on the management platform (this is an
> > > > > > independent platform provided by the vendor and has nothing to do with
> > > > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > > > This includes some user configurations, available bandwidth resources and other
> > > > > > information.
> > > > > >
> > > > > > ## Usage
> > > > > >
> > > > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > > > use a separate virtio-net device for performance.
> > > > > >
> > > > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > > > devices. This number reaches 1k+.
> > > > > >
> > > > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > > > the user creates a vf.
> > > > > >
> > > > > > ## Migration
> > > > > >
> > > > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > > > from another host, of course its corresponding virtio device is also migrated to
> > > > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > > > device.
> > > > > >
> > > > > > ## Abstraction
> > > > > >
> > > > > > So, this is how I understand the process of creating vf:
> > > > > >
> > > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > > 2. Create device or migrate device
> > > > > > 3. Bind the backend virtio device to the vf
> > > > >
> > > > > 3) should come before 2)?
> > > > >
> > > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > > does the user know its id?
> > > >
> > > > Got the id from the management platform.
> > >
> > > So it can do the binding via that management platform which this
> > > became a cloud vendor specific interface.
> >
> > In our scenario, this is bound by the user using this id and vf id in the os.
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > > > queue to modify its msix vector and other configurations.
> > > > > >
> > > > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > > > useful for live migration and virtio devices with special configurations.
> > > > >
> > > > > All of these could be addressed if a dynamic provisioning model is
> > > > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > > > in SR-IOV might be tricky.
> > > >
> > > >
> > > > SR-IOV vf is native PCI device, this is the advancement.
> > >
> > > The problem is that it doesn't support flexible provisioning, e.g
> > > create and destroy a single VF.
> >
> > YES. ^_^!!
>
> So sure, create it. Once you have created it, you can
> use the VF# to talk to it.
>
>
> I *suspect* that what this ID does is replace provisioning commands.
>
> So instead of saying "create VF#3 with MAC 0xABC and 0x1000VQs"
> you would have management say "ID 0xFACE refers to MAC ABC and 1000VQs"
> and later you will say "bind VF#3 to ID 0xFACE" and that will
> set it up.
>
> Is that it?
>
> But why is it important to do it in two steps like this?
> as opposed to in one step?  I have no idea.
>
> >
> > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > > > transport layer by nature. This is our advantage.
> > > > >
> > > > > Is it not switching the transport layer but about binding/unbinding
> > > > > vitio devices to VF?
> > > >
> > > > YES.
> > > >
> > > > >
> > > > > Is a new capability or similar admin cmd sufficient in this case?
> > > >
> > > > All is ok.
> > > >
> > > >
> > > > >
> > > > > struct virtio_pci_bind_cap {
> > > > >         struct virtio_pci_cap cap;
> > > > >         u16 bind; // virtio_device_id
> > > > >         u16 unbind; // virtio_device_id
> > > > > };
> > > >
> > > > You mean that the "bind" or "unbind" is writeable?
> >
> > This is a good idea.
> >
> > Thanks.
>
> So stealing valuable memory from limited pci config space, no error
> handling, no filtering... Ugh.  Let's not put a round peg in a square
> hole.

It's just quick example to demonstrate the idea, actually I meant it
could be done via admin cmd, see above

"""
Is a new capability or similar admin cmd sufficient in this case?
"""

Thanks

>
> For management I think we should use admin commands. They were built for
> the management use-case.
> Config space (pci and virtio) is better for driver slow path.
>
> --
> MST
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Wed, Jun 28, 2023 at 11:55 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 28, 2023 at 02:06:32PM +0800, Xuan Zhuo wrote:
> > On Wed, 28 Jun 2023 10:49:45 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 6:54 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Tue, 27 Jun 2023 17:00:06 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > > On Tue, Jun 27, 2023 at 4:28 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > >
> > > > > >
> > > > > > Thanks Parav for pointing it out. We may have some gaps on the case.
> > > > > >
> > > > > > Let me introduce our case, which I think it is simple and should be easy to
> > > > > > understand.
> > > > > >
> > > > > > First, the user (customer) purchased a bare metal machine.
> > > > > >
> > > > > > ## Bare metal machine
> > > > > >
> > > > > > Let me briefly explain the characteristics of a bare metal machine. It is not a
> > > > > > virtual machine, it is a physical machine, and the difference between it and a
> > > > > > general physical machine is that its PCI is connected to a device similar to a
> > > > > > DPU. This DPU provides devices such as virtio-blk/net to the host through PCI.
> > > > > > These devices are managed by the vendor, and must be created and purchased
> > > > > > on the vendor's management platform.
> > > > > >
> > > > > > ## DPU
> > > > > >
> > > > > > There is a software implementation in the DPU, which will respond to PCI
> > > > > > operations. But as mentioned above, resources such as network cards must be
> > > > > > purchased and created before they can exist. So users can create VF, which is
> > > > > > just a pci-level operation, but there may not be a corresponding backend.
> > > > > >
> > > > > > ## Management Platform
> > > > > >
> > > > > > The creation and configuration of devices is realized on the management
> > > > > > platform.
> > > > > >
> > > > > > After the user completed the purchase on the management platform (this is an
> > > > > > independent platform provided by the vendor and has nothing to do with
> > > > > > virtio), then there will be a corresponding device implementation in the DPU.
> > > > > > This includes some user configurations, available bandwidth resources and other
> > > > > > information.
> > > > > >
> > > > > > ## Usage
> > > > > >
> > > > > > Since the user is directly on the HOST, the user can create VMs, passthrough PF
> > > > > > or VF into the VM. Or users can create a large number of dockers, all of which
> > > > > > use a separate virtio-net device for performance.
> > > > > >
> > > > > > The reason why users use vf is that we need to use a large number of virtio-net
> > > > > > devices. This number reaches 1k+.
> > > > > >
> > > > > > Based on this scenario, we need to bind vf to the backend device. Because, we
> > > > > > cannot automatically complete the creation of the virtio-net backend device when
> > > > > > the user creates a vf.
> > > > > >
> > > > > > ## Migration
> > > > > >
> > > > > > In addition, let's consider another scenario of migration. If a vm is migrated
> > > > > > from another host, of course its corresponding virtio device is also migrated to
> > > > > > the DPU. At this time, our newly created vf can only be used by the vm after it
> > > > > > is bound to the migrated device. We do not want this vf to be a brand new
> > > > > > device.
> > > > > >
> > > > > > ## Abstraction
> > > > > >
> > > > > > So, this is how I understand the process of creating vf:
> > > > > >
> > > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > > 2. Create device or migrate device
> > > > > > 3. Bind the backend virtio device to the vf
> > > > >
> > > > > 3) should come before 2)?
> > > > >
> > > > > Who is going to do 3) btw, is it the user? If yes, for example, if a
> > > > > user wants another 4 queue virtio-net devices, after purchase, how
> > > > > does the user know its id?
> > > >
> > > > Got the id from the management platform.
> > >
> > > So it can do the binding via that management platform which this
> > > became a cloud vendor specific interface.
> >
> > In our scenario, this is bound by the user using this id and vf id in the os.
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > In most scenarios, the first step may be enough. We can make some fine-tuning on
> > > > > > this default device, such as modifying its mac. In the future, we can use admin
> > > > > > queue to modify its msix vector and other configurations.
> > > > > >
> > > > > > But we should allow, we bind a backend virtio device to a certain vf. This is
> > > > > > useful for live migration and virtio devices with special configurations.
> > > > >
> > > > > All of these could be addressed if a dynamic provisioning model is
> > > > > implemented (SIOV or transport virtqueue). Trying to have a workaround
> > > > > in SR-IOV might be tricky.
> > > >
> > > >
> > > > SR-IOV vf is native PCI device, this is the advancement.
> > >
> > > The problem is that it doesn't support flexible provisioning, e.g
> > > create and destroy a single VF.
> >
> > YES. ^_^!!
>
> So sure, create it. Once you have created it, you can
> use the VF# to talk to it.
>
>
> I *suspect* that what this ID does is replace provisioning commands.
>
> So instead of saying "create VF#3 with MAC 0xABC and 0x1000VQs"
> you would have management say "ID 0xFACE refers to MAC ABC and 1000VQs"
> and later you will say "bind VF#3 to ID 0xFACE" and that will
> set it up.
>
> Is that it?
>
> But why is it important to do it in two steps like this?
> as opposed to in one step?  I have no idea.
>
> >
> > >
> > > >
> > > >
> > > > >
> > > > > >
> > > > > > The design of virtio itself is two layers, and virtio should allow switching the
> > > > > > transport layer by nature. This is our advantage.
> > > > >
> > > > > Is it not switching the transport layer but about binding/unbinding
> > > > > vitio devices to VF?
> > > >
> > > > YES.
> > > >
> > > > >
> > > > > Is a new capability or similar admin cmd sufficient in this case?
> > > >
> > > > All is ok.
> > > >
> > > >
> > > > >
> > > > > struct virtio_pci_bind_cap {
> > > > >         struct virtio_pci_cap cap;
> > > > >         u16 bind; // virtio_device_id
> > > > >         u16 unbind; // virtio_device_id
> > > > > };
> > > >
> > > > You mean that the "bind" or "unbind" is writeable?
> >
> > This is a good idea.
> >
> > Thanks.
>
> So stealing valuable memory from limited pci config space, no error
> handling, no filtering... Ugh.  Let's not put a round peg in a square
> hole.

It's just quick example to demonstrate the idea, actually I meant it
could be done via admin cmd, see above

"""
Is a new capability or similar admin cmd sufficient in this case?
"""

Thanks

>
> For management I think we should use admin commands. They were built for
> the management use-case.
> Config space (pci and virtio) is better for driver slow path.
>
> --
> MST
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > So, this is how I understand the process of creating vf:
> > > >
> > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > >     is only a default backend. It does not fully meet our expectations.
> > > > 2. Create device or migrate device
> > > > 3. Bind the backend virtio device to the vf
> > >
> > > I can see this making sense as a feature bit that says VFs are not
> > > initialized by default and must first be setup through an admin command.
> > > This will likely need to be a feature bit because it's changing
> > > behaviour outside of admin commands.
> > >
> > > Then, we can have:
> > >
> > > 	ADMIN_SETUP VF#
> > > 	ADMIN_CLEANUP VF#
> > >
> > > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> >
> > Great!!
> >
> > >
> > >
> > > Why do we need an id as a level of indirection though? What is wrong
> > > with just using VF# directly?
> >
> > I think VF# is ok. I also need to use it. But we need an ID for virtio device
> > not the transport(PF, VF).
> >
> > What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> > to connect the virtio driver and the virtio device. Right?
> >
> > The virtio device does not necessarily exist depending on PCI. For example, a
> > virtio device is migrated from another DPU, and it is not associated with any
> > PCI. What I have always wanted to say is that this device(not PCI) must have its
> > own ID, which has nothing to do with the transport.
> >
> > Now we want to use this migrated device and connect it to the corresponding
> > vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> > tell our DPU to bind this migrated device with this vf?
> > We can specify the VF by the VF#, but how can we specify the virtio device?
> > Maybe there are two migrated virtio device.
> >
> > Maybe we should compare it to the case of using PF directly before. The biggest
> > difference here is that PF is directly hot-plugged by the DPU, so when a
> > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> > on the host. Now the vf is created by the user, and only the user knows how to
> > use each one. We cannot directly bind the device purchased or migrated by the
> > user to a certain vf. We need the user in the host to bind the vf(transport) to
> > a certain virtio device.
> >
> > Thanks.
> >
> >
>
> Maybe I didn't have enough coffee today but I can't figure out what all
> this means :( For example what does "exist depending" mean?  What does
> "device is migrated" mean?  What does it mean to be "migrated from the
> same host"? What is "any PCI"? E.g. I know people did vm migration
> moving virtio from a hardware device to a software
> implementation. Is that "not associated with any PCI" ?
> What is "user(custom)"? how is "the vf is created by the user"?
> what does it mean to "bind the device .. to a certain vf"?
> It looks like Parav understand what's going on though, so
> maybe it's my fault.
>
> But fundamentally, the problem is that the spec patch doesn't do
> anything useful by itself, it relies on some out of spec interface to
> make these id values make sense. So the TC is reduced to guessing: we
> could just trust you that it's useful somehow and at the same time
> serves the purpose whatever it is.
>
> But it would be better if instead of trying to explain what that does in
> vague terms, you post a spec patch that allows doing whatever needs
> doing for these IDs to make sense through e.g. admin commands.

I try to describe the requirement as a common requirement.

@Parav do you think this is a requirement in your devices?

The core point here is that our virtio-net device is created by customers on the
management platform. It is not a pure virtio-net device, but also includes many
other features, such as bandwidth, such as VPC information, etc. These are not
within the scope of virtio, we cannot configure these information based on the
admin queue after creating the vf. But we can bind the device created by the
user to vf, which completes the binding of many manufacturer features to vf.

In addition to this scenario, in the case of live migration, we can migrate a vm
from one host to another host. During this process, the virtio-net devce is also
migrated from the dpu of one host to the dpu of another host . At this time, the
purpose of creating a vf is to insert the migrated virtio-net into the host. It
is impossible for us to create a vf to get a brand new device. So also bind.

The core problem of this patch is that we need to assign an id to each device.
Is this reasonable? I think this is reasonable, each device can have an id.

Virtio device and transport can be unbound and re-bound.

Thanks.


>
> --
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo]

On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > So, this is how I understand the process of creating vf:
> > > >
> > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > >     is only a default backend. It does not fully meet our expectations.
> > > > 2. Create device or migrate device
> > > > 3. Bind the backend virtio device to the vf
> > >
> > > I can see this making sense as a feature bit that says VFs are not
> > > initialized by default and must first be setup through an admin command.
> > > This will likely need to be a feature bit because it's changing
> > > behaviour outside of admin commands.
> > >
> > > Then, we can have:
> > >
> > > 	ADMIN_SETUP VF#
> > > 	ADMIN_CLEANUP VF#
> > >
> > > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> >
> > Great!!
> >
> > >
> > >
> > > Why do we need an id as a level of indirection though? What is wrong
> > > with just using VF# directly?
> >
> > I think VF# is ok. I also need to use it. But we need an ID for virtio device
> > not the transport(PF, VF).
> >
> > What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> > to connect the virtio driver and the virtio device. Right?
> >
> > The virtio device does not necessarily exist depending on PCI. For example, a
> > virtio device is migrated from another DPU, and it is not associated with any
> > PCI. What I have always wanted to say is that this device(not PCI) must have its
> > own ID, which has nothing to do with the transport.
> >
> > Now we want to use this migrated device and connect it to the corresponding
> > vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> > tell our DPU to bind this migrated device with this vf?
> > We can specify the VF by the VF#, but how can we specify the virtio device?
> > Maybe there are two migrated virtio device.
> >
> > Maybe we should compare it to the case of using PF directly before. The biggest
> > difference here is that PF is directly hot-plugged by the DPU, so when a
> > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> > on the host. Now the vf is created by the user, and only the user knows how to
> > use each one. We cannot directly bind the device purchased or migrated by the
> > user to a certain vf. We need the user in the host to bind the vf(transport) to
> > a certain virtio device.
> >
> > Thanks.
> >
> >
>
> Maybe I didn't have enough coffee today but I can't figure out what all
> this means :( For example what does "exist depending" mean?  What does
> "device is migrated" mean?  What does it mean to be "migrated from the
> same host"? What is "any PCI"? E.g. I know people did vm migration
> moving virtio from a hardware device to a software
> implementation. Is that "not associated with any PCI" ?
> What is "user(custom)"? how is "the vf is created by the user"?
> what does it mean to "bind the device .. to a certain vf"?
> It looks like Parav understand what's going on though, so
> maybe it's my fault.
>
> But fundamentally, the problem is that the spec patch doesn't do
> anything useful by itself, it relies on some out of spec interface to
> make these id values make sense. So the TC is reduced to guessing: we
> could just trust you that it's useful somehow and at the same time
> serves the purpose whatever it is.
>
> But it would be better if instead of trying to explain what that does in
> vague terms, you post a spec patch that allows doing whatever needs
> doing for these IDs to make sense through e.g. admin commands.

I try to describe the requirement as a common requirement.

@Parav do you think this is a requirement in your devices?

The core point here is that our virtio-net device is created by customers on the
management platform. It is not a pure virtio-net device, but also includes many
other features, such as bandwidth, such as VPC information, etc. These are not
within the scope of virtio, we cannot configure these information based on the
admin queue after creating the vf. But we can bind the device created by the
user to vf, which completes the binding of many manufacturer features to vf.

In addition to this scenario, in the case of live migration, we can migrate a vm
from one host to another host. During this process, the virtio-net devce is also
migrated from the dpu of one host to the dpu of another host . At this time, the
purpose of creating a vf is to insert the migrated virtio-net into the host. It
is impossible for us to create a vf to get a brand new device. So also bind.

The core problem of this patch is that we need to assign an id to each device.
Is this reasonable? I think this is reasonable, each device can have an id.

Virtio device and transport can be unbound and re-bound.

Thanks.


>
> --
> MST
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jul 3, 2023 at 11:53 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > I can see this making sense as a feature bit that says VFs are not
> > > > initialized by default and must first be setup through an admin command.
> > > > This will likely need to be a feature bit because it's changing
> > > > behaviour outside of admin commands.
> > > >
> > > > Then, we can have:
> > > >
> > > >   ADMIN_SETUP VF#
> > > >   ADMIN_CLEANUP VF#
> > > >
> > > > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> > >
> > > Great!!
> > >
> > > >
> > > >
> > > > Why do we need an id as a level of indirection though? What is wrong
> > > > with just using VF# directly?
> > >
> > > I think VF# is ok. I also need to use it. But we need an ID for virtio device
> > > not the transport(PF, VF).
> > >
> > > What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> > > to connect the virtio driver and the virtio device. Right?
> > >
> > > The virtio device does not necessarily exist depending on PCI. For example, a
> > > virtio device is migrated from another DPU, and it is not associated with any
> > > PCI. What I have always wanted to say is that this device(not PCI) must have its
> > > own ID, which has nothing to do with the transport.
> > >
> > > Now we want to use this migrated device and connect it to the corresponding
> > > vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> > > tell our DPU to bind this migrated device with this vf?
> > > We can specify the VF by the VF#, but how can we specify the virtio device?
> > > Maybe there are two migrated virtio device.
> > >
> > > Maybe we should compare it to the case of using PF directly before. The biggest
> > > difference here is that PF is directly hot-plugged by the DPU, so when a
> > > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> > > on the host. Now the vf is created by the user, and only the user knows how to
> > > use each one. We cannot directly bind the device purchased or migrated by the
> > > user to a certain vf. We need the user in the host to bind the vf(transport) to
> > > a certain virtio device.
> > >
> > > Thanks.
> > >
> > >
> >
> > Maybe I didn't have enough coffee today but I can't figure out what all
> > this means :( For example what does "exist depending" mean?  What does
> > "device is migrated" mean?  What does it mean to be "migrated from the
> > same host"? What is "any PCI"? E.g. I know people did vm migration
> > moving virtio from a hardware device to a software
> > implementation. Is that "not associated with any PCI" ?
> > What is "user(custom)"? how is "the vf is created by the user"?
> > what does it mean to "bind the device .. to a certain vf"?
> > It looks like Parav understand what's going on though, so
> > maybe it's my fault.
> >
> > But fundamentally, the problem is that the spec patch doesn't do
> > anything useful by itself, it relies on some out of spec interface to
> > make these id values make sense. So the TC is reduced to guessing: we
> > could just trust you that it's useful somehow and at the same time
> > serves the purpose whatever it is.
> >
> > But it would be better if instead of trying to explain what that does in
> > vague terms, you post a spec patch that allows doing whatever needs
> > doing for these IDs to make sense through e.g. admin commands.
>
> I try to describe the requirement as a common requirement.
>
> @Parav do you think this is a requirement in your devices?
>
> The core point here is that our virtio-net device is created by customers on the
> management platform. It is not a pure virtio-net device, but also includes many
> other features, such as bandwidth, such as VPC information, etc. These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.

It sounds like your device is vDPA. Anyhow, it looks like you need
provision vf and the virtio via vendor specific way. Then we don't
need to bother it in the spec.

>
> In addition to this scenario, in the case of live migration, we can migrate a vm
> from one host to another host. During this process, the virtio-net devce is also
> migrated from the dpu of one host to the dpu of another host . At this time, the
> purpose of creating a vf is to insert the migrated virtio-net into the host. It
> is impossible for us to create a vf to get a brand new device. So also bind.

If there's are states beyond the scope of virtio, trying to workaround
it in the spec seems sub-optimal.

>
> The core problem of this patch is that we need to assign an id to each device.
> Is this reasonable? I think this is reasonable, each device can have an id.

Id seems to be fine but why bind is a must for the user? That looks
like two issues.

>
> Virtio device and transport can be unbound and re-bound.

This seems not to be supported by the spec at least now.

Thanks

>
> Thanks.
>
>
> >
> > --
> > MST
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Jason Wang]

On Mon, Jul 3, 2023 at 11:53 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Wed, 28 Jun 2023 11:41:02 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Wed, Jun 28, 2023 at 10:21:04AM +0800, Xuan Zhuo wrote:
> > > On Tue, 27 Jun 2023 12:02:41 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > On Tue, Jun 27, 2023 at 04:23:05PM +0800, Xuan Zhuo wrote:
> > > > > So, this is how I understand the process of creating vf:
> > > > >
> > > > > 1. Create a PCI VF, at this time there may be no backend virtio device, or there
> > > > >     is only a default backend. It does not fully meet our expectations.
> > > > > 2. Create device or migrate device
> > > > > 3. Bind the backend virtio device to the vf
> > > >
> > > > I can see this making sense as a feature bit that says VFs are not
> > > > initialized by default and must first be setup through an admin command.
> > > > This will likely need to be a feature bit because it's changing
> > > > behaviour outside of admin commands.
> > > >
> > > > Then, we can have:
> > > >
> > > >   ADMIN_SETUP VF#
> > > >   ADMIN_CLEANUP VF#
> > > >
> > > > I like this because this generalizes CREATE/DESTROY that SIOV guys proposed.
> > >
> > > Great!!
> > >
> > > >
> > > >
> > > > Why do we need an id as a level of indirection though? What is wrong
> > > > with just using VF# directly?
> > >
> > > I think VF# is ok. I also need to use it. But we need an ID for virtio device
> > > not the transport(PF, VF).
> > >
> > > What I want to emphasize is that PCI(pf or vf) is a transport, it is only used
> > > to connect the virtio driver and the virtio device. Right?
> > >
> > > The virtio device does not necessarily exist depending on PCI. For example, a
> > > virtio device is migrated from another DPU, and it is not associated with any
> > > PCI. What I have always wanted to say is that this device(not PCI) must have its
> > > own ID, which has nothing to do with the transport.
> > >
> > > Now we want to use this migrated device and connect it to the corresponding
> > > vm (migrated from the same host). We can passthrough vf to this vm. But how do I
> > > tell our DPU to bind this migrated device with this vf?
> > > We can specify the VF by the VF#, but how can we specify the virtio device?
> > > Maybe there are two migrated virtio device.
> > >
> > > Maybe we should compare it to the case of using PF directly before. The biggest
> > > difference here is that PF is directly hot-plugged by the DPU, so when a
> > > user(custom) buys a virtio-net device, the DPU directly inserts a new PCI device
> > > on the host. Now the vf is created by the user, and only the user knows how to
> > > use each one. We cannot directly bind the device purchased or migrated by the
> > > user to a certain vf. We need the user in the host to bind the vf(transport) to
> > > a certain virtio device.
> > >
> > > Thanks.
> > >
> > >
> >
> > Maybe I didn't have enough coffee today but I can't figure out what all
> > this means :( For example what does "exist depending" mean?  What does
> > "device is migrated" mean?  What does it mean to be "migrated from the
> > same host"? What is "any PCI"? E.g. I know people did vm migration
> > moving virtio from a hardware device to a software
> > implementation. Is that "not associated with any PCI" ?
> > What is "user(custom)"? how is "the vf is created by the user"?
> > what does it mean to "bind the device .. to a certain vf"?
> > It looks like Parav understand what's going on though, so
> > maybe it's my fault.
> >
> > But fundamentally, the problem is that the spec patch doesn't do
> > anything useful by itself, it relies on some out of spec interface to
> > make these id values make sense. So the TC is reduced to guessing: we
> > could just trust you that it's useful somehow and at the same time
> > serves the purpose whatever it is.
> >
> > But it would be better if instead of trying to explain what that does in
> > vague terms, you post a spec patch that allows doing whatever needs
> > doing for these IDs to make sense through e.g. admin commands.
>
> I try to describe the requirement as a common requirement.
>
> @Parav do you think this is a requirement in your devices?
>
> The core point here is that our virtio-net device is created by customers on the
> management platform. It is not a pure virtio-net device, but also includes many
> other features, such as bandwidth, such as VPC information, etc. These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.

It sounds like your device is vDPA. Anyhow, it looks like you need
provision vf and the virtio via vendor specific way. Then we don't
need to bother it in the spec.

>
> In addition to this scenario, in the case of live migration, we can migrate a vm
> from one host to another host. During this process, the virtio-net devce is also
> migrated from the dpu of one host to the dpu of another host . At this time, the
> purpose of creating a vf is to insert the migrated virtio-net into the host. It
> is impossible for us to create a vf to get a brand new device. So also bind.

If there's are states beyond the scope of virtio, trying to workaround
it in the spec seems sub-optimal.

>
> The core problem of this patch is that we need to assign an id to each device.
> Is this reasonable? I think this is reasonable, each device can have an id.

Id seems to be fine but why bind is a must for the user? That looks
like two issues.

>
> Virtio device and transport can be unbound and re-bound.

This seems not to be supported by the spec at least now.

Thanks

>
> Thanks.
>
>
> >
> > --
> > MST
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

Hi Xuan,

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Sunday, July 2, 2023 11:22 PM


> I try to describe the requirement as a common requirement.
> 
> @Parav do you think this is a requirement in your devices?
> 
> The core point here is that our virtio-net device is created by customers on the
> management platform. It is not a pure virtio-net device, but also includes many
> other features, such as bandwidth, such as VPC information, etc.
When these devices are on PCI transport, and if they are PF.
PCI VPD has well defined serial number, and few more fields.
This helps to cross identify the device between compute side where virtio device is used and other management side, who inserted this virtio PCI device in the compute.


>  These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.
> 
> In addition to this scenario, in the case of live migration, we can migrate a vm
> from one host to another host. During this process, the virtio-net devce is also
> migrated from the dpu of one host to the dpu of another host . At this time, the
> purpose of creating a vf is to insert the migrated virtio-net into the host. It is
> impossible for us to create a vf to get a brand new device. So also bind.
>
For the VF in this flow looks like below.

a. device id assignment is initiated by the compute side AQ to a VF (on destination)
b. management system received the command and queried some central place to get attributed (qos etc), that you described.

Here somehow the src side already had access to the management system in first place for the VF backend configuration.
And I believe same access is/should be possible on the destination side too.
So that same parameter can be configured as out of band command.

In fact for virtio device migration, a migrating sw may even need to inspect and figure out which is the best system to migrate to in the backend.

So out of band access is present and needed that can rely on the VF number to provision these things without explicit bind command.

So bind command is optional command that can help to do the work that is already done today out of band.

And if bind on specific system is identified, it is probably better to do this out of band.

Your key point is " virtio-net device is created by customers on the management platform".
If they are based on the VF, it can be identified uniquely by the VF number without explicit bind command.

For PF and VF I responded few other ideas before at [1], you probably missed it.

[1] https://lore.kernel.org/virtio-comment/20230628112107-mutt-send-email-mst@kernel.org/T/#m99518f22871a223c4dd014bb77e712b3befd9793

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-comment] RE: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Parav Pandit]

Hi Xuan,

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Sunday, July 2, 2023 11:22 PM


> I try to describe the requirement as a common requirement.
> 
> @Parav do you think this is a requirement in your devices?
> 
> The core point here is that our virtio-net device is created by customers on the
> management platform. It is not a pure virtio-net device, but also includes many
> other features, such as bandwidth, such as VPC information, etc.
When these devices are on PCI transport, and if they are PF.
PCI VPD has well defined serial number, and few more fields.
This helps to cross identify the device between compute side where virtio device is used and other management side, who inserted this virtio PCI device in the compute.


>  These are not
> within the scope of virtio, we cannot configure these information based on the
> admin queue after creating the vf. But we can bind the device created by the
> user to vf, which completes the binding of many manufacturer features to vf.
> 
> In addition to this scenario, in the case of live migration, we can migrate a vm
> from one host to another host. During this process, the virtio-net devce is also
> migrated from the dpu of one host to the dpu of another host . At this time, the
> purpose of creating a vf is to insert the migrated virtio-net into the host. It is
> impossible for us to create a vf to get a brand new device. So also bind.
>
For the VF in this flow looks like below.

a. device id assignment is initiated by the compute side AQ to a VF (on destination)
b. management system received the command and queried some central place to get attributed (qos etc), that you described.

Here somehow the src side already had access to the management system in first place for the VF backend configuration.
And I believe same access is/should be possible on the destination side too.
So that same parameter can be configured as out of band command.

In fact for virtio device migration, a migrating sw may even need to inspect and figure out which is the best system to migrate to in the backend.

So out of band access is present and needed that can rely on the VF number to provision these things without explicit bind command.

So bind command is optional command that can help to do the work that is already done today out of band.

And if bind on specific system is identified, it is probably better to do this out of band.

Your key point is " virtio-net device is created by customers on the management platform".
If they are based on the VF, it can be identified uniquely by the VF number without explicit bind command.

For PF and VF I responded few other ideas before at [1], you probably missed it.

[1] https://lore.kernel.org/virtio-comment/20230628112107-mutt-send-email-mst@kernel.org/T/#m99518f22871a223c4dd014bb77e712b3befd9793

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/