[virtio-comment] [PATCH 0/2] Selective queue enabling

[virtio-comment] [PATCH 0/2] Selective queue enabling

[Eugenio Pérez]

This series allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Eugenio Pérez (2):
  virtio: introduce selective queue enabling
  virtio: pci support virtqueue selective enabling

 content.tex       | 15 +++++++++++++--
 transport-pci.tex |  4 ++++
 2 files changed, 17 insertions(+), 2 deletions(-)

-- 
2.31.1



This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] [PATCH 1/2] virtio: introduce selective queue enabling

[Eugenio Pérez]

This patch allows the driver to start the device (as set DRIVER_OK) with only
some queues enabled, and then enable another queues later.

This is the current way to migrate net device state through control
virtqueue, in a software assisted framework with vDPA:
* First, only net CVQ is enabled at DRIVER_OK
* All the control commands (mac address, mq, etc) needed for the device
to behave the same as the source of migration are sent
* Finally all the dataplane queues are enabled.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 content.tex | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index d2ab9eb..ea4a6d8 100644
--- a/content.tex
+++ b/content.tex
@@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
 \begin{description}
 \item[0 to 23, and 50 to 127] Feature bits for the specific device type
 
-\item[24 to 41] Feature bits reserved for extensions to the queue and
+\item[24 to 42] Feature bits reserved for extensions to the queue and
   feature negotiation mechanisms
 
-\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
+\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
 \end{description}
 
 \begin{note}
@@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
 Every driver and device supports either the Packed or the Split
 Virtqueue format, or both.
 
+\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+
+If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can enable
+individual virtqueues both before and after the DRIVER\_OK. The way to enable a
+virtqueue is transport specific, but it mimics the re-enabling after an
+hypothetical reset.
+
 \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}
 
 When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
@@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
 	handling features reserved for future use.
 
+  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates
+  that the driver can enable a queue both before and after DRIVER\_OK.
+  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] [PATCH 2/2] virtio: pci support virtqueue selective enabling

[Eugenio Pérez]

Reusing virtqueue reset method to enabling a vq.

Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
---
 transport-pci.tex | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/transport-pci.tex b/transport-pci.tex
index a5c6719..04c0429 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 \item[\field{queue_enable}]
         The driver uses this to selectively prevent the device from executing requests from this virtqueue.
+        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
+        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
+        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
+        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
         1 - enabled; 0 - disabled.
 
 \item[\field{queue_notify_off}]
-- 
2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Parav Pandit]

Hi Eugenio,

> From: Eugenio Pérez <eperezma@redhat.com>
> Sent: Tuesday, June 6, 2023 1:55 PM
> 
> This patch allows the driver to start the device (as set DRIVER_OK) with only
> some queues enabled, and then enable another queues later.
> 
> This is the current way to migrate net device state through control virtqueue, in
> a software assisted framework with vDPA:
> * First, only net CVQ is enabled at DRIVER_OK
> * All the control commands (mac address, mq, etc) needed for the device to
> behave the same as the source of migration are sent
> * Finally all the dataplane queues are enabled.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
>  content.tex | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index d2ab9eb..ea4a6d8 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> Virtio Device / Feature B  \begin{description}
>  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> 
> -\item[24 to 41] Feature bits reserved for extensions to the queue and
> +\item[24 to 42] Feature bits reserved for extensions to the queue and
>    feature negotiation mechanisms
> 
> -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> +\item[43 to 49, and 128 and above] Feature bits reserved for future
> extensions.
>  \end{description}
> 
>  \begin{note}
> @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> or the Split  Virtqueue format, or both.
> 
> +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> +
> +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> +enable individual virtqueues both before and after the DRIVER\_OK. The
> +way to enable a virtqueue is transport specific, but it mimics the
> +re-enabling after an hypothetical reset.
> +
>  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> Virtqueues / Virtqueue Reset}
> 
>  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> Feature Bits}
>  	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
>  	handling features reserved for future use.
> 
> +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> + the driver can enable a queue both before and after DRIVER\_OK.
> +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> Virtqueue Enable}.
> +

Can you please highlight, current spec normative which indicates that all queues must be enabled before DRIVER_OK and more queues cannot be enabled after DRIVER_OK?

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

[Michael S. Tsirkin]

On Tue, Jun 06, 2023 at 07:55:11PM +0200, Eugenio Pérez wrote:
> Reusing virtqueue reset method to enabling a vq.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
>  transport-pci.tex | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/transport-pci.tex b/transport-pci.tex
> index a5c6719..04c0429 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>  
>  \item[\field{queue_enable}]
>          The driver uses this to selectively prevent the device from executing requests from this virtqueue.
> +        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
> +        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
> +        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> +        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
>          1 - enabled; 0 - disabled.

No, this is always used to enable queues even with no features.

>  \item[\field{queue_notify_off}]
> -- 
> 2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> This series allows the driver to start the device (as set DRIVER_OK) with only
> some queues enabled, and then enable another queues later.
> 
> This is the current way to migrate net device state through control
> virtqueue, in a software assisted framework with vDPA:
> * First, only net CVQ is enabled at DRIVER_OK
> * All the control commands (mac address, mq, etc) needed for the device
> to behave the same as the source of migration are sent
> * Finally all the dataplane queues are enabled.

In my opinion, this is somewhat problematic. Specifically, currently
devices tend to deduce how many queues are needed by looking
at the state at DRIVER_OK time.  
                                 
Question: what is wrong with enabling queues initially and then
doing a reset right after DRIVER_OK? You can even allocate
memory for just one queue (zeroing it out).

Granted this looks kind of ugly but side-steps this problem with
no need for spec changes.

> Eugenio Pérez (2):
>   virtio: introduce selective queue enabling
>   virtio: pci support virtqueue selective enabling
> 
>  content.tex       | 15 +++++++++++++--
>  transport-pci.tex |  4 ++++
>  2 files changed, 17 insertions(+), 2 deletions(-)
> 
> -- 
> 2.31.1
> 


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 06, 2023 at 07:55:10PM +0200, Eugenio Pérez wrote:
> This patch allows the driver to start the device (as set DRIVER_OK) with only
> some queues enabled, and then enable another queues later.
> 
> This is the current way to migrate net device state through control
> virtqueue, in a software assisted framework with vDPA:
> * First, only net CVQ is enabled at DRIVER_OK
> * All the control commands (mac address, mq, etc) needed for the device
> to behave the same as the source of migration are sent
> * Finally all the dataplane queues are enabled.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> ---
>  content.tex | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index d2ab9eb..ea4a6d8 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>  \begin{description}
>  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
>  
> -\item[24 to 41] Feature bits reserved for extensions to the queue and
> +\item[24 to 42] Feature bits reserved for extensions to the queue and
>    feature negotiation mechanisms
>  
> -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> +\item[43 to 49, and 128 and above] Feature bits reserved for future extensions.
>  \end{description}
>  
>  \begin{note}
> @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Virtqueues}
>  Every driver and device supports either the Packed or the Split
>  Virtqueue format, or both.
>  
> +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> +
> +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME,

No need for \_, we build with _ not being a special character.

> it can enable
> +individual virtqueues both before and after the DRIVER\_OK. The way to enable a
> +virtqueue is transport specific, but it mimics the re-enabling after an
> +hypothetical reset.
> +


This is not a good way to do it: spec describes a certain order
of initialization in several places. Please find all of them
and fix, don't just add a separate chapter as a kind of kludge.


>  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}
>  
>  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>  	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
>  	handling features reserved for future use.
>  
> +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates
> +  that the driver can enable a queue both before and after DRIVER\_OK.
> +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}.
> +
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> -- 
> 2.31.1


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 06, 2023 at 06:04:32PM +0000, Parav Pandit wrote:
> Hi Eugenio,
> 
> > From: Eugenio Pérez <eperezma@redhat.com>
> > Sent: Tuesday, June 6, 2023 1:55 PM
> > 
> > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > some queues enabled, and then enable another queues later.
> > 
> > This is the current way to migrate net device state through control virtqueue, in
> > a software assisted framework with vDPA:
> > * First, only net CVQ is enabled at DRIVER_OK
> > * All the control commands (mac address, mq, etc) needed for the device to
> > behave the same as the source of migration are sent
> > * Finally all the dataplane queues are enabled.
> > 
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > ---
> >  content.tex | 15 +++++++++++++--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/content.tex b/content.tex
> > index d2ab9eb..ea4a6d8 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > Virtio Device / Feature B  \begin{description}
> >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > 
> > -\item[24 to 41] Feature bits reserved for extensions to the queue and
> > +\item[24 to 42] Feature bits reserved for extensions to the queue and
> >    feature negotiation mechanisms
> > 
> > -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > +\item[43 to 49, and 128 and above] Feature bits reserved for future
> > extensions.
> >  \end{description}
> > 
> >  \begin{note}
> > @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> > Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> > or the Split  Virtqueue format, or both.
> > 
> > +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> > +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > +
> > +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> > +enable individual virtqueues both before and after the DRIVER\_OK. The
> > +way to enable a virtqueue is transport specific, but it mimics the
> > +re-enabling after an hypothetical reset.
> > +
> >  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> > Virtqueues / Virtqueue Reset}
> > 
> >  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > Feature Bits}
> >  	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> >  	handling features reserved for future use.
> > 
> > +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> > + the driver can enable a queue both before and after DRIVER\_OK.
> > +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> > Virtqueue Enable}.
> > +
> 
> Can you please highlight, current spec normative which indicates that
> all queues must be enabled before DRIVER_OK and more queues cannot be
> enabled after DRIVER_OK?


This one:
\drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}

it does not say "all queues" if you want to split hairs but I think
assuming it's safe to enable queues later is dangerous.

E.g. vhost user backends tend to assume no new queues appear
after DRIVER_OK.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, June 6, 2023 3:20 PM
> >
> > Can you please highlight, current spec normative which indicates that
> > all queues must be enabled before DRIVER_OK and more queues cannot be
> > enabled after DRIVER_OK?
> 
> 
> This one:
> \drivernormative{\subsection}{Device Initialization}{General Initialization And
> Device Operation / Device Initialization}
> 
> it does not say "all queues" if you want to split hairs but I think assuming it's
> safe to enable queues later is dangerous.
> 
Yes, I have read it few times and every time I concluded that:
it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.

> E.g. vhost user backends tend to assume no new queues appear after
> DRIVER_OK.

This backend implementation can be improved without introducing new bits on the guest side.
Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.

And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > This series allows the driver to start the device (as set DRIVER_OK) with only
> > some queues enabled, and then enable another queues later.
> >
> > This is the current way to migrate net device state through control
> > virtqueue, in a software assisted framework with vDPA:
> > * First, only net CVQ is enabled at DRIVER_OK
> > * All the control commands (mac address, mq, etc) needed for the device
> > to behave the same as the source of migration are sent
> > * Finally all the dataplane queues are enabled.
>
> In my opinion, this is somewhat problematic. Specifically, currently
> devices tend to deduce how many queues are needed by looking
> at the state at DRIVER_OK time.
>
> Question: what is wrong with enabling queues initially and then
> doing a reset right after DRIVER_OK? You can even allocate
> memory for just one queue (zeroing it out).
>
> Granted this looks kind of ugly but side-steps this problem with
> no need for spec changes.
>

The problem is that the rx queues can start receiving, as the guest
already has buffers there. Apart from that, the back and forth
introduces latencies.

Maybe a better angle is to start all the queues as if they're reset,
write 1 just to CVQ, configure the device, and then write 1 to all
dataplane vqs?

Thanks!

> > Eugenio Pérez (2):
> >   virtio: introduce selective queue enabling
> >   virtio: pci support virtqueue selective enabling
> >
> >  content.tex       | 15 +++++++++++++--
> >  transport-pci.tex |  4 ++++
> >  2 files changed, 17 insertions(+), 2 deletions(-)
> >
> > --
> > 2.31.1
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

[Eugenio Perez Martin]

On Tue, Jun 6, 2023 at 9:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 06, 2023 at 07:55:11PM +0200, Eugenio Pérez wrote:
> > Reusing virtqueue reset method to enabling a vq.
> >
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > ---
> >  transport-pci.tex | 4 ++++
> >  1 file changed, 4 insertions(+)
> >
> > diff --git a/transport-pci.tex b/transport-pci.tex
> > index a5c6719..04c0429 100644
> > --- a/transport-pci.tex
> > +++ b/transport-pci.tex
> > @@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >
> >  \item[\field{queue_enable}]
> >          The driver uses this to selectively prevent the device from executing requests from this virtqueue.
> > +        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
> > +        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
> > +        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > +        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> >          1 - enabled; 0 - disabled.
>
> No, this is always used to enable queues even with no features.
>

Yes. Now I see it is confusing if reading the spec and not the diff,
but my intention was to not modify the behavior without the feature
flags. Maybe there is a better way to achieve it.

> >  \item[\field{queue_notify_off}]
> > --
> > 2.31.1
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Xuan Zhuo]

On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > some queues enabled, and then enable another queues later.
> > >
> > > This is the current way to migrate net device state through control
> > > virtqueue, in a software assisted framework with vDPA:
> > > * First, only net CVQ is enabled at DRIVER_OK
> > > * All the control commands (mac address, mq, etc) needed for the device
> > > to behave the same as the source of migration are sent
> > > * Finally all the dataplane queues are enabled.
> >
> > In my opinion, this is somewhat problematic. Specifically, currently
> > devices tend to deduce how many queues are needed by looking
> > at the state at DRIVER_OK time.
> >
> > Question: what is wrong with enabling queues initially and then
> > doing a reset right after DRIVER_OK? You can even allocate
> > memory for just one queue (zeroing it out).
> >
> > Granted this looks kind of ugly but side-steps this problem with
> > no need for spec changes.
> >
>
> The problem is that the rx queues can start receiving, as the guest
> already has buffers there.

Can we reset the vq before filling buffers to it?

> Apart from that, the back and forth
> introduces latencies.
>
> Maybe a better angle is to start all the queues as if they're reset,
> write 1 just to CVQ, configure the device, and then write 1 to all
> dataplane vqs?

write to what?

Thanks.

>
> Thanks!
>
> > > Eugenio Pérez (2):
> > >   virtio: introduce selective queue enabling
> > >   virtio: pci support virtqueue selective enabling
> > >
> > >  content.tex       | 15 +++++++++++++--
> > >  transport-pci.tex |  4 ++++
> > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > >
> > > --
> > > 2.31.1
> > >
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > some queues enabled, and then enable another queues later.
> > > >
> > > > This is the current way to migrate net device state through control
> > > > virtqueue, in a software assisted framework with vDPA:
> > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > to behave the same as the source of migration are sent
> > > > * Finally all the dataplane queues are enabled.
> > >
> > > In my opinion, this is somewhat problematic. Specifically, currently
> > > devices tend to deduce how many queues are needed by looking
> > > at the state at DRIVER_OK time.
> > >
> > > Question: what is wrong with enabling queues initially and then
> > > doing a reset right after DRIVER_OK? You can even allocate
> > > memory for just one queue (zeroing it out).
> > >
> > > Granted this looks kind of ugly but side-steps this problem with
> > > no need for spec changes.
> > >
> >
> > The problem is that the rx queues can start receiving, as the guest
> > already has buffers there.
>
> Can we reset the vq before filling buffers to it?
>

They are passthrough from the guest so there is a window where the
device can process rx descriptors.

> > Apart from that, the back and forth
> > introduces latencies.
> >
> > Maybe a better angle is to start all the queues as if they're reset,
> > write 1 just to CVQ, configure the device, and then write 1 to all
> > dataplane vqs?
>
> write to what?
>

Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.

Thanks!

> Thanks.
>
> >
> > Thanks!
> >
> > > > Eugenio Pérez (2):
> > > >   virtio: introduce selective queue enabling
> > > >   virtio: pci support virtqueue selective enabling
> > > >
> > > >  content.tex       | 15 +++++++++++++--
> > > >  transport-pci.tex |  4 ++++
> > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > >
> > > > --
> > > > 2.31.1
> > > >
> > >
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > some queues enabled, and then enable another queues later.
> > > > >
> > > > > This is the current way to migrate net device state through control
> > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > to behave the same as the source of migration are sent
> > > > > * Finally all the dataplane queues are enabled.
> > > >
> > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > devices tend to deduce how many queues are needed by looking
> > > > at the state at DRIVER_OK time.
> > > >
> > > > Question: what is wrong with enabling queues initially and then
> > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > memory for just one queue (zeroing it out).
> > > >
> > > > Granted this looks kind of ugly but side-steps this problem with
> > > > no need for spec changes.
> > > >
> > >
> > > The problem is that the rx queues can start receiving, as the guest
> > > already has buffers there.
> >
> > Can we reset the vq before filling buffers to it?
> >
> 
> They are passthrough from the guest so there is a window where the
> device can process rx descriptors.

Not if there are no descriptors there.


> > > Apart from that, the back and forth
> > > introduces latencies.
> > >
> > > Maybe a better angle is to start all the queues as if they're reset,
> > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > dataplane vqs?
> >
> > write to what?
> >
> 
> Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> 
> Thanks!
> 
> > Thanks.
> >
> > >
> > > Thanks!
> > >
> > > > > Eugenio Pérez (2):
> > > > >   virtio: introduce selective queue enabling
> > > > >   virtio: pci support virtqueue selective enabling
> > > > >
> > > > >  content.tex       | 15 +++++++++++++--
> > > > >  transport-pci.tex |  4 ++++
> > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > >
> > > > > --
> > > > > 2.31.1
> > > > >
> > > >
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

[Michael S. Tsirkin]

On Wed, Jun 07, 2023 at 09:37:00AM +0200, Eugenio Perez Martin wrote:
> On Tue, Jun 6, 2023 at 9:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 06, 2023 at 07:55:11PM +0200, Eugenio Pérez wrote:
> > > Reusing virtqueue reset method to enabling a vq.
> > >
> > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > ---
> > >  transport-pci.tex | 4 ++++
> > >  1 file changed, 4 insertions(+)
> > >
> > > diff --git a/transport-pci.tex b/transport-pci.tex
> > > index a5c6719..04c0429 100644
> > > --- a/transport-pci.tex
> > > +++ b/transport-pci.tex
> > > @@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> > >
> > >  \item[\field{queue_enable}]
> > >          The driver uses this to selectively prevent the device from executing requests from this virtqueue.
> > > +        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
> > > +        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
> > > +        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > +        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> > >          1 - enabled; 0 - disabled.
> >
> > No, this is always used to enable queues even with no features.
> >
> 
> Yes. Now I see it is confusing if reading the spec and not the diff,
> but my intention was to not modify the behavior without the feature
> flags. Maybe there is a better way to achieve it.

This is bad:

 Driver always does X.
+If Y then driver does Z.


This is good:

-Driver always does X.
-If Y, then driver does Z, otherwise X.

even though it's a bigger patch, but patch review is
secondary to a readable spec.



> > >  \item[\field{queue_notify_off}]
> > > --
> > > 2.31.1
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > some queues enabled, and then enable another queues later.
> > > > > >
> > > > > > This is the current way to migrate net device state through control
> > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > to behave the same as the source of migration are sent
> > > > > > * Finally all the dataplane queues are enabled.
> > > > >
> > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > devices tend to deduce how many queues are needed by looking
> > > > > at the state at DRIVER_OK time.
> > > > >
> > > > > Question: what is wrong with enabling queues initially and then
> > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > memory for just one queue (zeroing it out).
> > > > >
> > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > no need for spec changes.
> > > > >
> > > >
> > > > The problem is that the rx queues can start receiving, as the guest
> > > > already has buffers there.
> > >
> > > Can we reset the vq before filling buffers to it?
> > >
> >
> > They are passthrough from the guest so there is a window where the
> > device can process rx descriptors.
>
> Not if there are no descriptors there.
>

But the migration is driven by the hypervisor, and it cannot control
that. The guest will likely have rx descriptors available.

>
> > > > Apart from that, the back and forth
> > > > introduces latencies.
> > > >
> > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > dataplane vqs?
> > >
> > > write to what?
> > >
> >
> > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> >
> > Thanks!
> >
> > > Thanks.
> > >
> > > >
> > > > Thanks!
> > > >
> > > > > > Eugenio Pérez (2):
> > > > > >   virtio: introduce selective queue enabling
> > > > > >   virtio: pci support virtqueue selective enabling
> > > > > >
> > > > > >  content.tex       | 15 +++++++++++++--
> > > > > >  transport-pci.tex |  4 ++++
> > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.31.1
> > > > > >
> > > > >
> > > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 2/2] virtio: pci support virtqueue selective enabling

[Eugenio Perez Martin]

On Wed, Jun 7, 2023 at 11:03 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 07, 2023 at 09:37:00AM +0200, Eugenio Perez Martin wrote:
> > On Tue, Jun 6, 2023 at 9:09 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 06, 2023 at 07:55:11PM +0200, Eugenio Pérez wrote:
> > > > Reusing virtqueue reset method to enabling a vq.
> > > >
> > > > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
> > > > ---
> > > >  transport-pci.tex | 4 ++++
> > > >  1 file changed, 4 insertions(+)
> > > >
> > > > diff --git a/transport-pci.tex b/transport-pci.tex
> > > > index a5c6719..04c0429 100644
> > > > --- a/transport-pci.tex
> > > > +++ b/transport-pci.tex
> > > > @@ -375,6 +375,10 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> > > >
> > > >  \item[\field{queue_enable}]
> > > >          The driver uses this to selectively prevent the device from executing requests from this virtqueue.
> > > > +        If the driver negotiates VIRTIO_F_RING_ENABLE_ANYTIME or
> > > > +        VIRTIO_F_RING_RESET, the driver writes 1 to this field to enable a
> > > > +        virtqueue.  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > > +        and \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Virtqueue Reset}.
> > > >          1 - enabled; 0 - disabled.
> > >
> > > No, this is always used to enable queues even with no features.
> > >
> >
> > Yes. Now I see it is confusing if reading the spec and not the diff,
> > but my intention was to not modify the behavior without the feature
> > flags. Maybe there is a better way to achieve it.
>
> This is bad:
>
>  Driver always does X.
> +If Y then driver does Z.
>
>
> This is good:
>
> -Driver always does X.
> -If Y, then driver does Z, otherwise X.
>
> even though it's a bigger patch, but patch review is
> secondary to a readable spec.
>

I agree, I'll modify it in that direction if we agree on using PCI
config space this way.

Thanks!

>
>
> > > >  \item[\field{queue_notify_off}]
> > > > --
> > > > 2.31.1
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > some queues enabled, and then enable another queues later.
> > > > > > >
> > > > > > > This is the current way to migrate net device state through control
> > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > to behave the same as the source of migration are sent
> > > > > > > * Finally all the dataplane queues are enabled.
> > > > > >
> > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > at the state at DRIVER_OK time.
> > > > > >
> > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > memory for just one queue (zeroing it out).
> > > > > >
> > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > no need for spec changes.
> > > > > >
> > > > >
> > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > already has buffers there.
> > > >
> > > > Can we reset the vq before filling buffers to it?
> > > >
> > >
> > > They are passthrough from the guest so there is a window where the
> > > device can process rx descriptors.
> >
> > Not if there are no descriptors there.
> >
> 
> But the migration is driven by the hypervisor, and it cannot control
> that. The guest will likely have rx descriptors available.

Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
rx queues? How do you do this - they are DMA from same VF no?


> >
> > > > > Apart from that, the back and forth
> > > > > introduces latencies.
> > > > >
> > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > dataplane vqs?
> > > >
> > > > write to what?
> > > >
> > >
> > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > >
> > > Thanks!
> > >
> > > > Thanks.
> > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > > > Eugenio Pérez (2):
> > > > > > >   virtio: introduce selective queue enabling
> > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > >
> > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > >  transport-pci.tex |  4 ++++
> > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > >
> > > > > > > --
> > > > > > > 2.31.1
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >
> > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > >
> > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > >
> > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > at the state at DRIVER_OK time.
> > > > > > >
> > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > memory for just one queue (zeroing it out).
> > > > > > >
> > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > no need for spec changes.
> > > > > > >
> > > > > >
> > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > already has buffers there.
> > > > >
> > > > > Can we reset the vq before filling buffers to it?
> > > > >
> > > >
> > > > They are passthrough from the guest so there is a window where the
> > > > device can process rx descriptors.
> > >
> > > Not if there are no descriptors there.
> > >
> >
> > But the migration is driven by the hypervisor, and it cannot control
> > that. The guest will likely have rx descriptors available.
>
> Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> rx queues?

No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So in this case if RX queues are enabled at the same time, the device
may try to queue packets to queue 0.

Thanks

> How do you do this - they are DMA from same VF no?
>
>
> > >
> > > > > > Apart from that, the back and forth
> > > > > > introduces latencies.
> > > > > >
> > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > dataplane vqs?
> > > > >
> > > > > write to what?
> > > > >
> > > >
> > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > >
> > > > Thanks!
> > > >
> > > > > Thanks.
> > > > >
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > > > Eugenio Pérez (2):
> > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > >
> > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > >
> > > > > > > > --
> > > > > > > > 2.31.1
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > >
> > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > >
> > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > >
> > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > at the state at DRIVER_OK time.
> > > > > > > >
> > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > >
> > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > no need for spec changes.
> > > > > > > >
> > > > > > >
> > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > already has buffers there.
> > > > > >
> > > > > > Can we reset the vq before filling buffers to it?
> > > > > >
> > > > >
> > > > > They are passthrough from the guest so there is a window where the
> > > > > device can process rx descriptors.
> > > >
> > > > Not if there are no descriptors there.
> > > >
> > >
> > > But the migration is driven by the hypervisor, and it cannot control
> > > that. The guest will likely have rx descriptors available.
> >
> > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > rx queues?
> 
> No, hypervisor tries to restore virtqueue states via cvq before guest can drive.

So cvq maps to hypervisor memory?


> So in this case if RX queues are enabled at the same time, the device
> may try to queue packets to queue 0.
> 
> Thanks
> 
> > How do you do this - they are DMA from same VF no?
> >
> >
> > > >
> > > > > > > Apart from that, the back and forth
> > > > > > > introduces latencies.
> > > > > > >
> > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > dataplane vqs?
> > > > > >
> > > > > > write to what?
> > > > > >
> > > > >
> > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > > > Eugenio Pérez (2):
> > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > >
> > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > 2.31.1
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > >
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > >
> > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > >
> > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > >
> > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > >
> > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > >
> > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > no need for spec changes.
> > > > > > > > >
> > > > > > > >
> > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > already has buffers there.
> > > > > > >
> > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > >
> > > > > >
> > > > > > They are passthrough from the guest so there is a window where the
> > > > > > device can process rx descriptors.
> > > > >
> > > > > Not if there are no descriptors there.
> > > > >
> > > >
> > > > But the migration is driven by the hypervisor, and it cannot control
> > > > that. The guest will likely have rx descriptors available.
> > >
> > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > rx queues?
> >
> > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
>
> So cvq maps to hypervisor memory?
>

From the device POV, yes, the CVQ vring is in hypervisor memory, not
in the guest's one. That allows the hypervisor to send the CVQ
commands to restore the device state in the destination, without the
guest intervention.

Data vqs, on the other hand, are passthrough. The device talks
directly to the guest's vrings.

Currently, this is done by emulating CVQ in the host's kernel and then
translating the commands in the way that suits better the vdpa device,
using its vendor vdpa driver in the host. Other methods like PASID are
possible too.

Thanks!

>
>
> > So in this case if RX queues are enabled at the same time, the device
> > may try to queue packets to queue 0.
> >
> > Thanks
> >
> > > How do you do this - they are DMA from same VF no?
> > >
> > >
> > > > >
> > > > > > > > Apart from that, the back and forth
> > > > > > > > introduces latencies.
> > > > > > > >
> > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > dataplane vqs?
> > > > > > >
> > > > > > > write to what?
> > > > > > >
> > > > > >
> > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > >
> > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > 2.31.1
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > >
> > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > before posting.
> > > > > > >
> > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > >
> > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > >
> > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > >
> > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > >
> > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > no need for spec changes.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > already has buffers there.
> > > > > > > >
> > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > >
> > > > > > >
> > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > device can process rx descriptors.
> > > > > >
> > > > > > Not if there are no descriptors there.
> > > > > >
> > > > >
> > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > that. The guest will likely have rx descriptors available.
> > > >
> > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > rx queues?
> > >
> > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> >
> > So cvq maps to hypervisor memory?
> >
> 
> >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> in the guest's one. That allows the hypervisor to send the CVQ
> commands to restore the device state in the destination, without the
> guest intervention.
> 
> Data vqs, on the other hand, are passthrough. The device talks
> directly to the guest's vrings.
> 
> Currently, this is done by emulating CVQ in the host's kernel and then
> translating the commands in the way that suits better the vdpa device,
> using its vendor vdpa driver in the host. Other methods like PASID are
> possible too.
> 
> Thanks!

OK. So my suggestion is simple: map data vrings to a zero page in
hypervisor memory initially. Later reset and map to guest.


If that does not work, then I am not sure this proposal is enough
since I think devices want to have a specific point in time
where they know which queues are going to be used.
Maybe we could use e.g. bit 1 in queue_enable to signal that?


> >
> >
> > > So in this case if RX queues are enabled at the same time, the device
> > > may try to queue packets to queue 0.
> > >
> > > Thanks
> > >
> > > > How do you do this - they are DMA from same VF no?
> > > >
> > > >
> > > > > >
> > > > > > > > > Apart from that, the back and forth
> > > > > > > > > introduces latencies.
> > > > > > > > >
> > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > dataplane vqs?
> > > > > > > >
> > > > > > > > write to what?
> > > > > > > >
> > > > > > >
> > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > >
> > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > 2.31.1
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > >
> > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > before posting.
> > > > > > > >
> > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > >
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Thu, Jun 8, 2023 at 2:04 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > >
> > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > >
> > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > >
> > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > >
> > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > >
> > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > no need for spec changes.
> > > > > > > > >
> > > > > > > >
> > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > already has buffers there.
> > > > > > >
> > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > >
> > > > > >
> > > > > > They are passthrough from the guest so there is a window where the
> > > > > > device can process rx descriptors.
> > > > >
> > > > > Not if there are no descriptors there.
> > > > >
> > > >
> > > > But the migration is driven by the hypervisor, and it cannot control
> > > > that. The guest will likely have rx descriptors available.
> > >
> > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > rx queues?
> >
> > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
>
> So cvq maps to hypervisor memory?

Yes.

Thanks

>
>
> > So in this case if RX queues are enabled at the same time, the device
> > may try to queue packets to queue 0.
> >
> > Thanks
> >
> > > How do you do this - they are DMA from same VF no?
> > >
> > >
> > > > >
> > > > > > > > Apart from that, the back and forth
> > > > > > > > introduces latencies.
> > > > > > > >
> > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > dataplane vqs?
> > > > > > >
> > > > > > > write to what?
> > > > > > >
> > > > > >
> > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > >
> > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > 2.31.1
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > >
> > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > before posting.
> > > > > > >
> > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Thu, Jun 8, 2023 at 3:19 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > >
> > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > >
> > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > >
> > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > >
> > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > no need for spec changes.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > already has buffers there.
> > > > > > > > >
> > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > >
> > > > > > > >
> > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > device can process rx descriptors.
> > > > > > >
> > > > > > > Not if there are no descriptors there.
> > > > > > >
> > > > > >
> > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > that. The guest will likely have rx descriptors available.
> > > > >
> > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > rx queues?
> > > >
> > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > >
> > > So cvq maps to hypervisor memory?
> > >
> >
> > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > in the guest's one. That allows the hypervisor to send the CVQ
> > commands to restore the device state in the destination, without the
> > guest intervention.
> >
> > Data vqs, on the other hand, are passthrough. The device talks
> > directly to the guest's vrings.
> >
> > Currently, this is done by emulating CVQ in the host's kernel and then
> > translating the commands in the way that suits better the vdpa device,
> > using its vendor vdpa driver in the host. Other methods like PASID are
> > possible too.
> >
> > Thanks!
>
> OK. So my suggestion is simple: map data vrings to a zero page in
> hypervisor memory initially. Later reset and map to guest.

Is there a spec section that guarantees the above behaviour? (I guess not).

Thanks

>
>
> If that does not work, then I am not sure this proposal is enough
> since I think devices want to have a specific point in time
> where they know which queues are going to be used.
> Maybe we could use e.g. bit 1 in queue_enable to signal that?
>
>
> > >
> > >
> > > > So in this case if RX queues are enabled at the same time, the device
> > > > may try to queue packets to queue 0.
> > > >
> > > > Thanks
> > > >
> > > > > How do you do this - they are DMA from same VF no?
> > > > >
> > > > >
> > > > > > >
> > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > introduces latencies.
> > > > > > > > > >
> > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > dataplane vqs?
> > > > > > > > >
> > > > > > > > > write to what?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > >
> > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > 2.31.1
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > >
> > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > before posting.
> > > > > > > > >
> > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Thu, Jun 8, 2023 at 9:19 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > >
> > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > >
> > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > >
> > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > >
> > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > no need for spec changes.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > already has buffers there.
> > > > > > > > >
> > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > >
> > > > > > > >
> > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > device can process rx descriptors.
> > > > > > >
> > > > > > > Not if there are no descriptors there.
> > > > > > >
> > > > > >
> > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > that. The guest will likely have rx descriptors available.
> > > > >
> > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > rx queues?
> > > >
> > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > >
> > > So cvq maps to hypervisor memory?
> > >
> >
> > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > in the guest's one. That allows the hypervisor to send the CVQ
> > commands to restore the device state in the destination, without the
> > guest intervention.
> >
> > Data vqs, on the other hand, are passthrough. The device talks
> > directly to the guest's vrings.
> >
> > Currently, this is done by emulating CVQ in the host's kernel and then
> > translating the commands in the way that suits better the vdpa device,
> > using its vendor vdpa driver in the host. Other methods like PASID are
> > possible too.
> >
> > Thanks!
>
> OK. So my suggestion is simple: map data vrings to a zero page in
> hypervisor memory initially. Later reset and map to guest.
>

The idea is interesting, but we lose the net configuration in a device
reset, so we need to send it again.

In the case of qemu+vDPA maybe it is possible with queue_reset, like:
* Map all the guest pages as usual.
* Map a new zero page, forbid the guest to write on that page. Even in
vIOMMU case, we can send all the CVQ commands before allowing the
guest to modify mappings. The guest has no way to write on that page
through the device as, well, dataplane is not initialized. It's the
way DPDK shadow virtqueues worked, so it should be valid.
* Reset the queues to the guest passthrough.

I don't like the complexity of it but I like it does require even less
changes to the device / spec.

>
> If that does not work, then I am not sure this proposal is enough
> since I think devices want to have a specific point in time
> where they know which queues are going to be used.

In the case of net, this should not be a problem since the spec
mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if
cvq and mq. virtio-blk also has num_queues.

Is it even valid to not enable some of the queues?

I've always felt that queue_enable has been redundant before
queue_reset for this reason actually.

> Maybe we could use e.g. bit 1 in queue_enable to signal that?
>

I'm totally ok to go in that direction.

Thanks!

>
> > >
> > >
> > > > So in this case if RX queues are enabled at the same time, the device
> > > > may try to queue packets to queue 0.
> > > >
> > > > Thanks
> > > >
> > > > > How do you do this - they are DMA from same VF no?
> > > > >
> > > > >
> > > > > > >
> > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > introduces latencies.
> > > > > > > > > >
> > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > dataplane vqs?
> > > > > > > > >
> > > > > > > > > write to what?
> > > > > > > > >
> > > > > > > >
> > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > >
> > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > 2.31.1
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > >
> > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > before posting.
> > > > > > > > >
> > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 6, 2023 at 9:25 PM Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, June 6, 2023 3:20 PM
> > >
> > > Can you please highlight, current spec normative which indicates that
> > > all queues must be enabled before DRIVER_OK and more queues cannot be
> > > enabled after DRIVER_OK?
> >
> >
> > This one:
> > \drivernormative{\subsection}{Device Initialization}{General Initialization And
> > Device Operation / Device Initialization}
> >
> > it does not say "all queues" if you want to split hairs but I think assuming it's
> > safe to enable queues later is dangerous.
> >
> Yes, I have read it few times and every time I concluded that:
> it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.
>

I also read it this way to be honest. From a POV, to allow it if a
device does not handle it could be considered a fix, rather than an
addition.

I'm assuming we need to modify the standard just because it was not
explicitly allowed. To explicitly allow it before DRIVER_OK was not
needed because it was already widespread.

It would be great for me if we agree the device must support it, but
maybe it is just too late for that.

Thanks!

> > E.g. vhost user backends tend to assume no new queues appear after
> > DRIVER_OK.
>
> This backend implementation can be improved without introducing new bits on the guest side.
> Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.
>
> And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Xuan Zhuo]

On Tue, 6 Jun 2023 15:19:53 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Jun 06, 2023 at 06:04:32PM +0000, Parav Pandit wrote:
> > Hi Eugenio,
> >
> > > From: Eugenio P閞ez <eperezma@redhat.com>
> > > Sent: Tuesday, June 6, 2023 1:55 PM
> > >
> > > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > > some queues enabled, and then enable another queues later.
> > >
> > > This is the current way to migrate net device state through control virtqueue, in
> > > a software assisted framework with vDPA:
> > > * First, only net CVQ is enabled at DRIVER_OK
> > > * All the control commands (mac address, mq, etc) needed for the device to
> > > behave the same as the source of migration are sent
> > > * Finally all the dataplane queues are enabled.
> > >
> > > Signed-off-by: Eugenio P閞ez <eperezma@redhat.com>
> > > ---
> > >  content.tex | 15 +++++++++++++--
> > >  1 file changed, 13 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/content.tex b/content.tex
> > > index d2ab9eb..ea4a6d8 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > > Virtio Device / Feature B  \begin{description}
> > >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > >
> > > -\item[24 to 41] Feature bits reserved for extensions to the queue and
> > > +\item[24 to 42] Feature bits reserved for extensions to the queue and
> > >    feature negotiation mechanisms
> > >
> > > -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > > +\item[43 to 49, and 128 and above] Feature bits reserved for future
> > > extensions.
> > >  \end{description}
> > >
> > >  \begin{note}
> > > @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> > > Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> > > or the Split  Virtqueue format, or both.
> > >
> > > +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> > > +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > +
> > > +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> > > +enable individual virtqueues both before and after the DRIVER\_OK. The
> > > +way to enable a virtqueue is transport specific, but it mimics the
> > > +re-enabling after an hypothetical reset.
> > > +
> > >  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> > > Virtqueues / Virtqueue Reset}
> > >
> > >  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > > @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > > Feature Bits}
> > >  	\ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> > >  	handling features reserved for future use.
> > >
> > > +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> > > + the driver can enable a queue both before and after DRIVER\_OK.
> > > +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> > > Virtqueue Enable}.
> > > +
> >
> > Can you please highlight, current spec normative which indicates that
> > all queues must be enabled before DRIVER_OK and more queues cannot be
> > enabled after DRIVER_OK?
>
>
> This one:
> \drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}
>
> it does not say "all queues" if you want to split hairs but I think
> assuming it's safe to enable queues later is dangerous.

Although there is no "all queues" here, if the queue is not ready, the device
may use all the queues after "Device OK". So I also think it might be dangerous.

But I also ran into this problem. We want the device to support 100 queues, but
we also want to use only 16 queues by default. The user can then manually scale
up to 100, if the user has a large memory.

So I think, the point is that "num_queues" is the max and default value. Can we
change "num_queues" to default, then we introduced a new value as max.

Does this way solve your problem? @Eugenio

Thanks.



>
> E.g. vhost user backends tend to assume no new queues appear
> after DRIVER_OK.
>
> --
> MST
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Parav Pandit]

> From: Eugenio Perez Martin <eperezma@redhat.com>
> Sent: Thursday, June 8, 2023 7:54 AM

> > Yes, I have read it few times and every time I concluded that:
> > it does not say that it must be done before DRIVER_OK and cannot be done
> after DRIVER_OK.
> >
> 
> I also read it this way to be honest. From a POV, to allow it if a device does not
> handle it could be considered a fix, rather than an addition.
> 
> I'm assuming we need to modify the standard just because it was not explicitly
> allowed. To explicitly allow it before DRIVER_OK was not needed because it was
> already widespread.
> 
It was explicit enough that queues can be enabled after DRIVER_OK.
And queue reset is the perfect example showed that.
Who ever implements the device should be ready to handle queue_enable writes after DRIVER_OK stage, it is really that simple.
There is no need of new feature bit.

> It would be great for me if we agree the device must support it, but maybe it is
> just too late for that.
Device must support it. There is nothing late about it.
And if you are considering LM flow, then for sure it is not late because LM is just not fully baked.

[virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Thursday, June 8, 2023 8:12 AM


> Although there is no "all queues" here, if the queue is not ready, the device may
> use all the queues after "Device OK". So I also think it might be dangerous.
>
Device can only used enabled queues at any stage before/after DRIVER_OK.
If the queue is not enabled, there is simply no rx packet or anything on that queue.
 
> But I also ran into this problem. We want the device to support 100 queues, but
> we also want to use only 16 queues by default. The user can then manually
> scale up to 100, if the user has a large memory.
> 
So, one config sequence :
a. driver can enable 16 queues before  driver ok 
b. driver does driver ok
c. few hours later driver enables remaining 84 queues

in second config sequence:
a. driver can enable 0 queues before  driver ok 
b. driver does driver ok
c. few hours later driver enables remaining 16+84 queues 

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 03:47:41PM +0800, Jason Wang wrote:
> On Thu, Jun 8, 2023 at 3:19 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > > >
> > > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > > >
> > > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > > >
> > > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > > no need for spec changes.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > > already has buffers there.
> > > > > > > > > >
> > > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > > device can process rx descriptors.
> > > > > > > >
> > > > > > > > Not if there are no descriptors there.
> > > > > > > >
> > > > > > >
> > > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > > that. The guest will likely have rx descriptors available.
> > > > > >
> > > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > > rx queues?
> > > > >
> > > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > > >
> > > > So cvq maps to hypervisor memory?
> > > >
> > >
> > > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > > in the guest's one. That allows the hypervisor to send the CVQ
> > > commands to restore the device state in the destination, without the
> > > guest intervention.
> > >
> > > Data vqs, on the other hand, are passthrough. The device talks
> > > directly to the guest's vrings.
> > >
> > > Currently, this is done by emulating CVQ in the host's kernel and then
> > > translating the commands in the way that suits better the vdpa device,
> > > using its vendor vdpa driver in the host. Other methods like PASID are
> > > possible too.
> > >
> > > Thanks!
> >
> > OK. So my suggestion is simple: map data vrings to a zero page in
> > hypervisor memory initially. Later reset and map to guest.
> 
> Is there a spec section that guarantees the above behaviour? (I guess not).
> 
> Thanks


yes, the initialization section. driver MUST first setup vqs then
set DRIVER_OK.


> >
> >
> > If that does not work, then I am not sure this proposal is enough
> > since I think devices want to have a specific point in time
> > where they know which queues are going to be used.
> > Maybe we could use e.g. bit 1 in queue_enable to signal that?
> >
> >
> > > >
> > > >
> > > > > So in this case if RX queues are enabled at the same time, the device
> > > > > may try to queue packets to queue 0.
> > > > >
> > > > > Thanks
> > > > >
> > > > > > How do you do this - they are DMA from same VF no?
> > > > > >
> > > > > >
> > > > > > > >
> > > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > > introduces latencies.
> > > > > > > > > > >
> > > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > > dataplane vqs?
> > > > > > > > > >
> > > > > > > > > > write to what?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > > >
> > > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > 2.31.1
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > >
> > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > before posting.
> > > > > > > > > >
> > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Eugenio Perez Martin <eperezma@redhat.com>
> Sent: Thursday, June 8, 2023 4:36 AM

> In the case of net, this should not be a problem since the spec mandates 2 if
> !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if cvq and mq. virtio-blk
> also has num_queues.
> 
num_queues is maximum supported count, not enabled count.

_F_RESET was introduced to enable/disable the queue after DRIVER_OK, which implies that we don’t need yet another feature bit.

> I've always felt that queue_enable has been redundant before queue_reset for
> this reason actually.
> 
Queue_reset was born after queue_enable register.
So before queue_reset, queue_enable cannot be redundant.

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Eugenio Perez Martin]

On Thu, Jun 8, 2023 at 2:24 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Tue, 6 Jun 2023 15:19:53 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, Jun 06, 2023 at 06:04:32PM +0000, Parav Pandit wrote:
> > > Hi Eugenio,
> > >
> > > > From: Eugenio P閞ez <eperezma@redhat.com>
> > > > Sent: Tuesday, June 6, 2023 1:55 PM
> > > >
> > > > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > > > some queues enabled, and then enable another queues later.
> > > >
> > > > This is the current way to migrate net device state through control virtqueue, in
> > > > a software assisted framework with vDPA:
> > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > * All the control commands (mac address, mq, etc) needed for the device to
> > > > behave the same as the source of migration are sent
> > > > * Finally all the dataplane queues are enabled.
> > > >
> > > > Signed-off-by: Eugenio P閞ez <eperezma@redhat.com>
> > > > ---
> > > >  content.tex | 15 +++++++++++++--
> > > >  1 file changed, 13 insertions(+), 2 deletions(-)
> > > >
> > > > diff --git a/content.tex b/content.tex
> > > > index d2ab9eb..ea4a6d8 100644
> > > > --- a/content.tex
> > > > +++ b/content.tex
> > > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > > > Virtio Device / Feature B  \begin{description}
> > > >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > > >
> > > > -\item[24 to 41] Feature bits reserved for extensions to the queue and
> > > > +\item[24 to 42] Feature bits reserved for extensions to the queue and
> > > >    feature negotiation mechanisms
> > > >
> > > > -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > > > +\item[43 to 49, and 128 and above] Feature bits reserved for future
> > > > extensions.
> > > >  \end{description}
> > > >
> > > >  \begin{note}
> > > > @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> > > > Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> > > > or the Split  Virtqueue format, or both.
> > > >
> > > > +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> > > > +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > > +
> > > > +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> > > > +enable individual virtqueues both before and after the DRIVER\_OK. The
> > > > +way to enable a virtqueue is transport specific, but it mimics the
> > > > +re-enabling after an hypothetical reset.
> > > > +
> > > >  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> > > > Virtqueues / Virtqueue Reset}
> > > >
> > > >  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > > > @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > > > Feature Bits}
> > > >   \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> > > >   handling features reserved for future use.
> > > >
> > > > +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> > > > + the driver can enable a queue both before and after DRIVER\_OK.
> > > > +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> > > > Virtqueue Enable}.
> > > > +
> > >
> > > Can you please highlight, current spec normative which indicates that
> > > all queues must be enabled before DRIVER_OK and more queues cannot be
> > > enabled after DRIVER_OK?
> >
> >
> > This one:
> > \drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}
> >
> > it does not say "all queues" if you want to split hairs but I think
> > assuming it's safe to enable queues later is dangerous.
>
> Although there is no "all queues" here, if the queue is not ready, the device
> may use all the queues after "Device OK". So I also think it might be dangerous.
>
> But I also ran into this problem. We want the device to support 100 queues, but
> we also want to use only 16 queues by default. The user can then manually scale
> up to 100, if the user has a large memory.
>
> So I think, the point is that "num_queues" is the max and default value. Can we
> change "num_queues" to default, then we introduced a new value as max.
>
> Does this way solve your problem? @Eugenio
>

It would need a tweak, but it gives me an idea.

I think you're talking about what the device offers, isn't it? I would
need to modify it from the driver.

But it has an interesting question, how to allocate "less vrings" than
virtio_pci_common_cfg.num_queues? I guess it is ok to set all vq
addresses to a vring that never increments avail_idx? The driver still
needs to allocate memory for that one and lose start time configuring
them.

Maybe for net is valid to leave them unconfigured? it should only use
one vq pair and CVQ by default.

My proposal could be done at the virtio-net level, allowing
max_virtqueue_pairs to be 0 from the beginning. For example, allowing
the driver to write it before DRIVER_OK. In that case, no data vq is
enabled and only CVQ process requests after DRIVER_OK. After all the
config is written, the driver changes the number of active queues with
VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.

I think it is better solved in a generic way, but I'm ok with this
-net specific too.

Thanks!

> Thanks.
>
>
>
> >
> > E.g. vhost user backends tend to assume no new queues appear
> > after DRIVER_OK.
> >
> > --
> > MST
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 04:18:23PM +0200, Eugenio Perez Martin wrote:
> On Thu, Jun 8, 2023 at 2:24 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 6 Jun 2023 15:19:53 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Tue, Jun 06, 2023 at 06:04:32PM +0000, Parav Pandit wrote:
> > > > Hi Eugenio,
> > > >
> > > > > From: Eugenio P閞ez <eperezma@redhat.com>
> > > > > Sent: Tuesday, June 6, 2023 1:55 PM
> > > > >
> > > > > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > > > > some queues enabled, and then enable another queues later.
> > > > >
> > > > > This is the current way to migrate net device state through control virtqueue, in
> > > > > a software assisted framework with vDPA:
> > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > * All the control commands (mac address, mq, etc) needed for the device to
> > > > > behave the same as the source of migration are sent
> > > > > * Finally all the dataplane queues are enabled.
> > > > >
> > > > > Signed-off-by: Eugenio P閞ez <eperezma@redhat.com>
> > > > > ---
> > > > >  content.tex | 15 +++++++++++++--
> > > > >  1 file changed, 13 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/content.tex b/content.tex
> > > > > index d2ab9eb..ea4a6d8 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > > > > Virtio Device / Feature B  \begin{description}
> > > > >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > > > >
> > > > > -\item[24 to 41] Feature bits reserved for extensions to the queue and
> > > > > +\item[24 to 42] Feature bits reserved for extensions to the queue and
> > > > >    feature negotiation mechanisms
> > > > >
> > > > > -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > > > > +\item[43 to 49, and 128 and above] Feature bits reserved for future
> > > > > extensions.
> > > > >  \end{description}
> > > > >
> > > > >  \begin{note}
> > > > > @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> > > > > Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> > > > > or the Split  Virtqueue format, or both.
> > > > >
> > > > > +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> > > > > +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > > > +
> > > > > +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> > > > > +enable individual virtqueues both before and after the DRIVER\_OK. The
> > > > > +way to enable a virtqueue is transport specific, but it mimics the
> > > > > +re-enabling after an hypothetical reset.
> > > > > +
> > > > >  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> > > > > Virtqueues / Virtqueue Reset}
> > > > >
> > > > >  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > > > > @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > > > > Feature Bits}
> > > > >   \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> > > > >   handling features reserved for future use.
> > > > >
> > > > > +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> > > > > + the driver can enable a queue both before and after DRIVER\_OK.
> > > > > +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> > > > > Virtqueue Enable}.
> > > > > +
> > > >
> > > > Can you please highlight, current spec normative which indicates that
> > > > all queues must be enabled before DRIVER_OK and more queues cannot be
> > > > enabled after DRIVER_OK?
> > >
> > >
> > > This one:
> > > \drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}
> > >
> > > it does not say "all queues" if you want to split hairs but I think
> > > assuming it's safe to enable queues later is dangerous.
> >
> > Although there is no "all queues" here, if the queue is not ready, the device
> > may use all the queues after "Device OK". So I also think it might be dangerous.
> >
> > But I also ran into this problem. We want the device to support 100 queues, but
> > we also want to use only 16 queues by default. The user can then manually scale
> > up to 100, if the user has a large memory.
> >
> > So I think, the point is that "num_queues" is the max and default value. Can we
> > change "num_queues" to default, then we introduced a new value as max.
> >
> > Does this way solve your problem? @Eugenio
> >
> 
> It would need a tweak, but it gives me an idea.
> 
> I think you're talking about what the device offers, isn't it? I would
> need to modify it from the driver.
> 
> But it has an interesting question, how to allocate "less vrings" than
> virtio_pci_common_cfg.num_queues?

Just allocate less, don't configure what you do not use.

> I guess it is ok to set all vq
> addresses to a vring that never increments avail_idx? The driver still
> needs to allocate memory for that one and lose start time configuring
> them.
> 
> Maybe for net is valid to leave them unconfigured? it should only use
> one vq pair and CVQ by default.

It's ok for all devices, and this is how it always worked.

> My proposal could be done at the virtio-net level, allowing
> max_virtqueue_pairs to be 0 from the beginning. For example, allowing
> the driver to write it before DRIVER_OK. In that case, no data vq is
> enabled and only CVQ process requests after DRIVER_OK. After all the
> config is written, the driver changes the number of active queues with
> VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
> 
> I think it is better solved in a generic way, but I'm ok with this
> -net specific too.
> 
> Thanks!
> 
> > Thanks.
> >
> >
> >
> > >
> > > E.g. vhost user backends tend to assume no new queues appear
> > > after DRIVER_OK.
> > >
> > > --
> > > MST
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 01:15:46PM +0000, Parav Pandit wrote:
> 
> 
> > From: Eugenio Perez Martin <eperezma@redhat.com>
> > Sent: Thursday, June 8, 2023 7:54 AM
> 
> > > Yes, I have read it few times and every time I concluded that:
> > > it does not say that it must be done before DRIVER_OK and cannot be done
> > after DRIVER_OK.
> > >
> > 
> > I also read it this way to be honest. From a POV, to allow it if a device does not
> > handle it could be considered a fix, rather than an addition.
> > 
> > I'm assuming we need to modify the standard just because it was not explicitly
> > allowed. To explicitly allow it before DRIVER_OK was not needed because it was
> > already widespread.
> > 
> It was explicit enough that queues can be enabled after DRIVER_OK.

Weird, there's spec text saying they can't.

> And queue reset is the perfect example showed that.
> Who ever implements the device should be ready to handle queue_enable writes after DRIVER_OK stage, it is really that simple.
> There is no need of new feature bit.

Hard to change the past, we didn't ask people to do it.

> > It would be great for me if we agree the device must support it, but maybe it is
> > just too late for that.
> Device must support it. There is nothing late about it.
> And if you are considering LM flow, then for sure it is not late because LM is just not fully baked.

Considering regular use, no LM.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, June 8, 2023 11:06 AM

> > > I'm assuming we need to modify the standard just because it was not
> > > explicitly allowed. To explicitly allow it before DRIVER_OK was not
> > > needed because it was already widespread.
> > >
> > It was explicit enough that queues can be enabled after DRIVER_OK.
> 
> Weird, there's spec text saying they can't.
> 
So far no one cited that text here. :)
You pointed to the general initialization section but it doesnt specify.

> > And queue reset is the perfect example showed that.
> > Who ever implements the device should be ready to handle queue_enable
> writes after DRIVER_OK stage, it is really that simple.
> > There is no need of new feature bit.
> 
> Hard to change the past, we didn't ask people to do it.
> 
> > > It would be great for me if we agree the device must support it, but
> > > maybe it is just too late for that.
> > Device must support it. There is nothing late about it.
> > And if you are considering LM flow, then for sure it is not late because LM is
> just not fully baked.
> 
> Considering regular use, no LM.

_RESET is the existing feature bit that allows one to enable queues after DRIVER_OK.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] RE: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 01:53:32PM +0200, Eugenio Perez Martin wrote:
> On Tue, Jun 6, 2023 at 9:25 PM Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, June 6, 2023 3:20 PM
> > > >
> > > > Can you please highlight, current spec normative which indicates that
> > > > all queues must be enabled before DRIVER_OK and more queues cannot be
> > > > enabled after DRIVER_OK?
> > >
> > >
> > > This one:
> > > \drivernormative{\subsection}{Device Initialization}{General Initialization And
> > > Device Operation / Device Initialization}
> > >
> > > it does not say "all queues" if you want to split hairs but I think assuming it's
> > > safe to enable queues later is dangerous.
> > >
> > Yes, I have read it few times and every time I concluded that:
> > it does not say that it must be done before DRIVER_OK and cannot be done after DRIVER_OK.
> >
> 
> I also read it this way to be honest.

You have been doing spec lawyering for too long, take a step
back, take a walk, then look at it from common sense POV.


Well spec says: you MUST follow this order:
- A
- B
- C

can one conclude that if C triggers then B already happened?

I would conclude yes.



and just because it does not say *and do not do it in any other
order* you are saying
- A
- C
- B

is also allowed?

MUST follow this rule to me means "and do not violate this rule".




> From a POV, to allow it if a
> device does not handle it could be considered a fix, rather than an
> addition.

I think vhost user at least is started at DRIVER_OK and at
that point it looks at # of enabled Qs and allocates a bunch
of arrays to match.


> I'm assuming we need to modify the standard just because it was not
> explicitly allowed. To explicitly allow it before DRIVER_OK was not
> needed because it was already widespread.
> 
> It would be great for me if we agree the device must support it, but
> maybe it is just too late for that.
> 
> Thanks!
> 
> > > E.g. vhost user backends tend to assume no new queues appear after
> > > DRIVER_OK.
> >
> > This backend implementation can be improved without introducing new bits on the guest side.
> > Q reset is already a second example that demonstrates that queues will be disabled/enabled after DRIVER_OK.
> >
> > And for PCI transport, PCI device and driver do not restrict queue_enable access after DRIVER_OK.
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 08, 2023 at 10:36:19AM +0200, Eugenio Perez Martin wrote:
> On Thu, Jun 8, 2023 at 9:19 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > > >
> > > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > > >
> > > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > > >
> > > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > > no need for spec changes.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > > already has buffers there.
> > > > > > > > > >
> > > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > > device can process rx descriptors.
> > > > > > > >
> > > > > > > > Not if there are no descriptors there.
> > > > > > > >
> > > > > > >
> > > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > > that. The guest will likely have rx descriptors available.
> > > > > >
> > > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > > rx queues?
> > > > >
> > > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > > >
> > > > So cvq maps to hypervisor memory?
> > > >
> > >
> > > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > > in the guest's one. That allows the hypervisor to send the CVQ
> > > commands to restore the device state in the destination, without the
> > > guest intervention.
> > >
> > > Data vqs, on the other hand, are passthrough. The device talks
> > > directly to the guest's vrings.
> > >
> > > Currently, this is done by emulating CVQ in the host's kernel and then
> > > translating the commands in the way that suits better the vdpa device,
> > > using its vendor vdpa driver in the host. Other methods like PASID are
> > > possible too.
> > >
> > > Thanks!
> >
> > OK. So my suggestion is simple: map data vrings to a zero page in
> > hypervisor memory initially. Later reset and map to guest.
> >
> 
> The idea is interesting, but we lose the net configuration in a device
> reset, so we need to send it again.

Ring reset, not device reset.

> In the case of qemu+vDPA maybe it is possible with queue_reset, like:
> * Map all the guest pages as usual.
> * Map a new zero page, forbid the guest to write on that page. Even in
> vIOMMU case, we can send all the CVQ commands before allowing the
> guest to modify mappings. The guest has no way to write on that page
> through the device as, well, dataplane is not initialized. It's the
> way DPDK shadow virtqueues worked, so it should be valid.
> * Reset the queues to the guest passthrough.
> 
> I don't like the complexity of it but I like it does require even less
> changes to the device / spec.

This is exactly what I meant.

> >
> > If that does not work, then I am not sure this proposal is enough
> > since I think devices want to have a specific point in time
> > where they know which queues are going to be used.
> 
> In the case of net, this should not be a problem since the spec
> mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if
> cvq and mq. virtio-blk also has num_queues.

This is max, not how many there are in practice.

> Is it even valid to not enable some of the queues?

Yes and linux will do that if max_virtqueue_pairs > #CPUs.

> I've always felt that queue_enable has been redundant before
> queue_reset for this reason actually.
> 
> > Maybe we could use e.g. bit 1 in queue_enable to signal that?
> >
> 
> I'm totally ok to go in that direction.
> 
> Thanks!
> 
> >
> > > >
> > > >
> > > > > So in this case if RX queues are enabled at the same time, the device
> > > > > may try to queue packets to queue 0.
> > > > >
> > > > > Thanks
> > > > >
> > > > > > How do you do this - they are DMA from same VF no?
> > > > > >
> > > > > >
> > > > > > > >
> > > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > > introduces latencies.
> > > > > > > > > > >
> > > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > > dataplane vqs?
> > > > > > > > > >
> > > > > > > > > > write to what?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > > >
> > > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > 2.31.1
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > >
> > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > before posting.
> > > > > > > > > >
> > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Xuan Zhuo]

On Thu, 8 Jun 2023 16:18:23 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> On Thu, Jun 8, 2023 at 2:24 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 6 Jun 2023 15:19:53 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Tue, Jun 06, 2023 at 06:04:32PM +0000, Parav Pandit wrote:
> > > > Hi Eugenio,
> > > >
> > > > > From: Eugenio P閞ez <eperezma@redhat.com>
> > > > > Sent: Tuesday, June 6, 2023 1:55 PM
> > > > >
> > > > > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > > > > some queues enabled, and then enable another queues later.
> > > > >
> > > > > This is the current way to migrate net device state through control virtqueue, in
> > > > > a software assisted framework with vDPA:
> > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > * All the control commands (mac address, mq, etc) needed for the device to
> > > > > behave the same as the source of migration are sent
> > > > > * Finally all the dataplane queues are enabled.
> > > > >
> > > > > Signed-off-by: Eugenio P閞ez <eperezma@redhat.com>
> > > > > ---
> > > > >  content.tex | 15 +++++++++++++--
> > > > >  1 file changed, 13 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/content.tex b/content.tex
> > > > > index d2ab9eb..ea4a6d8 100644
> > > > > --- a/content.tex
> > > > > +++ b/content.tex
> > > > > @@ -99,10 +99,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a
> > > > > Virtio Device / Feature B  \begin{description}
> > > > >  \item[0 to 23, and 50 to 127] Feature bits for the specific device type
> > > > >
> > > > > -\item[24 to 41] Feature bits reserved for extensions to the queue and
> > > > > +\item[24 to 42] Feature bits reserved for extensions to the queue and
> > > > >    feature negotiation mechanisms
> > > > >
> > > > > -\item[42 to 49, and 128 and above] Feature bits reserved for future extensions.
> > > > > +\item[43 to 49, and 128 and above] Feature bits reserved for future
> > > > > extensions.
> > > > >  \end{description}
> > > > >
> > > > >  \begin{note}
> > > > > @@ -398,6 +398,13 @@ \section{Virtqueues}\label{sec:Basic Facilities of a
> > > > > Virtio Device / Virtqueues}  Every driver and device supports either the Packed
> > > > > or the Split  Virtqueue format, or both.
> > > > >
> > > > > +\subsection{Virtqueue selective enabling}\label{sec:Basic Facilities of
> > > > > +a Virtio Device / Virtqueues / Selective Virtqueue Enable}
> > > > > +
> > > > > +If the driver negotiates VIRTIO\_F\_RING\_ENABLE\_ANYTIME, it can
> > > > > +enable individual virtqueues both before and after the DRIVER\_OK. The
> > > > > +way to enable a virtqueue is transport specific, but it mimics the
> > > > > +re-enabling after an hypothetical reset.
> > > > > +
> > > > >  \subsection{Virtqueue Reset}\label{sec:Basic Facilities of a Virtio Device /
> > > > > Virtqueues / Virtqueue Reset}
> > > > >
> > > > >  When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > > > > @@ -870,6 +877,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> > > > > Feature Bits}
> > > > >   \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> > > > >   handling features reserved for future use.
> > > > >
> > > > > +  \item[VIRTIO_F_RING_ENABLE_ANYTIME(42)] This feature indicates  that
> > > > > + the driver can enable a queue both before and after DRIVER\_OK.
> > > > > +  See \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Selective
> > > > > Virtqueue Enable}.
> > > > > +
> > > >
> > > > Can you please highlight, current spec normative which indicates that
> > > > all queues must be enabled before DRIVER_OK and more queues cannot be
> > > > enabled after DRIVER_OK?
> > >
> > >
> > > This one:
> > > \drivernormative{\subsection}{Device Initialization}{General Initialization And Device Operation / Device Initialization}
> > >
> > > it does not say "all queues" if you want to split hairs but I think
> > > assuming it's safe to enable queues later is dangerous.
> >
> > Although there is no "all queues" here, if the queue is not ready, the device
> > may use all the queues after "Device OK". So I also think it might be dangerous.
> >
> > But I also ran into this problem. We want the device to support 100 queues, but
> > we also want to use only 16 queues by default. The user can then manually scale
> > up to 100, if the user has a large memory.
> >
> > So I think, the point is that "num_queues" is the max and default value. Can we
> > change "num_queues" to default, then we introduced a new value as max.
> >
> > Does this way solve your problem? @Eugenio
> >
>
> It would need a tweak, but it gives me an idea.
>
> I think you're talking about what the device offers, isn't it?


Yes, I want the driver offers a default-num-queues and a max-num-queues.

Thanks


> I would
> need to modify it from the driver.
>
> But it has an interesting question, how to allocate "less vrings" than
> virtio_pci_common_cfg.num_queues? I guess it is ok to set all vq
> addresses to a vring that never increments avail_idx? The driver still
> needs to allocate memory for that one and lose start time configuring
> them.
>
> Maybe for net is valid to leave them unconfigured? it should only use
> one vq pair and CVQ by default.
>
> My proposal could be done at the virtio-net level, allowing
> max_virtqueue_pairs to be 0 from the beginning. For example, allowing
> the driver to write it before DRIVER_OK. In that case, no data vq is
> enabled and only CVQ process requests after DRIVER_OK. After all the
> config is written, the driver changes the number of active queues with
> VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET.
>
> I think it is better solved in a generic way, but I'm ok with this
> -net specific too.
>
> Thanks!
>
> > Thanks.
> >
> >
> >
> > >
> > > E.g. vhost user backends tend to assume no new queues appear
> > > after DRIVER_OK.
> > >
> > > --
> > > MST
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Fri, Jun 9, 2023 at 12:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 08, 2023 at 10:36:19AM +0200, Eugenio Perez Martin wrote:
> > On Thu, Jun 8, 2023 at 9:19 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > > > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > > > no need for spec changes.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > > > already has buffers there.
> > > > > > > > > > >
> > > > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > > > device can process rx descriptors.
> > > > > > > > >
> > > > > > > > > Not if there are no descriptors there.
> > > > > > > > >
> > > > > > > >
> > > > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > > > that. The guest will likely have rx descriptors available.
> > > > > > >
> > > > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > > > rx queues?
> > > > > >
> > > > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > > > >
> > > > > So cvq maps to hypervisor memory?
> > > > >
> > > >
> > > > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > > > in the guest's one. That allows the hypervisor to send the CVQ
> > > > commands to restore the device state in the destination, without the
> > > > guest intervention.
> > > >
> > > > Data vqs, on the other hand, are passthrough. The device talks
> > > > directly to the guest's vrings.
> > > >
> > > > Currently, this is done by emulating CVQ in the host's kernel and then
> > > > translating the commands in the way that suits better the vdpa device,
> > > > using its vendor vdpa driver in the host. Other methods like PASID are
> > > > possible too.
> > > >
> > > > Thanks!
> > >
> > > OK. So my suggestion is simple: map data vrings to a zero page in
> > > hypervisor memory initially. Later reset and map to guest.
> > >
> >
> > The idea is interesting, but we lose the net configuration in a device
> > reset, so we need to send it again.
>
> Ring reset, not device reset.
>
> > In the case of qemu+vDPA maybe it is possible with queue_reset, like:
> > * Map all the guest pages as usual.
> > * Map a new zero page, forbid the guest to write on that page. Even in
> > vIOMMU case, we can send all the CVQ commands before allowing the
> > guest to modify mappings. The guest has no way to write on that page
> > through the device as, well, dataplane is not initialized. It's the
> > way DPDK shadow virtqueues worked, so it should be valid.
> > * Reset the queues to the guest passthrough.
> >
> > I don't like the complexity of it but I like it does require even less
> > changes to the device / spec.
>
> This is exactly what I meant.
>

Ok, sorry for the misunderstanding.

Another drawback of that is the long time a device can spend to
reconfigure its memory maps. taking that into account, does it make
sense to send a new version with the bit 1 on enable?

Thanks!

> > >
> > > If that does not work, then I am not sure this proposal is enough
> > > since I think devices want to have a specific point in time
> > > where they know which queues are going to be used.
> >
> > In the case of net, this should not be a problem since the spec
> > mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if
> > cvq and mq. virtio-blk also has num_queues.
>
> This is max, not how many there are in practice.
>
> > Is it even valid to not enable some of the queues?
>
> Yes and linux will do that if max_virtqueue_pairs > #CPUs.
>
> > I've always felt that queue_enable has been redundant before
> > queue_reset for this reason actually.
> >
> > > Maybe we could use e.g. bit 1 in queue_enable to signal that?
> > >
> >
> > I'm totally ok to go in that direction.
> >
> > Thanks!
> >
> > >
> > > > >
> > > > >
> > > > > > So in this case if RX queues are enabled at the same time, the device
> > > > > > may try to queue packets to queue 0.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > > How do you do this - they are DMA from same VF no?
> > > > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > > > introduces latencies.
> > > > > > > > > > > >
> > > > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > > > dataplane vqs?
> > > > > > > > > > >
> > > > > > > > > > > write to what?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > 2.31.1
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > >
> > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > before posting.
> > > > > > > > > > >
> > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Fri, Jun 09, 2023 at 12:27:58PM +0200, Eugenio Perez Martin wrote:
> On Fri, Jun 9, 2023 at 12:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Jun 08, 2023 at 10:36:19AM +0200, Eugenio Perez Martin wrote:
> > > On Thu, Jun 8, 2023 at 9:19 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > > > > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > > > > no need for spec changes.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > > > > already has buffers there.
> > > > > > > > > > > >
> > > > > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > > > > device can process rx descriptors.
> > > > > > > > > >
> > > > > > > > > > Not if there are no descriptors there.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > > > > that. The guest will likely have rx descriptors available.
> > > > > > > >
> > > > > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > > > > rx queues?
> > > > > > >
> > > > > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > > > > >
> > > > > > So cvq maps to hypervisor memory?
> > > > > >
> > > > >
> > > > > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > > > > in the guest's one. That allows the hypervisor to send the CVQ
> > > > > commands to restore the device state in the destination, without the
> > > > > guest intervention.
> > > > >
> > > > > Data vqs, on the other hand, are passthrough. The device talks
> > > > > directly to the guest's vrings.
> > > > >
> > > > > Currently, this is done by emulating CVQ in the host's kernel and then
> > > > > translating the commands in the way that suits better the vdpa device,
> > > > > using its vendor vdpa driver in the host. Other methods like PASID are
> > > > > possible too.
> > > > >
> > > > > Thanks!
> > > >
> > > > OK. So my suggestion is simple: map data vrings to a zero page in
> > > > hypervisor memory initially. Later reset and map to guest.
> > > >
> > >
> > > The idea is interesting, but we lose the net configuration in a device
> > > reset, so we need to send it again.
> >
> > Ring reset, not device reset.
> >
> > > In the case of qemu+vDPA maybe it is possible with queue_reset, like:
> > > * Map all the guest pages as usual.
> > > * Map a new zero page, forbid the guest to write on that page. Even in
> > > vIOMMU case, we can send all the CVQ commands before allowing the
> > > guest to modify mappings. The guest has no way to write on that page
> > > through the device as, well, dataplane is not initialized. It's the
> > > way DPDK shadow virtqueues worked, so it should be valid.
> > > * Reset the queues to the guest passthrough.
> > >
> > > I don't like the complexity of it but I like it does require even less
> > > changes to the device / spec.
> >
> > This is exactly what I meant.
> >
> 
> Ok, sorry for the misunderstanding.
> 
> Another drawback of that is the long time a device can spend to
> reconfigure its memory maps.

The map is already used for cvq though, I don't see why it would take
much more with this.

> taking that into account, does it make
> sense to send a new version with the bit 1 on enable?
> 
> Thanks!

Current hardware won't support the extra bit though.

> > > >
> > > > If that does not work, then I am not sure this proposal is enough
> > > > since I think devices want to have a specific point in time
> > > > where they know which queues are going to be used.
> > >
> > > In the case of net, this should not be a problem since the spec
> > > mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if
> > > cvq and mq. virtio-blk also has num_queues.
> >
> > This is max, not how many there are in practice.
> >
> > > Is it even valid to not enable some of the queues?
> >
> > Yes and linux will do that if max_virtqueue_pairs > #CPUs.
> >
> > > I've always felt that queue_enable has been redundant before
> > > queue_reset for this reason actually.
> > >
> > > > Maybe we could use e.g. bit 1 in queue_enable to signal that?
> > > >
> > >
> > > I'm totally ok to go in that direction.
> > >
> > > Thanks!
> > >
> > > >
> > > > > >
> > > > > >
> > > > > > > So in this case if RX queues are enabled at the same time, the device
> > > > > > > may try to queue packets to queue 0.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > > How do you do this - they are DMA from same VF no?
> > > > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > > > > introduces latencies.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > > > > dataplane vqs?
> > > > > > > > > > > >
> > > > > > > > > > > > write to what?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > 2.31.1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > > >
> > > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > > before posting.
> > > > > > > > > > > >
> > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Fri, Jun 9, 2023 at 5:54 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Fri, Jun 09, 2023 at 12:27:58PM +0200, Eugenio Perez Martin wrote:
> > On Fri, Jun 9, 2023 at 12:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Jun 08, 2023 at 10:36:19AM +0200, Eugenio Perez Martin wrote:
> > > > On Thu, Jun 8, 2023 at 9:19 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Thu, Jun 08, 2023 at 08:43:18AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Thu, Jun 8, 2023 at 8:04 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Thu, Jun 08, 2023 at 08:44:41AM +0800, Jason Wang wrote:
> > > > > > > > On Thu, Jun 8, 2023 at 4:27 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Wed, Jun 07, 2023 at 11:41:39AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > On Wed, Jun 7, 2023 at 10:59 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jun 07, 2023 at 10:47:12AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > On Wed, Jun 7, 2023 at 10:23 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, 7 Jun 2023 07:35:58 +0200, Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > > > > > > > > > > > > > On Tue, Jun 6, 2023 at 9:10 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Jun 06, 2023 at 07:55:09PM +0200, Eugenio Pérez wrote:
> > > > > > > > > > > > > > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > > > > > > > > > > > > > some queues enabled, and then enable another queues later.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This is the current way to migrate net device state through control
> > > > > > > > > > > > > > > > virtqueue, in a software assisted framework with vDPA:
> > > > > > > > > > > > > > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > > > > > > > > > > > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > > > > > > > > > > > > > to behave the same as the source of migration are sent
> > > > > > > > > > > > > > > > * Finally all the dataplane queues are enabled.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In my opinion, this is somewhat problematic. Specifically, currently
> > > > > > > > > > > > > > > devices tend to deduce how many queues are needed by looking
> > > > > > > > > > > > > > > at the state at DRIVER_OK time.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Question: what is wrong with enabling queues initially and then
> > > > > > > > > > > > > > > doing a reset right after DRIVER_OK? You can even allocate
> > > > > > > > > > > > > > > memory for just one queue (zeroing it out).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Granted this looks kind of ugly but side-steps this problem with
> > > > > > > > > > > > > > > no need for spec changes.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The problem is that the rx queues can start receiving, as the guest
> > > > > > > > > > > > > > already has buffers there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Can we reset the vq before filling buffers to it?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > They are passthrough from the guest so there is a window where the
> > > > > > > > > > > > device can process rx descriptors.
> > > > > > > > > > >
> > > > > > > > > > > Not if there are no descriptors there.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > But the migration is driven by the hypervisor, and it cannot control
> > > > > > > > > > that. The guest will likely have rx descriptors available.
> > > > > > > > >
> > > > > > > > > Maybe I misunderstand. Is hypervisor driving cvq while guest is driving
> > > > > > > > > rx queues?
> > > > > > > >
> > > > > > > > No, hypervisor tries to restore virtqueue states via cvq before guest can drive.
> > > > > > >
> > > > > > > So cvq maps to hypervisor memory?
> > > > > > >
> > > > > >
> > > > > > >From the device POV, yes, the CVQ vring is in hypervisor memory, not
> > > > > > in the guest's one. That allows the hypervisor to send the CVQ
> > > > > > commands to restore the device state in the destination, without the
> > > > > > guest intervention.
> > > > > >
> > > > > > Data vqs, on the other hand, are passthrough. The device talks
> > > > > > directly to the guest's vrings.
> > > > > >
> > > > > > Currently, this is done by emulating CVQ in the host's kernel and then
> > > > > > translating the commands in the way that suits better the vdpa device,
> > > > > > using its vendor vdpa driver in the host. Other methods like PASID are
> > > > > > possible too.
> > > > > >
> > > > > > Thanks!
> > > > >
> > > > > OK. So my suggestion is simple: map data vrings to a zero page in
> > > > > hypervisor memory initially. Later reset and map to guest.
> > > > >
> > > >
> > > > The idea is interesting, but we lose the net configuration in a device
> > > > reset, so we need to send it again.
> > >
> > > Ring reset, not device reset.
> > >
> > > > In the case of qemu+vDPA maybe it is possible with queue_reset, like:
> > > > * Map all the guest pages as usual.
> > > > * Map a new zero page, forbid the guest to write on that page. Even in
> > > > vIOMMU case, we can send all the CVQ commands before allowing the
> > > > guest to modify mappings. The guest has no way to write on that page
> > > > through the device as, well, dataplane is not initialized. It's the
> > > > way DPDK shadow virtqueues worked, so it should be valid.
> > > > * Reset the queues to the guest passthrough.
> > > >
> > > > I don't like the complexity of it but I like it does require even less
> > > > changes to the device / spec.
> > >
> > > This is exactly what I meant.
> > >
> >
> > Ok, sorry for the misunderstanding.
> >
> > Another drawback of that is the long time a device can spend to
> > reconfigure its memory maps.
>
> The map is already used for cvq though, I don't see why it would take
> much more with this.
>

In the case of vDPA HW, CVQ maps may not need to reach the device's or
platform IOMMU, so the modification of that mapping is way cheaper.

> > taking that into account, does it make
> > sense to send a new version with the bit 1 on enable?
> >
> > Thanks!
>
> Current hardware won't support the extra bit though.
>
> > > > >
> > > > > If that does not work, then I am not sure this proposal is enough
> > > > > since I think devices want to have a specific point in time
> > > > > where they know which queues are going to be used.
> > > >
> > > > In the case of net, this should not be a problem since the spec
> > > > mandates 2 if !cvq, 3 if cvq but !mq, and max_virtqueue_pairs*2+1 if
> > > > cvq and mq. virtio-blk also has num_queues.
> > >
> > > This is max, not how many there are in practice.
> > >
> > > > Is it even valid to not enable some of the queues?
> > >
> > > Yes and linux will do that if max_virtqueue_pairs > #CPUs.
> > >
> > > > I've always felt that queue_enable has been redundant before
> > > > queue_reset for this reason actually.
> > > >
> > > > > Maybe we could use e.g. bit 1 in queue_enable to signal that?
> > > > >
> > > >
> > > > I'm totally ok to go in that direction.
> > > >
> > > > Thanks!
> > > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > So in this case if RX queues are enabled at the same time, the device
> > > > > > > > may try to queue packets to queue 0.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > > How do you do this - they are DMA from same VF no?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > > > Apart from that, the back and forth
> > > > > > > > > > > > > > introduces latencies.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Maybe a better angle is to start all the queues as if they're reset,
> > > > > > > > > > > > > > write 1 just to CVQ, configure the device, and then write 1 to all
> > > > > > > > > > > > > > dataplane vqs?
> > > > > > > > > > > > >
> > > > > > > > > > > > > write to what?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Sorry I was unclear, I mean to enable the vqs writing 1 to queue_enable.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Eugenio Pérez (2):
> > > > > > > > > > > > > > > >   virtio: introduce selective queue enabling
> > > > > > > > > > > > > > > >   virtio: pci support virtqueue selective enabling
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >  content.tex       | 15 +++++++++++++--
> > > > > > > > > > > > > > > >  transport-pci.tex |  4 ++++
> > > > > > > > > > > > > > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > 2.31.1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > > > before posting.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > Another drawback of that is the long time a device can spend to
> > > reconfigure its memory maps.
> >
> > The map is already used for cvq though, I don't see why it would take
> > much more with this.
> >
> 
> In the case of vDPA HW, CVQ maps may not need to reach the device's or
> platform IOMMU, so the modification of that mapping is way cheaper.

Oh, I get it. It's setup time, but you are right.

> > > taking that into account, does it make
> > > sense to send a new version with the bit 1 on enable?
> > >
> > > Thanks!
> >
> > Current hardware won't support the extra bit though.

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 06, 2023 at 07:55:10PM +0200, Eugenio Pérez wrote:
> This patch allows the driver to start the device (as set DRIVER_OK) with only
> some queues enabled, and then enable another queues later.
> 
> This is the current way to migrate net device state through control
> virtqueue, in a software assisted framework with vDPA:
> * First, only net CVQ is enabled at DRIVER_OK
> * All the control commands (mac address, mq, etc) needed for the device
> to behave the same as the source of migration are sent
> * Finally all the dataplane queues are enabled.
> 
> Signed-off-by: Eugenio Pérez <eperezma@redhat.com>


I realized I never asked. *Why* enable CVQ so early?
I think it's so that commands take effect before any
buffers are processed? So maybe it's a net specific
issue?

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > Another drawback of that is the long time a device can spend to
> > > > reconfigure its memory maps.
> > >
> > > The map is already used for cvq though, I don't see why it would take
> > > much more with this.
> > >
> > 
> > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > platform IOMMU, so the modification of that mapping is way cheaper.
> 
> Oh, I get it. It's setup time, but you are right.
> 
> > > > taking that into account, does it make
> > > > sense to send a new version with the bit 1 on enable?
> > > >
> > > > Thanks!
> > >
> > > Current hardware won't support the extra bit though.


Thinking more about this, I think the following works:

- enable all queues before DRIVER_OK
- reset all data queues before DRIVER_OK
- DRIVER_OK
- CVQ commands
- re-enable data queues


No?



> -- 
> MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[virtio-comment] Re: [PATCH 1/2] virtio: introduce selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 13, 2023 at 9:50 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 06, 2023 at 07:55:10PM +0200, Eugenio Pérez wrote:
> > This patch allows the driver to start the device (as set DRIVER_OK) with only
> > some queues enabled, and then enable another queues later.
> >
> > This is the current way to migrate net device state through control
> > virtqueue, in a software assisted framework with vDPA:
> > * First, only net CVQ is enabled at DRIVER_OK
> > * All the control commands (mac address, mq, etc) needed for the device
> > to behave the same as the source of migration are sent
> > * Finally all the dataplane queues are enabled.
> >
> > Signed-off-by: Eugenio Pérez <eperezma@redhat.com>
>
>
> I realized I never asked. *Why* enable CVQ so early?
> I think it's so that commands take effect before any
> buffers are processed? So maybe it's a net specific
> issue?
>

Yes, that's the reason. Can I add something to the patch message to
make it clearer?

It can be solved using a net specific solution for sure, but I think
we can make it general if other devices with CVQ have such needs.

Thanks.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > Another drawback of that is the long time a device can spend to
> > > > > reconfigure its memory maps.
> > > >
> > > > The map is already used for cvq though, I don't see why it would take
> > > > much more with this.
> > > >
> > >
> > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > platform IOMMU, so the modification of that mapping is way cheaper.
> >
> > Oh, I get it. It's setup time, but you are right.
> >
> > > > > taking that into account, does it make
> > > > > sense to send a new version with the bit 1 on enable?
> > > > >
> > > > > Thanks!
> > > >
> > > > Current hardware won't support the extra bit though.
>
>
> Thinking more about this, I think the following works:
>
> - enable all queues before DRIVER_OK
> - reset all data queues before DRIVER_OK
> - DRIVER_OK
> - CVQ commands
> - re-enable data queues
>
>
> No?
>

If it is valid to write to queue_reset before DRIVER_OK, then sure. I
asked for that in the past at
https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
.

>
>
> > --
> > MST
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > Another drawback of that is the long time a device can spend to
> > > > > > reconfigure its memory maps.
> > > > >
> > > > > The map is already used for cvq though, I don't see why it would take
> > > > > much more with this.
> > > > >
> > > >
> > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > >
> > > Oh, I get it. It's setup time, but you are right.
> > >
> > > > > > taking that into account, does it make
> > > > > > sense to send a new version with the bit 1 on enable?
> > > > > >
> > > > > > Thanks!
> > > > >
> > > > > Current hardware won't support the extra bit though.
> >
> >
> > Thinking more about this, I think the following works:
> >
> > - enable all queues before DRIVER_OK
> > - reset all data queues before DRIVER_OK
> > - DRIVER_OK
> > - CVQ commands
> > - re-enable data queues
> >
> >
> > No?
> >
> 
> If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> asked for that in the past at
> https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> .

Well then Jason said:
 we should make svq work without RING_RESET
but now apparently we went full circle and propose
a feature bit after all?


> >
> >
> > > --
> > > MST
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Eugenio Perez Martin <eperezma@redhat.com>
> Sent: Tuesday, June 13, 2023 6:13 AM

> > - enable all queues before DRIVER_OK
> > - reset all data queues before DRIVER_OK
> > - DRIVER_OK
> > - CVQ commands
> > - re-enable data queues
> >
> >
> > No?
> >
> 
> If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for
> that in the past at https://lists.oasis-open.org/archives/virtio-
> dev/202305/msg00329.html
> .

Before DRIVER_OK stage, queue_enable is already available so why to use queue reset?
They have duplicate functionality, hence there is no motivation to support queue_reset before DRIVER_OK stage?

It is not explicitly mentioned in the spec that one can setup the queue using queue reset instead of queue enable.
As Jason mentioned it is implementation specific, one device supports it and one doesn't.
Hence, it will break on those devices which doesnt support it.

Therefore, I would like to add it to the spec.
"queue_reset register MUST be accessed by the driver only after device has reached the DRIVER_OK stage."

> 
> >
> >
> > > --
> > > MST
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and to
> > minimize spam in the list archive, subscription is required before
> > posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License:
> > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines:
> > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 13, 2023 at 07:00:02PM +0000, Parav Pandit wrote:
> 
> > From: Eugenio Perez Martin <eperezma@redhat.com>
> > Sent: Tuesday, June 13, 2023 6:13 AM
> 
> > > - enable all queues before DRIVER_OK
> > > - reset all data queues before DRIVER_OK
> > > - DRIVER_OK
> > > - CVQ commands
> > > - re-enable data queues
> > >
> > >
> > > No?
> > >
> > 
> > If it is valid to write to queue_reset before DRIVER_OK, then sure. I asked for
> > that in the past at https://lists.oasis-open.org/archives/virtio-
> > dev/202305/msg00329.html
> > .
> 
> Before DRIVER_OK stage, queue_enable is already available so why to use queue reset?
> They have duplicate functionality, hence there is no motivation to support queue_reset before DRIVER_OK stage?

I do not get how.  Let's make sure we are talking about the same thing.
This is what I am saying:

Start operating:

- program queue address, size
- queue_enable

Now we change our mind about queue size or address. So:

- queue_reset
- program another queue address, size

Finally we are decided, let's start operating:

- DRIVER_OK


I don't see how it's possible with just queue_enable.


> 
> It is not explicitly mentioned in the spec that one can setup the queue using queue reset instead of queue enable.
> As Jason mentioned it is implementation specific, one device supports it and one doesn't.
> Hence, it will break on those devices which doesnt support it.
> 

Setup? No, and spec explicitly says to setup one has to use
queue_enable. But this is not what we discussed with Stefano here.

> Therefore, I would like to add it to the spec.
> "queue_reset register MUST be accessed by the driver only after device has reached the DRIVER_OK stage."

I don't think we can add MUST requirements after the proposal is
in the released spec. If device has stricter requirements than the
spec then it's not compliant.





> > 
> > >
> > >
> > > > --
> > > > MST
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and to
> > > minimize spam in the list archive, subscription is required before
> > > posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License:
> > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines:
> > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> 


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, June 13, 2023 3:55 PM
> 
> I do not get how.  Let's make sure we are talking about the same thing.
> This is what I am saying:
> 
> Start operating:
> 
> - program queue address, size
> - queue_enable
> 
> Now we change our mind about queue size or address. So:
> 
> - queue_reset

Why queue_reset is needed here?
I don't see spec saying that queue_enable must be done only once for a given Q before DRIVER_OK.

> - program another queue address, size
> 
> Finally we are decided, let's start operating:
> 
> - DRIVER_OK
> 
> 
> I don't see how it's possible with just queue_enable.
> 
> 
> >
> > It is not explicitly mentioned in the spec that one can setup the queue using
> queue reset instead of queue enable.
> > As Jason mentioned it is implementation specific, one device supports it and
> one doesn't.
> > Hence, it will break on those devices which doesnt support it.
> >
> 
> Setup? No, and spec explicitly says to setup one has to use queue_enable. But
> this is not what we discussed with Stefano here.
> 
The above sequence you described is not well documented.

> > Therefore, I would like to add it to the spec.
> > "queue_reset register MUST be accessed by the driver only after device has
> reached the DRIVER_OK stage."
> 
> I don't think we can add MUST requirements after the proposal is in the
> released spec. If device has stricter requirements than the spec then it's not
> compliant.
>
Lets resolve above question first about how many times one can use queue_enable before DRIVER_OK for a specific VQ.
Without this either way driver and device combination is broken when queue_reset is used before DRIVER_OK.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Parav Pandit
> Sent: Tuesday, June 13, 2023 5:09 PM

> Why queue_reset is needed here?
> I don't see spec saying that queue_enable must be done only once for a given Q
> before DRIVER_OK.

I remember now and I found the wording which was the source of adding queue_reset register.

Spec: "The driver MUST NOT write a 0 to queue_enable."

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Tue, Jun 13, 2023 at 09:09:05PM +0000, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, June 13, 2023 3:55 PM
> > 
> > I do not get how.  Let's make sure we are talking about the same thing.
> > This is what I am saying:
> > 
> > Start operating:
> > 
> > - program queue address, size
> > - queue_enable
> > 
> > Now we change our mind about queue size or address. So:
> > 
> > - queue_reset
> 
> Why queue_reset is needed here?
> I don't see spec saying that queue_enable must be done only once for a given Q before DRIVER_OK.

I see you found it.

> > - program another queue address, size
> > 
> > Finally we are decided, let's start operating:
> > 
> > - DRIVER_OK
> > 
> > 
> > I don't see how it's possible with just queue_enable.
> > 
> > 
> > >
> > > It is not explicitly mentioned in the spec that one can setup the queue using
> > queue reset instead of queue enable.
> > > As Jason mentioned it is implementation specific, one device supports it and
> > one doesn't.
> > > Hence, it will break on those devices which doesnt support it.
> > >
> > 
> > Setup? No, and spec explicitly says to setup one has to use queue_enable. But
> > this is not what we discussed with Stefano here.
> > 
> The above sequence you described is not well documented.

It's the standard sequence for queue reset, is it not?

For example:

	Virtqueue reset is divided into two parts. The driver first resets a queue and
	can afterwards optionally re-enable it.


it seems clear that queue has to be enabled before being reset.



> > > Therefore, I would like to add it to the spec.
> > > "queue_reset register MUST be accessed by the driver only after device has
> > reached the DRIVER_OK stage."
> > 
> > I don't think we can add MUST requirements after the proposal is in the
> > released spec. If device has stricter requirements than the spec then it's not
> > compliant.
> >
> Lets resolve above question first about how many times one can use queue_enable before DRIVER_OK for a specific VQ.
> Without this either way driver and device combination is broken when queue_reset is used before DRIVER_OK.

I'm not sure what's broken.


If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to
\field{queue_reset} to reset the queue, the driver MUST NOT consider queue
reset to be complete until it reads back 0 in \field{queue_reset}. The driver
MAY re-enable the queue by writing 1 to \field{queue_enable} after ensuring
that other virtqueue fields have been set up correctly. The driver MAY set
driver-writeable queue configuration values to different values than those that
were used before the queue reset.


So the limitation is that it happens after FEATURES_OK.


-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, June 13, 2023 5:48 PM

> > Why queue_reset is needed here?
> > I don't see spec saying that queue_enable must be done only once for a given
> Q before DRIVER_OK.
> 
> I see you found it.
> 
> > > - program another queue address, size
> > >
> > > Finally we are decided, let's start operating:
> > >
> > > - DRIVER_OK
> > >
> > >
> > > I don't see how it's possible with just queue_enable.
> > >
> > >
> > > >
> > > > It is not explicitly mentioned in the spec that one can setup the
> > > > queue using
> > > queue reset instead of queue enable.
> > > > As Jason mentioned it is implementation specific, one device
> > > > supports it and
> > > one doesn't.
> > > > Hence, it will break on those devices which doesnt support it.
> > > >
> > >
> > > Setup? No, and spec explicitly says to setup one has to use
> > > queue_enable. But this is not what we discussed with Stefano here.
> > >
> > The above sequence you described is not well documented.
> 
> It's the standard sequence for queue reset, is it not?
> 
> For example:
> 
> 	Virtqueue reset is divided into two parts. The driver first resets a queue
> and
> 	can afterwards optionally re-enable it.
> 
> 
> it seems clear that queue has to be enabled before being reset.
>
Yes, it is clear.
 
> I'm not sure what's broken.
>
If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live",
Such device needs the fix.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Zhu, Lingshan]

On 6/14/2023 5:54 AM, Parav Pandit wrote:
>
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Tuesday, June 13, 2023 5:48 PM
>>> Why queue_reset is needed here?
>>> I don't see spec saying that queue_enable must be done only once for a given
>> Q before DRIVER_OK.
>>
>> I see you found it.
>>
>>>> - program another queue address, size
>>>>
>>>> Finally we are decided, let's start operating:
>>>>
>>>> - DRIVER_OK
>>>>
>>>>
>>>> I don't see how it's possible with just queue_enable.
>>>>
>>>>
>>>>> It is not explicitly mentioned in the spec that one can setup the
>>>>> queue using
>>>> queue reset instead of queue enable.
>>>>> As Jason mentioned it is implementation specific, one device
>>>>> supports it and
>>>> one doesn't.
>>>>> Hence, it will break on those devices which doesnt support it.
>>>>>
>>>> Setup? No, and spec explicitly says to setup one has to use
>>>> queue_enable. But this is not what we discussed with Stefano here.
>>>>
>>> The above sequence you described is not well documented.
>> It's the standard sequence for queue reset, is it not?
>>
>> For example:
>>
>> 	Virtqueue reset is divided into two parts. The driver first resets a queue
>> and
>> 	can afterwards optionally re-enable it.
>>
>>
>> it seems clear that queue has to be enabled before being reset.
>>
> Yes, it is clear.
>   
>> I'm not sure what's broken.
>>
> If a device has assumed that queue_reset must be done after DRIVER_OK stage because until that point the device is not "live",
> Such device needs the fix.
I think queue_reset just resets the queue and its fields including the 
queue_enable. DRIVER_OK is not a must for performing a queue_reset,
My understanding is:
- before DRIVER_OK, the driver can re-config the queue anytime, doesn't 
need a reset_queue
- after DRIVER_OK, the driver has to reset the queue to re-config the 
queue. Because a queue is considered alive when both DRIVER_OK && 
queue_enable,
and resetting a queue also set queue_enable to zero.

At the end of the migration, hypervisor should reset the queues and 
config the queues with the guest configurations and enable the queues, 
right?

Thanks


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Wednesday, June 14, 2023 12:27 AM

> > If a device has assumed that queue_reset must be done after DRIVER_OK
> > stage because until that point the device is not "live", Such device needs the
> fix.
> I think queue_reset just resets the queue and its fields including the
> queue_enable. DRIVER_OK is not a must for performing a queue_reset, My
> understanding is:
> - before DRIVER_OK, the driver can re-config the queue anytime, doesn't need
> a reset_queue

Before DRIVER_OK stage, to reconfigure a queue, driver needs to write queue_enable = 0.
This is not allowed in the spec.
So driver need to use queue_reset to reconfigure.

> - after DRIVER_OK, the driver has to reset the queue to re-config the queue.
> Because a queue is considered alive when both DRIVER_OK && queue_enable,
> and resetting a queue also set queue_enable to zero.
> 
> At the end of the migration, hypervisor should reset the queues and config the
> queues with the guest configurations and enable the queues, right?

The migration for non vdpa world would not require any of the queue enable/disable flow because device is migrated (paused) with its current state.
And it can resume from where it left off.

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Zhu, Lingshan]

On 6/14/2023 12:32 PM, Parav Pandit wrote:
>
>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>> Sent: Wednesday, June 14, 2023 12:27 AM
>>> If a device has assumed that queue_reset must be done after DRIVER_OK
>>> stage because until that point the device is not "live", Such device needs the
>> fix.
>> I think queue_reset just resets the queue and its fields including the
>> queue_enable. DRIVER_OK is not a must for performing a queue_reset, My
>> understanding is:
>> - before DRIVER_OK, the driver can re-config the queue anytime, doesn't need
>> a reset_queue
> Before DRIVER_OK stage, to reconfigure a queue, driver needs to write queue_enable = 0.
> This is not allowed in the spec.
> So driver need to use queue_reset to reconfigure.
The spec requires the driver config all other fields before set 
queue_enable.
I am not sure whether there are real occurring cases that re-config the vq
fields between queue_enable and DRIVER_OK in the initialization.

But I agree maybe we should allow set queue_enable = 0.

This reset may happen after DRIVER_OK.
>
>> - after DRIVER_OK, the driver has to reset the queue to re-config the queue.
>> Because a queue is considered alive when both DRIVER_OK && queue_enable,
>> and resetting a queue also set queue_enable to zero.
>>
>> At the end of the migration, hypervisor should reset the queues and config the
>> queues with the guest configurations and enable the queues, right?
> The migration for non vdpa world would not require any of the queue enable/disable flow because device is migrated (paused) with its current state.
> And it can resume from where it left off.
For non-vdpa cases, it still require to migrate the states like MQ which 
is controlled by cvq.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Wednesday, June 14, 2023 2:11 AM

> 
> On 6/14/2023 12:32 PM, Parav Pandit wrote:
> >
> >> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >> Sent: Wednesday, June 14, 2023 12:27 AM
> >>> If a device has assumed that queue_reset must be done after
> >>> DRIVER_OK stage because until that point the device is not "live",
> >>> Such device needs the
> >> fix.
> >> I think queue_reset just resets the queue and its fields including
> >> the queue_enable. DRIVER_OK is not a must for performing a
> >> queue_reset, My understanding is:
> >> - before DRIVER_OK, the driver can re-config the queue anytime,
> >> doesn't need a reset_queue
> > Before DRIVER_OK stage, to reconfigure a queue, driver needs to write
> queue_enable = 0.
> > This is not allowed in the spec.
> > So driver need to use queue_reset to reconfigure.
> The spec requires the driver config all other fields before set queue_enable.
> I am not sure whether there are real occurring cases that re-config the vq fields
> between queue_enable and DRIVER_OK in the initialization.
> 
Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.

> But I agree maybe we should allow set queue_enable = 0.
>
This functionality of queue_enable = 0, is supported using queue_reset register.
 
> This reset may happen after DRIVER_OK.

Yes, the queue_reset after DRIVER_OK is not ambiguous.
Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.

For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Zhu, Lingshan]

On 6/14/2023 7:56 PM, Parav Pandit wrote:
>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>> Sent: Wednesday, June 14, 2023 2:11 AM
>> On 6/14/2023 12:32 PM, Parav Pandit wrote:
>>>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>>>> Sent: Wednesday, June 14, 2023 12:27 AM
>>>>> If a device has assumed that queue_reset must be done after
>>>>> DRIVER_OK stage because until that point the device is not "live",
>>>>> Such device needs the
>>>> fix.
>>>> I think queue_reset just resets the queue and its fields including
>>>> the queue_enable. DRIVER_OK is not a must for performing a
>>>> queue_reset, My understanding is:
>>>> - before DRIVER_OK, the driver can re-config the queue anytime,
>>>> doesn't need a reset_queue
>>> Before DRIVER_OK stage, to reconfigure a queue, driver needs to write
>> queue_enable = 0.
>>> This is not allowed in the spec.
>>> So driver need to use queue_reset to reconfigure.
>> The spec requires the driver config all other fields before set queue_enable.
>> I am not sure whether there are real occurring cases that re-config the vq fields
>> between queue_enable and DRIVER_OK in the initialization.
>>
> Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.
>
>> But I agree maybe we should allow set queue_enable = 0.
>>
> This functionality of queue_enable = 0, is supported using queue_reset register.
>   
>> This reset may happen after DRIVER_OK.
> Yes, the queue_reset after DRIVER_OK is not ambiguous.
> Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.
>
> For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.
I guess the intention may be:
Step 1: the hypervisor owns all queues, it enables all queues and cvq, 
set queue fields with phony host values.
Then set DRIVER_OK, config the device and send commands through cvq.
But don't do link announce. So the device would not consume the 
descriptors and external network will not send
any data to it.

Step 2: Reset the queues, set the fields of the queues with guest 
config, selectively enable the queues,
so the device works for the guest.

Does this sounds good?




This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > reconfigure its memory maps.
> > > > > >
> > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > much more with this.
> > > > > >
> > > > >
> > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > >
> > > > Oh, I get it. It's setup time, but you are right.
> > > >
> > > > > > > taking that into account, does it make
> > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > >
> > > > > > > Thanks!
> > > > > >
> > > > > > Current hardware won't support the extra bit though.
> > >
> > >
> > > Thinking more about this, I think the following works:
> > >
> > > - enable all queues before DRIVER_OK
> > > - reset all data queues before DRIVER_OK
> > > - DRIVER_OK
> > > - CVQ commands
> > > - re-enable data queues
> > >
> > >
> > > No?
> > >
> >
> > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > asked for that in the past at
> > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > .
>
> Well then Jason said:
>  we should make svq work without RING_RESET
> but now apparently we went full circle and propose
> a feature bit after all?
>

My understanding is that RING_RESET covers more than just delaying the
enabling of net data vqs, like to change the address or properties of
vqs after DRIVER_OK. While the late enabling may be easy to implement
in current hw, vq property changes may be more challenging. There are
current HW that only support late enabling but not reconfigure the
queues. Ideally, just the subset of late enabling is needed.

Does it make sense to change the proposal to something in the line of:
RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
If this feature is negotiated, all vqs start as if the vqs has been
reset before DRIVER_OK. This feature does not allow resetting a queue
once it is live.

Thanks!

>
> > >
> > >
> > > > --
> > > > MST
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Thu, Jun 15, 2023 at 7:56 AM Zhu, Lingshan <lingshan.zhu@intel.com> wrote:
>
>
>
> On 6/14/2023 7:56 PM, Parav Pandit wrote:
> >> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >> Sent: Wednesday, June 14, 2023 2:11 AM
> >> On 6/14/2023 12:32 PM, Parav Pandit wrote:
> >>>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> >>>> Sent: Wednesday, June 14, 2023 12:27 AM
> >>>>> If a device has assumed that queue_reset must be done after
> >>>>> DRIVER_OK stage because until that point the device is not "live",
> >>>>> Such device needs the
> >>>> fix.
> >>>> I think queue_reset just resets the queue and its fields including
> >>>> the queue_enable. DRIVER_OK is not a must for performing a
> >>>> queue_reset, My understanding is:
> >>>> - before DRIVER_OK, the driver can re-config the queue anytime,
> >>>> doesn't need a reset_queue
> >>> Before DRIVER_OK stage, to reconfigure a queue, driver needs to write
> >> queue_enable = 0.
> >>> This is not allowed in the spec.
> >>> So driver need to use queue_reset to reconfigure.
> >> The spec requires the driver config all other fields before set queue_enable.
> >> I am not sure whether there are real occurring cases that re-config the vq fields
> >> between queue_enable and DRIVER_OK in the initialization.
> >>
> > Probably there is no case today, but may be it can occur where driver tries to configure something and if it fails, it may need to roll back and re-configure queues.
> >
> >> But I agree maybe we should allow set queue_enable = 0.
> >>
> > This functionality of queue_enable = 0, is supported using queue_reset register.
> >
> >> This reset may happen after DRIVER_OK.
> > Yes, the queue_reset after DRIVER_OK is not ambiguous.
> > Queue_reset before DRIVER_OK has the ambiguity, and will clarify that it should be supported by device before DRIVER_OK.
> >
> > For the main motivation of this patch, the _RESET bit seems enough to selectively enable the queue before or after DRIVER_OK.
> I guess the intention may be:
> Step 1: the hypervisor owns all queues, it enables all queues and cvq,
> set queue fields with phony host values.
> Then set DRIVER_OK, config the device and send commands through cvq.
> But don't do link announce. So the device would not consume the
> descriptors and external network will not send
> any data to it.
>

Even without a link announcement, the device can process descriptors.
Some examples are receiving broadcast packets or just sending any
packet (which effectively acts as a link announcement for the network
topology).

Thanks!

> Step 2: Reset the queues, set the fields of the queues with guest
> config, selectively enable the queues,
> so the device works for the guest.
>
> Does this sounds good?
>
>
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > reconfigure its memory maps.
> > > > > > >
> > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > much more with this.
> > > > > > >
> > > > > >
> > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > >
> > > > > Oh, I get it. It's setup time, but you are right.
> > > > >
> > > > > > > > taking that into account, does it make
> > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > >
> > > > > > > Current hardware won't support the extra bit though.
> > > >
> > > >
> > > > Thinking more about this, I think the following works:
> > > >
> > > > - enable all queues before DRIVER_OK
> > > > - reset all data queues before DRIVER_OK
> > > > - DRIVER_OK
> > > > - CVQ commands
> > > > - re-enable data queues
> > > >
> > > >
> > > > No?
> > > >
> > >
> > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > asked for that in the past at
> > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > .
> >
> > Well then Jason said:
> >  we should make svq work without RING_RESET
> > but now apparently we went full circle and propose
> > a feature bit after all?
> >
> 
> My understanding is that RING_RESET covers more than just delaying the
> enabling of net data vqs, like to change the address or properties of
> vqs after DRIVER_OK. While the late enabling may be easy to implement
> in current hw, vq property changes may be more challenging. There are
> current HW that only support late enabling but not reconfigure the
> queues. Ideally, just the subset of late enabling is needed.
> 
> Does it make sense to change the proposal to something in the line of:
> RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> If this feature is negotiated, all vqs start as if the vqs has been
> reset before DRIVER_OK. This feature does not allow resetting a queue
> once it is live.
> 
> Thanks!

Maybe. No idea - I am not a hardware designer. Can some hardware
vendors chime in and confirm?
It seems like a very very narrow "make my hack work" use-case.
And I'm not even sure how well it works.
For example what if cvq command enables rss? If the queues in
question are not enabled packets will be dropped.
So - fix some setups, break others?


> >
> > > >
> > > >
> > > > > --
> > > > > MST
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > reconfigure its memory maps.
> > > > > > > >
> > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > much more with this.
> > > > > > > >
> > > > > > >
> > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > >
> > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > >
> > > > > > > > > taking that into account, does it make
> > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Current hardware won't support the extra bit though.
> > > > >
> > > > >
> > > > > Thinking more about this, I think the following works:
> > > > >
> > > > > - enable all queues before DRIVER_OK
> > > > > - reset all data queues before DRIVER_OK
> > > > > - DRIVER_OK
> > > > > - CVQ commands
> > > > > - re-enable data queues
> > > > >
> > > > >
> > > > > No?
> > > > >
> > > >
> > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > asked for that in the past at
> > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > .
> > >
> > > Well then Jason said:
> > >  we should make svq work without RING_RESET
> > > but now apparently we went full circle and propose
> > > a feature bit after all?
> > >
> >
> > My understanding is that RING_RESET covers more than just delaying the
> > enabling of net data vqs, like to change the address or properties of
> > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > in current hw, vq property changes may be more challenging. There are
> > current HW that only support late enabling but not reconfigure the
> > queues. Ideally, just the subset of late enabling is needed.
> >
> > Does it make sense to change the proposal to something in the line of:
> > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > If this feature is negotiated, all vqs start as if the vqs has been
> > reset before DRIVER_OK. This feature does not allow resetting a queue
> > once it is live.
> >
> > Thanks!
>
> Maybe. No idea - I am not a hardware designer. Can some hardware
> vendors chime in and confirm?
> It seems like a very very narrow "make my hack work" use-case.
> And I'm not even sure how well it works.
> For example what if cvq command enables rss? If the queues in
> question are not enabled packets will be dropped.
> So - fix some setups, break others?
>

I don't think it breaks, they are already dropped before setting
DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
the future, the device is supposed to drop packets meanwhile anyway,
isn't it?

>
> > >
> > > > >
> > > > >
> > > > > > --
> > > > > > MST
> > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > reconfigure its memory maps.
> > > > > > > > >
> > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > much more with this.
> > > > > > > > >
> > > > > > > >
> > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > >
> > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > >
> > > > > > > > > > taking that into account, does it make
> > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Current hardware won't support the extra bit though.
> > > > > >
> > > > > >
> > > > > > Thinking more about this, I think the following works:
> > > > > >
> > > > > > - enable all queues before DRIVER_OK
> > > > > > - reset all data queues before DRIVER_OK
> > > > > > - DRIVER_OK
> > > > > > - CVQ commands
> > > > > > - re-enable data queues
> > > > > >
> > > > > >
> > > > > > No?
> > > > > >
> > > > >
> > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > asked for that in the past at
> > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > .
> > > >
> > > > Well then Jason said:
> > > >  we should make svq work without RING_RESET
> > > > but now apparently we went full circle and propose
> > > > a feature bit after all?
> > > >
> > >
> > > My understanding is that RING_RESET covers more than just delaying the
> > > enabling of net data vqs, like to change the address or properties of
> > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > in current hw, vq property changes may be more challenging. There are
> > > current HW that only support late enabling but not reconfigure the
> > > queues. Ideally, just the subset of late enabling is needed.
> > >
> > > Does it make sense to change the proposal to something in the line of:
> > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > If this feature is negotiated, all vqs start as if the vqs has been
> > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > once it is live.
> > >
> > > Thanks!
> >
> > Maybe. No idea - I am not a hardware designer. Can some hardware
> > vendors chime in and confirm?
> > It seems like a very very narrow "make my hack work" use-case.
> > And I'm not even sure how well it works.
> > For example what if cvq command enables rss? If the queues in
> > question are not enabled packets will be dropped.
> > So - fix some setups, break others?
> >
> 
> I don't think it breaks, they are already dropped before setting
> DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> the future, the device is supposed to drop packets meanwhile anyway,
> isn't it?


I just thought of an idea. Devices generally don't poke at rings
by themselves. How about at init time we cause kicks on data queues
to vmexit? Then on 1st kick we map hardware so following ones will
go straight to hardware. This will make sure cvq is processed
first. Hmm?

> >
> > > >
> > > > > >
> > > > > >
> > > > > > > --
> > > > > > > MST
> > > > > >
> > > > > >
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > >
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > >
> > > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > >
> > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > much more with this.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > >
> > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > >
> > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > >
> > > > > > >
> > > > > > > Thinking more about this, I think the following works:
> > > > > > >
> > > > > > > - enable all queues before DRIVER_OK
> > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > - DRIVER_OK
> > > > > > > - CVQ commands
> > > > > > > - re-enable data queues
> > > > > > >
> > > > > > >
> > > > > > > No?
> > > > > > >
> > > > > >
> > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > asked for that in the past at
> > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > .
> > > > >
> > > > > Well then Jason said:
> > > > >  we should make svq work without RING_RESET
> > > > > but now apparently we went full circle and propose
> > > > > a feature bit after all?
> > > > >
> > > >
> > > > My understanding is that RING_RESET covers more than just delaying the
> > > > enabling of net data vqs, like to change the address or properties of
> > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > in current hw, vq property changes may be more challenging. There are
> > > > current HW that only support late enabling but not reconfigure the
> > > > queues. Ideally, just the subset of late enabling is needed.
> > > >
> > > > Does it make sense to change the proposal to something in the line of:
> > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > once it is live.
> > > >
> > > > Thanks!
> > >
> > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > vendors chime in and confirm?
> > > It seems like a very very narrow "make my hack work" use-case.
> > > And I'm not even sure how well it works.
> > > For example what if cvq command enables rss? If the queues in
> > > question are not enabled packets will be dropped.
> > > So - fix some setups, break others?
> > >
> >
> > I don't think it breaks, they are already dropped before setting
> > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > the future, the device is supposed to drop packets meanwhile anyway,
> > isn't it?
>
>
> I just thought of an idea. Devices generally don't poke at rings
> by themselves. How about at init time we cause kicks on data queues
> to vmexit? Then on 1st kick we map hardware so following ones will
> go straight to hardware. This will make sure cvq is processed
> first. Hmm?
>

That would be possible, but nothing prevents the hw to start
processing requests even without that first kick, isn't it?

I think that was discussed also in the past, as previous qemu patches
gave a spurious kick to start queues just in case. I think the
conclusion was that at least the vDPA device will start processing
requests at DRIVER_OK, no need to wait for a kick.

https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/

Thanks!

> > >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > > --
> > > > > > > > MST
> > > > > > >
> > > > > > >
> > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > >
> > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > before posting.
> > > > > > >
> > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > >
> > > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > >
> > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > much more with this.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > >
> > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > >
> > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thinking more about this, I think the following works:
> > > > > > > >
> > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > - DRIVER_OK
> > > > > > > > - CVQ commands
> > > > > > > > - re-enable data queues
> > > > > > > >
> > > > > > > >
> > > > > > > > No?
> > > > > > > >
> > > > > > >
> > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > asked for that in the past at
> > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > .
> > > > > >
> > > > > > Well then Jason said:
> > > > > >  we should make svq work without RING_RESET
> > > > > > but now apparently we went full circle and propose
> > > > > > a feature bit after all?
> > > > > >
> > > > >
> > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > enabling of net data vqs, like to change the address or properties of
> > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > in current hw, vq property changes may be more challenging. There are
> > > > > current HW that only support late enabling but not reconfigure the
> > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > >
> > > > > Does it make sense to change the proposal to something in the line of:
> > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > once it is live.
> > > > >
> > > > > Thanks!
> > > >
> > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > vendors chime in and confirm?
> > > > It seems like a very very narrow "make my hack work" use-case.
> > > > And I'm not even sure how well it works.
> > > > For example what if cvq command enables rss? If the queues in
> > > > question are not enabled packets will be dropped.
> > > > So - fix some setups, break others?
> > > >
> > >
> > > I don't think it breaks, they are already dropped before setting
> > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > the future, the device is supposed to drop packets meanwhile anyway,
> > > isn't it?
> >
> >
> > I just thought of an idea. Devices generally don't poke at rings
> > by themselves. How about at init time we cause kicks on data queues
> > to vmexit? Then on 1st kick we map hardware so following ones will
> > go straight to hardware. This will make sure cvq is processed
> > first. Hmm?

This is not the case at least for vhost-net. It doesn't need to be
kicked in order to work. And it's an example that we need clearly
define the behaviour instead of depending on any existing
implementation details.

> >
>
> That would be possible, but nothing prevents the hw to start
> processing requests even without that first kick, isn't it?

Yes.

Thanks

>
> I think that was discussed also in the past, as previous qemu patches
> gave a spurious kick to start queues just in case. I think the
> conclusion was that at least the vDPA device will start processing
> requests at DRIVER_OK, no need to wait for a kick.
>
> https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
>
> Thanks!
>
> > > >
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > --
> > > > > > > > > MST
> > > > > > > >
> > > > > > > >
> > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > >
> > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > before posting.
> > > > > > > >
> > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > >
> > > > > >
> > > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Sun, Jun 25, 2023 at 01:31:36PM +0800, Jason Wang wrote:
> On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
> <eperezma@redhat.com> wrote:
> >
> > On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > > >
> > > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > > much more with this.
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > > >
> > > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > > >
> > > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thinking more about this, I think the following works:
> > > > > > > > >
> > > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > > - DRIVER_OK
> > > > > > > > > - CVQ commands
> > > > > > > > > - re-enable data queues
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > No?
> > > > > > > > >
> > > > > > > >
> > > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > > asked for that in the past at
> > > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > > .
> > > > > > >
> > > > > > > Well then Jason said:
> > > > > > >  we should make svq work without RING_RESET
> > > > > > > but now apparently we went full circle and propose
> > > > > > > a feature bit after all?
> > > > > > >
> > > > > >
> > > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > > enabling of net data vqs, like to change the address or properties of
> > > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > > in current hw, vq property changes may be more challenging. There are
> > > > > > current HW that only support late enabling but not reconfigure the
> > > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > > >
> > > > > > Does it make sense to change the proposal to something in the line of:
> > > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > > once it is live.
> > > > > >
> > > > > > Thanks!
> > > > >
> > > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > > vendors chime in and confirm?
> > > > > It seems like a very very narrow "make my hack work" use-case.
> > > > > And I'm not even sure how well it works.
> > > > > For example what if cvq command enables rss? If the queues in
> > > > > question are not enabled packets will be dropped.
> > > > > So - fix some setups, break others?
> > > > >
> > > >
> > > > I don't think it breaks, they are already dropped before setting
> > > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > > the future, the device is supposed to drop packets meanwhile anyway,
> > > > isn't it?
> > >
> > >
> > > I just thought of an idea. Devices generally don't poke at rings
> > > by themselves. How about at init time we cause kicks on data queues
> > > to vmexit? Then on 1st kick we map hardware so following ones will
> > > go straight to hardware. This will make sure cvq is processed
> > > first. Hmm?
> 
> This is not the case at least for vhost-net. It doesn't need to be
> kicked in order to work. And it's an example that we need clearly
> define the behaviour instead of depending on any existing
> implementation details.

This kind of guarantee however sounds generally useful to
control which queues are accessed when.

This is only needed after restore, which is ATM a vdpa thing,
so can just be a vdpa flag, but I'm ok with it as a
virtio flag too.

> > >
> >
> > That would be possible, but nothing prevents the hw to start
> > processing requests even without that first kick, isn't it?
> 
> Yes.
> 
> Thanks
> 
> >
> > I think that was discussed also in the past, as previous qemu patches
> > gave a spurious kick to start queues just in case. I think the
> > conclusion was that at least the vDPA device will start processing
> > requests at DRIVER_OK, no need to wait for a kick.
> >
> > https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
> >
> > Thanks!
> >
> > > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > MST
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > >
> > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > before posting.
> > > > > > > > >
> > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Mon, Jun 26, 2023 at 5:32 AM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Sun, Jun 25, 2023 at 01:31:36PM +0800, Jason Wang wrote:
> > On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
> > <eperezma@redhat.com> wrote:
> > >
> > > On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > > > much more with this.
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > > > >
> > > > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > > > >
> > > > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thinking more about this, I think the following works:
> > > > > > > > > >
> > > > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > > > - DRIVER_OK
> > > > > > > > > > - CVQ commands
> > > > > > > > > > - re-enable data queues
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > No?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > > > asked for that in the past at
> > > > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > > > .
> > > > > > > >
> > > > > > > > Well then Jason said:
> > > > > > > >  we should make svq work without RING_RESET
> > > > > > > > but now apparently we went full circle and propose
> > > > > > > > a feature bit after all?
> > > > > > > >
> > > > > > >
> > > > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > > > enabling of net data vqs, like to change the address or properties of
> > > > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > > > in current hw, vq property changes may be more challenging. There are
> > > > > > > current HW that only support late enabling but not reconfigure the
> > > > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > > > >
> > > > > > > Does it make sense to change the proposal to something in the line of:
> > > > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > > > once it is live.
> > > > > > >
> > > > > > > Thanks!
> > > > > >
> > > > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > > > vendors chime in and confirm?
> > > > > > It seems like a very very narrow "make my hack work" use-case.
> > > > > > And I'm not even sure how well it works.
> > > > > > For example what if cvq command enables rss? If the queues in
> > > > > > question are not enabled packets will be dropped.
> > > > > > So - fix some setups, break others?
> > > > > >
> > > > >
> > > > > I don't think it breaks, they are already dropped before setting
> > > > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > > > the future, the device is supposed to drop packets meanwhile anyway,
> > > > > isn't it?
> > > >
> > > >
> > > > I just thought of an idea. Devices generally don't poke at rings
> > > > by themselves. How about at init time we cause kicks on data queues
> > > > to vmexit? Then on 1st kick we map hardware so following ones will
> > > > go straight to hardware. This will make sure cvq is processed
> > > > first. Hmm?
> >
> > This is not the case at least for vhost-net. It doesn't need to be
> > kicked in order to work. And it's an example that we need clearly
> > define the behaviour instead of depending on any existing
> > implementation details.
>
> This kind of guarantee however sounds generally useful to
> control which queues are accessed when.
>
> This is only needed after restore, which is ATM a vdpa thing,
> so can just be a vdpa flag, but I'm ok with it as a
> virtio flag too.

Great. Having a virtio flag may help to reduce the extra and
duplicated effort spent in vdpa.

Thanks

>
> > > >
> > >
> > > That would be possible, but nothing prevents the hw to start
> > > processing requests even without that first kick, isn't it?
> >
> > Yes.
> >
> > Thanks
> >
> > >
> > > I think that was discussed also in the past, as previous qemu patches
> > > gave a spurious kick to start queues just in case. I think the
> > > conclusion was that at least the vDPA device will start processing
> > > requests at DRIVER_OK, no need to wait for a kick.
> > >
> > > https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
> > >
> > > Thanks!
> > >
> > > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > MST
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > >
> > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > before posting.
> > > > > > > > > >
> > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Mon, Jun 26, 2023 at 4:53 AM Jason Wang <jasowang@redhat.com> wrote:
>
> On Mon, Jun 26, 2023 at 5:32 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, Jun 25, 2023 at 01:31:36PM +0800, Jason Wang wrote:
> > > On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
> > > <eperezma@redhat.com> wrote:
> > > >
> > > > On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > >
> > > > > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > > > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > > > > much more with this.
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > > > > >
> > > > > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > > > > >
> > > > > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thinking more about this, I think the following works:
> > > > > > > > > > >
> > > > > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > > > > - DRIVER_OK
> > > > > > > > > > > - CVQ commands
> > > > > > > > > > > - re-enable data queues
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > No?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > > > > asked for that in the past at
> > > > > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > > > > .
> > > > > > > > >
> > > > > > > > > Well then Jason said:
> > > > > > > > >  we should make svq work without RING_RESET
> > > > > > > > > but now apparently we went full circle and propose
> > > > > > > > > a feature bit after all?
> > > > > > > > >
> > > > > > > >
> > > > > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > > > > enabling of net data vqs, like to change the address or properties of
> > > > > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > > > > in current hw, vq property changes may be more challenging. There are
> > > > > > > > current HW that only support late enabling but not reconfigure the
> > > > > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > > > > >
> > > > > > > > Does it make sense to change the proposal to something in the line of:
> > > > > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > > > > once it is live.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > >
> > > > > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > > > > vendors chime in and confirm?
> > > > > > > It seems like a very very narrow "make my hack work" use-case.
> > > > > > > And I'm not even sure how well it works.
> > > > > > > For example what if cvq command enables rss? If the queues in
> > > > > > > question are not enabled packets will be dropped.
> > > > > > > So - fix some setups, break others?
> > > > > > >
> > > > > >
> > > > > > I don't think it breaks, they are already dropped before setting
> > > > > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > > > > the future, the device is supposed to drop packets meanwhile anyway,
> > > > > > isn't it?
> > > > >
> > > > >
> > > > > I just thought of an idea. Devices generally don't poke at rings
> > > > > by themselves. How about at init time we cause kicks on data queues
> > > > > to vmexit? Then on 1st kick we map hardware so following ones will
> > > > > go straight to hardware. This will make sure cvq is processed
> > > > > first. Hmm?
> > >
> > > This is not the case at least for vhost-net. It doesn't need to be
> > > kicked in order to work. And it's an example that we need clearly
> > > define the behaviour instead of depending on any existing
> > > implementation details.
> >
> > This kind of guarantee however sounds generally useful to
> > control which queues are accessed when.
> >
> > This is only needed after restore, which is ATM a vdpa thing,
> > so can just be a vdpa flag, but I'm ok with it as a
> > virtio flag too.
>
> Great. Having a virtio flag may help to reduce the extra and
> duplicated effort spent in vdpa.
>

So the flag should be something in the line of:
VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after
the first kick. Before that kick, it will not process any request.

?

Thanks!

> Thanks
>
> >
> > > > >
> > > >
> > > > That would be possible, but nothing prevents the hw to start
> > > > processing requests even without that first kick, isn't it?
> > >
> > > Yes.
> > >
> > > Thanks
> > >
> > > >
> > > > I think that was discussed also in the past, as previous qemu patches
> > > > gave a spurious kick to start queues just in case. I think the
> > > > conclusion was that at least the vDPA device will start processing
> > > > requests at DRIVER_OK, no need to wait for a kick.
> > > >
> > > > https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
> > > >
> > > > Thanks!
> > > >
> > > > > > >
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > MST
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > >
> > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > before posting.
> > > > > > > > > > >
> > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Mon, Jun 26, 2023 at 10:19:48AM +0200, Eugenio Perez Martin wrote:
> On Mon, Jun 26, 2023 at 4:53 AM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Mon, Jun 26, 2023 at 5:32 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Sun, Jun 25, 2023 at 01:31:36PM +0800, Jason Wang wrote:
> > > > On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
> > > > <eperezma@redhat.com> wrote:
> > > > >
> > > > > On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > >
> > > > > > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > > > > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > > > > > much more with this.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > > > > > >
> > > > > > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thinking more about this, I think the following works:
> > > > > > > > > > > >
> > > > > > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > > > > > - DRIVER_OK
> > > > > > > > > > > > - CVQ commands
> > > > > > > > > > > > - re-enable data queues
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > No?
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > > > > > asked for that in the past at
> > > > > > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > > > > > .
> > > > > > > > > >
> > > > > > > > > > Well then Jason said:
> > > > > > > > > >  we should make svq work without RING_RESET
> > > > > > > > > > but now apparently we went full circle and propose
> > > > > > > > > > a feature bit after all?
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > > > > > enabling of net data vqs, like to change the address or properties of
> > > > > > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > > > > > in current hw, vq property changes may be more challenging. There are
> > > > > > > > > current HW that only support late enabling but not reconfigure the
> > > > > > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > > > > > >
> > > > > > > > > Does it make sense to change the proposal to something in the line of:
> > > > > > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > > > > > once it is live.
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > > > > > vendors chime in and confirm?
> > > > > > > > It seems like a very very narrow "make my hack work" use-case.
> > > > > > > > And I'm not even sure how well it works.
> > > > > > > > For example what if cvq command enables rss? If the queues in
> > > > > > > > question are not enabled packets will be dropped.
> > > > > > > > So - fix some setups, break others?
> > > > > > > >
> > > > > > >
> > > > > > > I don't think it breaks, they are already dropped before setting
> > > > > > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > > > > > the future, the device is supposed to drop packets meanwhile anyway,
> > > > > > > isn't it?
> > > > > >
> > > > > >
> > > > > > I just thought of an idea. Devices generally don't poke at rings
> > > > > > by themselves. How about at init time we cause kicks on data queues
> > > > > > to vmexit? Then on 1st kick we map hardware so following ones will
> > > > > > go straight to hardware. This will make sure cvq is processed
> > > > > > first. Hmm?
> > > >
> > > > This is not the case at least for vhost-net. It doesn't need to be
> > > > kicked in order to work. And it's an example that we need clearly
> > > > define the behaviour instead of depending on any existing
> > > > implementation details.
> > >
> > > This kind of guarantee however sounds generally useful to
> > > control which queues are accessed when.
> > >
> > > This is only needed after restore, which is ATM a vdpa thing,
> > > so can just be a vdpa flag, but I'm ok with it as a
> > > virtio flag too.
> >
> > Great. Having a virtio flag may help to reduce the extra and
> > duplicated effort spent in vdpa.
> >
> 
> So the flag should be something in the line of:
> VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after
> the first kick. Before that kick, it will not process any request.
> 
> ?
> 
> Thanks!

That's not bad. I am not sure adding this to spec is worth
the effort but at least it's kind of reasonable.
Are there uses such as hardening where something like this could be useful?


> > Thanks
> >
> > >
> > > > > >
> > > > >
> > > > > That would be possible, but nothing prevents the hw to start
> > > > > processing requests even without that first kick, isn't it?
> > > >
> > > > Yes.
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > I think that was discussed also in the past, as previous qemu patches
> > > > > gave a spurious kick to start queues just in case. I think the
> > > > > conclusion was that at least the vDPA device will start processing
> > > > > requests at DRIVER_OK, no need to wait for a kick.
> > > > >
> > > > > https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
> > > > >
> > > > > Thanks!
> > > > >
> > > > > > > >
> > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > MST
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > > >
> > > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > > before posting.
> > > > > > > > > > > >
> > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] Re: [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Mon, Jun 26, 2023 at 5:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jun 26, 2023 at 10:19:48AM +0200, Eugenio Perez Martin wrote:
> > On Mon, Jun 26, 2023 at 4:53 AM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > > On Mon, Jun 26, 2023 at 5:32 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > >
> > > > On Sun, Jun 25, 2023 at 01:31:36PM +0800, Jason Wang wrote:
> > > > > On Sun, Jun 25, 2023 at 2:41 AM Eugenio Perez Martin
> > > > > <eperezma@redhat.com> wrote:
> > > > > >
> > > > > > On Sun, Jun 18, 2023 at 1:08 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > >
> > > > > > > On Sat, Jun 17, 2023 at 02:53:46PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > On Fri, Jun 16, 2023 at 4:40 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > >
> > > > > > > > > On Thu, Jun 15, 2023 at 10:35:09AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > On Tue, Jun 13, 2023 at 2:29 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 13, 2023 at 12:12:43PM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > On Tue, Jun 13, 2023 at 9:53 AM Michael S. Tsirkin <mst@redhat.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 13, 2023 at 03:46:35AM -0400, Michael S. Tsirkin wrote:
> > > > > > > > > > > > > > On Mon, Jun 12, 2023 at 09:56:57AM +0200, Eugenio Perez Martin wrote:
> > > > > > > > > > > > > > > > > Another drawback of that is the long time a device can spend to
> > > > > > > > > > > > > > > > > reconfigure its memory maps.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The map is already used for cvq though, I don't see why it would take
> > > > > > > > > > > > > > > > much more with this.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the case of vDPA HW, CVQ maps may not need to reach the device's or
> > > > > > > > > > > > > > > platform IOMMU, so the modification of that mapping is way cheaper.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Oh, I get it. It's setup time, but you are right.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > taking that into account, does it make
> > > > > > > > > > > > > > > > > sense to send a new version with the bit 1 on enable?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Current hardware won't support the extra bit though.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thinking more about this, I think the following works:
> > > > > > > > > > > > >
> > > > > > > > > > > > > - enable all queues before DRIVER_OK
> > > > > > > > > > > > > - reset all data queues before DRIVER_OK
> > > > > > > > > > > > > - DRIVER_OK
> > > > > > > > > > > > > - CVQ commands
> > > > > > > > > > > > > - re-enable data queues
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > No?
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > If it is valid to write to queue_reset before DRIVER_OK, then sure. I
> > > > > > > > > > > > asked for that in the past at
> > > > > > > > > > > > https://lists.oasis-open.org/archives/virtio-dev/202305/msg00329.html
> > > > > > > > > > > > .
> > > > > > > > > > >
> > > > > > > > > > > Well then Jason said:
> > > > > > > > > > >  we should make svq work without RING_RESET
> > > > > > > > > > > but now apparently we went full circle and propose
> > > > > > > > > > > a feature bit after all?
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > My understanding is that RING_RESET covers more than just delaying the
> > > > > > > > > > enabling of net data vqs, like to change the address or properties of
> > > > > > > > > > vqs after DRIVER_OK. While the late enabling may be easy to implement
> > > > > > > > > > in current hw, vq property changes may be more challenging. There are
> > > > > > > > > > current HW that only support late enabling but not reconfigure the
> > > > > > > > > > queues. Ideally, just the subset of late enabling is needed.
> > > > > > > > > >
> > > > > > > > > > Does it make sense to change the proposal to something in the line of:
> > > > > > > > > > RING_F_LATE_ENABLING: This feature flag is a subset of _F_RING_RESET.
> > > > > > > > > > If this feature is negotiated, all vqs start as if the vqs has been
> > > > > > > > > > reset before DRIVER_OK. This feature does not allow resetting a queue
> > > > > > > > > > once it is live.
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Maybe. No idea - I am not a hardware designer. Can some hardware
> > > > > > > > > vendors chime in and confirm?
> > > > > > > > > It seems like a very very narrow "make my hack work" use-case.
> > > > > > > > > And I'm not even sure how well it works.
> > > > > > > > > For example what if cvq command enables rss? If the queues in
> > > > > > > > > question are not enabled packets will be dropped.
> > > > > > > > > So - fix some setups, break others?
> > > > > > > > >
> > > > > > > >
> > > > > > > > I don't think it breaks, they are already dropped before setting
> > > > > > > > DRIVER_OK anyway. If we restore the configuration before DRIVER_OK in
> > > > > > > > the future, the device is supposed to drop packets meanwhile anyway,
> > > > > > > > isn't it?
> > > > > > >
> > > > > > >
> > > > > > > I just thought of an idea. Devices generally don't poke at rings
> > > > > > > by themselves. How about at init time we cause kicks on data queues
> > > > > > > to vmexit? Then on 1st kick we map hardware so following ones will
> > > > > > > go straight to hardware. This will make sure cvq is processed
> > > > > > > first. Hmm?
> > > > >
> > > > > This is not the case at least for vhost-net. It doesn't need to be
> > > > > kicked in order to work. And it's an example that we need clearly
> > > > > define the behaviour instead of depending on any existing
> > > > > implementation details.
> > > >
> > > > This kind of guarantee however sounds generally useful to
> > > > control which queues are accessed when.
> > > >
> > > > This is only needed after restore, which is ATM a vdpa thing,
> > > > so can just be a vdpa flag, but I'm ok with it as a
> > > > virtio flag too.
> > >
> > > Great. Having a virtio flag may help to reduce the extra and
> > > duplicated effort spent in vdpa.
> > >
> >
> > So the flag should be something in the line of:
> > VIRTIO_F_ACCESS_AFTER_KICK: The device will only access the ring after
> > the first kick. Before that kick, it will not process any request.
> >
> > ?
> >
> > Thanks!
>
> That's not bad. I am not sure adding this to spec is worth
> the effort but at least it's kind of reasonable.
> Are there uses such as hardening where something like this could be useful?

I think so, the driver can simply ignore any device notifications
before the first kick. But it doesn't solve all the problems, we still
need to take care about the device without this feature.

Thanks

>
>
> > > Thanks
> > >
> > > >
> > > > > > >
> > > > > >
> > > > > > That would be possible, but nothing prevents the hw to start
> > > > > > processing requests even without that first kick, isn't it?
> > > > >
> > > > > Yes.
> > > > >
> > > > > Thanks
> > > > >
> > > > > >
> > > > > > I think that was discussed also in the past, as previous qemu patches
> > > > > > gave a spurious kick to start queues just in case. I think the
> > > > > > conclusion was that at least the vDPA device will start processing
> > > > > > requests at DRIVER_OK, no need to wait for a kick.
> > > > > >
> > > > > > https://lore.kernel.org/all/CACGkMEt8cMM1UxVzxb0eHeaWSpR0ApvGzaF901vrM4m-uGMiPA@mail.gmail.com/
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > > MST
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > > > > > > > before posting.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Eugenio Perez Martin]

On Tue, Jun 6, 2023 at 7:55 PM Eugenio Pérez <eperezma@redhat.com> wrote:
>
> This series allows the driver to start the device (as set DRIVER_OK) with only
> some queues enabled, and then enable another queues later.
>
> This is the current way to migrate net device state through control
> virtqueue, in a software assisted framework with vDPA:
> * First, only net CVQ is enabled at DRIVER_OK
> * All the control commands (mac address, mq, etc) needed for the device
> to behave the same as the source of migration are sent
> * Finally all the dataplane queues are enabled.
>

Adding Yan to the thread, who mentioned that he is looking for
something similar in the virtio-networking upstream meeting.

> Eugenio Pérez (2):
>   virtio: introduce selective queue enabling
>   virtio: pci support virtqueue selective enabling
>
>  content.tex       | 15 +++++++++++++--
>  transport-pci.tex |  4 ++++
>  2 files changed, 17 insertions(+), 2 deletions(-)
>
> --
> 2.31.1
>
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Fri, Jul 7, 2023 at 2:19 AM Eugenio Perez Martin <eperezma@redhat.com> wrote:
>
> On Tue, Jun 6, 2023 at 7:55 PM Eugenio Pérez <eperezma@redhat.com> wrote:
> >
> > This series allows the driver to start the device (as set DRIVER_OK) with only
> > some queues enabled, and then enable another queues later.
> >
> > This is the current way to migrate net device state through control
> > virtqueue, in a software assisted framework with vDPA:
> > * First, only net CVQ is enabled at DRIVER_OK
> > * All the control commands (mac address, mq, etc) needed for the device
> > to behave the same as the source of migration are sent
> > * Finally all the dataplane queues are enabled.
> >
>
> Adding Yan to the thread, who mentioned that he is looking for
> something similar in the virtio-networking upstream meeting.

Would there be a V2?

Thanks

>
> > Eugenio Pérez (2):
> >   virtio: introduce selective queue enabling
> >   virtio: pci support virtqueue selective enabling
> >
> >  content.tex       | 15 +++++++++++++--
> >  transport-pci.tex |  4 ++++
> >  2 files changed, 17 insertions(+), 2 deletions(-)
> >
> > --
> > 2.31.1
> >
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Michael S. Tsirkin]

On Mon, Jul 10, 2023 at 11:55:50AM +0800, Jason Wang wrote:
> On Fri, Jul 7, 2023 at 2:19 AM Eugenio Perez Martin <eperezma@redhat.com> wrote:
> >
> > On Tue, Jun 6, 2023 at 7:55 PM Eugenio Pérez <eperezma@redhat.com> wrote:
> > >
> > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > some queues enabled, and then enable another queues later.
> > >
> > > This is the current way to migrate net device state through control
> > > virtqueue, in a software assisted framework with vDPA:
> > > * First, only net CVQ is enabled at DRIVER_OK
> > > * All the control commands (mac address, mq, etc) needed for the device
> > > to behave the same as the source of migration are sent
> > > * Finally all the dataplane queues are enabled.
> > >
> >
> > Adding Yan to the thread, who mentioned that he is looking for
> > something similar in the virtio-networking upstream meeting.
> 
> Would there be a V2?
> 
> Thanks

I'm not sure we need to bother at this point. In linux
we seem to have agreed to use a mix of RING_RESET and
a vdpa specific thing. Non-vdpa solutions don't seem to
need this. Right Parav?

> >
> > > Eugenio Pérez (2):
> > >   virtio: introduce selective queue enabling
> > >   virtio: pci support virtqueue selective enabling
> > >
> > >  content.tex       | 15 +++++++++++++--
> > >  transport-pci.tex |  4 ++++
> > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > >
> > > --
> > > 2.31.1
> > >
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, July 10, 2023 1:49 AM

> I'm not sure we need to bother at this point. In linux we seem to have agreed to
> use a mix of RING_RESET and a vdpa specific thing. Non-vdpa solutions don't
> seem to need this. Right Parav?

Couldn’t follow all the emails, but yes, I agree with Michael that F_RING_RESET seems enough which is serving the purpose of dynamically adding the q.

Re: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Mon, Jul 10, 2023 at 1:49 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Mon, Jul 10, 2023 at 11:55:50AM +0800, Jason Wang wrote:
> > On Fri, Jul 7, 2023 at 2:19 AM Eugenio Perez Martin <eperezma@redhat.com> wrote:
> > >
> > > On Tue, Jun 6, 2023 at 7:55 PM Eugenio Pérez <eperezma@redhat.com> wrote:
> > > >
> > > > This series allows the driver to start the device (as set DRIVER_OK) with only
> > > > some queues enabled, and then enable another queues later.
> > > >
> > > > This is the current way to migrate net device state through control
> > > > virtqueue, in a software assisted framework with vDPA:
> > > > * First, only net CVQ is enabled at DRIVER_OK
> > > > * All the control commands (mac address, mq, etc) needed for the device
> > > > to behave the same as the source of migration are sent
> > > > * Finally all the dataplane queues are enabled.
> > > >
> > >
> > > Adding Yan to the thread, who mentioned that he is looking for
> > > something similar in the virtio-networking upstream meeting.
> >
> > Would there be a V2?
> >
> > Thanks
>
> I'm not sure we need to bother at this point. In linux
> we seem to have agreed to use a mix of RING_RESET and
> a vdpa specific thing.

The question is:

1) This proposal is pretty compact and self contained
2) Some vDPA device and vhost devices already behave like this
3) Using RING_RESET requires a lot of codes and tricks

This makes me wonder if this proposal is still the better way to go.

Thanks

> Non-vdpa solutions don't seem to
> need this.
> Right Parav?
>
> > >
> > > > Eugenio Pérez (2):
> > > >   virtio: introduce selective queue enabling
> > > >   virtio: pci support virtqueue selective enabling
> > > >
> > > >  content.tex       | 15 +++++++++++++--
> > > >  transport-pci.tex |  4 ++++
> > > >  2 files changed, 17 insertions(+), 2 deletions(-)
> > > >
> > > > --
> > > > 2.31.1
> > > >
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH 0/2] Selective queue enabling

[Jason Wang]

On Mon, Jul 10, 2023 at 8:13 PM Parav Pandit <parav@nvidia.com> wrote:
>
>
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Monday, July 10, 2023 1:49 AM
>
> > I'm not sure we need to bother at this point. In linux we seem to have agreed to
> > use a mix of RING_RESET and a vdpa specific thing. Non-vdpa solutions don't
> > seem to need this. Right Parav?
>
> Couldn’t follow all the emails, but yes, I agree with Michael that F_RING_RESET seems enough which is serving the purpose of dynamically adding the q.

Not adding but enabling :)

For example, to be used for migration.

Thanks


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/