[virtio-dev] [PATCH v3 0/2] virtio: introduce VIRTIO_F_RING_RESET for reset queue

[virtio-dev] [PATCH v3 0/2] virtio: introduce VIRTIO_F_RING_RESET for reset queue

[Xuan Zhuo]

Hi All:

This is a new version to support VIRTIO_F_RING_RESET. The feautre
extends the basic facility to allow the driver to reset a virtqueue.
This main motivation is to support the reset function of the queue of the
network device.

Please review.

Thanks

Xuan Zhuo (2):
  virtio: introduce virtqueue reset as basic facility
  virtio: pci support virtqueue reset

 content.tex | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

--
2.31.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 1/2] virtio: introduce virtqueue reset as basic facility

[Xuan Zhuo]

This patch allows the driver to reset a queue individually.

This is very common on general network equipment. By disabling a queue,
you can quickly reclaim the buffer currently on the queue. If necessary,
we can reinitialize the queue separately.

For example, when virtio-net implements support for AF_XDP, we need to
disable a queue to release all the original buffers when AF_XDP setup.
And quickly release all the AF_XDP buffers that have been placed in the
queue when AF_XDP exits.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 content.tex | 41 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)

diff --git a/content.tex b/content.tex
index 37c45d3..603b1f1 100644
--- a/content.tex
+++ b/content.tex
@@ -407,6 +407,43 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Virtqueue Reset}\label{sec:Virtqueues / Virtqueue Reset}
+
+When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
+individually. The way to reset the virtqueue is transport specific.
+
+Virtqueue reset is divided into two parts. The driver MUST reset stop a queue
+first, and then re-enable the queue. The re-enable part is optional.
+
+\devicenormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
+The device MUST not executes the requests from this virtqueue, and notify the
+driver.
+
+And the device MUST initialize all states of this virtqueue, including the
+available state and the used state.
+
+\drivernormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
+
+The driver MUST first notify the device and confirm that we have successfully
+stopped a queue.
+
+Then the driver can release all the resources occupied by this virtqueue.
+And reinitialize the state of this queue.
+
+\devicenormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
+
+This process is the same as the initialization process of a single queue during
+the initialization of the entire device. After the device has accepted all the
+configuration from the driver, it will start working under the notification of the
+driver.
+
+\drivernormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
+
+This process is the same as the initialization process of a single queue during
+the entire device initialization process. After the driver allocates resources
+and completes the configuration of the queue status, it notifies the device to
+start working.
+
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
 
 We start with an overview of device initialization, then expand on the
@@ -6673,6 +6710,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
   transport specific.
   For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
 
+  \item[VIRTIO_F_RING_RESET(40)] This feature indicates
+  that the driver can reset a queue individually.
+  See \ref{sec:Virtqueues / Virtqueue Reset}.
+
 \end{description}
 
 \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
-- 
2.31.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Xuan Zhuo]

PCI support virtqueue reset.

virtio_pci_common_cfg add "queue_reset" to support virtqueue reset.
The driver uses this to selectively reset the queue.

Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
 content.tex | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/content.tex b/content.tex
index 603b1f1..35228dc 100644
--- a/content.tex
+++ b/content.tex
@@ -896,6 +896,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         le64 queue_driver;              /* read-write */
         le64 queue_device;              /* read-write */
         le16 queue_notify_data;         /* read-only for driver */
+        le16 queue_reset;               /* read-write */
 };
 \end{lstlisting}
 
@@ -975,6 +976,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
         may benefit from providing another value, for example an internal virtqueue
         identifier, or an internal offset related to the virtqueue number.
         \end{note}
+
+\item[\field{queue_reset}]
+        The driver uses this to selectively reset the queue.
+        This field exists only if VIRTIO_F_RING_RESET has been
+        negotiated. (see \ref{sec:Virtqueues / Virtqueue Reset}).
+
 \end{description}
 
 \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
@@ -1016,6 +1023,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 The device MUST present a 0 in \field{queue_enable} on reset.
 
+The device MUST present a 0 in \field{queue_reset} after the virtqueue is
+enabled with \field{queue_enable}. The device MUST reset the queue when 1 is
+written to \field{queue_reset}, and present a 1 in \field{queue_reset} until the
+queue is enabled with \field{queue_enable}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
+
+The device MUST present a 0 in \field{queue_enable} after the virtqueue is reset
+with \field{queue_reset}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
+
 The device MUST present a 0 in \field{queue_size} if the virtqueue
 corresponding to the current \field{queue_select} is unavailable.
 
@@ -1040,6 +1055,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
 
 The driver MUST NOT write a 0 to \field{queue_enable}.
 
+If the VIRTIO_F_RING_RESET feature bit is negotiated, the driver can stop using
+the queue by writing a 1 to \field{queue_reset} and MUST read the value back to
+ensure synchronization. Then optionally re-enable the queue by writing a 1 to
+\field{queue_enable} after configuring the other virtqueue fields. (see
+\ref{sec:Virtqueues / Virtqueue Reset}).
+
 \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 
 The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
-- 
2.31.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 1/2] virtio: introduce virtqueue reset as basic facility

[Cornelia Huck]

On Mon, Sep 27 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:

> This patch allows the driver to reset a queue individually.
>
> This is very common on general network equipment. By disabling a queue,
> you can quickly reclaim the buffer currently on the queue. If necessary,
> we can reinitialize the queue separately.
>
> For example, when virtio-net implements support for AF_XDP, we need to
> disable a queue to release all the original buffers when AF_XDP setup.
> And quickly release all the AF_XDP buffers that have been placed in the
> queue when AF_XDP exits.
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  content.tex | 41 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 37c45d3..603b1f1 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -407,6 +407,43 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\section{Virtqueue Reset}\label{sec:Virtqueues / Virtqueue Reset}
> +
> +When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> +individually. The way to reset the virtqueue is transport specific.
> +
> +Virtqueue reset is divided into two parts. The driver MUST reset stop a queue
> +first, and then re-enable the queue. The re-enable part is optional.

'MUST' is only for use within normative sections. Also, I'm not sure
'reset stop' is a good term; I think 'reset' and 're-enable' already pair
well enough.

> +
> +\devicenormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> +The device MUST not executes the requests from this virtqueue, and notify the
> +driver.

"After a queue has been reset by the driver, the device MUST NOT execute
any requests from that virtqueue, or notify the driver for it."

?

> +
> +And the device MUST initialize all states of this virtqueue, including the
> +available state and the used state.

"The device MUST re-initialize any state of a virtqueue that has been
reset, including available and used state."

?

> +
> +\drivernormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> +
> +The driver MUST first notify the device and confirm that we have successfully
> +stopped a queue.

I don't quite understand that sentence... do you mean that the driver
must
- tell the device to reset a queue
- and then verify that the queue has actually been reset (similar to a
  full device reset)?

> +
> +Then the driver can release all the resources occupied by this virtqueue.
> +And reinitialize the state of this queue.

"After the queue has been successfully reset, the driver MAY release any
resource associated with that virtqueue."

?

> +
> +\devicenormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> +
> +This process is the same as the initialization process of a single queue during
> +the initialization of the entire device. After the device has accepted all the
> +configuration from the driver, it will start working under the notification of the
> +driver.
> +
> +\drivernormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> +
> +This process is the same as the initialization process of a single queue during
> +the entire device initialization process. After the driver allocates resources
> +and completes the configuration of the queue status, it notifies the device to
> +start working.

Those two statements don't really read normative, more descriptive. I
would rather expect some statements as to what the device MUST do when
the driver does certain things etc. (and have a description of the
process outside of the normative sections.)

Are the queue characteristics the same when a queue has been reset and
is now re-enabled? I.e. may the device change things like the maximum
queue size while a queue is disabled? I assume the driver may use a
different size for the queue when it re-enables it?

> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>  
>  We start with an overview of device initialization, then expand on the
> @@ -6673,6 +6710,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    transport specific.
>    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>  
> +  \item[VIRTIO_F_RING_RESET(40)] This feature indicates
> +  that the driver can reset a queue individually.
> +  See \ref{sec:Virtqueues / Virtqueue Reset}.
> +
>  \end{description}
>  
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Cornelia Huck]

On Mon, Sep 27 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:

> PCI support virtqueue reset.
>
> virtio_pci_common_cfg add "queue_reset" to support virtqueue reset.
> The driver uses this to selectively reset the queue.
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  content.tex | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 603b1f1..35228dc 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -896,6 +896,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>          le64 queue_driver;              /* read-write */
>          le64 queue_device;              /* read-write */
>          le16 queue_notify_data;         /* read-only for driver */
> +        le16 queue_reset;               /* read-write */
>  };
>  \end{lstlisting}
>  
> @@ -975,6 +976,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>          may benefit from providing another value, for example an internal virtqueue
>          identifier, or an internal offset related to the virtqueue number.
>          \end{note}
> +
> +\item[\field{queue_reset}]
> +        The driver uses this to selectively reset the queue.
> +        This field exists only if VIRTIO_F_RING_RESET has been
> +        negotiated. (see \ref{sec:Virtqueues / Virtqueue Reset}).
> +
>  \end{description}
>  
>  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> @@ -1016,6 +1023,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>  
>  The device MUST present a 0 in \field{queue_enable} on reset.
>  
> +The device MUST present a 0 in \field{queue_reset} after the virtqueue is
> +enabled with \field{queue_enable}. The device MUST reset the queue when 1 is
> +written to \field{queue_reset}, and present a 1 in \field{queue_reset} until the
> +queue is enabled with \field{queue_enable}. (see \ref{sec:Virtqueues / Virtqueue Reset}).

"present a 1 in \field{queue_reset} after the queue has been reset,
until the driver re-enables the queue via \field{queue_enable}"

?

I suppose queue_reset is reset to 0 when the whole device is reset?

> +
> +The device MUST present a 0 in \field{queue_enable} after the virtqueue is reset
> +with \field{queue_reset}. (see \ref{sec:Virtqueues / Virtqueue Reset}).

"after the driver has reset the virtqueue via \field{queue_reset}" ?

This section should probably also refer to the feature bit at least once?

> +
>  The device MUST present a 0 in \field{queue_size} if the virtqueue
>  corresponding to the current \field{queue_select} is unavailable.
>  
> @@ -1040,6 +1055,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
>  
>  The driver MUST NOT write a 0 to \field{queue_enable}.
>  
> +If the VIRTIO_F_RING_RESET feature bit is negotiated, the driver can stop using
> +the queue by writing a 1 to \field{queue_reset} and MUST read the value back to
> +ensure synchronization. Then optionally re-enable the queue by writing a 1 to
> +\field{queue_enable} after configuring the other virtqueue fields. (see
> +\ref{sec:Virtqueues / Virtqueue Reset}).

"If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to
\field{queue_reset} to reset the queue, it MUST verify that the queue
has been reset by reading back \field{queue_reset} and ensuring that it
is 1. The driver MAY re-enable the queue by writing a 1 to
\field{queue_enable} after ensuring that the other virtqueue fields have
been set up correctly."

?

> +
>  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>  
>  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG

MMIO can probably use a similar mechanism.

For CCW, maybe we can add a payload to the existing RESET command if the
feature has been negotiated; the queue can be re-enabled by the normal
READ_VQ_CONF/SET_VQ procedure. I have not yet thought that through.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 1/2] virtio: introduce virtqueue reset as basic facility

[Xuan Zhuo]

On Mon, 27 Sep 2021 16:37:28 +0200, Cornelia Huck <cohuck@redhat.com> wrote:
> On Mon, Sep 27 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> > This patch allows the driver to reset a queue individually.
> >
> > This is very common on general network equipment. By disabling a queue,
> > you can quickly reclaim the buffer currently on the queue. If necessary,
> > we can reinitialize the queue separately.
> >
> > For example, when virtio-net implements support for AF_XDP, we need to
> > disable a queue to release all the original buffers when AF_XDP setup.
> > And quickly release all the AF_XDP buffers that have been placed in the
> > queue when AF_XDP exits.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  content.tex | 41 +++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 41 insertions(+)
> >
> > diff --git a/content.tex b/content.tex
> > index 37c45d3..603b1f1 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -407,6 +407,43 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> >  types. It is RECOMMENDED that devices generate version 4
> >  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >
> > +\section{Virtqueue Reset}\label{sec:Virtqueues / Virtqueue Reset}
> > +
> > +When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > +individually. The way to reset the virtqueue is transport specific.
> > +
> > +Virtqueue reset is divided into two parts. The driver MUST reset stop a queue
> > +first, and then re-enable the queue. The re-enable part is optional.
>
> 'MUST' is only for use within normative sections. Also, I'm not sure
> 'reset stop' is a good term; I think 'reset' and 're-enable' already pair
> well enough.
>
> > +
> > +\devicenormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> > +The device MUST not executes the requests from this virtqueue, and notify the
> > +driver.
>
> "After a queue has been reset by the driver, the device MUST NOT execute
> any requests from that virtqueue, or notify the driver for it."
>
> ?

Thanks, this is better.

>
> > +
> > +And the device MUST initialize all states of this virtqueue, including the
> > +available state and the used state.
>
> "The device MUST re-initialize any state of a virtqueue that has been
> reset, including available and used state."
>
> ?

Thanks, this is better.
>
> > +
> > +\drivernormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> > +
> > +The driver MUST first notify the device and confirm that we have successfully
> > +stopped a queue.
>
> I don't quite understand that sentence... do you mean that the driver
> must
> - tell the device to reset a queue
> - and then verify that the queue has actually been reset (similar to a
>   full device reset)?
>
> > +
> > +Then the driver can release all the resources occupied by this virtqueue.
> > +And reinitialize the state of this queue.
>
> "After the queue has been successfully reset, the driver MAY release any
> resource associated with that virtqueue."
>
> ?

Thanks, this is better.

>
> > +
> > +\devicenormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> > +
> > +This process is the same as the initialization process of a single queue during
> > +the initialization of the entire device. After the device has accepted all the
> > +configuration from the driver, it will start working under the notification of the
> > +driver.
> > +
> > +\drivernormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> > +
> > +This process is the same as the initialization process of a single queue during
> > +the entire device initialization process. After the driver allocates resources
> > +and completes the configuration of the queue status, it notifies the device to
> > +start working.
>
> Those two statements don't really read normative, more descriptive. I
> would rather expect some statements as to what the device MUST do when
> the driver does certain things etc. (and have a description of the
> process outside of the normative sections.)

OK.

>
> Are the queue characteristics the same when a queue has been reset and
> is now re-enabled? I.e. may the device change things like the maximum
> queue size while a queue is disabled? I assume the driver may use a
> different size for the queue when it re-enables it?

These features can be modified when re-enable. I will explain this in the next
version.

Thanks very much.

>
> > +
> >  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> >
> >  We start with an overview of device initialization, then expand on the
> > @@ -6673,6 +6710,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >    transport specific.
> >    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> >
> > +  \item[VIRTIO_F_RING_RESET(40)] This feature indicates
> > +  that the driver can reset a queue individually.
> > +  See \ref{sec:Virtqueues / Virtqueue Reset}.
> > +
> >  \end{description}
> >
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

[virtio-dev] Re: [PATCH v3 1/2] virtio: introduce virtqueue reset as basic facility

[Jason Wang]

On Mon, Sep 27, 2021 at 8:32 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> This patch allows the driver to reset a queue individually.
>
> This is very common on general network equipment. By disabling a queue,
> you can quickly reclaim the buffer currently on the queue. If necessary,
> we can reinitialize the queue separately.
>
> For example, when virtio-net implements support for AF_XDP, we need to
> disable a queue to release all the original buffers when AF_XDP setup.
> And quickly release all the AF_XDP buffers that have been placed in the
> queue when AF_XDP exits.
>
> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
>  content.tex | 41 +++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 41 insertions(+)
>
> diff --git a/content.tex b/content.tex
> index 37c45d3..603b1f1 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -407,6 +407,43 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>
> +\section{Virtqueue Reset}\label{sec:Virtqueues / Virtqueue Reset}
> +
> +When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> +individually. The way to reset the virtqueue is transport specific.
> +
> +Virtqueue reset is divided into two parts. The driver MUST reset stop a queue
> +first, and then re-enable the queue. The re-enable part is optional.
> +
> +\devicenormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> +The device MUST not executes the requests from this virtqueue, and notify the
> +driver.
> +
> +And the device MUST initialize all states of this virtqueue, including the
> +available state and the used state.
> +
> +\drivernormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> +
> +The driver MUST first notify

What did "notify" mean here? I guess it means the transport specific
way to stop a virtqueue?

Thanks

> the device and confirm that we have successfully
> +stopped a queue.
> +
> +Then the driver can release all the resources occupied by this virtqueue.
> +And reinitialize the state of this queue.
> +
> +\devicenormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> +
> +This process is the same as the initialization process of a single queue during
> +the initialization of the entire device. After the device has accepted all the
> +configuration from the driver, it will start working under the notification of the
> +driver.
> +
> +\drivernormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> +
> +This process is the same as the initialization process of a single queue during
> +the entire device initialization process. After the driver allocates resources
> +and completes the configuration of the queue status, it notifies the device to
> +start working.
> +
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
>
>  We start with an overview of device initialization, then expand on the
> @@ -6673,6 +6710,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>    transport specific.
>    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
>
> +  \item[VIRTIO_F_RING_RESET(40)] This feature indicates
> +  that the driver can reset a queue individually.
> +  See \ref{sec:Virtqueues / Virtqueue Reset}.
> +
>  \end{description}
>
>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> --
> 2.31.0
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Jason Wang]

On Mon, Sep 27, 2021 at 10:51 PM Cornelia Huck <cohuck@redhat.com> wrote:
>
> On Mon, Sep 27 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> > PCI support virtqueue reset.
> >
> > virtio_pci_common_cfg add "queue_reset" to support virtqueue reset.
> > The driver uses this to selectively reset the queue.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  content.tex | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> >
> > diff --git a/content.tex b/content.tex
> > index 603b1f1..35228dc 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -896,6 +896,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >          le64 queue_driver;              /* read-write */
> >          le64 queue_device;              /* read-write */
> >          le16 queue_notify_data;         /* read-only for driver */
> > +        le16 queue_reset;               /* read-write */
> >  };
> >  \end{lstlisting}
> >
> > @@ -975,6 +976,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >          may benefit from providing another value, for example an internal virtqueue
> >          identifier, or an internal offset related to the virtqueue number.
> >          \end{note}
> > +
> > +\item[\field{queue_reset}]
> > +        The driver uses this to selectively reset the queue.
> > +        This field exists only if VIRTIO_F_RING_RESET has been
> > +        negotiated. (see \ref{sec:Virtqueues / Virtqueue Reset}).
> > +
> >  \end{description}
> >
> >  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> > @@ -1016,6 +1023,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >
> >  The device MUST present a 0 in \field{queue_enable} on reset.
> >
> > +The device MUST present a 0 in \field{queue_reset} after the virtqueue is
> > +enabled with \field{queue_enable}. The device MUST reset the queue when 1 is
> > +written to \field{queue_reset}, and present a 1 in \field{queue_reset} until the
> > +queue is enabled with \field{queue_enable}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
>
> "present a 1 in \field{queue_reset} after the queue has been reset,
> until the driver re-enables the queue via \field{queue_enable}"
>
> ?
>
> I suppose queue_reset is reset to 0 when the whole device is reset?
>
> > +
> > +The device MUST present a 0 in \field{queue_enable} after the virtqueue is reset
> > +with \field{queue_reset}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
>
> "after the driver has reset the virtqueue via \field{queue_reset}" ?
>
> This section should probably also refer to the feature bit at least once?
>
> > +
> >  The device MUST present a 0 in \field{queue_size} if the virtqueue
> >  corresponding to the current \field{queue_select} is unavailable.
> >
> > @@ -1040,6 +1055,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >
> >  The driver MUST NOT write a 0 to \field{queue_enable}.
> >
> > +If the VIRTIO_F_RING_RESET feature bit is negotiated, the driver can stop using
> > +the queue by writing a 1 to \field{queue_reset} and MUST read the value back to
> > +ensure synchronization. Then optionally re-enable the queue by writing a 1 to
> > +\field{queue_enable} after configuring the other virtqueue fields. (see
> > +\ref{sec:Virtqueues / Virtqueue Reset}).
>
> "If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to
> \field{queue_reset} to reset the queue, it MUST verify that the queue
> has been reset by reading back \field{queue_reset} and ensuring that it
> is 1. The driver MAY re-enable the queue by writing a 1 to
> \field{queue_enable} after ensuring that the other virtqueue fields have
> been set up correctly."
>
> ?
>
> > +
> >  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
> >
> >  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
>
> MMIO can probably use a similar mechanism.
>
> For CCW, maybe we can add a payload to the existing RESET command if the
> feature has been negotiated;

I wonder if it's better for a new dedicated command. Consider device
reset is different from virtqueue reset.

Thanks

> the queue can be re-enabled by the normal
> READ_VQ_CONF/SET_VQ procedure. I have not yet thought that through.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v3 1/2] virtio: introduce virtqueue reset as basic facility

[Xuan Zhuo]

On Tue, 28 Sep 2021 11:01:55 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Mon, Sep 27, 2021 at 8:32 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > This patch allows the driver to reset a queue individually.
> >
> > This is very common on general network equipment. By disabling a queue,
> > you can quickly reclaim the buffer currently on the queue. If necessary,
> > we can reinitialize the queue separately.
> >
> > For example, when virtio-net implements support for AF_XDP, we need to
> > disable a queue to release all the original buffers when AF_XDP setup.
> > And quickly release all the AF_XDP buffers that have been placed in the
> > queue when AF_XDP exits.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  content.tex | 41 +++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 41 insertions(+)
> >
> > diff --git a/content.tex b/content.tex
> > index 37c45d3..603b1f1 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -407,6 +407,43 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
> >  types. It is RECOMMENDED that devices generate version 4
> >  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
> >
> > +\section{Virtqueue Reset}\label{sec:Virtqueues / Virtqueue Reset}
> > +
> > +When VIRTIO_F_RING_RESET is negotiated, the driver can reset a virtqueue
> > +individually. The way to reset the virtqueue is transport specific.
> > +
> > +Virtqueue reset is divided into two parts. The driver MUST reset stop a queue
> > +first, and then re-enable the queue. The re-enable part is optional.
> > +
> > +\devicenormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> > +The device MUST not executes the requests from this virtqueue, and notify the
> > +driver.
> > +
> > +And the device MUST initialize all states of this virtqueue, including the
> > +available state and the used state.
> > +
> > +\drivernormative{\subsection}{Virtqueue Reset Stop}{Virtqueues / Virtqueue Reset}
> > +
> > +The driver MUST first notify
>
> What did "notify" mean here? I guess it means the transport specific
> way to stop a virtqueue?

Yes.

Cornelia Huck suggested that this expression would be better:

	After the driver tell the device to reset a queue, the driver MUST verify that
	the queue has actually been reset.

Thanks.


>
> Thanks
>
> > the device and confirm that we have successfully
> > +stopped a queue.
> > +
> > +Then the driver can release all the resources occupied by this virtqueue.
> > +And reinitialize the state of this queue.
> > +
> > +\devicenormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> > +
> > +This process is the same as the initialization process of a single queue during
> > +the initialization of the entire device. After the device has accepted all the
> > +configuration from the driver, it will start working under the notification of the
> > +driver.
> > +
> > +\drivernormative{\subsection}{Virtqueue Reset Re-enable}{Virtqueues / Virtqueue Reset}
> > +
> > +This process is the same as the initialization process of a single queue during
> > +the entire device initialization process. After the driver allocates resources
> > +and completes the configuration of the queue status, it notifies the device to
> > +start working.
> > +
> >  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> >
> >  We start with an overview of device initialization, then expand on the
> > @@ -6673,6 +6710,10 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
> >    transport specific.
> >    For more details about driver notifications over PCI see \ref{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Available Buffer Notifications}.
> >
> > +  \item[VIRTIO_F_RING_RESET(40)] This feature indicates
> > +  that the driver can reset a queue individually.
> > +  See \ref{sec:Virtqueues / Virtqueue Reset}.
> > +
> >  \end{description}
> >
> >  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> > --
> > 2.31.0
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Xuan Zhuo]

On Mon, 27 Sep 2021 16:51:43 +0200, Cornelia Huck <cohuck@redhat.com> wrote:
> On Mon, Sep 27 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> > PCI support virtqueue reset.
> >
> > virtio_pci_common_cfg add "queue_reset" to support virtqueue reset.
> > The driver uses this to selectively reset the queue.
> >
> > Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> >  content.tex | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> >
> > diff --git a/content.tex b/content.tex
> > index 603b1f1..35228dc 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -896,6 +896,7 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >          le64 queue_driver;              /* read-write */
> >          le64 queue_device;              /* read-write */
> >          le16 queue_notify_data;         /* read-only for driver */
> > +        le16 queue_reset;               /* read-write */
> >  };
> >  \end{lstlisting}
> >
> > @@ -975,6 +976,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >          may benefit from providing another value, for example an internal virtqueue
> >          identifier, or an internal offset related to the virtqueue number.
> >          \end{note}
> > +
> > +\item[\field{queue_reset}]
> > +        The driver uses this to selectively reset the queue.
> > +        This field exists only if VIRTIO_F_RING_RESET has been
> > +        negotiated. (see \ref{sec:Virtqueues / Virtqueue Reset}).
> > +
> >  \end{description}
> >
> >  \devicenormative{\paragraph}{Common configuration structure layout}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Common configuration structure layout}
> > @@ -1016,6 +1023,14 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >
> >  The device MUST present a 0 in \field{queue_enable} on reset.
> >
> > +The device MUST present a 0 in \field{queue_reset} after the virtqueue is
> > +enabled with \field{queue_enable}. The device MUST reset the queue when 1 is
> > +written to \field{queue_reset}, and present a 1 in \field{queue_reset} until the
> > +queue is enabled with \field{queue_enable}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
>
> "present a 1 in \field{queue_reset} after the queue has been reset,
> until the driver re-enables the queue via \field{queue_enable}"
>
> ?
>

Thanks. This is better.

> I suppose queue_reset is reset to 0 when the whole device is reset?

Yes, I will explain this in the next version.

>
> > +
> > +The device MUST present a 0 in \field{queue_enable} after the virtqueue is reset
> > +with \field{queue_reset}. (see \ref{sec:Virtqueues / Virtqueue Reset}).
>
> "after the driver has reset the virtqueue via \field{queue_reset}" ?

Thanks. This is better.

>
> This section should probably also refer to the feature bit at least once?

OK.

>
> > +
> >  The device MUST present a 0 in \field{queue_size} if the virtqueue
> >  corresponding to the current \field{queue_select} is unavailable.
> >
> > @@ -1040,6 +1055,12 @@ \subsubsection{Common configuration structure layout}\label{sec:Virtio Transport
> >
> >  The driver MUST NOT write a 0 to \field{queue_enable}.
> >
> > +If the VIRTIO_F_RING_RESET feature bit is negotiated, the driver can stop using
> > +the queue by writing a 1 to \field{queue_reset} and MUST read the value back to
> > +ensure synchronization. Then optionally re-enable the queue by writing a 1 to
> > +\field{queue_enable} after configuring the other virtqueue fields. (see
> > +\ref{sec:Virtqueues / Virtqueue Reset}).
>
> "If VIRTIO_F_RING_RESET has been negotiated, after the driver writes 1 to
> \field{queue_reset} to reset the queue, it MUST verify that the queue
> has been reset by reading back \field{queue_reset} and ensuring that it
> is 1. The driver MAY re-enable the queue by writing a 1 to
> \field{queue_enable} after ensuring that the other virtqueue fields have
> been set up correctly."
>
> ?

Thanks. This is better.

>
> > +
> >  \subsubsection{Notification structure layout}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
> >
> >  The notification location is found using the VIRTIO_PCI_CAP_NOTIFY_CFG
>
> MMIO can probably use a similar mechanism.
>
> For CCW, maybe we can add a payload to the existing RESET command if the
> feature has been negotiated; the queue can be re-enabled by the normal
> READ_VQ_CONF/SET_VQ procedure. I have not yet thought that through.

Regarding CCW, can I consider adding this part later? This time I want to
complete PCI and MMIO support for queue reset first.

Thanks.

Re: [virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Cornelia Huck]

On Tue, Sep 28 2021, Jason Wang <jasowang@redhat.com> wrote:

> On Mon, Sep 27, 2021 at 10:51 PM Cornelia Huck <cohuck@redhat.com> wrote:
>> For CCW, maybe we can add a payload to the existing RESET command if the
>> feature has been negotiated;
>
> I wonder if it's better for a new dedicated command. Consider device
> reset is different from virtqueue reset.

Yes, either can work. I'm trying to be a bit conservative with
allocating new commands, but if it is too confusing, we can use a new
one. The important thing is that I think this can be implemented for ccw
as well.

>
> Thanks
>
>> the queue can be re-enabled by the normal
>> READ_VQ_CONF/SET_VQ procedure. I have not yet thought that through.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 2/2] virtio: pci support virtqueue reset

[Cornelia Huck]

On Tue, Sep 28 2021, Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:

> On Mon, 27 Sep 2021 16:51:43 +0200, Cornelia Huck <cohuck@redhat.com> wrote:
>> MMIO can probably use a similar mechanism.
>>
>> For CCW, maybe we can add a payload to the existing RESET command if the
>> feature has been negotiated; the queue can be re-enabled by the normal
>> READ_VQ_CONF/SET_VQ procedure. I have not yet thought that through.
>
> Regarding CCW, can I consider adding this part later? This time I want to
> complete PCI and MMIO support for queue reset first.

Sure, I do not really expect people to come up with procedures for ccw;
I think it can be made to work.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org