[virtio-dev] [PATCH 0/2] virtio-fs: add notification queue

[virtio-dev] [PATCH 0/2] virtio-fs: add notification queue

[Stefan Hajnoczi]

This patch series adds the notification queue to the VIRTIO specification.
This new virtqueue carries device->driver FUSE notify messages.  They are
currently unused but will be necessary for file locking, which can block for an
unbounded amount of time and therefore needs a asynchronous completion event
instead of a request/response buffer that consumes space in the request
virtqueue until the operation completes.

Patch 1 corrects an oversight I noticed: the file system device was not added
to the Conformance chapter.

Stefan Hajnoczi (2):
  virtio-fs: add file system device to Conformance chapter
  virtio-fs: add notification queue

 conformance.tex | 21 +++++++++++++++++
 virtio-fs.tex   | 63 +++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 74 insertions(+), 10 deletions(-)

-- 
2.23.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 1/2] virtio-fs: add file system device to Conformance chapter

[Stefan Hajnoczi]

The file system device is not listed in the Conformance chapter.  Fix
this.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
 conformance.tex | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/conformance.tex b/conformance.tex
index 0ac58aa..25d11ec 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -183,6 +183,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / Socket Device / Device Operation / Device Events}
 \end{itemize}
 
+\conformance{\subsection}{File System Driver Conformance}\label{sec:Conformance / Driver Conformance / File System Driver Conformance}
+
+A file system driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
+\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
+\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
+\end{itemize}
+
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
 A device MUST conform to the following normative statements:
@@ -338,6 +348,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / Socket Device / Device Operation / Receive and Transmit}
 \end{itemize}
 
+\conformance{\subsection}{File System Device Conformance}\label{sec:Conformance / Device Conformance / File System Device Conformance}
+
+A file system device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / File System Device / Device configuration layout}
+\item \ref{devicenormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
+\item \ref{devicenormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
+\end{itemize}
+
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
 non-transitional, see \ref{intro:Legacy
-- 
2.23.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH 2/2] virtio-fs: add notification queue

[Stefan Hajnoczi]

The FUSE protocol allows the file server (device) to initiate
communication with the client (driver) using FUSE notify messages.
Normally only the client can initiate communication.  This feature is
used to report asynchronous events that are not related to an in-flight
request.

This patch adds a notification queue that works like an rx queue in
other types of device.  The device can emit FUSE notify messages by
using a buffer from this queue.

This mechanism was designed by Vivek Goyal <vgoyal@redhat.com>.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
---
TODO:
 * Flow control?  What happens when the notification queue is empty when
   the device wants to emit a message.  We cannot assume unlimited
   device resources for holding back pending notifications.  Eventually
   they will have to be dropped and/or the device needs to report an
   error.
---
 conformance.tex |  1 +
 virtio-fs.tex   | 63 +++++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 54 insertions(+), 10 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index 25d11ec..a43eea3 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -190,6 +190,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \begin{itemize}
 \item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
 \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
+\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
 \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
 \end{itemize}
 
diff --git a/virtio-fs.tex b/virtio-fs.tex
index 158d066..a94a377 100644
--- a/virtio-fs.tex
+++ b/virtio-fs.tex
@@ -25,24 +25,33 @@ \subsection{Virtqueues}\label{sec:Device Types / File System Device / Virtqueues
 
 \begin{description}
 \item[0] hiprio
-\item[1\ldots n] request queues
+\item[1] notification queue
+\item[2\ldots n] request queues
 \end{description}
 
+The notification queue only exists if VIRTIO_FS_F_NOTIFICATION is set.
+
 \subsection{Feature bits}\label{sec:Device Types / File System Device / Feature bits}
 
-There are currently no feature bits defined.
+\begin{description}
+\item[VIRTIO_FS_F_NOTIFICATION (0)] Device has support for FUSE notify
+    messages.  The notification queue is virtqueue 1.
+\end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / File System Device / Device configuration layout}
 
-All fields of this configuration are always available.
-
 \begin{lstlisting}
 struct virtio_fs_config {
         char tag[36];
         le32 num_request_queues;
+        le32 notify_buf_size;
 };
 \end{lstlisting}
 
+The \field{tag} and \field{num_request_queues} fields are always available.
+The \field{notify_buf_size} field is only available when
+VIRTIO_FS_F_NOTIFICATION is set.
+
 \begin{description}
 \item[\field{tag}] is the name associated with this file system.  The tag is
     encoded in UTF-8 and padded with NUL bytes if shorter than the
@@ -53,6 +62,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
     there are no ordering guarantees between requests made available on
     different queues.  Use of multiple queues is intended to increase
     performance.
+\item[\field{notify_buf_size}] is the minimum number of bytes required for each
+    buffer in the notification queue.
 \end{description}
 
 \drivernormative{\subsubsection}{Device configuration layout}{Device Types / File System Device / Device configuration layout}
@@ -65,13 +76,20 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
 
 The device MUST set \field{num_request_queues} to 1 or greater.
 
+The device MUST set \field{notify_buf_size} to be large enough to hold any of
+the FUSE notify messages that this device emits.
+
 \subsection{Device Initialization}\label{Device Types / File System Device / Device Initialization}
 
-On initialization the driver first discovers the device's virtqueues.  The FUSE
-session is started by sending a FUSE\_INIT request as defined by the FUSE
-protocol on one request virtqueue.  All virtqueues provide access to the same
-FUSE session and therefore only one FUSE\_INIT request is required regardless
-of the number of available virtqueues.
+On initialization the driver first discovers the device's virtqueues.
+
+The driver populates the notification queue with buffers for receiving FUSE
+notify messages if VIRTIO_FS_F_NOTIFICATION is set.
+
+The FUSE session is started by sending a FUSE\_INIT request as defined by the
+FUSE protocol on one request virtqueue.  All virtqueues provide access to the
+same FUSE session and therefore only one FUSE\_INIT request is required
+regardless of the number of available virtqueues.
 
 \subsection{Device Operation}\label{sec:Device Types / File System Device / Device Operation}
 
@@ -88,7 +106,8 @@ \subsection{Device Operation}\label{sec:Device Types / File System Device / Devi
       full.
 \end{itemize}
 
-Note that FUSE notification requests are not supported.
+FUSE notify messages are received on the notification queue if
+VIRTIO_FS_F_NOTIFICATION is set.
 
 \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Request Queues}
 
@@ -179,6 +198,30 @@ \subsubsection{Device Operation: High Priority Queue}\label{sec:Device Types / F
 
 The driver MUST anticipate that request queues are processed concurrently with the hiprio queue.
 
+\subsubsection{Device Operation: Notification Queue}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
+
+The notification queue is populated with buffers by the driver and these
+buffers are used by the device to emit FUSE notify messages.  Notification
+queue buffer layout is as follows:
+
+\begin{lstlisting}
+struct virtio_fs_notify {
+        // Device-writable part
+        struct fuse_out_header out_hdr;
+        char outarg[notify_buf_size - sizeof(struct fuse_out_header)];
+};
+\end{lstlisting}
+
+\field{outarg} contains the FUSE notify message payload that depends on the
+type of notification being emitted.
+
+\drivernormative{\paragraph}{Device Operation: Notification Queue}{Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
+
+The driver MUST provide buffers of at least \field{notify_buf_size} bytes.
+
+The driver SHOULD replenish notification queue buffers sufficiently quickly so
+that there is always at least one available buffer.
+
 \subsubsection{Device Operation: DAX Window}\label{sec:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
 
 FUSE\_READ and FUSE\_WRITE requests transfer file contents between the
-- 
2.23.0


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 1/2] virtio-fs: add file system device to Conformance chapter

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@redhat.com) wrote:
> The file system device is not listed in the Conformance chapter.  Fix
> this.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
>  conformance.tex | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/conformance.tex b/conformance.tex
> index 0ac58aa..25d11ec 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -183,6 +183,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \item \ref{drivernormative:Device Types / Socket Device / Device Operation / Device Events}
>  \end{itemize}
>  
> +\conformance{\subsection}{File System Driver Conformance}\label{sec:Conformance / Driver Conformance / File System Driver Conformance}
> +
> +A file system driver MUST conform to the following normative statements:
> +
> +\begin{itemize}
> +\item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
> +\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
> +\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
> +\end{itemize}
> +
>  \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
>  
>  A device MUST conform to the following normative statements:
> @@ -338,6 +348,16 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \item \ref{devicenormative:Device Types / Socket Device / Device Operation / Receive and Transmit}
>  \end{itemize}
>  
> +\conformance{\subsection}{File System Device Conformance}\label{sec:Conformance / Device Conformance / File System Device Conformance}
> +
> +A file system device MUST conform to the following normative statements:
> +
> +\begin{itemize}
> +\item \ref{devicenormative:Device Types / File System Device / Device configuration layout}
> +\item \ref{devicenormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
> +\item \ref{devicenormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
> +\end{itemize}
> +
>  \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
>  A conformant implementation MUST be either transitional or
>  non-transitional, see \ref{intro:Legacy

These look OK.

Do you then need to add these entries to the list of 'one of clauses' in
7.1 ?

Dave

> -- 
> 2.23.0
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH 2/2] virtio-fs: add notification queue

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@redhat.com) wrote:
> The FUSE protocol allows the file server (device) to initiate
> communication with the client (driver) using FUSE notify messages.
> Normally only the client can initiate communication.  This feature is
> used to report asynchronous events that are not related to an in-flight
> request.
> 
> This patch adds a notification queue that works like an rx queue in
> other types of device.  The device can emit FUSE notify messages by
> using a buffer from this queue.
> 
> This mechanism was designed by Vivek Goyal <vgoyal@redhat.com>.
> 
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> ---
> TODO:
>  * Flow control?  What happens when the notification queue is empty when
>    the device wants to emit a message.  We cannot assume unlimited
>    device resources for holding back pending notifications.  Eventually
>    they will have to be dropped and/or the device needs to report an
>    error.
> ---
>  conformance.tex |  1 +
>  virtio-fs.tex   | 63 +++++++++++++++++++++++++++++++++++++++++--------
>  2 files changed, 54 insertions(+), 10 deletions(-)
> 
> diff --git a/conformance.tex b/conformance.tex
> index 25d11ec..a43eea3 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -190,6 +190,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \begin{itemize}
>  \item \ref{drivernormative:Device Types / File System Device / Device configuration layout}
>  \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: High Priority Queue}
> +\item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
>  \item \ref{drivernormative:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
>  \end{itemize}
>  
> diff --git a/virtio-fs.tex b/virtio-fs.tex
> index 158d066..a94a377 100644
> --- a/virtio-fs.tex
> +++ b/virtio-fs.tex
> @@ -25,24 +25,33 @@ \subsection{Virtqueues}\label{sec:Device Types / File System Device / Virtqueues
>  
>  \begin{description}
>  \item[0] hiprio
> -\item[1\ldots n] request queues
> +\item[1] notification queue
> +\item[2\ldots n] request queues
>  \end{description}
>  
> +The notification queue only exists if VIRTIO_FS_F_NOTIFICATION is set.
> +
>  \subsection{Feature bits}\label{sec:Device Types / File System Device / Feature bits}
>  
> -There are currently no feature bits defined.
> +\begin{description}
> +\item[VIRTIO_FS_F_NOTIFICATION (0)] Device has support for FUSE notify
> +    messages.  The notification queue is virtqueue 1.
> +\end{description}
>  
>  \subsection{Device configuration layout}\label{sec:Device Types / File System Device / Device configuration layout}
>  
> -All fields of this configuration are always available.
> -
>  \begin{lstlisting}
>  struct virtio_fs_config {
>          char tag[36];
>          le32 num_request_queues;
> +        le32 notify_buf_size;
>  };
>  \end{lstlisting}
>  
> +The \field{tag} and \field{num_request_queues} fields are always available.
> +The \field{notify_buf_size} field is only available when
> +VIRTIO_FS_F_NOTIFICATION is set.
> +
>  \begin{description}
>  \item[\field{tag}] is the name associated with this file system.  The tag is
>      encoded in UTF-8 and padded with NUL bytes if shorter than the
> @@ -53,6 +62,8 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
>      there are no ordering guarantees between requests made available on
>      different queues.  Use of multiple queues is intended to increase
>      performance.
> +\item[\field{notify_buf_size}] is the minimum number of bytes required for each
> +    buffer in the notification queue.
>  \end{description}
>  
>  \drivernormative{\subsubsection}{Device configuration layout}{Device Types / File System Device / Device configuration layout}
> @@ -65,13 +76,20 @@ \subsection{Device configuration layout}\label{sec:Device Types / File System De
>  
>  The device MUST set \field{num_request_queues} to 1 or greater.
>  
> +The device MUST set \field{notify_buf_size} to be large enough to hold any of
> +the FUSE notify messages that this device emits.
> +

Is there some ordering requirement here to ensure that the device has
set notify_buf_size before the driver starts trying to populate the
queue with empties?

Dave

>  \subsection{Device Initialization}\label{Device Types / File System Device / Device Initialization}
>  
> -On initialization the driver first discovers the device's virtqueues.  The FUSE
> -session is started by sending a FUSE\_INIT request as defined by the FUSE
> -protocol on one request virtqueue.  All virtqueues provide access to the same
> -FUSE session and therefore only one FUSE\_INIT request is required regardless
> -of the number of available virtqueues.
> +On initialization the driver first discovers the device's virtqueues.
> +
> +The driver populates the notification queue with buffers for receiving FUSE
> +notify messages if VIRTIO_FS_F_NOTIFICATION is set.
> +
> +The FUSE session is started by sending a FUSE\_INIT request as defined by the
> +FUSE protocol on one request virtqueue.  All virtqueues provide access to the
> +same FUSE session and therefore only one FUSE\_INIT request is required
> +regardless of the number of available virtqueues.
>  
>  \subsection{Device Operation}\label{sec:Device Types / File System Device / Device Operation}
>  
> @@ -88,7 +106,8 @@ \subsection{Device Operation}\label{sec:Device Types / File System Device / Devi
>        full.
>  \end{itemize}
>  
> -Note that FUSE notification requests are not supported.
> +FUSE notify messages are received on the notification queue if
> +VIRTIO_FS_F_NOTIFICATION is set.
>  
>  \subsubsection{Device Operation: Request Queues}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Request Queues}
>  
> @@ -179,6 +198,30 @@ \subsubsection{Device Operation: High Priority Queue}\label{sec:Device Types / F
>  
>  The driver MUST anticipate that request queues are processed concurrently with the hiprio queue.
>  
> +\subsubsection{Device Operation: Notification Queue}\label{sec:Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
> +
> +The notification queue is populated with buffers by the driver and these
> +buffers are used by the device to emit FUSE notify messages.  Notification
> +queue buffer layout is as follows:
> +
> +\begin{lstlisting}
> +struct virtio_fs_notify {
> +        // Device-writable part
> +        struct fuse_out_header out_hdr;
> +        char outarg[notify_buf_size - sizeof(struct fuse_out_header)];
> +};
> +\end{lstlisting}
> +
> +\field{outarg} contains the FUSE notify message payload that depends on the
> +type of notification being emitted.
> +
> +\drivernormative{\paragraph}{Device Operation: Notification Queue}{Device Types / File System Device / Device Operation / Device Operation: Notification Queue}
> +
> +The driver MUST provide buffers of at least \field{notify_buf_size} bytes.
> +
> +The driver SHOULD replenish notification queue buffers sufficiently quickly so
> +that there is always at least one available buffer.
> +
>  \subsubsection{Device Operation: DAX Window}\label{sec:Device Types / File System Device / Device Operation / Device Operation: DAX Window}
>  
>  FUSE\_READ and FUSE\_WRITE requests transfer file contents between the
> -- 
> 2.23.0
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org