On Thu, May 04, 2023 at 04:19:00PM +0800, zhenwei pi wrote:
> In the past years, virtio supports lots of device specifications by
> PCI/MMIO/CCW. These devices work fine in the virtualization environment.
> 
> Introduce Virtio Over Fabrics transport to support "network defined
> peripheral devices". With this transport, Many Virtio based devices
> transparently work over fabrics. Note that the balloon device may not
> make sense. Shared memory regions won't work.
> 
> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> ---
>  content.tex           |  1 +
>  transport-fabrics.tex | 31 +++++++++++++++++++++++++++++++
>  2 files changed, 32 insertions(+)
>  create mode 100644 transport-fabrics.tex
> 
> diff --git a/content.tex b/content.tex
> index cff548a..f899c3a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -582,6 +582,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options}
>  \input{transport-pci.tex}
>  \input{transport-mmio.tex}
>  \input{transport-ccw.tex}
> +\input{transport-fabrics.tex}
>  
>  \chapter{Device Types}\label{sec:Device Types}
>  
> diff --git a/transport-fabrics.tex b/transport-fabrics.tex
> new file mode 100644
> index 0000000..0dc031b
> --- /dev/null
> +++ b/transport-fabrics.tex
> @@ -0,0 +1,31 @@
> +\section{Virtio Over Fabrics}\label{sec:Virtio Transport Options / Virtio Over Fabrics}
> +
> +This section defines specification to Virtio that enables operation over other
> +interconnects. A central goal of Virtio Over Fabrics is to maintain consistency
> +with the PCI device, so Virtio based devices transparently work over PCI or
> +fabrics.

The reader wants to know what VIRTIO Over Fabrics is, not how it relates
to other Transports that they may not be very familiar with.

Fabrics is a Transport and any Transport is capable of supporting the
VIRTIO device model. Therefore I don't think the stated aim should be to
match PCI specifically. Just being a Transport is already enough. PCI is
not special.

I suggest something like:

  Virtio Over Fabrics enables operation over interconnects that rely
  primarily on message passing. Supported interconnects include TODO.

> +
> +Virtio Over Fabrics uses reliable connection to transmit data, the reliable

"uses a reliable connection"

> +connection betweens two rules:

"connection facilitates communication between entities playing the following roles:"

> +
> +\begin{itemize}
> +\item An initiator functions as an Virtio Over Fabrics client. An initiator

"as a Virtio ..."

> +typically serves the same purpose to a machine as a Virtio device, issues
> +commands to remote side.

This says that the driver talks to the initiator instead of a local
device and the initiator forwards commands to the actual device on the
remote side?

I find this sentence confusing because I associate the initiator with
the driver, not the device.

Maybe:

  The initiator sends commands from the driver to the target.

> +\item A target functions as an Virtio Over Fabrics server. An target typically

"A target"

> +handles commands from the initiator side and responses completions.

The concept of the device is missing here. For symmetry it may be good
to say something like:

  The target forwards commands to the device and sends responses back to
  the initiator.

> +\end{itemize}
> +
> +Virtio Over Fabrics has the following differences from the PCI based
> +specification:
> +
> +\begin{itemize}
> +\item Instead of memory sharing mechanism of virtqueue, there is a one-to-one
> +mapping between virtqueue and the reliable connection which executes the vring
> +data transmission.
> +\item An additional control connection is required to execute control commands
> +which is similar to read/write register on a PCI device.
> +\item Virtio Over Fabrics does not define an interrupt mechanism that allows an
> +initiator to generate a host interrupt. It is the responsibility of the host
> +fabric interface to generate host interrupts.
> +\end{itemize}

As mentioned above, comparing against PCI requires that the reader is
familiar with PCI. I think it would be preferrable to explain the unique
characteristics of Virtio Over Fabrics in a self-contained way:

  The basic organization of Virtio Over Fabrics is as follows:

  \begin{itemize}
  \item A reliable connection carries control commands that are not specific to a virtqueue.
  \item Each virtqueue has its own reliable connection.
  \item There is no interrupt mechanism since the arrival of data on the fabric already indicates when there is activity.
  \end{itemize}

Stefan