[RFC v4] virtio-vsock: add description for datagram type

[RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang]

From: "jiang.wang" <jiang.wang@bytedance.com>

Add supports for datagram type for virtio-vsock. Datagram
sockets are connectionless and unreliable. To avoid contention
with stream and other sockets, add two more virtqueues and
a new feature bit to identify if those two new queues exist or not.

Also add descriptions for resource management of datagram, which
does not use the existing credit update mechanism associated with
stream sockets.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
---

V2: addressed the comments for the previous version.
V3: add description for the mergeable receive buffer.
V4: add a feature bit for stream and reserver a bit for seqpacket.
Fix mrg_rxbuf related sentences.

 virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 142 insertions(+), 13 deletions(-)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index da7e641..bacac3c 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
 
 \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
 \begin{description}
-\item[0] rx
-\item[1] tx
+\item[0] stream rx
+\item[1] stream tx
+\item[2] datagram rx
+\item[3] datagram tx
+\item[4] event
+\end{description}
+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
+only uses 3 queues, as the following.
+
+\begin{description}
+\item[0] stream rx
+\item[1] stream tx
 \item[2] event
 \end{description}
 
+When behavior differs between stream and datagram rx/tx virtqueues
+their full names are used. Common behavior is simply described in
+terms of rx/tx virtqueues and applies to both stream and datagram
+virtqueues.
+
 \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 
-There are currently no feature bits defined for this device.
+\begin{description}
+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
+\end{description}
+
+\begin{description}
+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket type.
+\end{description}
+
+\begin{description}
+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
+\end{description}
+
+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM is set.
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
 
@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 Packets transmitted or received contain a header before the payload:
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
+
 \begin{lstlisting}
 struct virtio_vsock_hdr {
 	le64 src_cid;
@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 };
 \end{lstlisting}
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
+\begin{lstlisting}
+struct virtio_vsock_hdr_mrg_rxbuf {
+	struct virtio_vsock_hdr hdr;
+	le16 num_buffers;
+};
+\end{lstlisting}
+
+
 The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
 
 Most packets simply transfer data but control packets are also used for
@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
 
+Flow control applies to stream sockets; datagram sockets do not have
+flow control.
+
 The tx virtqueue carries packets initiated by applications and replies to
 received packets.  The rx virtqueue carries packets initiated by the device and
 replies to previously transmitted packets.
@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
 consists of a (cid, port number) tuple. The header fields used for this are
 \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
 
-Currently only stream sockets are supported. \field{type} is 1 for stream
-socket types.
+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
+socket types. \field{type} is 3 for dgram socket types.
 
 Stream sockets provide in-order, guaranteed, connection-oriented delivery
 without message boundaries.
 
+Datagram sockets provide unordered, unreliable, connectionless messages 
+with message boundaries and a maximum length.
+
 \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
 \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
 stream sockets. The guest and the device publish how much buffer space is
@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
 \end{lstlisting}
 
-If there is insufficient buffer space, the sender waits until virtqueue buffers
+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
 are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
 the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
 available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
 communicating updates any time a change in buffer space occurs.
 
+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
+is split to two parts: tx side and rx side. For the tx side, if the 
+virtqueue is full, the packet will be dropped.
+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
+is dropped by the receiver.
+
+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+\begin{itemize}
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
+      with buffers of at least 1526 bytes for stream sockets and 4096 bytes for datagram sockets.
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
+\end{itemize}
+
+\begin{note}
+Obviously each buffer can be split across multiple descriptor elements.
+\end{note}
+
+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+The device MUST set \field{num_buffers} to the number of descriptors used when
+transmitting the  packet.
+
+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
+is not negotiated.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and driver will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and the device will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
-The driver queues outgoing packets on the tx virtqueue and incoming packet
+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
 \begin{lstlisting}
 struct virtio_vsock_packet {
     struct virtio_vsock_hdr hdr;
@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 };
 \end{lstlisting}
 
+Otherwise, use the following form:
+\begin{lstlisting}
+struct virtio_vsock_packet_mrg_rxbuf {
+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
+    u8 data[];
+};
+\end{lstlisting}
+
+
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
+When transmitting packets to the device, \field{num_buffers} is not used.
+
+\begin{enumerate}
+\item \field{num_buffers} indicates how many descriptors
+  this packet is spread over (including this one). 
+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
+  This allows receipt of large packets without having to allocate large
+  buffers: a packet that does not fit in a single buffer can flow
+  over to the next buffer, and so on. In this case, there will be
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
+  chains them together to form a single packet in a way similar to
+  how it would store it in a single buffer spread over multiple
+  descriptors.
+  The other buffers will not begin with a struct virtio_vsock_hdr.
+
+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
+  descriptor is used.
+
+\item If
+  \field{num_buffers} is one, then the entire packet will be
+  contained within this buffer, immediately following the struct
+  virtio_vsock_hdr.
+\end{enumerate}
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
 sending outgoing packets.
 
-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
 \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
 
-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
+\field{num_buffers} to indicate the number of buffers
+the packet (including the header) is spread over.
+
+If a receive packet is spread over multiple buffers, the device
+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
+1$ buffers) completely up to the full length of each buffer
+supplied by the driver.
+
+The device MUST use all buffers used by a single receive
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
+
 \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
 
 Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
 destination) address tuple for a new connection while the other peer is still
 processing the old connection.
 
+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
+
+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
+a receiving socket does not exist on the destination.
+If the transmission or receiving buffers are full, the packets
+are dropped.
+
 \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
 
 Certain events are communicated by the device to the driver using the event
-- 
2.11.0

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang]

From: "jiang.wang" <jiang.wang@bytedance.com>

Add supports for datagram type for virtio-vsock. Datagram
sockets are connectionless and unreliable. To avoid contention
with stream and other sockets, add two more virtqueues and
a new feature bit to identify if those two new queues exist or not.

Also add descriptions for resource management of datagram, which
does not use the existing credit update mechanism associated with
stream sockets.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
---

V2: addressed the comments for the previous version.
V3: add description for the mergeable receive buffer.
V4: add a feature bit for stream and reserver a bit for seqpacket.
Fix mrg_rxbuf related sentences.

 virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 142 insertions(+), 13 deletions(-)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index da7e641..bacac3c 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
 
 \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
 \begin{description}
-\item[0] rx
-\item[1] tx
+\item[0] stream rx
+\item[1] stream tx
+\item[2] datagram rx
+\item[3] datagram tx
+\item[4] event
+\end{description}
+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
+only uses 3 queues, as the following.
+
+\begin{description}
+\item[0] stream rx
+\item[1] stream tx
 \item[2] event
 \end{description}
 
+When behavior differs between stream and datagram rx/tx virtqueues
+their full names are used. Common behavior is simply described in
+terms of rx/tx virtqueues and applies to both stream and datagram
+virtqueues.
+
 \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 
-There are currently no feature bits defined for this device.
+\begin{description}
+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
+\end{description}
+
+\begin{description}
+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket type.
+\end{description}
+
+\begin{description}
+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
+\end{description}
+
+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM is set.
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
 
@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 Packets transmitted or received contain a header before the payload:
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
+
 \begin{lstlisting}
 struct virtio_vsock_hdr {
 	le64 src_cid;
@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 };
 \end{lstlisting}
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
+\begin{lstlisting}
+struct virtio_vsock_hdr_mrg_rxbuf {
+	struct virtio_vsock_hdr hdr;
+	le16 num_buffers;
+};
+\end{lstlisting}
+
+
 The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
 
 Most packets simply transfer data but control packets are also used for
@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
 
+Flow control applies to stream sockets; datagram sockets do not have
+flow control.
+
 The tx virtqueue carries packets initiated by applications and replies to
 received packets.  The rx virtqueue carries packets initiated by the device and
 replies to previously transmitted packets.
@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
 consists of a (cid, port number) tuple. The header fields used for this are
 \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
 
-Currently only stream sockets are supported. \field{type} is 1 for stream
-socket types.
+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
+socket types. \field{type} is 3 for dgram socket types.
 
 Stream sockets provide in-order, guaranteed, connection-oriented delivery
 without message boundaries.
 
+Datagram sockets provide unordered, unreliable, connectionless messages 
+with message boundaries and a maximum length.
+
 \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
 \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
 stream sockets. The guest and the device publish how much buffer space is
@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
 \end{lstlisting}
 
-If there is insufficient buffer space, the sender waits until virtqueue buffers
+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
 are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
 the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
 available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
 communicating updates any time a change in buffer space occurs.
 
+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
+is split to two parts: tx side and rx side. For the tx side, if the 
+virtqueue is full, the packet will be dropped.
+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
+is dropped by the receiver.
+
+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+\begin{itemize}
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
+      with buffers of at least 1526 bytes for stream sockets and 4096 bytes for datagram sockets.
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
+\end{itemize}
+
+\begin{note}
+Obviously each buffer can be split across multiple descriptor elements.
+\end{note}
+
+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+The device MUST set \field{num_buffers} to the number of descriptors used when
+transmitting the  packet.
+
+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
+is not negotiated.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and driver will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and the device will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
-The driver queues outgoing packets on the tx virtqueue and incoming packet
+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
 \begin{lstlisting}
 struct virtio_vsock_packet {
     struct virtio_vsock_hdr hdr;
@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 };
 \end{lstlisting}
 
+Otherwise, use the following form:
+\begin{lstlisting}
+struct virtio_vsock_packet_mrg_rxbuf {
+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
+    u8 data[];
+};
+\end{lstlisting}
+
+
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
+When transmitting packets to the device, \field{num_buffers} is not used.
+
+\begin{enumerate}
+\item \field{num_buffers} indicates how many descriptors
+  this packet is spread over (including this one). 
+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
+  This allows receipt of large packets without having to allocate large
+  buffers: a packet that does not fit in a single buffer can flow
+  over to the next buffer, and so on. In this case, there will be
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
+  chains them together to form a single packet in a way similar to
+  how it would store it in a single buffer spread over multiple
+  descriptors.
+  The other buffers will not begin with a struct virtio_vsock_hdr.
+
+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
+  descriptor is used.
+
+\item If
+  \field{num_buffers} is one, then the entire packet will be
+  contained within this buffer, immediately following the struct
+  virtio_vsock_hdr.
+\end{enumerate}
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
 sending outgoing packets.
 
-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
 \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
 
-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
+\field{num_buffers} to indicate the number of buffers
+the packet (including the header) is spread over.
+
+If a receive packet is spread over multiple buffers, the device
+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
+1$ buffers) completely up to the full length of each buffer
+supplied by the driver.
+
+The device MUST use all buffers used by a single receive
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
+
 \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
 
 Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
 destination) address tuple for a new connection while the other peer is still
 processing the old connection.
 
+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
+
+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
+a receiving socket does not exist on the destination.
+If the transmission or receiving buffers are full, the packets
+are dropped.
+
 \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
 
 Certain events are communicated by the device to the driver using the event
-- 
2.11.0


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

Any comments? Thanks.

On Thu, May 27, 2021 at 9:01 PM Jiang Wang <jiang.wang@bytedance.com> wrote:
>
> From: "jiang.wang" <jiang.wang@bytedance.com>
>
> Add supports for datagram type for virtio-vsock. Datagram
> sockets are connectionless and unreliable. To avoid contention
> with stream and other sockets, add two more virtqueues and
> a new feature bit to identify if those two new queues exist or not.
>
> Also add descriptions for resource management of datagram, which
> does not use the existing credit update mechanism associated with
> stream sockets.
>
> Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> ---
>
> V2: addressed the comments for the previous version.
> V3: add description for the mergeable receive buffer.
> V4: add a feature bit for stream and reserver a bit for seqpacket.
> Fix mrg_rxbuf related sentences.
>
>  virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 142 insertions(+), 13 deletions(-)
>
> diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> index da7e641..bacac3c 100644
> --- a/virtio-vsock.tex
> +++ b/virtio-vsock.tex
> @@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
>  \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>  \begin{description}
> -\item[0] rx
> -\item[1] tx
> +\item[0] stream rx
> +\item[1] stream tx
> +\item[2] datagram rx
> +\item[3] datagram tx
> +\item[4] event
> +\end{description}
> +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> +only uses 3 queues, as the following.
> +
> +\begin{description}
> +\item[0] stream rx
> +\item[1] stream tx
>  \item[2] event
>  \end{description}
>
> +When behavior differs between stream and datagram rx/tx virtqueues
> +their full names are used. Common behavior is simply described in
> +terms of rx/tx virtqueues and applies to both stream and datagram
> +virtqueues.
> +
>  \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
> -There are currently no feature bits defined for this device.
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> +\end{description}
> +
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket type.
> +\end{description}
> +
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> +\end{description}
> +
> +If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM is set.
>
>  \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>
> @@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
>  Packets transmitted or received contain a header before the payload:
>
> +If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> +
>  \begin{lstlisting}
>  struct virtio_vsock_hdr {
>         le64 src_cid;
> @@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>  };
>  \end{lstlisting}
>
> +If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> +\begin{lstlisting}
> +struct virtio_vsock_hdr_mrg_rxbuf {
> +       struct virtio_vsock_hdr hdr;
> +       le16 num_buffers;
> +};
> +\end{lstlisting}
> +
> +
>  The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
>  Most packets simply transfer data but control packets are also used for
> @@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
>  \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
> +Flow control applies to stream sockets; datagram sockets do not have
> +flow control.
> +
>  The tx virtqueue carries packets initiated by applications and replies to
>  received packets.  The rx virtqueue carries packets initiated by the device and
>  replies to previously transmitted packets.
> @@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>  consists of a (cid, port number) tuple. The header fields used for this are
>  \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
> -Currently only stream sockets are supported. \field{type} is 1 for stream
> -socket types.
> +Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> +socket types. \field{type} is 3 for dgram socket types.
>
>  Stream sockets provide in-order, guaranteed, connection-oriented delivery
>  without message boundaries.
>
> +Datagram sockets provide unordered, unreliable, connectionless messages
> +with message boundaries and a maximum length.
> +
>  \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>  \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
>  stream sockets. The guest and the device publish how much buffer space is
> @@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>  u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>  \end{lstlisting}
>
> -If there is insufficient buffer space, the sender waits until virtqueue buffers
> +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>  are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>  the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>  available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> @@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>  previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>  communicating updates any time a change in buffer space occurs.
>
> +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> +VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> +is split to two parts: tx side and rx side. For the tx side, if the
> +virtqueue is full, the packet will be dropped.
> +For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> +is dropped by the receiver.
> +
> +\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> +\begin{itemize}
> +\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> +      with buffers of at least 1526 bytes for stream sockets and 4096 bytes for datagram sockets.
> +\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> +least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
> +\end{itemize}
> +
> +\begin{note}
> +Obviously each buffer can be split across multiple descriptor elements.
> +\end{note}
> +
> +\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> +The device MUST set \field{num_buffers} to the number of descriptors used when
> +transmitting the  packet.
> +
> +The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> +is not negotiated.
> +
>  \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> -sufficient free buffer space for the payload.
> +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> +and driver will not get any notification.
>
>  All packets associated with a stream flow MUST contain valid information in
>  \field{buf_alloc} and \field{fwd_cnt} fields.
>
>  \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> -sufficient free buffer space for the payload.
> +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> +and the device will not get any notification.
>
>  All packets associated with a stream flow MUST contain valid information in
>  \field{buf_alloc} and \field{fwd_cnt} fields.
>
>  \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> -The driver queues outgoing packets on the tx virtqueue and incoming packet
> +The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
>  receive buffers on the rx virtqueue. Packets are of the following form:
>
> +If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>  \begin{lstlisting}
>  struct virtio_vsock_packet {
>      struct virtio_vsock_hdr hdr;
> @@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>  };
>  \end{lstlisting}
>
> +Otherwise, use the following form:
> +\begin{lstlisting}
> +struct virtio_vsock_packet_mrg_rxbuf {
> +    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> +    u8 data[];
> +};
> +\end{lstlisting}
> +
> +
>  Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>  incoming packets are write-only.
>
> +When transmitting packets to the device, \field{num_buffers} is not used.
> +
> +\begin{enumerate}
> +\item \field{num_buffers} indicates how many descriptors
> +  this packet is spread over (including this one).
> +  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> +  This allows receipt of large packets without having to allocate large
> +  buffers: a packet that does not fit in a single buffer can flow
> +  over to the next buffer, and so on. In this case, there will be
> +  at least \field{num_buffers} used buffers in the virtqueue, and the device
> +  chains them together to form a single packet in a way similar to
> +  how it would store it in a single buffer spread over multiple
> +  descriptors.
> +  The other buffers will not begin with a struct virtio_vsock_hdr.
> +
> +  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> +  descriptor is used.
> +
> +\item If
> +  \field{num_buffers} is one, then the entire packet will be
> +  contained within this buffer, immediately following the struct
> +  virtio_vsock_hdr.
> +\end{enumerate}
> +
>  \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
>  The \field{guest_cid} configuration field MUST be used as the source CID when
>  sending outgoing packets.
>
> -A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> +For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>  unknown \field{type} value.
>
>  \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
>  The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>
> -A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> +For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>  unknown \field{type} value.
>
> +If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> +\field{num_buffers} to indicate the number of buffers
> +the packet (including the header) is spread over.
> +
> +If a receive packet is spread over multiple buffers, the device
> +MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> +1$ buffers) completely up to the full length of each buffer
> +supplied by the driver.
> +
> +The device MUST use all buffers used by a single receive
> +packet together, such that at least \field{num_buffers} are
> +observed by driver as used.
> +
>  \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
>  Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> @@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
>  destination) address tuple for a new connection while the other peer is still
>  processing the old connection.
>
> +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> +
> +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> +a receiving socket does not exist on the destination.
> +If the transmission or receiving buffers are full, the packets
> +are dropped.
> +
>  \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>
>  Certain events are communicated by the device to the driver using the event
> --
> 2.11.0
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

Any comments? Thanks.

On Thu, May 27, 2021 at 9:01 PM Jiang Wang <jiang.wang@bytedance.com> wrote:
>
> From: "jiang.wang" <jiang.wang@bytedance.com>
>
> Add supports for datagram type for virtio-vsock. Datagram
> sockets are connectionless and unreliable. To avoid contention
> with stream and other sockets, add two more virtqueues and
> a new feature bit to identify if those two new queues exist or not.
>
> Also add descriptions for resource management of datagram, which
> does not use the existing credit update mechanism associated with
> stream sockets.
>
> Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> ---
>
> V2: addressed the comments for the previous version.
> V3: add description for the mergeable receive buffer.
> V4: add a feature bit for stream and reserver a bit for seqpacket.
> Fix mrg_rxbuf related sentences.
>
>  virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>  1 file changed, 142 insertions(+), 13 deletions(-)
>
> diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> index da7e641..bacac3c 100644
> --- a/virtio-vsock.tex
> +++ b/virtio-vsock.tex
> @@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
>  \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>  \begin{description}
> -\item[0] rx
> -\item[1] tx
> +\item[0] stream rx
> +\item[1] stream tx
> +\item[2] datagram rx
> +\item[3] datagram tx
> +\item[4] event
> +\end{description}
> +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> +only uses 3 queues, as the following.
> +
> +\begin{description}
> +\item[0] stream rx
> +\item[1] stream tx
>  \item[2] event
>  \end{description}
>
> +When behavior differs between stream and datagram rx/tx virtqueues
> +their full names are used. Common behavior is simply described in
> +terms of rx/tx virtqueues and applies to both stream and datagram
> +virtqueues.
> +
>  \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
> -There are currently no feature bits defined for this device.
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> +\end{description}
> +
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket type.
> +\end{description}
> +
> +\begin{description}
> +\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> +\end{description}
> +
> +If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM is set.
>
>  \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>
> @@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
>  Packets transmitted or received contain a header before the payload:
>
> +If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> +
>  \begin{lstlisting}
>  struct virtio_vsock_hdr {
>         le64 src_cid;
> @@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>  };
>  \end{lstlisting}
>
> +If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> +\begin{lstlisting}
> +struct virtio_vsock_hdr_mrg_rxbuf {
> +       struct virtio_vsock_hdr hdr;
> +       le16 num_buffers;
> +};
> +\end{lstlisting}
> +
> +
>  The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
>  Most packets simply transfer data but control packets are also used for
> @@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
>  \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
> +Flow control applies to stream sockets; datagram sockets do not have
> +flow control.
> +
>  The tx virtqueue carries packets initiated by applications and replies to
>  received packets.  The rx virtqueue carries packets initiated by the device and
>  replies to previously transmitted packets.
> @@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>  consists of a (cid, port number) tuple. The header fields used for this are
>  \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
> -Currently only stream sockets are supported. \field{type} is 1 for stream
> -socket types.
> +Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> +socket types. \field{type} is 3 for dgram socket types.
>
>  Stream sockets provide in-order, guaranteed, connection-oriented delivery
>  without message boundaries.
>
> +Datagram sockets provide unordered, unreliable, connectionless messages
> +with message boundaries and a maximum length.
> +
>  \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>  \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
>  stream sockets. The guest and the device publish how much buffer space is
> @@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>  u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>  \end{lstlisting}
>
> -If there is insufficient buffer space, the sender waits until virtqueue buffers
> +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>  are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>  the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>  available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> @@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>  previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>  communicating updates any time a change in buffer space occurs.
>
> +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> +VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> +is split to two parts: tx side and rx side. For the tx side, if the
> +virtqueue is full, the packet will be dropped.
> +For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> +is dropped by the receiver.
> +
> +\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> +\begin{itemize}
> +\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> +      with buffers of at least 1526 bytes for stream sockets and 4096 bytes for datagram sockets.
> +\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> +least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
> +\end{itemize}
> +
> +\begin{note}
> +Obviously each buffer can be split across multiple descriptor elements.
> +\end{note}
> +
> +\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> +The device MUST set \field{num_buffers} to the number of descriptors used when
> +transmitting the  packet.
> +
> +The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> +is not negotiated.
> +
>  \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> -sufficient free buffer space for the payload.
> +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> +and driver will not get any notification.
>
>  All packets associated with a stream flow MUST contain valid information in
>  \field{buf_alloc} and \field{fwd_cnt} fields.
>
>  \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> -sufficient free buffer space for the payload.
> +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> +and the device will not get any notification.
>
>  All packets associated with a stream flow MUST contain valid information in
>  \field{buf_alloc} and \field{fwd_cnt} fields.
>
>  \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> -The driver queues outgoing packets on the tx virtqueue and incoming packet
> +The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
>  receive buffers on the rx virtqueue. Packets are of the following form:
>
> +If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>  \begin{lstlisting}
>  struct virtio_vsock_packet {
>      struct virtio_vsock_hdr hdr;
> @@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>  };
>  \end{lstlisting}
>
> +Otherwise, use the following form:
> +\begin{lstlisting}
> +struct virtio_vsock_packet_mrg_rxbuf {
> +    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> +    u8 data[];
> +};
> +\end{lstlisting}
> +
> +
>  Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>  incoming packets are write-only.
>
> +When transmitting packets to the device, \field{num_buffers} is not used.
> +
> +\begin{enumerate}
> +\item \field{num_buffers} indicates how many descriptors
> +  this packet is spread over (including this one).
> +  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> +  This allows receipt of large packets without having to allocate large
> +  buffers: a packet that does not fit in a single buffer can flow
> +  over to the next buffer, and so on. In this case, there will be
> +  at least \field{num_buffers} used buffers in the virtqueue, and the device
> +  chains them together to form a single packet in a way similar to
> +  how it would store it in a single buffer spread over multiple
> +  descriptors.
> +  The other buffers will not begin with a struct virtio_vsock_hdr.
> +
> +  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> +  descriptor is used.
> +
> +\item If
> +  \field{num_buffers} is one, then the entire packet will be
> +  contained within this buffer, immediately following the struct
> +  virtio_vsock_hdr.
> +\end{enumerate}
> +
>  \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
>  The \field{guest_cid} configuration field MUST be used as the source CID when
>  sending outgoing packets.
>
> -A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> +For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>  unknown \field{type} value.
>
>  \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
>  The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>
> -A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> +For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>  unknown \field{type} value.
>
> +If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> +\field{num_buffers} to indicate the number of buffers
> +the packet (including the header) is spread over.
> +
> +If a receive packet is spread over multiple buffers, the device
> +MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> +1$ buffers) completely up to the full length of each buffer
> +supplied by the driver.
> +
> +The device MUST use all buffers used by a single receive
> +packet together, such that at least \field{num_buffers} are
> +observed by driver as used.
> +
>  \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
>  Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> @@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
>  destination) address tuple for a new connection while the other peer is still
>  processing the old connection.
>
> +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> +
> +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> +a receiving socket does not exist on the destination.
> +If the transmission or receiving buffers are full, the packets
> +are dropped.
> +
>  \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>
>  Certain events are communicated by the device to the driver using the event
> --
> 2.11.0
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>From: "jiang.wang" <jiang.wang@bytedance.com>
>
>Add supports for datagram type for virtio-vsock. Datagram
>sockets are connectionless and unreliable. To avoid contention
>with stream and other sockets, add two more virtqueues and
>a new feature bit to identify if those two new queues exist or not.
>
>Also add descriptions for resource management of datagram, which
>does not use the existing credit update mechanism associated with
>stream sockets.
>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>---
>
>V2: addressed the comments for the previous version.
>V3: add description for the mergeable receive buffer.
>V4: add a feature bit for stream and reserver a bit for seqpacket.
>Fix mrg_rxbuf related sentences.
>
> virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> 1 file changed, 142 insertions(+), 13 deletions(-)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index da7e641..bacac3c 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
> \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> \begin{description}
>-\item[0] rx
>-\item[1] tx
>+\item[0] stream rx
>+\item[1] stream tx
>+\item[2] datagram rx
>+\item[3] datagram tx
>+\item[4] event

Is there a particular reason to always have the event queue as the last 
one?

Maybe it's better to add the datagram queues at the bottom, so the first 
3 queues are always the same.

>+\end{description}
>+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>+only uses 3 queues, as the following.
>+
>+\begin{description}
>+\item[0] stream rx
>+\item[1] stream tx
> \item[2] event
> \end{description}
>
>+When behavior differs between stream and datagram rx/tx virtqueues
>+their full names are used. Common behavior is simply described in
>+terms of rx/tx virtqueues and applies to both stream and datagram
>+virtqueues.
>+
> \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
>-There are currently no feature bits defined for this device.
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>+\end{description}
>+
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket 
>type.
>+\end{description}
>+
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>+\end{description}
>+
>+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM 
>is set.

I'd say more like socket streams are supported, without reference to the 
feature bit, something like: "If no feature bits are defined, then 
assume device only supports stream socket type."

>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>
>@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> Packets transmitted or received contain a header before the payload:
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>+
> \begin{lstlisting}
> struct virtio_vsock_hdr {
> 	le64 src_cid;
>@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> };
> \end{lstlisting}
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>+\begin{lstlisting}
>+struct virtio_vsock_hdr_mrg_rxbuf {
>+	struct virtio_vsock_hdr hdr;
>+	le16 num_buffers;
>+};
>+\end{lstlisting}
>+
>+
> The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
> Most packets simply transfer data but control packets are also used for
>@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
>+Flow control applies to stream sockets; datagram sockets do not have
>+flow control.
>+
> The tx virtqueue carries packets initiated by applications and replies to
> received packets.  The rx virtqueue carries packets initiated by the device and
> replies to previously transmitted packets.
>@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> consists of a (cid, port number) tuple. The header fields used for this are
> \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
>-Currently only stream sockets are supported. \field{type} is 1 for 
>stream
>-socket types.
>+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>+socket types. \field{type} is 3 for dgram socket types.

When Arseny's change will merged, we can define and use 
VIRTIO_VSOCK_TYPE_DGRAM,.

>
> Stream sockets provide in-order, guaranteed, connection-oriented 
> delivery
> without message boundaries.
>
>+Datagram sockets provide unordered, unreliable, connectionless 
>messages
>+with message boundaries and a maximum length.
>+
> \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> \field{buf_alloc} and \field{fwd_cnt} are used for buffer space 
> management of
> stream sockets. The guest and the device publish how much buffer space is
>@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> \end{lstlisting}
>
>-If there is insufficient buffer space, the sender waits until virtqueue buffers
>+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> communicating updates any time a change in buffer space occurs.
>
>+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>+is split to two parts: tx side and rx side. For the tx side, if the

Maybe better to use sender and receiver, since we use tx and rx to 
identify the queues.

>+virtqueue is full, the packet will be dropped.
>+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>+is dropped by the receiver.

This sentence is a bit unclear.
`buf_alloc` for stream socket is used to inform the other peer about the 
receive buffer space. In this case we are using the local information, 
so there is no need to refer to `buf_alloc`. We can write something 
like: "The packet is dropped by the receiver if there is no space in the 
receive buffer".

>+
>+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+\begin{itemize}
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>+      with buffers of at least 1526 bytes for stream sockets and 4096 
>bytes for datagram sockets.

Where does 1526 come from?

We're adding a requirement for socket streams that wasn't there until 
now.

>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>+\end{itemize}
>+
>+\begin{note}
>+Obviously each buffer can be split across multiple descriptor elements.
>+\end{note}
>+
>+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+The device MUST set \field{num_buffers} to the number of descriptors used when
>+transmitting the  packet.
>+
>+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>+is not negotiated.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and driver will not get any notification.
>
> All packets associated with a stream flow MUST contain valid 
> information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and the device will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>-The driver queues outgoing packets on the tx virtqueue and incoming packet
>+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> receive buffers on the rx virtqueue. Packets are of the following form:
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.

Please use present as in the rest of the document,

> \begin{lstlisting}
> struct virtio_vsock_packet {
>     struct virtio_vsock_hdr hdr;
>@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> };
> \end{lstlisting}
>
>+Otherwise, use the following form:

Maybe better to repeat:

If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...

>+\begin{lstlisting}
>+struct virtio_vsock_packet_mrg_rxbuf {
>+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>+    u8 data[];
>+};
>+\end{lstlisting}
>+
>+
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
>+When transmitting packets to the device, \field{num_buffers} is not 
>used.
>+
>+\begin{enumerate}
>+\item \field{num_buffers} indicates how many descriptors
>+  this packet is spread over (including this one).
>+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
>+  This allows receipt of large packets without having to allocate large
>+  buffers: a packet that does not fit in a single buffer can flow
>+  over to the next buffer, and so on. In this case, there will be
>+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>+  chains them together to form a single packet in a way similar to
>+  how it would store it in a single buffer spread over multiple
>+  descriptors.
>+  The other buffers will not begin with a struct virtio_vsock_hdr.
>+
>+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
>+  descriptor is used.
>+
>+\item If
>+  \field{num_buffers} is one, then the entire packet will be
>+  contained within this buffer, immediately following the struct
>+  virtio_vsock_hdr.
>+\end{enumerate}
>+
> \drivernormative{\paragraph}{Device Operation: Receive and 
> Transmit}{Device Types / Socket Device / Device Operation / Receive 
> and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
> sending outgoing packets.
>
>-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be 

Perhaps we can leave it as it was before, since the `type` field 
identifies the socket type itself.

>sent if a packet is received with an
> unknown \field{type} value.
>
> \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>
>-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> unknown \field{type} value.

Ditto.

>
>+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>+\field{num_buffers} to indicate the number of buffers
>+the packet (including the header) is spread over.
>+
>+If a receive packet is spread over multiple buffers, the device
>+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>+1$ buffers) completely up to the full length of each buffer
>+supplied by the driver.
>+
>+The device MUST use all buffers used by a single receive
>+packet together, such that at least \field{num_buffers} are
>+observed by driver as used.
>+
> \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
> Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> destination) address tuple for a new connection while the other peer is still
> processing the old connection.
>
>+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>+
>+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>+a receiving socket does not exist on the destination.
>+If the transmission or receiving buffers are full, the packets
>+are dropped.
>+
> \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>
> Certain events are communicated by the device to the driver using the event
>-- 
>2.11.0
>

I don't know if maybe it's better to break this patch in two, one where 
we add datagram and one for mergeable buffer.

But let's see what others think.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>From: "jiang.wang" <jiang.wang@bytedance.com>
>
>Add supports for datagram type for virtio-vsock. Datagram
>sockets are connectionless and unreliable. To avoid contention
>with stream and other sockets, add two more virtqueues and
>a new feature bit to identify if those two new queues exist or not.
>
>Also add descriptions for resource management of datagram, which
>does not use the existing credit update mechanism associated with
>stream sockets.
>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>---
>
>V2: addressed the comments for the previous version.
>V3: add description for the mergeable receive buffer.
>V4: add a feature bit for stream and reserver a bit for seqpacket.
>Fix mrg_rxbuf related sentences.
>
> virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> 1 file changed, 142 insertions(+), 13 deletions(-)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index da7e641..bacac3c 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
> \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> \begin{description}
>-\item[0] rx
>-\item[1] tx
>+\item[0] stream rx
>+\item[1] stream tx
>+\item[2] datagram rx
>+\item[3] datagram tx
>+\item[4] event

Is there a particular reason to always have the event queue as the last 
one?

Maybe it's better to add the datagram queues at the bottom, so the first 
3 queues are always the same.

>+\end{description}
>+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>+only uses 3 queues, as the following.
>+
>+\begin{description}
>+\item[0] stream rx
>+\item[1] stream tx
> \item[2] event
> \end{description}
>
>+When behavior differs between stream and datagram rx/tx virtqueues
>+their full names are used. Common behavior is simply described in
>+terms of rx/tx virtqueues and applies to both stream and datagram
>+virtqueues.
>+
> \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
>-There are currently no feature bits defined for this device.
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>+\end{description}
>+
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket 
>type.
>+\end{description}
>+
>+\begin{description}
>+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>+\end{description}
>+
>+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM 
>is set.

I'd say more like socket streams are supported, without reference to the 
feature bit, something like: "If no feature bits are defined, then 
assume device only supports stream socket type."

>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>
>@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> Packets transmitted or received contain a header before the payload:
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>+
> \begin{lstlisting}
> struct virtio_vsock_hdr {
> 	le64 src_cid;
>@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> };
> \end{lstlisting}
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>+\begin{lstlisting}
>+struct virtio_vsock_hdr_mrg_rxbuf {
>+	struct virtio_vsock_hdr hdr;
>+	le16 num_buffers;
>+};
>+\end{lstlisting}
>+
>+
> The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
> Most packets simply transfer data but control packets are also used for
>@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
>+Flow control applies to stream sockets; datagram sockets do not have
>+flow control.
>+
> The tx virtqueue carries packets initiated by applications and replies to
> received packets.  The rx virtqueue carries packets initiated by the device and
> replies to previously transmitted packets.
>@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> consists of a (cid, port number) tuple. The header fields used for this are
> \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
>-Currently only stream sockets are supported. \field{type} is 1 for 
>stream
>-socket types.
>+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>+socket types. \field{type} is 3 for dgram socket types.

When Arseny's change will merged, we can define and use 
VIRTIO_VSOCK_TYPE_DGRAM,.

>
> Stream sockets provide in-order, guaranteed, connection-oriented 
> delivery
> without message boundaries.
>
>+Datagram sockets provide unordered, unreliable, connectionless 
>messages
>+with message boundaries and a maximum length.
>+
> \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> \field{buf_alloc} and \field{fwd_cnt} are used for buffer space 
> management of
> stream sockets. The guest and the device publish how much buffer space is
>@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> \end{lstlisting}
>
>-If there is insufficient buffer space, the sender waits until virtqueue buffers
>+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> communicating updates any time a change in buffer space occurs.
>
>+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>+is split to two parts: tx side and rx side. For the tx side, if the

Maybe better to use sender and receiver, since we use tx and rx to 
identify the queues.

>+virtqueue is full, the packet will be dropped.
>+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>+is dropped by the receiver.

This sentence is a bit unclear.
`buf_alloc` for stream socket is used to inform the other peer about the 
receive buffer space. In this case we are using the local information, 
so there is no need to refer to `buf_alloc`. We can write something 
like: "The packet is dropped by the receiver if there is no space in the 
receive buffer".

>+
>+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+\begin{itemize}
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>+      with buffers of at least 1526 bytes for stream sockets and 4096 
>bytes for datagram sockets.

Where does 1526 come from?

We're adding a requirement for socket streams that wasn't there until 
now.

>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>+\end{itemize}
>+
>+\begin{note}
>+Obviously each buffer can be split across multiple descriptor elements.
>+\end{note}
>+
>+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+The device MUST set \field{num_buffers} to the number of descriptors used when
>+transmitting the  packet.
>+
>+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>+is not negotiated.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and driver will not get any notification.
>
> All packets associated with a stream flow MUST contain valid 
> information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and the device will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>-The driver queues outgoing packets on the tx virtqueue and incoming packet
>+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> receive buffers on the rx virtqueue. Packets are of the following form:
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.

Please use present as in the rest of the document,

> \begin{lstlisting}
> struct virtio_vsock_packet {
>     struct virtio_vsock_hdr hdr;
>@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> };
> \end{lstlisting}
>
>+Otherwise, use the following form:

Maybe better to repeat:

If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...

>+\begin{lstlisting}
>+struct virtio_vsock_packet_mrg_rxbuf {
>+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>+    u8 data[];
>+};
>+\end{lstlisting}
>+
>+
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
>+When transmitting packets to the device, \field{num_buffers} is not 
>used.
>+
>+\begin{enumerate}
>+\item \field{num_buffers} indicates how many descriptors
>+  this packet is spread over (including this one).
>+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
>+  This allows receipt of large packets without having to allocate large
>+  buffers: a packet that does not fit in a single buffer can flow
>+  over to the next buffer, and so on. In this case, there will be
>+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>+  chains them together to form a single packet in a way similar to
>+  how it would store it in a single buffer spread over multiple
>+  descriptors.
>+  The other buffers will not begin with a struct virtio_vsock_hdr.
>+
>+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
>+  descriptor is used.
>+
>+\item If
>+  \field{num_buffers} is one, then the entire packet will be
>+  contained within this buffer, immediately following the struct
>+  virtio_vsock_hdr.
>+\end{enumerate}
>+
> \drivernormative{\paragraph}{Device Operation: Receive and 
> Transmit}{Device Types / Socket Device / Device Operation / Receive 
> and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
> sending outgoing packets.
>
>-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be 

Perhaps we can leave it as it was before, since the `type` field 
identifies the socket type itself.

>sent if a packet is received with an
> unknown \field{type} value.
>
> \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>
>-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> unknown \field{type} value.

Ditto.

>
>+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>+\field{num_buffers} to indicate the number of buffers
>+the packet (including the header) is spread over.
>+
>+If a receive packet is spread over multiple buffers, the device
>+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>+1$ buffers) completely up to the full length of each buffer
>+supplied by the driver.
>+
>+The device MUST use all buffers used by a single receive
>+packet together, such that at least \field{num_buffers} are
>+observed by driver as used.
>+
> \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
> Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> destination) address tuple for a new connection while the other peer is still
> processing the old connection.
>
>+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>+
>+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>+a receiving socket does not exist on the destination.
>+If the transmission or receiving buffers are full, the packets
>+are dropped.
>+
> \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>
> Certain events are communicated by the device to the driver using the event
>-- 
>2.11.0
>

I don't know if maybe it's better to break this patch in two, one where 
we add datagram and one for mergeable buffer.

But let's see what others think.

Thanks,
Stefano


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
> >From: "jiang.wang" <jiang.wang@bytedance.com>
> >
> >Add supports for datagram type for virtio-vsock. Datagram
> >sockets are connectionless and unreliable. To avoid contention
> >with stream and other sockets, add two more virtqueues and
> >a new feature bit to identify if those two new queues exist or not.
> >
> >Also add descriptions for resource management of datagram, which
> >does not use the existing credit update mechanism associated with
> >stream sockets.
> >
> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> >---
> >
> >V2: addressed the comments for the previous version.
> >V3: add description for the mergeable receive buffer.
> >V4: add a feature bit for stream and reserver a bit for seqpacket.
> >Fix mrg_rxbuf related sentences.
> >
> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> > 1 file changed, 142 insertions(+), 13 deletions(-)
> >
> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> >index da7e641..bacac3c 100644
> >--- a/virtio-vsock.tex
> >+++ b/virtio-vsock.tex
> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> >
> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> > \begin{description}
> >-\item[0] rx
> >-\item[1] tx
> >+\item[0] stream rx
> >+\item[1] stream tx
> >+\item[2] datagram rx
> >+\item[3] datagram tx
> >+\item[4] event
>
> Is there a particular reason to always have the event queue as the last
> one?
>
> Maybe it's better to add the datagram queues at the bottom, so the first
> 3 queues are always the same.
>
I am not sure. I think Linux kernel should be fine with what you described.
But I am not sure about QEMU. From the code, I see virtqueue is allocated
as an array, like following,

+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
+    struct vhost_virtqueue vhost_vqs[4];
+ #else
    struct vhost_virtqueue vhost_vqs[2];
+ #endi

so I assume the virtqueues for tx/rx should be
continuous? I can try to put the new queues at the end and see if it
works or not.

btw, my qemu change is here:
https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84

> >+\end{description}
> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> >+only uses 3 queues, as the following.
> >+
> >+\begin{description}
> >+\item[0] stream rx
> >+\item[1] stream tx
> > \item[2] event
> > \end{description}
> >
> >+When behavior differs between stream and datagram rx/tx virtqueues
> >+their full names are used. Common behavior is simply described in
> >+terms of rx/tx virtqueues and applies to both stream and datagram
> >+virtqueues.
> >+
> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> >
> >-There are currently no feature bits defined for this device.
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> >+\end{description}
> >+
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
> >type.
> >+\end{description}
> >+
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> >+\end{description}
> >+
> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
> >is set.
>
> I'd say more like socket streams are supported, without reference to the
> feature bit, something like: "If no feature bits are defined, then
> assume device only supports stream socket type."
>
OK.

> >
> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
> >
> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >
> > Packets transmitted or received contain a header before the payload:
> >
> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> >+
> > \begin{lstlisting}
> > struct virtio_vsock_hdr {
> >       le64 src_cid;
> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> > };
> > \end{lstlisting}
> >
> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> >+\begin{lstlisting}
> >+struct virtio_vsock_hdr_mrg_rxbuf {
> >+      struct virtio_vsock_hdr hdr;
> >+      le16 num_buffers;
> >+};
> >+\end{lstlisting}
> >+
> >+
> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
> >
> > Most packets simply transfer data but control packets are also used for
> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >
> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
> >
> >+Flow control applies to stream sockets; datagram sockets do not have
> >+flow control.
> >+
> > The tx virtqueue carries packets initiated by applications and replies to
> > received packets.  The rx virtqueue carries packets initiated by the device and
> > replies to previously transmitted packets.
> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> > consists of a (cid, port number) tuple. The header fields used for this are
> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
> >
> >-Currently only stream sockets are supported. \field{type} is 1 for
> >stream
> >-socket types.
> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> >+socket types. \field{type} is 3 for dgram socket types.
>
> When Arseny's change will merged, we can define and use
> VIRTIO_VSOCK_TYPE_DGRAM,.

Sure.
> >
> > Stream sockets provide in-order, guaranteed, connection-oriented
> > delivery
> > without message boundaries.
> >
> >+Datagram sockets provide unordered, unreliable, connectionless
> >messages
> >+with message boundaries and a maximum length.
> >+
> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
> > management of
> > stream sockets. The guest and the device publish how much buffer space is
> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> > \end{lstlisting}
> >
> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> > communicating updates any time a change in buffer space occurs.
> >
> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> >+is split to two parts: tx side and rx side. For the tx side, if the
>
> Maybe better to use sender and receiver, since we use tx and rx to
> identify the queues.

OK.

> >+virtqueue is full, the packet will be dropped.
> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> >+is dropped by the receiver.
>
> This sentence is a bit unclear.
> `buf_alloc` for stream socket is used to inform the other peer about the
> receive buffer space. In this case we are using the local information,
> so there is no need to refer to `buf_alloc`. We can write something
> like: "The packet is dropped by the receiver if there is no space in the
> receive buffer".

OK.

> >+
> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >+\begin{itemize}
> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> >+      with buffers of at least 1526 bytes for stream sockets and 4096
> >bytes for datagram sockets.
>
> Where does 1526 come from?

No specific reason. Any recommendations?

> We're adding a requirement for socket streams that wasn't there until
> now.

This is only when mergeable rxbuf bit is used. I think before this, the stream
rx buf should be at least bigger than the pkt header. We just did not put that
into the spec.

> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
> >+\end{itemize}
> >+
> >+\begin{note}
> >+Obviously each buffer can be split across multiple descriptor elements.
> >+\end{note}
> >+
> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >+The device MUST set \field{num_buffers} to the number of descriptors used when
> >+transmitting the  packet.
> >+
> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> >+is not negotiated.
> >+
> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >-sufficient free buffer space for the payload.
> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >+and driver will not get any notification.
> >
> > All packets associated with a stream flow MUST contain valid
> > information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >
> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >-sufficient free buffer space for the payload.
> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >+and the device will not get any notification.
> >
> > All packets associated with a stream flow MUST contain valid information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >
> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> >-The driver queues outgoing packets on the tx virtqueue and incoming packet
> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> > receive buffers on the rx virtqueue. Packets are of the following form:
> >
> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>
> Please use present as in the rest of the document,

Sure. I see both past tense and present tense in the spec, so I was a little bit
confused.

> > \begin{lstlisting}
> > struct virtio_vsock_packet {
> >     struct virtio_vsock_hdr hdr;
> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> > };
> > \end{lstlisting}
> >
> >+Otherwise, use the following form:
>
> Maybe better to repeat:
>
> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
>
OK.

> >+\begin{lstlisting}
> >+struct virtio_vsock_packet_mrg_rxbuf {
> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> >+    u8 data[];
> >+};
> >+\end{lstlisting}
> >+
> >+
> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> > incoming packets are write-only.
> >
> >+When transmitting packets to the device, \field{num_buffers} is not
> >used.
> >+
> >+\begin{enumerate}
> >+\item \field{num_buffers} indicates how many descriptors
> >+  this packet is spread over (including this one).
> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> >+  This allows receipt of large packets without having to allocate large
> >+  buffers: a packet that does not fit in a single buffer can flow
> >+  over to the next buffer, and so on. In this case, there will be
> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
> >+  chains them together to form a single packet in a way similar to
> >+  how it would store it in a single buffer spread over multiple
> >+  descriptors.
> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
> >+
> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> >+  descriptor is used.
> >+
> >+\item If
> >+  \field{num_buffers} is one, then the entire packet will be
> >+  contained within this buffer, immediately following the struct
> >+  virtio_vsock_hdr.
> >+\end{enumerate}
> >+
> > \drivernormative{\paragraph}{Device Operation: Receive and
> > Transmit}{Device Types / Socket Device / Device Operation / Receive
> > and Transmit}
> >
> > The \field{guest_cid} configuration field MUST be used as the source CID when
> > sending outgoing packets.
> >
> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
>
> Perhaps we can leave it as it was before, since the `type` field
> identifies the socket type itself.

OK.

> >sent if a packet is received with an
> > unknown \field{type} value.
> >
> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> >
> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
> >
> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> > unknown \field{type} value.
>
> Ditto.

OK.

> >
> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> >+\field{num_buffers} to indicate the number of buffers
> >+the packet (including the header) is spread over.
> >+
> >+If a receive packet is spread over multiple buffers, the device
> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> >+1$ buffers) completely up to the full length of each buffer
> >+supplied by the driver.
> >+
> >+The device MUST use all buffers used by a single receive
> >+packet together, such that at least \field{num_buffers} are
> >+observed by driver as used.
> >+
> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
> >
> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> > destination) address tuple for a new connection while the other peer is still
> > processing the old connection.
> >
> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> >+
> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> >+a receiving socket does not exist on the destination.
> >+If the transmission or receiving buffers are full, the packets
> >+are dropped.
> >+
> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
> >
> > Certain events are communicated by the device to the driver using the event
> >--
> >2.11.0
> >
>
> I don't know if maybe it's better to break this patch in two, one where
> we add datagram and one for mergeable buffer.
>
> But let's see what others think.

OK. I can definitely break it to two patches if necessary. Thanks for
all the comments.

Regards,

Jiang

>
> Thanks,
> Stefano
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
> >From: "jiang.wang" <jiang.wang@bytedance.com>
> >
> >Add supports for datagram type for virtio-vsock. Datagram
> >sockets are connectionless and unreliable. To avoid contention
> >with stream and other sockets, add two more virtqueues and
> >a new feature bit to identify if those two new queues exist or not.
> >
> >Also add descriptions for resource management of datagram, which
> >does not use the existing credit update mechanism associated with
> >stream sockets.
> >
> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> >---
> >
> >V2: addressed the comments for the previous version.
> >V3: add description for the mergeable receive buffer.
> >V4: add a feature bit for stream and reserver a bit for seqpacket.
> >Fix mrg_rxbuf related sentences.
> >
> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> > 1 file changed, 142 insertions(+), 13 deletions(-)
> >
> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> >index da7e641..bacac3c 100644
> >--- a/virtio-vsock.tex
> >+++ b/virtio-vsock.tex
> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> >
> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> > \begin{description}
> >-\item[0] rx
> >-\item[1] tx
> >+\item[0] stream rx
> >+\item[1] stream tx
> >+\item[2] datagram rx
> >+\item[3] datagram tx
> >+\item[4] event
>
> Is there a particular reason to always have the event queue as the last
> one?
>
> Maybe it's better to add the datagram queues at the bottom, so the first
> 3 queues are always the same.
>
I am not sure. I think Linux kernel should be fine with what you described.
But I am not sure about QEMU. From the code, I see virtqueue is allocated
as an array, like following,

+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
+    struct vhost_virtqueue vhost_vqs[4];
+ #else
    struct vhost_virtqueue vhost_vqs[2];
+ #endi

so I assume the virtqueues for tx/rx should be
continuous? I can try to put the new queues at the end and see if it
works or not.

btw, my qemu change is here:
https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84

> >+\end{description}
> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> >+only uses 3 queues, as the following.
> >+
> >+\begin{description}
> >+\item[0] stream rx
> >+\item[1] stream tx
> > \item[2] event
> > \end{description}
> >
> >+When behavior differs between stream and datagram rx/tx virtqueues
> >+their full names are used. Common behavior is simply described in
> >+terms of rx/tx virtqueues and applies to both stream and datagram
> >+virtqueues.
> >+
> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> >
> >-There are currently no feature bits defined for this device.
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> >+\end{description}
> >+
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
> >type.
> >+\end{description}
> >+
> >+\begin{description}
> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> >+\end{description}
> >+
> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
> >is set.
>
> I'd say more like socket streams are supported, without reference to the
> feature bit, something like: "If no feature bits are defined, then
> assume device only supports stream socket type."
>
OK.

> >
> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
> >
> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >
> > Packets transmitted or received contain a header before the payload:
> >
> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> >+
> > \begin{lstlisting}
> > struct virtio_vsock_hdr {
> >       le64 src_cid;
> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> > };
> > \end{lstlisting}
> >
> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> >+\begin{lstlisting}
> >+struct virtio_vsock_hdr_mrg_rxbuf {
> >+      struct virtio_vsock_hdr hdr;
> >+      le16 num_buffers;
> >+};
> >+\end{lstlisting}
> >+
> >+
> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
> >
> > Most packets simply transfer data but control packets are also used for
> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >
> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
> >
> >+Flow control applies to stream sockets; datagram sockets do not have
> >+flow control.
> >+
> > The tx virtqueue carries packets initiated by applications and replies to
> > received packets.  The rx virtqueue carries packets initiated by the device and
> > replies to previously transmitted packets.
> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> > consists of a (cid, port number) tuple. The header fields used for this are
> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
> >
> >-Currently only stream sockets are supported. \field{type} is 1 for
> >stream
> >-socket types.
> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> >+socket types. \field{type} is 3 for dgram socket types.
>
> When Arseny's change will merged, we can define and use
> VIRTIO_VSOCK_TYPE_DGRAM,.

Sure.
> >
> > Stream sockets provide in-order, guaranteed, connection-oriented
> > delivery
> > without message boundaries.
> >
> >+Datagram sockets provide unordered, unreliable, connectionless
> >messages
> >+with message boundaries and a maximum length.
> >+
> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
> > management of
> > stream sockets. The guest and the device publish how much buffer space is
> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> > \end{lstlisting}
> >
> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> > communicating updates any time a change in buffer space occurs.
> >
> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> >+is split to two parts: tx side and rx side. For the tx side, if the
>
> Maybe better to use sender and receiver, since we use tx and rx to
> identify the queues.

OK.

> >+virtqueue is full, the packet will be dropped.
> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> >+is dropped by the receiver.
>
> This sentence is a bit unclear.
> `buf_alloc` for stream socket is used to inform the other peer about the
> receive buffer space. In this case we are using the local information,
> so there is no need to refer to `buf_alloc`. We can write something
> like: "The packet is dropped by the receiver if there is no space in the
> receive buffer".

OK.

> >+
> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >+\begin{itemize}
> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> >+      with buffers of at least 1526 bytes for stream sockets and 4096
> >bytes for datagram sockets.
>
> Where does 1526 come from?

No specific reason. Any recommendations?

> We're adding a requirement for socket streams that wasn't there until
> now.

This is only when mergeable rxbuf bit is used. I think before this, the stream
rx buf should be at least bigger than the pkt header. We just did not put that
into the spec.

> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
> >+\end{itemize}
> >+
> >+\begin{note}
> >+Obviously each buffer can be split across multiple descriptor elements.
> >+\end{note}
> >+
> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >+The device MUST set \field{num_buffers} to the number of descriptors used when
> >+transmitting the  packet.
> >+
> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> >+is not negotiated.
> >+
> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >-sufficient free buffer space for the payload.
> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >+and driver will not get any notification.
> >
> > All packets associated with a stream flow MUST contain valid
> > information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >
> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >-sufficient free buffer space for the payload.
> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >+and the device will not get any notification.
> >
> > All packets associated with a stream flow MUST contain valid information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >
> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> >-The driver queues outgoing packets on the tx virtqueue and incoming packet
> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> > receive buffers on the rx virtqueue. Packets are of the following form:
> >
> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>
> Please use present as in the rest of the document,

Sure. I see both past tense and present tense in the spec, so I was a little bit
confused.

> > \begin{lstlisting}
> > struct virtio_vsock_packet {
> >     struct virtio_vsock_hdr hdr;
> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> > };
> > \end{lstlisting}
> >
> >+Otherwise, use the following form:
>
> Maybe better to repeat:
>
> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
>
OK.

> >+\begin{lstlisting}
> >+struct virtio_vsock_packet_mrg_rxbuf {
> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> >+    u8 data[];
> >+};
> >+\end{lstlisting}
> >+
> >+
> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> > incoming packets are write-only.
> >
> >+When transmitting packets to the device, \field{num_buffers} is not
> >used.
> >+
> >+\begin{enumerate}
> >+\item \field{num_buffers} indicates how many descriptors
> >+  this packet is spread over (including this one).
> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> >+  This allows receipt of large packets without having to allocate large
> >+  buffers: a packet that does not fit in a single buffer can flow
> >+  over to the next buffer, and so on. In this case, there will be
> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
> >+  chains them together to form a single packet in a way similar to
> >+  how it would store it in a single buffer spread over multiple
> >+  descriptors.
> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
> >+
> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> >+  descriptor is used.
> >+
> >+\item If
> >+  \field{num_buffers} is one, then the entire packet will be
> >+  contained within this buffer, immediately following the struct
> >+  virtio_vsock_hdr.
> >+\end{enumerate}
> >+
> > \drivernormative{\paragraph}{Device Operation: Receive and
> > Transmit}{Device Types / Socket Device / Device Operation / Receive
> > and Transmit}
> >
> > The \field{guest_cid} configuration field MUST be used as the source CID when
> > sending outgoing packets.
> >
> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
>
> Perhaps we can leave it as it was before, since the `type` field
> identifies the socket type itself.

OK.

> >sent if a packet is received with an
> > unknown \field{type} value.
> >
> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> >
> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
> >
> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> > unknown \field{type} value.
>
> Ditto.

OK.

> >
> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> >+\field{num_buffers} to indicate the number of buffers
> >+the packet (including the header) is spread over.
> >+
> >+If a receive packet is spread over multiple buffers, the device
> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> >+1$ buffers) completely up to the full length of each buffer
> >+supplied by the driver.
> >+
> >+The device MUST use all buffers used by a single receive
> >+packet together, such that at least \field{num_buffers} are
> >+observed by driver as used.
> >+
> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
> >
> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> > destination) address tuple for a new connection while the other peer is still
> > processing the old connection.
> >
> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> >+
> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> >+a receiving socket does not exist on the destination.
> >+If the transmission or receiving buffers are full, the packets
> >+are dropped.
> >+
> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
> >
> > Certain events are communicated by the device to the driver using the event
> >--
> >2.11.0
> >
>
> I don't know if maybe it's better to break this patch in two, one where
> we add datagram and one for mergeable buffer.
>
> But let's see what others think.

OK. I can definitely break it to two patches if necessary. Thanks for
all the comments.

Regards,

Jiang

>
> Thanks,
> Stefano
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
>On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>> >From: "jiang.wang" <jiang.wang@bytedance.com>
>> >
>> >Add supports for datagram type for virtio-vsock. Datagram
>> >sockets are connectionless and unreliable. To avoid contention
>> >with stream and other sockets, add two more virtqueues and
>> >a new feature bit to identify if those two new queues exist or not.
>> >
>> >Also add descriptions for resource management of datagram, which
>> >does not use the existing credit update mechanism associated with
>> >stream sockets.
>> >
>> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>> >---
>> >
>> >V2: addressed the comments for the previous version.
>> >V3: add description for the mergeable receive buffer.
>> >V4: add a feature bit for stream and reserver a bit for seqpacket.
>> >Fix mrg_rxbuf related sentences.
>> >
>> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>> > 1 file changed, 142 insertions(+), 13 deletions(-)
>> >
>> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> >index da7e641..bacac3c 100644
>> >--- a/virtio-vsock.tex
>> >+++ b/virtio-vsock.tex
>> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >
>> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> > \begin{description}
>> >-\item[0] rx
>> >-\item[1] tx
>> >+\item[0] stream rx
>> >+\item[1] stream tx
>> >+\item[2] datagram rx
>> >+\item[3] datagram tx
>> >+\item[4] event
>>
>> Is there a particular reason to always have the event queue as the last
>> one?
>>
>> Maybe it's better to add the datagram queues at the bottom, so the first
>> 3 queues are always the same.
>>
>I am not sure. I think Linux kernel should be fine with what you described.
>But I am not sure about QEMU. From the code, I see virtqueue is allocated
>as an array, like following,
>
>+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
>+    struct vhost_virtqueue vhost_vqs[4];
>+ #else
>    struct vhost_virtqueue vhost_vqs[2];
>+ #endi

I see, also vhost_dev_init() requires an array, so I agree that this is 
the best approach, sorry for the noise.

Just to be sure to check that anything is working if 
CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver 
that doesn't support DGRAM, and viceversa.

>
>so I assume the virtqueues for tx/rx should be
>continuous? I can try to put the new queues at the end and see if it
>works or not.
>
>btw, my qemu change is here:
>https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
>
>> >+\end{description}
>> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> >+only uses 3 queues, as the following.
>> >+
>> >+\begin{description}
>> >+\item[0] stream rx
>> >+\item[1] stream tx
>> > \item[2] event
>> > \end{description}
>> >
>> >+When behavior differs between stream and datagram rx/tx virtqueues
>> >+their full names are used. Common behavior is simply described in
>> >+terms of rx/tx virtqueues and applies to both stream and datagram
>> >+virtqueues.
>> >+
>> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>> >
>> >-There are currently no feature bits defined for this device.
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>> >+\end{description}
>> >+
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
>> >type.
>> >+\end{description}
>> >+
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>> >+\end{description}
>> >+
>> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
>> >is set.
>>
>> I'd say more like socket streams are supported, without reference to the
>> feature bit, something like: "If no feature bits are defined, then
>> assume device only supports stream socket type."
>>
>OK.
>
>> >
>> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>> >
>> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >
>> > Packets transmitted or received contain a header before the payload:
>> >
>> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>> >+
>> > \begin{lstlisting}
>> > struct virtio_vsock_hdr {
>> >       le64 src_cid;
>> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> > };
>> > \end{lstlisting}
>> >
>> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>> >+\begin{lstlisting}
>> >+struct virtio_vsock_hdr_mrg_rxbuf {
>> >+      struct virtio_vsock_hdr hdr;
>> >+      le16 num_buffers;
>> >+};
>> >+\end{lstlisting}
>> >+
>> >+
>> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>> >
>> > Most packets simply transfer data but control packets are also used for
>> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >
>> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>> >
>> >+Flow control applies to stream sockets; datagram sockets do not have
>> >+flow control.
>> >+
>> > The tx virtqueue carries packets initiated by applications and replies to
>> > received packets.  The rx virtqueue carries packets initiated by the device and
>> > replies to previously transmitted packets.
>> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>> > consists of a (cid, port number) tuple. The header fields used for this are
>> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>> >
>> >-Currently only stream sockets are supported. \field{type} is 1 for
>> >stream
>> >-socket types.
>> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>> >+socket types. \field{type} is 3 for dgram socket types.
>>
>> When Arseny's change will merged, we can define and use
>> VIRTIO_VSOCK_TYPE_DGRAM,.
>
>Sure.
>> >
>> > Stream sockets provide in-order, guaranteed, connection-oriented
>> > delivery
>> > without message boundaries.
>> >
>> >+Datagram sockets provide unordered, unreliable, connectionless
>> >messages
>> >+with message boundaries and a maximum length.
>> >+
>> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
>> > management of
>> > stream sockets. The guest and the device publish how much buffer space is
>> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> > \end{lstlisting}
>> >
>> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
>> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> > communicating updates any time a change in buffer space occurs.
>> >
>> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>> >+is split to two parts: tx side and rx side. For the tx side, if the
>>
>> Maybe better to use sender and receiver, since we use tx and rx to
>> identify the queues.
>
>OK.
>
>> >+virtqueue is full, the packet will be dropped.
>> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>> >+is dropped by the receiver.
>>
>> This sentence is a bit unclear.
>> `buf_alloc` for stream socket is used to inform the other peer about the
>> receive buffer space. In this case we are using the local information,
>> so there is no need to refer to `buf_alloc`. We can write something
>> like: "The packet is dropped by the receiver if there is no space in the
>> receive buffer".
>
>OK.
>
>> >+
>> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >+\begin{itemize}
>> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>> >+      with buffers of at least 1526 bytes for stream sockets and 4096
>> >bytes for datagram sockets.
>>
>> Where does 1526 come from?
>
>No specific reason. Any recommendations?
>
>> We're adding a requirement for socket streams that wasn't there until
>> now.
>
>This is only when mergeable rxbuf bit is used. I think before this, the stream
>rx buf should be at least bigger than the pkt header. We just did not put that
>into the spec.

Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
is not negotiated"
    ^

Is it a typo?


IIUC when VIRTIO_VSOCK_F_MRG_RXBUF is negotiated the minimum buffer size 
is virtio_vsock_hdr_mgr_rxbuf...

>
>> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.

... from this ^ statement.

And I agree that rx buf should be at least the virtio_vsock_hdr size.

>> >+\end{itemize}
>> >+
>> >+\begin{note}
>> >+Obviously each buffer can be split across multiple descriptor 
>> >elements.
>> >+\end{note}
>> >+
>> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >+The device MUST set \field{num_buffers} to the number of descriptors used when
>> >+transmitting the  packet.
>> >+
>> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>> >+is not negotiated.
>> >+
>> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >-sufficient free buffer space for the payload.
>> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> >+and driver will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid
>> > information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >-sufficient free buffer space for the payload.
>> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >+sufficient free buffer space for the payload. For dgram sockets, 
>> >VIRTIO_VSOCK_OP_RW data packets
>> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> >+and the device will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >-The driver queues outgoing packets on the tx virtqueue and incoming 
>> >packet
>> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
>> > receive buffers on the rx virtqueue. Packets are of the following form:
>> >
>> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>>
>> Please use present as in the rest of the document,
>
>Sure. I see both past tense and present tense in the spec, so I was a 
>little bit confused.

Sorry about that. I generally always saw the present tense and seemed to 
use it everywhere.

>
>> > \begin{lstlisting}
>> > struct virtio_vsock_packet {
>> >     struct virtio_vsock_hdr hdr;
>> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>> > };
>> > \end{lstlisting}
>> >
>> >+Otherwise, use the following form:
>>
>> Maybe better to repeat:
>>
>> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
>>
>OK.
>
>> >+\begin{lstlisting}
>> >+struct virtio_vsock_packet_mrg_rxbuf {
>> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>> >+    u8 data[];
>> >+};
>> >+\end{lstlisting}
>> >+
>> >+
>> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>> > incoming packets are write-only.
>> >
>> >+When transmitting packets to the device, \field{num_buffers} is not
>> >used.
>> >+
>> >+\begin{enumerate}
>> >+\item \field{num_buffers} indicates how many descriptors
>> >+  this packet is spread over (including this one).
>> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
>> >+  This allows receipt of large packets without having to allocate large
>> >+  buffers: a packet that does not fit in a single buffer can flow
>> >+  over to the next buffer, and so on. In this case, there will be
>> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>> >+  chains them together to form a single packet in a way similar to
>> >+  how it would store it in a single buffer spread over multiple
>> >+  descriptors.
>> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
>> >+
>> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
>> >+  descriptor is used.
>> >+
>> >+\item If
>> >+  \field{num_buffers} is one, then the entire packet will be
>> >+  contained within this buffer, immediately following the struct
>> >+  virtio_vsock_hdr.
>> >+\end{enumerate}
>> >+
>> > \drivernormative{\paragraph}{Device Operation: Receive and
>> > Transmit}{Device Types / Socket Device / Device Operation / Receive
>> > and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST be used as the source CID when
>> > sending outgoing packets.
>> >
>> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
>>
>> Perhaps we can leave it as it was before, since the `type` field
>> identifies the socket type itself.
>
>OK.
>
>> >sent if a packet is received with an
>> > unknown \field{type} value.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>> >
>> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST 
>> >be sent if a packet is received with an
>> > unknown \field{type} value.
>>
>> Ditto.
>
>OK.
>
>> >
>> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>> >+\field{num_buffers} to indicate the number of buffers
>> >+the packet (including the header) is spread over.
>> >+
>> >+If a receive packet is spread over multiple buffers, the device
>> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>> >+1$ buffers) completely up to the full length of each buffer
>> >+supplied by the driver.
>> >+
>> >+The device MUST use all buffers used by a single receive
>> >+packet together, such that at least \field{num_buffers} are
>> >+observed by driver as used.
>> >+
>> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket 
>> > Device / Device Operation / Stream Sockets}
>> >
>> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
>> > destination) address tuple for a new connection while the other peer is still
>> > processing the old connection.
>> >
>> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>> >+
>> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>> >+a receiving socket does not exist on the destination.
>> >+If the transmission or receiving buffers are full, the packets
>> >+are dropped.
>> >+
>> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>> >
>> > Certain events are communicated by the device to the driver using the event
>> >--
>> >2.11.0
>> >
>>
>> I don't know if maybe it's better to break this patch in two, one 
>> where
>> we add datagram and one for mergeable buffer.
>>
>> But let's see what others think.
>
>OK. I can definitely break it to two patches if necessary. Thanks for
>all the comments.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
>On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>> >From: "jiang.wang" <jiang.wang@bytedance.com>
>> >
>> >Add supports for datagram type for virtio-vsock. Datagram
>> >sockets are connectionless and unreliable. To avoid contention
>> >with stream and other sockets, add two more virtqueues and
>> >a new feature bit to identify if those two new queues exist or not.
>> >
>> >Also add descriptions for resource management of datagram, which
>> >does not use the existing credit update mechanism associated with
>> >stream sockets.
>> >
>> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>> >---
>> >
>> >V2: addressed the comments for the previous version.
>> >V3: add description for the mergeable receive buffer.
>> >V4: add a feature bit for stream and reserver a bit for seqpacket.
>> >Fix mrg_rxbuf related sentences.
>> >
>> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>> > 1 file changed, 142 insertions(+), 13 deletions(-)
>> >
>> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> >index da7e641..bacac3c 100644
>> >--- a/virtio-vsock.tex
>> >+++ b/virtio-vsock.tex
>> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >
>> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> > \begin{description}
>> >-\item[0] rx
>> >-\item[1] tx
>> >+\item[0] stream rx
>> >+\item[1] stream tx
>> >+\item[2] datagram rx
>> >+\item[3] datagram tx
>> >+\item[4] event
>>
>> Is there a particular reason to always have the event queue as the last
>> one?
>>
>> Maybe it's better to add the datagram queues at the bottom, so the first
>> 3 queues are always the same.
>>
>I am not sure. I think Linux kernel should be fine with what you described.
>But I am not sure about QEMU. From the code, I see virtqueue is allocated
>as an array, like following,
>
>+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
>+    struct vhost_virtqueue vhost_vqs[4];
>+ #else
>    struct vhost_virtqueue vhost_vqs[2];
>+ #endi

I see, also vhost_dev_init() requires an array, so I agree that this is 
the best approach, sorry for the noise.

Just to be sure to check that anything is working if 
CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver 
that doesn't support DGRAM, and viceversa.

>
>so I assume the virtqueues for tx/rx should be
>continuous? I can try to put the new queues at the end and see if it
>works or not.
>
>btw, my qemu change is here:
>https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
>
>> >+\end{description}
>> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> >+only uses 3 queues, as the following.
>> >+
>> >+\begin{description}
>> >+\item[0] stream rx
>> >+\item[1] stream tx
>> > \item[2] event
>> > \end{description}
>> >
>> >+When behavior differs between stream and datagram rx/tx virtqueues
>> >+their full names are used. Common behavior is simply described in
>> >+terms of rx/tx virtqueues and applies to both stream and datagram
>> >+virtqueues.
>> >+
>> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>> >
>> >-There are currently no feature bits defined for this device.
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>> >+\end{description}
>> >+
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
>> >type.
>> >+\end{description}
>> >+
>> >+\begin{description}
>> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>> >+\end{description}
>> >+
>> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
>> >is set.
>>
>> I'd say more like socket streams are supported, without reference to the
>> feature bit, something like: "If no feature bits are defined, then
>> assume device only supports stream socket type."
>>
>OK.
>
>> >
>> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>> >
>> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >
>> > Packets transmitted or received contain a header before the payload:
>> >
>> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>> >+
>> > \begin{lstlisting}
>> > struct virtio_vsock_hdr {
>> >       le64 src_cid;
>> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> > };
>> > \end{lstlisting}
>> >
>> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>> >+\begin{lstlisting}
>> >+struct virtio_vsock_hdr_mrg_rxbuf {
>> >+      struct virtio_vsock_hdr hdr;
>> >+      le16 num_buffers;
>> >+};
>> >+\end{lstlisting}
>> >+
>> >+
>> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>> >
>> > Most packets simply transfer data but control packets are also used for
>> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >
>> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>> >
>> >+Flow control applies to stream sockets; datagram sockets do not have
>> >+flow control.
>> >+
>> > The tx virtqueue carries packets initiated by applications and replies to
>> > received packets.  The rx virtqueue carries packets initiated by the device and
>> > replies to previously transmitted packets.
>> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>> > consists of a (cid, port number) tuple. The header fields used for this are
>> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>> >
>> >-Currently only stream sockets are supported. \field{type} is 1 for
>> >stream
>> >-socket types.
>> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>> >+socket types. \field{type} is 3 for dgram socket types.
>>
>> When Arseny's change will merged, we can define and use
>> VIRTIO_VSOCK_TYPE_DGRAM,.
>
>Sure.
>> >
>> > Stream sockets provide in-order, guaranteed, connection-oriented
>> > delivery
>> > without message boundaries.
>> >
>> >+Datagram sockets provide unordered, unreliable, connectionless
>> >messages
>> >+with message boundaries and a maximum length.
>> >+
>> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
>> > management of
>> > stream sockets. The guest and the device publish how much buffer space is
>> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> > \end{lstlisting}
>> >
>> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
>> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> > communicating updates any time a change in buffer space occurs.
>> >
>> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>> >+is split to two parts: tx side and rx side. For the tx side, if the
>>
>> Maybe better to use sender and receiver, since we use tx and rx to
>> identify the queues.
>
>OK.
>
>> >+virtqueue is full, the packet will be dropped.
>> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>> >+is dropped by the receiver.
>>
>> This sentence is a bit unclear.
>> `buf_alloc` for stream socket is used to inform the other peer about the
>> receive buffer space. In this case we are using the local information,
>> so there is no need to refer to `buf_alloc`. We can write something
>> like: "The packet is dropped by the receiver if there is no space in the
>> receive buffer".
>
>OK.
>
>> >+
>> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >+\begin{itemize}
>> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>> >+      with buffers of at least 1526 bytes for stream sockets and 4096
>> >bytes for datagram sockets.
>>
>> Where does 1526 come from?
>
>No specific reason. Any recommendations?
>
>> We're adding a requirement for socket streams that wasn't there until
>> now.
>
>This is only when mergeable rxbuf bit is used. I think before this, the stream
>rx buf should be at least bigger than the pkt header. We just did not put that
>into the spec.

Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
is not negotiated"
    ^

Is it a typo?


IIUC when VIRTIO_VSOCK_F_MRG_RXBUF is negotiated the minimum buffer size 
is virtio_vsock_hdr_mgr_rxbuf...

>
>> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.

... from this ^ statement.

And I agree that rx buf should be at least the virtio_vsock_hdr size.

>> >+\end{itemize}
>> >+
>> >+\begin{note}
>> >+Obviously each buffer can be split across multiple descriptor 
>> >elements.
>> >+\end{note}
>> >+
>> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >+The device MUST set \field{num_buffers} to the number of descriptors used when
>> >+transmitting the  packet.
>> >+
>> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>> >+is not negotiated.
>> >+
>> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >-sufficient free buffer space for the payload.
>> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> >+and driver will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid
>> > information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >-sufficient free buffer space for the payload.
>> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> >+sufficient free buffer space for the payload. For dgram sockets, 
>> >VIRTIO_VSOCK_OP_RW data packets
>> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> >+and the device will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >-The driver queues outgoing packets on the tx virtqueue and incoming 
>> >packet
>> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
>> > receive buffers on the rx virtqueue. Packets are of the following form:
>> >
>> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
>>
>> Please use present as in the rest of the document,
>
>Sure. I see both past tense and present tense in the spec, so I was a 
>little bit confused.

Sorry about that. I generally always saw the present tense and seemed to 
use it everywhere.

>
>> > \begin{lstlisting}
>> > struct virtio_vsock_packet {
>> >     struct virtio_vsock_hdr hdr;
>> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>> > };
>> > \end{lstlisting}
>> >
>> >+Otherwise, use the following form:
>>
>> Maybe better to repeat:
>>
>> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
>>
>OK.
>
>> >+\begin{lstlisting}
>> >+struct virtio_vsock_packet_mrg_rxbuf {
>> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>> >+    u8 data[];
>> >+};
>> >+\end{lstlisting}
>> >+
>> >+
>> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>> > incoming packets are write-only.
>> >
>> >+When transmitting packets to the device, \field{num_buffers} is not
>> >used.
>> >+
>> >+\begin{enumerate}
>> >+\item \field{num_buffers} indicates how many descriptors
>> >+  this packet is spread over (including this one).
>> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
>> >+  This allows receipt of large packets without having to allocate large
>> >+  buffers: a packet that does not fit in a single buffer can flow
>> >+  over to the next buffer, and so on. In this case, there will be
>> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>> >+  chains them together to form a single packet in a way similar to
>> >+  how it would store it in a single buffer spread over multiple
>> >+  descriptors.
>> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
>> >+
>> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
>> >+  descriptor is used.
>> >+
>> >+\item If
>> >+  \field{num_buffers} is one, then the entire packet will be
>> >+  contained within this buffer, immediately following the struct
>> >+  virtio_vsock_hdr.
>> >+\end{enumerate}
>> >+
>> > \drivernormative{\paragraph}{Device Operation: Receive and
>> > Transmit}{Device Types / Socket Device / Device Operation / Receive
>> > and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST be used as the source CID when
>> > sending outgoing packets.
>> >
>> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
>>
>> Perhaps we can leave it as it was before, since the `type` field
>> identifies the socket type itself.
>
>OK.
>
>> >sent if a packet is received with an
>> > unknown \field{type} value.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
>> >
>> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
>> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST 
>> >be sent if a packet is received with an
>> > unknown \field{type} value.
>>
>> Ditto.
>
>OK.
>
>> >
>> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>> >+\field{num_buffers} to indicate the number of buffers
>> >+the packet (including the header) is spread over.
>> >+
>> >+If a receive packet is spread over multiple buffers, the device
>> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>> >+1$ buffers) completely up to the full length of each buffer
>> >+supplied by the driver.
>> >+
>> >+The device MUST use all buffers used by a single receive
>> >+packet together, such that at least \field{num_buffers} are
>> >+observed by driver as used.
>> >+
>> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket 
>> > Device / Device Operation / Stream Sockets}
>> >
>> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
>> > destination) address tuple for a new connection while the other peer is still
>> > processing the old connection.
>> >
>> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>> >+
>> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>> >+a receiving socket does not exist on the destination.
>> >+If the transmission or receiving buffers are full, the packets
>> >+are dropped.
>> >+
>> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
>> >
>> > Certain events are communicated by the device to the driver using the event
>> >--
>> >2.11.0
>> >
>>
>> I don't know if maybe it's better to break this patch in two, one 
>> where
>> we add datagram and one for mergeable buffer.
>>
>> But let's see what others think.
>
>OK. I can definitely break it to two patches if necessary. Thanks for
>all the comments.

Thanks,
Stefano


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

On Wed, Jun 9, 2021 at 12:17 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
> >On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >>
> >> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
> >> >From: "jiang.wang" <jiang.wang@bytedance.com>
> >> >
> >> >Add supports for datagram type for virtio-vsock. Datagram
> >> >sockets are connectionless and unreliable. To avoid contention
> >> >with stream and other sockets, add two more virtqueues and
> >> >a new feature bit to identify if those two new queues exist or not.
> >> >
> >> >Also add descriptions for resource management of datagram, which
> >> >does not use the existing credit update mechanism associated with
> >> >stream sockets.
> >> >
> >> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> >> >---
> >> >
> >> >V2: addressed the comments for the previous version.
> >> >V3: add description for the mergeable receive buffer.
> >> >V4: add a feature bit for stream and reserver a bit for seqpacket.
> >> >Fix mrg_rxbuf related sentences.
> >> >
> >> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> >> > 1 file changed, 142 insertions(+), 13 deletions(-)
> >> >
> >> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> >> >index da7e641..bacac3c 100644
> >> >--- a/virtio-vsock.tex
> >> >+++ b/virtio-vsock.tex
> >> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> >> >
> >> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> >> > \begin{description}
> >> >-\item[0] rx
> >> >-\item[1] tx
> >> >+\item[0] stream rx
> >> >+\item[1] stream tx
> >> >+\item[2] datagram rx
> >> >+\item[3] datagram tx
> >> >+\item[4] event
> >>
> >> Is there a particular reason to always have the event queue as the last
> >> one?
> >>
> >> Maybe it's better to add the datagram queues at the bottom, so the first
> >> 3 queues are always the same.
> >>
> >I am not sure. I think Linux kernel should be fine with what you described.
> >But I am not sure about QEMU. From the code, I see virtqueue is allocated
> >as an array, like following,
> >
> >+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
> >+    struct vhost_virtqueue vhost_vqs[4];
> >+ #else
> >    struct vhost_virtqueue vhost_vqs[2];
> >+ #endi
>
> I see, also vhost_dev_init() requires an array, so I agree that this is
> the best approach, sorry for the noise.
>
> Just to be sure to check that anything is working if
> CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver
> that doesn't support DGRAM, and viceversa.

Sure.  I just want to mention that the QEMU should be consistent
with the device (host). If QEMU enabled CONFIG_VHOST_VSOCK_DGRAM,
the device also needs to enable a similar option. Than the driver can
be old or new versions.

> >
> >so I assume the virtqueues for tx/rx should be
> >continuous? I can try to put the new queues at the end and see if it
> >works or not.
> >
> >btw, my qemu change is here:
> >https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
> >
> >> >+\end{description}
> >> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> >> >+only uses 3 queues, as the following.
> >> >+
> >> >+\begin{description}
> >> >+\item[0] stream rx
> >> >+\item[1] stream tx
> >> > \item[2] event
> >> > \end{description}
> >> >
> >> >+When behavior differs between stream and datagram rx/tx virtqueues
> >> >+their full names are used. Common behavior is simply described in
> >> >+terms of rx/tx virtqueues and applies to both stream and datagram
> >> >+virtqueues.
> >> >+
> >> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> >> >
> >> >-There are currently no feature bits defined for this device.
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> >> >+\end{description}
> >> >+
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
> >> >type.
> >> >+\end{description}
> >> >+
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> >> >+\end{description}
> >> >+
> >> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
> >> >is set.
> >>
> >> I'd say more like socket streams are supported, without reference to the
> >> feature bit, something like: "If no feature bits are defined, then
> >> assume device only supports stream socket type."
> >>
> >OK.
> >
> >> >
> >> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
> >> >
> >> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> >
> >> > Packets transmitted or received contain a header before the payload:
> >> >
> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> >> >+
> >> > \begin{lstlisting}
> >> > struct virtio_vsock_hdr {
> >> >       le64 src_cid;
> >> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> > };
> >> > \end{lstlisting}
> >> >
> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> >> >+\begin{lstlisting}
> >> >+struct virtio_vsock_hdr_mrg_rxbuf {
> >> >+      struct virtio_vsock_hdr hdr;
> >> >+      le16 num_buffers;
> >> >+};
> >> >+\end{lstlisting}
> >> >+
> >> >+
> >> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
> >> >
> >> > Most packets simply transfer data but control packets are also used for
> >> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> >
> >> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
> >> >
> >> >+Flow control applies to stream sockets; datagram sockets do not have
> >> >+flow control.
> >> >+
> >> > The tx virtqueue carries packets initiated by applications and replies to
> >> > received packets.  The rx virtqueue carries packets initiated by the device and
> >> > replies to previously transmitted packets.
> >> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> >> > consists of a (cid, port number) tuple. The header fields used for this are
> >> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
> >> >
> >> >-Currently only stream sockets are supported. \field{type} is 1 for
> >> >stream
> >> >-socket types.
> >> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> >> >+socket types. \field{type} is 3 for dgram socket types.
> >>
> >> When Arseny's change will merged, we can define and use
> >> VIRTIO_VSOCK_TYPE_DGRAM,.
> >
> >Sure.
> >> >
> >> > Stream sockets provide in-order, guaranteed, connection-oriented
> >> > delivery
> >> > without message boundaries.
> >> >
> >> >+Datagram sockets provide unordered, unreliable, connectionless
> >> >messages
> >> >+with message boundaries and a maximum length.
> >> >+
> >> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
> >> > management of
> >> > stream sockets. The guest and the device publish how much buffer space is
> >> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> >> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> >> > \end{lstlisting}
> >> >
> >> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
> >> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> >> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> >> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> >> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> >> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> >> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> >> > communicating updates any time a change in buffer space occurs.
> >> >
> >> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> >> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> >> >+is split to two parts: tx side and rx side. For the tx side, if the
> >>
> >> Maybe better to use sender and receiver, since we use tx and rx to
> >> identify the queues.
> >
> >OK.
> >
> >> >+virtqueue is full, the packet will be dropped.
> >> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> >> >+is dropped by the receiver.
> >>
> >> This sentence is a bit unclear.
> >> `buf_alloc` for stream socket is used to inform the other peer about the
> >> receive buffer space. In this case we are using the local information,
> >> so there is no need to refer to `buf_alloc`. We can write something
> >> like: "The packet is dropped by the receiver if there is no space in the
> >> receive buffer".
> >
> >OK.
> >
> >> >+
> >> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >> >+\begin{itemize}
> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> >> >+      with buffers of at least 1526 bytes for stream sockets and 4096
> >> >bytes for datagram sockets.
> >>
> >> Where does 1526 come from?
> >
> >No specific reason. Any recommendations?
> >
> >> We're adding a requirement for socket streams that wasn't there until
> >> now.
> >
> >This is only when mergeable rxbuf bit is used. I think before this, the stream
> >rx buf should be at least bigger than the pkt header. We just did not put that
> >into the spec.
>
> Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
> is not negotiated"
>     ^
My bad. You are right, this is a new requirement for stream sockets.
Not a typo.  Maybe remove the part for stream sockets? Something
like following:

If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD
populate the receive queue(s)
with buffers of  at least 4096 bytes for datagram sockets.

> Is it a typo?
>
>
> IIUC when VIRTIO_VSOCK_F_MRG_RXBUF is negotiated the minimum buffer size
> is virtio_vsock_hdr_mgr_rxbuf...
>
> >
> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> >> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>
> ... from this ^ statement.
>
> And I agree that rx buf should be at least the virtio_vsock_hdr size.
>
> >> >+\end{itemize}
> >> >+
> >> >+\begin{note}
> >> >+Obviously each buffer can be split across multiple descriptor
> >> >elements.
> >> >+\end{note}
> >> >+
> >> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >> >+The device MUST set \field{num_buffers} to the number of descriptors used when
> >> >+transmitting the  packet.
> >> >+
> >> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> >> >+is not negotiated.
> >> >+
> >> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >-sufficient free buffer space for the payload.
> >> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >> >+and driver will not get any notification.
> >> >
> >> > All packets associated with a stream flow MUST contain valid
> >> > information in
> >> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >> >
> >> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >-sufficient free buffer space for the payload.
> >> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >+sufficient free buffer space for the payload. For dgram sockets,
> >> >VIRTIO_VSOCK_OP_RW data packets
> >> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >> >+and the device will not get any notification.
> >> >
> >> > All packets associated with a stream flow MUST contain valid information in
> >> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >> >
> >> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> >> >-The driver queues outgoing packets on the tx virtqueue and incoming
> >> >packet
> >> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> >> > receive buffers on the rx virtqueue. Packets are of the following form:
> >> >
> >> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
> >>
> >> Please use present as in the rest of the document,
> >
> >Sure. I see both past tense and present tense in the spec, so I was a
> >little bit confused.
>
> Sorry about that. I generally always saw the present tense and seemed to
> use it everywhere.
>
Sure. No problem.

> >> > \begin{lstlisting}
> >> > struct virtio_vsock_packet {
> >> >     struct virtio_vsock_hdr hdr;
> >> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> >> > };
> >> > \end{lstlisting}
> >> >
> >> >+Otherwise, use the following form:
> >>
> >> Maybe better to repeat:
> >>
> >> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
> >>
> >OK.
> >
> >> >+\begin{lstlisting}
> >> >+struct virtio_vsock_packet_mrg_rxbuf {
> >> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> >> >+    u8 data[];
> >> >+};
> >> >+\end{lstlisting}
> >> >+
> >> >+
> >> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> >> > incoming packets are write-only.
> >> >
> >> >+When transmitting packets to the device, \field{num_buffers} is not
> >> >used.
> >> >+
> >> >+\begin{enumerate}
> >> >+\item \field{num_buffers} indicates how many descriptors
> >> >+  this packet is spread over (including this one).
> >> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> >> >+  This allows receipt of large packets without having to allocate large
> >> >+  buffers: a packet that does not fit in a single buffer can flow
> >> >+  over to the next buffer, and so on. In this case, there will be
> >> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
> >> >+  chains them together to form a single packet in a way similar to
> >> >+  how it would store it in a single buffer spread over multiple
> >> >+  descriptors.
> >> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
> >> >+
> >> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> >> >+  descriptor is used.
> >> >+
> >> >+\item If
> >> >+  \field{num_buffers} is one, then the entire packet will be
> >> >+  contained within this buffer, immediately following the struct
> >> >+  virtio_vsock_hdr.
> >> >+\end{enumerate}
> >> >+
> >> > \drivernormative{\paragraph}{Device Operation: Receive and
> >> > Transmit}{Device Types / Socket Device / Device Operation / Receive
> >> > and Transmit}
> >> >
> >> > The \field{guest_cid} configuration field MUST be used as the source CID when
> >> > sending outgoing packets.
> >> >
> >> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
> >>
> >> Perhaps we can leave it as it was before, since the `type` field
> >> identifies the socket type itself.
> >
> >OK.
> >
> >> >sent if a packet is received with an
> >> > unknown \field{type} value.
> >> >
> >> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> >> >
> >> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
> >> >
> >> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST
> >> >be sent if a packet is received with an
> >> > unknown \field{type} value.
> >>
> >> Ditto.
> >
> >OK.
> >
> >> >
> >> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> >> >+\field{num_buffers} to indicate the number of buffers
> >> >+the packet (including the header) is spread over.
> >> >+
> >> >+If a receive packet is spread over multiple buffers, the device
> >> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> >> >+1$ buffers) completely up to the full length of each buffer
> >> >+supplied by the driver.
> >> >+
> >> >+The device MUST use all buffers used by a single receive
> >> >+packet together, such that at least \field{num_buffers} are
> >> >+observed by driver as used.
> >> >+
> >> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket
> >> > Device / Device Operation / Stream Sockets}
> >> >
> >> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> >> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> >> > destination) address tuple for a new connection while the other peer is still
> >> > processing the old connection.
> >> >
> >> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> >> >+
> >> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> >> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> >> >+a receiving socket does not exist on the destination.
> >> >+If the transmission or receiving buffers are full, the packets
> >> >+are dropped.
> >> >+
> >> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
> >> >
> >> > Certain events are communicated by the device to the driver using the event
> >> >--
> >> >2.11.0
> >> >
> >>
> >> I don't know if maybe it's better to break this patch in two, one
> >> where
> >> we add datagram and one for mergeable buffer.
> >>
> >> But let's see what others think.
> >
> >OK. I can definitely break it to two patches if necessary. Thanks for
> >all the comments.
>
> Thanks,
> Stefano
>
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Jiang Wang .]

On Wed, Jun 9, 2021 at 12:17 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>
> On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
> >On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
> >>
> >> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
> >> >From: "jiang.wang" <jiang.wang@bytedance.com>
> >> >
> >> >Add supports for datagram type for virtio-vsock. Datagram
> >> >sockets are connectionless and unreliable. To avoid contention
> >> >with stream and other sockets, add two more virtqueues and
> >> >a new feature bit to identify if those two new queues exist or not.
> >> >
> >> >Also add descriptions for resource management of datagram, which
> >> >does not use the existing credit update mechanism associated with
> >> >stream sockets.
> >> >
> >> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
> >> >---
> >> >
> >> >V2: addressed the comments for the previous version.
> >> >V3: add description for the mergeable receive buffer.
> >> >V4: add a feature bit for stream and reserver a bit for seqpacket.
> >> >Fix mrg_rxbuf related sentences.
> >> >
> >> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
> >> > 1 file changed, 142 insertions(+), 13 deletions(-)
> >> >
> >> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> >> >index da7e641..bacac3c 100644
> >> >--- a/virtio-vsock.tex
> >> >+++ b/virtio-vsock.tex
> >> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> >> >
> >> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> >> > \begin{description}
> >> >-\item[0] rx
> >> >-\item[1] tx
> >> >+\item[0] stream rx
> >> >+\item[1] stream tx
> >> >+\item[2] datagram rx
> >> >+\item[3] datagram tx
> >> >+\item[4] event
> >>
> >> Is there a particular reason to always have the event queue as the last
> >> one?
> >>
> >> Maybe it's better to add the datagram queues at the bottom, so the first
> >> 3 queues are always the same.
> >>
> >I am not sure. I think Linux kernel should be fine with what you described.
> >But I am not sure about QEMU. From the code, I see virtqueue is allocated
> >as an array, like following,
> >
> >+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
> >+    struct vhost_virtqueue vhost_vqs[4];
> >+ #else
> >    struct vhost_virtqueue vhost_vqs[2];
> >+ #endi
>
> I see, also vhost_dev_init() requires an array, so I agree that this is
> the best approach, sorry for the noise.
>
> Just to be sure to check that anything is working if
> CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver
> that doesn't support DGRAM, and viceversa.

Sure.  I just want to mention that the QEMU should be consistent
with the device (host). If QEMU enabled CONFIG_VHOST_VSOCK_DGRAM,
the device also needs to enable a similar option. Than the driver can
be old or new versions.

> >
> >so I assume the virtqueues for tx/rx should be
> >continuous? I can try to put the new queues at the end and see if it
> >works or not.
> >
> >btw, my qemu change is here:
> >https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
> >
> >> >+\end{description}
> >> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> >> >+only uses 3 queues, as the following.
> >> >+
> >> >+\begin{description}
> >> >+\item[0] stream rx
> >> >+\item[1] stream tx
> >> > \item[2] event
> >> > \end{description}
> >> >
> >> >+When behavior differs between stream and datagram rx/tx virtqueues
> >> >+their full names are used. Common behavior is simply described in
> >> >+terms of rx/tx virtqueues and applies to both stream and datagram
> >> >+virtqueues.
> >> >+
> >> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> >> >
> >> >-There are currently no feature bits defined for this device.
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
> >> >+\end{description}
> >> >+
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
> >> >type.
> >> >+\end{description}
> >> >+
> >> >+\begin{description}
> >> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
> >> >+\end{description}
> >> >+
> >> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
> >> >is set.
> >>
> >> I'd say more like socket streams are supported, without reference to the
> >> feature bit, something like: "If no feature bits are defined, then
> >> assume device only supports stream socket type."
> >>
> >OK.
> >
> >> >
> >> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
> >> >
> >> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> >
> >> > Packets transmitted or received contain a header before the payload:
> >> >
> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
> >> >+
> >> > \begin{lstlisting}
> >> > struct virtio_vsock_hdr {
> >> >       le64 src_cid;
> >> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> > };
> >> > \end{lstlisting}
> >> >
> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
> >> >+\begin{lstlisting}
> >> >+struct virtio_vsock_hdr_mrg_rxbuf {
> >> >+      struct virtio_vsock_hdr hdr;
> >> >+      le16 num_buffers;
> >> >+};
> >> >+\end{lstlisting}
> >> >+
> >> >+
> >> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
> >> >
> >> > Most packets simply transfer data but control packets are also used for
> >> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> >> >
> >> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
> >> >
> >> >+Flow control applies to stream sockets; datagram sockets do not have
> >> >+flow control.
> >> >+
> >> > The tx virtqueue carries packets initiated by applications and replies to
> >> > received packets.  The rx virtqueue carries packets initiated by the device and
> >> > replies to previously transmitted packets.
> >> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> >> > consists of a (cid, port number) tuple. The header fields used for this are
> >> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
> >> >
> >> >-Currently only stream sockets are supported. \field{type} is 1 for
> >> >stream
> >> >-socket types.
> >> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
> >> >+socket types. \field{type} is 3 for dgram socket types.
> >>
> >> When Arseny's change will merged, we can define and use
> >> VIRTIO_VSOCK_TYPE_DGRAM,.
> >
> >Sure.
> >> >
> >> > Stream sockets provide in-order, guaranteed, connection-oriented
> >> > delivery
> >> > without message boundaries.
> >> >
> >> >+Datagram sockets provide unordered, unreliable, connectionless
> >> >messages
> >> >+with message boundaries and a maximum length.
> >> >+
> >> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
> >> > management of
> >> > stream sockets. The guest and the device publish how much buffer space is
> >> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> >> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> >> > \end{lstlisting}
> >> >
> >> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
> >> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> >> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> >> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> >> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> >> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> >> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> >> > communicating updates any time a change in buffer space occurs.
> >> >
> >> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
> >> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
> >> >+is split to two parts: tx side and rx side. For the tx side, if the
> >>
> >> Maybe better to use sender and receiver, since we use tx and rx to
> >> identify the queues.
> >
> >OK.
> >
> >> >+virtqueue is full, the packet will be dropped.
> >> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
> >> >+is dropped by the receiver.
> >>
> >> This sentence is a bit unclear.
> >> `buf_alloc` for stream socket is used to inform the other peer about the
> >> receive buffer space. In this case we are using the local information,
> >> so there is no need to refer to `buf_alloc`. We can write something
> >> like: "The packet is dropped by the receiver if there is no space in the
> >> receive buffer".
> >
> >OK.
> >
> >> >+
> >> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >> >+\begin{itemize}
> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
> >> >+      with buffers of at least 1526 bytes for stream sockets and 4096
> >> >bytes for datagram sockets.
> >>
> >> Where does 1526 come from?
> >
> >No specific reason. Any recommendations?
> >
> >> We're adding a requirement for socket streams that wasn't there until
> >> now.
> >
> >This is only when mergeable rxbuf bit is used. I think before this, the stream
> >rx buf should be at least bigger than the pkt header. We just did not put that
> >into the spec.
>
> Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
> is not negotiated"
>     ^
My bad. You are right, this is a new requirement for stream sockets.
Not a typo.  Maybe remove the part for stream sockets? Something
like following:

If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD
populate the receive queue(s)
with buffers of  at least 4096 bytes for datagram sockets.

> Is it a typo?
>
>
> IIUC when VIRTIO_VSOCK_F_MRG_RXBUF is negotiated the minimum buffer size
> is virtio_vsock_hdr_mgr_rxbuf...
>
> >
> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
> >> >+least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>
> ... from this ^ statement.
>
> And I agree that rx buf should be at least the virtio_vsock_hdr size.
>
> >> >+\end{itemize}
> >> >+
> >> >+\begin{note}
> >> >+Obviously each buffer can be split across multiple descriptor
> >> >elements.
> >> >+\end{note}
> >> >+
> >> >+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
> >> >+The device MUST set \field{num_buffers} to the number of descriptors used when
> >> >+transmitting the  packet.
> >> >+
> >> >+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
> >> >+is not negotiated.
> >> >+
> >> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >-sufficient free buffer space for the payload.
> >> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> >> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >> >+and driver will not get any notification.
> >> >
> >> > All packets associated with a stream flow MUST contain valid
> >> > information in
> >> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >> >
> >> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> >> >-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >-sufficient free buffer space for the payload.
> >> >+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> >> >+sufficient free buffer space for the payload. For dgram sockets,
> >> >VIRTIO_VSOCK_OP_RW data packets
> >> >+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> >> >+and the device will not get any notification.
> >> >
> >> > All packets associated with a stream flow MUST contain valid information in
> >> > \field{buf_alloc} and \field{fwd_cnt} fields.
> >> >
> >> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> >> >-The driver queues outgoing packets on the tx virtqueue and incoming
> >> >packet
> >> >+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> >> > receive buffers on the rx virtqueue. Packets are of the following form:
> >> >
> >> >+If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, use the following.
> >>
> >> Please use present as in the rest of the document,
> >
> >Sure. I see both past tense and present tense in the spec, so I was a
> >little bit confused.
>
> Sorry about that. I generally always saw the present tense and seemed to
> use it everywhere.
>
Sure. No problem.

> >> > \begin{lstlisting}
> >> > struct virtio_vsock_packet {
> >> >     struct virtio_vsock_hdr hdr;
> >> >@@ -195,24 +270,70 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> >> > };
> >> > \end{lstlisting}
> >> >
> >> >+Otherwise, use the following form:
> >>
> >> Maybe better to repeat:
> >>
> >> If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated...
> >>
> >OK.
> >
> >> >+\begin{lstlisting}
> >> >+struct virtio_vsock_packet_mrg_rxbuf {
> >> >+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
> >> >+    u8 data[];
> >> >+};
> >> >+\end{lstlisting}
> >> >+
> >> >+
> >> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> >> > incoming packets are write-only.
> >> >
> >> >+When transmitting packets to the device, \field{num_buffers} is not
> >> >used.
> >> >+
> >> >+\begin{enumerate}
> >> >+\item \field{num_buffers} indicates how many descriptors
> >> >+  this packet is spread over (including this one).
> >> >+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF was negotiated.
> >> >+  This allows receipt of large packets without having to allocate large
> >> >+  buffers: a packet that does not fit in a single buffer can flow
> >> >+  over to the next buffer, and so on. In this case, there will be
> >> >+  at least \field{num_buffers} used buffers in the virtqueue, and the device
> >> >+  chains them together to form a single packet in a way similar to
> >> >+  how it would store it in a single buffer spread over multiple
> >> >+  descriptors.
> >> >+  The other buffers will not begin with a struct virtio_vsock_hdr.
> >> >+
> >> >+  If VIRTIO_VSOCK_F_MRG_RXBUF was not negotiated, then only one
> >> >+  descriptor is used.
> >> >+
> >> >+\item If
> >> >+  \field{num_buffers} is one, then the entire packet will be
> >> >+  contained within this buffer, immediately following the struct
> >> >+  virtio_vsock_hdr.
> >> >+\end{enumerate}
> >> >+
> >> > \drivernormative{\paragraph}{Device Operation: Receive and
> >> > Transmit}{Device Types / Socket Device / Device Operation / Receive
> >> > and Transmit}
> >> >
> >> > The \field{guest_cid} configuration field MUST be used as the source CID when
> >> > sending outgoing packets.
> >> >
> >> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST be
> >>
> >> Perhaps we can leave it as it was before, since the `type` field
> >> identifies the socket type itself.
> >
> >OK.
> >
> >> >sent if a packet is received with an
> >> > unknown \field{type} value.
> >> >
> >> > \devicenormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> >> >
> >> > The \field{guest_cid} configuration field MUST NOT contain a reserved CID as listed in \ref{sec:Device Types / Socket Device / Device configuration layout}.
> >> >
> >> >-A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> >> >+For stream and datagram sockets, A VIRTIO_VSOCK_OP_RST reply MUST
> >> >be sent if a packet is received with an
> >> > unknown \field{type} value.
> >>
> >> Ditto.
> >
> >OK.
> >
> >> >
> >> >+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
> >> >+\field{num_buffers} to indicate the number of buffers
> >> >+the packet (including the header) is spread over.
> >> >+
> >> >+If a receive packet is spread over multiple buffers, the device
> >> >+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
> >> >+1$ buffers) completely up to the full length of each buffer
> >> >+supplied by the driver.
> >> >+
> >> >+The device MUST use all buffers used by a single receive
> >> >+packet together, such that at least \field{num_buffers} are
> >> >+observed by driver as used.
> >> >+
> >> > \subsubsection{Stream Sockets}\label{sec:Device Types / Socket
> >> > Device / Device Operation / Stream Sockets}
> >> >
> >> > Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
> >> >@@ -240,6 +361,14 @@ \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device O
> >> > destination) address tuple for a new connection while the other peer is still
> >> > processing the old connection.
> >> >
> >> >+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> >> >+
> >> >+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> >> >+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> >> >+a receiving socket does not exist on the destination.
> >> >+If the transmission or receiving buffers are full, the packets
> >> >+are dropped.
> >> >+
> >> > \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
> >> >
> >> > Certain events are communicated by the device to the driver using the event
> >> >--
> >> >2.11.0
> >> >
> >>
> >> I don't know if maybe it's better to break this patch in two, one
> >> where
> >> we add datagram and one for mergeable buffer.
> >>
> >> But let's see what others think.
> >
> >OK. I can definitely break it to two patches if necessary. Thanks for
> >all the comments.
>
> Thanks,
> Stefano
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Wed, Jun 09, 2021 at 08:31:27PM -0700, Jiang Wang . wrote:
>On Wed, Jun 9, 2021 at 12:17 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
>> >On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> >>
>> >> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>> >> >From: "jiang.wang" <jiang.wang@bytedance.com>
>> >> >
>> >> >Add supports for datagram type for virtio-vsock. Datagram
>> >> >sockets are connectionless and unreliable. To avoid contention
>> >> >with stream and other sockets, add two more virtqueues and
>> >> >a new feature bit to identify if those two new queues exist or not.
>> >> >
>> >> >Also add descriptions for resource management of datagram, which
>> >> >does not use the existing credit update mechanism associated with
>> >> >stream sockets.
>> >> >
>> >> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>> >> >---
>> >> >
>> >> >V2: addressed the comments for the previous version.
>> >> >V3: add description for the mergeable receive buffer.
>> >> >V4: add a feature bit for stream and reserver a bit for seqpacket.
>> >> >Fix mrg_rxbuf related sentences.
>> >> >
>> >> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>> >> > 1 file changed, 142 insertions(+), 13 deletions(-)
>> >> >
>> >> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> >> >index da7e641..bacac3c 100644
>> >> >--- a/virtio-vsock.tex
>> >> >+++ b/virtio-vsock.tex
>> >> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >> >
>> >> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> >> > \begin{description}
>> >> >-\item[0] rx
>> >> >-\item[1] tx
>> >> >+\item[0] stream rx
>> >> >+\item[1] stream tx
>> >> >+\item[2] datagram rx
>> >> >+\item[3] datagram tx
>> >> >+\item[4] event
>> >>
>> >> Is there a particular reason to always have the event queue as the last
>> >> one?
>> >>
>> >> Maybe it's better to add the datagram queues at the bottom, so the first
>> >> 3 queues are always the same.
>> >>
>> >I am not sure. I think Linux kernel should be fine with what you described.
>> >But I am not sure about QEMU. From the code, I see virtqueue is allocated
>> >as an array, like following,
>> >
>> >+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
>> >+    struct vhost_virtqueue vhost_vqs[4];
>> >+ #else
>> >    struct vhost_virtqueue vhost_vqs[2];
>> >+ #endi
>>
>> I see, also vhost_dev_init() requires an array, so I agree that this is
>> the best approach, sorry for the noise.
>>
>> Just to be sure to check that anything is working if
>> CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver
>> that doesn't support DGRAM, and viceversa.
>
>Sure.  I just want to mention that the QEMU should be consistent
>with the device (host). If QEMU enabled CONFIG_VHOST_VSOCK_DGRAM,
>the device also needs to enable a similar option. Than the driver can
>be old or new versions.

Okay, but we should allow to run an old QEMU (without DGRAM) with a new 
kernel (with DGRAM support built it) and viceversa.
The features bit are used to guarantee compatibility and to enable and 
disable features at runtime depending on what the device or driver 
supports.

>
>> >
>> >so I assume the virtqueues for tx/rx should be
>> >continuous? I can try to put the new queues at the end and see if it
>> >works or not.
>> >
>> >btw, my qemu change is here:
>> >https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
>> >
>> >> >+\end{description}
>> >> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> >> >+only uses 3 queues, as the following.
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[0] stream rx
>> >> >+\item[1] stream tx
>> >> > \item[2] event
>> >> > \end{description}
>> >> >
>> >> >+When behavior differs between stream and datagram rx/tx virtqueues
>> >> >+their full names are used. Common behavior is simply described in
>> >> >+terms of rx/tx virtqueues and applies to both stream and datagram
>> >> >+virtqueues.
>> >> >+
>> >> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>> >> >
>> >> >-There are currently no feature bits defined for this device.
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>> >> >+\end{description}
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
>> >> >type.
>> >> >+\end{description}
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>> >> >+\end{description}
>> >> >+
>> >> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
>> >> >is set.
>> >>
>> >> I'd say more like socket streams are supported, without reference to the
>> >> feature bit, something like: "If no feature bits are defined, then
>> >> assume device only supports stream socket type."
>> >>
>> >OK.
>> >
>> >> >
>> >> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>> >> >
>> >> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> >
>> >> > Packets transmitted or received contain a header before the payload:
>> >> >
>> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>> >> >+
>> >> > \begin{lstlisting}
>> >> > struct virtio_vsock_hdr {
>> >> >       le64 src_cid;
>> >> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> > };
>> >> > \end{lstlisting}
>> >> >
>> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>> >> >+\begin{lstlisting}
>> >> >+struct virtio_vsock_hdr_mrg_rxbuf {
>> >> >+      struct virtio_vsock_hdr hdr;
>> >> >+      le16 num_buffers;
>> >> >+};
>> >> >+\end{lstlisting}
>> >> >+
>> >> >+
>> >> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>> >> >
>> >> > Most packets simply transfer data but control packets are also used for
>> >> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> >
>> >> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>> >> >
>> >> >+Flow control applies to stream sockets; datagram sockets do not have
>> >> >+flow control.
>> >> >+
>> >> > The tx virtqueue carries packets initiated by applications and replies to
>> >> > received packets.  The rx virtqueue carries packets initiated by the device and
>> >> > replies to previously transmitted packets.
>> >> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>> >> > consists of a (cid, port number) tuple. The header fields used for this are
>> >> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>> >> >
>> >> >-Currently only stream sockets are supported. \field{type} is 1 for
>> >> >stream
>> >> >-socket types.
>> >> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>> >> >+socket types. \field{type} is 3 for dgram socket types.
>> >>
>> >> When Arseny's change will merged, we can define and use
>> >> VIRTIO_VSOCK_TYPE_DGRAM,.
>> >
>> >Sure.
>> >> >
>> >> > Stream sockets provide in-order, guaranteed, connection-oriented
>> >> > delivery
>> >> > without message boundaries.
>> >> >
>> >> >+Datagram sockets provide unordered, unreliable, connectionless
>> >> >messages
>> >> >+with message boundaries and a maximum length.
>> >> >+
>> >> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
>> >> > management of
>> >> > stream sockets. The guest and the device publish how much buffer space is
>> >> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> >> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> >> > \end{lstlisting}
>> >> >
>> >> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
>> >> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>> >> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> >> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> >> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> >> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> >> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> >> > communicating updates any time a change in buffer space occurs.
>> >> >
>> >> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>> >> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>> >> >+is split to two parts: tx side and rx side. For the tx side, if the
>> >>
>> >> Maybe better to use sender and receiver, since we use tx and rx to
>> >> identify the queues.
>> >
>> >OK.
>> >
>> >> >+virtqueue is full, the packet will be dropped.
>> >> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>> >> >+is dropped by the receiver.
>> >>
>> >> This sentence is a bit unclear.
>> >> `buf_alloc` for stream socket is used to inform the other peer about the
>> >> receive buffer space. In this case we are using the local information,
>> >> so there is no need to refer to `buf_alloc`. We can write something
>> >> like: "The packet is dropped by the receiver if there is no space in the
>> >> receive buffer".
>> >
>> >OK.
>> >
>> >> >+
>> >> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >> >+\begin{itemize}
>> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>> >> >+      with buffers of at least 1526 bytes for stream sockets and 4096
>> >> >bytes for datagram sockets.
>> >>
>> >> Where does 1526 come from?
>> >
>> >No specific reason. Any recommendations?
>> >
>> >> We're adding a requirement for socket streams that wasn't there until
>> >> now.
>> >
>> >This is only when mergeable rxbuf bit is used. I think before this, the stream
>> >rx buf should be at least bigger than the pkt header. We just did not put that
>> >into the spec.
>>
>> Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
>> is not negotiated"
>>     ^
>My bad. You are right, this is a new requirement for stream sockets.
>Not a typo.  Maybe remove the part for stream sockets? Something
>like following:
>
>If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD
>populate the receive queue(s)
>with buffers of  at least 4096 bytes for datagram sockets.

Make sense to me. Maybe we can replase s/receive queue(s)/datagram rx 
queue/.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[virtio-comment] Re: [External] Re: [RFC v4] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Wed, Jun 09, 2021 at 08:31:27PM -0700, Jiang Wang . wrote:
>On Wed, Jun 9, 2021 at 12:17 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>>
>> On Tue, Jun 08, 2021 at 09:22:26PM -0700, Jiang Wang . wrote:
>> >On Tue, Jun 8, 2021 at 6:46 AM Stefano Garzarella <sgarzare@redhat.com> wrote:
>> >>
>> >> On Fri, May 28, 2021 at 04:01:18AM +0000, Jiang Wang wrote:
>> >> >From: "jiang.wang" <jiang.wang@bytedance.com>
>> >> >
>> >> >Add supports for datagram type for virtio-vsock. Datagram
>> >> >sockets are connectionless and unreliable. To avoid contention
>> >> >with stream and other sockets, add two more virtqueues and
>> >> >a new feature bit to identify if those two new queues exist or not.
>> >> >
>> >> >Also add descriptions for resource management of datagram, which
>> >> >does not use the existing credit update mechanism associated with
>> >> >stream sockets.
>> >> >
>> >> >Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>> >> >---
>> >> >
>> >> >V2: addressed the comments for the previous version.
>> >> >V3: add description for the mergeable receive buffer.
>> >> >V4: add a feature bit for stream and reserver a bit for seqpacket.
>> >> >Fix mrg_rxbuf related sentences.
>> >> >
>> >> > virtio-vsock.tex | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++-----
>> >> > 1 file changed, 142 insertions(+), 13 deletions(-)
>> >> >
>> >> >diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> >> >index da7e641..bacac3c 100644
>> >> >--- a/virtio-vsock.tex
>> >> >+++ b/virtio-vsock.tex
>> >> >@@ -9,14 +9,41 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >> >
>> >> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> >> > \begin{description}
>> >> >-\item[0] rx
>> >> >-\item[1] tx
>> >> >+\item[0] stream rx
>> >> >+\item[1] stream tx
>> >> >+\item[2] datagram rx
>> >> >+\item[3] datagram tx
>> >> >+\item[4] event
>> >>
>> >> Is there a particular reason to always have the event queue as the last
>> >> one?
>> >>
>> >> Maybe it's better to add the datagram queues at the bottom, so the first
>> >> 3 queues are always the same.
>> >>
>> >I am not sure. I think Linux kernel should be fine with what you described.
>> >But I am not sure about QEMU. From the code, I see virtqueue is allocated
>> >as an array, like following,
>> >
>> >+ #ifdef CONFIG_VHOST_VSOCK_DGRAM
>> >+    struct vhost_virtqueue vhost_vqs[4];
>> >+ #else
>> >    struct vhost_virtqueue vhost_vqs[2];
>> >+ #endi
>>
>> I see, also vhost_dev_init() requires an array, so I agree that this is
>> the best approach, sorry for the noise.
>>
>> Just to be sure to check that anything is working if
>> CONFIG_VHOST_VSOCK_DGRAM is defined, but the guest has an old driver
>> that doesn't support DGRAM, and viceversa.
>
>Sure.  I just want to mention that the QEMU should be consistent
>with the device (host). If QEMU enabled CONFIG_VHOST_VSOCK_DGRAM,
>the device also needs to enable a similar option. Than the driver can
>be old or new versions.

Okay, but we should allow to run an old QEMU (without DGRAM) with a new 
kernel (with DGRAM support built it) and viceversa.
The features bit are used to guarantee compatibility and to enable and 
disable features at runtime depending on what the device or driver 
supports.

>
>> >
>> >so I assume the virtqueues for tx/rx should be
>> >continuous? I can try to put the new queues at the end and see if it
>> >works or not.
>> >
>> >btw, my qemu change is here:
>> >https://github.com/Jiang1155/qemu/commit/6307aa7a0c347905a31f3ca6577923e2f6dd9d84
>> >
>> >> >+\end{description}
>> >> >+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> >> >+only uses 3 queues, as the following.
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[0] stream rx
>> >> >+\item[1] stream tx
>> >> > \item[2] event
>> >> > \end{description}
>> >> >
>> >> >+When behavior differs between stream and datagram rx/tx virtqueues
>> >> >+their full names are used. Common behavior is simply described in
>> >> >+terms of rx/tx virtqueues and applies to both stream and datagram
>> >> >+virtqueues.
>> >> >+
>> >> > \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>> >> >
>> >> >-There are currently no feature bits defined for this device.
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_STREAM (0)] Device has support for stream socket type.
>> >> >+\end{description}
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_DGRAM (2)] Device has support for datagram socket
>> >> >type.
>> >> >+\end{description}
>> >> >+
>> >> >+\begin{description}
>> >> >+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] Driver can merge receive buffers.
>> >> >+\end{description}
>> >> >+
>> >> >+If no feature bits are defined, then assume only VIRTIO_VSOCK_F_STREAM
>> >> >is set.
>> >>
>> >> I'd say more like socket streams are supported, without reference to the
>> >> feature bit, something like: "If no feature bits are defined, then
>> >> assume device only supports stream socket type."
>> >>
>> >OK.
>> >
>> >> >
>> >> > \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>> >> >
>> >> >@@ -64,6 +91,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> >
>> >> > Packets transmitted or received contain a header before the payload:
>> >> >
>> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>> >> >+
>> >> > \begin{lstlisting}
>> >> > struct virtio_vsock_hdr {
>> >> >       le64 src_cid;
>> >> >@@ -79,6 +108,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> > };
>> >> > \end{lstlisting}
>> >> >
>> >> >+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>> >> >+\begin{lstlisting}
>> >> >+struct virtio_vsock_hdr_mrg_rxbuf {
>> >> >+      struct virtio_vsock_hdr hdr;
>> >> >+      le16 num_buffers;
>> >> >+};
>> >> >+\end{lstlisting}
>> >> >+
>> >> >+
>> >> > The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>> >> >
>> >> > Most packets simply transfer data but control packets are also used for
>> >> >@@ -107,6 +145,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>> >> >
>> >> > \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>> >> >
>> >> >+Flow control applies to stream sockets; datagram sockets do not have
>> >> >+flow control.
>> >> >+
>> >> > The tx virtqueue carries packets initiated by applications and replies to
>> >> > received packets.  The rx virtqueue carries packets initiated by the device and
>> >> > replies to previously transmitted packets.
>> >> >@@ -140,12 +181,15 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
>> >> > consists of a (cid, port number) tuple. The header fields used for this are
>> >> > \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>> >> >
>> >> >-Currently only stream sockets are supported. \field{type} is 1 for
>> >> >stream
>> >> >-socket types.
>> >> >+Currently stream and datagram (dgram) sockets are supported. \field{type} is 1 for stream
>> >> >+socket types. \field{type} is 3 for dgram socket types.
>> >>
>> >> When Arseny's change will merged, we can define and use
>> >> VIRTIO_VSOCK_TYPE_DGRAM,.
>> >
>> >Sure.
>> >> >
>> >> > Stream sockets provide in-order, guaranteed, connection-oriented
>> >> > delivery
>> >> > without message boundaries.
>> >> >
>> >> >+Datagram sockets provide unordered, unreliable, connectionless
>> >> >messages
>> >> >+with message boundaries and a maximum length.
>> >> >+
>> >> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> >> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space
>> >> > management of
>> >> > stream sockets. The guest and the device publish how much buffer space is
>> >> >@@ -162,7 +206,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> >> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> >> > \end{lstlisting}
>> >> >
>> >> >-If there is insufficient buffer space, the sender waits until virtqueue buffers
>> >> >+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>> >> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> >> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> >> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> >> >@@ -170,24 +214,55 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> >> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> >> > communicating updates any time a change in buffer space occurs.
>> >> >
>> >> >+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE or
>> >> >+VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management
>> >> >+is split to two parts: tx side and rx side. For the tx side, if the
>> >>
>> >> Maybe better to use sender and receiver, since we use tx and rx to
>> >> identify the queues.
>> >
>> >OK.
>> >
>> >> >+virtqueue is full, the packet will be dropped.
>> >> >+For the rx side, dgram also uses the \field{buf_alloc}. If it is full, the packet
>> >> >+is dropped by the receiver.
>> >>
>> >> This sentence is a bit unclear.
>> >> `buf_alloc` for stream socket is used to inform the other peer about the
>> >> receive buffer space. In this case we are using the local information,
>> >> so there is no need to refer to `buf_alloc`. We can write something
>> >> like: "The packet is dropped by the receiver if there is no space in the
>> >> receive buffer".
>> >
>> >OK.
>> >
>> >> >+
>> >> >+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>> >> >+\begin{itemize}
>> >> >+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the receive queue(s)
>> >> >+      with buffers of at least 1526 bytes for stream sockets and 4096
>> >> >bytes for datagram sockets.
>> >>
>> >> Where does 1526 come from?
>> >
>> >No specific reason. Any recommendations?
>> >
>> >> We're adding a requirement for socket streams that wasn't there until
>> >> now.
>> >
>> >This is only when mergeable rxbuf bit is used. I think before this, the stream
>> >rx buf should be at least bigger than the pkt header. We just did not put that
>> >into the spec.
>>
>> Mmm, I'm confused now. The statement says "If VIRTIO_VSOCK_F_MRG_RXBUF
>> is not negotiated"
>>     ^
>My bad. You are right, this is a new requirement for stream sockets.
>Not a typo.  Maybe remove the part for stream sockets? Something
>like following:
>
>If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD
>populate the receive queue(s)
>with buffers of  at least 4096 bytes for datagram sockets.

Make sense to me. Maybe we can replase s/receive queue(s)/datagram rx 
queue/.

Thanks,
Stefano


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

[PATCH v5 0/2] Support vsock datagram and mergeable buffers

[beshleman.devbox]

From: Bobby Eshleman <bobby.eshleman@bytedance.com>

This series adds two features to vsock: datagrams and mergeable buffers.

The first patch adds a new socket type (the datagram) to vsock. The second
patch adds a feature bit that supports mergeable buffers (similar to
virtio-net).

Jiang Wang (2):
  virtio-vsock: add description for datagram type
  virtio-vsock: add mergeable buffer feature bit

 virtio-vsock.tex | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 129 insertions(+), 10 deletions(-)

-- 
2.11.0

[PATCH v5 1/2] virtio-vsock: add description for datagram type

[beshleman.devbox]

From: Jiang Wang <jiang.wang@bytedance.com>

Add supports for datagram type for virtio-vsock. Datagram
sockets are connectionless and unreliable. To avoid contention
with stream and other sockets, add two more virtqueues and
a new feature bit to identify if those two new queues exist or not.

Also add descriptions for resource management of datagram, which
does not use the existing credit update mechanism associated with
stream sockets.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
---

V2: Addressed the comments for the previous version.
V3: Add description for the mergeable receive buffer.
V4: Add a feature bit for stream and reserver a bit for seqpacket.
    Fix mrg_rxbuf related sentences.
V5: Rebase onto head (change to more nicely accompany seqpacket changes).
    Remove statement about no feature bits being set (already made by seqpacket patches).
    Clarify \field{type} declaration.
    Use words "sender/receiver" instead of "tx/rx" for clarity, and other prose changes.
    Directly state that full buffers result in dropped packets.
    Change verbs to present tense.
    Clarify if-else pairs (e.g., "If XYZ is negotiated" is followed by "If XYZ
    is not negotiated" instead of "Otherwise").
    Mergeable buffer changes are split off into a separate patch.

 virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 53 insertions(+), 10 deletions(-)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index d79984d..1a66a1b 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
 
 \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
 \begin{description}
-\item[0] rx
-\item[1] tx
+\item[0] stream rx
+\item[1] stream tx
+\item[2] datagram rx
+\item[3] datagram tx
+\item[4] event
+\end{description}
+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
+only uses 3 queues, as the following.
+
+\begin{description}
+\item[0] stream rx
+\item[1] stream tx
 \item[2] event
 \end{description}
 
+When behavior differs between stream and datagram rx/tx virtqueues
+their full names are used. Common behavior is simply described in
+terms of rx/tx virtqueues and applies to both stream and datagram
+virtqueues.
+
 \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 
 If no feature bit is set, only stream socket type is supported.
@@ -23,6 +38,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 \begin{description}
 \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
 \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
+\item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
 \end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
@@ -109,6 +125,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
 
+Flow control applies to stream sockets; datagram sockets do not have
+flow control.
+
 The tx virtqueue carries packets initiated by applications and replies to
 received packets.  The rx virtqueue carries packets initiated by the device and
 replies to previously transmitted packets.
@@ -142,18 +161,22 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
 consists of a (cid, port number) tuple. The header fields used for this are
 \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
 
-Currently stream and seqpacket sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
-for stream socket types, and 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types.
+Currently stream, seqpacket, and dgram sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
+for stream socket types, 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types, and 3 (VIRTIO_VSOCK_TYPE_DGRAM) for dgram socket types.
 
 \begin{lstlisting}
 #define VIRTIO_VSOCK_TYPE_STREAM    1
 #define VIRTIO_VSOCK_TYPE_SEQPACKET 2
+#define VIRTIO_VSOCK_TYPE_DGRAM     3
 \end{lstlisting}
 
 Stream sockets provide in-order, guaranteed, connection-oriented delivery
 without message boundaries. Seqpacket sockets provide in-order, guaranteed,
 connection-oriented delivery with message and record boundaries.
 
+Datagram sockets provide unordered, unreliable, connectionless messages
+with message boundaries and a maximum length.
+
 \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
 \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
 stream sockets. The guest and the device publish how much buffer space is
@@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
 \end{lstlisting}
 
-If there is insufficient buffer space, the sender waits until virtqueue buffers
+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
 are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
 the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
 available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
@@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
 communicating updates any time a change in buffer space occurs.
 
+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
+or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
+into two parts: senders and receivers. For senders, the packet is dropped if the
+virtqueue is full. For receivers, the packet is dropped if there is no space
+in the receive buffer.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and driver will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and the device will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
-The driver queues outgoing packets on the tx virtqueue and incoming packet
+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
 \begin{lstlisting}
@@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
+When transmitting packets to the device, \field{num_buffers} is not used.
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
@@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
 #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
 \end{lstlisting}
 
+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
+
+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
+a receiving socket does not exist on the destination.
+If the transmission or receiving buffers are full, the packets
+are dropped.
+
 \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
 
 Certain events are communicated by the device to the driver using the event
-- 
2.11.0

[PATCH v5 2/2] virtio-vsock: add mergeable buffer feature bit

[beshleman.devbox]

From: Jiang Wang <jiang.wang@bytedance.com>

Add support for mergeable buffers for virtio-vsock. Mergeable buffers
allow individual large packets to be spread across multiple buffers
while still using only a single packet header. This avoids
artificially restraining packet size to the size of a single
buffer and offers a performant fragmentation/defragmentation
scheme.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
---
 virtio-vsock.tex | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index 1a66a1b..bf44d5d 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -39,6 +39,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
 \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
 \item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] driver can merge receive buffers.
 \end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
@@ -87,6 +88,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 Packets transmitted or received contain a header before the payload:
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
+
 \begin{lstlisting}
 struct virtio_vsock_hdr {
 	le64 src_cid;
@@ -102,6 +105,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 };
 \end{lstlisting}
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
+\begin{lstlisting}
+struct virtio_vsock_hdr_mrg_rxbuf {
+	struct virtio_vsock_hdr hdr;
+	le16 num_buffers;
+};
+\end{lstlisting}
+
+
 The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
 
 Most packets simply transfer data but control packets are also used for
@@ -207,6 +219,25 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 virtqueue is full. For receivers, the packet is dropped if there is no space
 in the receive buffer.
 
+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+\begin{itemize}
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the datagram rx queue
+      with buffers of at least 4096 bytes.
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
+      least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
+\end{itemize}
+
+\begin{note}
+Each buffer may be split across multiple descriptor elements.
+\end{note}
+
+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+The device MUST set \field{num_buffers} to the number of descriptors used when
+transmitting the packet.
+
+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
+is not negotiated.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
 For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
 sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
@@ -229,6 +260,7 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
+If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following.
 \begin{lstlisting}
 struct virtio_vsock_packet {
     struct virtio_vsock_hdr hdr;
@@ -236,11 +268,42 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 };
 \end{lstlisting}
 
+If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following form:
+\begin{lstlisting}
+struct virtio_vsock_packet_mrg_rxbuf {
+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
+    u8 data[];
+};
+\end{lstlisting}
+
+
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
 When transmitting packets to the device, \field{num_buffers} is not used.
 
+\begin{enumerate}
+\item \field{num_buffers} indicates how many descriptors
+  this packet is spread over (including this one).
+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF is negotiated.
+  This allows receipt of large packets without having to allocate large
+  buffers: a packet that does not fit in a single buffer can flow
+  over to the next buffer, and so on. In this case, there will be
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
+  chains them together to form a single packet in a way similar to
+  how it would store it in a single buffer spread over multiple
+  descriptors.
+  The other buffers will not begin with a struct virtio_vsock_hdr.
+
+  If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, then only one
+  descriptor is used.
+
+\item If
+  \field{num_buffers} is one, then the entire packet will be
+  contained within this buffer, immediately following the struct
+  virtio_vsock_hdr.
+\end{enumerate}
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
@@ -256,6 +319,19 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
+\field{num_buffers} to indicate the number of buffers
+the packet (including the header) is spread over.
+
+If a receive packet is spread over multiple buffers, the device
+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
+1$ buffers) completely up to the full length of each buffer
+supplied by the driver.
+
+The device MUST use all buffers used by a single receive
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
+
 \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
 
 Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
-- 
2.11.0

[PATCH v5 0/2] Support vsock datagram and mergeable buffers

[beshleman.devbox]

From: Bobby Eshleman <bobby.eshleman@bytedance.com>

This series adds two features to vsock: datagrams and mergeable buffers.

The first patch adds a new socket type (the datagram) to vsock. The second
patch adds a feature bit that supports mergeable buffers (similar to
virtio-net).

PS: please excuse the double send, I wasn't properly subscribed to the ML
so the first thread failed to get through.

Jiang Wang (2):
  virtio-vsock: add description for datagram type
  virtio-vsock: add mergeable buffer feature bit

 virtio-vsock.tex | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 129 insertions(+), 10 deletions(-)

-- 
2.11.0

[PATCH v5 1/2] virtio-vsock: add description for datagram type

[beshleman.devbox]

From: Jiang Wang <jiang.wang@bytedance.com>

Add supports for datagram type for virtio-vsock. Datagram
sockets are connectionless and unreliable. To avoid contention
with stream and other sockets, add two more virtqueues and
a new feature bit to identify if those two new queues exist or not.

Also add descriptions for resource management of datagram, which
does not use the existing credit update mechanism associated with
stream sockets.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
---

V2: Addressed the comments for the previous version.
V3: Add description for the mergeable receive buffer.
V4: Add a feature bit for stream and reserver a bit for seqpacket.
    Fix mrg_rxbuf related sentences.
V5: Rebase onto head (change to more nicely accompany seqpacket changes).
    Remove statement about no feature bits being set (already made by seqpacket patches).
    Clarify \field{type} declaration.
    Use words "sender/receiver" instead of "tx/rx" for clarity, and other prose changes.
    Directly state that full buffers result in dropped packets.
    Change verbs to present tense.
    Clarify if-else pairs (e.g., "If XYZ is negotiated" is followed by "If XYZ
    is not negotiated" instead of "Otherwise").
    Mergeable buffer changes are split off into a separate patch.

 virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 53 insertions(+), 10 deletions(-)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index d79984d..1a66a1b 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
 
 \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
 \begin{description}
-\item[0] rx
-\item[1] tx
+\item[0] stream rx
+\item[1] stream tx
+\item[2] datagram rx
+\item[3] datagram tx
+\item[4] event
+\end{description}
+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
+only uses 3 queues, as the following.
+
+\begin{description}
+\item[0] stream rx
+\item[1] stream tx
 \item[2] event
 \end{description}
 
+When behavior differs between stream and datagram rx/tx virtqueues
+their full names are used. Common behavior is simply described in
+terms of rx/tx virtqueues and applies to both stream and datagram
+virtqueues.
+
 \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 
 If no feature bit is set, only stream socket type is supported.
@@ -23,6 +38,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 \begin{description}
 \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
 \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
+\item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
 \end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
@@ -109,6 +125,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
 
+Flow control applies to stream sockets; datagram sockets do not have
+flow control.
+
 The tx virtqueue carries packets initiated by applications and replies to
 received packets.  The rx virtqueue carries packets initiated by the device and
 replies to previously transmitted packets.
@@ -142,18 +161,22 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
 consists of a (cid, port number) tuple. The header fields used for this are
 \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
 
-Currently stream and seqpacket sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
-for stream socket types, and 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types.
+Currently stream, seqpacket, and dgram sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
+for stream socket types, 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types, and 3 (VIRTIO_VSOCK_TYPE_DGRAM) for dgram socket types.
 
 \begin{lstlisting}
 #define VIRTIO_VSOCK_TYPE_STREAM    1
 #define VIRTIO_VSOCK_TYPE_SEQPACKET 2
+#define VIRTIO_VSOCK_TYPE_DGRAM     3
 \end{lstlisting}
 
 Stream sockets provide in-order, guaranteed, connection-oriented delivery
 without message boundaries. Seqpacket sockets provide in-order, guaranteed,
 connection-oriented delivery with message and record boundaries.
 
+Datagram sockets provide unordered, unreliable, connectionless messages
+with message boundaries and a maximum length.
+
 \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
 \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
 stream sockets. The guest and the device publish how much buffer space is
@@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
 \end{lstlisting}
 
-If there is insufficient buffer space, the sender waits until virtqueue buffers
+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
 are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
 the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
 available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
@@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
 communicating updates any time a change in buffer space occurs.
 
+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
+or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
+into two parts: senders and receivers. For senders, the packet is dropped if the
+virtqueue is full. For receivers, the packet is dropped if there is no space
+in the receive buffer.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and driver will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
-sufficient free buffer space for the payload.
+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
+and the device will not get any notification.
 
 All packets associated with a stream flow MUST contain valid information in
 \field{buf_alloc} and \field{fwd_cnt} fields.
 
 \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
-The driver queues outgoing packets on the tx virtqueue and incoming packet
+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
 \begin{lstlisting}
@@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
+When transmitting packets to the device, \field{num_buffers} is not used.
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
@@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
 #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
 \end{lstlisting}
 
+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
+
+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
+a receiving socket does not exist on the destination.
+If the transmission or receiving buffers are full, the packets
+are dropped.
+
 \subsubsection{Device Events}\label{sec:Device Types / Socket Device / Device Operation / Device Events}
 
 Certain events are communicated by the device to the driver using the event
-- 
2.11.0

[PATCH v5 2/2] virtio-vsock: add mergeable buffer feature bit

[beshleman.devbox]

From: Jiang Wang <jiang.wang@bytedance.com>

Add support for mergeable buffers for virtio-vsock. Mergeable buffers
allow individual large packets to be spread across multiple buffers
while still using only a single packet header. This avoids
artificially restraining packet size to the size of a single
buffer and offers a performant fragmentation/defragmentation
scheme.

Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
---
 virtio-vsock.tex | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)

diff --git a/virtio-vsock.tex b/virtio-vsock.tex
index 1a66a1b..bf44d5d 100644
--- a/virtio-vsock.tex
+++ b/virtio-vsock.tex
@@ -39,6 +39,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
 \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
 \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
 \item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] driver can merge receive buffers.
 \end{description}
 
 \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
@@ -87,6 +88,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 
 Packets transmitted or received contain a header before the payload:
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
+
 \begin{lstlisting}
 struct virtio_vsock_hdr {
 	le64 src_cid;
@@ -102,6 +105,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
 };
 \end{lstlisting}
 
+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
+\begin{lstlisting}
+struct virtio_vsock_hdr_mrg_rxbuf {
+	struct virtio_vsock_hdr hdr;
+	le16 num_buffers;
+};
+\end{lstlisting}
+
+
 The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
 
 Most packets simply transfer data but control packets are also used for
@@ -207,6 +219,25 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
 virtqueue is full. For receivers, the packet is dropped if there is no space
 in the receive buffer.
 
+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+\begin{itemize}
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the datagram rx queue
+      with buffers of at least 4096 bytes.
+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
+      least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
+\end{itemize}
+
+\begin{note}
+Each buffer may be split across multiple descriptor elements.
+\end{note}
+
+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
+The device MUST set \field{num_buffers} to the number of descriptors used when
+transmitting the packet.
+
+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
+is not negotiated.
+
 \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
 For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
 sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
@@ -229,6 +260,7 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
 receive buffers on the rx virtqueue. Packets are of the following form:
 
+If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following.
 \begin{lstlisting}
 struct virtio_vsock_packet {
     struct virtio_vsock_hdr hdr;
@@ -236,11 +268,42 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 };
 \end{lstlisting}
 
+If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following form:
+\begin{lstlisting}
+struct virtio_vsock_packet_mrg_rxbuf {
+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
+    u8 data[];
+};
+\end{lstlisting}
+
+
 Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
 incoming packets are write-only.
 
 When transmitting packets to the device, \field{num_buffers} is not used.
 
+\begin{enumerate}
+\item \field{num_buffers} indicates how many descriptors
+  this packet is spread over (including this one).
+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF is negotiated.
+  This allows receipt of large packets without having to allocate large
+  buffers: a packet that does not fit in a single buffer can flow
+  over to the next buffer, and so on. In this case, there will be
+  at least \field{num_buffers} used buffers in the virtqueue, and the device
+  chains them together to form a single packet in a way similar to
+  how it would store it in a single buffer spread over multiple
+  descriptors.
+  The other buffers will not begin with a struct virtio_vsock_hdr.
+
+  If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, then only one
+  descriptor is used.
+
+\item If
+  \field{num_buffers} is one, then the entire packet will be
+  contained within this buffer, immediately following the struct
+  virtio_vsock_hdr.
+\end{enumerate}
+
 \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
 
 The \field{guest_cid} configuration field MUST be used as the source CID when
@@ -256,6 +319,19 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
 A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
 unknown \field{type} value.
 
+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
+\field{num_buffers} to indicate the number of buffers
+the packet (including the header) is spread over.
+
+If a receive packet is spread over multiple buffers, the device
+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
+1$ buffers) completely up to the full length of each buffer
+supplied by the driver.
+
+The device MUST use all buffers used by a single receive
+packet together, such that at least \field{num_buffers} are
+observed by driver as used.
+
 \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
 
 Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
-- 
2.11.0

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Stefano Garzarella]

Hi Bobby,
Sorry for the delay, but I saw these patches today.
Please, can you keep me in CC?

On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
>From: Jiang Wang <jiang.wang@bytedance.com>
>
>Add supports for datagram type for virtio-vsock. Datagram
>sockets are connectionless and unreliable. To avoid contention
>with stream and other sockets, add two more virtqueues and
>a new feature bit to identify if those two new queues exist or not.
>
>Also add descriptions for resource management of datagram, which
>does not use the existing credit update mechanism associated with
>stream sockets.
>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
>---
>
>V2: Addressed the comments for the previous version.
>V3: Add description for the mergeable receive buffer.
>V4: Add a feature bit for stream and reserver a bit for seqpacket.
>    Fix mrg_rxbuf related sentences.
>V5: Rebase onto head (change to more nicely accompany seqpacket changes).
>    Remove statement about no feature bits being set (already made by seqpacket patches).
>    Clarify \field{type} declaration.
>    Use words "sender/receiver" instead of "tx/rx" for clarity, and other prose changes.
>    Directly state that full buffers result in dropped packets.
>    Change verbs to present tense.
>    Clarify if-else pairs (e.g., "If XYZ is negotiated" is followed by "If XYZ
>    is not negotiated" instead of "Otherwise").
>    Mergeable buffer changes are split off into a separate patch.
>
> virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
> 1 file changed, 53 insertions(+), 10 deletions(-)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index d79984d..1a66a1b 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
> \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> \begin{description}
>-\item[0] rx
>-\item[1] tx
>+\item[0] stream rx
>+\item[1] stream tx
>+\item[2] datagram rx
>+\item[3] datagram tx
>+\item[4] event
>+\end{description}
>+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>+only uses 3 queues, as the following.

We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
provide the possibility to support for example only dgrams.

So I think we should consider the case where we have only DGRAM queues
(and it will be similar to the stream only case so 3 queues).

Maybe we could describe this part better and say that if we have both
STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
only 3 queues.

>+
>+\begin{description}
>+\item[0] stream rx
>+\item[1] stream tx
> \item[2] event
> \end{description}
>
>+When behavior differs between stream and datagram rx/tx virtqueues
>+their full names are used. Common behavior is simply described in
>+terms of rx/tx virtqueues and applies to both stream and datagram
>+virtqueues.
>+
> \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
> If no feature bit is set, only stream socket type is supported.
>@@ -23,6 +38,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> \begin{description}
> \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
> \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
>+\item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
> \end{description}
>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>@@ -109,6 +125,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
>+Flow control applies to stream sockets; datagram sockets do not have
>+flow control.
>+
> The tx virtqueue carries packets initiated by applications and replies to
> received packets.  The rx virtqueue carries packets initiated by the device and
> replies to previously transmitted packets.
>@@ -142,18 +161,22 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> consists of a (cid, port number) tuple. The header fields used for this are
> \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
>-Currently stream and seqpacket sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
>-for stream socket types, and 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types.
>+Currently stream, seqpacket, and dgram sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
>+for stream socket types, 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types, and 3 (VIRTIO_VSOCK_TYPE_DGRAM) for dgram socket types.
>
> \begin{lstlisting}
> #define VIRTIO_VSOCK_TYPE_STREAM    1
> #define VIRTIO_VSOCK_TYPE_SEQPACKET 2
>+#define VIRTIO_VSOCK_TYPE_DGRAM     3
> \end{lstlisting}
>
> Stream sockets provide in-order, guaranteed, connection-oriented delivery
> without message boundaries. Seqpacket sockets provide in-order, guaranteed,
> connection-oriented delivery with message and record boundaries.
>
>+Datagram sockets provide unordered, unreliable, connectionless messages
>+with message boundaries and a maximum length.
>+
> \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
> stream sockets. The guest and the device publish how much buffer space is
>@@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> \end{lstlisting}
>
>-If there is insufficient buffer space, the sender waits until virtqueue buffers
>+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers

stream and seqpacket

> are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>@@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> communicating updates any time a change in buffer space occurs.
>
>+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
>+or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
>+into two parts: senders and receivers. For senders, the packet is dropped if the
>+virtqueue is full. For receivers, the packet is dropped if there is no space
>+in the receive buffer.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has

stream and seqpacket

>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and driver will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has

stream and seqpacket

>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and the device will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>-The driver queues outgoing packets on the tx virtqueue and incoming packet
>+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet

Is this change related?

> receive buffers on the rx virtqueue. Packets are of the following form:
>
> \begin{lstlisting}
>@@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
>+When transmitting packets to the device, \field{num_buffers} is not used.
>+

Leftover? Perhaps it should go in patch 2.

> \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
>@@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> \end{lstlisting}
>
>+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>+
>+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>+a receiving socket does not exist on the destination.
>+If the transmission or receiving buffers are full, the packets
>+are dropped.
>+

I'm not sure we should respond with RST if there's no socket bind on
the port.

What happens with UDP if we do a sendto to a closed port?

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Stefano Garzarella]

Hi Bobby,
Sorry for the delay, but I saw these patches today.
Please, can you keep me in CC?

On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
>From: Jiang Wang <jiang.wang@bytedance.com>
>
>Add supports for datagram type for virtio-vsock. Datagram
>sockets are connectionless and unreliable. To avoid contention
>with stream and other sockets, add two more virtqueues and
>a new feature bit to identify if those two new queues exist or not.
>
>Also add descriptions for resource management of datagram, which
>does not use the existing credit update mechanism associated with
>stream sockets.
>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
>---
>
>V2: Addressed the comments for the previous version.
>V3: Add description for the mergeable receive buffer.
>V4: Add a feature bit for stream and reserver a bit for seqpacket.
>    Fix mrg_rxbuf related sentences.
>V5: Rebase onto head (change to more nicely accompany seqpacket changes).
>    Remove statement about no feature bits being set (already made by seqpacket patches).
>    Clarify \field{type} declaration.
>    Use words "sender/receiver" instead of "tx/rx" for clarity, and other prose changes.
>    Directly state that full buffers result in dropped packets.
>    Change verbs to present tense.
>    Clarify if-else pairs (e.g., "If XYZ is negotiated" is followed by "If XYZ
>    is not negotiated" instead of "Otherwise").
>    Mergeable buffer changes are split off into a separate patch.
>
> virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
> 1 file changed, 53 insertions(+), 10 deletions(-)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index d79984d..1a66a1b 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>
> \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> \begin{description}
>-\item[0] rx
>-\item[1] tx
>+\item[0] stream rx
>+\item[1] stream tx
>+\item[2] datagram rx
>+\item[3] datagram tx
>+\item[4] event
>+\end{description}
>+The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>+only uses 3 queues, as the following.

We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
provide the possibility to support for example only dgrams.

So I think we should consider the case where we have only DGRAM queues
(and it will be similar to the stream only case so 3 queues).

Maybe we could describe this part better and say that if we have both
STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
only 3 queues.

>+
>+\begin{description}
>+\item[0] stream rx
>+\item[1] stream tx
> \item[2] event
> \end{description}
>
>+When behavior differs between stream and datagram rx/tx virtqueues
>+their full names are used. Common behavior is simply described in
>+terms of rx/tx virtqueues and applies to both stream and datagram
>+virtqueues.
>+
> \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
>
> If no feature bit is set, only stream socket type is supported.
>@@ -23,6 +38,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> \begin{description}
> \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
> \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
>+\item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
> \end{description}
>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>@@ -109,6 +125,9 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> \subsubsection{Virtqueue Flow Control}\label{sec:Device Types / Socket Device / Device Operation / Virtqueue Flow Control}
>
>+Flow control applies to stream sockets; datagram sockets do not have
>+flow control.
>+
> The tx virtqueue carries packets initiated by applications and replies to
> received packets.  The rx virtqueue carries packets initiated by the device and
> replies to previously transmitted packets.
>@@ -142,18 +161,22 @@ \subsubsection{Addressing}\label{sec:Device Types / Socket Device / Device Opera
> consists of a (cid, port number) tuple. The header fields used for this are
> \field{src_cid}, \field{src_port}, \field{dst_cid}, and \field{dst_port}.
>
>-Currently stream and seqpacket sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
>-for stream socket types, and 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types.
>+Currently stream, seqpacket, and dgram sockets are supported. \field{type} is 1 (VIRTIO_VSOCK_TYPE_STREAM)
>+for stream socket types, 2 (VIRTIO_VSOCK_TYPE_SEQPACKET) for seqpacket socket types, and 3 (VIRTIO_VSOCK_TYPE_DGRAM) for dgram socket types.
>
> \begin{lstlisting}
> #define VIRTIO_VSOCK_TYPE_STREAM    1
> #define VIRTIO_VSOCK_TYPE_SEQPACKET 2
>+#define VIRTIO_VSOCK_TYPE_DGRAM     3
> \end{lstlisting}
>
> Stream sockets provide in-order, guaranteed, connection-oriented delivery
> without message boundaries. Seqpacket sockets provide in-order, guaranteed,
> connection-oriented delivery with message and record boundaries.
>
>+Datagram sockets provide unordered, unreliable, connectionless messages
>+with message boundaries and a maximum length.
>+
> \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
> stream sockets. The guest and the device publish how much buffer space is
>@@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> \end{lstlisting}
>
>-If there is insufficient buffer space, the sender waits until virtqueue buffers
>+For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers

stream and seqpacket

> are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>@@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> communicating updates any time a change in buffer space occurs.
>
>+Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
>+or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
>+into two parts: senders and receivers. For senders, the packet is dropped if the
>+virtqueue is full. For receivers, the packet is dropped if there is no space
>+in the receive buffer.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has

stream and seqpacket

>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and driver will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>-VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>-sufficient free buffer space for the payload.
>+For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has

stream and seqpacket

>+sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>+MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>+and the device will not get any notification.
>
> All packets associated with a stream flow MUST contain valid information in
> \field{buf_alloc} and \field{fwd_cnt} fields.
>
> \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>-The driver queues outgoing packets on the tx virtqueue and incoming packet
>+The driver queues outgoing packets on the tx virtqueue and allocates incoming packet

Is this change related?

> receive buffers on the rx virtqueue. Packets are of the following form:
>
> \begin{lstlisting}
>@@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
>+When transmitting packets to the device, \field{num_buffers} is not used.
>+

Leftover? Perhaps it should go in patch 2.

> \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
>@@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> \end{lstlisting}
>
>+\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>+
>+Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>+a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>+a receiving socket does not exist on the destination.
>+If the transmission or receiving buffers are full, the packets
>+are dropped.
>+

I'm not sure we should respond with RST if there's no socket bind on
the port.

What happens with UDP if we do a sendto to a closed port?

Thanks,
Stefano

Re: [virtio-comment] [PATCH v5 2/2] virtio-vsock: add mergeable buffer feature bit

[Stefano Garzarella]

On Thu, Feb 24, 2022 at 10:15:47PM +0000, beshleman.devbox@gmail.com wrote:
>From: Jiang Wang <jiang.wang@bytedance.com>
>
>Add support for mergeable buffers for virtio-vsock. Mergeable buffers
>allow individual large packets to be spread across multiple buffers
>while still using only a single packet header. This avoids
>artificially restraining packet size to the size of a single
>buffer and offers a performant fragmentation/defragmentation
>scheme.

We need this new functionality because we want to fragment a datagram 
packet over multiple buffers, right?

This reminded me that we don't have a maximum size for now, in this case 
what would it be?

Maybe it would be helpful to define it as Laura is planning to do here:
https://lists.oasis-open.org/archives/virtio-comment/202202/msg00053.html

>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
>---
> virtio-vsock.tex | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 76 insertions(+)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index 1a66a1b..bf44d5d 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -39,6 +39,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
> \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
> \item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
>+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] driver can merge receive buffers.
> \end{description}
>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>@@ -87,6 +88,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> Packets transmitted or received contain a header before the payload:
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>+
> \begin{lstlisting}
> struct virtio_vsock_hdr {
> 	le64 src_cid;
>@@ -102,6 +105,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> };
> \end{lstlisting}
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>+\begin{lstlisting}
>+struct virtio_vsock_hdr_mrg_rxbuf {
>+	struct virtio_vsock_hdr hdr;
>+	le16 num_buffers;
>+};
>+\end{lstlisting}
>+
>+
> The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
> Most packets simply transfer data but control packets are also used for
>@@ -207,6 +219,25 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> virtqueue is full. For receivers, the packet is dropped if there is no space
> in the receive buffer.
>
>+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+\begin{itemize}
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the datagram rx queue
>+      with buffers of at least 4096 bytes.
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>+      least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>+\end{itemize}
>+
>+\begin{note}
>+Each buffer may be split across multiple descriptor elements.
>+\end{note}
>+
>+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+The device MUST set \field{num_buffers} to the number of descriptors used when
>+transmitting the packet.
>+
>+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>+is not negotiated.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>@@ -229,6 +260,7 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> receive buffers on the rx virtqueue. Packets are of the following form:
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following.
> \begin{lstlisting}
> struct virtio_vsock_packet {
>     struct virtio_vsock_hdr hdr;
>@@ -236,11 +268,42 @@ \subsubsection{Receive and 
>Transmit}\label{sec:Device Types / Socket Device / De
> };
> \end{lstlisting}
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following form:
>+\begin{lstlisting}
>+struct virtio_vsock_packet_mrg_rxbuf {
>+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>+    u8 data[];
>+};
>+\end{lstlisting}
>+
>+
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
> When transmitting packets to the device, \field{num_buffers} is not used.
>
>+\begin{enumerate}
>+\item \field{num_buffers} indicates how many descriptors
>+  this packet is spread over (including this one).
>+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF is negotiated.
>+  This allows receipt of large packets without having to allocate large
>+  buffers: a packet that does not fit in a single buffer can flow
>+  over to the next buffer, and so on. In this case, there will be
>+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>+  chains them together to form a single packet in a way similar to
>+  how it would store it in a single buffer spread over multiple
>+  descriptors.
>+  The other buffers will not begin with a struct virtio_vsock_hdr.
>+
>+  If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, then only one
>+  descriptor is used.
>+
>+\item If
>+  \field{num_buffers} is one, then the entire packet will be
>+  contained within this buffer, immediately following the struct
>+  virtio_vsock_hdr.
     ^
Should it be virtio_vsock_hdr_mrg_rxbuf?

>+\end{enumerate}
>+
> \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
>@@ -256,6 +319,19 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> unknown \field{type} value.
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>+\field{num_buffers} to indicate the number of buffers
>+the packet (including the header) is spread over.
>+
>+If a receive packet is spread over multiple buffers, the device
>+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>+1$ buffers) completely up to the full length of each buffer
>+supplied by the driver.
>+
>+The device MUST use all buffers used by a single receive
>+packet together, such that at least \field{num_buffers} are
>+observed by driver as used.
>+
> \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
> Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>-- 
>2.11.0
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 2/2] virtio-vsock: add mergeable buffer feature bit

[Stefano Garzarella]

On Thu, Feb 24, 2022 at 10:15:47PM +0000, beshleman.devbox@gmail.com wrote:
>From: Jiang Wang <jiang.wang@bytedance.com>
>
>Add support for mergeable buffers for virtio-vsock. Mergeable buffers
>allow individual large packets to be spread across multiple buffers
>while still using only a single packet header. This avoids
>artificially restraining packet size to the size of a single
>buffer and offers a performant fragmentation/defragmentation
>scheme.

We need this new functionality because we want to fragment a datagram 
packet over multiple buffers, right?

This reminded me that we don't have a maximum size for now, in this case 
what would it be?

Maybe it would be helpful to define it as Laura is planning to do here:
https://lists.oasis-open.org/archives/virtio-comment/202202/msg00053.html

>
>Signed-off-by: Jiang Wang <jiang.wang@bytedance.com>
>Signed-off-by: Bobby Eshleman <bobby.eshleman@bytedance.com>
>---
> virtio-vsock.tex | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 76 insertions(+)
>
>diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>index 1a66a1b..bf44d5d 100644
>--- a/virtio-vsock.tex
>+++ b/virtio-vsock.tex
>@@ -39,6 +39,7 @@ \subsection{Feature bits}\label{sec:Device Types / Socket Device / Feature bits}
> \item[VIRTIO_VSOCK_F_STREAM (0)] stream socket type is supported.
> \item[VIRTIO_VSOCK_F_SEQPACKET (1)] seqpacket socket type is supported.
> \item[VIRTIO_VSOCK_F_DGRAM (2)] datagram socket type is supported.
>+\item[VIRTIO_VSOCK_F_MRG_RXBUF (3)] driver can merge receive buffers.
> \end{description}
>
> \subsection{Device configuration layout}\label{sec:Device Types / Socket Device / Device configuration layout}
>@@ -87,6 +88,8 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
>
> Packets transmitted or received contain a header before the payload:
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following header.
>+
> \begin{lstlisting}
> struct virtio_vsock_hdr {
> 	le64 src_cid;
>@@ -102,6 +105,15 @@ \subsection{Device Operation}\label{sec:Device Types / Socket Device / Device Op
> };
> \end{lstlisting}
>
>+If feature VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following header.
>+\begin{lstlisting}
>+struct virtio_vsock_hdr_mrg_rxbuf {
>+	struct virtio_vsock_hdr hdr;
>+	le16 num_buffers;
>+};
>+\end{lstlisting}
>+
>+
> The upper 32 bits of src_cid and dst_cid are reserved and zeroed.
>
> Most packets simply transfer data but control packets are also used for
>@@ -207,6 +219,25 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> virtqueue is full. For receivers, the packet is dropped if there is no space
> in the receive buffer.
>
>+\drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+\begin{itemize}
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, the driver SHOULD populate the datagram rx queue
>+      with buffers of at least 4096 bytes.
>+\item If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, each buffer MUST be at
>+      least the size of the struct virtio_vsock_hdr_mgr_rxbuf.
>+\end{itemize}
>+
>+\begin{note}
>+Each buffer may be split across multiple descriptor elements.
>+\end{note}
>+
>+\devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Setting Up Receive Buffers}
>+The device MUST set \field{num_buffers} to the number of descriptors used when
>+transmitting the packet.
>+
>+The device MUST use only a single descriptor if VIRTIO_VSOCK_F_MRG_RXBUF
>+is not negotiated.
>+
> \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>@@ -229,6 +260,7 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> receive buffers on the rx virtqueue. Packets are of the following form:
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, use the following.
> \begin{lstlisting}
> struct virtio_vsock_packet {
>     struct virtio_vsock_hdr hdr;
>@@ -236,11 +268,42 @@ \subsubsection{Receive and 
>Transmit}\label{sec:Device Types / Socket Device / De
> };
> \end{lstlisting}
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF is negotiated, use the following form:
>+\begin{lstlisting}
>+struct virtio_vsock_packet_mrg_rxbuf {
>+    struct virtio_vsock_hdr_mrg_rxbuf hdr;
>+    u8 data[];
>+};
>+\end{lstlisting}
>+
>+
> Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> incoming packets are write-only.
>
> When transmitting packets to the device, \field{num_buffers} is not used.
>
>+\begin{enumerate}
>+\item \field{num_buffers} indicates how many descriptors
>+  this packet is spread over (including this one).
>+  This is valid only if VIRTIO_VSOCK_F_MRG_RXBUF is negotiated.
>+  This allows receipt of large packets without having to allocate large
>+  buffers: a packet that does not fit in a single buffer can flow
>+  over to the next buffer, and so on. In this case, there will be
>+  at least \field{num_buffers} used buffers in the virtqueue, and the device
>+  chains them together to form a single packet in a way similar to
>+  how it would store it in a single buffer spread over multiple
>+  descriptors.
>+  The other buffers will not begin with a struct virtio_vsock_hdr.
>+
>+  If VIRTIO_VSOCK_F_MRG_RXBUF is not negotiated, then only one
>+  descriptor is used.
>+
>+\item If
>+  \field{num_buffers} is one, then the entire packet will be
>+  contained within this buffer, immediately following the struct
>+  virtio_vsock_hdr.
     ^
Should it be virtio_vsock_hdr_mrg_rxbuf?

>+\end{enumerate}
>+
> \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>
> The \field{guest_cid} configuration field MUST be used as the source CID when
>@@ -256,6 +319,19 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> A VIRTIO_VSOCK_OP_RST reply MUST be sent if a packet is received with an
> unknown \field{type} value.
>
>+If VIRTIO_VSOCK_F_MRG_RXBUF has been negotiated, the device MUST set
>+\field{num_buffers} to indicate the number of buffers
>+the packet (including the header) is spread over.
>+
>+If a receive packet is spread over multiple buffers, the device
>+MUST use all buffers but the last (i.e. the first $\field{num_buffers} -
>+1$ buffers) completely up to the full length of each buffer
>+supplied by the driver.
>+
>+The device MUST use all buffers used by a single receive
>+packet together, such that at least \field{num_buffers} are
>+observed by driver as used.
>+
> \subsubsection{Stream Sockets}\label{sec:Device Types / Socket Device / Device Operation / Stream Sockets}
>
> Connections are established by sending a VIRTIO_VSOCK_OP_REQUEST packet. If a
>-- 
>2.11.0
>
>
>This publicly archived list offers a means to provide input to the
>OASIS Virtual I/O Device (VIRTIO) TC.
>
>In order to verify user consent to the Feedback License terms and
>to minimize spam in the list archive, subscription is required
>before posting.
>
>Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>List help: virtio-comment-help@lists.oasis-open.org
>List archive: https://lists.oasis-open.org/archives/virtio-comment/
>Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>Committee: https://www.oasis-open.org/committees/virtio/
>Join OASIS: https://www.oasis-open.org/join/
>

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Bobby Eshleman]

On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> Hi Bobby,
> Sorry for the delay, but I saw these patches today.
> Please, can you keep me in CC?
> 

Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
one. I'll make sure you are there moving forward.

I want to mention that I'm taking a look at
https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
here. After sending out this series we noticed potential overlap between
the two issues. The additional dgram queues may become redundant if a
fairness mechanism that solves issue #1 above also applies to
connection-less protocols (similar to how the TC subsystem works). I've
just begun sorting out potential solutions so no hard results yet. Just
putting on your radar that the proposal here in v5 may be impacted if my
investigation into issue #1 yields something adequate.

> On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
> > From: Jiang Wang <jiang.wang@bytedance.com>
> > 

... snip ...

> > 
> > virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
> > 1 file changed, 53 insertions(+), 10 deletions(-)
> > 
> > diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> > index d79984d..1a66a1b 100644
> > --- a/virtio-vsock.tex
> > +++ b/virtio-vsock.tex
> > @@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> > 
> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> > \begin{description}
> > -\item[0] rx
> > -\item[1] tx
> > +\item[0] stream rx
> > +\item[1] stream tx
> > +\item[2] datagram rx
> > +\item[3] datagram tx
> > +\item[4] event
> > +\end{description}
> > +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> > +only uses 3 queues, as the following.
> 
> We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
> provide the possibility to support for example only dgrams.
> 
> So I think we should consider the case where we have only DGRAM queues
> (and it will be similar to the stream only case so 3 queues).
> 
> Maybe we could describe this part better and say that if we have both
> STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
> only 3 queues.
> 

Roger that.

> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
> > stream sockets. The guest and the device publish how much buffer space is
> > @@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> > \end{lstlisting}
> > 
> > -If there is insufficient buffer space, the sender waits until virtqueue buffers
> > +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> 
> stream and seqpacket
> 
> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> > @@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> > communicating updates any time a change in buffer space occurs.
> > 
> > +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
> > +or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
> > +into two parts: senders and receivers. For senders, the packet is dropped if the
> > +virtqueue is full. For receivers, the packet is dropped if there is no space
> > +in the receive buffer.
> > +
> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > -sufficient free buffer space for the payload.
> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> 
> stream and seqpacket
> 
> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > +and driver will not get any notification.
> > 
> > All packets associated with a stream flow MUST contain valid information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> > 
> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > -sufficient free buffer space for the payload.
> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> 
> stream and seqpacket
> 

Roger that to all three instances above.

> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > +and the device will not get any notification.
> > 
> > All packets associated with a stream flow MUST contain valid information in
> > \field{buf_alloc} and \field{fwd_cnt} fields.
> > 
> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> > -The driver queues outgoing packets on the tx virtqueue and incoming packet
> > +The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> 
> Is this change related?
> 

I think we can remove this change.


> > receive buffers on the rx virtqueue. Packets are of the following form:
> > 
> > \begin{lstlisting}
> > @@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> > incoming packets are write-only.
> > 
> > +When transmitting packets to the device, \field{num_buffers} is not used.
> > +
> 
> Leftover? Perhaps it should go in patch 2.
> 

Ah yes, I thought I had the two well-separated but this snuck out from
under me.

> > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> > 
> > The \field{guest_cid} configuration field MUST be used as the source CID when
> > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> > \end{lstlisting}
> > 
> > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> > +
> > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> > +a receiving socket does not exist on the destination.
> > +If the transmission or receiving buffers are full, the packets
> > +are dropped.
> > +
> 
> I'm not sure we should respond with RST if there's no socket bind on
> the port.
> 
> What happens with UDP if we do a sendto to a closed port?
> 
> Thanks,
> Stefano
> 

With UDP this results in an ICMP Destination Unreachable message, which
is explicitly not UDP but is experienced by the application nonetheless.
There was some discussion from v1, and the design choice essentially
came down to "how much do we want to be emulating of ICMP inside
vsock?"


- Bobby

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > Hi Bobby,
> > Sorry for the delay, but I saw these patches today.
> > Please, can you keep me in CC?
> > 
> 
> Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> one. I'll make sure you are there moving forward.
> 
> I want to mention that I'm taking a look at
> https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> here. After sending out this series we noticed potential overlap between
> the two issues. The additional dgram queues may become redundant if a
> fairness mechanism that solves issue #1 above also applies to
> connection-less protocols (similar to how the TC subsystem works). I've
> just begun sorting out potential solutions so no hard results yet. Just
> putting on your radar that the proposal here in v5 may be impacted if my
> investigation into issue #1 yields something adequate.


Well not mergeable, but datagram is upstream in Linux, is it not?
So we do want it in the spec IMHO, even if in the future there
will be a better protocol.

> > On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
> > > From: Jiang Wang <jiang.wang@bytedance.com>
> > > 
> 
> ... snip ...
> 
> > > 
> > > virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
> > > 1 file changed, 53 insertions(+), 10 deletions(-)
> > > 
> > > diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> > > index d79984d..1a66a1b 100644
> > > --- a/virtio-vsock.tex
> > > +++ b/virtio-vsock.tex
> > > @@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> > > 
> > > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> > > \begin{description}
> > > -\item[0] rx
> > > -\item[1] tx
> > > +\item[0] stream rx
> > > +\item[1] stream tx
> > > +\item[2] datagram rx
> > > +\item[3] datagram tx
> > > +\item[4] event
> > > +\end{description}
> > > +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> > > +only uses 3 queues, as the following.
> > 
> > We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
> > provide the possibility to support for example only dgrams.
> > 
> > So I think we should consider the case where we have only DGRAM queues
> > (and it will be similar to the stream only case so 3 queues).
> > 
> > Maybe we could describe this part better and say that if we have both
> > STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
> > only 3 queues.
> > 
> 
> Roger that.
> 
> > > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
> > > stream sockets. The guest and the device publish how much buffer space is
> > > @@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> > > \end{lstlisting}
> > > 
> > > -If there is insufficient buffer space, the sender waits until virtqueue buffers
> > > +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> > 
> > stream and seqpacket
> > 
> > > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> > > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> > > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> > > @@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> > > communicating updates any time a change in buffer space occurs.
> > > 
> > > +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
> > > +or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
> > > +into two parts: senders and receivers. For senders, the packet is dropped if the
> > > +virtqueue is full. For receivers, the packet is dropped if there is no space
> > > +in the receive buffer.
> > > +
> > > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > > -sufficient free buffer space for the payload.
> > > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > 
> > stream and seqpacket
> > 
> > > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > > +and driver will not get any notification.
> > > 
> > > All packets associated with a stream flow MUST contain valid information in
> > > \field{buf_alloc} and \field{fwd_cnt} fields.
> > > 
> > > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > > -sufficient free buffer space for the payload.
> > > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > 
> > stream and seqpacket
> > 
> 
> Roger that to all three instances above.
> 
> > > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > > +and the device will not get any notification.
> > > 
> > > All packets associated with a stream flow MUST contain valid information in
> > > \field{buf_alloc} and \field{fwd_cnt} fields.
> > > 
> > > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> > > -The driver queues outgoing packets on the tx virtqueue and incoming packet
> > > +The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> > 
> > Is this change related?
> > 
> 
> I think we can remove this change.
> 
> 
> > > receive buffers on the rx virtqueue. Packets are of the following form:
> > > 
> > > \begin{lstlisting}
> > > @@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> > > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> > > incoming packets are write-only.
> > > 
> > > +When transmitting packets to the device, \field{num_buffers} is not used.
> > > +
> > 
> > Leftover? Perhaps it should go in patch 2.
> > 
> 
> Ah yes, I thought I had the two well-separated but this snuck out from
> under me.
> 
> > > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> > > 
> > > The \field{guest_cid} configuration field MUST be used as the source CID when
> > > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> > > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> > > \end{lstlisting}
> > > 
> > > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> > > +
> > > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> > > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> > > +a receiving socket does not exist on the destination.
> > > +If the transmission or receiving buffers are full, the packets
> > > +are dropped.
> > > +
> > 
> > I'm not sure we should respond with RST if there's no socket bind on
> > the port.
> > 
> > What happens with UDP if we do a sendto to a closed port?
> > 
> > Thanks,
> > Stefano
> > 
> 
> With UDP this results in an ICMP Destination Unreachable message, which
> is explicitly not UDP but is experienced by the application nonetheless.
> There was some discussion from v1, and the design choice essentially
> came down to "how much do we want to be emulating of ICMP inside
> vsock?"
> 
> 
> - Bobby

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > Hi Bobby,
> > Sorry for the delay, but I saw these patches today.
> > Please, can you keep me in CC?
> > 
> 
> Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> one. I'll make sure you are there moving forward.
> 
> I want to mention that I'm taking a look at
> https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> here. After sending out this series we noticed potential overlap between
> the two issues. The additional dgram queues may become redundant if a
> fairness mechanism that solves issue #1 above also applies to
> connection-less protocols (similar to how the TC subsystem works). I've
> just begun sorting out potential solutions so no hard results yet. Just
> putting on your radar that the proposal here in v5 may be impacted if my
> investigation into issue #1 yields something adequate.


Well not mergeable, but datagram is upstream in Linux, is it not?
So we do want it in the spec IMHO, even if in the future there
will be a better protocol.

> > On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
> > > From: Jiang Wang <jiang.wang@bytedance.com>
> > > 
> 
> ... snip ...
> 
> > > 
> > > virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
> > > 1 file changed, 53 insertions(+), 10 deletions(-)
> > > 
> > > diff --git a/virtio-vsock.tex b/virtio-vsock.tex
> > > index d79984d..1a66a1b 100644
> > > --- a/virtio-vsock.tex
> > > +++ b/virtio-vsock.tex
> > > @@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
> > > 
> > > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
> > > \begin{description}
> > > -\item[0] rx
> > > -\item[1] tx
> > > +\item[0] stream rx
> > > +\item[1] stream tx
> > > +\item[2] datagram rx
> > > +\item[3] datagram tx
> > > +\item[4] event
> > > +\end{description}
> > > +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
> > > +only uses 3 queues, as the following.
> > 
> > We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
> > provide the possibility to support for example only dgrams.
> > 
> > So I think we should consider the case where we have only DGRAM queues
> > (and it will be similar to the stream only case so 3 queues).
> > 
> > Maybe we could describe this part better and say that if we have both
> > STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
> > only 3 queues.
> > 
> 
> Roger that.
> 
> > > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
> > > stream sockets. The guest and the device publish how much buffer space is
> > > @@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
> > > \end{lstlisting}
> > > 
> > > -If there is insufficient buffer space, the sender waits until virtqueue buffers
> > > +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
> > 
> > stream and seqpacket
> > 
> > > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
> > > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
> > > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
> > > @@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
> > > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
> > > communicating updates any time a change in buffer space occurs.
> > > 
> > > +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
> > > +or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
> > > +into two parts: senders and receivers. For senders, the packet is dropped if the
> > > +virtqueue is full. For receivers, the packet is dropped if there is no space
> > > +in the receive buffer.
> > > +
> > > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > > -sufficient free buffer space for the payload.
> > > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > 
> > stream and seqpacket
> > 
> > > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > > +and driver will not get any notification.
> > > 
> > > All packets associated with a stream flow MUST contain valid information in
> > > \field{buf_alloc} and \field{fwd_cnt} fields.
> > > 
> > > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
> > > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > > -sufficient free buffer space for the payload.
> > > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
> > 
> > stream and seqpacket
> > 
> 
> Roger that to all three instances above.
> 
> > > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
> > > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
> > > +and the device will not get any notification.
> > > 
> > > All packets associated with a stream flow MUST contain valid information in
> > > \field{buf_alloc} and \field{fwd_cnt} fields.
> > > 
> > > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
> > > -The driver queues outgoing packets on the tx virtqueue and incoming packet
> > > +The driver queues outgoing packets on the tx virtqueue and allocates incoming packet
> > 
> > Is this change related?
> > 
> 
> I think we can remove this change.
> 
> 
> > > receive buffers on the rx virtqueue. Packets are of the following form:
> > > 
> > > \begin{lstlisting}
> > > @@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
> > > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
> > > incoming packets are write-only.
> > > 
> > > +When transmitting packets to the device, \field{num_buffers} is not used.
> > > +
> > 
> > Leftover? Perhaps it should go in patch 2.
> > 
> 
> Ah yes, I thought I had the two well-separated but this snuck out from
> under me.
> 
> > > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> > > 
> > > The \field{guest_cid} configuration field MUST be used as the source CID when
> > > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> > > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> > > \end{lstlisting}
> > > 
> > > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> > > +
> > > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> > > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> > > +a receiving socket does not exist on the destination.
> > > +If the transmission or receiving buffers are full, the packets
> > > +are dropped.
> > > +
> > 
> > I'm not sure we should respond with RST if there's no socket bind on
> > the port.
> > 
> > What happens with UDP if we do a sendto to a closed port?
> > 
> > Thanks,
> > Stefano
> > 
> 
> With UDP this results in an ICMP Destination Unreachable message, which
> is explicitly not UDP but is experienced by the application nonetheless.
> There was some discussion from v1, and the design choice essentially
> came down to "how much do we want to be emulating of ICMP inside
> vsock?"
> 
> 
> - Bobby

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
>On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
>> Hi Bobby,
>> Sorry for the delay, but I saw these patches today.
>> Please, can you keep me in CC?
>>
>
>Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
>one. I'll make sure you are there moving forward.

No problem :-)

>
>I want to mention that I'm taking a look at
>https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
>here. After sending out this series we noticed potential overlap between
>the two issues. The additional dgram queues may become redundant if a
>fairness mechanism that solves issue #1 above also applies to
>connection-less protocols (similar to how the TC subsystem works). I've
>just begun sorting out potential solutions so no hard results yet. Just
>putting on your radar that the proposal here in v5 may be impacted if my
>investigation into issue #1 yields something adequate.

Oh, this would be great!

>
>> On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
>> > From: Jiang Wang <jiang.wang@bytedance.com>
>> >
>
>... snip ...
>
>> >
>> > virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
>> > 1 file changed, 53 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> > index d79984d..1a66a1b 100644
>> > --- a/virtio-vsock.tex
>> > +++ b/virtio-vsock.tex
>> > @@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >
>> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> > \begin{description}
>> > -\item[0] rx
>> > -\item[1] tx
>> > +\item[0] stream rx
>> > +\item[1] stream tx
>> > +\item[2] datagram rx
>> > +\item[3] datagram tx
>> > +\item[4] event
>> > +\end{description}
>> > +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> > +only uses 3 queues, as the following.
>>
>> We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
>> provide the possibility to support for example only dgrams.
>>
>> So I think we should consider the case where we have only DGRAM queues
>> (and it will be similar to the stream only case so 3 queues).
>>
>> Maybe we could describe this part better and say that if we have both
>> STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
>> only 3 queues.
>>
>
>Roger that.
>
>> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
>> > stream sockets. The guest and the device publish how much buffer space is
>> > @@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> > \end{lstlisting}
>> >
>> > -If there is insufficient buffer space, the sender waits until virtqueue buffers
>> > +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>>
>> stream and seqpacket
>>
>> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> > @@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> > communicating updates any time a change in buffer space occurs.
>> >
>> > +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
>> > +or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
>> > +into two parts: senders and receivers. For senders, the packet is dropped if the
>> > +virtqueue is full. For receivers, the packet is dropped if there is no space
>> > +in the receive buffer.
>> > +
>> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> > -sufficient free buffer space for the payload.
>> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>>
>> stream and seqpacket
>>
>> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> > +and driver will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> > -sufficient free buffer space for the payload.
>> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>>
>> stream and seqpacket
>>
>
>Roger that to all three instances above.
>
>> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> > +and the device will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>> > -The driver queues outgoing packets on the tx virtqueue and incoming packet
>> > +The driver queues outgoing packets on the tx virtqueue and 
>> > allocates incoming packet
>>
>> Is this change related?
>>
>
>I think we can remove this change.
>
>
>> > receive buffers on the rx virtqueue. Packets are of the following form:
>> >
>> > \begin{lstlisting}
>> > @@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>> > incoming packets are write-only.
>> >
>> > +When transmitting packets to the device, \field{num_buffers} is not used.
>> > +
>>
>> Leftover? Perhaps it should go in patch 2.
>>
>
>Ah yes, I thought I had the two well-separated but this snuck out from
>under me.
>
>> > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST be used as the source CID when
>> > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
>> > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
>> > \end{lstlisting}
>> >
>> > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>> > +
>> > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>> > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>> > +a receiving socket does not exist on the destination.
>> > +If the transmission or receiving buffers are full, the packets
>> > +are dropped.
>> > +
>>
>> I'm not sure we should respond with RST if there's no socket bind on
>> the port.
>>
>> What happens with UDP if we do a sendto to a closed port?
>>
>> Thanks,
>> Stefano
>>
>
>With UDP this results in an ICMP Destination Unreachable message, which
>is explicitly not UDP but is experienced by the application nonetheless.
>There was some discussion from v1, and the design choice essentially
>came down to "how much do we want to be emulating of ICMP inside
>vsock?"

Okay, I see, but how this is propagate to the userspace?

IIUC for UDP the user should open a RAW socket with IPPROTO_ICMP and 
wait for an error message.

Have you taken a look at a possible implementation with AF_VSOCK yet?

In any case if it can be useful we could include it in the spec and then 
implement it later in Linux.

Thanks,
Stefano

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Stefano Garzarella]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
>On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
>> Hi Bobby,
>> Sorry for the delay, but I saw these patches today.
>> Please, can you keep me in CC?
>>
>
>Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
>one. I'll make sure you are there moving forward.

No problem :-)

>
>I want to mention that I'm taking a look at
>https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
>here. After sending out this series we noticed potential overlap between
>the two issues. The additional dgram queues may become redundant if a
>fairness mechanism that solves issue #1 above also applies to
>connection-less protocols (similar to how the TC subsystem works). I've
>just begun sorting out potential solutions so no hard results yet. Just
>putting on your radar that the proposal here in v5 may be impacted if my
>investigation into issue #1 yields something adequate.

Oh, this would be great!

>
>> On Thu, Feb 24, 2022 at 10:15:46PM +0000, beshleman.devbox@gmail.com wrote:
>> > From: Jiang Wang <jiang.wang@bytedance.com>
>> >
>
>... snip ...
>
>> >
>> > virtio-vsock.tex | 63 +++++++++++++++++++++++++++++++++++++++++++++++---------
>> > 1 file changed, 53 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/virtio-vsock.tex b/virtio-vsock.tex
>> > index d79984d..1a66a1b 100644
>> > --- a/virtio-vsock.tex
>> > +++ b/virtio-vsock.tex
>> > @@ -9,11 +9,26 @@ \subsection{Device ID}\label{sec:Device Types / Socket Device / Device ID}
>> >
>> > \subsection{Virtqueues}\label{sec:Device Types / Socket Device / Virtqueues}
>> > \begin{description}
>> > -\item[0] rx
>> > -\item[1] tx
>> > +\item[0] stream rx
>> > +\item[1] stream tx
>> > +\item[2] datagram rx
>> > +\item[3] datagram tx
>> > +\item[4] event
>> > +\end{description}
>> > +The virtio socket device uses 5 queues if feature bit VIRTIO_VSOCK_F_DRGAM is set. Otherwise, it
>> > +only uses 3 queues, as the following.
>>
>> We are also adding a new flag (VIRTIO_VSOCK_F_NO_IMPLIED_STREAM) to
>> provide the possibility to support for example only dgrams.
>>
>> So I think we should consider the case where we have only DGRAM queues
>> (and it will be similar to the stream only case so 3 queues).
>>
>> Maybe we could describe this part better and say that if we have both
>> STREAM (or SEQPACKET) and DGRAM set we have 5 queues, otherwise
>> only 3 queues.
>>
>
>Roger that.
>
>> > \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > \field{buf_alloc} and \field{fwd_cnt} are used for buffer space management of
>> > stream sockets. The guest and the device publish how much buffer space is
>> > @@ -170,7 +193,7 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > u32 peer_free = peer_buf_alloc - (tx_cnt - peer_fwd_cnt);
>> > \end{lstlisting}
>> >
>> > -If there is insufficient buffer space, the sender waits until virtqueue buffers
>> > +For stream sockets, if there is insufficient buffer space, the sender waits until virtqueue buffers
>>
>> stream and seqpacket
>>
>> > are returned and checks \field{buf_alloc} and \field{fwd_cnt} again. Sending
>> > the VIRTIO_VSOCK_OP_CREDIT_REQUEST packet queries how much buffer space is
>> > available. The reply to this query is a VIRTIO_VSOCK_OP_CREDIT_UPDATE packet.
>> > @@ -178,22 +201,32 @@ \subsubsection{Buffer Space Management}\label{sec:Device Types / Socket Device /
>> > previously receiving a VIRTIO_VSOCK_OP_CREDIT_REQUEST packet. This allows
>> > communicating updates any time a change in buffer space occurs.
>> >
>> > +Unlike stream sockets, dgram sockets do not use VIRTIO_VSOCK_OP_CREDIT_UPDATE
>> > +or VIRTIO_VSOCK_OP_CREDIT_REQUEST packets. The dgram buffer management is split
>> > +into two parts: senders and receivers. For senders, the packet is dropped if the
>> > +virtqueue is full. For receivers, the packet is dropped if there is no space
>> > +in the receive buffer.
>> > +
>> > \drivernormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> > -sufficient free buffer space for the payload.
>> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>>
>> stream and seqpacket
>>
>> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> > +and driver will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \devicenormative{\paragraph}{Device Operation: Buffer Space Management}{Device Types / Socket Device / Device Operation / Buffer Space Management}
>> > -VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>> > -sufficient free buffer space for the payload.
>> > +For stream sockets, VIRTIO_VSOCK_OP_RW data packets MUST only be transmitted when the peer has
>>
>> stream and seqpacket
>>
>
>Roger that to all three instances above.
>
>> > +sufficient free buffer space for the payload. For dgram sockets, VIRTIO_VSOCK_OP_RW data packets
>> > +MAY be transmitted when the peer rx buffer is full. Then the packet will be dropped by the peer,
>> > +and the device will not get any notification.
>> >
>> > All packets associated with a stream flow MUST contain valid information in
>> > \field{buf_alloc} and \field{fwd_cnt} fields.
>> >
>> > \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / Device Operation / Receive and Transmit}
>> > -The driver queues outgoing packets on the tx virtqueue and incoming packet
>> > +The driver queues outgoing packets on the tx virtqueue and 
>> > allocates incoming packet
>>
>> Is this change related?
>>
>
>I think we can remove this change.
>
>
>> > receive buffers on the rx virtqueue. Packets are of the following form:
>> >
>> > \begin{lstlisting}
>> > @@ -206,6 +239,8 @@ \subsubsection{Receive and Transmit}\label{sec:Device Types / Socket Device / De
>> > Virtqueue buffers for outgoing packets are read-only. Virtqueue buffers for
>> > incoming packets are write-only.
>> >
>> > +When transmitting packets to the device, \field{num_buffers} is not used.
>> > +
>>
>> Leftover? Perhaps it should go in patch 2.
>>
>
>Ah yes, I thought I had the two well-separated but this snuck out from
>under me.
>
>> > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
>> >
>> > The \field{guest_cid} configuration field MUST be used as the source CID when
>> > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
>> > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
>> > \end{lstlisting}
>> >
>> > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
>> > +
>> > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
>> > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
>> > +a receiving socket does not exist on the destination.
>> > +If the transmission or receiving buffers are full, the packets
>> > +are dropped.
>> > +
>>
>> I'm not sure we should respond with RST if there's no socket bind on
>> the port.
>>
>> What happens with UDP if we do a sendto to a closed port?
>>
>> Thanks,
>> Stefano
>>
>
>With UDP this results in an ICMP Destination Unreachable message, which
>is explicitly not UDP but is experienced by the application nonetheless.
>There was some discussion from v1, and the design choice essentially
>came down to "how much do we want to be emulating of ICMP inside
>vsock?"

Okay, I see, but how this is propagate to the userspace?

IIUC for UDP the user should open a RAW socket with IPPROTO_ICMP and 
wait for an error message.

Have you taken a look at a possible implementation with AF_VSOCK yet?

In any case if it can be useful we could include it in the spec and then 
implement it later in Linux.

Thanks,
Stefano

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > Hi Bobby,
> > Sorry for the delay, but I saw these patches today.
> > Please, can you keep me in CC?
> > 
> 
> Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> one. I'll make sure you are there moving forward.
> 
> I want to mention that I'm taking a look at
> https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> here. After sending out this series we noticed potential overlap between
> the two issues. The additional dgram queues may become redundant if a
> fairness mechanism that solves issue #1 above also applies to
> connection-less protocols (similar to how the TC subsystem works). I've
> just begun sorting out potential solutions so no hard results yet. Just
> putting on your radar that the proposal here in v5 may be impacted if my
> investigation into issue #1 yields something adequate.


With respect to datagram, there is actually another issue which also
exists for stream but is smaller there - per message overhead is not
accounted for.  For stream we can work around that by copying payload
data.  For datagram we can't as we need to preserve message boundaries.
One way to address that is to add config for host/guest per-message
overhead, and have sender decrement fwd counter by that value for
each message sent.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > Hi Bobby,
> > Sorry for the delay, but I saw these patches today.
> > Please, can you keep me in CC?
> > 
> 
> Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> one. I'll make sure you are there moving forward.
> 
> I want to mention that I'm taking a look at
> https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> here. After sending out this series we noticed potential overlap between
> the two issues. The additional dgram queues may become redundant if a
> fairness mechanism that solves issue #1 above also applies to
> connection-less protocols (similar to how the TC subsystem works). I've
> just begun sorting out potential solutions so no hard results yet. Just
> putting on your radar that the proposal here in v5 may be impacted if my
> investigation into issue #1 yields something adequate.


With respect to datagram, there is actually another issue which also
exists for stream but is smaller there - per message overhead is not
accounted for.  For stream we can work around that by copying payload
data.  For datagram we can't as we need to preserve message boundaries.
One way to address that is to add config for host/guest per-message
overhead, and have sender decrement fwd counter by that value for
each message sent.

-- 
MST

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Bobby Eshleman]

On Thu, Mar 03, 2022 at 02:15:24AM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> > On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > > Hi Bobby,
> > > Sorry for the delay, but I saw these patches today.
> > > Please, can you keep me in CC?
> > > 
> > 
> > Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> > one. I'll make sure you are there moving forward.
> > 
> > I want to mention that I'm taking a look at
> > https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> > here. After sending out this series we noticed potential overlap between
> > the two issues. The additional dgram queues may become redundant if a
> > fairness mechanism that solves issue #1 above also applies to
> > connection-less protocols (similar to how the TC subsystem works). I've
> > just begun sorting out potential solutions so no hard results yet. Just
> > putting on your radar that the proposal here in v5 may be impacted if my
> > investigation into issue #1 yields something adequate.
> 
> 
> Well not mergeable, but datagram is upstream in Linux, is it not?
> So we do want it in the spec IMHO, even if in the future there
> will be a better protocol.
> 

As of right now, it is not upstream in Linux. The virtio transport just passes
-EOPNOTSUPP up the stack when the sock invokes it.

I think what you're thinking of is the vsock dgram in VMWare's vmci and
Hyper-V. They support dgrams, but are not compatible with virtio (e.g., don't
use virtqueues).

-Bobby

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Bobby Eshleman]

On Thu, Mar 03, 2022 at 11:23:39AM +0100, Stefano Garzarella wrote:
> On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> > 
> > > > \drivernormative{\paragraph}{Device Operation: Receive and Transmit}{Device Types / Socket Device / Device Operation / Receive and Transmit}
> > > >
> > > > The \field{guest_cid} configuration field MUST be used as the source CID when
> > > > @@ -274,6 +309,14 @@ \subsubsection{Seqpacket Sockets}\label{sec:Device Types / Socket Device / Devic
> > > > #define VIRTIO_VSOCK_SEQ_EOR (1 << 1)
> > > > \end{lstlisting}
> > > >
> > > > +\subsubsection{Datagram Sockets}\label{sec:Device Types / Socket Device / Device Operation / Datagram Sockets}
> > > > +
> > > > +Datagram (dgram) sockets are connectionless and unreliable. The sender just sends
> > > > +a message to the peer and hopes it will be delivered. A VIRTIO_VSOCK_OP_RST reply is sent if
> > > > +a receiving socket does not exist on the destination.
> > > > +If the transmission or receiving buffers are full, the packets
> > > > +are dropped.
> > > > +
> > > 
> > > I'm not sure we should respond with RST if there's no socket bind on
> > > the port.
> > > 
> > > What happens with UDP if we do a sendto to a closed port?
> > > 
> > > Thanks,
> > > Stefano
> > > 
> > 
> > With UDP this results in an ICMP Destination Unreachable message, which
> > is explicitly not UDP but is experienced by the application nonetheless.
> > There was some discussion from v1, and the design choice essentially
> > came down to "how much do we want to be emulating of ICMP inside
> > vsock?"
> 
> Okay, I see, but how this is propagate to the userspace?
> 
> IIUC for UDP the user should open a RAW socket with IPPROTO_ICMP and wait
> for an error message.
> 

I'll write a sample test for this and get back to you, just to be certain.

... snip ...

> Have you taken a look at a possible implementation with AF_VSOCK yet?
> 
> In any case if it can be useful we could include it in the spec and then
> implement it later in Linux.

I have not yet, and from my reading of the code it was not in Jiang's Linux
series yet.

Although I understand the logic behind it, I can't say I'm totally confident in
this yet. It is carry over from v4 and has roots in a discussion between
stefanha and cong.wong in the RFC thread "[RFC PATCH] virtio-vsock: add
description for datagram type" (sorry I don't have the Message-Id, it was just
forwarded to me, and it is missing from lore.kernel.org for some reason).

I'll look into it more deeply in terms of a potential AF_VSOCK implementation
and build a stronger case for/against in the next rev.

Thanks,
Bobby

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Sat, Mar 05, 2022 at 01:25:44AM +0000, Bobby Eshleman wrote:
> On Thu, Mar 03, 2022 at 02:15:24AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> > > On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > > > Hi Bobby,
> > > > Sorry for the delay, but I saw these patches today.
> > > > Please, can you keep me in CC?
> > > > 
> > > 
> > > Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> > > one. I'll make sure you are there moving forward.
> > > 
> > > I want to mention that I'm taking a look at
> > > https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> > > here. After sending out this series we noticed potential overlap between
> > > the two issues. The additional dgram queues may become redundant if a
> > > fairness mechanism that solves issue #1 above also applies to
> > > connection-less protocols (similar to how the TC subsystem works). I've
> > > just begun sorting out potential solutions so no hard results yet. Just
> > > putting on your radar that the proposal here in v5 may be impacted if my
> > > investigation into issue #1 yields something adequate.
> > 
> > 
> > Well not mergeable, but datagram is upstream in Linux, is it not?
> > So we do want it in the spec IMHO, even if in the future there
> > will be a better protocol.
> > 
> 
> As of right now, it is not upstream in Linux. The virtio transport just passes
> -EOPNOTSUPP up the stack when the sock invokes it.
> 
> I think what you're thinking of is the vsock dgram in VMWare's vmci and
> Hyper-V. They support dgrams, but are not compatible with virtio (e.g., don't
> use virtqueues).
> 
> -Bobby

Oh no, I was thinking about SEQPACKET actually. Which has the same issue
I noted on another thread with memory accounting as datagram by the way.

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Michael S. Tsirkin]

On Sat, Mar 05, 2022 at 01:25:44AM +0000, Bobby Eshleman wrote:
> On Thu, Mar 03, 2022 at 02:15:24AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> > > On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > > > Hi Bobby,
> > > > Sorry for the delay, but I saw these patches today.
> > > > Please, can you keep me in CC?
> > > > 
> > > 
> > > Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> > > one. I'll make sure you are there moving forward.
> > > 
> > > I want to mention that I'm taking a look at
> > > https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> > > here. After sending out this series we noticed potential overlap between
> > > the two issues. The additional dgram queues may become redundant if a
> > > fairness mechanism that solves issue #1 above also applies to
> > > connection-less protocols (similar to how the TC subsystem works). I've
> > > just begun sorting out potential solutions so no hard results yet. Just
> > > putting on your radar that the proposal here in v5 may be impacted if my
> > > investigation into issue #1 yields something adequate.
> > 
> > 
> > Well not mergeable, but datagram is upstream in Linux, is it not?
> > So we do want it in the spec IMHO, even if in the future there
> > will be a better protocol.
> > 
> 
> As of right now, it is not upstream in Linux. The virtio transport just passes
> -EOPNOTSUPP up the stack when the sock invokes it.
> 
> I think what you're thinking of is the vsock dgram in VMWare's vmci and
> Hyper-V. They support dgrams, but are not compatible with virtio (e.g., don't
> use virtqueues).
> 
> -Bobby

Oh no, I was thinking about SEQPACKET actually. Which has the same issue
I noted on another thread with memory accounting as datagram by the way.

-- 
MST

Re: [virtio-comment] [PATCH v5 1/2] virtio-vsock: add description for datagram type

[Bobby Eshleman]

On Thu, Mar 03, 2022 at 06:41:11AM -0500, Michael S. Tsirkin wrote:
> On Thu, Mar 03, 2022 at 03:29:31AM +0000, Bobby Eshleman wrote:
> > On Wed, Mar 02, 2022 at 05:09:58PM +0100, Stefano Garzarella wrote:
> > > Hi Bobby,
> > > Sorry for the delay, but I saw these patches today.
> > > Please, can you keep me in CC?
> > > 
> > 
> > Hey Stefano, sorry about that. I'm not sure how I lost your CC on this
> > one. I'll make sure you are there moving forward.
> > 
> > I want to mention that I'm taking a look at
> > https://gitlab.com/vsock/vsock/-/issues/1 in parallel with my dgram work
> > here. After sending out this series we noticed potential overlap between
> > the two issues. The additional dgram queues may become redundant if a
> > fairness mechanism that solves issue #1 above also applies to
> > connection-less protocols (similar to how the TC subsystem works). I've
> > just begun sorting out potential solutions so no hard results yet. Just
> > putting on your radar that the proposal here in v5 may be impacted if my
> > investigation into issue #1 yields something adequate.
> 
> 
> With respect to datagram, there is actually another issue which also
> exists for stream but is smaller there - per message overhead is not
> accounted for.  For stream we can work around that by copying payload
> data.  For datagram we can't as we need to preserve message boundaries.
> One way to address that is to add config for host/guest per-message
> overhead, and have sender decrement fwd counter by that value for
> each message sent.
> 

As I'm just becoming acquainted with the code excuse me if this question is
naive: is the overhead not knowable by the implementation, and that is why a
configuration would be required? My naive assumption was that overhead is
coming from headers (and maybe other packet metadata?) which are known by the
implementation?

Thanks,
Bobby

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/