RE: [virtio-comment] Re: [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission

[Parav Pandit <parav@nvidia.com>]

> From: zhenwei pi <pizhenwei@bytedance.com>
> Sent: Monday, June 5, 2023 4:41 AM
> 
> On 6/5/23 10:41, Parav Pandit wrote:
> >
> >
> > On 5/4/2023 4:19 AM, zhenwei pi wrote:
> >> Keyed transmission is used for message oriented communication(Ex
> >> RDMA), also add virtio-blk read/write 8K example.
> >>
> >> Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
> >> ---
> >
> >> +An example of a virtio-blk write 8K request(message size:
> >> sizeof(Command) +
> >> +4 * sizeof(Descriptor)):
> >> +\begin{lstlisting}
> >> + COMMAND            +------+
> >> +                    |opcode|  ->  virtio_of_op_vring
> >> +                    +------+
> >> +                    |cmd id|  ->  10
> >> +                    +------+
> >> +                    |length|  ->  0
> >> +                    +------+
> >> +                    |ndesc |  ->  4
> >> +                    +------+
> >> +                    |rsvd  |
> >> +                    +------+
> >> +
> >> + DESC0              +------+
> >> +                    |addr  |  -> 0xffff012345670000
> >> +                    +------+
> >> +                    |length|  -> 16 (virtio blk write command)
> >> +                    +------+
> >> +                    |id    |  -> 0
> >> +                    +------+
> >
> > for RDMA this id is not useful. It can be omitted.
> > still parsing the rest.
> >
> > if we talk blk as an example, above command descriptor can be of 32
> > bytes, such as struct virtio_of_cmd {
> >      u8 opcode;
> >      u8 rsvd;
> >      le16 cmd_id;
> >      u8 inline_desc_cnt;
> >      u8 rsvd[3];
> >      /* some padding/metadata for long desc list if any */ };
> >
> > struct virtio_of_rdma_desc {
> >      le64 addr;
> >      le32 length;
> >      le32 rdma_key;
> > };
> >
> > struct virtio_rdma_op {
> >      struct virtio_of_cmd cmd;
> >      struct virtio_of_rdma_desc desc[1 or 3]; /* count can be
> > negotiated */ };
> >
> > With this a send and receive queue on initiator and target can
> > exchange, cmd descriptor for read/writes.
> >
> 
> Hi,
> 
> Do you mean that separating a Virtio Over RDMA queue into 2 QP, one for
> sending, another one for receiving?
> 
No. just one QP.

Initiator_QP_A -> target_QP_B.

When initiator QP A sends 32B cmd, it lands in the target QP B's receive queue.

After this target can do one or more read/write DMA using RDMA read/write from the initiator's memory.

Finally target_QP_B sends 8B completion, it arrives in the QP_A's receive queue.