RE: RE: [virtio-comment] Re: [PATCH v2 05/11] transport-fabrics: introduce Keyed Transmission

[Parav Pandit <parav@nvidia.com>]

> From: zhenwei pi <pizhenwei@bytedance.com>
> Sent: Monday, June 5, 2023 8:50 AM


> >>> if we talk blk as an example, above command descriptor can be of 32
> >>> bytes, such as struct virtio_of_cmd {
> >>>       u8 opcode;
> >>>       u8 rsvd;
> >>>       le16 cmd_id;
> >>>       u8 inline_desc_cnt;
> >>>       u8 rsvd[3];
> >>>       /* some padding/metadata for long desc list if any */ };
> >>>
> >>> struct virtio_of_rdma_desc {
> >>>       le64 addr;
> >>>       le32 length;
> >>>       le32 rdma_key;
> >>> };
> >>>
> >>> struct virtio_rdma_op {
> >>>       struct virtio_of_cmd cmd;
> >>>       struct virtio_of_rdma_desc desc[1 or 3]; /* count can be
> >>> negotiated */ };
> >>>
> >>> With this a send and receive queue on initiator and target can
> >>> exchange, cmd descriptor for read/writes.
> >>>
> >>
> >> Hi,
> >>
> >> Do you mean that separating a Virtio Over RDMA queue into 2 QP, one
> >> for sending, another one for receiving?
> >>
> > No. just one QP.
> >
> > Initiator_QP_A -> target_QP_B.
> >
> > When initiator QP A sends 32B cmd, it lands in the target QP B's receive
> queue.
> >
> > After this target can do one or more read/write DMA using RDMA read/write
> from the initiator's memory.
> >
> 
> Hi, I have several questions:
> 1, how to tell the target to read/write DMA using RDMA read/write? is
> virtio_of_rdma_desc missing?
> 
Virtio_of_rdma_desc is part of the 32B struct virtio_rdma_op in above example.

> 2, if several virtio_of_rdma_desc arrives, the target need to distinguish READ *
> m + WRITE * n descriptors. but *flags* field has been removed ...
> 
The idea is to not have multiple virtio_of_rdma_desc.
An initiator can represent 1B to 4GB of noncontiguous buffer using a single rdma mkey.
Hence, only one virtio_of_rdma_desc is enough from initiator to target.

> 3, if I understand correctly, Initiator_QP_A -> Target_QP_B(CMD),
> Target_QP_B(RDMA READ), Target_QP_B(RDMA WRITE), Target_QP_B ->
> Initiator_QP_A(COMP). this uses 4 RTT.
> 
RDMA read and writes are for the actual variable size data of 512B, 4K, 1MB etc.

Optionally, a target can expose a constant size buffer where initiator can directly write the data of 512B, 4KB as well.
However, this doesn't scale very well always, but sure it is possible, and it only works for blk write commands.

In a more advanced scheme target can dynamically add such buffers and advertise it to the initiator.
I would think to make it incremental once the basic data flow model is established.

> > Finally target_QP_B sends 8B completion, it arrives in the QP_A's receive
> queue.
> 
> --
> zhenwei pi