UDP with IB verbs lib

UDP with IB verbs lib

[Dimitris Dimitropoulos]

Hi,

I'm looking at UDP using the IB verbs library. If I send UDP packets
that are intercepted by the IB verbs layer, what happens with the
completion notifications ?

For example, say I create a list of 10 ibv_recv_wr objects and each
has num_sge = 30, with each SGE having a 4K size. And I setup for
reception with ibv_post_recv(). If I transmit 30 UDP packets each 4K
in size will I receive one CQ event ? Or 30 ?

Will the UDP packets be written in consecutive SGEs of the first
ibv_recv_wr object ? Or will they be written in consecutive
ibv_recv_wr objects (in their first SGE) ?

Thank you.

Dimitris

Re: UDP with IB verbs lib

[Zhu Yanjun]

Please refer to SoftRoCE. The source code is here 
./drivers/infiniband/sw/rxe/

On 3/19/2020 7:07 AM, Dimitris Dimitropoulos wrote:
> Hi,
>
> I'm looking at UDP using the IB verbs library. If I send UDP packets
> that are intercepted by the IB verbs layer, what happens with the
> completion notifications ?
>
> For example, say I create a list of 10 ibv_recv_wr objects and each
> has num_sge = 30, with each SGE having a 4K size. And I setup for
> reception with ibv_post_recv(). If I transmit 30 UDP packets each 4K
> in size will I receive one CQ event ? Or 30 ?
>
> Will the UDP packets be written in consecutive SGEs of the first
> ibv_recv_wr object ? Or will they be written in consecutive
> ibv_recv_wr objects (in their first SGE) ?
>
> Thank you.
>
> Dimitris

Re: UDP with IB verbs lib

[Jason Gunthorpe]

On Wed, Mar 18, 2020 at 04:07:37PM -0700, Dimitris Dimitropoulos wrote:
> Hi,
> 
> I'm looking at UDP using the IB verbs library. If I send UDP packets
> that are intercepted by the IB verbs layer, what happens with the
> completion notifications ?

Each recv that requests a completion gets a completion.

> For example, say I create a list of 10 ibv_recv_wr objects and each
> has num_sge = 30, with each SGE having a 4K size. And I setup for
> reception with ibv_post_recv(). If I transmit 30 UDP packets each 4K
> in size will I receive one CQ event ? Or 30 ?

One per ibv_recv_wr requesting completion

> Will the UDP packets be written in consecutive SGEs of the first
> ibv_recv_wr object ? Or will they be written in consecutive
> ibv_recv_wr objects (in their first SGE) ?

Only the first. Each recv handles a single incoming packet.

DPDK has a hyper-optimized version of processing IP/UDP packets via
mlx5dv

Jason

Re: UDP with IB verbs lib

[Dimitris Dimitropoulos]

That's pretty clear. Thank you.

Dimitris

On Thu, Mar 19, 2020 at 6:52 AM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Wed, Mar 18, 2020 at 04:07:37PM -0700, Dimitris Dimitropoulos wrote:
> > Hi,
> >
> > I'm looking at UDP using the IB verbs library. If I send UDP packets
> > that are intercepted by the IB verbs layer, what happens with the
> > completion notifications ?
>
> Each recv that requests a completion gets a completion.
>
> > For example, say I create a list of 10 ibv_recv_wr objects and each
> > has num_sge = 30, with each SGE having a 4K size. And I setup for
> > reception with ibv_post_recv(). If I transmit 30 UDP packets each 4K
> > in size will I receive one CQ event ? Or 30 ?
>
> One per ibv_recv_wr requesting completion
>
> > Will the UDP packets be written in consecutive SGEs of the first
> > ibv_recv_wr object ? Or will they be written in consecutive
> > ibv_recv_wr objects (in their first SGE) ?
>
> Only the first. Each recv handles a single incoming packet.
>
> DPDK has a hyper-optimized version of processing IP/UDP packets via
> mlx5dv
>
> Jason