Question about extra 40 bytes needed for UD receive buffer

Question about extra 40 bytes needed for UD receive buffer

[Honggang LI]

hi,

According to volume 1 of IBAT specification of release 1.4,

1) 11.4.1.2 POST RECEIVE REQUEST
Note that for UD QPs, the first 40 bytes of the buffer(s) referred
to by the Scatter/Gather list will contain the GRH of the incoming message.
If no GRH is present, the contents of first 40 bytes of the buffer(s) will
be undefined. The presence of the GRH will be indicated by a bit in the
Work Completion.

2) 11.4.2.1 POLL FOR COMPLETION
GRH Present indicator, for UD RQs only. If this indicator is
set, the first 40 bytes of the buffer(s) referred to by the Scatter/Gather
list will contain the GRH of the incoming message. If it is not set, the
contents of first 40 bytes of the buffer(s) will be undefined. Contents 
of the payload of the message will begin after the first 40 bytes

3) A17.4.5.2 SCATTERING OF THE L3 HEADER IN UD
The first 40 bytes of user posted UD Receive Buffers are reserved for the
L3 header of the incoming packet (as per the InfiniBand Spec Section
11.4.1.2). In RoCEv2, this area is filled up with the IP header. IPv6 header
uses the entire 40 bytes. IPv4 headers use the 20 bytes in the second half
of the reserved 40 bytes area (i.e. offset 20 from the beginning of the
receive buffer). In this case, the content of the first 20 bytes is undefined.



After function `ibv_poll_cq` return 1, the dump of receive buffer of
`ibv_ud_pingpong` shows there is 40 bytes GRH in the head of receive
buffer. The first 20 bytes are zeros and the second 20 bytes is a valid
IP header. At this point, everything works as required by the
specification rules 1), 2) and 3).

However, I'm confused by the dump of `mckey` receive buffer. The flag
`IBV_WC_GRH` is set in `ibv_wc.wc_flags`, but there is no GRH in
the receive buffer. Received data starts from the *first* byte of
receive buffer. As multicast over UD QP only, can someone please explian
why there is no GRH in receive buffer and the data starts from the first
bytes of receive buffer with `IBV_WC_GRH` was set?

Thanks

Re: Question about extra 40 bytes needed for UD receive buffer

[Honggang LI]

On Mon, Apr 15, 2024 at 10:55:12AM +0800, Honggang LI wrote:
> Subject: Question about extra 40 bytes needed for UD receive buffer
> From: Honggang LI <honggangli@163.com>
> Date: Mon, 15 Apr 2024 10:55:12 +0800
> 
> However, I'm confused by the dump of `mckey` receive buffer. The flag
> `IBV_WC_GRH` is set in `ibv_wc.wc_flags`, but there is no GRH in
> the receive buffer. Received data starts from the *first* byte of
> receive buffer. As multicast over UD QP only, can someone please explian
> why there is no GRH in receive buffer and the data starts from the first
> bytes of receive buffer with `IBV_WC_GRH` was set?

Please ignore this. I made a mistake with the offset of the grh.

There is a valid IP header in the receive buffer of 'mckey'.