Re: ibv_req_notify_cq clarification

[Gal Pressman <galpress@amazon.com>]

On 18/02/2021 14:38, Bernard Metzler wrote:
> -----"Gal Pressman" <galpress@amazon.com> wrote: -----
> 
>> To: "RDMA mailing list" <linux-rdma@vger.kernel.org>
>> From: "Gal Pressman" <galpress@amazon.com>
>> Date: 02/18/2021 11:26AM
>> Subject: [EXTERNAL] ibv_req_notify_cq clarification
>>
>> I'm a bit confused about the meaning of the ibv_req_notify_cq() verb:
>> "Upon the addition of a new CQ entry (CQE) to cq, a completion event
>> will be
>> added to the completion channel associated with the CQ."
>>
>> What is considered a new CQE in this case?
>> The next CQE from the user's perspective, i.e. any new CQE that
>> wasn't consumed
>> by the user's poll cq?
>> Or any new CQE from the device's perspective?
>>
>> For example, if at the time of ibv_req_notify_cq() call the CQ has
>> received 100
>> completions, but the user hasn't polled his CQ yet, when should he be
>> notified?
>> On the 101 completion or immediately (since there are completions
>> waiting on the
>> CQ)?
> 
> It is from drivers perspective. So notification when 101
> became available and CQ is marked for notification.
> Applications tend to poll the CQ after requesting
> notification, since there might be a race
> from appl perspective...

Thanks Bernard,

Looking at the implementation of rdma-core providers, most of them pass the CQ
consumer index as part of the arm doorbell which made me think it arms the CQ to
notify on a completion newer than what the app polled, not on a completion newer
than the producer index.
Any idea why the consumer index is needed if that's not the case?