Re: Re: ibv_req_notify_cq clarification

["Bernard Metzler" <BMT@zurich.ibm.com>]

-----"Jason Gunthorpe" <jgg@ziepe.ca> wrote: -----

>To: "Gal Pressman" <galpress@amazon.com>
>From: "Jason Gunthorpe" <jgg@ziepe.ca>
>Date: 02/18/2021 07:35PM
>Cc: "RDMA mailing list" <linux-rdma@vger.kernel.org>
>Subject: [EXTERNAL] Re: ibv_req_notify_cq clarification
>
>On Thu, Feb 18, 2021 at 05:52:16PM +0200, Gal Pressman wrote:
>> On 18/02/2021 14:53, Jason Gunthorpe wrote:
>> > On Thu, Feb 18, 2021 at 11:13:43AM +0200, Gal Pressman wrote:
>> >> I'm a bit confused about the meaning of the ibv_req_notify_cq()
>verb:
>> >> "Upon the addition of a new CQ entry (CQE) to cq, a completion
>event will be
>> >> added to the completion channel associated with the CQ."
>> >>
>> >> What is considered a new CQE in this case?
>> >> The next CQE from the user's perspective, i.e. any new CQE that
>wasn't consumed
>> >> by the user's poll cq?
>> >> Or any new CQE from the device's perspective?
>> > 
>> > new CQE from the device perspective.
>> > 
>> >> For example, if at the time of ibv_req_notify_cq() call the CQ
>has received 100
>> >> completions, but the user hasn't polled his CQ yet, when should
>he be notified?
>> >> On the 101 completion or immediately (since there are
>completions waiting on the
>> >> CQ)?
>> > 
>> > 101 completion
>> > 
>> > It is only meaningful to call it when the CQ is empty.
>> 
>> Thanks, so there's an inherent race between the user's CQ poll and
>the next arm?
>
>I think the specs or man pages talk about this, the application has
>to
>observe empty, do arm, then poll again then sleep on the cq if empty.
>

I'd love to see the IB_CQ_REPORT_MISSED_EVENTS flag mechanics
available for user land verbs. I learned about this potential
race the hard way when a well known distributed FS sometimes
hung ;) 



>> Do you know what's the purpose of the consumer index in the arm
>doorbell that's
>> implemented by many providers?
>
>The consumer index is needed by HW to prevent CQ overflow, presumably
>the drivers push to reduce the cases where the HW has to read it from
>PCI
>
>Jason
>