Re: [PATCH v6 06/25] rtrs: client: main functionality

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Dec 30, 2019 at 03:53:10PM -0800, Bart Van Assche wrote:
> > +	if (req->sg_cnt) {
> > +		if (unlikely(req->dir == DMA_FROM_DEVICE && req->need_inv)) {
> > +			/*
> > +			 * We are here to invalidate RDMA read requests
> > +			 * ourselves.  In normal scenario server should
> > +			 * send INV for all requested RDMA reads, but
> > +			 * we are here, thus two things could happen:
> > +			 *
> > +			 *    1.  this is failover, when errno != 0
> > +			 *        and can_wait == 1,
> > +			 *
> > +			 *    2.  something totally bad happened and
> > +			 *        server forgot to send INV, so we
> > +			 *        should do that ourselves.
> > +			 */
> 
> Please document in the protocol documentation when RDMA reads are used.
> 
> What does "server forgot to send INV" mean?
> 
> Additionally, if I remember correctly Jason considers it very important
> that invalidation happens from the submitting context because otherwise
> the RDMA retry mechanism can't work.

I think my point has usually been you can't use completions on the RQ
to deduce the state of the SQ

But if you are doing inv by posting it on the same SQ then things will
get ordered OK as the HW shouldn't progress the INV until any work
touching that rkey is also concluded.

Jason