[PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Adit Ranadive]

From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>

There is a chance of a race between arming the CQ and receiving
completions. By reporting CQ missed events any ULPs should poll
again to get the completions.

Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
---
v0 -> v1:
 - Check for invalid ring index.
---
 drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
index 69bda61..90aa326 100644
--- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
+++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
@@ -65,13 +65,28 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
 	struct pvrdma_dev *dev = to_vdev(ibcq->device);
 	struct pvrdma_cq *cq = to_vcq(ibcq);
 	u32 val = cq->cq_handle;
+	unsigned long flags;
+	int has_data = 0;
 
 	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
 		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
 
+	spin_lock_irqsave(&cq->cq_lock, flags);
+
 	pvrdma_write_uar_cq(dev, val);
 
-	return 0;
+	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
+		unsigned int head;
+
+		has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
+						    cq->ibcq.cqe, &head);
+		if (unlikely(has_data == PVRDMA_INVALID_IDX))
+			dev_err(&dev->pdev->dev, "CQ ring state invalid\n");
+	}
+
+	spin_unlock_irqrestore(&cq->cq_lock, flags);
+
+	return has_data;
 }
 
 /**
-- 
2.7.4

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Yuval Shaia]

On Thu, Aug 10, 2017 at 12:05:02PM -0700, Adit Ranadive wrote:
> From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> 
> There is a chance of a race between arming the CQ and receiving
> completions. By reporting CQ missed events any ULPs should poll
> again to get the completions.
> 
> Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> ---
> v0 -> v1:
>  - Check for invalid ring index.
> ---
>  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 17 ++++++++++++++++-
>  1 file changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> index 69bda61..90aa326 100644
> --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> @@ -65,13 +65,28 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
>  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
>  	struct pvrdma_cq *cq = to_vcq(ibcq);
>  	u32 val = cq->cq_handle;
> +	unsigned long flags;
> +	int has_data = 0;
>  
>  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
>  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
>  
> +	spin_lock_irqsave(&cq->cq_lock, flags);
> +
>  	pvrdma_write_uar_cq(dev, val);
>  
> -	return 0;
> +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> +		unsigned int head;
> +
> +		has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> +						    cq->ibcq.cqe, &head);
> +		if (unlikely(has_data == PVRDMA_INVALID_IDX))
> +			dev_err(&dev->pdev->dev, "CQ ring state invalid\n");

I see the point of checking the return value but per my understanding, and
correct me if i'm wrong, this rare case points to a corrupted ring which
can happen *only* in case of a bug so it is not "error" by nature.
If this is correct then i don't see the point of having this "question" on
every call to ib_notify_cq.

Do you agree to move this check to pvrdma_idx_ring_has_data and even make
the function use BUG_ON?

> +	}
> +
> +	spin_unlock_irqrestore(&cq->cq_lock, flags);
> +
> +	return has_data;
>  }
>  
>  /**
> -- 
> 2.7.4
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Adit Ranadive]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2704 bytes --]

On 8/10/17, 1:02 PM, "Yuval Shaia" <yuval.shaia@oracle.com> wrote:
> On Thu, Aug 10, 2017 at 12:05:02PM -0700, Adit Ranadive wrote:
> > From: Bryan Tan <bryantan@vmware.com>
> > 
> > There is a chance of a race between arming the CQ and receiving
> > completions. By reporting CQ missed events any ULPs should poll
> > again to get the completions.
> > 
> > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> > Acked-by: Aditya Sarwade <asarwade@vmware.com>
> > Signed-off-by: Bryan Tan <bryantan@vmware.com>
> > Signed-off-by: Adit Ranadive <aditr@vmware.com>
> > ---
> > v0 -> v1:
> >  - Check for invalid ring index.
> > ---
> >  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 17 ++++++++++++++++-
> >  1 file changed, 16 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > index 69bda61..90aa326 100644
> > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > @@ -65,13 +65,28 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
> >  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
> >  	struct pvrdma_cq *cq = to_vcq(ibcq);
> >  	u32 val = cq->cq_handle;
> > +	unsigned long flags;
> > +	int has_data = 0;
> >  
> >  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
> >  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
> >  
> > +	spin_lock_irqsave(&cq->cq_lock, flags);
> > +
> >  	pvrdma_write_uar_cq(dev, val);
> >  
> > -	return 0;
> > +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> > +		unsigned int head;
> > +
> > +		has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> > +						    cq->ibcq.cqe, &head);
> > +		if (unlikely(has_data == PVRDMA_INVALID_IDX))
> > +			dev_err(&dev->pdev->dev, "CQ ring state invalid\n");
> 
> I see the point of checking the return value but per my understanding, and
> correct me if i'm wrong, this rare case points to a corrupted ring which
> can happen *only* in case of a bug so it is not "error" by nature.
> If this is correct then i don't see the point of having this "question" on
> every call to ib_notify_cq.
> 
> Do you agree to move this check to pvrdma_idx_ring_has_data and even make
> the function use BUG_ON?

I'll concede that while it points to a corrupted ring (through a device bug, 
memory corruption) but we want to report it as a device error to maintain
consistency in our driver and give ULPs a chance to clean up. Also, the compiler
optimization should help here.

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±­ÙšŠ{ayº\x1dʇڙë,j\a­¢f£¢·hš‹»öì\x17/oSc¾™Ú³9˜uÀ¦æå‰È&jw¨®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿïêäz¹Þ–Šàþf£¢·hšˆ§~ˆmš

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Yuval Shaia]

On Thu, Aug 10, 2017 at 08:55:42PM +0000, Adit Ranadive wrote:
> On 8/10/17, 1:02 PM, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> > On Thu, Aug 10, 2017 at 12:05:02PM -0700, Adit Ranadive wrote:
> > > From: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > 
> > > There is a chance of a race between arming the CQ and receiving
> > > completions. By reporting CQ missed events any ULPs should poll
> > > again to get the completions.
> > > 
> > > Fixes: 29c8d9eba550 ("IB: Add vmw_pvrdma driver")
> > > Acked-by: Aditya Sarwade <asarwade-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > Signed-off-by: Bryan Tan <bryantan-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > Signed-off-by: Adit Ranadive <aditr-pghWNbHTmq7QT0dZR+AlfA@public.gmane.org>
> > > ---
> > > v0 -> v1:
> > >  - Check for invalid ring index.
> > > ---
> > >  drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c | 17 ++++++++++++++++-
> > >  1 file changed, 16 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > index 69bda61..90aa326 100644
> > > --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_cq.c
> > > @@ -65,13 +65,28 @@ int pvrdma_req_notify_cq(struct ib_cq *ibcq,
> > >  	struct pvrdma_dev *dev = to_vdev(ibcq->device);
> > >  	struct pvrdma_cq *cq = to_vcq(ibcq);
> > >  	u32 val = cq->cq_handle;
> > > +	unsigned long flags;
> > > +	int has_data = 0;
> > >  
> > >  	val |= (notify_flags & IB_CQ_SOLICITED_MASK) == IB_CQ_SOLICITED ?
> > >  		PVRDMA_UAR_CQ_ARM_SOL : PVRDMA_UAR_CQ_ARM;
> > >  
> > > +	spin_lock_irqsave(&cq->cq_lock, flags);
> > > +
> > >  	pvrdma_write_uar_cq(dev, val);
> > >  
> > > -	return 0;
> > > +	if (notify_flags & IB_CQ_REPORT_MISSED_EVENTS) {
> > > +		unsigned int head;
> > > +
> > > +		has_data = pvrdma_idx_ring_has_data(&cq->ring_state->rx,
> > > +						    cq->ibcq.cqe, &head);
> > > +		if (unlikely(has_data == PVRDMA_INVALID_IDX))
> > > +			dev_err(&dev->pdev->dev, "CQ ring state invalid\n");
> > 
> > I see the point of checking the return value but per my understanding, and
> > correct me if i'm wrong, this rare case points to a corrupted ring which
> > can happen *only* in case of a bug so it is not "error" by nature.
> > If this is correct then i don't see the point of having this "question" on
> > every call to ib_notify_cq.
> > 
> > Do you agree to move this check to pvrdma_idx_ring_has_data and even make
> > the function use BUG_ON?
> 
> I'll concede that while it points to a corrupted ring (through a device bug, 
> memory corruption) but we want to report it as a device error to maintain
> consistency in our driver and give ULPs a chance to clean up. Also, the compiler
> optimization should help here.

Great, i understand that.
So, at least can you consider moving this dev_err into
pvrdma_idx_ring_has_data so callers do not need to handle errors?

btw, same apply to pvrdma_idx_ring_has_space

> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Adit Ranadive]

On 8/10/17, 2:01 PM, "Yuval Shaia" <yuval.shaia@oracle.com> wrote:
> Great, i understand that.
> So, at least can you consider moving this dev_err into
> pvrdma_idx_ring_has_data so callers do not need to handle errors?

Thanks for the suggestion. It's something we will have to discuss internally,
though I'm not sure if BUG_ON is the right way to go. That just seems a really
big hammer to me.
As it stands the patch should be added since it does fix a potential race
condition.

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Yuval Shaia]

On Thu, Aug 10, 2017 at 09:26:06PM +0000, Adit Ranadive wrote:
> On 8/10/17, 2:01 PM, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> > Great, i understand that.
> > So, at least can you consider moving this dev_err into
> > pvrdma_idx_ring_has_data so callers do not need to handle errors?
> 
> Thanks for the suggestion. It's something we will have to discuss internally,
> though I'm not sure if BUG_ON is the right way to go. That just seems a really

No, i agree, BUG_ON is a mistake as ring corruption can be caused by HW, my
suggestion is just to place the check and error print inside the function
pvrdma_idx_ring_has_data so callers will not need to handle such errors.

Anyways, with or without taking the suggestion - fix lgtm.

Reviewed-by: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

> big hammer to me.
> As it stands the patch should be added since it does fix a potential race
> condition.
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Doug Ledford]

On Fri, 2017-08-11 at 00:33 +0300, Yuval Shaia wrote:
> On Thu, Aug 10, 2017 at 09:26:06PM +0000, Adit Ranadive wrote:
> > On 8/10/17, 2:01 PM, "Yuval Shaia" <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> > > Great, i understand that.
> > > So, at least can you consider moving this dev_err into
> > > pvrdma_idx_ring_has_data so callers do not need to handle errors?
> > 
> > Thanks for the suggestion. It's something we will have to discuss
> > internally,
> > though I'm not sure if BUG_ON is the right way to go. That just
> > seems a really
> 
> No, i agree, BUG_ON is a mistake as ring corruption can be caused by
> HW, 

BUG_ON is generally a mistake regardless.  Even if it was an issue in
the driver, another option would be to forcibly remove the ib_device
and shut it all down.  That would prevent a driver bug from taking the
machine down, and therefore is preferable to a BUG_ON.  If I see a
BUG_ON, I'm immediately going to be asking why that particular
condition is so bad that you think someone's Oracle server should die
entirely versus limping along with some disabled device or something
(because I know that's exactly what Linus will ask when he sees it).

> my
> suggestion is just to place the check and error print inside the
> function
> pvrdma_idx_ring_has_data so callers will not need to handle such
> errors.
> 
> Anyways, with or without taking the suggestion - fix lgtm.
> 
> Reviewed-by: Yuval Shaia <yuval.shaia-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>

Pulled in for -rc.  Thanks.

> > big hammer to me.
> > As it stands the patch should be added since it does fix a
> > potential race
> > condition.
> > 
-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
    GPG KeyID: B826A3330E572FDD
    Key fingerprint = AE6B 1BDA 122B 23B4 265B  1274 B826 A333 0E57 2FDD

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Pv-drivers] [PATCH v1 for-rc] RDMA/vmw_pvrdma: Report CQ missed events

[Steven Rostedt]

On Wed, 16 Aug 2017 10:59:52 -0400
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:

> (because I know that's exactly what Linus will ask when he sees it).

  https://lwn.net/Articles/13183/

Never ever use BUG_ON(), unless you are ready to really defend it. And
be prepared to be yelled at by Linus. Just use WARN_ON() and exit the
routine nicely.

-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html