Re: [PATCH net 1/3] ibmvnic: rework to ensure SCRQ entry reads are properly ordered

[Lijun Pan <lijunp213@gmail.com>]

On Sat, Jan 23, 2021 at 11:10 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu, 21 Jan 2021 00:17:08 -0600 Lijun Pan wrote:
> > Move the dma_rmb() between pending_scrq() and ibmvnic_next_scrq()
> > into the end of pending_scrq(), and explain why.
> > Explain in detail why the dma_rmb() is placed at the end of
> > ibmvnic_next_scrq().
> >
> > Fixes: b71ec9522346 ("ibmvnic: Ensure that SCRQ entry reads are correctly ordered")
> > Signed-off-by: Lijun Pan <ljp@linux.ibm.com>
>
> I fail to see why this is a fix. You move the barrier from caller to
> callee but there are no new barriers here. Did I miss some?

I want to save the code since it is used more than twice.
Maybe I should send to net-next.

>
> > diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
> > index 9778c83150f1..8e043683610f 100644
> > --- a/drivers/net/ethernet/ibm/ibmvnic.c
> > +++ b/drivers/net/ethernet/ibm/ibmvnic.c
> > @@ -2511,12 +2511,6 @@ static int ibmvnic_poll(struct napi_struct *napi, int budget)
> >
> >               if (!pending_scrq(adapter, rx_scrq))
> >                       break;
> > -             /* The queue entry at the current index is peeked at above
> > -              * to determine that there is a valid descriptor awaiting
> > -              * processing. We want to be sure that the current slot
> > -              * holds a valid descriptor before reading its contents.
> > -              */
> > -             dma_rmb();
> >               next = ibmvnic_next_scrq(adapter, rx_scrq);
> >               rx_buff =
> >                   (struct ibmvnic_rx_buff *)be64_to_cpu(next->
> > @@ -3256,13 +3250,6 @@ static int ibmvnic_complete_tx(struct ibmvnic_adapter *adapter,
> >               int total_bytes = 0;
> >               int num_packets = 0;
> >
> > -             /* The queue entry at the current index is peeked at above
> > -              * to determine that there is a valid descriptor awaiting
> > -              * processing. We want to be sure that the current slot
> > -              * holds a valid descriptor before reading its contents.
> > -              */
> > -             dma_rmb();
> > -
> >               next = ibmvnic_next_scrq(adapter, scrq);
> >               for (i = 0; i < next->tx_comp.num_comps; i++) {
> >                       if (next->tx_comp.rcs[i])
> > @@ -3636,11 +3623,25 @@ static int pending_scrq(struct ibmvnic_adapter *adapter,
> >                       struct ibmvnic_sub_crq_queue *scrq)
> >  {
> >       union sub_crq *entry = &scrq->msgs[scrq->cur];
> > +     int rc;
> >
> >       if (entry->generic.first & IBMVNIC_CRQ_CMD_RSP)
> > -             return 1;
> > +             rc = 1;
> >       else
> > -             return 0;
> > +             rc = 0;
> > +
> > +     /* Ensure that the entire SCRQ descriptor scrq->msgs
> > +      * has been loaded before reading its contents.
>
> This comment is confusing. IMHO the comments you're removing are much
> clearer.

I did not quite get what fields in the data structure are supposed to
be guarded from the old comment. Then I asked around and figured it out.
So I updated it with the new comment, which tells specifically what
the barrier protects against.

>
> > +      * This barrier makes sure this function's entry, esp.
> > +      * entry->generic.first & IBMVNIC_CRQ_CMD_RSP
> > +      * 1. is loaded before ibmvnic_next_scrq()'s
> > +      * entry->generic.first & IBMVNIC_CRQ_CMD_RSP;
> > +      * 2. OR is loaded before ibmvnic_poll()'s
> > +      * disable_scrq_irq()'s scrq->hw_irq.
> > +      */
> > +     dma_rmb();
> > +
> > +     return rc;
> >  }
> >
> >  static union sub_crq *ibmvnic_next_scrq(struct ibmvnic_adapter *adapter,
>