Re: [PATCH v2] net: mlx5: Use iowriteXbe() to ring doorbell and remove reduntant wmb()

[Saeed Mahameed <saeedm@mellanox.com>]

On Sat, 2020-01-04 at 00:28 +0200, Liran Alon wrote:
> > On 3 Jan 2020, at 21:17, Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > 
> > On Fri, Jan 03, 2020 at 07:52:07PM +0200, Liran Alon wrote:
> > > diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
> > > index 40748fc1b11b..4631ad35da53 100644
> > > +++ b/include/linux/mlx5/cq.h
> > > @@ -162,13 +162,8 @@ static inline void mlx5_cq_arm(struct
> > > mlx5_core_cq *cq, u32 cmd,
> > > 
> > > 	*cq->arm_db = cpu_to_be32(sn << 28 | cmd | ci);
> > > 
> > > -	/* Make sure that the doorbell record in host memory is
> > > -	 * written before ringing the doorbell via PCI MMIO.
> > > -	 */
> > > -	wmb();
> > > -
> > > -	doorbell[0] = cpu_to_be32(sn << 28 | cmd | ci);
> > > -	doorbell[1] = cpu_to_be32(cq->cqn);
> > > +	doorbell[0] = sn << 28 | cmd | ci;
> > > +	doorbell[1] = cq->cqn;
> > 
> > This does actually have to change to a u64 otherwise it is not the
> > same.
> > 
> > On x86 LE, it was
> > db[0] = swab(a)
> > db[1] = swab(b)
> > __raw_writel(db)
> > 
> > Now it is
> > db[0] = a
> > db[1] = b
> > __raw_writel(swab(db))
> > 
> > Putting the swab around the u64 swaps the order of a/b in the TLP.
> > 
> > It might be tempting to swap db[0]/db[1] but IIRC this messed it up
> > on
> > BE.
> 
> Oops. You are right...
> 
> > The sanest, simplest solution is to use a u64 natively, as the
> > example
> > I gave did.
> 
> I agree.
> 
> > There is also the issue of casting a u32 to a u64 and possibly
> > triggering a unaligned kernel access, presumably this doesn't
> > happen
> > today only by some lucky chance..
> > 
> > > 	mlx5_write64(doorbell, uar_page + MLX5_CQ_DOORBELL);
> > > }
> > > diff --git a/include/linux/mlx5/doorbell.h
> > > b/include/linux/mlx5/doorbell.h
> > > index 5c267707e1df..9c1d35777323 100644
> > > +++ b/include/linux/mlx5/doorbell.h
> > > @@ -43,17 +43,15 @@
> > >  * Note that the write is not atomic on 32-bit systems! In
> > > contrast to 64-bit
> > >  * ones, it requires proper locking. mlx5_write64 doesn't do any
> > > locking, so use
> > >  * it at your own discretion, protected by some kind of lock on
> > > 32 bits.
> > > - *
> > > - * TODO: use write{q,l}_relaxed()
> > >  */
> > > 
> > > -static inline void mlx5_write64(__be32 val[2], void __iomem
> > > *dest)
> > > +static inline void mlx5_write64(u32 val[2], void __iomem *dest)
> > > {
> > 
> > So this should accept a straight u64, the goofy arrays have to go
> > away
> 
> I agree.
> 
> > > #if BITS_PER_LONG == 64
> > > -	__raw_writeq(*(u64 *)val, dest);
> > > +	iowrite64be(*(u64 *)val, dest);
> > > #else
> > > -	__raw_writel((__force u32) val[0], dest);
> > > -	__raw_writel((__force u32) val[1], dest + 4);
> > > +	iowrite32be(val[0], dest);
> > > +	iowrite32be(val[1], dest + 4);
> > 
> > With a u64 input this fallback is written as
> > 
> >  iowrite32be(val >> 32, dest)
> >  iowrite32be((u32)val, dest + 4)
> > 
> > Which matches the definition for how write64 must construct a TLP.
> > 
> > And arguably the first one should be _relaxed (but nobody cares
> > about
> > this code path)
> 
> I agree with everything. Will fix on v3.
> 

Hi Liran,

As agreed in a separate email thread, we will run performance
regression, before i can merge this patch, CCed Moshe and Tal for
perofmance feedback.

Thanks,
Saeed.