RE: [PATCH v2 rdma-next] RDMA/mlx5: Enable Relaxed Ordering by default for kernel ULPs

From: David Laight <David.Laight@ACULAB.COM>
To: 'Christoph Hellwig' <hch@lst.de>, Leon Romanovsky <leon@kernel.org>
Cc: Doug Ledford <dledford@redhat.com>,
	Jason Gunthorpe <jgg@nvidia.com>,
	Avihai Horon <avihaih@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Bart Van Assche <bvanassche@acm.org>,
	"Tom Talpey" <tom@talpey.com>,
	Santosh Shilimkar <santosh.shilimkar@oracle.com>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Keith Busch <kbusch@kernel.org>, Honggang LI <honli@redhat.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>
Subject: RE: [PATCH v2 rdma-next] RDMA/mlx5: Enable Relaxed Ordering by default for kernel ULPs
Date: Wed, 9 Jun 2021 14:10:43 +0000	[thread overview]
Message-ID: <6b370a8fde1e406192d37c748b79ad01@AcuMS.aculab.com> (raw)
In-Reply-To: <20210609125241.GA1347@lst.de>

From: Christoph Hellwig
> Sent: 09 June 2021 13:53
> 
> On Wed, Jun 09, 2021 at 02:05:03PM +0300, Leon Romanovsky wrote:
> > From: Avihai Horon <avihaih@nvidia.com>
> >
> > Relaxed Ordering is a capability that can only benefit users that support
> > it. All kernel ULPs should support Relaxed Ordering, as they are designed
> > to read data only after observing the CQE and use the DMA API correctly.
> >
> > Hence, implicitly enable Relaxed Ordering by default for kernel ULPs.
> >
> > Signed-off-by: Avihai Horon <avihaih@nvidia.com>
> > Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> > ---
> > Changelog:
> > v2:
> >  * Dropped IB/core patch and set RO implicitly in mlx5 exactly like in
> >    eth side of mlx5 driver.
> 
> This looks great in terms of code changes.  But can we please also add a
> patch to document that PCIe relaxed ordering is fine for kernel ULP usage
> somewhere?

Several things need to happen to use relaxed ordering:
1) The pcie target has to be able to use RO for buffer writes
   and non-RO for control writes.
2) The Linux driver has to enable (1).
3) The generic Linux kernel has to 'not mask' RO on all the PCIe
   bridges and root.
4) The hardware memory system has to 'not break' when passes a RO write.

The comments about the DMA API are almost pointless.
They'd only be relevant if the driver has looking for the last
byte of a buffer to change - and then assuming the rest is valid.

This patch looks like a driver patch - so is changing (2) above.

I've seen another patch that (I think) is enabling (3).
Although some X86 cpu are known to be broken (aka 4).

And I still don't know what a ULP is.

I actually know a bit about TLP.
I found (and fixed) a bug in the Altera/Intel fpga logic
where it didn't like receiving two data TLP in response
to a single read TLP.
We've also (now) got logic in the fpga that traces all received
and sent TLP to an internal memory block.
So I can see what the cpus we have actually generate.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)