Re: [PATCH rdma-next v1 0/2] Enable relaxed ordering for ULPs

From: Jason Gunthorpe <jgg@nvidia.com>
To: David Laight <David.Laight@ACULAB.COM>
Cc: Leon Romanovsky <leon@kernel.org>,
	Doug Ledford <dledford@redhat.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	Avihai Horon <avihaih@nvidia.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	Christoph Hellwig <hch@lst.de>,
	Bart Van Assche <bvanassche@acm.org>, Tom Talpey <tom@talpey.com>,
	Santosh Shilimkar <santosh.shilimkar@oracle.com>,
	Chuck Lever III <chuck.lever@oracle.com>,
	Keith Busch <kbusch@kernel.org>, Honggang LI <honli@redhat.com>,
	Max Gurtovoy <mgurtovoy@nvidia.com>
Subject: Re: [PATCH rdma-next v1 0/2] Enable relaxed ordering for ULPs
Date: Mon, 31 May 2021 19:44:20 -0300	[thread overview]
Message-ID: <20210531224420.GG1002214@nvidia.com> (raw)
In-Reply-To: <2fe802bdebbd44619447c83ed7e30a74@AcuMS.aculab.com>

On Mon, May 31, 2021 at 09:45:47PM +0000, David Laight wrote:
> From: Jason Gunthorpe
> > Sent: 31 May 2021 19:14
> > 
> > On Thu, May 27, 2021 at 08:11:14AM +0000, David Laight wrote:
> > > > There was such a big discussion on the last version I wondered why
> > > > this was so quiet. I guess because the cc list isn't very big..
> > > >
> > > > Adding the people from the original thread, here is the patches:
> > > >
> > > > https://lore.kernel.org/linux-rdma/cover.1621505111.git.leonro@nvidia.com/
> > > >
> > > > I think this is the general approach that was asked for, to special case
> > > > uverbs and turn it on in kernel universally
> > >
> > > I'm still not sure which PCIe transactions you are enabling relaxed
> > > ordering for.  Nothing has ever said that in layman's terms.
> > >
> > > IIRC PCIe targets (like ethernet chips) can use relaxed ordered
> > > writes for frame contents but must use strongly ordered writes
> > > for the corresponding ring (control structure) updates.
> > 
> > Right, it is exactly like this, just not expressed in ethernet
> > specific terms.
> > 
> > Data transfer TLPs are relaxed ordered and control structure TLPs are
> > normal ordered.
> 
> So exactly what is this patch doing?

It allows RDMA devices to set the relaxed ordering bit in their PCI
TLPs following the rules outlined above, but specified in detail, in
the InfiniBand Architecture spec.

Today the Linux model prevents devices from using the PCI relaxed
ordering bit in their TLPs at all.

Jason