Re: dynamic-sg patch has broken rdma_rxe

From: Jason Gunthorpe <jgg@nvidia.com>
To: Maor Gottlieb <maorg@nvidia.com>,
	Tvrtko Ursulin <tvrtko.ursulin@intel.com>,
	Christoph Hellwig <hch@lst.de>
Cc: Gal Pressman <galpress@amazon.com>,
	Bob Pearson <rpearsonhpe@gmail.com>,
	Leon Romanovsky <leonro@nvidia.com>, <linux-rdma@vger.kernel.org>
Subject: Re: dynamic-sg patch has broken rdma_rxe
Date: Thu, 15 Oct 2020 21:31:27 -0300	[thread overview]
Message-ID: <20201016003127.GD6219@nvidia.com> (raw)
In-Reply-To: <8cf4796d-4dcb-ef5a-83ac-e11134eac99b@nvidia.com>

On Thu, Oct 15, 2020 at 03:21:34PM +0300, Maor Gottlieb wrote:
> 
> On 10/15/2020 2:23 PM, Gal Pressman wrote:
> > On 15/10/2020 10:44, Maor Gottlieb wrote:
> > > On 10/15/2020 1:51 AM, Jason Gunthorpe wrote:
> > > > On Tue, Oct 13, 2020 at 09:33:14AM -0500, Bob Pearson wrote:
> > > > > Jason,
> > > > > 
> > > > > Just pulled for-next and now hit the following warning.
> > > > > Register user space memory is not longer working.
> > > > > I am trying to debug this but if you have any idea where to look let me know.
> > > > The offset_in_page is wrong, but it is protecting some other logic..
> > > > 
> > > > Maor? Leon? Can you sort it out tomorrow?
> > > Leon and I investigated it. This check existed before my series to protect the
> > > alloc_table_from_pages logic. It's still relevant.
> > > This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the
> > > dma_device to ib_register_device"), and according to below link it was
> > > expected.  The safest approach is to set the max_segment_size back the 2GB in
> > > all drivers. What do you think?
> > > 
> > > https://lore.kernel.org/linux-rdma/20200923072111.GA31828@infradead.org/
> > FWIW, EFA is broken as well (same call trace) so it's not just software drivers.
> 
> This is true to all drivers that call to ib_umem_get and set UINT_MAX  as
> max_segment_size.
> Jason,  maybe instead of set UINT_MAX as max_segment_size, need to set
> SCATTERLIST_MAX_SEGMENT which does the required alignment.

SCATTERLIST_MAX_SEGMENT is almost never used, however there are lots
of places passing UINT_MAX or similar as the max_segsize for DMA.

The only place that does use it looks goofy to me:

	dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
					     SCATTERLIST_MAX_SEGMENT));

The seg_size should reflect the HW capability, not be mixed in with
knowledge about SGL internals. If the SGL can't build up to the HW
limit then it is fine to internally silently reduce it.

So I think we need to fix the scatterlist code, like below, and
just remove SCATTERLIST_MAX_SEGMENT completely.

It fixes things? Are you OK with this Christoph?

I need to get this fixed for the merge window PR I want to send on
Friday.

diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index e102fdfaa75be7..d158033834cdbc 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -435,7 +435,9 @@ struct scatterlist *__sg_alloc_table_from_pages(struct sg_table *sgt,
 	unsigned int added_nents = 0;
 	struct scatterlist *s = prv;
 
-	if (WARN_ON(!max_segment || offset_in_page(max_segment)))
+	/* Avoid overflow when computing sg_len + PAGE_SIZE */
+	max_segment = max_segment & PAGE_MASK;
+	if (WARN_ON(max_segment < PAGE_SIZE))
 		return ERR_PTR(-EINVAL);
 
 	if (IS_ENABLED(CONFIG_ARCH_NO_SG_CHAIN) && prv)
