Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Christoph Hellwig <hch@lst.de>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Maor Gottlieb <maorg@nvidia.com>,
	Tvrtko Ursulin <tvrtko.ursulin@intel.com>,
	Christoph Hellwig <hch@lst.de>,
	Gal Pressman <galpress@amazon.com>,
	Bob Pearson <rpearsonhpe@gmail.com>,
	Leon Romanovsky <leonro@nvidia.com>,
	linux-rdma@vger.kernel.org
Subject: Re: dynamic-sg patch has broken rdma_rxe
Date: Fri, 16 Oct 2020 09:13:47 +0200
Message-ID: <20201016071347.GA8001@lst.de> (raw)
In-Reply-To: <20201016003127.GD6219@nvidia.com>

On Thu, Oct 15, 2020 at 09:31:27PM -0300, Jason Gunthorpe wrote:
> On Thu, Oct 15, 2020 at 03:21:34PM +0300, Maor Gottlieb wrote:
> > 
> > On 10/15/2020 2:23 PM, Gal Pressman wrote:
> > > On 15/10/2020 10:44, Maor Gottlieb wrote:
> > > > On 10/15/2020 1:51 AM, Jason Gunthorpe wrote:
> > > > > On Tue, Oct 13, 2020 at 09:33:14AM -0500, Bob Pearson wrote:
> > > > > > Jason,
> > > > > > 
> > > > > > Just pulled for-next and now hit the following warning.
> > > > > > Register user space memory is not longer working.
> > > > > > I am trying to debug this but if you have any idea where to look let me know.
> > > > > The offset_in_page is wrong, but it is protecting some other logic..
> > > > > 
> > > > > Maor? Leon? Can you sort it out tomorrow?
> > > > Leon and I investigated it. This check existed before my series to protect the
> > > > alloc_table_from_pages logic. It's still relevant.
> > > > This patch that broke it:  54816d3e69d1 ("RDMA: Explicitly pass in the
> > > > dma_device to ib_register_device"), and according to below link it was
> > > > expected.  The safest approach is to set the max_segment_size back the 2GB in
> > > > all drivers. What do you think?
> > > > 
> > > > https://lore.kernel.org/linux-rdma/20200923072111.GA31828@infradead.org/
> > > FWIW, EFA is broken as well (same call trace) so it's not just software drivers.
> > 
> > This is true to all drivers that call to ib_umem_get and set UINT_MAX  as
> > max_segment_size.
> > Jason,  maybe instead of set UINT_MAX as max_segment_size, need to set
> > SCATTERLIST_MAX_SEGMENT which does the required alignment.
> 
> SCATTERLIST_MAX_SEGMENT is almost never used, however there are lots
> of places passing UINT_MAX or similar as the max_segsize for DMA.
> 
> The only place that does use it looks goofy to me:
> 
> 	dma_set_max_seg_size(dev->dev, min_t(unsigned int, U32_MAX & PAGE_MASK,
> 					     SCATTERLIST_MAX_SEGMENT));
> 
> The seg_size should reflect the HW capability, not be mixed in with
> knowledge about SGL internals. If the SGL can't build up to the HW
> limit then it is fine to internally silently reduce it.
> 
> So I think we need to fix the scatterlist code, like below, and
> just remove SCATTERLIST_MAX_SEGMENT completely.
> 
> It fixes things? Are you OK with this Christoph?
> 
> I need to get this fixed for the merge window PR I want to send on
> Friday.

This looks ok.  Not critical, but I think we should follow up and kill
off the confusing SCATTERLIST_MAX_SEGMENT as well soon

  reply index

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-13 14:33 Bob Pearson
2020-10-13 16:34 ` Bob Pearson
2020-10-14 22:51 ` Jason Gunthorpe
2020-10-15  7:44   ` Maor Gottlieb
2020-10-15 11:23     ` Gal Pressman
2020-10-15 12:21       ` Maor Gottlieb
2020-10-16  0:31         ` Jason Gunthorpe
2020-10-16  7:13           ` Christoph Hellwig [this message]
     [not found]           ` <796ca31aed8f469c957cb850385b9d09@intel.com>
2020-10-16 11:58             ` Jason Gunthorpe
2020-10-19  9:50               ` Tvrtko Ursulin
2020-10-19 12:12                 ` Jason Gunthorpe
2020-10-19 12:29                   ` Tvrtko Ursulin
2020-10-19 12:48                     ` Jason Gunthorpe
2020-10-20 11:37                       ` Tvrtko Ursulin
2020-10-20 11:47                         ` Jason Gunthorpe
2020-10-20 12:31                           ` Tvrtko Ursulin
2020-10-20 12:56                             ` Jason Gunthorpe
2020-10-20 13:09                               ` Tvrtko Ursulin
2020-10-20 13:32                                 ` Jason Gunthorpe
2020-10-15 15:35     ` Bob Pearson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201016071347.GA8001@lst.de \
    --to=hch@lst.de \
    --cc=galpress@amazon.com \
    --cc=jgg@nvidia.com \
    --cc=leonro@nvidia.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=maorg@nvidia.com \
    --cc=rpearsonhpe@gmail.com \
    --cc=tvrtko.ursulin@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git