Linux-RDMA Archive on lore.kernel.org
 help / color / Atom feed
From: Jason Gunthorpe <jgg@nvidia.com>
To: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Leon Romanovsky <leon@kernel.org>, Christoph Hellwig <hch@lst.de>,
	"Doug Ledford" <dledford@redhat.com>,
	<linux-rdma@vger.kernel.org>, <intel-gfx@lists.freedesktop.org>,
	Roland Scheidegger <sroland@vmware.com>,
	<dri-devel@lists.freedesktop.org>,
	David Airlie <airlied@linux.ie>,
	"VMware Graphics" <linux-graphics-maintainer@vmware.com>,
	Maor Gottlieb <maorg@nvidia.com>,
	Maor Gottlieb <maorg@mellanox.com>
Subject: Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages
Date: Fri, 25 Sep 2020 09:34:10 -0300
Message-ID: <20200925123410.GB9475@nvidia.com> (raw)
In-Reply-To: <c5956163-1769-ee40-e4ed-45532d8c4e19@linux.intel.com>

On Fri, Sep 25, 2020 at 01:29:49PM +0100, Tvrtko Ursulin wrote:
> 
> On 25/09/2020 12:58, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote:
> > > 
> > > On 25/09/2020 08:13, Leon Romanovsky wrote:
> > > > On Thu, Sep 24, 2020 at 09:21:20AM +0100, Tvrtko Ursulin wrote:
> > > > > 
> > > > > On 22/09/2020 09:39, Leon Romanovsky wrote:
> > > > > > From: Maor Gottlieb <maorg@mellanox.com>
> > > > > > 
> > > > > > Extend __sg_alloc_table_from_pages to support dynamic allocation of
> > > > > > SG table from pages. It should be used by drivers that can't supply
> > > > > > all the pages at one time.
> > > > > > 
> > > > > > This function returns the last populated SGE in the table. Users should
> > > > > > pass it as an argument to the function from the second call and forward.
> > > > > > As before, nents will be equal to the number of populated SGEs (chunks).
> > > > > 
> > > > > So it's appending and growing the "list", did I get that right? Sounds handy
> > > > > indeed. Some comments/questions below.
> > > > 
> > > > Yes, we (RDMA) use this function to chain contiguous pages.
> > > 
> > > I will eveluate if i915 could start using it. We have some loops which build
> > > page by page and coalesce.
> > 
> > Christoph H doesn't like it, but if there are enough cases we should
> > really have a pin_user_pages_to_sg() rather than open code this all
> > over the place.
> > 
> > With THP the chance of getting a coalescing SG is much higher, and
> > everything is more efficient with larger SGEs.
> 
> Right, I was actually referring to i915 sites where we build sg tables out
> of shmem and plain kernel pages. In those areas we have some open coded
> coalescing loops (see for instance our shmem_get_pages). Plus a local "trim"
> to discard the unused entries, since we allocate pessimistically not knowing
> how coalescing will pan out. This kind of core function which appends pages
> could replace some of that. Maybe it would be slightly less efficient but I
> will pencil in to at least evaluate it.
> 
> Otherwise I do agree that coalescing is a win and in the past I have
> measured savings in a few MiB range just for struct scatterlist storage.

I think the eventual dream is to have a pin_user_pages_bvec or similar
that is integrated into the GUP logic so avoids all the extra work,
just allocates pages of bvecs on the fly. No extra step through a
linear array of page *'s

Starting to structuring things to take advantage of that makes some
sense

Jason

  reply index

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-22  8:39 [PATCH rdma-next v3 0/2] Dynamicaly allocate SG table from the pages Leon Romanovsky
2020-09-22  8:39 ` [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages Leon Romanovsky
2020-09-23  5:42   ` Christoph Hellwig
     [not found]   ` <118a03ef-d160-e202-81cc-16c9c39359fc@linux.intel.com>
2020-09-25  7:13     ` [Intel-gfx] " Leon Romanovsky
2020-09-25 11:55       ` Jason Gunthorpe
2020-09-25 12:18         ` Maor Gottlieb
2020-09-25 12:33           ` Tvrtko Ursulin
2020-09-25 13:39             ` Maor Gottlieb
2020-09-25 13:54               ` Tvrtko Ursulin
     [not found]       ` <adff5752-582c-2065-89e2-924ef732911a@linux.intel.com>
2020-09-25 11:58         ` Jason Gunthorpe
2020-09-25 12:29           ` Tvrtko Ursulin
2020-09-25 12:34             ` Jason Gunthorpe [this message]
2020-09-25 12:13         ` Maor Gottlieb
2020-09-22  8:39 ` [PATCH rdma-next v3 2/2] RDMA/umem: Move to allocate " Leon Romanovsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200925123410.GB9475@nvidia.com \
    --to=jgg@nvidia.com \
    --cc=airlied@linux.ie \
    --cc=dledford@redhat.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=hch@lst.de \
    --cc=intel-gfx@lists.freedesktop.org \
    --cc=leon@kernel.org \
    --cc=linux-graphics-maintainer@vmware.com \
    --cc=linux-rdma@vger.kernel.org \
    --cc=maorg@mellanox.com \
    --cc=maorg@nvidia.com \
    --cc=sroland@vmware.com \
    --cc=tvrtko.ursulin@linux.intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-RDMA Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-rdma/0 linux-rdma/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-rdma linux-rdma/ https://lore.kernel.org/linux-rdma \
		linux-rdma@vger.kernel.org
	public-inbox-index linux-rdma

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-rdma


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git