From: Jason Gunthorpe <jgg@nvidia.com>
To: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Leon Romanovsky <leon@kernel.org>, Christoph Hellwig <hch@lst.de>,
"Doug Ledford" <dledford@redhat.com>,
<linux-rdma@vger.kernel.org>, <intel-gfx@lists.freedesktop.org>,
Roland Scheidegger <sroland@vmware.com>,
<dri-devel@lists.freedesktop.org>,
David Airlie <airlied@linux.ie>,
"VMware Graphics" <linux-graphics-maintainer@vmware.com>,
Maor Gottlieb <maorg@nvidia.com>,
Maor Gottlieb <maorg@mellanox.com>
Subject: Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages
Date: Fri, 25 Sep 2020 09:34:10 -0300 [thread overview]
Message-ID: <20200925123410.GB9475@nvidia.com> (raw)
In-Reply-To: <c5956163-1769-ee40-e4ed-45532d8c4e19@linux.intel.com>
On Fri, Sep 25, 2020 at 01:29:49PM +0100, Tvrtko Ursulin wrote:
>
> On 25/09/2020 12:58, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote:
> > >
> > > On 25/09/2020 08:13, Leon Romanovsky wrote:
> > > > On Thu, Sep 24, 2020 at 09:21:20AM +0100, Tvrtko Ursulin wrote:
> > > > >
> > > > > On 22/09/2020 09:39, Leon Romanovsky wrote:
> > > > > > From: Maor Gottlieb <maorg@mellanox.com>
> > > > > >
> > > > > > Extend __sg_alloc_table_from_pages to support dynamic allocation of
> > > > > > SG table from pages. It should be used by drivers that can't supply
> > > > > > all the pages at one time.
> > > > > >
> > > > > > This function returns the last populated SGE in the table. Users should
> > > > > > pass it as an argument to the function from the second call and forward.
> > > > > > As before, nents will be equal to the number of populated SGEs (chunks).
> > > > >
> > > > > So it's appending and growing the "list", did I get that right? Sounds handy
> > > > > indeed. Some comments/questions below.
> > > >
> > > > Yes, we (RDMA) use this function to chain contiguous pages.
> > >
> > > I will eveluate if i915 could start using it. We have some loops which build
> > > page by page and coalesce.
> >
> > Christoph H doesn't like it, but if there are enough cases we should
> > really have a pin_user_pages_to_sg() rather than open code this all
> > over the place.
> >
> > With THP the chance of getting a coalescing SG is much higher, and
> > everything is more efficient with larger SGEs.
>
> Right, I was actually referring to i915 sites where we build sg tables out
> of shmem and plain kernel pages. In those areas we have some open coded
> coalescing loops (see for instance our shmem_get_pages). Plus a local "trim"
> to discard the unused entries, since we allocate pessimistically not knowing
> how coalescing will pan out. This kind of core function which appends pages
> could replace some of that. Maybe it would be slightly less efficient but I
> will pencil in to at least evaluate it.
>
> Otherwise I do agree that coalescing is a win and in the past I have
> measured savings in a few MiB range just for struct scatterlist storage.
I think the eventual dream is to have a pin_user_pages_bvec or similar
that is integrated into the GUP logic so avoids all the extra work,
just allocates pages of bvecs on the fly. No extra step through a
linear array of page *'s
Starting to structuring things to take advantage of that makes some
sense
Jason
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@nvidia.com>
To: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Leon Romanovsky <leon@kernel.org>,
linux-rdma@vger.kernel.org, intel-gfx@lists.freedesktop.org,
Roland Scheidegger <sroland@vmware.com>,
dri-devel@lists.freedesktop.org,
Maor Gottlieb <maorg@mellanox.com>,
David Airlie <airlied@linux.ie>,
Doug Ledford <dledford@redhat.com>,
VMware Graphics <linux-graphics-maintainer@vmware.com>,
Maor Gottlieb <maorg@nvidia.com>, Christoph Hellwig <hch@lst.de>
Subject: Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages
Date: Fri, 25 Sep 2020 09:34:10 -0300 [thread overview]
Message-ID: <20200925123410.GB9475@nvidia.com> (raw)
In-Reply-To: <c5956163-1769-ee40-e4ed-45532d8c4e19@linux.intel.com>
On Fri, Sep 25, 2020 at 01:29:49PM +0100, Tvrtko Ursulin wrote:
>
> On 25/09/2020 12:58, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote:
> > >
> > > On 25/09/2020 08:13, Leon Romanovsky wrote:
> > > > On Thu, Sep 24, 2020 at 09:21:20AM +0100, Tvrtko Ursulin wrote:
> > > > >
> > > > > On 22/09/2020 09:39, Leon Romanovsky wrote:
> > > > > > From: Maor Gottlieb <maorg@mellanox.com>
> > > > > >
> > > > > > Extend __sg_alloc_table_from_pages to support dynamic allocation of
> > > > > > SG table from pages. It should be used by drivers that can't supply
> > > > > > all the pages at one time.
> > > > > >
> > > > > > This function returns the last populated SGE in the table. Users should
> > > > > > pass it as an argument to the function from the second call and forward.
> > > > > > As before, nents will be equal to the number of populated SGEs (chunks).
> > > > >
> > > > > So it's appending and growing the "list", did I get that right? Sounds handy
> > > > > indeed. Some comments/questions below.
> > > >
> > > > Yes, we (RDMA) use this function to chain contiguous pages.
> > >
> > > I will eveluate if i915 could start using it. We have some loops which build
> > > page by page and coalesce.
> >
> > Christoph H doesn't like it, but if there are enough cases we should
> > really have a pin_user_pages_to_sg() rather than open code this all
> > over the place.
> >
> > With THP the chance of getting a coalescing SG is much higher, and
> > everything is more efficient with larger SGEs.
>
> Right, I was actually referring to i915 sites where we build sg tables out
> of shmem and plain kernel pages. In those areas we have some open coded
> coalescing loops (see for instance our shmem_get_pages). Plus a local "trim"
> to discard the unused entries, since we allocate pessimistically not knowing
> how coalescing will pan out. This kind of core function which appends pages
> could replace some of that. Maybe it would be slightly less efficient but I
> will pencil in to at least evaluate it.
>
> Otherwise I do agree that coalescing is a win and in the past I have
> measured savings in a few MiB range just for struct scatterlist storage.
I think the eventual dream is to have a pin_user_pages_bvec or similar
that is integrated into the GUP logic so avoids all the extra work,
just allocates pages of bvecs on the fly. No extra step through a
linear array of page *'s
Starting to structuring things to take advantage of that makes some
sense
Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
WARNING: multiple messages have this Message-ID (diff)
From: Jason Gunthorpe <jgg@nvidia.com>
To: Tvrtko Ursulin <tvrtko.ursulin@linux.intel.com>
Cc: Leon Romanovsky <leon@kernel.org>,
linux-rdma@vger.kernel.org, intel-gfx@lists.freedesktop.org,
Roland Scheidegger <sroland@vmware.com>,
dri-devel@lists.freedesktop.org,
Maor Gottlieb <maorg@mellanox.com>,
David Airlie <airlied@linux.ie>,
Doug Ledford <dledford@redhat.com>,
VMware Graphics <linux-graphics-maintainer@vmware.com>,
Maor Gottlieb <maorg@nvidia.com>, Christoph Hellwig <hch@lst.de>
Subject: Re: [Intel-gfx] [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages
Date: Fri, 25 Sep 2020 09:34:10 -0300 [thread overview]
Message-ID: <20200925123410.GB9475@nvidia.com> (raw)
In-Reply-To: <c5956163-1769-ee40-e4ed-45532d8c4e19@linux.intel.com>
On Fri, Sep 25, 2020 at 01:29:49PM +0100, Tvrtko Ursulin wrote:
>
> On 25/09/2020 12:58, Jason Gunthorpe wrote:
> > On Fri, Sep 25, 2020 at 12:41:29PM +0100, Tvrtko Ursulin wrote:
> > >
> > > On 25/09/2020 08:13, Leon Romanovsky wrote:
> > > > On Thu, Sep 24, 2020 at 09:21:20AM +0100, Tvrtko Ursulin wrote:
> > > > >
> > > > > On 22/09/2020 09:39, Leon Romanovsky wrote:
> > > > > > From: Maor Gottlieb <maorg@mellanox.com>
> > > > > >
> > > > > > Extend __sg_alloc_table_from_pages to support dynamic allocation of
> > > > > > SG table from pages. It should be used by drivers that can't supply
> > > > > > all the pages at one time.
> > > > > >
> > > > > > This function returns the last populated SGE in the table. Users should
> > > > > > pass it as an argument to the function from the second call and forward.
> > > > > > As before, nents will be equal to the number of populated SGEs (chunks).
> > > > >
> > > > > So it's appending and growing the "list", did I get that right? Sounds handy
> > > > > indeed. Some comments/questions below.
> > > >
> > > > Yes, we (RDMA) use this function to chain contiguous pages.
> > >
> > > I will eveluate if i915 could start using it. We have some loops which build
> > > page by page and coalesce.
> >
> > Christoph H doesn't like it, but if there are enough cases we should
> > really have a pin_user_pages_to_sg() rather than open code this all
> > over the place.
> >
> > With THP the chance of getting a coalescing SG is much higher, and
> > everything is more efficient with larger SGEs.
>
> Right, I was actually referring to i915 sites where we build sg tables out
> of shmem and plain kernel pages. In those areas we have some open coded
> coalescing loops (see for instance our shmem_get_pages). Plus a local "trim"
> to discard the unused entries, since we allocate pessimistically not knowing
> how coalescing will pan out. This kind of core function which appends pages
> could replace some of that. Maybe it would be slightly less efficient but I
> will pencil in to at least evaluate it.
>
> Otherwise I do agree that coalescing is a win and in the past I have
> measured savings in a few MiB range just for struct scatterlist storage.
I think the eventual dream is to have a pin_user_pages_bvec or similar
that is integrated into the GUP logic so avoids all the extra work,
just allocates pages of bvecs on the fly. No extra step through a
linear array of page *'s
Starting to structuring things to take advantage of that makes some
sense
Jason
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
next prev parent reply other threads:[~2020-09-25 12:34 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-09-22 8:39 [PATCH rdma-next v3 0/2] Dynamicaly allocate SG table from the pages Leon Romanovsky
2020-09-22 8:39 ` [Intel-gfx] " Leon Romanovsky
2020-09-22 8:39 ` Leon Romanovsky
2020-09-22 8:39 ` [PATCH rdma-next v3 1/2] lib/scatterlist: Add support in dynamic allocation of SG table from pages Leon Romanovsky
2020-09-22 8:39 ` [Intel-gfx] " Leon Romanovsky
2020-09-22 8:39 ` Leon Romanovsky
2020-09-23 5:42 ` Christoph Hellwig
2020-09-23 5:42 ` [Intel-gfx] " Christoph Hellwig
2020-09-24 8:21 ` Tvrtko Ursulin
2020-09-24 8:21 ` Tvrtko Ursulin
2020-09-25 7:13 ` Leon Romanovsky
2020-09-25 7:13 ` Leon Romanovsky
2020-09-25 7:13 ` Leon Romanovsky
2020-09-25 11:41 ` Tvrtko Ursulin
2020-09-25 11:41 ` Tvrtko Ursulin
2020-09-25 11:58 ` Jason Gunthorpe
2020-09-25 11:58 ` Jason Gunthorpe
2020-09-25 11:58 ` Jason Gunthorpe
2020-09-25 12:29 ` Tvrtko Ursulin
2020-09-25 12:29 ` Tvrtko Ursulin
2020-09-25 12:29 ` Tvrtko Ursulin
2020-09-25 12:34 ` Jason Gunthorpe [this message]
2020-09-25 12:34 ` Jason Gunthorpe
2020-09-25 12:34 ` Jason Gunthorpe
2020-09-25 12:13 ` Maor Gottlieb
2020-09-25 12:13 ` Maor Gottlieb
2020-09-25 12:13 ` Maor Gottlieb
2020-09-25 11:55 ` Jason Gunthorpe
2020-09-25 11:55 ` Jason Gunthorpe
2020-09-25 11:55 ` Jason Gunthorpe
2020-09-25 12:18 ` Maor Gottlieb
2020-09-25 12:18 ` Maor Gottlieb
2020-09-25 12:18 ` Maor Gottlieb
2020-09-25 12:33 ` Tvrtko Ursulin
2020-09-25 12:33 ` Tvrtko Ursulin
2020-09-25 12:33 ` Tvrtko Ursulin
2020-09-25 13:39 ` Maor Gottlieb
2020-09-25 13:39 ` Maor Gottlieb
2020-09-25 13:39 ` Maor Gottlieb
2020-09-25 13:54 ` Tvrtko Ursulin
2020-09-25 13:54 ` Tvrtko Ursulin
2020-09-25 13:54 ` Tvrtko Ursulin
2020-09-22 8:39 ` [PATCH rdma-next v3 2/2] RDMA/umem: Move to allocate " Leon Romanovsky
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200925123410.GB9475@nvidia.com \
--to=jgg@nvidia.com \
--cc=airlied@linux.ie \
--cc=dledford@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=hch@lst.de \
--cc=intel-gfx@lists.freedesktop.org \
--cc=leon@kernel.org \
--cc=linux-graphics-maintainer@vmware.com \
--cc=linux-rdma@vger.kernel.org \
--cc=maorg@mellanox.com \
--cc=maorg@nvidia.com \
--cc=sroland@vmware.com \
--cc=tvrtko.ursulin@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.