From: Boris Brezillon <boris.brezillon@collabora.com>
To: Steven Price <steven.price@arm.com>
Cc: "Dmitry Osipenko" <dmitry.osipenko@collabora.com>,
"David Airlie" <airlied@gmail.com>,
"Gerd Hoffmann" <kraxel@redhat.com>,
"Gurchetan Singh" <gurchetansingh@chromium.org>,
"Chia-I Wu" <olvaffe@gmail.com>,
"Daniel Vetter" <daniel@ffwll.ch>,
"Maarten Lankhorst" <maarten.lankhorst@linux.intel.com>,
"Maxime Ripard" <mripard@kernel.org>,
"Thomas Zimmermann" <tzimmermann@suse.de>,
"Christian König" <christian.koenig@amd.com>,
"Qiang Yu" <yuq825@gmail.com>, "Emma Anholt" <emma@anholt.net>,
"Melissa Wen" <mwen@igalia.com>,
dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
kernel@collabora.com, virtualization@lists.linux-foundation.org
Subject: Re: [PATCH v19 18/30] drm/panfrost: Explicitly get and put drm-shmem pages
Date: Fri, 26 Jan 2024 10:39:24 +0100 [thread overview]
Message-ID: <20240126103924.0b911a4f@collabora.com> (raw)
In-Reply-To: <7144dd9b-62d1-4968-9b94-0313e2475f7e@arm.com>
On Thu, 25 Jan 2024 16:47:24 +0000
Steven Price <steven.price@arm.com> wrote:
> On 05/01/2024 18:46, Dmitry Osipenko wrote:
> > To simplify the drm-shmem refcnt handling, we're moving away from
> > the implicit get_pages() that is used by get_pages_sgt(). From now on
> > drivers will have to pin pages while they use sgt. Panfrost's shrinker
> > doesn't support swapping out BOs, hence pages are pinned and sgt is valid
> > as long as pages' use-count > 0.
> >
> > In Panfrost, panfrost_gem_mapping, which is the object representing a
> > GPU mapping of a BO, owns a pages ref. This guarantees that any BO being
> > mapped GPU side has its pages retained till the mapping is destroyed.
> >
> > Since pages are no longer guaranteed to stay pinned for the BO lifetime,
> > and MADVISE(DONT_NEED) flagging remains after the GEM handle has been
> > destroyed, we need to add an extra 'is_purgeable' check in
> > panfrost_gem_purge(), to make sure we're not trying to purge a BO that
> > already had its pages released.
> >
> > Signed-off-by: Dmitry Osipenko <dmitry.osipenko@collabora.com>
>
> Reviewed-by: Steven Price <steven.price@arm.com>
>
> Although I don't like the condition in panfrost_gem_mapping_release()
> for drm_gem_shmem_put_pages() and assigning NULL to bo->sgts - it feels
> very fragile. See below.
>
> > ---
> > drivers/gpu/drm/panfrost/panfrost_gem.c | 63 ++++++++++++++-----
> > .../gpu/drm/panfrost/panfrost_gem_shrinker.c | 6 ++
> > 2 files changed, 52 insertions(+), 17 deletions(-)
> >
> > diff --git a/drivers/gpu/drm/panfrost/panfrost_gem.c b/drivers/gpu/drm/panfrost/panfrost_gem.c
> > index f268bd5c2884..7edfc12f7c1f 100644
> > --- a/drivers/gpu/drm/panfrost/panfrost_gem.c
> > +++ b/drivers/gpu/drm/panfrost/panfrost_gem.c
> > @@ -35,20 +35,6 @@ static void panfrost_gem_free_object(struct drm_gem_object *obj)
> > */
> > WARN_ON_ONCE(!list_empty(&bo->mappings.list));
> >
> > - if (bo->sgts) {
> > - int i;
> > - int n_sgt = bo->base.base.size / SZ_2M;
> > -
> > - for (i = 0; i < n_sgt; i++) {
> > - if (bo->sgts[i].sgl) {
> > - dma_unmap_sgtable(pfdev->dev, &bo->sgts[i],
> > - DMA_BIDIRECTIONAL, 0);
> > - sg_free_table(&bo->sgts[i]);
> > - }
> > - }
> > - kvfree(bo->sgts);
> > - }
> > -
> > drm_gem_shmem_free(&bo->base);
> > }
> >
> > @@ -85,11 +71,40 @@ panfrost_gem_teardown_mapping(struct panfrost_gem_mapping *mapping)
> >
> > static void panfrost_gem_mapping_release(struct kref *kref)
> > {
> > - struct panfrost_gem_mapping *mapping;
> > -
> > - mapping = container_of(kref, struct panfrost_gem_mapping, refcount);
> > + struct panfrost_gem_mapping *mapping =
> > + container_of(kref, struct panfrost_gem_mapping, refcount);
> > + struct panfrost_gem_object *bo = mapping->obj;
> > + struct panfrost_device *pfdev = bo->base.base.dev->dev_private;
> >
> > panfrost_gem_teardown_mapping(mapping);
> > +
> > + /* On heap BOs, release the sgts created in the fault handler path. */
> > + if (bo->sgts) {
> > + int i, n_sgt = bo->base.base.size / SZ_2M;
> > +
> > + for (i = 0; i < n_sgt; i++) {
> > + if (bo->sgts[i].sgl) {
> > + dma_unmap_sgtable(pfdev->dev, &bo->sgts[i],
> > + DMA_BIDIRECTIONAL, 0);
> > + sg_free_table(&bo->sgts[i]);
> > + }
> > + }
> > + kvfree(bo->sgts);
> > + }
> > +
> > + /* Pages ref is owned by the panfrost_gem_mapping object. We must
> > + * release our pages ref (if any), before releasing the object
> > + * ref.
> > + * Non-heap BOs acquired the pages at panfrost_gem_mapping creation
> > + * time, and heap BOs may have acquired pages if the fault handler
> > + * was called, in which case bo->sgts should be non-NULL.
> > + */
> > + if (!bo->base.base.import_attach && (!bo->is_heap || bo->sgts) &&
> > + bo->base.madv >= 0) {
> > + drm_gem_shmem_put_pages(&bo->base);
> > + bo->sgts = NULL;
>
> The assignment of NULL here really ought to be unconditional - it isn't
> a valid pointer because of the kvfree() above.
Fair enough. How about we drop the '|| bo->sgts' and add an
drm_gem_shmem_put_pages() to the above if (bo->sgts) block, where we'll
also assign bo->sgts to NULL?
>
> I also feel that the big condition above suggests there's a need for a
> better state machine to keep track of what's going on.
I'm planning to extend drm_gem_shmem to support the alloc-on-fault use
case that all Mali GPUs seem to rely on (lima, panfrost and soon
panthor would use those helpers). The idea is to:
- make the allocation non-blocking, so we can kill the blocking
allocation in the dma signalling path (basically what intel does)
- allow dynamic extension of the pages array using an xarray instead of
a plain array
Hopefully this makes the state tracking a lot easier, and we can also
get rid of the hack we have in panfrost/lima where we manipulate
drm_gem_shmem_object refcounts directly.
>
> But having said that I do think this series as a whole is an
> improvement, it's nice to get the shrinker code generic. And sadly I
> don't have an immediate idea for cleaning this up, hence my R-b.
>
> Steve
>
> > + }
> > +
> > drm_gem_object_put(&mapping->obj->base.base);
> > panfrost_mmu_ctx_put(mapping->mmu);
> > kfree(mapping);
> > @@ -125,6 +140,20 @@ int panfrost_gem_open(struct drm_gem_object *obj, struct drm_file *file_priv)
> > if (!mapping)
> > return -ENOMEM;
> >
> > + if (!bo->is_heap && !bo->base.base.import_attach) {
> > + /* Pages ref is owned by the panfrost_gem_mapping object.
> > + * For non-heap BOs, we request pages at mapping creation
> > + * time, such that the panfrost_mmu_map() call, further down in
> > + * this function, is guaranteed to have pages_use_count > 0
> > + * when drm_gem_shmem_get_pages_sgt() is called.
> > + */
> > + ret = drm_gem_shmem_get_pages(&bo->base);
> > + if (ret) {
> > + kfree(mapping);
> > + return ret;
> > + }
> > + }
> > +
> > INIT_LIST_HEAD(&mapping->node);
> > kref_init(&mapping->refcount);
> > drm_gem_object_get(obj);
> > diff --git a/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c b/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
> > index 02b60ea1433a..d4fb0854cf2f 100644
> > --- a/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
> > +++ b/drivers/gpu/drm/panfrost/panfrost_gem_shrinker.c
> > @@ -50,6 +50,12 @@ static bool panfrost_gem_purge(struct drm_gem_object *obj)
> > if (!dma_resv_trylock(shmem->base.resv))
> > goto unlock_mappings;
> >
> > + /* BO might have become unpurgeable if the last pages_use_count ref
> > + * was dropped, but the BO hasn't been destroyed yet.
> > + */
> > + if (!drm_gem_shmem_is_purgeable(shmem))
> > + goto unlock_mappings;
> > +
> > panfrost_gem_teardown_mappings_locked(bo);
> > drm_gem_shmem_purge_locked(&bo->base);
> > ret = true;
>
next prev parent reply other threads:[~2024-01-26 9:39 UTC|newest]
Thread overview: 70+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-01-05 18:45 [PATCH v19 00/30] Add generic memory shrinker to VirtIO-GPU and Panfrost DRM drivers Dmitry Osipenko
2024-01-05 18:45 ` [PATCH v19 01/30] drm/gem: Change locked/unlocked postfix of drm_gem_v/unmap() function names Dmitry Osipenko
2024-01-05 18:45 ` [PATCH v19 02/30] drm/gem: Add _locked postfix to functions that have unlocked counterpart Dmitry Osipenko
2024-01-05 18:45 ` [PATCH v19 03/30] drm/gem: Document locking rule of vmap and evict callbacks Dmitry Osipenko
2024-01-25 7:55 ` Boris Brezillon
2024-01-05 18:45 ` [PATCH v19 04/30] drm/shmem-helper: Make all exported symbols GPL Dmitry Osipenko
2024-01-05 18:45 ` [PATCH v19 05/30] drm/shmem-helper: Refactor locked/unlocked functions Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 06/30] drm/shmem-helper: Remove obsoleted is_iomem test Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 07/30] drm/shmem-helper: Add and use pages_pin_count Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 08/30] drm/shmem-helper: Use refcount_t for pages_use_count Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 09/30] drm/shmem-helper: Add and use lockless drm_gem_shmem_get_pages() Dmitry Osipenko
2024-01-25 17:24 ` Daniel Vetter
2024-01-25 21:52 ` Dmitry Osipenko
2024-01-26 10:18 ` Boris Brezillon
2024-01-26 16:43 ` Dmitry Osipenko
2024-01-30 8:34 ` Daniel Vetter
2024-01-30 10:05 ` Dmitry Osipenko
2024-01-30 10:10 ` Boris Brezillon
2024-02-01 18:53 ` Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 10/30] drm/shmem-helper: Switch drm_gem_shmem_vmap/vunmap to use pin/unpin Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 11/30] drm/shmem-helper: Use refcount_t for vmap_use_count Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 12/30] drm/shmem-helper: Prepare drm_gem_shmem_free() to shrinker addition Dmitry Osipenko
2024-01-25 8:01 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 13/30] drm/shmem-helper: Make drm_gem_shmem_get_pages() public Dmitry Osipenko
2024-01-25 8:02 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 14/30] drm/shmem-helper: Add drm_gem_shmem_put_pages() Dmitry Osipenko
2024-01-25 8:02 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 15/30] drm/shmem-helper: Avoid lockdep warning when pages are released Dmitry Osipenko
2024-01-25 8:02 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 16/30] drm/lima: Explicitly get and put drm-shmem pages Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 17/30] drm/panfrost: Fix the error path in panfrost_mmu_map_fault_addr() Dmitry Osipenko
2024-01-25 16:47 ` Steven Price
2024-01-25 21:41 ` Dmitry Osipenko
2024-01-25 21:44 ` Dmitry Osipenko
2024-01-26 9:08 ` AngeloGioacchino Del Regno
2024-04-04 15:23 ` Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 18/30] drm/panfrost: Explicitly get and put drm-shmem pages Dmitry Osipenko
2024-01-25 8:10 ` Boris Brezillon
2024-01-25 16:47 ` Steven Price
2024-01-26 9:39 ` Boris Brezillon [this message]
2024-01-26 11:26 ` Steven Price
2024-01-05 18:46 ` [PATCH v19 19/30] drm/virtio: " Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 20/30] drm/v3d: " Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 21/30] drm/shmem-helper: Change sgt allocation policy Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 22/30] drm/shmem-helper: Add common memory shrinker Dmitry Osipenko
2024-01-25 9:07 ` Boris Brezillon
2024-01-30 8:39 ` Daniel Vetter
2024-01-30 10:04 ` Dmitry Osipenko
2024-01-25 10:19 ` Boris Brezillon
2024-01-25 21:56 ` Dmitry Osipenko
2024-01-26 9:55 ` Boris Brezillon
2024-01-26 16:27 ` Dmitry Osipenko
2024-01-26 18:12 ` Boris Brezillon
2024-01-29 6:16 ` Dmitry Osipenko
2024-01-29 8:55 ` Boris Brezillon
2024-01-29 9:06 ` Boris Brezillon
2024-01-29 9:01 ` Boris Brezillon
2024-01-29 9:25 ` Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 23/30] drm/shmem-helper: Export drm_gem_shmem_get_pages_sgt_locked() Dmitry Osipenko
2024-01-25 9:09 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 24/30] drm/shmem-helper: Optimize unlocked get_pages_sgt() Dmitry Osipenko
2024-01-25 9:28 ` Boris Brezillon
2024-01-05 18:46 ` [PATCH v19 25/30] drm/shmem-helper: Don't free refcounted GEM Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 26/30] drm/shmem-helper: Turn warnings about imported GEM into errors Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 27/30] drm/virtio: Pin display framebuffer BO Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 28/30] drm/virtio: Attach shmem BOs dynamically Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 29/30] drm/virtio: Support shmem shrinking Dmitry Osipenko
2024-01-05 18:46 ` [PATCH v19 30/30] drm/panfrost: Switch to generic memory shrinker Dmitry Osipenko
2024-01-25 9:49 ` Boris Brezillon
2024-01-25 12:04 ` Dmitry Osipenko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240126103924.0b911a4f@collabora.com \
--to=boris.brezillon@collabora.com \
--cc=airlied@gmail.com \
--cc=christian.koenig@amd.com \
--cc=daniel@ffwll.ch \
--cc=dmitry.osipenko@collabora.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=emma@anholt.net \
--cc=gurchetansingh@chromium.org \
--cc=kernel@collabora.com \
--cc=kraxel@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=mripard@kernel.org \
--cc=mwen@igalia.com \
--cc=olvaffe@gmail.com \
--cc=steven.price@arm.com \
--cc=tzimmermann@suse.de \
--cc=virtualization@lists.linux-foundation.org \
--cc=yuq825@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).