Re: [PATCH 7/8] drm/panfrost: Add the panfrost_gem_mapping concept

From: Boris Brezillon <boris.brezillon@collabora.com>
To: Daniel Vetter <daniel@ffwll.ch>
Cc: Rob Herring <robh+dt@kernel.org>,
	Tomeu Vizoso <tomeu@tomeuvizoso.net>,
	Alyssa Rosenzweig <alyssa.rosenzweig@collabora.com>,
	Steven Price <steven.price@arm.com>,
	stable@vger.kernel.org, dri-devel@lists.freedesktop.org
Subject: Re: [PATCH 7/8] drm/panfrost: Add the panfrost_gem_mapping concept
Date: Fri, 29 Nov 2019 22:36:29 +0100	[thread overview]
Message-ID: <20191129223629.3aaab761@collabora.com> (raw)
In-Reply-To: <20191129201459.GS624164@phenom.ffwll.local>

On Fri, 29 Nov 2019 21:14:59 +0100
Daniel Vetter <daniel@ffwll.ch> wrote:

> On Fri, Nov 29, 2019 at 02:59:07PM +0100, Boris Brezillon wrote:
> > With the introduction of per-FD address space, the same BO can be mapped
> > in different address space if the BO is globally visible (GEM_FLINK)  
> 
> Also dma-buf self-imports for wayland/dri3 ...

Indeed, I'll extend the commit message to mention that case.

> 
> > and opened in different context. The current implementation does not
> > take case into account, and attaches the mapping directly to the
> > panfrost_gem_object.
> > 
> > Let's create a panfrost_gem_mapping struct and allow multiple mappings
> > per BO.
> > 
> > The mappings are refcounted, which helps solve another problem where
> > mappings were teared down (GEM handle closed by userspace) while GPU
> > jobs accessing those BOs were still in-flight. Jobs now keep a
> > reference on the mappings they use.  
> 
> uh what.
> 
> tbh this sounds bad enough (as in how did a desktop on panfrost ever work)

Well, we didn't discover this problem until recently because:

1/ We have a BO cache in mesa, and until recently, this cache could
only grow (no entry eviction and no MADVISE support), meaning that BOs
were staying around forever until the app was killed.

2/ Mappings were teared down at BO destruction time before commit
a5efb4c9a562 ("drm/panfrost: Restructure the GEM object creation"), and
jobs are retaining references to all the BO they access.

3/ The mesa driver was serializing GPU jobs, and only releasing the BO
reference when the job was done (wait on the completion fence). This
has recently been changed, and now BOs are returned to the cache as
soon as the job has been submitted to the kernel. When that
happens,those BOs are marked purgeable which means the kernel can
reclaim them when it's under memory pressure.

So yes, kernel 5.4 with a recent mesa version is currently subject to
GPU page-fault storms when the system starts reclaiming memory.

> that I think you really want a few igts to test this stuff.

I'll see what I can come up with (not sure how to easily detect
pagefaults from userspace).