Re: [PATCH v6 01/10] mm: add zone device coherent type memory support

From: Jason Gunthorpe <jgg@nvidia.com>
To: David Hildenbrand <david@redhat.com>
Cc: Alistair Popple <apopple@nvidia.com>,
	Felix Kuehling <felix.kuehling@amd.com>,
	Christoph Hellwig <hch@lst.de>, Alex Sierra <alex.sierra@amd.com>,
	akpm@linux-foundation.org, linux-mm@kvack.org,
	rcampbell@nvidia.com, linux-ext4@vger.kernel.org,
	linux-xfs@vger.kernel.org, amd-gfx@lists.freedesktop.org,
	dri-devel@lists.freedesktop.org, jglisse@redhat.com,
	willy@infradead.org
Subject: Re: [PATCH v6 01/10] mm: add zone device coherent type memory support
Date: Wed, 16 Feb 2022 08:26:00 -0400	[thread overview]
Message-ID: <20220216122600.GG4160@nvidia.com> (raw)
In-Reply-To: <98d8bbc5-ffc2-8966-fdc1-a844874e7ae8@redhat.com>

On Wed, Feb 16, 2022 at 09:31:03AM +0100, David Hildenbrand wrote:
> On 16.02.22 03:36, Alistair Popple wrote:
> > On Wednesday, 16 February 2022 1:03:57 PM AEDT Jason Gunthorpe wrote:
> >> On Wed, Feb 16, 2022 at 12:23:44PM +1100, Alistair Popple wrote:
> >>
> >>> Device private and device coherent pages are not marked with pte_devmap and they
> >>> are backed by a struct page. The only way of inserting them is via migrate_vma.
> >>> The refcount is decremented in zap_pte_range() on munmap() with special handling
> >>> for device private pages. Looking at it again though I wonder if there is any
> >>> special treatment required in zap_pte_range() for device coherent pages given
> >>> they count as present pages.
> >>
> >> This is what I guessed, but we shouldn't be able to just drop
> >> pte_devmap on these pages without any other work?? Granted it does
> >> very little already..
> > 
> > Yes, I agree we need to check this more closely. For device private pages
> > not having pte_devmap is fine, because they are non-present swap entries so
> > they always get special handling in the swap entry paths but the same isn't
> > true for coherent device pages.
> 
> I'm curious, how does the refcount of a PageAnon() DEVICE_COHERENT page
> look like when mapped? I'd assume it's also (currently) still offset by
> one, meaning, if it's mapped into a single page table it's always at
> least 2.

Christoph fixed this offset by one and updated the DEVICE_COHERENT
patchset, I hope we will see that version merged.

> >> I thought at least gup_fast needed to be touched or did this get
> >> handled by scanning the page list after the fact?
> > 
> > Right, for gup I think the only special handling required is to prevent
> > pinning. I had assumed that check_and_migrate_movable_pages() would still get
> > called for gup_fast but unless I've missed something I don't think it does.
> > That means gup_fast could still pin movable and coherent pages. Technically
> > that is ok for coherent pages, but it's undesirable.
> 
> We really should have the same pinning rules for GUP vs. GUP-fast.
> is_pinnable_page() should be the right place for such checks (similarly
> as indicated in my reply to the migration series).

Yes, I think this is a bug too.

The other place that needs careful audit is all the callers using
vm_normal_page() - they must all be able to accept a ZONE_DEVICE page
if we don't set pte_devmap.

Jason 