Nouveau Archive on
 help / color / Atom feed
From: Alistair Popple <>
To: Ralph Campbell <>
Subject: Re: [Nouveau] [PATCH v4 5/8] mm: Device exclusive memory access
Date: Tue, 9 Mar 2021 12:17:16 +1100
Message-ID: <1795020.LnfgZAJ4CS@nvdebian> (raw)
In-Reply-To: <>

On Tuesday, 9 March 2021 6:44:41 AM AEDT Ralph Campbell wrote:
> On 3/3/21 10:16 PM, Alistair Popple wrote:
> > Some devices require exclusive write access to shared virtual
> > memory (SVM) ranges to perform atomic operations on that memory. This
> > requires CPU page tables to be updated to deny access whilst atomic
> > operations are occurring.
> > 
> > In order to do this introduce a new swap entry
> > type (SWP_DEVICE_EXCLUSIVE). When a SVM range needs to be marked for
> > exclusive access by a device all page table mappings for the particular
> > range are replaced with device exclusive swap entries. This causes any
> > CPU access to the page to result in a fault.
> > 
> > Faults are resovled by replacing the faulting entry with the original
> > mapping. This results in MMU notifiers being called which a driver uses
> > to update access permissions such as revoking atomic access. After
> > notifiers have been called the device will no longer have exclusive
> > access to the region.
> > 
> > Signed-off-by: Alistair Popple <>
> I see in the next two patches how make_device_exclusive_entry() and
> check_device_exclusive_range() are used. This points out a similar
> problem that migrate_vma_setup() had before I added the
> mmu_notifier_range_init_migrate() helper to pass a cookie from
> migrate_vma_setup() to the invalidation callback so the device driver
> could ignore an invalidation callback triggered by the caller and thus
> resulting in a deadlock or having to invalidate device PTEs that
> wouldn't be migrating.
> I think you can eliminate the need for check_device_exclusive_range() in
> the same way by adding a "void *" pointer to make_device_exclusive_entry()
> and passing that through to try_to_protect(), setting rmap_walk_control 
> and then passing arg to mmu_notifier_range_init_migrate().

Thanks for the idea, I had missed there was already a way of passing a "void 
*" as part of mmu_notifier_range_init_migrate(). Agree that should allow a 
single pass without needing check_device_exclusive_range().

As Jason points out still need to check the GUP page is mapped at the expected 
address but that can be done as part of installing the exclusive swap entry in 

> Although, maybe it would be better to define a new
> mmu_notifier_range_init_exclusive() and event type MMU_NOTIFY_EXCLUSIVE so
> that a device driver can revoke atomic/exclusive access but keep read/write
> access to other parts of the page.

Agree, I don't think overloading mmu_notifier_range_init_migrate() with the 
exclusive usage is correct. Better to define a new helper.

> I thought about how make_device_exclusive_entry() is similar to 
> and whether it would be possible to add a new HMM_PFN_REQ_EXCLUSIVE flag but 
> see that make_device_exclusive_entry() returns the pages locked and with an
> additional get_page() reference. This doesn't fit well with the other
> hmm_range_fault() entries being returned as a "snapshot" so having a 
> API makes sense. I think it would be useful to add a HMM_PFN_EXCLUSIVE flag 
> that snapshots of the page tables can at least report that a page is 
> being accessed by *some* device. Unfortunately, there is no pgmap pointer to 
> able to tell which device has exclusive access (since any struct page could 
> exclusively accessed, not just device private ones).
I have also experimented with integrating this with HMM but it just didn't end 
up being a good fit for the reasons you mention.

I also don't think adding HMM_PFN_EXCLUSIVE to read page table snapshots is 
that useful because there is no way to tell *which* device has exclusive 
access. So unless I've missed some particular usage for it now I think it can 
probably be added as a future improvement to HMM if/when it is needed.

 - Alistair

Nouveau mailing list

  reply index

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-03-04  6:16 [Nouveau] [PATCH v4 0/8] Add support for SVM atomics in Nouveau Alistair Popple
2021-03-04  6:16 ` [Nouveau] [PATCH v4 1/8] mm: Remove special swap entry functions Alistair Popple
2021-03-08 18:24   ` Ralph Campbell
2021-03-04  6:16 ` [Nouveau] [PATCH v4 2/8] mm/swapops: Rework swap entry manipulation code Alistair Popple
2021-03-08 18:30   ` Ralph Campbell
2021-03-04  6:16 ` [Nouveau] [PATCH v4 3/8] mm/rmap: Split try_to_munlock from try_to_unmap Alistair Popple
2021-03-08 18:39   ` Ralph Campbell
2021-03-04  6:16 ` [Nouveau] [PATCH v4 4/8] mm/rmap: Split migration into its own function Alistair Popple
2021-03-08 18:58   ` Ralph Campbell
2021-03-09  0:01     ` Alistair Popple
2021-03-04  6:16 ` [Nouveau] [PATCH v4 5/8] mm: Device exclusive memory access Alistair Popple
2021-03-08 19:44   ` Ralph Campbell
2021-03-09  1:17     ` Alistair Popple [this message]
2021-03-04  6:16 ` [Nouveau] [PATCH v4 6/8] mm: Selftests for exclusive device memory Alistair Popple
2021-03-04  6:16 ` [Nouveau] [PATCH v4 7/8] nouveau/svm: Refactor nouveau_range_fault Alistair Popple
2021-03-04  6:16 ` [Nouveau] [PATCH v4 8/8] nouveau/svm: Implement atomic SVM access Alistair Popple

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1795020.LnfgZAJ4CS@nvdebian \ \ \ \ \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Nouveau Archive on

Archives are clonable:
	git clone --mirror nouveau/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 nouveau nouveau/ \
	public-inbox-index nouveau

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone