From: Alistair Popple <apopple@nvidia.com> To: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org> Cc: Ralph Campbell <rcampbell@nvidia.com>, John Hubbard <jhubbard@nvidia.com>, nouveau@lists.freedesktop.org, Alistair Popple <apopple@nvidia.com>, stable@vger.kernel.org Subject: [PATCH] mm: Take a page reference when removing device exclusive entries Date: Tue, 28 Mar 2023 13:14:34 +1100 [thread overview] Message-ID: <20230328021434.292971-1-apopple@nvidia.com> (raw) Device exclusive page table entries are used to prevent CPU access to a page whilst it is being accessed from a device. Typically this is used to implement atomic operations when the underlying bus does not support atomic access. When a CPU thread encounters a device exclusive entry it locks the page and restores the original entry after calling mmu notifiers to signal drivers that exclusive access is no longer available. The device exclusive entry holds a reference to the page making it safe to access the struct page whilst the entry is present. However the fault handling code does not hold the PTL when taking the page lock. This means if there are multiple threads faulting concurrently on the device exclusive entry one will remove the entry whilst others will wait on the page lock without holding a reference. This can lead to threads locking or waiting on a page with a zero refcount. Whilst mmap_lock prevents the pages getting freed via munmap() they may still be freed by a migration. This leads to warnings such as PAGE_FLAGS_CHECK_AT_FREE due to the page being locked when the refcount drops to zero. Note that during removal of the device exclusive entry the PTE is currently re-checked under the PTL so no futher bad page accesses occur once it is locked. Signed-off-by: Alistair Popple <apopple@nvidia.com> Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Cc: stable@vger.kernel.org --- mm/memory.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 8c8420934d60..b499bd283d8e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3623,8 +3623,19 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct mmu_notifier_range range; - if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) + /* + * We need a page reference to lock the page because we don't + * hold the PTL so a racing thread can remove the + * device-exclusive entry and unmap the page. If the page is + * free the entry must have been removed already. + */ + if (!get_page_unless_zero(vmf->page)) + return 0; + + if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) { + put_page(vmf->page); return VM_FAULT_RETRY; + } mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, vma, vma->vm_mm, vmf->address & PAGE_MASK, (vmf->address & PAGE_MASK) + PAGE_SIZE, NULL); @@ -3637,6 +3648,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); + put_page(vmf->page); mmu_notifier_invalidate_range_end(&range); return 0; -- 2.39.2
WARNING: multiple messages have this Message-ID (diff)
From: Alistair Popple <apopple@nvidia.com> To: linux-mm@kvack.org, Andrew Morton <akpm@linux-foundation.org> Cc: Ralph Campbell <rcampbell@nvidia.com>, Alistair Popple <apopple@nvidia.com>, stable@vger.kernel.org, nouveau@lists.freedesktop.org Subject: [Nouveau] [PATCH] mm: Take a page reference when removing device exclusive entries Date: Tue, 28 Mar 2023 13:14:34 +1100 [thread overview] Message-ID: <20230328021434.292971-1-apopple@nvidia.com> (raw) Device exclusive page table entries are used to prevent CPU access to a page whilst it is being accessed from a device. Typically this is used to implement atomic operations when the underlying bus does not support atomic access. When a CPU thread encounters a device exclusive entry it locks the page and restores the original entry after calling mmu notifiers to signal drivers that exclusive access is no longer available. The device exclusive entry holds a reference to the page making it safe to access the struct page whilst the entry is present. However the fault handling code does not hold the PTL when taking the page lock. This means if there are multiple threads faulting concurrently on the device exclusive entry one will remove the entry whilst others will wait on the page lock without holding a reference. This can lead to threads locking or waiting on a page with a zero refcount. Whilst mmap_lock prevents the pages getting freed via munmap() they may still be freed by a migration. This leads to warnings such as PAGE_FLAGS_CHECK_AT_FREE due to the page being locked when the refcount drops to zero. Note that during removal of the device exclusive entry the PTE is currently re-checked under the PTL so no futher bad page accesses occur once it is locked. Signed-off-by: Alistair Popple <apopple@nvidia.com> Fixes: b756a3b5e7ea ("mm: device exclusive memory access") Cc: stable@vger.kernel.org --- mm/memory.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 8c8420934d60..b499bd283d8e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3623,8 +3623,19 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) struct vm_area_struct *vma = vmf->vma; struct mmu_notifier_range range; - if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) + /* + * We need a page reference to lock the page because we don't + * hold the PTL so a racing thread can remove the + * device-exclusive entry and unmap the page. If the page is + * free the entry must have been removed already. + */ + if (!get_page_unless_zero(vmf->page)) + return 0; + + if (!folio_lock_or_retry(folio, vma->vm_mm, vmf->flags)) { + put_page(vmf->page); return VM_FAULT_RETRY; + } mmu_notifier_range_init_owner(&range, MMU_NOTIFY_EXCLUSIVE, 0, vma, vma->vm_mm, vmf->address & PAGE_MASK, (vmf->address & PAGE_MASK) + PAGE_SIZE, NULL); @@ -3637,6 +3648,7 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); + put_page(vmf->page); mmu_notifier_invalidate_range_end(&range); return 0; -- 2.39.2
next reply other threads:[~2023-03-28 2:14 UTC|newest] Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-03-28 2:14 Alistair Popple [this message] 2023-03-28 2:14 ` [Nouveau] [PATCH] mm: Take a page reference when removing device exclusive entries Alistair Popple 2023-03-28 6:25 ` John Hubbard 2023-03-28 6:25 ` [Nouveau] " John Hubbard 2023-03-28 19:55 ` Andrew Morton 2023-03-28 19:55 ` [Nouveau] " Andrew Morton 2023-03-29 1:45 ` Alistair Popple 2023-03-29 1:45 ` [Nouveau] " Alistair Popple 2023-03-28 19:59 ` Ralph Campbell 2023-03-28 19:59 ` Ralph Campbell 2023-03-29 3:16 ` Matthew Wilcox 2023-03-29 3:16 ` [Nouveau] " Matthew Wilcox 2023-03-29 3:45 ` John Hubbard 2023-03-29 3:45 ` [Nouveau] " John Hubbard 2023-03-30 0:26 ` Alistair Popple 2023-03-30 0:26 ` [Nouveau] " Alistair Popple 2023-03-30 0:55 ` Alistair Popple 2023-03-30 0:55 ` [Nouveau] " Alistair Popple
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20230328021434.292971-1-apopple@nvidia.com \ --to=apopple@nvidia.com \ --cc=akpm@linux-foundation.org \ --cc=jhubbard@nvidia.com \ --cc=linux-mm@kvack.org \ --cc=nouveau@lists.freedesktop.org \ --cc=rcampbell@nvidia.com \ --cc=stable@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.