* [RFC PATCH 0/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization
@ 2020-10-13 23:10 Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 1/3] hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync Mike Kravetz
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Mike Kravetz @ 2020-10-13 23:10 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Hugh Dickins, Michal Hocko, Naoya Horiguchi, Aneesh Kumar K . V,
Andrea Arcangeli, Kirill A . Shutemov, Davidlohr Bueso,
Prakash Sangappa, Andrew Morton, Mike Kravetz
In commit c0d0381ade79, changes were made to use i_mmap_rwsem for pmd
sharing synchronization. This required changes to mm locking order that
are hugetlb specific. Specifically, i_mmap_rwsem must be taken before
the page lock. This is not not a huge issue in hugetlb specific code,
but becomes more problematic in the areas of page migration and memory
failure where generic mm code had to deal with this change to lock
ordering. An ugly routine 'hugetlb_page_mapping_lock_write' was added
to help with these issues.
Recently, Hugh Dickins diagnosed a migration BUG as caused by code
introduced with hugetlb i_mmap_rwsem synchronization [1]. Subsequent
discussion in that thread pointed out additional problems in the code.
Adding a rw_semaphore to the hugetlbfs inode for this type of synchronization
was mentioned. Such an approach is actually 'cleaner' as it can be
inserted in the lock hierarchy where needed. And, there is no issue
with other parts of the mm using this rw_semaphore.
This series adds a rw_semaphore (hinode_rwsem) to the hugetlbfs inode.
The first patch reverts all commits having to deal with the current use
of i_mmap_rwsem for pmd sharing and fault/truncate synchronization. The
revert of 5 commits was combined into a single patch. I am looking for
feedback on this approach. I considered:
- 5 Patches to revert the 5 commits
- Reverting patches depending on c0d0381ade79, then having a patch to
change from i_mmap_rwsem to hinode_rwsem.
To me, a 'clean slate' approach seemed best but I am open to whatever
would be easiest to review.
[1] https://lore.kernel.org/linux-mm/alpine.LSU.2.11.2010071833100.2214@eggly.anvils/
Mike Kravetz (3):
hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync
hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization
huegtlbfs: handle page fault/truncate races
fs/hugetlbfs/inode.c | 72 ++++++------
include/linux/fs.h | 15 ---
include/linux/hugetlb.h | 72 ++++++++++--
mm/hugetlb.c | 243 +++++++++++++++-------------------------
mm/memory-failure.c | 34 ++----
mm/migrate.c | 34 +++---
mm/rmap.c | 17 +--
mm/userfaultfd.c | 14 +--
8 files changed, 220 insertions(+), 281 deletions(-)
--
2.25.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* [RFC PATCH 1/3] hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync
2020-10-13 23:10 [RFC PATCH 0/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
@ 2020-10-13 23:10 ` Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
2020-10-13 23:11 ` [RFC PATCH 3/3] huegtlbfs: handle page fault/truncate races Mike Kravetz
2 siblings, 0 replies; 6+ messages in thread
From: Mike Kravetz @ 2020-10-13 23:10 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Hugh Dickins, Michal Hocko, Naoya Horiguchi, Aneesh Kumar K . V,
Andrea Arcangeli, Kirill A . Shutemov, Davidlohr Bueso,
Prakash Sangappa, Andrew Morton, Mike Kravetz
i_mmap_rwsem is being used for synchronization of huge pmd sharing.
This required changing the locking order such that i_mmap_rwsem needed
to be taken before a page lock. This can be accommodated in hugetlbfs
specific code, but becomes problematic in the areas of page migration
and memory failure where it is hard to get around the normal locking
order.
In subsequent patches, a new rw semaphore will be added to the hugetlbfs
inode for this type of synchronization. This patch will revert use of
i_mmap_rwsem for pmd sharing and fault/truncate synchronization.
The following commits are reverted:
commit xxxxxxxxxxxx ("hugetlb: add lockdep check for i_mmap_rwsem held in
huge_pmd_share")
commit 34ae204f1851 ("hugetlbfs: remove call to huge_pte_alloc without
i_mmap_rwsem")
commit 1139d336fff4 ("mm/hugetlb.c: fix pages per hugetlb calculation")
commit 87bf91d39bb5 ("hugetlbfs: Use i_mmap_rwsem to address page
fault/truncate race")
commit c0d0381ade79 ("hugetlbfs: use i_mmap_rwsem for more pmd sharing
synchronization")
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 30 ++----
include/linux/fs.h | 15 ---
include/linux/hugetlb.h | 16 +---
mm/hugetlb.c | 197 ++++++----------------------------------
mm/memory-failure.c | 29 +-----
mm/migrate.c | 25 +----
mm/rmap.c | 18 +---
mm/userfaultfd.c | 11 +--
8 files changed, 49 insertions(+), 292 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index b5c109703daa..3a1246aeedc4 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -444,9 +444,10 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end)
* In this case, we first scan the range and release found pages.
* After releasing pages, hugetlb_unreserve_pages cleans up region/reserv
* maps and global counts. Page faults can not race with truncation
- * in this routine. hugetlb_no_page() holds i_mmap_rwsem and prevents
- * page faults in the truncated range by checking i_size. i_size is
- * modified while holding i_mmap_rwsem.
+ * in this routine. hugetlb_no_page() prevents page faults in the
+ * truncated range. It checks i_size before allocation, and again after
+ * with the page table lock for the page held. The same lock must be
+ * acquired to unmap a page.
* hole punch is indicated if end is not LLONG_MAX
* In the hole punch case we scan the range and release found pages.
* Only when releasing a page is the associated region/reserv map
@@ -486,15 +487,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
index = page->index;
hash = hugetlb_fault_mutex_hash(mapping, index);
- if (!truncate_op) {
- /*
- * Only need to hold the fault mutex in the
- * hole punch case. This prevents races with
- * page faults. Races are not possible in the
- * case of truncation.
- */
- mutex_lock(&hugetlb_fault_mutex_table[hash]);
- }
+ mutex_lock(&hugetlb_fault_mutex_table[hash]);
/*
* If page is mapped, it was faulted in after being
@@ -508,9 +501,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
if (unlikely(page_mapped(page))) {
BUG_ON(truncate_op);
- mutex_unlock(&hugetlb_fault_mutex_table[hash]);
i_mmap_lock_write(mapping);
- mutex_lock(&hugetlb_fault_mutex_table[hash]);
hugetlb_vmdelete_list(&mapping->i_mmap,
index * pages_per_huge_page(h),
(index + 1) * pages_per_huge_page(h));
@@ -537,8 +528,7 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
}
unlock_page(page);
- if (!truncate_op)
- mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ mutex_unlock(&hugetlb_fault_mutex_table[hash]);
}
huge_pagevec_release(&pvec);
cond_resched();
@@ -576,8 +566,8 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
BUG_ON(offset & ~huge_page_mask(h));
pgoff = offset >> PAGE_SHIFT;
- i_mmap_lock_write(mapping);
i_size_write(inode, offset);
+ i_mmap_lock_write(mapping);
if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
@@ -699,11 +689,7 @@ static long hugetlbfs_fallocate(struct file *file, int mode, loff_t offset,
/* addr is the offset within the file (zero based) */
addr = index * hpage_size;
- /*
- * fault mutex taken here, protects against fault path
- * and hole punch. inode_lock previously taken protects
- * against truncation.
- */
+ /* mutex taken here, fault path and hole punch */
hash = hugetlb_fault_mutex_hash(mapping, index);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 3dac62137768..b2b820dda8ab 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -493,11 +493,6 @@ static inline void i_mmap_lock_write(struct address_space *mapping)
down_write(&mapping->i_mmap_rwsem);
}
-static inline int i_mmap_trylock_write(struct address_space *mapping)
-{
- return down_write_trylock(&mapping->i_mmap_rwsem);
-}
-
static inline void i_mmap_unlock_write(struct address_space *mapping)
{
up_write(&mapping->i_mmap_rwsem);
@@ -513,16 +508,6 @@ static inline void i_mmap_unlock_read(struct address_space *mapping)
up_read(&mapping->i_mmap_rwsem);
}
-static inline void i_mmap_assert_locked(struct address_space *mapping)
-{
- lockdep_assert_held(&mapping->i_mmap_rwsem);
-}
-
-static inline void i_mmap_assert_write_locked(struct address_space *mapping)
-{
- lockdep_assert_held_write(&mapping->i_mmap_rwsem);
-}
-
/*
* Might pages of this file be mapped into userspace?
*/
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index ebca2ef02212..bf79a5601091 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -154,8 +154,6 @@ u32 hugetlb_fault_mutex_hash(struct address_space *mapping, pgoff_t idx);
pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud);
-struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage);
-
extern int sysctl_hugetlb_shm_group;
extern struct list_head huge_boot_pages;
@@ -165,8 +163,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
pte_t *huge_pte_offset(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep);
+int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep);
void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
unsigned long *start, unsigned long *end);
struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -199,15 +196,8 @@ static inline unsigned long hugetlb_total_pages(void)
return 0;
}
-static inline struct address_space *hugetlb_page_mapping_lock_write(
- struct page *hpage)
-{
- return NULL;
-}
-
-static inline int huge_pmd_unshare(struct mm_struct *mm,
- struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+static inline int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr,
+ pte_t *ptep)
{
return 0;
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index fe76f8fd5a73..9a316b6d0b51 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1568,106 +1568,6 @@ int PageHeadHuge(struct page *page_head)
return page_head[1].compound_dtor == HUGETLB_PAGE_DTOR;
}
-/*
- * Find address_space associated with hugetlbfs page.
- * Upon entry page is locked and page 'was' mapped although mapped state
- * could change. If necessary, use anon_vma to find vma and associated
- * address space. The returned mapping may be stale, but it can not be
- * invalid as page lock (which is held) is required to destroy mapping.
- */
-static struct address_space *_get_hugetlb_page_mapping(struct page *hpage)
-{
- struct anon_vma *anon_vma;
- pgoff_t pgoff_start, pgoff_end;
- struct anon_vma_chain *avc;
- struct address_space *mapping = page_mapping(hpage);
-
- /* Simple file based mapping */
- if (mapping)
- return mapping;
-
- /*
- * Even anonymous hugetlbfs mappings are associated with an
- * underlying hugetlbfs file (see hugetlb_file_setup in mmap
- * code). Find a vma associated with the anonymous vma, and
- * use the file pointer to get address_space.
- */
- anon_vma = page_lock_anon_vma_read(hpage);
- if (!anon_vma)
- return mapping; /* NULL */
-
- /* Use first found vma */
- pgoff_start = page_to_pgoff(hpage);
- pgoff_end = pgoff_start + pages_per_huge_page(page_hstate(hpage)) - 1;
- anon_vma_interval_tree_foreach(avc, &anon_vma->rb_root,
- pgoff_start, pgoff_end) {
- struct vm_area_struct *vma = avc->vma;
-
- mapping = vma->vm_file->f_mapping;
- break;
- }
-
- anon_vma_unlock_read(anon_vma);
- return mapping;
-}
-
-/*
- * Find and lock address space (mapping) in write mode.
- *
- * Upon entry, the page is locked which allows us to find the mapping
- * even in the case of an anon page. However, locking order dictates
- * the i_mmap_rwsem be acquired BEFORE the page lock. This is hugetlbfs
- * specific. So, we first try to lock the sema while still holding the
- * page lock. If this works, great! If not, then we need to drop the
- * page lock and then acquire i_mmap_rwsem and reacquire page lock. Of
- * course, need to revalidate state along the way.
- */
-struct address_space *hugetlb_page_mapping_lock_write(struct page *hpage)
-{
- struct address_space *mapping, *mapping2;
-
- mapping = _get_hugetlb_page_mapping(hpage);
-retry:
- if (!mapping)
- return mapping;
-
- /*
- * If no contention, take lock and return
- */
- if (i_mmap_trylock_write(mapping))
- return mapping;
-
- /*
- * Must drop page lock and wait on mapping sema.
- * Note: Once page lock is dropped, mapping could become invalid.
- * As a hack, increase map count until we lock page again.
- */
- atomic_inc(&hpage->_mapcount);
- unlock_page(hpage);
- i_mmap_lock_write(mapping);
- lock_page(hpage);
- atomic_add_negative(-1, &hpage->_mapcount);
-
- /* verify page is still mapped */
- if (!page_mapped(hpage)) {
- i_mmap_unlock_write(mapping);
- return NULL;
- }
-
- /*
- * Get address space again and verify it is the same one
- * we locked. If not, drop lock and retry.
- */
- mapping2 = _get_hugetlb_page_mapping(hpage);
- if (mapping2 != mapping) {
- i_mmap_unlock_write(mapping);
- mapping = mapping2;
- goto retry;
- }
-
- return mapping;
-}
-
pgoff_t __basepage_index(struct page *page)
{
struct page *page_head = compound_head(page);
@@ -3818,7 +3718,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
int cow;
struct hstate *h = hstate_vma(vma);
unsigned long sz = huge_page_size(h);
- struct address_space *mapping = vma->vm_file->f_mapping;
struct mmu_notifier_range range;
int ret = 0;
@@ -3829,14 +3728,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
vma->vm_start,
vma->vm_end);
mmu_notifier_invalidate_range_start(&range);
- } else {
- /*
- * For shared mappings i_mmap_rwsem must be held to call
- * huge_pte_alloc, otherwise the returned ptep could go
- * away if part of a shared pmd and another thread calls
- * huge_pmd_unshare.
- */
- i_mmap_lock_read(mapping);
}
for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) {
@@ -3914,8 +3805,6 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
if (cow)
mmu_notifier_invalidate_range_end(&range);
- else
- i_mmap_unlock_read(mapping);
return ret;
}
@@ -3959,7 +3848,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, &address, ptep)) {
spin_unlock(ptl);
/*
* We just unmapped a page of PMDs by clearing a PUD.
@@ -4335,17 +4224,16 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
}
/*
- * We can not race with truncation due to holding i_mmap_rwsem.
- * i_size is modified when holding i_mmap_rwsem, so check here
- * once for faults beyond end of file.
+ * Use page lock to guard against racing truncation
+ * before we get page_table_lock.
*/
- size = i_size_read(mapping->host) >> huge_page_shift(h);
- if (idx >= size)
- goto out;
-
retry:
page = find_lock_page(mapping, idx);
if (!page) {
+ size = i_size_read(mapping->host) >> huge_page_shift(h);
+ if (idx >= size)
+ goto out;
+
/*
* Check for page in userfault range
*/
@@ -4365,15 +4253,13 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
};
/*
- * hugetlb_fault_mutex and i_mmap_rwsem must be
- * dropped before handling userfault. Reacquire
- * after handling fault to make calling code simpler.
+ * hugetlb_fault_mutex must be dropped before
+ * handling userfault. Reacquire after handling
+ * fault to make calling code simpler.
*/
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- i_mmap_unlock_read(mapping);
ret = handle_userfault(&vmf, VM_UFFD_MISSING);
- i_mmap_lock_read(mapping);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
goto out;
}
@@ -4451,6 +4337,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
}
ptl = huge_pte_lock(h, mm, ptep);
+ size = i_size_read(mapping->host) >> huge_page_shift(h);
+ if (idx >= size)
+ goto backout;
+
ret = 0;
if (!huge_pte_none(huge_ptep_get(ptep)))
goto backout;
@@ -4534,11 +4424,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
if (ptep) {
- /*
- * Since we hold no locks, ptep could be stale. That is
- * OK as we are only making decisions based on content and
- * not actually modifying content here.
- */
entry = huge_ptep_get(ptep);
if (unlikely(is_hugetlb_entry_migration(entry))) {
migration_entry_wait_huge(vma, mm, ptep);
@@ -4546,33 +4431,20 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
} else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
return VM_FAULT_HWPOISON_LARGE |
VM_FAULT_SET_HINDEX(hstate_index(h));
+ } else {
+ ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
+ if (!ptep)
+ return VM_FAULT_OOM;
}
- /*
- * Acquire i_mmap_rwsem before calling huge_pte_alloc and hold
- * until finished with ptep. This serves two purposes:
- * 1) It prevents huge_pmd_unshare from being called elsewhere
- * and making the ptep no longer valid.
- * 2) It synchronizes us with i_size modifications during truncation.
- *
- * ptep could have already be assigned via huge_pte_offset. That
- * is OK, as huge_pte_alloc will return the same value unless
- * something has changed.
- */
mapping = vma->vm_file->f_mapping;
- i_mmap_lock_read(mapping);
- ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
- if (!ptep) {
- i_mmap_unlock_read(mapping);
- return VM_FAULT_OOM;
- }
+ idx = vma_hugecache_offset(h, vma, haddr);
/*
* Serialize hugepage allocation and instantiation, so that we don't
* get spurious allocation failures if two CPUs race to instantiate
* the same page in the page cache.
*/
- idx = vma_hugecache_offset(h, vma, haddr);
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
@@ -4660,7 +4532,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
}
out_mutex:
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- i_mmap_unlock_read(mapping);
/*
* Generally it's safe to hold refcount during waiting page lock. But
* here we just wait to defer the next page fault to avoid busy loop and
@@ -5022,7 +4893,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, vma, &address, ptep)) {
+ if (huge_pmd_unshare(mm, &address, ptep)) {
pages++;
spin_unlock(ptl);
shared_pmd = true;
@@ -5337,18 +5208,10 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
* Search for a shareable pmd page for hugetlb. In any case calls pmd_alloc()
* and returns the corresponding pte. While this is not necessary for the
* !shared pmd case because we can allocate the pmd later as well, it makes the
- * code much cleaner.
- *
- * This routine must be called with i_mmap_rwsem held in at least read mode if
- * sharing is possible. For hugetlbfs, this prevents removal of any page
- * table entries associated with the address space. This is important as we
- * are setting up sharing based on existing page table entries (mappings).
- *
- * NOTE: This routine is only called from huge_pte_alloc. Some callers of
- * huge_pte_alloc know that sharing is not possible and do not take
- * i_mmap_rwsem as a performance optimization. This is handled by the
- * if !vma_shareable check at the beginning of the routine. i_mmap_rwsem is
- * only required for subsequent processing.
+ * code much cleaner. pmd allocation is essential for the shared case because
+ * pud has to be populated inside the same i_mmap_rwsem section - otherwise
+ * racing tasks could either miss the sharing (see huge_pte_offset) or select a
+ * bad pmd for sharing.
*/
pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
{
@@ -5365,7 +5228,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
if (!vma_shareable(vma, addr))
return (pte_t *)pmd_alloc(mm, pud, addr);
- i_mmap_assert_locked(mapping);
+ i_mmap_lock_read(mapping);
vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
if (svma == vma)
continue;
@@ -5395,6 +5258,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
spin_unlock(ptl);
out:
pte = (pte_t *)pmd_alloc(mm, pud, addr);
+ i_mmap_unlock_read(mapping);
return pte;
}
@@ -5405,19 +5269,17 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
* indicated by page_count > 1, unmap is achieved by clearing pud and
* decrementing the ref count. If count == 1, the pte page is not shared.
*
- * Called with page table lock held and i_mmap_rwsem held in write mode.
+ * called with page table lock held.
*
* returns: 1 successfully unmapped a shared pte page
* 0 the underlying pte page is not shared, or it is the last user
*/
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
{
pgd_t *pgd = pgd_offset(mm, *addr);
p4d_t *p4d = p4d_offset(pgd, *addr);
pud_t *pud = pud_offset(p4d, *addr);
- i_mmap_assert_write_locked(vma->vm_file->f_mapping);
BUG_ON(page_count(virt_to_page(ptep)) == 0);
if (page_count(virt_to_page(ptep)) == 1)
return 0;
@@ -5435,8 +5297,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
return NULL;
}
-int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
- unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
{
return 0;
}
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index c0bb186bba62..0c4f8cbc772e 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -992,7 +992,7 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn,
enum ttu_flags ttu = TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS;
struct address_space *mapping;
LIST_HEAD(tokill);
- bool unmap_success = true;
+ bool unmap_success;
int kill = 1, forcekill;
struct page *hpage = *hpagep;
bool mlocked = PageMlocked(hpage);
@@ -1054,32 +1054,7 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn,
if (kill)
collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED);
- if (!PageHuge(hpage)) {
- unmap_success = try_to_unmap(hpage, ttu);
- } else {
- /*
- * For hugetlb pages, try_to_unmap could potentially call
- * huge_pmd_unshare. Because of this, take semaphore in
- * write mode here and set TTU_RMAP_LOCKED to indicate we
- * have taken the lock at this higer level.
- *
- * Note that the call to hugetlb_page_mapping_lock_write
- * is necessary even if mapping is already set. It handles
- * ugliness of potentially having to drop page lock to obtain
- * i_mmap_rwsem.
- */
- mapping = hugetlb_page_mapping_lock_write(hpage);
-
- if (mapping) {
- unmap_success = try_to_unmap(hpage,
- ttu|TTU_RMAP_LOCKED);
- i_mmap_unlock_write(mapping);
- } else {
- pr_info("Memory failure: %#lx: could not find mapping for mapped huge page\n",
- pfn);
- unmap_success = false;
- }
- }
+ unmap_success = try_to_unmap(hpage, ttu);
if (!unmap_success)
pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n",
pfn, page_mapcount(hpage));
diff --git a/mm/migrate.c b/mm/migrate.c
index 5ca5842df5db..05c3c2e569df 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1280,7 +1280,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
int page_was_mapped = 0;
struct page *new_hpage;
struct anon_vma *anon_vma = NULL;
- struct address_space *mapping = NULL;
/*
* Migratability of hugepages depends on architectures and their size.
@@ -1328,36 +1327,18 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
goto put_anon;
if (page_mapped(hpage)) {
- /*
- * try_to_unmap could potentially call huge_pmd_unshare.
- * Because of this, take semaphore in write mode here and
- * set TTU_RMAP_LOCKED to let lower levels know we have
- * taken the lock.
- */
- mapping = hugetlb_page_mapping_lock_write(hpage);
- if (unlikely(!mapping))
- goto unlock_put_anon;
-
try_to_unmap(hpage,
- TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS|
- TTU_RMAP_LOCKED);
+ TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
page_was_mapped = 1;
- /*
- * Leave mapping locked until after subsequent call to
- * remove_migration_ptes()
- */
}
if (!page_mapped(hpage))
rc = move_to_new_page(new_hpage, hpage, mode);
- if (page_was_mapped) {
+ if (page_was_mapped)
remove_migration_ptes(hpage,
- rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, true);
- i_mmap_unlock_write(mapping);
- }
+ rc == MIGRATEPAGE_SUCCESS ? new_hpage : hpage, false);
-unlock_put_anon:
unlock_page(new_hpage);
put_anon:
diff --git a/mm/rmap.c b/mm/rmap.c
index 1b84945d655c..8ef12940e357 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -22,7 +22,7 @@
*
* inode->i_mutex (while writing or truncating, not reading or faulting)
* mm->mmap_lock
- * page->flags PG_locked (lock_page) * (see huegtlbfs below)
+ * page->flags PG_locked (lock_page)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
* hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
@@ -44,11 +44,6 @@
* anon_vma->rwsem,mapping->i_mutex (memory_failure, collect_procs_anon)
* ->tasklist_lock
* pte map lock
- *
- * * hugetlbfs PageHuge() pages take locks in this order:
- * mapping->i_mmap_rwsem
- * hugetlb_fault_mutex (hugetlbfs specific page fault mutex)
- * page->flags PG_locked (lock_page)
*/
#include <linux/mm.h>
@@ -1413,9 +1408,6 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
/*
* If sharing is possible, start and end will be adjusted
* accordingly.
- *
- * If called for a huge page, caller must hold i_mmap_rwsem
- * in write mode as it is possible to call huge_pmd_unshare.
*/
adjust_range_if_pmd_sharing_possible(vma, &range.start,
&range.end);
@@ -1463,13 +1455,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
address = pvmw.address;
if (PageHuge(page)) {
- /*
- * To call huge_pmd_unshare, i_mmap_rwsem must be
- * held in write mode. Caller needs to explicitly
- * do this outside rmap routines.
- */
- VM_BUG_ON(!(flags & TTU_RMAP_LOCKED));
- if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
+ if (huge_pmd_unshare(mm, &address, pvmw.pte)) {
/*
* huge_pmd_unshare unmapped an entire PMD
* page. There is no way of knowing exactly
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 9a3d451402d7..ab9e0496d601 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -278,14 +278,10 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
BUG_ON(dst_addr >= dst_start + len);
/*
- * Serialize via i_mmap_rwsem and hugetlb_fault_mutex.
- * i_mmap_rwsem ensures the dst_pte remains valid even
- * in the case of shared pmds. fault mutex prevents
- * races with other faulting threads.
+ * Serialize via hugetlb_fault_mutex
*/
- mapping = dst_vma->vm_file->f_mapping;
- i_mmap_lock_read(mapping);
idx = linear_page_index(dst_vma, dst_addr);
+ mapping = dst_vma->vm_file->f_mapping;
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
@@ -293,7 +289,6 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_pte = huge_pte_alloc(dst_mm, dst_addr, vma_hpagesize);
if (!dst_pte) {
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- i_mmap_unlock_read(mapping);
goto out_unlock;
}
@@ -301,7 +296,6 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_pteval = huge_ptep_get(dst_pte);
if (!huge_pte_none(dst_pteval)) {
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- i_mmap_unlock_read(mapping);
goto out_unlock;
}
@@ -309,7 +303,6 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_addr, src_addr, &page);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
- i_mmap_unlock_read(mapping);
vm_alloc_shared = vm_shared;
cond_resched();
--
2.25.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization
2020-10-13 23:10 [RFC PATCH 0/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 1/3] hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync Mike Kravetz
@ 2020-10-13 23:10 ` Mike Kravetz
2020-10-15 23:05 ` HORIGUCHI NAOYA(堀口 直也)
2020-10-13 23:11 ` [RFC PATCH 3/3] huegtlbfs: handle page fault/truncate races Mike Kravetz
2 siblings, 1 reply; 6+ messages in thread
From: Mike Kravetz @ 2020-10-13 23:10 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Hugh Dickins, Michal Hocko, Naoya Horiguchi, Aneesh Kumar K . V,
Andrea Arcangeli, Kirill A . Shutemov, Davidlohr Bueso,
Prakash Sangappa, Andrew Morton, Mike Kravetz
Due to pmd sharing, the huge PTE pointer returned by huge_pte_alloc
may not be valid. This can happen if a call to huge_pmd_unshare for
the same pmd is made in another thread.
To address this issue, add a rw_semaphore (hinode_rwsem) to the hugetlbfs
inode.
- hinode_rwsem is taken in read mode before calling huge_pte_alloc, and
held until finished with the returned pte pointer.
- hinode_rwsem is held in write mode whenever huge_pmd_unshare is called.
In the locking hierarchy, hinode_rwsem must be taken before a page lock.
In an effort to minimize performance impacts, hinode_rwsem is not taken
if the caller knows the target can not possibly be part of a shared pmd.
lockdep_assert calls are added to huge_pmd_share and huge_pmd_unshare to
help catch callers not using the proper locking.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 8 ++++
include/linux/hugetlb.h | 66 +++++++++++++++++++++++++++++--
mm/hugetlb.c | 86 +++++++++++++++++++++++++++++------------
mm/memory-failure.c | 15 ++++++-
mm/migrate.c | 15 +++++++
mm/rmap.c | 3 +-
mm/userfaultfd.c | 9 ++++-
7 files changed, 171 insertions(+), 31 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 3a1246aeedc4..d6bb675d4872 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -499,13 +499,17 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
* Getting here in a truncate operation is a bug.
*/
if (unlikely(page_mapped(page))) {
+ struct hugetlbfs_inode_info *info =
+ HUGETLBFS_I(inode);
BUG_ON(truncate_op);
+ down_write(&info->hinode_rwsem);
i_mmap_lock_write(mapping);
hugetlb_vmdelete_list(&mapping->i_mmap,
index * pages_per_huge_page(h),
(index + 1) * pages_per_huge_page(h));
i_mmap_unlock_write(mapping);
+ up_write(&info->hinode_rwsem);
}
lock_page(page);
@@ -562,15 +566,18 @@ static int hugetlb_vmtruncate(struct inode *inode, loff_t offset)
pgoff_t pgoff;
struct address_space *mapping = inode->i_mapping;
struct hstate *h = hstate_inode(inode);
+ struct hugetlbfs_inode_info *info = HUGETLBFS_I(inode);
BUG_ON(offset & ~huge_page_mask(h));
pgoff = offset >> PAGE_SHIFT;
+ down_write(&info->hinode_rwsem);
i_size_write(inode, offset);
i_mmap_lock_write(mapping);
if (!RB_EMPTY_ROOT(&mapping->i_mmap.rb_root))
hugetlb_vmdelete_list(&mapping->i_mmap, pgoff, 0);
i_mmap_unlock_write(mapping);
+ up_write(&info->hinode_rwsem);
remove_inode_hugepages(inode, offset, LLONG_MAX);
return 0;
}
@@ -829,6 +836,7 @@ static struct inode *hugetlbfs_get_inode(struct super_block *sb,
inode->i_atime = inode->i_mtime = inode->i_ctime = current_time(inode);
inode->i_mapping->private_data = resv_map;
info->seals = F_SEAL_SEAL;
+ init_rwsem(&info->hinode_rwsem);
switch (mode & S_IFMT) {
default:
init_special_inode(inode, mode, dev);
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index bf79a5601091..10322fdcf3d9 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -163,7 +163,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
pte_t *huge_pte_offset(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
-int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep);
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep);
void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
unsigned long *start, unsigned long *end);
struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
@@ -196,8 +197,9 @@ static inline unsigned long hugetlb_total_pages(void)
return 0;
}
-static inline int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr,
- pte_t *ptep)
+static inline int huge_pmd_unshare(struct mm_struct *mm,
+ struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
return 0;
}
@@ -414,6 +416,7 @@ struct hugetlbfs_inode_info {
struct shared_policy policy;
struct inode vfs_inode;
unsigned int seals;
+ struct rw_semaphore hinode_rwsem;
};
static inline struct hugetlbfs_inode_info *HUGETLBFS_I(struct inode *inode)
@@ -439,6 +442,51 @@ static inline struct hstate *hstate_inode(struct inode *i)
{
return HUGETLBFS_SB(i->i_sb)->hstate;
}
+
+static inline bool vma_shareable(struct vm_area_struct *vma, unsigned long addr)
+{
+ unsigned long base = addr & PUD_MASK;
+ unsigned long end = base + PUD_SIZE;
+
+ /* check on proper vm_flags and page table alignment */
+ if (vma->vm_flags & VM_MAYSHARE && range_in_vma(vma, base, end))
+ return true;
+ return false;
+}
+
+static inline void hinode_lock_read(struct address_space *mapping,
+ struct vm_area_struct *vma,
+ unsigned long addr)
+{
+ if (!mapping)
+ return;
+ if (vma && !vma_shareable(vma, addr))
+ return;
+
+ down_read(&HUGETLBFS_I(mapping->host)->hinode_rwsem);
+}
+
+static inline void hinode_unlock_read(struct address_space *mapping,
+ struct vm_area_struct *vma,
+ unsigned long addr)
+{
+ if (!mapping)
+ return;
+ if (vma && !vma_shareable(vma, addr))
+ return;
+
+ up_read(&HUGETLBFS_I(mapping->host)->hinode_rwsem);
+}
+
+static inline void hinode_assert_locked(struct address_space *mapping)
+{
+ lockdep_assert_held(&HUGETLBFS_I(mapping->host)->hinode_rwsem);
+}
+
+static inline void hinode_assert_write_locked(struct address_space *mapping)
+{
+ lockdep_assert_held_write(&HUGETLBFS_I(mapping->host)->hinode_rwsem);
+}
#else /* !CONFIG_HUGETLBFS */
#define is_file_hugepages(file) false
@@ -913,6 +961,18 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr
pte_t *ptep, pte_t pte, unsigned long sz)
{
}
+
+static inline void hinode_lock_read(struct address_space *mapping,
+ struct vm_area_struct *vma,
+ unsigned long addr)
+{
+}
+
+static inline void hinode_unlock_read(struct address_space *mapping,
+ struct vm_area_struct *vma,
+ unsigned long addr)
+{
+}
#endif /* CONFIG_HUGETLB_PAGE */
static inline spinlock_t *huge_pte_lock(struct hstate *h,
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 9a316b6d0b51..940c037418f8 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -3728,6 +3728,18 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
vma->vm_start,
vma->vm_end);
mmu_notifier_invalidate_range_start(&range);
+ } else {
+ /*
+ * For shared mappings hinode_rwsem must be held to call
+ * huge_pte_alloc, otherwise the returned ptep could go
+ * away if part of a shared pmd and another thread calls
+ * huge_pmd_unshare.
+ *
+ * Note: As an optimization We could check at each PUD_SIZE
+ * boundary to see if sharing is possible before taking
+ * hinode_rwsem.
+ */
+ hinode_lock_read(vma->vm_file->f_mapping, NULL, 0UL);
}
for (addr = vma->vm_start; addr < vma->vm_end; addr += sz) {
@@ -3805,6 +3817,8 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
if (cow)
mmu_notifier_invalidate_range_end(&range);
+ else
+ hinode_unlock_read(vma->vm_file->f_mapping, NULL, 0UL);
return ret;
}
@@ -3848,7 +3862,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma,
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, &address, ptep)) {
spin_unlock(ptl);
/*
* We just unmapped a page of PMDs by clearing a PUD.
@@ -4253,13 +4267,17 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
};
/*
- * hugetlb_fault_mutex must be dropped before
- * handling userfault. Reacquire after handling
+ * hugetlb_fault_mutex and inode mutex must be dropped
+ * before handling userfault. Reacquire after handling
* fault to make calling code simpler.
*/
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ hinode_unlock_read(mapping, vma, address);
+
ret = handle_userfault(&vmf, VM_UFFD_MISSING);
+
+ hinode_lock_read(mapping, vma, address);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
goto out;
}
@@ -4424,6 +4442,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
if (ptep) {
+ /*
+ * Since we hold no locks, ptep could be stale. That is
+ * OK as we are only making decisions based on content and
+ * not actually modifying content here.
+ */
entry = huge_ptep_get(ptep);
if (unlikely(is_hugetlb_entry_migration(entry))) {
migration_entry_wait_huge(vma, mm, ptep);
@@ -4431,20 +4454,32 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
} else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
return VM_FAULT_HWPOISON_LARGE |
VM_FAULT_SET_HINDEX(hstate_index(h));
- } else {
- ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
- if (!ptep)
- return VM_FAULT_OOM;
}
+ /*
+ * Acquire hinode_sema before calling huge_pte_alloc and hold
+ * until finished with ptep. This prevents huge_pmd_unshare from
+ * being called elsewhere and making the ptep no longer valid.
+ *
+ * ptep could have already be assigned via huge_pte_offset. That
+ * is OK, as huge_pte_alloc will return the same value unless
+ * something has changed.
+ */
mapping = vma->vm_file->f_mapping;
- idx = vma_hugecache_offset(h, vma, haddr);
+ hinode_lock_read(mapping, vma, address);
+ ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
+ if (!ptep) {
+ ret = VM_FAULT_OOM;
+ goto out_mutex;
+ }
+
/*
* Serialize hugepage allocation and instantiation, so that we don't
* get spurious allocation failures if two CPUs race to instantiate
* the same page in the page cache.
*/
+ idx = vma_hugecache_offset(h, vma, haddr);
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
@@ -4532,6 +4567,7 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
}
out_mutex:
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ hinode_unlock_read(mapping, vma, address);
/*
* Generally it's safe to hold refcount during waiting page lock. But
* here we just wait to defer the next page fault to avoid busy loop and
@@ -4872,6 +4908,8 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
unsigned long pages = 0;
bool shared_pmd = false;
struct mmu_notifier_range range;
+ struct hugetlbfs_inode_info *hinfo =
+ HUGETLBFS_I(vma->vm_file->f_inode);
/*
* In the case of shared PMDs, the area to flush could be beyond
@@ -4886,6 +4924,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
flush_cache_range(vma, range.start, range.end);
mmu_notifier_invalidate_range_start(&range);
+ down_write(&hinfo->hinode_rwsem);
i_mmap_lock_write(vma->vm_file->f_mapping);
for (; address < end; address += huge_page_size(h)) {
spinlock_t *ptl;
@@ -4893,7 +4932,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
if (!ptep)
continue;
ptl = huge_pte_lock(h, mm, ptep);
- if (huge_pmd_unshare(mm, &address, ptep)) {
+ if (huge_pmd_unshare(mm, vma, &address, ptep)) {
pages++;
spin_unlock(ptl);
shared_pmd = true;
@@ -4948,6 +4987,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma,
* See Documentation/vm/mmu_notifier.rst
*/
i_mmap_unlock_write(vma->vm_file->f_mapping);
+ up_write(&hinfo->hinode_rwsem);
mmu_notifier_invalidate_range_end(&range);
return pages << h->order;
@@ -5166,19 +5206,6 @@ static unsigned long page_table_shareable(struct vm_area_struct *svma,
return saddr;
}
-static bool vma_shareable(struct vm_area_struct *vma, unsigned long addr)
-{
- unsigned long base = addr & PUD_MASK;
- unsigned long end = base + PUD_SIZE;
-
- /*
- * check on proper vm_flags and page table alignment
- */
- if (vma->vm_flags & VM_MAYSHARE && range_in_vma(vma, base, end))
- return true;
- return false;
-}
-
/*
* Determine if start,end range within vma could be mapped by shared pmd.
* If yes, adjust start and end to cover range associated with possible
@@ -5212,6 +5239,10 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
* pud has to be populated inside the same i_mmap_rwsem section - otherwise
* racing tasks could either miss the sharing (see huge_pte_offset) or select a
* bad pmd for sharing.
+ *
+ * This should be called with hinode_rwsem held in read mode. Otherwise, it
+ * could race with huge_pmd_unshare and the pte_t pointer could become invalid
+ * before being returned to the caller.
*/
pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
{
@@ -5227,6 +5258,7 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
if (!vma_shareable(vma, addr))
return (pte_t *)pmd_alloc(mm, pud, addr);
+ hinode_assert_locked(mapping);
i_mmap_lock_read(mapping);
vma_interval_tree_foreach(svma, &mapping->i_mmap, idx, idx) {
@@ -5269,12 +5301,13 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
* indicated by page_count > 1, unmap is achieved by clearing pud and
* decrementing the ref count. If count == 1, the pte page is not shared.
*
- * called with page table lock held.
+ * Called with page table lock held and hinode_rwsem held in write mode.
*
* returns: 1 successfully unmapped a shared pte page
* 0 the underlying pte page is not shared, or it is the last user
*/
-int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
pgd_t *pgd = pgd_offset(mm, *addr);
p4d_t *p4d = p4d_offset(pgd, *addr);
@@ -5284,6 +5317,8 @@ int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
if (page_count(virt_to_page(ptep)) == 1)
return 0;
+ hinode_assert_write_locked(vma->vm_file->f_mapping);
+
pud_clear(pud);
put_page(virt_to_page(ptep));
mm_dec_nr_pmds(mm);
@@ -5297,7 +5332,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, unsigned long addr, pud_t *pud)
return NULL;
}
-int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
+int huge_pmd_unshare(struct mm_struct *mm, struct vm_area_struct *vma,
+ unsigned long *addr, pte_t *ptep)
{
return 0;
}
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 0c4f8cbc772e..fd6ee4d941a0 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -1054,7 +1054,20 @@ static bool hwpoison_user_mappings(struct page *p, unsigned long pfn,
if (kill)
collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED);
- unmap_success = try_to_unmap(hpage, ttu);
+ if (PageHuge(hpage) && !PageAnon(hpage) && mapping) {
+ struct hugetlbfs_inode_info *hinode_info =
+ HUGETLBFS_I(mapping->host);
+ /*
+ * For hugetlb pages, try_to_unmap could potentially call
+ * huge_pmd_unshare. Because of this, take hinode_rwsem
+ * in write mode before calling.
+ */
+ down_write(&hinode_info->hinode_rwsem);
+ unmap_success = try_to_unmap(hpage, ttu);
+ up_write(&hinode_info->hinode_rwsem);
+ } else {
+ unmap_success = try_to_unmap(hpage, ttu);
+ }
if (!unmap_success)
pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n",
pfn, page_mapcount(hpage));
diff --git a/mm/migrate.c b/mm/migrate.c
index 05c3c2e569df..f068727293c2 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1327,9 +1327,24 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
goto put_anon;
if (page_mapped(hpage)) {
+ struct address_space *mapping = NULL;
+ struct hugetlbfs_inode_info *hinode_info;
+ /*
+ * try_to_unmap could potentially call huge_pmd_unshare.
+ * Take hinode_rwsem if sharing is possible.
+ */
+ if (!PageAnon(hpage)) {
+ mapping = page_mapping(hpage);
+ if (mapping) {
+ hinode_info = HUGETLBFS_I(mapping->host);
+ down_write(&hinode_info->hinode_rwsem);
+ }
+ }
try_to_unmap(hpage,
TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
page_was_mapped = 1;
+ if (mapping)
+ up_write(&hinode_info->hinode_rwsem);
}
if (!page_mapped(hpage))
diff --git a/mm/rmap.c b/mm/rmap.c
index 8ef12940e357..9e65c9c95449 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -22,6 +22,7 @@
*
* inode->i_mutex (while writing or truncating, not reading or faulting)
* mm->mmap_lock
+ * hugetlbfs inode->hinode_rwsem (hugetlbfs specific)
* page->flags PG_locked (lock_page)
* hugetlbfs_i_mmap_rwsem_key (in huge_pmd_share)
* mapping->i_mmap_rwsem
@@ -1455,7 +1456,7 @@ static bool try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
address = pvmw.address;
if (PageHuge(page)) {
- if (huge_pmd_unshare(mm, &address, pvmw.pte)) {
+ if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) {
/*
* huge_pmd_unshare unmapped an entire PMD
* page. There is no way of knowing exactly
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index ab9e0496d601..3ca789934f6f 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -278,10 +278,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
BUG_ON(dst_addr >= dst_start + len);
/*
- * Serialize via hugetlb_fault_mutex
+ * Serialize via hinode_rwsem hugetlb_fault_mutex.
+ * hinode_rwsem ensures the dst_pte remains valid even
+ * in the case of shared pmds. fault mutex prevents
+ * races with other faulting threads.
*/
idx = linear_page_index(dst_vma, dst_addr);
mapping = dst_vma->vm_file->f_mapping;
+ hinode_lock_read(mapping, dst_vma, dst_addr);
hash = hugetlb_fault_mutex_hash(mapping, idx);
mutex_lock(&hugetlb_fault_mutex_table[hash]);
@@ -289,6 +293,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_pte = huge_pte_alloc(dst_mm, dst_addr, vma_hpagesize);
if (!dst_pte) {
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ hinode_unlock_read(mapping, dst_vma, dst_addr);
goto out_unlock;
}
@@ -296,6 +301,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_pteval = huge_ptep_get(dst_pte);
if (!huge_pte_none(dst_pteval)) {
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ hinode_unlock_read(mapping, dst_vma, dst_addr);
goto out_unlock;
}
@@ -303,6 +309,7 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
dst_addr, src_addr, &page);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
+ hinode_unlock_read(mapping, dst_vma, dst_addr);
vm_alloc_shared = vm_shared;
cond_resched();
--
2.25.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [RFC PATCH 3/3] huegtlbfs: handle page fault/truncate races
2020-10-13 23:10 [RFC PATCH 0/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 1/3] hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
@ 2020-10-13 23:11 ` Mike Kravetz
2 siblings, 0 replies; 6+ messages in thread
From: Mike Kravetz @ 2020-10-13 23:11 UTC (permalink / raw)
To: linux-mm, linux-kernel
Cc: Hugh Dickins, Michal Hocko, Naoya Horiguchi, Aneesh Kumar K . V,
Andrea Arcangeli, Kirill A . Shutemov, Davidlohr Bueso,
Prakash Sangappa, Andrew Morton, Mike Kravetz
A huegtlb page fault can race with page truncation. Make the code
identifying and handling these races more robust.
Page fault handling needs to back out pages added to page cache beyond
file size (i_size). When backing out the page, take care to restore
reserve map entries and counts as necessary.
File truncation (remove_inode_hugepages) needs to handle page mapping
changes that could have happened before locking the page. This could
happen if page was added to page cache and later backed out in fault
processing.
Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
---
fs/hugetlbfs/inode.c | 34 ++++++++++++++++++++--------------
mm/hugetlb.c | 40 ++++++++++++++++++++++++++++++++++++++--
2 files changed, 58 insertions(+), 16 deletions(-)
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index d6bb675d4872..f6ca2892e833 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -514,23 +514,29 @@ static void remove_inode_hugepages(struct inode *inode, loff_t lstart,
lock_page(page);
/*
- * We must free the huge page and remove from page
- * cache (remove_huge_page) BEFORE removing the
- * region/reserve map (hugetlb_unreserve_pages). In
- * rare out of memory conditions, removal of the
- * region/reserve map could fail. Correspondingly,
- * the subpool and global reserve usage count can need
- * to be adjusted.
+ * After locking page, make sure mapping is the same.
+ * We could have raced with page fault populate and
+ * backout code.
*/
- VM_BUG_ON(PagePrivate(page));
- remove_huge_page(page);
- freed++;
- if (!truncate_op) {
- if (unlikely(hugetlb_unreserve_pages(inode,
+ if (page_mapping(page) == mapping) {
+ /*
+ * We must free the huge page and remove from
+ * page cache (remove_huge_page) BEFORE
+ * removing the region/reserve map. In rare
+ * out of memory conditions, removal of the
+ * region/reserve map could fail and the
+ * subpool and global reserve usage count
+ * will need to be adjusted.
+ */
+ VM_BUG_ON(PagePrivate(page));
+ remove_huge_page(page);
+ freed++;
+ if (!truncate_op) {
+ if (unlikely(hugetlb_unreserve_pages(inode,
index, index + 1, 1)))
- hugetlb_fix_reserve_counts(inode);
+ hugetlb_fix_reserve_counts(inode);
+ }
}
-
unlock_page(page);
mutex_unlock(&hugetlb_fault_mutex_table[hash]);
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 940c037418f8..e0ba58385036 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -4225,6 +4225,9 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
spinlock_t *ptl;
unsigned long haddr = address & huge_page_mask(h);
bool new_page = false;
+ bool page_cache = false;
+ bool reserve_alloc = false;
+ bool beyond_i_size = false;
/*
* Currently, we are forced to kill the process in the event the
@@ -4309,6 +4312,8 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
clear_huge_page(page, address, pages_per_huge_page(h));
__SetPageUptodate(page);
new_page = true;
+ if (PagePrivate(page))
+ reserve_alloc = true;
if (vma->vm_flags & VM_MAYSHARE) {
int err = huge_add_to_page_cache(page, mapping, idx);
@@ -4318,6 +4323,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
goto retry;
goto out;
}
+ page_cache = true;
} else {
lock_page(page);
if (unlikely(anon_vma_prepare(vma))) {
@@ -4356,8 +4362,10 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
ptl = huge_pte_lock(h, mm, ptep);
size = i_size_read(mapping->host) >> huge_page_shift(h);
- if (idx >= size)
+ if (idx >= size) {
+ beyond_i_size = true;
goto backout;
+ }
ret = 0;
if (!huge_pte_none(huge_ptep_get(ptep)))
@@ -4395,8 +4403,36 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm,
backout:
spin_unlock(ptl);
backout_unlocked:
+ if (new_page) {
+ if (page_cache && beyond_i_size) {
+ /*
+ * Back out pages added to page cache beyond i_size.
+ * Otherwise, they will 'sit' there until the file
+ * is removed.
+ */
+ ClearPageDirty(page);
+ ClearPageUptodate(page);
+ delete_from_page_cache(page);
+ }
+
+ if (reserve_alloc) {
+ /*
+ * If reserve was consumed, set PagePrivate so that
+ * it will be restored in free_huge_page().
+ */
+ SetPagePrivate(page);
+ }
+
+ if (!beyond_i_size) {
+ /*
+ * Do not restore reserve map entries beyond i_size.
+ * there will be leaks when the file is removed.
+ */
+ restore_reserve_on_error(h, vma, haddr, page);
+ }
+
+ }
unlock_page(page);
- restore_reserve_on_error(h, vma, haddr, page);
put_page(page);
goto out;
}
--
2.25.4
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization
2020-10-13 23:10 ` [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
@ 2020-10-15 23:05 ` HORIGUCHI NAOYA(堀口 直也)
2020-10-16 3:59 ` Mike Kravetz
0 siblings, 1 reply; 6+ messages in thread
From: HORIGUCHI NAOYA(堀口 直也) @ 2020-10-15 23:05 UTC (permalink / raw)
To: Mike Kravetz
Cc: linux-mm, linux-kernel, Hugh Dickins, Michal Hocko,
Naoya Horiguchi, Aneesh Kumar K . V, Andrea Arcangeli,
Kirill A . Shutemov, Davidlohr Bueso, Prakash Sangappa,
Andrew Morton
On Tue, Oct 13, 2020 at 04:10:59PM -0700, Mike Kravetz wrote:
> Due to pmd sharing, the huge PTE pointer returned by huge_pte_alloc
> may not be valid. This can happen if a call to huge_pmd_unshare for
> the same pmd is made in another thread.
>
> To address this issue, add a rw_semaphore (hinode_rwsem) to the hugetlbfs
> inode.
> - hinode_rwsem is taken in read mode before calling huge_pte_alloc, and
> held until finished with the returned pte pointer.
> - hinode_rwsem is held in write mode whenever huge_pmd_unshare is called.
>
> In the locking hierarchy, hinode_rwsem must be taken before a page lock.
>
> In an effort to minimize performance impacts, hinode_rwsem is not taken
> if the caller knows the target can not possibly be part of a shared pmd.
> lockdep_assert calls are added to huge_pmd_share and huge_pmd_unshare to
> help catch callers not using the proper locking.
>
> Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
Hi Mike,
I didn't find a problem on main idea of introducing hinode_rwsem, so
I'm fine if the known problems are fixed.
I have one question. This patch seems to make sure that huge_pmd_unshare()
are called under holding hinode_rwsem in write mode for some case. Some
callers of try_to_unmap() seem not to hold it like shrink_page_list(),
unmap_page(), which is OK because they never call try_to_unmap() for hugetlb
pages. And unmap_ref_private() doesn't takes hinode_rwsem either, and
that's also OK because this function never handles pmd sharing case. So
what about unmap_single_vma()? It seems that this generic function could
reach huge_pmd_unshare() without hinode_rwsem, so what prevents the race here?
(Maybe I might miss some assumption or condition over this race...)
I left a few other minor comments below ...
> @@ -4424,6 +4442,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
>
> ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
> if (ptep) {
> + /*
> + * Since we hold no locks, ptep could be stale. That is
> + * OK as we are only making decisions based on content and
> + * not actually modifying content here.
> + */
nice comment, thank you.
> entry = huge_ptep_get(ptep);
> if (unlikely(is_hugetlb_entry_migration(entry))) {
> migration_entry_wait_huge(vma, mm, ptep);
> @@ -4431,20 +4454,32 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
> } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
> return VM_FAULT_HWPOISON_LARGE |
> VM_FAULT_SET_HINDEX(hstate_index(h));
> - } else {
> - ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
> - if (!ptep)
> - return VM_FAULT_OOM;
> }
>
> + /*
> + * Acquire hinode_sem before calling huge_pte_alloc and hold
hinode_rwsem?
> + * until finished with ptep. This prevents huge_pmd_unshare from
> + * being called elsewhere and making the ptep no longer valid.
> + *
> + * ptep could have already be assigned via huge_pte_offset. That
> + * is OK, as huge_pte_alloc will return the same value unless
> + * something has changed.
> + */
...
> @@ -278,10 +278,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
> BUG_ON(dst_addr >= dst_start + len);
>
> /*
> - * Serialize via hugetlb_fault_mutex
> + * Serialize via hinode_rwsem hugetlb_fault_mutex.
^ "and" here?
> + * hinode_rwsem ensures the dst_pte remains valid even
> + * in the case of shared pmds. fault mutex prevents
> + * races with other faulting threads.
> */
> idx = linear_page_index(dst_vma, dst_addr);
> mapping = dst_vma->vm_file->f_mapping;
> + hinode_lock_read(mapping, dst_vma, dst_addr);
> hash = hugetlb_fault_mutex_hash(mapping, idx);
> mutex_lock(&hugetlb_fault_mutex_table[hash]);
Thanks,
Naoya Horiguchi
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization
2020-10-15 23:05 ` HORIGUCHI NAOYA(堀口 直也)
@ 2020-10-16 3:59 ` Mike Kravetz
0 siblings, 0 replies; 6+ messages in thread
From: Mike Kravetz @ 2020-10-16 3:59 UTC (permalink / raw)
To: HORIGUCHI NAOYA (堀口 直也)
Cc: linux-mm, linux-kernel, Hugh Dickins, Michal Hocko,
Naoya Horiguchi, Aneesh Kumar K.V, Andrea Arcangeli,
Kirill A.Shutemov, Davidlohr Bueso, Prakash Sangappa,
Andrew Morton
On 10/15/20 4:05 PM, HORIGUCHI NAOYA(堀口 直也) wrote:
> On Tue, Oct 13, 2020 at 04:10:59PM -0700, Mike Kravetz wrote:
>> Due to pmd sharing, the huge PTE pointer returned by huge_pte_alloc
>> may not be valid. This can happen if a call to huge_pmd_unshare for
>> the same pmd is made in another thread.
>>
>> To address this issue, add a rw_semaphore (hinode_rwsem) to the hugetlbfs
>> inode.
>> - hinode_rwsem is taken in read mode before calling huge_pte_alloc, and
>> held until finished with the returned pte pointer.
>> - hinode_rwsem is held in write mode whenever huge_pmd_unshare is called.
>>
>> In the locking hierarchy, hinode_rwsem must be taken before a page lock.
>>
>> In an effort to minimize performance impacts, hinode_rwsem is not taken
>> if the caller knows the target can not possibly be part of a shared pmd.
>> lockdep_assert calls are added to huge_pmd_share and huge_pmd_unshare to
>> help catch callers not using the proper locking.
>>
>> Signed-off-by: Mike Kravetz <mike.kravetz@oracle.com>
>
> Hi Mike,
>
> I didn't find a problem on main idea of introducing hinode_rwsem, so
> I'm fine if the known problems are fixed.
Thank you for taking a look Naoya!
I have been trying to address these race issues for some time. The issues
have been there since the pmd sharing code was introduced. Fortunately,
it is not easy to hit the issue. However, targeted test programs can cause
BUGs.
> I have one question. This patch seems to make sure that huge_pmd_unshare()
> are called under holding hinode_rwsem in write mode for some case. Some
> callers of try_to_unmap() seem not to hold it like shrink_page_list(),
> unmap_page(), which is OK because they never call try_to_unmap() for hugetlb
> pages. And unmap_ref_private() doesn't takes hinode_rwsem either, and
> that's also OK because this function never handles pmd sharing case. So
> what about unmap_single_vma()? It seems that this generic function could
> reach huge_pmd_unshare() without hinode_rwsem, so what prevents the race here?
> (Maybe I might miss some assumption or condition over this race...)
You are not missing anything. I mistakingly left out the locking code in
of unmap_single_vma(). If I would have run some tests with lockdep enabled,
the new lock checking code would have noticed.
>
> I left a few other minor comments below ...
I will address the below issues in the next revision.
Thanks again for taking a look.
--
Mike Kravetz
>
>> @@ -4424,6 +4442,11 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
>>
>> ptep = huge_pte_offset(mm, haddr, huge_page_size(h));
>> if (ptep) {
>> + /*
>> + * Since we hold no locks, ptep could be stale. That is
>> + * OK as we are only making decisions based on content and
>> + * not actually modifying content here.
>> + */
>
> nice comment, thank you.
>
>> entry = huge_ptep_get(ptep);
>> if (unlikely(is_hugetlb_entry_migration(entry))) {
>> migration_entry_wait_huge(vma, mm, ptep);
>> @@ -4431,20 +4454,32 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
>> } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry)))
>> return VM_FAULT_HWPOISON_LARGE |
>> VM_FAULT_SET_HINDEX(hstate_index(h));
>> - } else {
>> - ptep = huge_pte_alloc(mm, haddr, huge_page_size(h));
>> - if (!ptep)
>> - return VM_FAULT_OOM;
>> }
>>
>> + /*
>> + * Acquire hinode_sem before calling huge_pte_alloc and hold
>
> hinode_rwsem?
>
>> + * until finished with ptep. This prevents huge_pmd_unshare from
>> + * being called elsewhere and making the ptep no longer valid.
>> + *
>> + * ptep could have already be assigned via huge_pte_offset. That
>> + * is OK, as huge_pte_alloc will return the same value unless
>> + * something has changed.
>> + */
>
> ...
>
>> @@ -278,10 +278,14 @@ static __always_inline ssize_t __mcopy_atomic_hugetlb(struct mm_struct *dst_mm,
>> BUG_ON(dst_addr >= dst_start + len);
>>
>> /*
>> - * Serialize via hugetlb_fault_mutex
>> + * Serialize via hinode_rwsem hugetlb_fault_mutex.
> ^ "and" here?
>
>> + * hinode_rwsem ensures the dst_pte remains valid even
>> + * in the case of shared pmds. fault mutex prevents
>> + * races with other faulting threads.
>> */
>> idx = linear_page_index(dst_vma, dst_addr);
>> mapping = dst_vma->vm_file->f_mapping;
>> + hinode_lock_read(mapping, dst_vma, dst_addr);
>> hash = hugetlb_fault_mutex_hash(mapping, idx);
>> mutex_lock(&hugetlb_fault_mutex_table[hash]);
>
>
> Thanks,
> Naoya Horiguchi
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-10-16 3:59 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-13 23:10 [RFC PATCH 0/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 1/3] hugetlbfs: revert use of i_mmap_rwsem for pmd sharing and more sync Mike Kravetz
2020-10-13 23:10 ` [RFC PATCH 2/3] hugetlbfs: introduce hinode_rwsem for pmd sharing synchronization Mike Kravetz
2020-10-15 23:05 ` HORIGUCHI NAOYA(堀口 直也)
2020-10-16 3:59 ` Mike Kravetz
2020-10-13 23:11 ` [RFC PATCH 3/3] huegtlbfs: handle page fault/truncate races Mike Kravetz
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).