* [PATCH 0/21 v4 RESEND] dax: Clear dirty bits after flushing caches
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Hello,
I'm resending v4 of my DAX write-protection patches since I've messed up
previous posting, mixing old and new version of the patch series...
This is the fourth revision of my patches to clear dirty bits from radix tree
of DAX inodes when caches for corresponding pfns have been flushed. This patch
set is significantly larger than the previous version because I'm changing how
->fault, ->page_mkwrite, and ->pfn_mkwrite handlers may choose to handle the
fault so that we don't have to leak details about DAX locking into the generic
code. In principle, these patches enable handlers to easily update PTEs and do
other work necessary to finish the fault without duplicating the functionality
present in the generic code. I'd be really like feedback from mm folks whether
such changes to fault handling code are fine or what they'd do differently.
The patches are based on 4.9-rc1 + Ross' DAX PMD page fault series [1] + ext4
conversion of DAX IO patch to the iomap infrastructure [2]. For testing,
I've pushed out a tree including all these patches and further DAX fixes
to:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git dax
The patches pass testing with xfstests on ext4 and xfs on my end. I'd be
grateful for review so that we can push these patches for the next merge
window.
[1] http://www.spinics.net/lists/linux-mm/msg115247.html
[2] Posted an hour ago - look for "ext4: Convert ext4 DAX IO to iomap framework"
Changes since v3:
* rebased on top of 4.9-rc1 + DAX PMD fault series + ext4 iomap conversion
* reordered some of the patches
* killed ->virtual_address field in vm_fault structure as requested by
Christoph
Changes since v2:
* rebased on top of 4.8-rc8 - this involved dealing with new fault_env
structure
* changed calling convention for fault helpers
Changes since v1:
* make sure all PTE updates happen under radix tree entry lock to protect
against races between faults & write-protecting code
* remove information about DAX locking from mm/memory.c
* smaller updates based on Ross' feedback
----
Background information regarding the motivation:
Currently we never clear dirty bits in the radix tree of a DAX inode. Thus
fsync(2) flushes all the dirty pfns again and again. This patches implement
clearing of the dirty tag in the radix tree so that we issue flush only when
needed.
The difficulty with clearing the dirty tag is that we have to protect against
a concurrent page fault setting the dirty tag and writing new data into the
page. So we need a lock serializing page fault and clearing of the dirty tag
and write-protecting PTEs (so that we get another pagefault when pfn is written
to again and we have to set the dirty tag again).
The effect of the patch set is easily visible:
Writing 1 GB of data via mmap, then fsync twice.
Before this patch set both fsyncs take ~205 ms on my test machine, after the
patch set the first fsync takes ~283 ms (the additional cost of walking PTEs,
clearing dirty bits etc. is very noticeable), the second fsync takes below
1 us.
As a bonus, these patches make filesystem freezing for DAX filesystems
reliable because mappings are now properly writeprotected while freezing the
fs.
Honza
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH 0/21 v4 RESEND] dax: Clear dirty bits after flushing caches
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Hello,
I'm resending v4 of my DAX write-protection patches since I've messed up
previous posting, mixing old and new version of the patch series...
This is the fourth revision of my patches to clear dirty bits from radix tree
of DAX inodes when caches for corresponding pfns have been flushed. This patch
set is significantly larger than the previous version because I'm changing how
->fault, ->page_mkwrite, and ->pfn_mkwrite handlers may choose to handle the
fault so that we don't have to leak details about DAX locking into the generic
code. In principle, these patches enable handlers to easily update PTEs and do
other work necessary to finish the fault without duplicating the functionality
present in the generic code. I'd be really like feedback from mm folks whether
such changes to fault handling code are fine or what they'd do differently.
The patches are based on 4.9-rc1 + Ross' DAX PMD page fault series [1] + ext4
conversion of DAX IO patch to the iomap infrastructure [2]. For testing,
I've pushed out a tree including all these patches and further DAX fixes
to:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git dax
The patches pass testing with xfstests on ext4 and xfs on my end. I'd be
grateful for review so that we can push these patches for the next merge
window.
[1] http://www.spinics.net/lists/linux-mm/msg115247.html
[2] Posted an hour ago - look for "ext4: Convert ext4 DAX IO to iomap framework"
Changes since v3:
* rebased on top of 4.9-rc1 + DAX PMD fault series + ext4 iomap conversion
* reordered some of the patches
* killed ->virtual_address field in vm_fault structure as requested by
Christoph
Changes since v2:
* rebased on top of 4.8-rc8 - this involved dealing with new fault_env
structure
* changed calling convention for fault helpers
Changes since v1:
* make sure all PTE updates happen under radix tree entry lock to protect
against races between faults & write-protecting code
* remove information about DAX locking from mm/memory.c
* smaller updates based on Ross' feedback
----
Background information regarding the motivation:
Currently we never clear dirty bits in the radix tree of a DAX inode. Thus
fsync(2) flushes all the dirty pfns again and again. This patches implement
clearing of the dirty tag in the radix tree so that we issue flush only when
needed.
The difficulty with clearing the dirty tag is that we have to protect against
a concurrent page fault setting the dirty tag and writing new data into the
page. So we need a lock serializing page fault and clearing of the dirty tag
and write-protecting PTEs (so that we get another pagefault when pfn is written
to again and we have to set the dirty tag again).
The effect of the patch set is easily visible:
Writing 1 GB of data via mmap, then fsync twice.
Before this patch set both fsyncs take ~205 ms on my test machine, after the
patch set the first fsync takes ~283 ms (the additional cost of walking PTEs,
clearing dirty bits etc. is very noticeable), the second fsync takes below
1 us.
As a bonus, these patches make filesystem freezing for DAX filesystems
reliable because mappings are now properly writeprotected while freezing the
fs.
Honza
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH 0/21 v4 RESEND] dax: Clear dirty bits after flushing caches
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Hello,
I'm resending v4 of my DAX write-protection patches since I've messed up
previous posting, mixing old and new version of the patch series...
This is the fourth revision of my patches to clear dirty bits from radix tree
of DAX inodes when caches for corresponding pfns have been flushed. This patch
set is significantly larger than the previous version because I'm changing how
->fault, ->page_mkwrite, and ->pfn_mkwrite handlers may choose to handle the
fault so that we don't have to leak details about DAX locking into the generic
code. In principle, these patches enable handlers to easily update PTEs and do
other work necessary to finish the fault without duplicating the functionality
present in the generic code. I'd be really like feedback from mm folks whether
such changes to fault handling code are fine or what they'd do differently.
The patches are based on 4.9-rc1 + Ross' DAX PMD page fault series [1] + ext4
conversion of DAX IO patch to the iomap infrastructure [2]. For testing,
I've pushed out a tree including all these patches and further DAX fixes
to:
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git dax
The patches pass testing with xfstests on ext4 and xfs on my end. I'd be
grateful for review so that we can push these patches for the next merge
window.
[1] http://www.spinics.net/lists/linux-mm/msg115247.html
[2] Posted an hour ago - look for "ext4: Convert ext4 DAX IO to iomap framework"
Changes since v3:
* rebased on top of 4.9-rc1 + DAX PMD fault series + ext4 iomap conversion
* reordered some of the patches
* killed ->virtual_address field in vm_fault structure as requested by
Christoph
Changes since v2:
* rebased on top of 4.8-rc8 - this involved dealing with new fault_env
structure
* changed calling convention for fault helpers
Changes since v1:
* make sure all PTE updates happen under radix tree entry lock to protect
against races between faults & write-protecting code
* remove information about DAX locking from mm/memory.c
* smaller updates based on Ross' feedback
----
Background information regarding the motivation:
Currently we never clear dirty bits in the radix tree of a DAX inode. Thus
fsync(2) flushes all the dirty pfns again and again. This patches implement
clearing of the dirty tag in the radix tree so that we issue flush only when
needed.
The difficulty with clearing the dirty tag is that we have to protect against
a concurrent page fault setting the dirty tag and writing new data into the
page. So we need a lock serializing page fault and clearing of the dirty tag
and write-protecting PTEs (so that we get another pagefault when pfn is written
to again and we have to set the dirty tag again).
The effect of the patch set is easily visible:
Writing 1 GB of data via mmap, then fsync twice.
Before this patch set both fsyncs take ~205 ms on my test machine, after the
patch set the first fsync takes ~283 ms (the additional cost of walking PTEs,
clearing dirty bits etc. is very noticeable), the second fsync takes below
1 us.
As a bonus, these patches make filesystem freezing for DAX filesystems
reliable because mappings are now properly writeprotected while freezing the
fs.
Honza
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:24 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we have two different structures for passing fault information
around - struct vm_fault and struct fault_env. DAX will need more
information in struct vm_fault to handle its faults so the content of
that structure would become event closer to fault_env. Furthermore it
would need to generate struct fault_env to be able to call some of the
generic functions. So at this point I don't think there's much use in
keeping these two structures separate. Just embed into struct vm_fault
all that is needed to use it for both purposes.
Signed-off-by: Jan Kara <jack@suse.cz>
---
Documentation/filesystems/Locking | 2 +-
fs/userfaultfd.c | 22 +-
include/linux/huge_mm.h | 10 +-
include/linux/mm.h | 28 +-
include/linux/userfaultfd_k.h | 4 +-
mm/filemap.c | 14 +-
mm/huge_memory.c | 173 ++++++------
mm/internal.h | 2 +-
mm/khugepaged.c | 20 +-
mm/memory.c | 549 +++++++++++++++++++-------------------
mm/nommu.c | 2 +-
11 files changed, 414 insertions(+), 412 deletions(-)
diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 14cdc101d165..ac3d080eabaa 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -557,7 +557,7 @@ till "end_pgoff". ->map_pages() is called with page table locked and must
not block. If it's not possible to reach a page without blocking,
filesystem should skip it. Filesystem should use do_set_pte() to setup
page table entry. Pointer to entry associated with the page is passed in
-"pte" field in fault_env structure. Pointers to entries for other offsets
+"pte" field in vm_fault structure. Pointers to entries for other offsets
should be calculated relative to "pte".
->page_mkwrite() is called when a previously read-only pte is
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 85959d8324df..d96e2f30084b 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -257,9 +257,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
* fatal_signal_pending()s, and the mmap_sem must be released before
* returning it.
*/
-int handle_userfault(struct fault_env *fe, unsigned long reason)
+int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
- struct mm_struct *mm = fe->vma->vm_mm;
+ struct mm_struct *mm = vmf->vma->vm_mm;
struct userfaultfd_ctx *ctx;
struct userfaultfd_wait_queue uwq;
int ret;
@@ -268,7 +268,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
ret = VM_FAULT_SIGBUS;
- ctx = fe->vma->vm_userfaultfd_ctx.ctx;
+ ctx = vmf->vma->vm_userfaultfd_ctx.ctx;
if (!ctx)
goto out;
@@ -301,17 +301,18 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* without first stopping userland access to the memory. For
* VM_UFFD_MISSING userfaults this is enough for now.
*/
- if (unlikely(!(fe->flags & FAULT_FLAG_ALLOW_RETRY))) {
+ if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
/*
* Validate the invariant that nowait must allow retry
* to be sure not to return SIGBUS erroneously on
* nowait invocations.
*/
- BUG_ON(fe->flags & FAULT_FLAG_RETRY_NOWAIT);
+ BUG_ON(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
#ifdef CONFIG_DEBUG_VM
if (printk_ratelimit()) {
printk(KERN_WARNING
- "FAULT_FLAG_ALLOW_RETRY missing %x\n", fe->flags);
+ "FAULT_FLAG_ALLOW_RETRY missing %x\n",
+ vmf->flags);
dump_stack();
}
#endif
@@ -323,7 +324,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* and wait.
*/
ret = VM_FAULT_RETRY;
- if (fe->flags & FAULT_FLAG_RETRY_NOWAIT)
+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
goto out;
/* take the reference before dropping the mmap_sem */
@@ -331,11 +332,11 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
uwq.wq.private = current;
- uwq.msg = userfault_msg(fe->address, fe->flags, reason);
+ uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
uwq.ctx = ctx;
return_to_userland =
- (fe->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
+ (vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
(FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
spin_lock(&ctx->fault_pending_wqh.lock);
@@ -353,7 +354,8 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
TASK_KILLABLE);
spin_unlock(&ctx->fault_pending_wqh.lock);
- must_wait = userfaultfd_must_wait(ctx, fe->address, fe->flags, reason);
+ must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
+ reason);
up_read(&mm->mmap_sem);
if (likely(must_wait && !ACCESS_ONCE(ctx->released) &&
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 9b9f65d99873..3237a39f94a4 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -1,12 +1,12 @@
#ifndef _LINUX_HUGE_MM_H
#define _LINUX_HUGE_MM_H
-extern int do_huge_pmd_anonymous_page(struct fault_env *fe);
+extern int do_huge_pmd_anonymous_page(struct vm_fault *vmf);
extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
struct vm_area_struct *vma);
-extern void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd);
-extern int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd);
+extern void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
+extern int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmd,
@@ -142,7 +142,7 @@ static inline int hpage_nr_pages(struct page *page)
return 1;
}
-extern int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd);
+extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *huge_zero_page;
@@ -210,7 +210,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
return NULL;
}
-static inline int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd)
+static inline int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
return 0;
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a92c8d73aeaf..657eb69eb87e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
* pgoff should be used in favour of virtual_address, if possible.
*/
struct vm_fault {
+ struct vm_area_struct *vma; /* Target VMA */
unsigned int flags; /* FAULT_FLAG_xxx flags */
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
- void __user *virtual_address; /* Faulting virtual address */
+ unsigned long address; /* Faulting virtual address */
+ void __user *virtual_address; /* Faulting virtual address masked by
+ * PAGE_MASK */
+ pmd_t *pmd; /* Pointer to pmd entry matching
+ * the 'address'
+ */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
@@ -309,19 +315,7 @@ struct vm_fault {
* VM_FAULT_DAX_LOCKED and fill in
* entry here.
*/
-};
-
-/*
- * Page fault context: passes though page fault handler instead of endless list
- * of function arguments.
- */
-struct fault_env {
- struct vm_area_struct *vma; /* Target VMA */
- unsigned long address; /* Faulting virtual address */
- unsigned int flags; /* FAULT_FLAG_xxx flags */
- pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ /* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
* table hasn't been allocated.
@@ -351,7 +345,7 @@ struct vm_operations_struct {
int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf);
int (*pmd_fault)(struct vm_area_struct *, unsigned long address,
pmd_t *, unsigned int flags);
- void (*map_pages)(struct fault_env *fe,
+ void (*map_pages)(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
/* notification that a previously read-only page is about to become
@@ -625,7 +619,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
return pte;
}
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
#endif
@@ -2097,7 +2091,7 @@ extern void truncate_inode_pages_final(struct address_space *);
/* generic vm_area_ops exported for stackable file systems */
extern int filemap_fault(struct vm_area_struct *, struct vm_fault *);
-extern void filemap_map_pages(struct fault_env *fe,
+extern void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
extern int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf);
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index dd66a952e8cd..11b92b047a1e 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -27,7 +27,7 @@
#define UFFD_SHARED_FCNTL_FLAGS (O_CLOEXEC | O_NONBLOCK)
#define UFFD_FLAGS_SET (EFD_SHARED_FCNTL_FLAGS)
-extern int handle_userfault(struct fault_env *fe, unsigned long reason);
+extern int handle_userfault(struct vm_fault *vmf, unsigned long reason);
extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,
unsigned long src_start, unsigned long len);
@@ -55,7 +55,7 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
#else /* CONFIG_USERFAULTFD */
/* mm helpers */
-static inline int handle_userfault(struct fault_env *fe, unsigned long reason)
+static inline int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
return VM_FAULT_SIGBUS;
}
diff --git a/mm/filemap.c b/mm/filemap.c
index db26ebc6c62f..1426fb1a99b3 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2209,12 +2209,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
struct radix_tree_iter iter;
void **slot;
- struct file *file = fe->vma->vm_file;
+ struct file *file = vmf->vma->vm_file;
struct address_space *mapping = file->f_mapping;
pgoff_t last_pgoff = start_pgoff;
loff_t size;
@@ -2270,11 +2270,11 @@ void filemap_map_pages(struct fault_env *fe,
if (file->f_ra.mmap_miss > 0)
file->f_ra.mmap_miss--;
- fe->address += (iter.index - last_pgoff) << PAGE_SHIFT;
- if (fe->pte)
- fe->pte += iter.index - last_pgoff;
+ vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
+ if (vmf->pte)
+ vmf->pte += iter.index - last_pgoff;
last_pgoff = iter.index;
- if (alloc_set_pte(fe, NULL, page))
+ if (alloc_set_pte(vmf, NULL, page))
goto unlock;
unlock_page(page);
goto next;
@@ -2284,7 +2284,7 @@ void filemap_map_pages(struct fault_env *fe,
put_page(page);
next:
/* Huge page is mapped? No need to proceed. */
- if (pmd_trans_huge(*fe->pmd))
+ if (pmd_trans_huge(*vmf->pmd))
break;
if (iter.index == end_pgoff)
break;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index cdcd25cb30fe..e286b09f9d24 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -532,13 +532,13 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
}
EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
-static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
+static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page,
gfp_t gfp)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
pgtable_t pgtable;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
VM_BUG_ON_PAGE(!PageCompound(page), page);
@@ -563,9 +563,9 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
*/
__SetPageUptodate(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
@@ -576,11 +576,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
if (userfaultfd_missing(vma)) {
int ret;
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
return ret;
}
@@ -590,11 +590,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
page_add_new_anon_rmap(page, vma, haddr, true);
mem_cgroup_commit_charge(page, memcg, false, true);
lru_cache_add_active_or_unevictable(page, vma);
- pgtable_trans_huge_deposit(vma->vm_mm, fe->pmd, pgtable);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
atomic_long_inc(&vma->vm_mm->nr_ptes);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
count_vm_event(THP_FAULT_ALLOC);
}
@@ -641,12 +641,12 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
return true;
}
-int do_huge_pmd_anonymous_page(struct fault_env *fe)
+int do_huge_pmd_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
gfp_t gfp;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
return VM_FAULT_FALLBACK;
@@ -654,7 +654,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_OOM;
if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
return VM_FAULT_OOM;
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm) &&
transparent_hugepage_use_zero_page()) {
pgtable_t pgtable;
@@ -670,22 +670,22 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
count_vm_event(THP_FAULT_FALLBACK);
return VM_FAULT_FALLBACK;
}
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
ret = 0;
set = false;
- if (pmd_none(*fe->pmd)) {
+ if (pmd_none(*vmf->pmd)) {
if (userfaultfd_missing(vma)) {
- spin_unlock(fe->ptl);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ spin_unlock(vmf->ptl);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
} else {
set_huge_zero_page(pgtable, vma->vm_mm, vma,
- haddr, fe->pmd, zero_page);
- spin_unlock(fe->ptl);
+ haddr, vmf->pmd, zero_page);
+ spin_unlock(vmf->ptl);
set = true;
}
} else
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
if (!set)
pte_free(vma->vm_mm, pgtable);
return ret;
@@ -697,7 +697,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_FALLBACK;
}
prep_transhuge_page(page);
- return __do_huge_pmd_anonymous_page(fe, page, gfp);
+ return __do_huge_pmd_anonymous_page(vmf, page, gfp);
}
static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
@@ -868,30 +868,30 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
return ret;
}
-void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd)
+void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)
{
pmd_t entry;
unsigned long haddr;
- fe->ptl = pmd_lock(fe->vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto unlock;
entry = pmd_mkyoung(orig_pmd);
- haddr = fe->address & HPAGE_PMD_MASK;
- if (pmdp_set_access_flags(fe->vma, haddr, fe->pmd, entry,
- fe->flags & FAULT_FLAG_WRITE))
- update_mmu_cache_pmd(fe->vma, fe->address, fe->pmd);
+ haddr = vmf->address & HPAGE_PMD_MASK;
+ if (pmdp_set_access_flags(vmf->vma, haddr, vmf->pmd, entry,
+ vmf->flags & FAULT_FLAG_WRITE))
+ update_mmu_cache_pmd(vmf->vma, vmf->address, vmf->pmd);
unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
}
-static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
+static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
struct mem_cgroup *memcg;
pgtable_t pgtable;
pmd_t _pmd;
@@ -910,7 +910,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
for (i = 0; i < HPAGE_PMD_NR; i++) {
pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE |
__GFP_OTHER_NODE, vma,
- fe->address, page_to_nid(page));
+ vmf->address, page_to_nid(page));
if (unlikely(!pages[i] ||
mem_cgroup_try_charge(pages[i], vma->vm_mm,
GFP_KERNEL, &memcg, false))) {
@@ -941,15 +941,15 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_free_pages;
VM_BUG_ON_PAGE(!PageHead(page), page);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
/* leave pmd empty until pte is filled */
- pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, fe->pmd);
+ pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, vmf->pmd);
pmd_populate(vma->vm_mm, &_pmd, pgtable);
for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
@@ -958,20 +958,20 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
memcg = (void *)page_private(pages[i]);
set_page_private(pages[i], 0);
- page_add_new_anon_rmap(pages[i], fe->vma, haddr, false);
+ page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false);
mem_cgroup_commit_charge(pages[i], memcg, false, false);
lru_cache_add_active_or_unevictable(pages[i], vma);
- fe->pte = pte_offset_map(&_pmd, haddr);
- VM_BUG_ON(!pte_none(*fe->pte));
- set_pte_at(vma->vm_mm, haddr, fe->pte, entry);
- pte_unmap(fe->pte);
+ vmf->pte = pte_offset_map(&_pmd, haddr);
+ VM_BUG_ON(!pte_none(*vmf->pte));
+ set_pte_at(vma->vm_mm, haddr, vmf->pte, entry);
+ pte_unmap(vmf->pte);
}
kfree(pages);
smp_wmb(); /* make pte visible before pmd */
- pmd_populate(vma->vm_mm, fe->pmd, pgtable);
+ pmd_populate(vma->vm_mm, vmf->pmd, pgtable);
page_remove_rmap(page, true);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
@@ -982,7 +982,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
return ret;
out_free_pages:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
for (i = 0; i < HPAGE_PMD_NR; i++) {
memcg = (void *)page_private(pages[i]);
@@ -994,23 +994,23 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
goto out;
}
-int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
+int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL, *new_page;
struct mem_cgroup *memcg;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
unsigned long mmun_start; /* For mmu_notifiers */
unsigned long mmun_end; /* For mmu_notifiers */
gfp_t huge_gfp; /* for allocation and charge */
int ret = 0;
- fe->ptl = pmd_lockptr(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd);
VM_BUG_ON_VMA(!vma->anon_vma, vma);
if (is_huge_zero_pmd(orig_pmd))
goto alloc;
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_unlock;
page = pmd_page(orig_pmd);
@@ -1023,13 +1023,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = pmd_mkyoung(orig_pmd);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- if (pmdp_set_access_flags(vma, haddr, fe->pmd, entry, 1))
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1))
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
ret |= VM_FAULT_WRITE;
goto out_unlock;
}
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
alloc:
if (transparent_hugepage_enabled(vma) &&
!transparent_hugepage_debug_cow()) {
@@ -1042,12 +1042,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
prep_transhuge_page(new_page);
} else {
if (!page) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
} else {
- ret = do_huge_pmd_wp_page_fallback(fe, orig_pmd, page);
+ ret = do_huge_pmd_wp_page_fallback(vmf, orig_pmd, page);
if (ret & VM_FAULT_OOM) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
}
put_page(page);
@@ -1059,7 +1059,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
huge_gfp, &memcg, true))) {
put_page(new_page);
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
if (page)
put_page(page);
ret |= VM_FAULT_FALLBACK;
@@ -1079,11 +1079,11 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- spin_lock(fe->ptl);
+ spin_lock(vmf->ptl);
if (page)
put_page(page);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd))) {
- spin_unlock(fe->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(new_page, memcg, true);
put_page(new_page);
goto out_mn;
@@ -1091,12 +1091,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = mk_huge_pmd(new_page, vma->vm_page_prot);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
page_add_new_anon_rmap(new_page, vma, haddr, true);
mem_cgroup_commit_charge(new_page, memcg, false, true);
lru_cache_add_active_or_unevictable(new_page, vma);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
if (!page) {
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
} else {
@@ -1106,13 +1106,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
}
ret |= VM_FAULT_WRITE;
}
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out_mn:
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
out:
return ret;
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
@@ -1185,12 +1185,12 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
}
/* NUMA hinting page fault entry point for trans huge pmds */
-int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
+int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct anon_vma *anon_vma = NULL;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
int page_nid = -1, this_nid = numa_node_id();
int target_nid, last_cpupid = -1;
bool page_locked;
@@ -1198,8 +1198,8 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
bool was_writable;
int flags = 0;
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(pmd, *fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd)))
goto out_unlock;
/*
@@ -1207,9 +1207,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* without disrupting NUMA hinting information. Do not relock and
* check_same as the page may no longer be mapped.
*/
- if (unlikely(pmd_trans_migrating(*fe->pmd))) {
- page = pmd_page(*fe->pmd);
- spin_unlock(fe->ptl);
+ if (unlikely(pmd_trans_migrating(*vmf->pmd))) {
+ page = pmd_page(*vmf->pmd);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
goto out;
}
@@ -1242,7 +1242,7 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
/* Migration could have started since the pmd_trans_migrating check */
if (!page_locked) {
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
page_nid = -1;
goto out;
@@ -1253,12 +1253,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* to serialises splits
*/
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
anon_vma = page_lock_anon_vma_read(page);
/* Confirm the PMD did not change while page_table_lock was released */
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(pmd, *fe->pmd))) {
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd))) {
unlock_page(page);
put_page(page);
page_nid = -1;
@@ -1276,9 +1276,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* Migrate the THP to the requested node, returns with page unlocked
* and access rights restored.
*/
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma,
- fe->pmd, pmd, fe->address, page, target_nid);
+ vmf->pmd, pmd, vmf->address, page, target_nid);
if (migrated) {
flags |= TNF_MIGRATED;
page_nid = target_nid;
@@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
pmd = pmd_mkyoung(pmd);
if (was_writable)
pmd = pmd_mkwrite(pmd);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, pmd);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
unlock_page(page);
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out:
if (anon_vma)
page_unlock_anon_vma_read(anon_vma);
if (page_nid != -1)
- task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, fe->flags);
+ task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
+ vmf->flags);
return 0;
}
diff --git a/mm/internal.h b/mm/internal.h
index 537ac9951f5f..093b1eacc91b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct fault_env *fe, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 728d7790dc2d..f88b2d3810a7 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -875,7 +875,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
{
pte_t pteval;
int swapped_in = 0, ret = 0;
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
@@ -887,19 +887,19 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
}
- fe.pte = pte_offset_map(pmd, address);
- for (; fe.address < address + HPAGE_PMD_NR*PAGE_SIZE;
- fe.pte++, fe.address += PAGE_SIZE) {
- pteval = *fe.pte;
+ vmf.pte = pte_offset_map(pmd, address);
+ for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
+ vmf.pte++, vmf.address += PAGE_SIZE) {
+ pteval = *vmf.pte;
if (!is_swap_pte(pteval))
continue;
swapped_in++;
- ret = do_swap_page(&fe, pteval);
+ ret = do_swap_page(&vmf, pteval);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
down_read(&mm->mmap_sem);
- if (hugepage_vma_revalidate(mm, address, &fe.vma)) {
+ if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
/* vma is no longer available, don't continue to swapin */
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
@@ -913,10 +913,10 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
return false;
}
/* pte is unmapped now, we need to map it */
- fe.pte = pte_offset_map(pmd, fe.address);
+ vmf.pte = pte_offset_map(pmd, vmf.address);
}
- fe.pte--;
- pte_unmap(fe.pte);
+ vmf.pte--;
+ pte_unmap(vmf.pte);
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
return true;
}
diff --git a/mm/memory.c b/mm/memory.c
index e18c57bdc75c..fad45cd59ba7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,11 +2074,11 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
+static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
struct page *page, int page_mkwrite, int dirty_shared)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2088,12 +2088,12 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = pte_mkyoung(orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
- if (ptep_set_access_flags(vma, fe->address, fe->pte, entry, 1))
- update_mmu_cache(vma, fe->address, fe->pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
+ update_mmu_cache(vma, vmf->address, vmf->pte);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
struct address_space *mapping;
@@ -2139,15 +2139,15 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
- const unsigned long mmun_start = fe->address & PAGE_MASK;
+ const unsigned long mmun_start = vmf->address & PAGE_MASK;
const unsigned long mmun_end = mmun_start + PAGE_SIZE;
struct mem_cgroup *memcg;
@@ -2155,15 +2155,16 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
goto oom;
if (is_zero_pfn(pte_pfn(orig_pte))) {
- new_page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ new_page = alloc_zeroed_user_highpage_movable(vma,
+ vmf->address);
if (!new_page)
goto oom;
} else {
new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
- fe->address);
+ vmf->address);
if (!new_page)
goto oom;
- cow_user_page(new_page, old_page, fe->address, vma);
+ cow_user_page(new_page, old_page, vmf->address, vma);
}
if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false))
@@ -2176,8 +2177,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
/*
* Re-check the pte - we dropped the lock
*/
- fe->pte = pte_offset_map_lock(mm, fe->pmd, fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte))) {
+ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2187,7 +2188,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2196,8 +2197,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* seen in the presence of one thread doing SMC and another
* thread doing COW.
*/
- ptep_clear_flush_notify(vma, fe->address, fe->pte);
- page_add_new_anon_rmap(new_page, vma, fe->address, false);
+ ptep_clear_flush_notify(vma, vmf->address, vmf->pte);
+ page_add_new_anon_rmap(new_page, vma, vmf->address, false);
mem_cgroup_commit_charge(new_page, memcg, false, false);
lru_cache_add_active_or_unevictable(new_page, vma);
/*
@@ -2205,8 +2206,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* mmu page tables (such as kvm shadow page tables), we want the
* new page to be mapped directly into the secondary page table.
*/
- set_pte_at_notify(mm, fe->address, fe->pte, entry);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at_notify(mm, vmf->address, vmf->pte, entry);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
if (old_page) {
/*
* Only after switching the pte to the new page may
@@ -2243,7 +2244,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
if (new_page)
put_page(new_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
if (old_page) {
/*
@@ -2271,43 +2272,43 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct fault_env *fe, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf = {
+ struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, fe->address),
+ .pgoff = linear_page_index(vma, vmf->address),
.virtual_address =
- (void __user *)(fe->address & PAGE_MASK),
+ (void __user *)(vmf->address & PAGE_MASK),
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
- pte_unmap_unlock(fe->pte, fe->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
if (ret & VM_FAULT_ERROR)
return ret;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
/*
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*fe->pte, orig_pte)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(fe, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
}
-static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
get_page(old_page);
@@ -2315,8 +2316,8 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
int tmp;
- pte_unmap_unlock(fe->pte, fe->ptl);
- tmp = do_page_mkwrite(vma, old_page, fe->address);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ tmp = do_page_mkwrite(vma, old_page, vmf->address);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -2328,18 +2329,18 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* they did, we just return, as we can count on the
* MMU to tell us if they didn't also make it writable.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(fe, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
}
/*
@@ -2360,13 +2361,13 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
- __releases(fe->ptl)
+static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, fe->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2377,10 +2378,10 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(fe, orig_pte);
+ return wp_pfn_shared(vmf, orig_pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
/*
@@ -2391,13 +2392,13 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
int total_mapcount;
if (!trylock_page(old_page)) {
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
lock_page(old_page);
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
@@ -2415,12 +2416,12 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(fe, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(fe, orig_pte, old_page);
+ return wp_page_shared(vmf, orig_pte, old_page);
}
/*
@@ -2428,8 +2429,8 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2517,9 +2518,9 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct fault_env *fe, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
struct mem_cgroup *memcg;
swp_entry_t entry;
@@ -2528,17 +2529,18 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, fe->pmd, fe->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
goto out;
entry = pte_to_swp_entry(orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
- migration_entry_wait(vma->vm_mm, fe->pmd, fe->address);
+ migration_entry_wait(vma->vm_mm, vmf->pmd,
+ vmf->address);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, fe->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2546,16 +2548,16 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
delayacct_set_flag(DELAYACCT_PF_SWAPIN);
page = lookup_swap_cache(entry);
if (!page) {
- page = swapin_readahead(entry,
- GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vma,
+ vmf->address);
if (!page) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2577,7 +2579,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
}
swapcache = page;
- locked = lock_page_or_retry(page, vma->vm_mm, fe->flags);
+ locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
if (!locked) {
@@ -2594,7 +2596,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
if (unlikely(!PageSwapCache(page) || page_private(page) != entry.val))
goto out_page;
- page = ksm_might_need_to_copy(page, vma, fe->address);
+ page = ksm_might_need_to_copy(page, vma, vmf->address);
if (unlikely(!page)) {
ret = VM_FAULT_OOM;
page = swapcache;
@@ -2610,9 +2612,9 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
/*
* Back out if somebody else already faulted in this pte.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (unlikely(!pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2633,22 +2635,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
pte = mk_pte(page, vma->vm_page_prot);
- if ((fe->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
+ if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
pte = maybe_mkwrite(pte_mkdirty(pte), vma);
- fe->flags &= ~FAULT_FLAG_WRITE;
+ vmf->flags &= ~FAULT_FLAG_WRITE;
ret |= VM_FAULT_WRITE;
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
if (pte_swp_soft_dirty(orig_pte))
pte = pte_mksoft_dirty(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
if (page == swapcache) {
- do_page_add_anon_rmap(page, vma, fe->address, exclusive);
+ do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
activate_page(page);
} else { /* ksm created a completely new copy */
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
}
@@ -2671,22 +2673,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
put_page(swapcache);
}
- if (fe->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(fe, pte);
+ if (vmf->flags & FAULT_FLAG_WRITE) {
+ ret |= do_wp_page(vmf, pte);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
}
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
return ret;
out_nomap:
mem_cgroup_cancel_charge(page, memcg, false);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out_page:
unlock_page(page);
out_release:
@@ -2737,9 +2739,9 @@ static inline int check_stack_guard_page(struct vm_area_struct *vma, unsigned lo
* but allow concurrent faults), and pte mapped but not yet locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_anonymous_page(struct fault_env *fe)
+static int do_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
struct page *page;
pte_t entry;
@@ -2749,7 +2751,7 @@ static int do_anonymous_page(struct fault_env *fe)
return VM_FAULT_SIGBUS;
/* Check if we need to add a guard page to the stack */
- if (check_stack_guard_page(vma, fe->address) < 0)
+ if (check_stack_guard_page(vma, vmf->address) < 0)
return VM_FAULT_SIGSEGV;
/*
@@ -2762,26 +2764,26 @@ static int do_anonymous_page(struct fault_env *fe)
*
* Here we only have down_read(mmap_sem).
*/
- if (pte_alloc(vma->vm_mm, fe->pmd, fe->address))
+ if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
return VM_FAULT_OOM;
/* See the comment in pte_alloc_one_map() */
- if (unlikely(pmd_trans_unstable(fe->pmd)))
+ if (unlikely(pmd_trans_unstable(vmf->pmd)))
return 0;
/* Use the zero-page for reads */
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm)) {
- entry = pte_mkspecial(pfn_pte(my_zero_pfn(fe->address),
+ entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
vma->vm_page_prot));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto unlock;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
goto setpte;
}
@@ -2789,7 +2791,7 @@ static int do_anonymous_page(struct fault_env *fe)
/* Allocate our own private page. */
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
if (!page)
goto oom;
@@ -2807,30 +2809,30 @@ static int do_anonymous_page(struct fault_env *fe)
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto release;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, false);
put_page(page);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
setpte:
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
release:
mem_cgroup_cancel_charge(page, memcg, false);
@@ -2847,62 +2849,62 @@ static int do_anonymous_page(struct fault_env *fe)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct fault_env *fe, pgoff_t pgoff,
+static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct page *cow_page, struct page **page, void **entry)
{
- struct vm_area_struct *vma = fe->vma;
- struct vm_fault vmf;
+ struct vm_area_struct *vma = vmf->vma;
+ struct vm_fault vmf2;
int ret;
- vmf.virtual_address = (void __user *)(fe->address & PAGE_MASK);
- vmf.pgoff = pgoff;
- vmf.flags = fe->flags;
- vmf.page = NULL;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.cow_page = cow_page;
+ vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.pgoff = pgoff;
+ vmf2.flags = vmf->flags;
+ vmf2.page = NULL;
+ vmf2.gfp_mask = __get_fault_gfp_mask(vma);
+ vmf2.cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf);
+ ret = vma->vm_ops->fault(vma, &vmf2);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf.entry;
+ *entry = vmf2.entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf.page))) {
+ if (unlikely(PageHWPoison(vmf2.page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf.page);
- put_page(vmf.page);
+ unlock_page(vmf2.page);
+ put_page(vmf2.page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf.page);
+ lock_page(vmf2.page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
- *page = vmf.page;
+ *page = vmf2.page;
return ret;
}
-static int pte_alloc_one_map(struct fault_env *fe)
+static int pte_alloc_one_map(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
- if (!pmd_none(*fe->pmd))
+ if (!pmd_none(*vmf->pmd))
goto map_pte;
- if (fe->prealloc_pte) {
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ if (vmf->prealloc_pte) {
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
goto map_pte;
}
atomic_long_inc(&vma->vm_mm->nr_ptes);
- pmd_populate(vma->vm_mm, fe->pmd, fe->prealloc_pte);
- spin_unlock(fe->ptl);
- fe->prealloc_pte = 0;
- } else if (unlikely(pte_alloc(vma->vm_mm, fe->pmd, fe->address))) {
+ pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
+ spin_unlock(vmf->ptl);
+ vmf->prealloc_pte = 0;
+ } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
return VM_FAULT_OOM;
}
map_pte:
@@ -2917,11 +2919,11 @@ static int pte_alloc_one_map(struct fault_env *fe)
* through an atomic read in C, which is what pmd_trans_unstable()
* provides.
*/
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return VM_FAULT_NOPAGE;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
return 0;
}
@@ -2939,11 +2941,11 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
return true;
}
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
pmd_t entry;
int i, ret;
@@ -2953,8 +2955,8 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
ret = VM_FAULT_FALLBACK;
page = compound_head(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd)))
goto out;
for (i = 0; i < HPAGE_PMD_NR; i++)
@@ -2967,19 +2969,19 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
add_mm_counter(vma->vm_mm, MM_FILEPAGES, HPAGE_PMD_NR);
page_add_file_rmap(page, true);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
- update_mmu_cache_pmd(vma, haddr, fe->pmd);
+ update_mmu_cache_pmd(vma, haddr, vmf->pmd);
/* fault is handled */
ret = 0;
count_vm_event(THP_FILE_MAPPED);
out:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
#else
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
BUILD_BUG();
return 0;
@@ -2990,41 +2992,42 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
* alloc_set_pte - setup new PTE entry for given page and add reverse page
* mapping. If needed, the fucntion allocates page table or use pre-allocated.
*
- * @fe: fault environment
+ * @vmf: fault environment
* @memcg: memcg to charge page (only for private mappings)
* @page: page to map
*
- * Caller must take care of unlocking fe->ptl, if fe->pte is non-NULL on return.
+ * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
+ * return.
*
* Target users are page handler itself and implementations of
* vm_ops->map_pages.
*/
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
pte_t entry;
int ret;
- if (pmd_none(*fe->pmd) && PageTransCompound(page) &&
+ if (pmd_none(*vmf->pmd) && PageTransCompound(page) &&
IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE)) {
/* THP on COW? */
VM_BUG_ON_PAGE(memcg, page);
- ret = do_set_pmd(fe, page);
+ ret = do_set_pmd(vmf, page);
if (ret != VM_FAULT_FALLBACK)
return ret;
}
- if (!fe->pte) {
- ret = pte_alloc_one_map(fe);
+ if (!vmf->pte) {
+ ret = pte_alloc_one_map(vmf);
if (ret)
return ret;
}
/* Re-check under ptl */
- if (unlikely(!pte_none(*fe->pte)))
+ if (unlikely(!pte_none(*vmf->pte)))
return VM_FAULT_NOPAGE;
flush_icache_page(vma, page);
@@ -3034,17 +3037,17 @@ int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
} else {
inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page));
page_add_file_rmap(page, false);
}
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* no need to invalidate: a not-present page won't be cached */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
return 0;
}
@@ -3113,17 +3116,17 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
{
- unsigned long address = fe->address, nr_pages, mask;
+ unsigned long address = vmf->address, nr_pages, mask;
pgoff_t end_pgoff;
int off, ret = 0;
nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
- fe->address = max(address & mask, fe->vma->vm_start);
- off = ((address - fe->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
+ vmf->address = max(address & mask, vmf->vma->vm_start);
+ off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
start_pgoff -= off;
/*
@@ -3131,49 +3134,51 @@ static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
* or fault_around_pages() from start_pgoff, depending what is nearest.
*/
end_pgoff = start_pgoff -
- ((fe->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
+ ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
PTRS_PER_PTE - 1;
- end_pgoff = min3(end_pgoff, vma_pages(fe->vma) + fe->vma->vm_pgoff - 1,
+ end_pgoff = min3(end_pgoff,
+ vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
start_pgoff + nr_pages - 1);
- if (pmd_none(*fe->pmd)) {
- fe->prealloc_pte = pte_alloc_one(fe->vma->vm_mm, fe->address);
- if (!fe->prealloc_pte)
+ if (pmd_none(*vmf->pmd)) {
+ vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
+ vmf->address);
+ if (!vmf->prealloc_pte)
goto out;
smp_wmb(); /* See comment in __pte_alloc() */
}
- fe->vma->vm_ops->map_pages(fe, start_pgoff, end_pgoff);
+ vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
/* preallocated pagetable is unused: free it */
- if (fe->prealloc_pte) {
- pte_free(fe->vma->vm_mm, fe->prealloc_pte);
- fe->prealloc_pte = 0;
+ if (vmf->prealloc_pte) {
+ pte_free(vmf->vma->vm_mm, vmf->prealloc_pte);
+ vmf->prealloc_pte = 0;
}
/* Huge page is mapped? Page fault is solved */
- if (pmd_trans_huge(*fe->pmd)) {
+ if (pmd_trans_huge(*vmf->pmd)) {
ret = VM_FAULT_NOPAGE;
goto out;
}
/* ->map_pages() haven't done anything useful. Cold page cache? */
- if (!fe->pte)
+ if (!vmf->pte)
goto out;
/* check if the page fault is solved */
- fe->pte -= (fe->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
- if (!pte_none(*fe->pte))
+ vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
+ if (!pte_none(*vmf->pte))
ret = VM_FAULT_NOPAGE;
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
- fe->address = address;
- fe->pte = NULL;
+ vmf->address = address;
+ vmf->pte = NULL;
return ret;
}
-static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
int ret = 0;
@@ -3183,27 +3188,27 @@ static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(fe, pgoff);
+ ret = do_fault_around(vmf, pgoff);
if (ret)
return ret;
}
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
unlock_page(fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(fault_page);
return ret;
}
-static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
void *fault_entry;
struct mem_cgroup *memcg;
@@ -3212,7 +3217,7 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
if (unlikely(anon_vma_prepare(vma)))
return VM_FAULT_OOM;
- new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
if (!new_page)
return VM_FAULT_OOM;
@@ -3222,17 +3227,17 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(fe, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, fe->address, vma);
+ copy_user_highpage(new_page, fault_page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(fe, memcg, new_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, memcg, new_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(fault_page);
put_page(fault_page);
@@ -3248,15 +3253,15 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3266,7 +3271,7 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, fe->address);
+ tmp = do_page_mkwrite(vma, fault_page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(fault_page);
@@ -3274,9 +3279,9 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
}
}
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(fault_page);
@@ -3314,19 +3319,19 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
* The mmap_sem may have been released depending on flags and our
* return value. See filemap_fault() and __lock_page_or_retry().
*/
-static int do_fault(struct fault_env *fe)
+static int do_fault(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
- pgoff_t pgoff = linear_page_index(vma, fe->address);
+ struct vm_area_struct *vma = vmf->vma;
+ pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
- if (!(fe->flags & FAULT_FLAG_WRITE))
- return do_read_fault(fe, pgoff);
+ if (!(vmf->flags & FAULT_FLAG_WRITE))
+ return do_read_fault(vmf, pgoff);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(fe, pgoff);
- return do_shared_fault(fe, pgoff);
+ return do_cow_fault(vmf, pgoff);
+ return do_shared_fault(vmf, pgoff);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3344,9 +3349,9 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct fault_env *fe, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf, pte_t pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
int page_nid = -1;
int last_cpupid;
@@ -3364,10 +3369,10 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
* page table entry is not accessible, so there would be no
* concurrent hardware modifications to the PTE.
*/
- fe->ptl = pte_lockptr(vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, pte))) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, pte))) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
goto out;
}
@@ -3376,18 +3381,18 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
pte = pte_mkyoung(pte);
if (was_writable)
pte = pte_mkwrite(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
- page = vm_normal_page(vma, fe->address, pte);
+ page = vm_normal_page(vma, vmf->address, pte);
if (!page) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
/* TODO: handle PTE-mapped THP */
if (PageCompound(page)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3411,9 +3416,9 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
last_cpupid = page_cpupid_last(page);
page_nid = page_to_nid(page);
- target_nid = numa_migrate_prep(page, vma, fe->address, page_nid,
+ target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
&flags);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (target_nid == -1) {
put_page(page);
goto out;
@@ -3433,28 +3438,28 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
return 0;
}
-static int create_huge_pmd(struct fault_env *fe)
+static int create_huge_pmd(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma_is_anonymous(vma))
- return do_huge_pmd_anonymous_page(fe);
+ return do_huge_pmd_anonymous_page(vmf);
if (vma->vm_ops->pmd_fault)
- return vma->vm_ops->pmd_fault(vma, fe->address, fe->pmd,
- fe->flags);
+ return vma->vm_ops->pmd_fault(vma, vmf->address, vmf->pmd,
+ vmf->flags);
return VM_FAULT_FALLBACK;
}
-static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
+static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
{
- if (vma_is_anonymous(fe->vma))
- return do_huge_pmd_wp_page(fe, orig_pmd);
- if (fe->vma->vm_ops->pmd_fault)
- return fe->vma->vm_ops->pmd_fault(fe->vma, fe->address, fe->pmd,
- fe->flags);
+ if (vma_is_anonymous(vmf->vma))
+ return do_huge_pmd_wp_page(vmf, orig_pmd);
+ if (vmf->vma->vm_ops->pmd_fault)
+ return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
+ vmf->pmd, vmf->flags);
/* COW handled on pte level: split pmd */
- VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
- split_huge_pmd(fe->vma, fe->pmd, fe->address);
+ VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
+ split_huge_pmd(vmf->vma, vmf->pmd, vmf->address);
return VM_FAULT_FALLBACK;
}
@@ -3479,21 +3484,21 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
* The mmap_sem may have been released depending on flags and our return value.
* See filemap_fault() and __lock_page_or_retry().
*/
-static int handle_pte_fault(struct fault_env *fe)
+static int handle_pte_fault(struct vm_fault *vmf)
{
pte_t entry;
- if (unlikely(pmd_none(*fe->pmd))) {
+ if (unlikely(pmd_none(*vmf->pmd))) {
/*
* Leave __pte_alloc() until later: because vm_ops->fault may
* want to allocate huge page, and if we expose page table
* for an instant, it will be difficult to retract from
* concurrent faults and from rmap lookups.
*/
- fe->pte = NULL;
+ vmf->pte = NULL;
} else {
/* See comment in pte_alloc_one_map() */
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return 0;
/*
* A regular pmd is established and it can't morph into a huge
@@ -3501,9 +3506,9 @@ static int handle_pte_fault(struct fault_env *fe)
* mmap_sem read mode and khugepaged takes it in write mode.
* So now it's safe to run pte_offset_map().
*/
- fe->pte = pte_offset_map(fe->pmd, fe->address);
+ vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
- entry = *fe->pte;
+ entry = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3515,37 +3520,37 @@ static int handle_pte_fault(struct fault_env *fe)
*/
barrier();
if (pte_none(entry)) {
- pte_unmap(fe->pte);
- fe->pte = NULL;
+ pte_unmap(vmf->pte);
+ vmf->pte = NULL;
}
}
- if (!fe->pte) {
- if (vma_is_anonymous(fe->vma))
- return do_anonymous_page(fe);
+ if (!vmf->pte) {
+ if (vma_is_anonymous(vmf->vma))
+ return do_anonymous_page(vmf);
else
- return do_fault(fe);
+ return do_fault(vmf);
}
if (!pte_present(entry))
- return do_swap_page(fe, entry);
+ return do_swap_page(vmf, entry);
- if (pte_protnone(entry) && vma_is_accessible(fe->vma))
- return do_numa_page(fe, entry);
+ if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf, entry);
- fe->ptl = pte_lockptr(fe->vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, entry)))
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
- if (fe->flags & FAULT_FLAG_WRITE) {
+ if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(fe, entry);
+ return do_wp_page(vmf, entry);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
- if (ptep_set_access_flags(fe->vma, fe->address, fe->pte, entry,
- fe->flags & FAULT_FLAG_WRITE)) {
- update_mmu_cache(fe->vma, fe->address, fe->pte);
+ if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
+ vmf->flags & FAULT_FLAG_WRITE)) {
+ update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
} else {
/*
* This is needed only for protection faults but the arch code
@@ -3553,11 +3558,11 @@ static int handle_pte_fault(struct fault_env *fe)
* This still avoids useless tlb flushes for .text page faults
* with threads.
*/
- if (fe->flags & FAULT_FLAG_WRITE)
- flush_tlb_fix_spurious_fault(fe->vma, fe->address);
+ if (vmf->flags & FAULT_FLAG_WRITE)
+ flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
}
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3570,7 +3575,7 @@ static int handle_pte_fault(struct fault_env *fe)
static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
unsigned int flags)
{
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = flags,
@@ -3583,35 +3588,35 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
pud = pud_alloc(mm, pgd, address);
if (!pud)
return VM_FAULT_OOM;
- fe.pmd = pmd_alloc(mm, pud, address);
- if (!fe.pmd)
+ vmf.pmd = pmd_alloc(mm, pud, address);
+ if (!vmf.pmd)
return VM_FAULT_OOM;
- if (pmd_none(*fe.pmd) && transparent_hugepage_enabled(vma)) {
- int ret = create_huge_pmd(&fe);
+ if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
+ int ret = create_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- pmd_t orig_pmd = *fe.pmd;
+ pmd_t orig_pmd = *vmf.pmd;
int ret;
barrier();
if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
if (pmd_protnone(orig_pmd) && vma_is_accessible(vma))
- return do_huge_pmd_numa_page(&fe, orig_pmd);
+ return do_huge_pmd_numa_page(&vmf, orig_pmd);
- if ((fe.flags & FAULT_FLAG_WRITE) &&
+ if ((vmf.flags & FAULT_FLAG_WRITE) &&
!pmd_write(orig_pmd)) {
- ret = wp_huge_pmd(&fe, orig_pmd);
+ ret = wp_huge_pmd(&vmf, orig_pmd);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- huge_pmd_set_accessed(&fe, orig_pmd);
+ huge_pmd_set_accessed(&vmf, orig_pmd);
return 0;
}
}
}
- return handle_pte_fault(&fe);
+ return handle_pte_fault(&vmf);
}
/*
diff --git a/mm/nommu.c b/mm/nommu.c
index 8b8faaf2a9e9..077d0dbe4c28 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1801,7 +1801,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
BUG();
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we have two different structures for passing fault information
around - struct vm_fault and struct fault_env. DAX will need more
information in struct vm_fault to handle its faults so the content of
that structure would become event closer to fault_env. Furthermore it
would need to generate struct fault_env to be able to call some of the
generic functions. So at this point I don't think there's much use in
keeping these two structures separate. Just embed into struct vm_fault
all that is needed to use it for both purposes.
Signed-off-by: Jan Kara <jack@suse.cz>
---
Documentation/filesystems/Locking | 2 +-
fs/userfaultfd.c | 22 +-
include/linux/huge_mm.h | 10 +-
include/linux/mm.h | 28 +-
include/linux/userfaultfd_k.h | 4 +-
mm/filemap.c | 14 +-
mm/huge_memory.c | 173 ++++++------
mm/internal.h | 2 +-
mm/khugepaged.c | 20 +-
mm/memory.c | 549 +++++++++++++++++++-------------------
mm/nommu.c | 2 +-
11 files changed, 414 insertions(+), 412 deletions(-)
diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 14cdc101d165..ac3d080eabaa 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -557,7 +557,7 @@ till "end_pgoff". ->map_pages() is called with page table locked and must
not block. If it's not possible to reach a page without blocking,
filesystem should skip it. Filesystem should use do_set_pte() to setup
page table entry. Pointer to entry associated with the page is passed in
-"pte" field in fault_env structure. Pointers to entries for other offsets
+"pte" field in vm_fault structure. Pointers to entries for other offsets
should be calculated relative to "pte".
->page_mkwrite() is called when a previously read-only pte is
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 85959d8324df..d96e2f30084b 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -257,9 +257,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
* fatal_signal_pending()s, and the mmap_sem must be released before
* returning it.
*/
-int handle_userfault(struct fault_env *fe, unsigned long reason)
+int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
- struct mm_struct *mm = fe->vma->vm_mm;
+ struct mm_struct *mm = vmf->vma->vm_mm;
struct userfaultfd_ctx *ctx;
struct userfaultfd_wait_queue uwq;
int ret;
@@ -268,7 +268,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
ret = VM_FAULT_SIGBUS;
- ctx = fe->vma->vm_userfaultfd_ctx.ctx;
+ ctx = vmf->vma->vm_userfaultfd_ctx.ctx;
if (!ctx)
goto out;
@@ -301,17 +301,18 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* without first stopping userland access to the memory. For
* VM_UFFD_MISSING userfaults this is enough for now.
*/
- if (unlikely(!(fe->flags & FAULT_FLAG_ALLOW_RETRY))) {
+ if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
/*
* Validate the invariant that nowait must allow retry
* to be sure not to return SIGBUS erroneously on
* nowait invocations.
*/
- BUG_ON(fe->flags & FAULT_FLAG_RETRY_NOWAIT);
+ BUG_ON(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
#ifdef CONFIG_DEBUG_VM
if (printk_ratelimit()) {
printk(KERN_WARNING
- "FAULT_FLAG_ALLOW_RETRY missing %x\n", fe->flags);
+ "FAULT_FLAG_ALLOW_RETRY missing %x\n",
+ vmf->flags);
dump_stack();
}
#endif
@@ -323,7 +324,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* and wait.
*/
ret = VM_FAULT_RETRY;
- if (fe->flags & FAULT_FLAG_RETRY_NOWAIT)
+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
goto out;
/* take the reference before dropping the mmap_sem */
@@ -331,11 +332,11 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
uwq.wq.private = current;
- uwq.msg = userfault_msg(fe->address, fe->flags, reason);
+ uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
uwq.ctx = ctx;
return_to_userland =
- (fe->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
+ (vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
(FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
spin_lock(&ctx->fault_pending_wqh.lock);
@@ -353,7 +354,8 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
TASK_KILLABLE);
spin_unlock(&ctx->fault_pending_wqh.lock);
- must_wait = userfaultfd_must_wait(ctx, fe->address, fe->flags, reason);
+ must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
+ reason);
up_read(&mm->mmap_sem);
if (likely(must_wait && !ACCESS_ONCE(ctx->released) &&
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 9b9f65d99873..3237a39f94a4 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -1,12 +1,12 @@
#ifndef _LINUX_HUGE_MM_H
#define _LINUX_HUGE_MM_H
-extern int do_huge_pmd_anonymous_page(struct fault_env *fe);
+extern int do_huge_pmd_anonymous_page(struct vm_fault *vmf);
extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
struct vm_area_struct *vma);
-extern void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd);
-extern int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd);
+extern void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
+extern int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmd,
@@ -142,7 +142,7 @@ static inline int hpage_nr_pages(struct page *page)
return 1;
}
-extern int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd);
+extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *huge_zero_page;
@@ -210,7 +210,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
return NULL;
}
-static inline int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd)
+static inline int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
return 0;
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a92c8d73aeaf..657eb69eb87e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
* pgoff should be used in favour of virtual_address, if possible.
*/
struct vm_fault {
+ struct vm_area_struct *vma; /* Target VMA */
unsigned int flags; /* FAULT_FLAG_xxx flags */
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
- void __user *virtual_address; /* Faulting virtual address */
+ unsigned long address; /* Faulting virtual address */
+ void __user *virtual_address; /* Faulting virtual address masked by
+ * PAGE_MASK */
+ pmd_t *pmd; /* Pointer to pmd entry matching
+ * the 'address'
+ */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
@@ -309,19 +315,7 @@ struct vm_fault {
* VM_FAULT_DAX_LOCKED and fill in
* entry here.
*/
-};
-
-/*
- * Page fault context: passes though page fault handler instead of endless list
- * of function arguments.
- */
-struct fault_env {
- struct vm_area_struct *vma; /* Target VMA */
- unsigned long address; /* Faulting virtual address */
- unsigned int flags; /* FAULT_FLAG_xxx flags */
- pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ /* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
* table hasn't been allocated.
@@ -351,7 +345,7 @@ struct vm_operations_struct {
int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf);
int (*pmd_fault)(struct vm_area_struct *, unsigned long address,
pmd_t *, unsigned int flags);
- void (*map_pages)(struct fault_env *fe,
+ void (*map_pages)(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
/* notification that a previously read-only page is about to become
@@ -625,7 +619,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
return pte;
}
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
#endif
@@ -2097,7 +2091,7 @@ extern void truncate_inode_pages_final(struct address_space *);
/* generic vm_area_ops exported for stackable file systems */
extern int filemap_fault(struct vm_area_struct *, struct vm_fault *);
-extern void filemap_map_pages(struct fault_env *fe,
+extern void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
extern int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf);
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index dd66a952e8cd..11b92b047a1e 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -27,7 +27,7 @@
#define UFFD_SHARED_FCNTL_FLAGS (O_CLOEXEC | O_NONBLOCK)
#define UFFD_FLAGS_SET (EFD_SHARED_FCNTL_FLAGS)
-extern int handle_userfault(struct fault_env *fe, unsigned long reason);
+extern int handle_userfault(struct vm_fault *vmf, unsigned long reason);
extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,
unsigned long src_start, unsigned long len);
@@ -55,7 +55,7 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
#else /* CONFIG_USERFAULTFD */
/* mm helpers */
-static inline int handle_userfault(struct fault_env *fe, unsigned long reason)
+static inline int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
return VM_FAULT_SIGBUS;
}
diff --git a/mm/filemap.c b/mm/filemap.c
index db26ebc6c62f..1426fb1a99b3 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2209,12 +2209,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
struct radix_tree_iter iter;
void **slot;
- struct file *file = fe->vma->vm_file;
+ struct file *file = vmf->vma->vm_file;
struct address_space *mapping = file->f_mapping;
pgoff_t last_pgoff = start_pgoff;
loff_t size;
@@ -2270,11 +2270,11 @@ void filemap_map_pages(struct fault_env *fe,
if (file->f_ra.mmap_miss > 0)
file->f_ra.mmap_miss--;
- fe->address += (iter.index - last_pgoff) << PAGE_SHIFT;
- if (fe->pte)
- fe->pte += iter.index - last_pgoff;
+ vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
+ if (vmf->pte)
+ vmf->pte += iter.index - last_pgoff;
last_pgoff = iter.index;
- if (alloc_set_pte(fe, NULL, page))
+ if (alloc_set_pte(vmf, NULL, page))
goto unlock;
unlock_page(page);
goto next;
@@ -2284,7 +2284,7 @@ void filemap_map_pages(struct fault_env *fe,
put_page(page);
next:
/* Huge page is mapped? No need to proceed. */
- if (pmd_trans_huge(*fe->pmd))
+ if (pmd_trans_huge(*vmf->pmd))
break;
if (iter.index == end_pgoff)
break;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index cdcd25cb30fe..e286b09f9d24 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -532,13 +532,13 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
}
EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
-static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
+static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page,
gfp_t gfp)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
pgtable_t pgtable;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
VM_BUG_ON_PAGE(!PageCompound(page), page);
@@ -563,9 +563,9 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
*/
__SetPageUptodate(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
@@ -576,11 +576,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
if (userfaultfd_missing(vma)) {
int ret;
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
return ret;
}
@@ -590,11 +590,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
page_add_new_anon_rmap(page, vma, haddr, true);
mem_cgroup_commit_charge(page, memcg, false, true);
lru_cache_add_active_or_unevictable(page, vma);
- pgtable_trans_huge_deposit(vma->vm_mm, fe->pmd, pgtable);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
atomic_long_inc(&vma->vm_mm->nr_ptes);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
count_vm_event(THP_FAULT_ALLOC);
}
@@ -641,12 +641,12 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
return true;
}
-int do_huge_pmd_anonymous_page(struct fault_env *fe)
+int do_huge_pmd_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
gfp_t gfp;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
return VM_FAULT_FALLBACK;
@@ -654,7 +654,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_OOM;
if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
return VM_FAULT_OOM;
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm) &&
transparent_hugepage_use_zero_page()) {
pgtable_t pgtable;
@@ -670,22 +670,22 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
count_vm_event(THP_FAULT_FALLBACK);
return VM_FAULT_FALLBACK;
}
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
ret = 0;
set = false;
- if (pmd_none(*fe->pmd)) {
+ if (pmd_none(*vmf->pmd)) {
if (userfaultfd_missing(vma)) {
- spin_unlock(fe->ptl);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ spin_unlock(vmf->ptl);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
} else {
set_huge_zero_page(pgtable, vma->vm_mm, vma,
- haddr, fe->pmd, zero_page);
- spin_unlock(fe->ptl);
+ haddr, vmf->pmd, zero_page);
+ spin_unlock(vmf->ptl);
set = true;
}
} else
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
if (!set)
pte_free(vma->vm_mm, pgtable);
return ret;
@@ -697,7 +697,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_FALLBACK;
}
prep_transhuge_page(page);
- return __do_huge_pmd_anonymous_page(fe, page, gfp);
+ return __do_huge_pmd_anonymous_page(vmf, page, gfp);
}
static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
@@ -868,30 +868,30 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
return ret;
}
-void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd)
+void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)
{
pmd_t entry;
unsigned long haddr;
- fe->ptl = pmd_lock(fe->vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto unlock;
entry = pmd_mkyoung(orig_pmd);
- haddr = fe->address & HPAGE_PMD_MASK;
- if (pmdp_set_access_flags(fe->vma, haddr, fe->pmd, entry,
- fe->flags & FAULT_FLAG_WRITE))
- update_mmu_cache_pmd(fe->vma, fe->address, fe->pmd);
+ haddr = vmf->address & HPAGE_PMD_MASK;
+ if (pmdp_set_access_flags(vmf->vma, haddr, vmf->pmd, entry,
+ vmf->flags & FAULT_FLAG_WRITE))
+ update_mmu_cache_pmd(vmf->vma, vmf->address, vmf->pmd);
unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
}
-static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
+static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
struct mem_cgroup *memcg;
pgtable_t pgtable;
pmd_t _pmd;
@@ -910,7 +910,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
for (i = 0; i < HPAGE_PMD_NR; i++) {
pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE |
__GFP_OTHER_NODE, vma,
- fe->address, page_to_nid(page));
+ vmf->address, page_to_nid(page));
if (unlikely(!pages[i] ||
mem_cgroup_try_charge(pages[i], vma->vm_mm,
GFP_KERNEL, &memcg, false))) {
@@ -941,15 +941,15 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_free_pages;
VM_BUG_ON_PAGE(!PageHead(page), page);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
/* leave pmd empty until pte is filled */
- pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, fe->pmd);
+ pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, vmf->pmd);
pmd_populate(vma->vm_mm, &_pmd, pgtable);
for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
@@ -958,20 +958,20 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
memcg = (void *)page_private(pages[i]);
set_page_private(pages[i], 0);
- page_add_new_anon_rmap(pages[i], fe->vma, haddr, false);
+ page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false);
mem_cgroup_commit_charge(pages[i], memcg, false, false);
lru_cache_add_active_or_unevictable(pages[i], vma);
- fe->pte = pte_offset_map(&_pmd, haddr);
- VM_BUG_ON(!pte_none(*fe->pte));
- set_pte_at(vma->vm_mm, haddr, fe->pte, entry);
- pte_unmap(fe->pte);
+ vmf->pte = pte_offset_map(&_pmd, haddr);
+ VM_BUG_ON(!pte_none(*vmf->pte));
+ set_pte_at(vma->vm_mm, haddr, vmf->pte, entry);
+ pte_unmap(vmf->pte);
}
kfree(pages);
smp_wmb(); /* make pte visible before pmd */
- pmd_populate(vma->vm_mm, fe->pmd, pgtable);
+ pmd_populate(vma->vm_mm, vmf->pmd, pgtable);
page_remove_rmap(page, true);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
@@ -982,7 +982,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
return ret;
out_free_pages:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
for (i = 0; i < HPAGE_PMD_NR; i++) {
memcg = (void *)page_private(pages[i]);
@@ -994,23 +994,23 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
goto out;
}
-int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
+int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL, *new_page;
struct mem_cgroup *memcg;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
unsigned long mmun_start; /* For mmu_notifiers */
unsigned long mmun_end; /* For mmu_notifiers */
gfp_t huge_gfp; /* for allocation and charge */
int ret = 0;
- fe->ptl = pmd_lockptr(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd);
VM_BUG_ON_VMA(!vma->anon_vma, vma);
if (is_huge_zero_pmd(orig_pmd))
goto alloc;
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_unlock;
page = pmd_page(orig_pmd);
@@ -1023,13 +1023,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = pmd_mkyoung(orig_pmd);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- if (pmdp_set_access_flags(vma, haddr, fe->pmd, entry, 1))
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1))
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
ret |= VM_FAULT_WRITE;
goto out_unlock;
}
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
alloc:
if (transparent_hugepage_enabled(vma) &&
!transparent_hugepage_debug_cow()) {
@@ -1042,12 +1042,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
prep_transhuge_page(new_page);
} else {
if (!page) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
} else {
- ret = do_huge_pmd_wp_page_fallback(fe, orig_pmd, page);
+ ret = do_huge_pmd_wp_page_fallback(vmf, orig_pmd, page);
if (ret & VM_FAULT_OOM) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
}
put_page(page);
@@ -1059,7 +1059,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
huge_gfp, &memcg, true))) {
put_page(new_page);
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
if (page)
put_page(page);
ret |= VM_FAULT_FALLBACK;
@@ -1079,11 +1079,11 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- spin_lock(fe->ptl);
+ spin_lock(vmf->ptl);
if (page)
put_page(page);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd))) {
- spin_unlock(fe->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(new_page, memcg, true);
put_page(new_page);
goto out_mn;
@@ -1091,12 +1091,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = mk_huge_pmd(new_page, vma->vm_page_prot);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
page_add_new_anon_rmap(new_page, vma, haddr, true);
mem_cgroup_commit_charge(new_page, memcg, false, true);
lru_cache_add_active_or_unevictable(new_page, vma);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
if (!page) {
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
} else {
@@ -1106,13 +1106,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
}
ret |= VM_FAULT_WRITE;
}
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out_mn:
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
out:
return ret;
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
@@ -1185,12 +1185,12 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
}
/* NUMA hinting page fault entry point for trans huge pmds */
-int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
+int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct anon_vma *anon_vma = NULL;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
int page_nid = -1, this_nid = numa_node_id();
int target_nid, last_cpupid = -1;
bool page_locked;
@@ -1198,8 +1198,8 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
bool was_writable;
int flags = 0;
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(pmd, *fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd)))
goto out_unlock;
/*
@@ -1207,9 +1207,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* without disrupting NUMA hinting information. Do not relock and
* check_same as the page may no longer be mapped.
*/
- if (unlikely(pmd_trans_migrating(*fe->pmd))) {
- page = pmd_page(*fe->pmd);
- spin_unlock(fe->ptl);
+ if (unlikely(pmd_trans_migrating(*vmf->pmd))) {
+ page = pmd_page(*vmf->pmd);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
goto out;
}
@@ -1242,7 +1242,7 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
/* Migration could have started since the pmd_trans_migrating check */
if (!page_locked) {
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
page_nid = -1;
goto out;
@@ -1253,12 +1253,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* to serialises splits
*/
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
anon_vma = page_lock_anon_vma_read(page);
/* Confirm the PMD did not change while page_table_lock was released */
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(pmd, *fe->pmd))) {
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd))) {
unlock_page(page);
put_page(page);
page_nid = -1;
@@ -1276,9 +1276,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* Migrate the THP to the requested node, returns with page unlocked
* and access rights restored.
*/
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma,
- fe->pmd, pmd, fe->address, page, target_nid);
+ vmf->pmd, pmd, vmf->address, page, target_nid);
if (migrated) {
flags |= TNF_MIGRATED;
page_nid = target_nid;
@@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
pmd = pmd_mkyoung(pmd);
if (was_writable)
pmd = pmd_mkwrite(pmd);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, pmd);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
unlock_page(page);
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out:
if (anon_vma)
page_unlock_anon_vma_read(anon_vma);
if (page_nid != -1)
- task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, fe->flags);
+ task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
+ vmf->flags);
return 0;
}
diff --git a/mm/internal.h b/mm/internal.h
index 537ac9951f5f..093b1eacc91b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct fault_env *fe, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 728d7790dc2d..f88b2d3810a7 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -875,7 +875,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
{
pte_t pteval;
int swapped_in = 0, ret = 0;
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
@@ -887,19 +887,19 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
}
- fe.pte = pte_offset_map(pmd, address);
- for (; fe.address < address + HPAGE_PMD_NR*PAGE_SIZE;
- fe.pte++, fe.address += PAGE_SIZE) {
- pteval = *fe.pte;
+ vmf.pte = pte_offset_map(pmd, address);
+ for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
+ vmf.pte++, vmf.address += PAGE_SIZE) {
+ pteval = *vmf.pte;
if (!is_swap_pte(pteval))
continue;
swapped_in++;
- ret = do_swap_page(&fe, pteval);
+ ret = do_swap_page(&vmf, pteval);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
down_read(&mm->mmap_sem);
- if (hugepage_vma_revalidate(mm, address, &fe.vma)) {
+ if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
/* vma is no longer available, don't continue to swapin */
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
@@ -913,10 +913,10 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
return false;
}
/* pte is unmapped now, we need to map it */
- fe.pte = pte_offset_map(pmd, fe.address);
+ vmf.pte = pte_offset_map(pmd, vmf.address);
}
- fe.pte--;
- pte_unmap(fe.pte);
+ vmf.pte--;
+ pte_unmap(vmf.pte);
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
return true;
}
diff --git a/mm/memory.c b/mm/memory.c
index e18c57bdc75c..fad45cd59ba7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,11 +2074,11 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
+static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
struct page *page, int page_mkwrite, int dirty_shared)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2088,12 +2088,12 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = pte_mkyoung(orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
- if (ptep_set_access_flags(vma, fe->address, fe->pte, entry, 1))
- update_mmu_cache(vma, fe->address, fe->pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
+ update_mmu_cache(vma, vmf->address, vmf->pte);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
struct address_space *mapping;
@@ -2139,15 +2139,15 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
- const unsigned long mmun_start = fe->address & PAGE_MASK;
+ const unsigned long mmun_start = vmf->address & PAGE_MASK;
const unsigned long mmun_end = mmun_start + PAGE_SIZE;
struct mem_cgroup *memcg;
@@ -2155,15 +2155,16 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
goto oom;
if (is_zero_pfn(pte_pfn(orig_pte))) {
- new_page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ new_page = alloc_zeroed_user_highpage_movable(vma,
+ vmf->address);
if (!new_page)
goto oom;
} else {
new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
- fe->address);
+ vmf->address);
if (!new_page)
goto oom;
- cow_user_page(new_page, old_page, fe->address, vma);
+ cow_user_page(new_page, old_page, vmf->address, vma);
}
if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false))
@@ -2176,8 +2177,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
/*
* Re-check the pte - we dropped the lock
*/
- fe->pte = pte_offset_map_lock(mm, fe->pmd, fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte))) {
+ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2187,7 +2188,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2196,8 +2197,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* seen in the presence of one thread doing SMC and another
* thread doing COW.
*/
- ptep_clear_flush_notify(vma, fe->address, fe->pte);
- page_add_new_anon_rmap(new_page, vma, fe->address, false);
+ ptep_clear_flush_notify(vma, vmf->address, vmf->pte);
+ page_add_new_anon_rmap(new_page, vma, vmf->address, false);
mem_cgroup_commit_charge(new_page, memcg, false, false);
lru_cache_add_active_or_unevictable(new_page, vma);
/*
@@ -2205,8 +2206,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* mmu page tables (such as kvm shadow page tables), we want the
* new page to be mapped directly into the secondary page table.
*/
- set_pte_at_notify(mm, fe->address, fe->pte, entry);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at_notify(mm, vmf->address, vmf->pte, entry);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
if (old_page) {
/*
* Only after switching the pte to the new page may
@@ -2243,7 +2244,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
if (new_page)
put_page(new_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
if (old_page) {
/*
@@ -2271,43 +2272,43 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct fault_env *fe, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf = {
+ struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, fe->address),
+ .pgoff = linear_page_index(vma, vmf->address),
.virtual_address =
- (void __user *)(fe->address & PAGE_MASK),
+ (void __user *)(vmf->address & PAGE_MASK),
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
- pte_unmap_unlock(fe->pte, fe->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
if (ret & VM_FAULT_ERROR)
return ret;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
/*
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*fe->pte, orig_pte)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(fe, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
}
-static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
get_page(old_page);
@@ -2315,8 +2316,8 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
int tmp;
- pte_unmap_unlock(fe->pte, fe->ptl);
- tmp = do_page_mkwrite(vma, old_page, fe->address);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ tmp = do_page_mkwrite(vma, old_page, vmf->address);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -2328,18 +2329,18 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* they did, we just return, as we can count on the
* MMU to tell us if they didn't also make it writable.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(fe, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
}
/*
@@ -2360,13 +2361,13 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
- __releases(fe->ptl)
+static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, fe->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2377,10 +2378,10 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(fe, orig_pte);
+ return wp_pfn_shared(vmf, orig_pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
/*
@@ -2391,13 +2392,13 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
int total_mapcount;
if (!trylock_page(old_page)) {
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
lock_page(old_page);
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
@@ -2415,12 +2416,12 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(fe, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(fe, orig_pte, old_page);
+ return wp_page_shared(vmf, orig_pte, old_page);
}
/*
@@ -2428,8 +2429,8 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2517,9 +2518,9 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct fault_env *fe, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
struct mem_cgroup *memcg;
swp_entry_t entry;
@@ -2528,17 +2529,18 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, fe->pmd, fe->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
goto out;
entry = pte_to_swp_entry(orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
- migration_entry_wait(vma->vm_mm, fe->pmd, fe->address);
+ migration_entry_wait(vma->vm_mm, vmf->pmd,
+ vmf->address);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, fe->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2546,16 +2548,16 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
delayacct_set_flag(DELAYACCT_PF_SWAPIN);
page = lookup_swap_cache(entry);
if (!page) {
- page = swapin_readahead(entry,
- GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vma,
+ vmf->address);
if (!page) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2577,7 +2579,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
}
swapcache = page;
- locked = lock_page_or_retry(page, vma->vm_mm, fe->flags);
+ locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
if (!locked) {
@@ -2594,7 +2596,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
if (unlikely(!PageSwapCache(page) || page_private(page) != entry.val))
goto out_page;
- page = ksm_might_need_to_copy(page, vma, fe->address);
+ page = ksm_might_need_to_copy(page, vma, vmf->address);
if (unlikely(!page)) {
ret = VM_FAULT_OOM;
page = swapcache;
@@ -2610,9 +2612,9 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
/*
* Back out if somebody else already faulted in this pte.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (unlikely(!pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2633,22 +2635,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
pte = mk_pte(page, vma->vm_page_prot);
- if ((fe->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
+ if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
pte = maybe_mkwrite(pte_mkdirty(pte), vma);
- fe->flags &= ~FAULT_FLAG_WRITE;
+ vmf->flags &= ~FAULT_FLAG_WRITE;
ret |= VM_FAULT_WRITE;
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
if (pte_swp_soft_dirty(orig_pte))
pte = pte_mksoft_dirty(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
if (page == swapcache) {
- do_page_add_anon_rmap(page, vma, fe->address, exclusive);
+ do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
activate_page(page);
} else { /* ksm created a completely new copy */
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
}
@@ -2671,22 +2673,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
put_page(swapcache);
}
- if (fe->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(fe, pte);
+ if (vmf->flags & FAULT_FLAG_WRITE) {
+ ret |= do_wp_page(vmf, pte);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
}
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
return ret;
out_nomap:
mem_cgroup_cancel_charge(page, memcg, false);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out_page:
unlock_page(page);
out_release:
@@ -2737,9 +2739,9 @@ static inline int check_stack_guard_page(struct vm_area_struct *vma, unsigned lo
* but allow concurrent faults), and pte mapped but not yet locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_anonymous_page(struct fault_env *fe)
+static int do_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
struct page *page;
pte_t entry;
@@ -2749,7 +2751,7 @@ static int do_anonymous_page(struct fault_env *fe)
return VM_FAULT_SIGBUS;
/* Check if we need to add a guard page to the stack */
- if (check_stack_guard_page(vma, fe->address) < 0)
+ if (check_stack_guard_page(vma, vmf->address) < 0)
return VM_FAULT_SIGSEGV;
/*
@@ -2762,26 +2764,26 @@ static int do_anonymous_page(struct fault_env *fe)
*
* Here we only have down_read(mmap_sem).
*/
- if (pte_alloc(vma->vm_mm, fe->pmd, fe->address))
+ if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
return VM_FAULT_OOM;
/* See the comment in pte_alloc_one_map() */
- if (unlikely(pmd_trans_unstable(fe->pmd)))
+ if (unlikely(pmd_trans_unstable(vmf->pmd)))
return 0;
/* Use the zero-page for reads */
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm)) {
- entry = pte_mkspecial(pfn_pte(my_zero_pfn(fe->address),
+ entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
vma->vm_page_prot));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto unlock;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
goto setpte;
}
@@ -2789,7 +2791,7 @@ static int do_anonymous_page(struct fault_env *fe)
/* Allocate our own private page. */
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
if (!page)
goto oom;
@@ -2807,30 +2809,30 @@ static int do_anonymous_page(struct fault_env *fe)
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto release;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, false);
put_page(page);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
setpte:
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
release:
mem_cgroup_cancel_charge(page, memcg, false);
@@ -2847,62 +2849,62 @@ static int do_anonymous_page(struct fault_env *fe)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct fault_env *fe, pgoff_t pgoff,
+static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct page *cow_page, struct page **page, void **entry)
{
- struct vm_area_struct *vma = fe->vma;
- struct vm_fault vmf;
+ struct vm_area_struct *vma = vmf->vma;
+ struct vm_fault vmf2;
int ret;
- vmf.virtual_address = (void __user *)(fe->address & PAGE_MASK);
- vmf.pgoff = pgoff;
- vmf.flags = fe->flags;
- vmf.page = NULL;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.cow_page = cow_page;
+ vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.pgoff = pgoff;
+ vmf2.flags = vmf->flags;
+ vmf2.page = NULL;
+ vmf2.gfp_mask = __get_fault_gfp_mask(vma);
+ vmf2.cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf);
+ ret = vma->vm_ops->fault(vma, &vmf2);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf.entry;
+ *entry = vmf2.entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf.page))) {
+ if (unlikely(PageHWPoison(vmf2.page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf.page);
- put_page(vmf.page);
+ unlock_page(vmf2.page);
+ put_page(vmf2.page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf.page);
+ lock_page(vmf2.page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
- *page = vmf.page;
+ *page = vmf2.page;
return ret;
}
-static int pte_alloc_one_map(struct fault_env *fe)
+static int pte_alloc_one_map(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
- if (!pmd_none(*fe->pmd))
+ if (!pmd_none(*vmf->pmd))
goto map_pte;
- if (fe->prealloc_pte) {
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ if (vmf->prealloc_pte) {
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
goto map_pte;
}
atomic_long_inc(&vma->vm_mm->nr_ptes);
- pmd_populate(vma->vm_mm, fe->pmd, fe->prealloc_pte);
- spin_unlock(fe->ptl);
- fe->prealloc_pte = 0;
- } else if (unlikely(pte_alloc(vma->vm_mm, fe->pmd, fe->address))) {
+ pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
+ spin_unlock(vmf->ptl);
+ vmf->prealloc_pte = 0;
+ } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
return VM_FAULT_OOM;
}
map_pte:
@@ -2917,11 +2919,11 @@ static int pte_alloc_one_map(struct fault_env *fe)
* through an atomic read in C, which is what pmd_trans_unstable()
* provides.
*/
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return VM_FAULT_NOPAGE;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
return 0;
}
@@ -2939,11 +2941,11 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
return true;
}
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
pmd_t entry;
int i, ret;
@@ -2953,8 +2955,8 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
ret = VM_FAULT_FALLBACK;
page = compound_head(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd)))
goto out;
for (i = 0; i < HPAGE_PMD_NR; i++)
@@ -2967,19 +2969,19 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
add_mm_counter(vma->vm_mm, MM_FILEPAGES, HPAGE_PMD_NR);
page_add_file_rmap(page, true);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
- update_mmu_cache_pmd(vma, haddr, fe->pmd);
+ update_mmu_cache_pmd(vma, haddr, vmf->pmd);
/* fault is handled */
ret = 0;
count_vm_event(THP_FILE_MAPPED);
out:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
#else
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
BUILD_BUG();
return 0;
@@ -2990,41 +2992,42 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
* alloc_set_pte - setup new PTE entry for given page and add reverse page
* mapping. If needed, the fucntion allocates page table or use pre-allocated.
*
- * @fe: fault environment
+ * @vmf: fault environment
* @memcg: memcg to charge page (only for private mappings)
* @page: page to map
*
- * Caller must take care of unlocking fe->ptl, if fe->pte is non-NULL on return.
+ * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
+ * return.
*
* Target users are page handler itself and implementations of
* vm_ops->map_pages.
*/
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
pte_t entry;
int ret;
- if (pmd_none(*fe->pmd) && PageTransCompound(page) &&
+ if (pmd_none(*vmf->pmd) && PageTransCompound(page) &&
IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE)) {
/* THP on COW? */
VM_BUG_ON_PAGE(memcg, page);
- ret = do_set_pmd(fe, page);
+ ret = do_set_pmd(vmf, page);
if (ret != VM_FAULT_FALLBACK)
return ret;
}
- if (!fe->pte) {
- ret = pte_alloc_one_map(fe);
+ if (!vmf->pte) {
+ ret = pte_alloc_one_map(vmf);
if (ret)
return ret;
}
/* Re-check under ptl */
- if (unlikely(!pte_none(*fe->pte)))
+ if (unlikely(!pte_none(*vmf->pte)))
return VM_FAULT_NOPAGE;
flush_icache_page(vma, page);
@@ -3034,17 +3037,17 @@ int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
} else {
inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page));
page_add_file_rmap(page, false);
}
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* no need to invalidate: a not-present page won't be cached */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
return 0;
}
@@ -3113,17 +3116,17 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
{
- unsigned long address = fe->address, nr_pages, mask;
+ unsigned long address = vmf->address, nr_pages, mask;
pgoff_t end_pgoff;
int off, ret = 0;
nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
- fe->address = max(address & mask, fe->vma->vm_start);
- off = ((address - fe->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
+ vmf->address = max(address & mask, vmf->vma->vm_start);
+ off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
start_pgoff -= off;
/*
@@ -3131,49 +3134,51 @@ static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
* or fault_around_pages() from start_pgoff, depending what is nearest.
*/
end_pgoff = start_pgoff -
- ((fe->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
+ ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
PTRS_PER_PTE - 1;
- end_pgoff = min3(end_pgoff, vma_pages(fe->vma) + fe->vma->vm_pgoff - 1,
+ end_pgoff = min3(end_pgoff,
+ vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
start_pgoff + nr_pages - 1);
- if (pmd_none(*fe->pmd)) {
- fe->prealloc_pte = pte_alloc_one(fe->vma->vm_mm, fe->address);
- if (!fe->prealloc_pte)
+ if (pmd_none(*vmf->pmd)) {
+ vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
+ vmf->address);
+ if (!vmf->prealloc_pte)
goto out;
smp_wmb(); /* See comment in __pte_alloc() */
}
- fe->vma->vm_ops->map_pages(fe, start_pgoff, end_pgoff);
+ vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
/* preallocated pagetable is unused: free it */
- if (fe->prealloc_pte) {
- pte_free(fe->vma->vm_mm, fe->prealloc_pte);
- fe->prealloc_pte = 0;
+ if (vmf->prealloc_pte) {
+ pte_free(vmf->vma->vm_mm, vmf->prealloc_pte);
+ vmf->prealloc_pte = 0;
}
/* Huge page is mapped? Page fault is solved */
- if (pmd_trans_huge(*fe->pmd)) {
+ if (pmd_trans_huge(*vmf->pmd)) {
ret = VM_FAULT_NOPAGE;
goto out;
}
/* ->map_pages() haven't done anything useful. Cold page cache? */
- if (!fe->pte)
+ if (!vmf->pte)
goto out;
/* check if the page fault is solved */
- fe->pte -= (fe->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
- if (!pte_none(*fe->pte))
+ vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
+ if (!pte_none(*vmf->pte))
ret = VM_FAULT_NOPAGE;
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
- fe->address = address;
- fe->pte = NULL;
+ vmf->address = address;
+ vmf->pte = NULL;
return ret;
}
-static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
int ret = 0;
@@ -3183,27 +3188,27 @@ static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(fe, pgoff);
+ ret = do_fault_around(vmf, pgoff);
if (ret)
return ret;
}
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
unlock_page(fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(fault_page);
return ret;
}
-static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
void *fault_entry;
struct mem_cgroup *memcg;
@@ -3212,7 +3217,7 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
if (unlikely(anon_vma_prepare(vma)))
return VM_FAULT_OOM;
- new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
if (!new_page)
return VM_FAULT_OOM;
@@ -3222,17 +3227,17 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(fe, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, fe->address, vma);
+ copy_user_highpage(new_page, fault_page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(fe, memcg, new_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, memcg, new_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(fault_page);
put_page(fault_page);
@@ -3248,15 +3253,15 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3266,7 +3271,7 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, fe->address);
+ tmp = do_page_mkwrite(vma, fault_page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(fault_page);
@@ -3274,9 +3279,9 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
}
}
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(fault_page);
@@ -3314,19 +3319,19 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
* The mmap_sem may have been released depending on flags and our
* return value. See filemap_fault() and __lock_page_or_retry().
*/
-static int do_fault(struct fault_env *fe)
+static int do_fault(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
- pgoff_t pgoff = linear_page_index(vma, fe->address);
+ struct vm_area_struct *vma = vmf->vma;
+ pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
- if (!(fe->flags & FAULT_FLAG_WRITE))
- return do_read_fault(fe, pgoff);
+ if (!(vmf->flags & FAULT_FLAG_WRITE))
+ return do_read_fault(vmf, pgoff);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(fe, pgoff);
- return do_shared_fault(fe, pgoff);
+ return do_cow_fault(vmf, pgoff);
+ return do_shared_fault(vmf, pgoff);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3344,9 +3349,9 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct fault_env *fe, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf, pte_t pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
int page_nid = -1;
int last_cpupid;
@@ -3364,10 +3369,10 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
* page table entry is not accessible, so there would be no
* concurrent hardware modifications to the PTE.
*/
- fe->ptl = pte_lockptr(vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, pte))) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, pte))) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
goto out;
}
@@ -3376,18 +3381,18 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
pte = pte_mkyoung(pte);
if (was_writable)
pte = pte_mkwrite(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
- page = vm_normal_page(vma, fe->address, pte);
+ page = vm_normal_page(vma, vmf->address, pte);
if (!page) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
/* TODO: handle PTE-mapped THP */
if (PageCompound(page)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3411,9 +3416,9 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
last_cpupid = page_cpupid_last(page);
page_nid = page_to_nid(page);
- target_nid = numa_migrate_prep(page, vma, fe->address, page_nid,
+ target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
&flags);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (target_nid == -1) {
put_page(page);
goto out;
@@ -3433,28 +3438,28 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
return 0;
}
-static int create_huge_pmd(struct fault_env *fe)
+static int create_huge_pmd(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma_is_anonymous(vma))
- return do_huge_pmd_anonymous_page(fe);
+ return do_huge_pmd_anonymous_page(vmf);
if (vma->vm_ops->pmd_fault)
- return vma->vm_ops->pmd_fault(vma, fe->address, fe->pmd,
- fe->flags);
+ return vma->vm_ops->pmd_fault(vma, vmf->address, vmf->pmd,
+ vmf->flags);
return VM_FAULT_FALLBACK;
}
-static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
+static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
{
- if (vma_is_anonymous(fe->vma))
- return do_huge_pmd_wp_page(fe, orig_pmd);
- if (fe->vma->vm_ops->pmd_fault)
- return fe->vma->vm_ops->pmd_fault(fe->vma, fe->address, fe->pmd,
- fe->flags);
+ if (vma_is_anonymous(vmf->vma))
+ return do_huge_pmd_wp_page(vmf, orig_pmd);
+ if (vmf->vma->vm_ops->pmd_fault)
+ return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
+ vmf->pmd, vmf->flags);
/* COW handled on pte level: split pmd */
- VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
- split_huge_pmd(fe->vma, fe->pmd, fe->address);
+ VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
+ split_huge_pmd(vmf->vma, vmf->pmd, vmf->address);
return VM_FAULT_FALLBACK;
}
@@ -3479,21 +3484,21 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
* The mmap_sem may have been released depending on flags and our return value.
* See filemap_fault() and __lock_page_or_retry().
*/
-static int handle_pte_fault(struct fault_env *fe)
+static int handle_pte_fault(struct vm_fault *vmf)
{
pte_t entry;
- if (unlikely(pmd_none(*fe->pmd))) {
+ if (unlikely(pmd_none(*vmf->pmd))) {
/*
* Leave __pte_alloc() until later: because vm_ops->fault may
* want to allocate huge page, and if we expose page table
* for an instant, it will be difficult to retract from
* concurrent faults and from rmap lookups.
*/
- fe->pte = NULL;
+ vmf->pte = NULL;
} else {
/* See comment in pte_alloc_one_map() */
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return 0;
/*
* A regular pmd is established and it can't morph into a huge
@@ -3501,9 +3506,9 @@ static int handle_pte_fault(struct fault_env *fe)
* mmap_sem read mode and khugepaged takes it in write mode.
* So now it's safe to run pte_offset_map().
*/
- fe->pte = pte_offset_map(fe->pmd, fe->address);
+ vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
- entry = *fe->pte;
+ entry = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3515,37 +3520,37 @@ static int handle_pte_fault(struct fault_env *fe)
*/
barrier();
if (pte_none(entry)) {
- pte_unmap(fe->pte);
- fe->pte = NULL;
+ pte_unmap(vmf->pte);
+ vmf->pte = NULL;
}
}
- if (!fe->pte) {
- if (vma_is_anonymous(fe->vma))
- return do_anonymous_page(fe);
+ if (!vmf->pte) {
+ if (vma_is_anonymous(vmf->vma))
+ return do_anonymous_page(vmf);
else
- return do_fault(fe);
+ return do_fault(vmf);
}
if (!pte_present(entry))
- return do_swap_page(fe, entry);
+ return do_swap_page(vmf, entry);
- if (pte_protnone(entry) && vma_is_accessible(fe->vma))
- return do_numa_page(fe, entry);
+ if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf, entry);
- fe->ptl = pte_lockptr(fe->vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, entry)))
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
- if (fe->flags & FAULT_FLAG_WRITE) {
+ if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(fe, entry);
+ return do_wp_page(vmf, entry);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
- if (ptep_set_access_flags(fe->vma, fe->address, fe->pte, entry,
- fe->flags & FAULT_FLAG_WRITE)) {
- update_mmu_cache(fe->vma, fe->address, fe->pte);
+ if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
+ vmf->flags & FAULT_FLAG_WRITE)) {
+ update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
} else {
/*
* This is needed only for protection faults but the arch code
@@ -3553,11 +3558,11 @@ static int handle_pte_fault(struct fault_env *fe)
* This still avoids useless tlb flushes for .text page faults
* with threads.
*/
- if (fe->flags & FAULT_FLAG_WRITE)
- flush_tlb_fix_spurious_fault(fe->vma, fe->address);
+ if (vmf->flags & FAULT_FLAG_WRITE)
+ flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
}
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3570,7 +3575,7 @@ static int handle_pte_fault(struct fault_env *fe)
static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
unsigned int flags)
{
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = flags,
@@ -3583,35 +3588,35 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
pud = pud_alloc(mm, pgd, address);
if (!pud)
return VM_FAULT_OOM;
- fe.pmd = pmd_alloc(mm, pud, address);
- if (!fe.pmd)
+ vmf.pmd = pmd_alloc(mm, pud, address);
+ if (!vmf.pmd)
return VM_FAULT_OOM;
- if (pmd_none(*fe.pmd) && transparent_hugepage_enabled(vma)) {
- int ret = create_huge_pmd(&fe);
+ if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
+ int ret = create_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- pmd_t orig_pmd = *fe.pmd;
+ pmd_t orig_pmd = *vmf.pmd;
int ret;
barrier();
if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
if (pmd_protnone(orig_pmd) && vma_is_accessible(vma))
- return do_huge_pmd_numa_page(&fe, orig_pmd);
+ return do_huge_pmd_numa_page(&vmf, orig_pmd);
- if ((fe.flags & FAULT_FLAG_WRITE) &&
+ if ((vmf.flags & FAULT_FLAG_WRITE) &&
!pmd_write(orig_pmd)) {
- ret = wp_huge_pmd(&fe, orig_pmd);
+ ret = wp_huge_pmd(&vmf, orig_pmd);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- huge_pmd_set_accessed(&fe, orig_pmd);
+ huge_pmd_set_accessed(&vmf, orig_pmd);
return 0;
}
}
}
- return handle_pte_fault(&fe);
+ return handle_pte_fault(&vmf);
}
/*
diff --git a/mm/nommu.c b/mm/nommu.c
index 8b8faaf2a9e9..077d0dbe4c28 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1801,7 +1801,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
BUG();
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we have two different structures for passing fault information
around - struct vm_fault and struct fault_env. DAX will need more
information in struct vm_fault to handle its faults so the content of
that structure would become event closer to fault_env. Furthermore it
would need to generate struct fault_env to be able to call some of the
generic functions. So at this point I don't think there's much use in
keeping these two structures separate. Just embed into struct vm_fault
all that is needed to use it for both purposes.
Signed-off-by: Jan Kara <jack@suse.cz>
---
Documentation/filesystems/Locking | 2 +-
fs/userfaultfd.c | 22 +-
include/linux/huge_mm.h | 10 +-
include/linux/mm.h | 28 +-
include/linux/userfaultfd_k.h | 4 +-
mm/filemap.c | 14 +-
mm/huge_memory.c | 173 ++++++------
mm/internal.h | 2 +-
mm/khugepaged.c | 20 +-
mm/memory.c | 549 +++++++++++++++++++-------------------
mm/nommu.c | 2 +-
11 files changed, 414 insertions(+), 412 deletions(-)
diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
index 14cdc101d165..ac3d080eabaa 100644
--- a/Documentation/filesystems/Locking
+++ b/Documentation/filesystems/Locking
@@ -557,7 +557,7 @@ till "end_pgoff". ->map_pages() is called with page table locked and must
not block. If it's not possible to reach a page without blocking,
filesystem should skip it. Filesystem should use do_set_pte() to setup
page table entry. Pointer to entry associated with the page is passed in
-"pte" field in fault_env structure. Pointers to entries for other offsets
+"pte" field in vm_fault structure. Pointers to entries for other offsets
should be calculated relative to "pte".
->page_mkwrite() is called when a previously read-only pte is
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 85959d8324df..d96e2f30084b 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -257,9 +257,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
* fatal_signal_pending()s, and the mmap_sem must be released before
* returning it.
*/
-int handle_userfault(struct fault_env *fe, unsigned long reason)
+int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
- struct mm_struct *mm = fe->vma->vm_mm;
+ struct mm_struct *mm = vmf->vma->vm_mm;
struct userfaultfd_ctx *ctx;
struct userfaultfd_wait_queue uwq;
int ret;
@@ -268,7 +268,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
ret = VM_FAULT_SIGBUS;
- ctx = fe->vma->vm_userfaultfd_ctx.ctx;
+ ctx = vmf->vma->vm_userfaultfd_ctx.ctx;
if (!ctx)
goto out;
@@ -301,17 +301,18 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* without first stopping userland access to the memory. For
* VM_UFFD_MISSING userfaults this is enough for now.
*/
- if (unlikely(!(fe->flags & FAULT_FLAG_ALLOW_RETRY))) {
+ if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
/*
* Validate the invariant that nowait must allow retry
* to be sure not to return SIGBUS erroneously on
* nowait invocations.
*/
- BUG_ON(fe->flags & FAULT_FLAG_RETRY_NOWAIT);
+ BUG_ON(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
#ifdef CONFIG_DEBUG_VM
if (printk_ratelimit()) {
printk(KERN_WARNING
- "FAULT_FLAG_ALLOW_RETRY missing %x\n", fe->flags);
+ "FAULT_FLAG_ALLOW_RETRY missing %x\n",
+ vmf->flags);
dump_stack();
}
#endif
@@ -323,7 +324,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
* and wait.
*/
ret = VM_FAULT_RETRY;
- if (fe->flags & FAULT_FLAG_RETRY_NOWAIT)
+ if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
goto out;
/* take the reference before dropping the mmap_sem */
@@ -331,11 +332,11 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
uwq.wq.private = current;
- uwq.msg = userfault_msg(fe->address, fe->flags, reason);
+ uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
uwq.ctx = ctx;
return_to_userland =
- (fe->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
+ (vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
(FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
spin_lock(&ctx->fault_pending_wqh.lock);
@@ -353,7 +354,8 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
TASK_KILLABLE);
spin_unlock(&ctx->fault_pending_wqh.lock);
- must_wait = userfaultfd_must_wait(ctx, fe->address, fe->flags, reason);
+ must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
+ reason);
up_read(&mm->mmap_sem);
if (likely(must_wait && !ACCESS_ONCE(ctx->released) &&
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 9b9f65d99873..3237a39f94a4 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -1,12 +1,12 @@
#ifndef _LINUX_HUGE_MM_H
#define _LINUX_HUGE_MM_H
-extern int do_huge_pmd_anonymous_page(struct fault_env *fe);
+extern int do_huge_pmd_anonymous_page(struct vm_fault *vmf);
extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
struct vm_area_struct *vma);
-extern void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd);
-extern int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd);
+extern void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
+extern int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
unsigned long addr,
pmd_t *pmd,
@@ -142,7 +142,7 @@ static inline int hpage_nr_pages(struct page *page)
return 1;
}
-extern int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd);
+extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
extern struct page *huge_zero_page;
@@ -210,7 +210,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
return NULL;
}
-static inline int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd)
+static inline int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
return 0;
}
diff --git a/include/linux/mm.h b/include/linux/mm.h
index a92c8d73aeaf..657eb69eb87e 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
* pgoff should be used in favour of virtual_address, if possible.
*/
struct vm_fault {
+ struct vm_area_struct *vma; /* Target VMA */
unsigned int flags; /* FAULT_FLAG_xxx flags */
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
- void __user *virtual_address; /* Faulting virtual address */
+ unsigned long address; /* Faulting virtual address */
+ void __user *virtual_address; /* Faulting virtual address masked by
+ * PAGE_MASK */
+ pmd_t *pmd; /* Pointer to pmd entry matching
+ * the 'address'
+ */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
@@ -309,19 +315,7 @@ struct vm_fault {
* VM_FAULT_DAX_LOCKED and fill in
* entry here.
*/
-};
-
-/*
- * Page fault context: passes though page fault handler instead of endless list
- * of function arguments.
- */
-struct fault_env {
- struct vm_area_struct *vma; /* Target VMA */
- unsigned long address; /* Faulting virtual address */
- unsigned int flags; /* FAULT_FLAG_xxx flags */
- pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ /* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
* table hasn't been allocated.
@@ -351,7 +345,7 @@ struct vm_operations_struct {
int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf);
int (*pmd_fault)(struct vm_area_struct *, unsigned long address,
pmd_t *, unsigned int flags);
- void (*map_pages)(struct fault_env *fe,
+ void (*map_pages)(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
/* notification that a previously read-only page is about to become
@@ -625,7 +619,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
return pte;
}
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
#endif
@@ -2097,7 +2091,7 @@ extern void truncate_inode_pages_final(struct address_space *);
/* generic vm_area_ops exported for stackable file systems */
extern int filemap_fault(struct vm_area_struct *, struct vm_fault *);
-extern void filemap_map_pages(struct fault_env *fe,
+extern void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff);
extern int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf);
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index dd66a952e8cd..11b92b047a1e 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -27,7 +27,7 @@
#define UFFD_SHARED_FCNTL_FLAGS (O_CLOEXEC | O_NONBLOCK)
#define UFFD_FLAGS_SET (EFD_SHARED_FCNTL_FLAGS)
-extern int handle_userfault(struct fault_env *fe, unsigned long reason);
+extern int handle_userfault(struct vm_fault *vmf, unsigned long reason);
extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,
unsigned long src_start, unsigned long len);
@@ -55,7 +55,7 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
#else /* CONFIG_USERFAULTFD */
/* mm helpers */
-static inline int handle_userfault(struct fault_env *fe, unsigned long reason)
+static inline int handle_userfault(struct vm_fault *vmf, unsigned long reason)
{
return VM_FAULT_SIGBUS;
}
diff --git a/mm/filemap.c b/mm/filemap.c
index db26ebc6c62f..1426fb1a99b3 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -2209,12 +2209,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
struct radix_tree_iter iter;
void **slot;
- struct file *file = fe->vma->vm_file;
+ struct file *file = vmf->vma->vm_file;
struct address_space *mapping = file->f_mapping;
pgoff_t last_pgoff = start_pgoff;
loff_t size;
@@ -2270,11 +2270,11 @@ void filemap_map_pages(struct fault_env *fe,
if (file->f_ra.mmap_miss > 0)
file->f_ra.mmap_miss--;
- fe->address += (iter.index - last_pgoff) << PAGE_SHIFT;
- if (fe->pte)
- fe->pte += iter.index - last_pgoff;
+ vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
+ if (vmf->pte)
+ vmf->pte += iter.index - last_pgoff;
last_pgoff = iter.index;
- if (alloc_set_pte(fe, NULL, page))
+ if (alloc_set_pte(vmf, NULL, page))
goto unlock;
unlock_page(page);
goto next;
@@ -2284,7 +2284,7 @@ void filemap_map_pages(struct fault_env *fe,
put_page(page);
next:
/* Huge page is mapped? No need to proceed. */
- if (pmd_trans_huge(*fe->pmd))
+ if (pmd_trans_huge(*vmf->pmd))
break;
if (iter.index == end_pgoff)
break;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index cdcd25cb30fe..e286b09f9d24 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -532,13 +532,13 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
}
EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
-static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
+static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page,
gfp_t gfp)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
pgtable_t pgtable;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
VM_BUG_ON_PAGE(!PageCompound(page), page);
@@ -563,9 +563,9 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
*/
__SetPageUptodate(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
@@ -576,11 +576,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
if (userfaultfd_missing(vma)) {
int ret;
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, true);
put_page(page);
pte_free(vma->vm_mm, pgtable);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
return ret;
}
@@ -590,11 +590,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
page_add_new_anon_rmap(page, vma, haddr, true);
mem_cgroup_commit_charge(page, memcg, false, true);
lru_cache_add_active_or_unevictable(page, vma);
- pgtable_trans_huge_deposit(vma->vm_mm, fe->pmd, pgtable);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
atomic_long_inc(&vma->vm_mm->nr_ptes);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
count_vm_event(THP_FAULT_ALLOC);
}
@@ -641,12 +641,12 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
return true;
}
-int do_huge_pmd_anonymous_page(struct fault_env *fe)
+int do_huge_pmd_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
gfp_t gfp;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
return VM_FAULT_FALLBACK;
@@ -654,7 +654,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_OOM;
if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
return VM_FAULT_OOM;
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm) &&
transparent_hugepage_use_zero_page()) {
pgtable_t pgtable;
@@ -670,22 +670,22 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
count_vm_event(THP_FAULT_FALLBACK);
return VM_FAULT_FALLBACK;
}
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
ret = 0;
set = false;
- if (pmd_none(*fe->pmd)) {
+ if (pmd_none(*vmf->pmd)) {
if (userfaultfd_missing(vma)) {
- spin_unlock(fe->ptl);
- ret = handle_userfault(fe, VM_UFFD_MISSING);
+ spin_unlock(vmf->ptl);
+ ret = handle_userfault(vmf, VM_UFFD_MISSING);
VM_BUG_ON(ret & VM_FAULT_FALLBACK);
} else {
set_huge_zero_page(pgtable, vma->vm_mm, vma,
- haddr, fe->pmd, zero_page);
- spin_unlock(fe->ptl);
+ haddr, vmf->pmd, zero_page);
+ spin_unlock(vmf->ptl);
set = true;
}
} else
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
if (!set)
pte_free(vma->vm_mm, pgtable);
return ret;
@@ -697,7 +697,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
return VM_FAULT_FALLBACK;
}
prep_transhuge_page(page);
- return __do_huge_pmd_anonymous_page(fe, page, gfp);
+ return __do_huge_pmd_anonymous_page(vmf, page, gfp);
}
static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
@@ -868,30 +868,30 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
return ret;
}
-void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd)
+void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)
{
pmd_t entry;
unsigned long haddr;
- fe->ptl = pmd_lock(fe->vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto unlock;
entry = pmd_mkyoung(orig_pmd);
- haddr = fe->address & HPAGE_PMD_MASK;
- if (pmdp_set_access_flags(fe->vma, haddr, fe->pmd, entry,
- fe->flags & FAULT_FLAG_WRITE))
- update_mmu_cache_pmd(fe->vma, fe->address, fe->pmd);
+ haddr = vmf->address & HPAGE_PMD_MASK;
+ if (pmdp_set_access_flags(vmf->vma, haddr, vmf->pmd, entry,
+ vmf->flags & FAULT_FLAG_WRITE))
+ update_mmu_cache_pmd(vmf->vma, vmf->address, vmf->pmd);
unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
}
-static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
+static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
struct mem_cgroup *memcg;
pgtable_t pgtable;
pmd_t _pmd;
@@ -910,7 +910,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
for (i = 0; i < HPAGE_PMD_NR; i++) {
pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE |
__GFP_OTHER_NODE, vma,
- fe->address, page_to_nid(page));
+ vmf->address, page_to_nid(page));
if (unlikely(!pages[i] ||
mem_cgroup_try_charge(pages[i], vma->vm_mm,
GFP_KERNEL, &memcg, false))) {
@@ -941,15 +941,15 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_free_pages;
VM_BUG_ON_PAGE(!PageHead(page), page);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
/* leave pmd empty until pte is filled */
- pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, fe->pmd);
+ pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, vmf->pmd);
pmd_populate(vma->vm_mm, &_pmd, pgtable);
for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
@@ -958,20 +958,20 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
memcg = (void *)page_private(pages[i]);
set_page_private(pages[i], 0);
- page_add_new_anon_rmap(pages[i], fe->vma, haddr, false);
+ page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false);
mem_cgroup_commit_charge(pages[i], memcg, false, false);
lru_cache_add_active_or_unevictable(pages[i], vma);
- fe->pte = pte_offset_map(&_pmd, haddr);
- VM_BUG_ON(!pte_none(*fe->pte));
- set_pte_at(vma->vm_mm, haddr, fe->pte, entry);
- pte_unmap(fe->pte);
+ vmf->pte = pte_offset_map(&_pmd, haddr);
+ VM_BUG_ON(!pte_none(*vmf->pte));
+ set_pte_at(vma->vm_mm, haddr, vmf->pte, entry);
+ pte_unmap(vmf->pte);
}
kfree(pages);
smp_wmb(); /* make pte visible before pmd */
- pmd_populate(vma->vm_mm, fe->pmd, pgtable);
+ pmd_populate(vma->vm_mm, vmf->pmd, pgtable);
page_remove_rmap(page, true);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
@@ -982,7 +982,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
return ret;
out_free_pages:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
for (i = 0; i < HPAGE_PMD_NR; i++) {
memcg = (void *)page_private(pages[i]);
@@ -994,23 +994,23 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
goto out;
}
-int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
+int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL, *new_page;
struct mem_cgroup *memcg;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
unsigned long mmun_start; /* For mmu_notifiers */
unsigned long mmun_end; /* For mmu_notifiers */
gfp_t huge_gfp; /* for allocation and charge */
int ret = 0;
- fe->ptl = pmd_lockptr(vma->vm_mm, fe->pmd);
+ vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd);
VM_BUG_ON_VMA(!vma->anon_vma, vma);
if (is_huge_zero_pmd(orig_pmd))
goto alloc;
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
goto out_unlock;
page = pmd_page(orig_pmd);
@@ -1023,13 +1023,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = pmd_mkyoung(orig_pmd);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- if (pmdp_set_access_flags(vma, haddr, fe->pmd, entry, 1))
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1))
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
ret |= VM_FAULT_WRITE;
goto out_unlock;
}
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
alloc:
if (transparent_hugepage_enabled(vma) &&
!transparent_hugepage_debug_cow()) {
@@ -1042,12 +1042,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
prep_transhuge_page(new_page);
} else {
if (!page) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
} else {
- ret = do_huge_pmd_wp_page_fallback(fe, orig_pmd, page);
+ ret = do_huge_pmd_wp_page_fallback(vmf, orig_pmd, page);
if (ret & VM_FAULT_OOM) {
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
ret |= VM_FAULT_FALLBACK;
}
put_page(page);
@@ -1059,7 +1059,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
huge_gfp, &memcg, true))) {
put_page(new_page);
- split_huge_pmd(vma, fe->pmd, fe->address);
+ split_huge_pmd(vma, vmf->pmd, vmf->address);
if (page)
put_page(page);
ret |= VM_FAULT_FALLBACK;
@@ -1079,11 +1079,11 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
mmun_end = haddr + HPAGE_PMD_SIZE;
mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
- spin_lock(fe->ptl);
+ spin_lock(vmf->ptl);
if (page)
put_page(page);
- if (unlikely(!pmd_same(*fe->pmd, orig_pmd))) {
- spin_unlock(fe->ptl);
+ if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) {
+ spin_unlock(vmf->ptl);
mem_cgroup_cancel_charge(new_page, memcg, true);
put_page(new_page);
goto out_mn;
@@ -1091,12 +1091,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
pmd_t entry;
entry = mk_huge_pmd(new_page, vma->vm_page_prot);
entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
- pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
+ pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
page_add_new_anon_rmap(new_page, vma, haddr, true);
mem_cgroup_commit_charge(new_page, memcg, false, true);
lru_cache_add_active_or_unevictable(new_page, vma);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
if (!page) {
add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
} else {
@@ -1106,13 +1106,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
}
ret |= VM_FAULT_WRITE;
}
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out_mn:
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
out:
return ret;
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
@@ -1185,12 +1185,12 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
}
/* NUMA hinting page fault entry point for trans huge pmds */
-int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
+int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct anon_vma *anon_vma = NULL;
struct page *page;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
int page_nid = -1, this_nid = numa_node_id();
int target_nid, last_cpupid = -1;
bool page_locked;
@@ -1198,8 +1198,8 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
bool was_writable;
int flags = 0;
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_same(pmd, *fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd)))
goto out_unlock;
/*
@@ -1207,9 +1207,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* without disrupting NUMA hinting information. Do not relock and
* check_same as the page may no longer be mapped.
*/
- if (unlikely(pmd_trans_migrating(*fe->pmd))) {
- page = pmd_page(*fe->pmd);
- spin_unlock(fe->ptl);
+ if (unlikely(pmd_trans_migrating(*vmf->pmd))) {
+ page = pmd_page(*vmf->pmd);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
goto out;
}
@@ -1242,7 +1242,7 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
/* Migration could have started since the pmd_trans_migrating check */
if (!page_locked) {
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
wait_on_page_locked(page);
page_nid = -1;
goto out;
@@ -1253,12 +1253,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* to serialises splits
*/
get_page(page);
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
anon_vma = page_lock_anon_vma_read(page);
/* Confirm the PMD did not change while page_table_lock was released */
- spin_lock(fe->ptl);
- if (unlikely(!pmd_same(pmd, *fe->pmd))) {
+ spin_lock(vmf->ptl);
+ if (unlikely(!pmd_same(pmd, *vmf->pmd))) {
unlock_page(page);
put_page(page);
page_nid = -1;
@@ -1276,9 +1276,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
* Migrate the THP to the requested node, returns with page unlocked
* and access rights restored.
*/
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma,
- fe->pmd, pmd, fe->address, page, target_nid);
+ vmf->pmd, pmd, vmf->address, page, target_nid);
if (migrated) {
flags |= TNF_MIGRATED;
page_nid = target_nid;
@@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
pmd = pmd_mkyoung(pmd);
if (was_writable)
pmd = pmd_mkwrite(pmd);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, pmd);
- update_mmu_cache_pmd(vma, fe->address, fe->pmd);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
+ update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
unlock_page(page);
out_unlock:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
out:
if (anon_vma)
page_unlock_anon_vma_read(anon_vma);
if (page_nid != -1)
- task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, fe->flags);
+ task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
+ vmf->flags);
return 0;
}
diff --git a/mm/internal.h b/mm/internal.h
index 537ac9951f5f..093b1eacc91b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct fault_env *fe, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 728d7790dc2d..f88b2d3810a7 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -875,7 +875,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
{
pte_t pteval;
int swapped_in = 0, ret = 0;
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
@@ -887,19 +887,19 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
}
- fe.pte = pte_offset_map(pmd, address);
- for (; fe.address < address + HPAGE_PMD_NR*PAGE_SIZE;
- fe.pte++, fe.address += PAGE_SIZE) {
- pteval = *fe.pte;
+ vmf.pte = pte_offset_map(pmd, address);
+ for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
+ vmf.pte++, vmf.address += PAGE_SIZE) {
+ pteval = *vmf.pte;
if (!is_swap_pte(pteval))
continue;
swapped_in++;
- ret = do_swap_page(&fe, pteval);
+ ret = do_swap_page(&vmf, pteval);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
down_read(&mm->mmap_sem);
- if (hugepage_vma_revalidate(mm, address, &fe.vma)) {
+ if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
/* vma is no longer available, don't continue to swapin */
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
return false;
@@ -913,10 +913,10 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
return false;
}
/* pte is unmapped now, we need to map it */
- fe.pte = pte_offset_map(pmd, fe.address);
+ vmf.pte = pte_offset_map(pmd, vmf.address);
}
- fe.pte--;
- pte_unmap(fe.pte);
+ vmf.pte--;
+ pte_unmap(vmf.pte);
trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
return true;
}
diff --git a/mm/memory.c b/mm/memory.c
index e18c57bdc75c..fad45cd59ba7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,11 +2074,11 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
+static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
struct page *page, int page_mkwrite, int dirty_shared)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2088,12 +2088,12 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = pte_mkyoung(orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
- if (ptep_set_access_flags(vma, fe->address, fe->pte, entry, 1))
- update_mmu_cache(vma, fe->address, fe->pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
+ update_mmu_cache(vma, vmf->address, vmf->pte);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
struct address_space *mapping;
@@ -2139,15 +2139,15 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
- const unsigned long mmun_start = fe->address & PAGE_MASK;
+ const unsigned long mmun_start = vmf->address & PAGE_MASK;
const unsigned long mmun_end = mmun_start + PAGE_SIZE;
struct mem_cgroup *memcg;
@@ -2155,15 +2155,16 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
goto oom;
if (is_zero_pfn(pte_pfn(orig_pte))) {
- new_page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ new_page = alloc_zeroed_user_highpage_movable(vma,
+ vmf->address);
if (!new_page)
goto oom;
} else {
new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
- fe->address);
+ vmf->address);
if (!new_page)
goto oom;
- cow_user_page(new_page, old_page, fe->address, vma);
+ cow_user_page(new_page, old_page, vmf->address, vma);
}
if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false))
@@ -2176,8 +2177,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
/*
* Re-check the pte - we dropped the lock
*/
- fe->pte = pte_offset_map_lock(mm, fe->pmd, fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte))) {
+ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2187,7 +2188,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2196,8 +2197,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* seen in the presence of one thread doing SMC and another
* thread doing COW.
*/
- ptep_clear_flush_notify(vma, fe->address, fe->pte);
- page_add_new_anon_rmap(new_page, vma, fe->address, false);
+ ptep_clear_flush_notify(vma, vmf->address, vmf->pte);
+ page_add_new_anon_rmap(new_page, vma, vmf->address, false);
mem_cgroup_commit_charge(new_page, memcg, false, false);
lru_cache_add_active_or_unevictable(new_page, vma);
/*
@@ -2205,8 +2206,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* mmu page tables (such as kvm shadow page tables), we want the
* new page to be mapped directly into the secondary page table.
*/
- set_pte_at_notify(mm, fe->address, fe->pte, entry);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at_notify(mm, vmf->address, vmf->pte, entry);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
if (old_page) {
/*
* Only after switching the pte to the new page may
@@ -2243,7 +2244,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
if (new_page)
put_page(new_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
if (old_page) {
/*
@@ -2271,43 +2272,43 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct fault_env *fe, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf = {
+ struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, fe->address),
+ .pgoff = linear_page_index(vma, vmf->address),
.virtual_address =
- (void __user *)(fe->address & PAGE_MASK),
+ (void __user *)(vmf->address & PAGE_MASK),
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
- pte_unmap_unlock(fe->pte, fe->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
if (ret & VM_FAULT_ERROR)
return ret;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
/*
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*fe->pte, orig_pte)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(fe, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
}
-static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
+static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
struct page *old_page)
- __releases(fe->ptl)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
get_page(old_page);
@@ -2315,8 +2316,8 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
int tmp;
- pte_unmap_unlock(fe->pte, fe->ptl);
- tmp = do_page_mkwrite(vma, old_page, fe->address);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ tmp = do_page_mkwrite(vma, old_page, vmf->address);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -2328,18 +2329,18 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* they did, we just return, as we can count on the
* MMU to tell us if they didn't also make it writable.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(fe, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
}
/*
@@ -2360,13 +2361,13 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
- __releases(fe->ptl)
+static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+ __releases(vmf->ptl)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, fe->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2377,10 +2378,10 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(fe, orig_pte);
+ return wp_pfn_shared(vmf, orig_pte);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
/*
@@ -2391,13 +2392,13 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
int total_mapcount;
if (!trylock_page(old_page)) {
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
lock_page(old_page);
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (!pte_same(*fe->pte, orig_pte)) {
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_same(*vmf->pte, orig_pte)) {
unlock_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
return 0;
}
@@ -2415,12 +2416,12 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(fe, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(fe, orig_pte, old_page);
+ return wp_page_shared(vmf, orig_pte, old_page);
}
/*
@@ -2428,8 +2429,8 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
*/
get_page(old_page);
- pte_unmap_unlock(fe->pte, fe->ptl);
- return wp_page_copy(fe, orig_pte, old_page);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return wp_page_copy(vmf, orig_pte, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2517,9 +2518,9 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct fault_env *fe, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
struct mem_cgroup *memcg;
swp_entry_t entry;
@@ -2528,17 +2529,18 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, fe->pmd, fe->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
goto out;
entry = pte_to_swp_entry(orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
- migration_entry_wait(vma->vm_mm, fe->pmd, fe->address);
+ migration_entry_wait(vma->vm_mm, vmf->pmd,
+ vmf->address);
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, fe->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2546,16 +2548,16 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
delayacct_set_flag(DELAYACCT_PF_SWAPIN);
page = lookup_swap_cache(entry);
if (!page) {
- page = swapin_readahead(entry,
- GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vma,
+ vmf->address);
if (!page) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
- fe->address, &fe->ptl);
- if (likely(pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (likely(pte_same(*vmf->pte, orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2577,7 +2579,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
}
swapcache = page;
- locked = lock_page_or_retry(page, vma->vm_mm, fe->flags);
+ locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
if (!locked) {
@@ -2594,7 +2596,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
if (unlikely(!PageSwapCache(page) || page_private(page) != entry.val))
goto out_page;
- page = ksm_might_need_to_copy(page, vma, fe->address);
+ page = ksm_might_need_to_copy(page, vma, vmf->address);
if (unlikely(!page)) {
ret = VM_FAULT_OOM;
page = swapcache;
@@ -2610,9 +2612,9 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
/*
* Back out if somebody else already faulted in this pte.
*/
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (unlikely(!pte_same(*fe->pte, orig_pte)))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2633,22 +2635,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
pte = mk_pte(page, vma->vm_page_prot);
- if ((fe->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
+ if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
pte = maybe_mkwrite(pte_mkdirty(pte), vma);
- fe->flags &= ~FAULT_FLAG_WRITE;
+ vmf->flags &= ~FAULT_FLAG_WRITE;
ret |= VM_FAULT_WRITE;
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
if (pte_swp_soft_dirty(orig_pte))
pte = pte_mksoft_dirty(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
if (page == swapcache) {
- do_page_add_anon_rmap(page, vma, fe->address, exclusive);
+ do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
activate_page(page);
} else { /* ksm created a completely new copy */
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
}
@@ -2671,22 +2673,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
put_page(swapcache);
}
- if (fe->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(fe, pte);
+ if (vmf->flags & FAULT_FLAG_WRITE) {
+ ret |= do_wp_page(vmf, pte);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
}
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
return ret;
out_nomap:
mem_cgroup_cancel_charge(page, memcg, false);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out_page:
unlock_page(page);
out_release:
@@ -2737,9 +2739,9 @@ static inline int check_stack_guard_page(struct vm_area_struct *vma, unsigned lo
* but allow concurrent faults), and pte mapped but not yet locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_anonymous_page(struct fault_env *fe)
+static int do_anonymous_page(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct mem_cgroup *memcg;
struct page *page;
pte_t entry;
@@ -2749,7 +2751,7 @@ static int do_anonymous_page(struct fault_env *fe)
return VM_FAULT_SIGBUS;
/* Check if we need to add a guard page to the stack */
- if (check_stack_guard_page(vma, fe->address) < 0)
+ if (check_stack_guard_page(vma, vmf->address) < 0)
return VM_FAULT_SIGSEGV;
/*
@@ -2762,26 +2764,26 @@ static int do_anonymous_page(struct fault_env *fe)
*
* Here we only have down_read(mmap_sem).
*/
- if (pte_alloc(vma->vm_mm, fe->pmd, fe->address))
+ if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
return VM_FAULT_OOM;
/* See the comment in pte_alloc_one_map() */
- if (unlikely(pmd_trans_unstable(fe->pmd)))
+ if (unlikely(pmd_trans_unstable(vmf->pmd)))
return 0;
/* Use the zero-page for reads */
- if (!(fe->flags & FAULT_FLAG_WRITE) &&
+ if (!(vmf->flags & FAULT_FLAG_WRITE) &&
!mm_forbids_zeropage(vma->vm_mm)) {
- entry = pte_mkspecial(pfn_pte(my_zero_pfn(fe->address),
+ entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
vma->vm_page_prot));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+ vmf->address, &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto unlock;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
goto setpte;
}
@@ -2789,7 +2791,7 @@ static int do_anonymous_page(struct fault_env *fe)
/* Allocate our own private page. */
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- page = alloc_zeroed_user_highpage_movable(vma, fe->address);
+ page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
if (!page)
goto oom;
@@ -2807,30 +2809,30 @@ static int do_anonymous_page(struct fault_env *fe)
if (vma->vm_flags & VM_WRITE)
entry = pte_mkwrite(pte_mkdirty(entry));
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
- if (!pte_none(*fe->pte))
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ if (!pte_none(*vmf->pte))
goto release;
/* Deliver the page fault to userland, check inside PT lock */
if (userfaultfd_missing(vma)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
mem_cgroup_cancel_charge(page, memcg, false);
put_page(page);
- return handle_userfault(fe, VM_UFFD_MISSING);
+ return handle_userfault(vmf, VM_UFFD_MISSING);
}
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
setpte:
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* No need to invalidate - it was non-present before */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
release:
mem_cgroup_cancel_charge(page, memcg, false);
@@ -2847,62 +2849,62 @@ static int do_anonymous_page(struct fault_env *fe)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct fault_env *fe, pgoff_t pgoff,
+static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct page *cow_page, struct page **page, void **entry)
{
- struct vm_area_struct *vma = fe->vma;
- struct vm_fault vmf;
+ struct vm_area_struct *vma = vmf->vma;
+ struct vm_fault vmf2;
int ret;
- vmf.virtual_address = (void __user *)(fe->address & PAGE_MASK);
- vmf.pgoff = pgoff;
- vmf.flags = fe->flags;
- vmf.page = NULL;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.cow_page = cow_page;
+ vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.pgoff = pgoff;
+ vmf2.flags = vmf->flags;
+ vmf2.page = NULL;
+ vmf2.gfp_mask = __get_fault_gfp_mask(vma);
+ vmf2.cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf);
+ ret = vma->vm_ops->fault(vma, &vmf2);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf.entry;
+ *entry = vmf2.entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf.page))) {
+ if (unlikely(PageHWPoison(vmf2.page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf.page);
- put_page(vmf.page);
+ unlock_page(vmf2.page);
+ put_page(vmf2.page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf.page);
+ lock_page(vmf2.page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
- *page = vmf.page;
+ *page = vmf2.page;
return ret;
}
-static int pte_alloc_one_map(struct fault_env *fe)
+static int pte_alloc_one_map(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
- if (!pmd_none(*fe->pmd))
+ if (!pmd_none(*vmf->pmd))
goto map_pte;
- if (fe->prealloc_pte) {
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd))) {
- spin_unlock(fe->ptl);
+ if (vmf->prealloc_pte) {
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd))) {
+ spin_unlock(vmf->ptl);
goto map_pte;
}
atomic_long_inc(&vma->vm_mm->nr_ptes);
- pmd_populate(vma->vm_mm, fe->pmd, fe->prealloc_pte);
- spin_unlock(fe->ptl);
- fe->prealloc_pte = 0;
- } else if (unlikely(pte_alloc(vma->vm_mm, fe->pmd, fe->address))) {
+ pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
+ spin_unlock(vmf->ptl);
+ vmf->prealloc_pte = 0;
+ } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
return VM_FAULT_OOM;
}
map_pte:
@@ -2917,11 +2919,11 @@ static int pte_alloc_one_map(struct fault_env *fe)
* through an atomic read in C, which is what pmd_trans_unstable()
* provides.
*/
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return VM_FAULT_NOPAGE;
- fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
- &fe->ptl);
+ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
return 0;
}
@@ -2939,11 +2941,11 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
return true;
}
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
- unsigned long haddr = fe->address & HPAGE_PMD_MASK;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
+ unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
pmd_t entry;
int i, ret;
@@ -2953,8 +2955,8 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
ret = VM_FAULT_FALLBACK;
page = compound_head(page);
- fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
- if (unlikely(!pmd_none(*fe->pmd)))
+ vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+ if (unlikely(!pmd_none(*vmf->pmd)))
goto out;
for (i = 0; i < HPAGE_PMD_NR; i++)
@@ -2967,19 +2969,19 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
add_mm_counter(vma->vm_mm, MM_FILEPAGES, HPAGE_PMD_NR);
page_add_file_rmap(page, true);
- set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
+ set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
- update_mmu_cache_pmd(vma, haddr, fe->pmd);
+ update_mmu_cache_pmd(vma, haddr, vmf->pmd);
/* fault is handled */
ret = 0;
count_vm_event(THP_FILE_MAPPED);
out:
- spin_unlock(fe->ptl);
+ spin_unlock(vmf->ptl);
return ret;
}
#else
-static int do_set_pmd(struct fault_env *fe, struct page *page)
+static int do_set_pmd(struct vm_fault *vmf, struct page *page)
{
BUILD_BUG();
return 0;
@@ -2990,41 +2992,42 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
* alloc_set_pte - setup new PTE entry for given page and add reverse page
* mapping. If needed, the fucntion allocates page table or use pre-allocated.
*
- * @fe: fault environment
+ * @vmf: fault environment
* @memcg: memcg to charge page (only for private mappings)
* @page: page to map
*
- * Caller must take care of unlocking fe->ptl, if fe->pte is non-NULL on return.
+ * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
+ * return.
*
* Target users are page handler itself and implementations of
* vm_ops->map_pages.
*/
-int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
+int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page)
{
- struct vm_area_struct *vma = fe->vma;
- bool write = fe->flags & FAULT_FLAG_WRITE;
+ struct vm_area_struct *vma = vmf->vma;
+ bool write = vmf->flags & FAULT_FLAG_WRITE;
pte_t entry;
int ret;
- if (pmd_none(*fe->pmd) && PageTransCompound(page) &&
+ if (pmd_none(*vmf->pmd) && PageTransCompound(page) &&
IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE)) {
/* THP on COW? */
VM_BUG_ON_PAGE(memcg, page);
- ret = do_set_pmd(fe, page);
+ ret = do_set_pmd(vmf, page);
if (ret != VM_FAULT_FALLBACK)
return ret;
}
- if (!fe->pte) {
- ret = pte_alloc_one_map(fe);
+ if (!vmf->pte) {
+ ret = pte_alloc_one_map(vmf);
if (ret)
return ret;
}
/* Re-check under ptl */
- if (unlikely(!pte_none(*fe->pte)))
+ if (unlikely(!pte_none(*vmf->pte)))
return VM_FAULT_NOPAGE;
flush_icache_page(vma, page);
@@ -3034,17 +3037,17 @@ int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
/* copy-on-write page */
if (write && !(vma->vm_flags & VM_SHARED)) {
inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
- page_add_new_anon_rmap(page, vma, fe->address, false);
+ page_add_new_anon_rmap(page, vma, vmf->address, false);
mem_cgroup_commit_charge(page, memcg, false, false);
lru_cache_add_active_or_unevictable(page, vma);
} else {
inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page));
page_add_file_rmap(page, false);
}
- set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
/* no need to invalidate: a not-present page won't be cached */
- update_mmu_cache(vma, fe->address, fe->pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
return 0;
}
@@ -3113,17 +3116,17 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
{
- unsigned long address = fe->address, nr_pages, mask;
+ unsigned long address = vmf->address, nr_pages, mask;
pgoff_t end_pgoff;
int off, ret = 0;
nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
- fe->address = max(address & mask, fe->vma->vm_start);
- off = ((address - fe->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
+ vmf->address = max(address & mask, vmf->vma->vm_start);
+ off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
start_pgoff -= off;
/*
@@ -3131,49 +3134,51 @@ static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
* or fault_around_pages() from start_pgoff, depending what is nearest.
*/
end_pgoff = start_pgoff -
- ((fe->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
+ ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
PTRS_PER_PTE - 1;
- end_pgoff = min3(end_pgoff, vma_pages(fe->vma) + fe->vma->vm_pgoff - 1,
+ end_pgoff = min3(end_pgoff,
+ vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
start_pgoff + nr_pages - 1);
- if (pmd_none(*fe->pmd)) {
- fe->prealloc_pte = pte_alloc_one(fe->vma->vm_mm, fe->address);
- if (!fe->prealloc_pte)
+ if (pmd_none(*vmf->pmd)) {
+ vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
+ vmf->address);
+ if (!vmf->prealloc_pte)
goto out;
smp_wmb(); /* See comment in __pte_alloc() */
}
- fe->vma->vm_ops->map_pages(fe, start_pgoff, end_pgoff);
+ vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
/* preallocated pagetable is unused: free it */
- if (fe->prealloc_pte) {
- pte_free(fe->vma->vm_mm, fe->prealloc_pte);
- fe->prealloc_pte = 0;
+ if (vmf->prealloc_pte) {
+ pte_free(vmf->vma->vm_mm, vmf->prealloc_pte);
+ vmf->prealloc_pte = 0;
}
/* Huge page is mapped? Page fault is solved */
- if (pmd_trans_huge(*fe->pmd)) {
+ if (pmd_trans_huge(*vmf->pmd)) {
ret = VM_FAULT_NOPAGE;
goto out;
}
/* ->map_pages() haven't done anything useful. Cold page cache? */
- if (!fe->pte)
+ if (!vmf->pte)
goto out;
/* check if the page fault is solved */
- fe->pte -= (fe->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
- if (!pte_none(*fe->pte))
+ vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
+ if (!pte_none(*vmf->pte))
ret = VM_FAULT_NOPAGE;
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
out:
- fe->address = address;
- fe->pte = NULL;
+ vmf->address = address;
+ vmf->pte = NULL;
return ret;
}
-static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
int ret = 0;
@@ -3183,27 +3188,27 @@ static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(fe, pgoff);
+ ret = do_fault_around(vmf, pgoff);
if (ret)
return ret;
}
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
unlock_page(fault_page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(fault_page);
return ret;
}
-static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
void *fault_entry;
struct mem_cgroup *memcg;
@@ -3212,7 +3217,7 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
if (unlikely(anon_vma_prepare(vma)))
return VM_FAULT_OOM;
- new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, fe->address);
+ new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
if (!new_page)
return VM_FAULT_OOM;
@@ -3222,17 +3227,17 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(fe, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, fe->address, vma);
+ copy_user_highpage(new_page, fault_page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(fe, memcg, new_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, memcg, new_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(fault_page);
put_page(fault_page);
@@ -3248,15 +3253,15 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3266,7 +3271,7 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, fe->address);
+ tmp = do_page_mkwrite(vma, fault_page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(fault_page);
@@ -3274,9 +3279,9 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
}
}
- ret |= alloc_set_pte(fe, NULL, fault_page);
- if (fe->pte)
- pte_unmap_unlock(fe->pte, fe->ptl);
+ ret |= alloc_set_pte(vmf, NULL, fault_page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(fault_page);
@@ -3314,19 +3319,19 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
* The mmap_sem may have been released depending on flags and our
* return value. See filemap_fault() and __lock_page_or_retry().
*/
-static int do_fault(struct fault_env *fe)
+static int do_fault(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
- pgoff_t pgoff = linear_page_index(vma, fe->address);
+ struct vm_area_struct *vma = vmf->vma;
+ pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
- if (!(fe->flags & FAULT_FLAG_WRITE))
- return do_read_fault(fe, pgoff);
+ if (!(vmf->flags & FAULT_FLAG_WRITE))
+ return do_read_fault(vmf, pgoff);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(fe, pgoff);
- return do_shared_fault(fe, pgoff);
+ return do_cow_fault(vmf, pgoff);
+ return do_shared_fault(vmf, pgoff);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3344,9 +3349,9 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct fault_env *fe, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf, pte_t pte)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
int page_nid = -1;
int last_cpupid;
@@ -3364,10 +3369,10 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
* page table entry is not accessible, so there would be no
* concurrent hardware modifications to the PTE.
*/
- fe->ptl = pte_lockptr(vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, pte))) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, pte))) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
goto out;
}
@@ -3376,18 +3381,18 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
pte = pte_mkyoung(pte);
if (was_writable)
pte = pte_mkwrite(pte);
- set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
- update_mmu_cache(vma, fe->address, fe->pte);
+ set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ update_mmu_cache(vma, vmf->address, vmf->pte);
- page = vm_normal_page(vma, fe->address, pte);
+ page = vm_normal_page(vma, vmf->address, pte);
if (!page) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
/* TODO: handle PTE-mapped THP */
if (PageCompound(page)) {
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3411,9 +3416,9 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
last_cpupid = page_cpupid_last(page);
page_nid = page_to_nid(page);
- target_nid = numa_migrate_prep(page, vma, fe->address, page_nid,
+ target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
&flags);
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
if (target_nid == -1) {
put_page(page);
goto out;
@@ -3433,28 +3438,28 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
return 0;
}
-static int create_huge_pmd(struct fault_env *fe)
+static int create_huge_pmd(struct vm_fault *vmf)
{
- struct vm_area_struct *vma = fe->vma;
+ struct vm_area_struct *vma = vmf->vma;
if (vma_is_anonymous(vma))
- return do_huge_pmd_anonymous_page(fe);
+ return do_huge_pmd_anonymous_page(vmf);
if (vma->vm_ops->pmd_fault)
- return vma->vm_ops->pmd_fault(vma, fe->address, fe->pmd,
- fe->flags);
+ return vma->vm_ops->pmd_fault(vma, vmf->address, vmf->pmd,
+ vmf->flags);
return VM_FAULT_FALLBACK;
}
-static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
+static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
{
- if (vma_is_anonymous(fe->vma))
- return do_huge_pmd_wp_page(fe, orig_pmd);
- if (fe->vma->vm_ops->pmd_fault)
- return fe->vma->vm_ops->pmd_fault(fe->vma, fe->address, fe->pmd,
- fe->flags);
+ if (vma_is_anonymous(vmf->vma))
+ return do_huge_pmd_wp_page(vmf, orig_pmd);
+ if (vmf->vma->vm_ops->pmd_fault)
+ return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
+ vmf->pmd, vmf->flags);
/* COW handled on pte level: split pmd */
- VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
- split_huge_pmd(fe->vma, fe->pmd, fe->address);
+ VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
+ split_huge_pmd(vmf->vma, vmf->pmd, vmf->address);
return VM_FAULT_FALLBACK;
}
@@ -3479,21 +3484,21 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
* The mmap_sem may have been released depending on flags and our return value.
* See filemap_fault() and __lock_page_or_retry().
*/
-static int handle_pte_fault(struct fault_env *fe)
+static int handle_pte_fault(struct vm_fault *vmf)
{
pte_t entry;
- if (unlikely(pmd_none(*fe->pmd))) {
+ if (unlikely(pmd_none(*vmf->pmd))) {
/*
* Leave __pte_alloc() until later: because vm_ops->fault may
* want to allocate huge page, and if we expose page table
* for an instant, it will be difficult to retract from
* concurrent faults and from rmap lookups.
*/
- fe->pte = NULL;
+ vmf->pte = NULL;
} else {
/* See comment in pte_alloc_one_map() */
- if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
+ if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
return 0;
/*
* A regular pmd is established and it can't morph into a huge
@@ -3501,9 +3506,9 @@ static int handle_pte_fault(struct fault_env *fe)
* mmap_sem read mode and khugepaged takes it in write mode.
* So now it's safe to run pte_offset_map().
*/
- fe->pte = pte_offset_map(fe->pmd, fe->address);
+ vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
- entry = *fe->pte;
+ entry = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3515,37 +3520,37 @@ static int handle_pte_fault(struct fault_env *fe)
*/
barrier();
if (pte_none(entry)) {
- pte_unmap(fe->pte);
- fe->pte = NULL;
+ pte_unmap(vmf->pte);
+ vmf->pte = NULL;
}
}
- if (!fe->pte) {
- if (vma_is_anonymous(fe->vma))
- return do_anonymous_page(fe);
+ if (!vmf->pte) {
+ if (vma_is_anonymous(vmf->vma))
+ return do_anonymous_page(vmf);
else
- return do_fault(fe);
+ return do_fault(vmf);
}
if (!pte_present(entry))
- return do_swap_page(fe, entry);
+ return do_swap_page(vmf, entry);
- if (pte_protnone(entry) && vma_is_accessible(fe->vma))
- return do_numa_page(fe, entry);
+ if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf, entry);
- fe->ptl = pte_lockptr(fe->vma->vm_mm, fe->pmd);
- spin_lock(fe->ptl);
- if (unlikely(!pte_same(*fe->pte, entry)))
+ vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
+ spin_lock(vmf->ptl);
+ if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
- if (fe->flags & FAULT_FLAG_WRITE) {
+ if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(fe, entry);
+ return do_wp_page(vmf, entry);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
- if (ptep_set_access_flags(fe->vma, fe->address, fe->pte, entry,
- fe->flags & FAULT_FLAG_WRITE)) {
- update_mmu_cache(fe->vma, fe->address, fe->pte);
+ if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
+ vmf->flags & FAULT_FLAG_WRITE)) {
+ update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
} else {
/*
* This is needed only for protection faults but the arch code
@@ -3553,11 +3558,11 @@ static int handle_pte_fault(struct fault_env *fe)
* This still avoids useless tlb flushes for .text page faults
* with threads.
*/
- if (fe->flags & FAULT_FLAG_WRITE)
- flush_tlb_fix_spurious_fault(fe->vma, fe->address);
+ if (vmf->flags & FAULT_FLAG_WRITE)
+ flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
}
unlock:
- pte_unmap_unlock(fe->pte, fe->ptl);
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
@@ -3570,7 +3575,7 @@ static int handle_pte_fault(struct fault_env *fe)
static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
unsigned int flags)
{
- struct fault_env fe = {
+ struct vm_fault vmf = {
.vma = vma,
.address = address,
.flags = flags,
@@ -3583,35 +3588,35 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
pud = pud_alloc(mm, pgd, address);
if (!pud)
return VM_FAULT_OOM;
- fe.pmd = pmd_alloc(mm, pud, address);
- if (!fe.pmd)
+ vmf.pmd = pmd_alloc(mm, pud, address);
+ if (!vmf.pmd)
return VM_FAULT_OOM;
- if (pmd_none(*fe.pmd) && transparent_hugepage_enabled(vma)) {
- int ret = create_huge_pmd(&fe);
+ if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
+ int ret = create_huge_pmd(&vmf);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- pmd_t orig_pmd = *fe.pmd;
+ pmd_t orig_pmd = *vmf.pmd;
int ret;
barrier();
if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
if (pmd_protnone(orig_pmd) && vma_is_accessible(vma))
- return do_huge_pmd_numa_page(&fe, orig_pmd);
+ return do_huge_pmd_numa_page(&vmf, orig_pmd);
- if ((fe.flags & FAULT_FLAG_WRITE) &&
+ if ((vmf.flags & FAULT_FLAG_WRITE) &&
!pmd_write(orig_pmd)) {
- ret = wp_huge_pmd(&fe, orig_pmd);
+ ret = wp_huge_pmd(&vmf, orig_pmd);
if (!(ret & VM_FAULT_FALLBACK))
return ret;
} else {
- huge_pmd_set_accessed(&fe, orig_pmd);
+ huge_pmd_set_accessed(&vmf, orig_pmd);
return 0;
}
}
}
- return handle_pte_fault(&fe);
+ return handle_pte_fault(&vmf);
}
/*
diff --git a/mm/nommu.c b/mm/nommu.c
index 8b8faaf2a9e9..077d0dbe4c28 100644
--- a/mm/nommu.c
+++ b/mm/nommu.c
@@ -1801,7 +1801,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
EXPORT_SYMBOL(filemap_fault);
-void filemap_map_pages(struct fault_env *fe,
+void filemap_map_pages(struct vm_fault *vmf,
pgoff_t start_pgoff, pgoff_t end_pgoff)
{
BUG();
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:24 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Every single user of vmf->virtual_address typed that entry to unsigned
long before doing anything with it so the type of virtual_address does
not really provide us any additional safety. Just use masked
vmf->address which already has the appropriate type.
Signed-off-by: Jan Kara <jack@suse.cz>
---
arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
arch/x86/entry/vdso/vma.c | 4 ++--
drivers/char/agp/alpha-agp.c | 2 +-
drivers/char/mspec.c | 2 +-
drivers/dax/dax.c | 2 +-
drivers/gpu/drm/armada/armada_gem.c | 2 +-
drivers/gpu/drm/drm_vm.c | 11 ++++++-----
drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
drivers/gpu/drm/gma500/framebuffer.c | 2 +-
drivers/gpu/drm/gma500/gem.c | 5 ++---
drivers/gpu/drm/i915/i915_gem.c | 2 +-
drivers/gpu/drm/msm/msm_gem.c | 7 +++----
drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
drivers/gpu/drm/tegra/gem.c | 4 ++--
drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
drivers/gpu/drm/udl/udl_gem.c | 5 ++---
drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
drivers/misc/cxl/context.c | 2 +-
drivers/misc/sgi-gru/grumain.c | 2 +-
drivers/staging/android/ion/ion.c | 2 +-
drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
drivers/xen/privcmd.c | 2 +-
fs/dax.c | 4 ++--
include/linux/mm.h | 2 --
mm/memory.c | 7 +++----
27 files changed, 59 insertions(+), 65 deletions(-)
diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
index 06254467e4dd..e8a31fffcdda 100644
--- a/arch/powerpc/platforms/cell/spufs/file.c
+++ b/arch/powerpc/platforms/cell/spufs/file.c
@@ -236,7 +236,7 @@ static int
spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct spu_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
unsigned long pfn, offset;
offset = vmf->pgoff << PAGE_SHIFT;
@@ -355,7 +355,7 @@ static int spufs_ps_fault(struct vm_area_struct *vma,
down_read(¤t->mm->mmap_sem);
} else {
area = ctx->spu->problem_phys + ps_offs;
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK,
(area + offset) >> PAGE_SHIFT);
spu_context_trace(spufs_ps_fault__insert, ctx, ctx->spu);
}
diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c
index 23c881caabd1..e20a5cb6cd31 100644
--- a/arch/x86/entry/vdso/vma.c
+++ b/arch/x86/entry/vdso/vma.c
@@ -109,7 +109,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
return VM_FAULT_SIGBUS;
if (sym_offset == image->sym_vvar_page) {
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK,
__pa_symbol(&__vvar_page) >> PAGE_SHIFT);
} else if (sym_offset == image->sym_pvclock_page) {
struct pvclock_vsyscall_time_info *pvti =
@@ -117,7 +117,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
if (pvti && vclock_was_used(VCLOCK_PVCLOCK)) {
ret = vm_insert_pfn(
vma,
- (unsigned long)vmf->virtual_address,
+ vmf->address & PAGE_MASK,
__pa(pvti) >> PAGE_SHIFT);
}
}
diff --git a/drivers/char/agp/alpha-agp.c b/drivers/char/agp/alpha-agp.c
index 199b8e99f7d7..372d9378d997 100644
--- a/drivers/char/agp/alpha-agp.c
+++ b/drivers/char/agp/alpha-agp.c
@@ -19,7 +19,7 @@ static int alpha_core_agp_vm_fault(struct vm_area_struct *vma,
unsigned long pa;
struct page *page;
- dma_addr = (unsigned long)vmf->virtual_address - vma->vm_start
+ dma_addr = (vmf->address & PAGE_MASK) - vma->vm_start
+ agp->aperture.bus_base;
pa = agp->ops->translate(agp, dma_addr);
diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c
index f3f92d5fcda0..2b7e1bc9ac5c 100644
--- a/drivers/char/mspec.c
+++ b/drivers/char/mspec.c
@@ -227,7 +227,7 @@ mspec_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* be because another thread has installed the pte first, so it
* is no problem.
*/
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
return VM_FAULT_NOPAGE;
}
diff --git a/drivers/dax/dax.c b/drivers/dax/dax.c
index 0e499bfca41c..6100c6bb52c5 100644
--- a/drivers/dax/dax.c
+++ b/drivers/dax/dax.c
@@ -328,7 +328,7 @@ static phys_addr_t pgoff_to_phys(struct dax_dev *dax_dev, pgoff_t pgoff,
static int __dax_dev_fault(struct dax_dev *dax_dev, struct vm_area_struct *vma,
struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long) vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct device *dev = &dax_dev->dev;
struct dax_region *dax_region;
int rc = VM_FAULT_SIGBUS;
diff --git a/drivers/gpu/drm/armada/armada_gem.c b/drivers/gpu/drm/armada/armada_gem.c
index 806791897304..6ccb70dce013 100644
--- a/drivers/gpu/drm/armada/armada_gem.c
+++ b/drivers/gpu/drm/armada/armada_gem.c
@@ -17,7 +17,7 @@
static int armada_gem_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct armada_gem_object *obj = drm_to_armada_gem(vma->vm_private_data);
- unsigned long addr = (unsigned long)vmf->virtual_address;
+ unsigned long addr = vmf->address & PAGE_MASK;
unsigned long pfn = obj->phys_addr >> PAGE_SHIFT;
int ret;
diff --git a/drivers/gpu/drm/drm_vm.c b/drivers/gpu/drm/drm_vm.c
index caa4e4ca616d..7a67e6198819 100644
--- a/drivers/gpu/drm/drm_vm.c
+++ b/drivers/gpu/drm/drm_vm.c
@@ -124,8 +124,8 @@ static int drm_do_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* Using vm_pgoff as a selector forces us to use this unusual
* addressing scheme.
*/
- resource_size_t offset = (unsigned long)vmf->virtual_address -
- vma->vm_start;
+ resource_size_t offset = (vmf->address & PAGE_MASK) -
+ vma->vm_start;
resource_size_t baddr = map->offset + offset;
struct drm_agp_mem *agpmem;
struct page *page;
@@ -195,7 +195,7 @@ static int drm_do_vm_shm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!map)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
i = (unsigned long)map->handle + offset;
page = vmalloc_to_page((void *)i);
if (!page)
@@ -301,7 +301,8 @@ static int drm_do_vm_dma_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!dma->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start; /* vm_[pg]off[set] should be 0 */
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
+ /* vm_[pg]off[set] should be 0 */
page_nr = offset >> PAGE_SHIFT; /* page_nr could just be vmf->pgoff */
page = virt_to_page((void *)dma->pagelist[page_nr]);
@@ -337,7 +338,7 @@ static int drm_do_vm_sg_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!entry->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
map_offset = map->offset - (unsigned long)dev->sg->virtual;
page_offset = (offset >> PAGE_SHIFT) + (map_offset >> PAGE_SHIFT);
page = entry->pagelist[page_offset];
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
index 0370b842d9cc..9a338a91dc08 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
@@ -202,15 +202,14 @@ int etnaviv_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
page = pages[pgoff];
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
page_to_pfn(page), page_to_pfn(page) << PAGE_SHIFT);
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
out:
switch (ret) {
diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c
index f2ae72ba7d5a..2e57d5067170 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_gem.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c
@@ -455,8 +455,8 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
pgoff_t page_offset;
int ret;
- page_offset = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK) - vma->vm_start) >>
+ PAGE_SHIFT;
if (page_offset >= (exynos_gem->size >> PAGE_SHIFT)) {
DRM_ERROR("invalid page offset\n");
@@ -465,7 +465,7 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
pfn = page_to_pfn(exynos_gem->pages[page_offset]);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out:
diff --git a/drivers/gpu/drm/gma500/framebuffer.c b/drivers/gpu/drm/gma500/framebuffer.c
index 3a44e705db53..fbc336ee151d 100644
--- a/drivers/gpu/drm/gma500/framebuffer.c
+++ b/drivers/gpu/drm/gma500/framebuffer.c
@@ -125,7 +125,7 @@ static int psbfb_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
psbfb->gtt->offset;
page_num = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
- address = (unsigned long)vmf->virtual_address - (vmf->pgoff << PAGE_SHIFT);
+ address = (vmf->address & PAGE_MASK) - (vmf->pgoff << PAGE_SHIFT);
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
diff --git a/drivers/gpu/drm/gma500/gem.c b/drivers/gpu/drm/gma500/gem.c
index 6d1cb6b370b1..0064dcbcbd2b 100644
--- a/drivers/gpu/drm/gma500/gem.c
+++ b/drivers/gpu/drm/gma500/gem.c
@@ -197,15 +197,14 @@ int psb_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
/* Page relative to the VMA start - we must calculate this ourselves
because vmf->pgoff is the fake GEM offset */
- page_offset = ((unsigned long) vmf->virtual_address - vma->vm_start)
- >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
/* CPU view of the page, don't go via the GART for CPU writes */
if (r->stolen)
pfn = (dev_priv->stolen_base + r->offset) >> PAGE_SHIFT;
else
pfn = page_to_pfn(r->pages[page_offset]);
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
fail:
mutex_unlock(&dev_priv->mmap_mutex);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 947e82c2b175..9733b0274494 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1763,7 +1763,7 @@ int i915_gem_fault(struct vm_area_struct *area, struct vm_fault *vmf)
int ret;
/* We don't use vmf->pgoff since that has the fake offset */
- page_offset = ((unsigned long)vmf->virtual_address - area->vm_start) >>
+ page_offset = ((vmf->address & PAGE_MASK)- area->vm_start) >>
PAGE_SHIFT;
trace_i915_gem_object_fault(obj, page_offset, true, write);
diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c
index b6ac27e31929..84890d604fd3 100644
--- a/drivers/gpu/drm/msm/msm_gem.c
+++ b/drivers/gpu/drm/msm/msm_gem.c
@@ -225,15 +225,14 @@ int msm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
pfn = page_to_pfn(pages[pgoff]);
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out_unlock:
diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index 505dee0db973..86cdd3e9248f 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -396,8 +396,7 @@ static int fault_1d(struct drm_gem_object *obj,
pgoff_t pgoff;
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (omap_obj->pages) {
omap_gem_cpu_sync(obj, pgoff);
@@ -407,10 +406,10 @@ static int fault_1d(struct drm_gem_object *obj,
pfn = (omap_obj->paddr >> PAGE_SHIFT) + pgoff;
}
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- return vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ return vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
}
@@ -425,7 +424,7 @@ static int fault_2d(struct drm_gem_object *obj,
struct page *pages[64]; /* XXX is this too much to have on stack? */
unsigned long pfn;
pgoff_t pgoff, base_pgoff;
- void __user *vaddr;
+ unsigned long vaddr;
int i, ret, slots;
/*
@@ -445,8 +444,7 @@ static int fault_2d(struct drm_gem_object *obj,
const int m = 1 + ((omap_obj->width << fmt) / PAGE_SIZE);
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
/*
* Actual address we start mapping at is rounded down to previous slot
@@ -457,7 +455,8 @@ static int fault_2d(struct drm_gem_object *obj,
/* figure out buffer width in slots */
slots = omap_obj->width >> priv->usergart[fmt].slot_shift;
- vaddr = vmf->virtual_address - ((pgoff - base_pgoff) << PAGE_SHIFT);
+ vaddr = (vmf->address & PAGE_MASK) -
+ ((pgoff - base_pgoff) << PAGE_SHIFT);
entry = &priv->usergart[fmt].entry[priv->usergart[fmt].last];
@@ -501,12 +500,11 @@ static int fault_2d(struct drm_gem_object *obj,
pfn = entry->paddr >> PAGE_SHIFT;
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
for (i = n; i > 0; i--) {
- vm_insert_mixed(vma, (unsigned long)vaddr,
- __pfn_to_pfn_t(pfn, PFN_DEV));
+ vm_insert_mixed(vma, vaddr, __pfn_to_pfn_t(pfn, PFN_DEV));
pfn += priv->usergart[fmt].stride_pfn;
vaddr += PAGE_SIZE * m;
}
diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c
index 95e622e31931..6d1a5b467ff2 100644
--- a/drivers/gpu/drm/tegra/gem.c
+++ b/drivers/gpu/drm/tegra/gem.c
@@ -427,10 +427,10 @@ static int tegra_bo_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!bo->pages)
return VM_FAULT_SIGBUS;
- offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >> PAGE_SHIFT;
+ offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
page = bo->pages[offset];
- err = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ err = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (err) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c
index a6ed9d5e5167..4fe0bbef7119 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_vm.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c
@@ -101,7 +101,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
int ret;
int i;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
int retval = VM_FAULT_NOPAGE;
struct ttm_mem_type_manager *man =
&bdev->man[bo->mem.mem_type];
diff --git a/drivers/gpu/drm/udl/udl_gem.c b/drivers/gpu/drm/udl/udl_gem.c
index 818e70712b18..091e38a04fe6 100644
--- a/drivers/gpu/drm/udl/udl_gem.c
+++ b/drivers/gpu/drm/udl/udl_gem.c
@@ -107,14 +107,13 @@ int udl_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned int page_offset;
int ret = 0;
- page_offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >>
- PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (!obj->pages)
return VM_FAULT_SIGBUS;
page = obj->pages[page_offset];
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (ret) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/vgem/vgem_drv.c b/drivers/gpu/drm/vgem/vgem_drv.c
index f36c14729b55..46b4d16e4e17 100644
--- a/drivers/gpu/drm/vgem/vgem_drv.c
+++ b/drivers/gpu/drm/vgem/vgem_drv.c
@@ -54,7 +54,7 @@ static int vgem_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct drm_vgem_gem_object *obj = vma->vm_private_data;
/* We don't use vmf->pgoff since that has the fake offset */
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct page *page;
page = shmem_read_mapping_page(file_inode(obj->base.filp)->i_mapping,
diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c
index 1db0af6c7f94..b11e0008e436 100644
--- a/drivers/media/v4l2-core/videobuf-dma-sg.c
+++ b/drivers/media/v4l2-core/videobuf-dma-sg.c
@@ -439,13 +439,12 @@ static int videobuf_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
dprintk(3, "fault: fault @ %08lx [vma %08lx-%08lx]\n",
- (unsigned long)vmf->virtual_address,
- vma->vm_start, vma->vm_end);
+ vmf->address & PAGE_MASK, vma->vm_start, vma->vm_end);
page = alloc_page(GFP_USER | __GFP_DMA32);
if (!page)
return VM_FAULT_OOM;
- clear_user_highpage(page, (unsigned long)vmf->virtual_address);
+ clear_user_highpage(page, vmf->address & PAGE_MASK);
vmf->page = page;
return 0;
diff --git a/drivers/misc/cxl/context.c b/drivers/misc/cxl/context.c
index 5e506c19108a..0de6d1334fcd 100644
--- a/drivers/misc/cxl/context.c
+++ b/drivers/misc/cxl/context.c
@@ -117,7 +117,7 @@ int cxl_context_init(struct cxl_context *ctx, struct cxl_afu *afu, bool master,
static int cxl_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct cxl_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
u64 area, offset;
offset = vmf->pgoff << PAGE_SHIFT;
diff --git a/drivers/misc/sgi-gru/grumain.c b/drivers/misc/sgi-gru/grumain.c
index 33741ad4a74a..677c4dd39561 100644
--- a/drivers/misc/sgi-gru/grumain.c
+++ b/drivers/misc/sgi-gru/grumain.c
@@ -932,7 +932,7 @@ int gru_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned long paddr, vaddr;
unsigned long expires;
- vaddr = (unsigned long)vmf->virtual_address;
+ vaddr = vmf->address & PAGE_MASK;
gru_dbg(grudev, "vma %p, vaddr 0x%lx (0x%lx)\n",
vma, vaddr, GSEG_BASE(vaddr));
STAT(nopfn);
diff --git a/drivers/staging/android/ion/ion.c b/drivers/staging/android/ion/ion.c
index 209a8f7ef02b..5aebbb380271 100644
--- a/drivers/staging/android/ion/ion.c
+++ b/drivers/staging/android/ion/ion.c
@@ -882,7 +882,7 @@ static int ion_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
BUG_ON(!buffer->pages || !buffer->pages[vmf->pgoff]);
pfn = page_to_pfn(ion_buffer_page(buffer->pages[vmf->pgoff]));
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
mutex_unlock(&buffer->lock);
if (ret)
return VM_FAULT_ERROR;
diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index 2b7f182a15e2..1491d788dcab 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -984,7 +984,8 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
"page %p map %p index %lu flags %lx count %u priv %0lx: got addr %p type NOPAGE\n",
vmf->page, vmf->page->mapping, vmf->page->index,
(long)vmf->page->flags, page_count(vmf->page),
- page_private(vmf->page), vmf->virtual_address);
+ page_private(vmf->page),
+ (void *)(vmf->address & PAGE_MASK));
if (unlikely(!(cfio->ft_flags & VM_FAULT_LOCKED))) {
lock_page(vmf->page);
cfio->ft_flags |= VM_FAULT_LOCKED;
@@ -995,12 +996,14 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
}
if (cfio->ft_flags & (VM_FAULT_SIGBUS | VM_FAULT_SIGSEGV)) {
- CDEBUG(D_PAGE, "got addr %p - SIGBUS\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - SIGBUS\n",
+ (void *)(vmf->address & PAGE_MASK));
return -EFAULT;
}
if (cfio->ft_flags & VM_FAULT_OOM) {
- CDEBUG(D_PAGE, "got addr %p - OOM\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - OOM\n",
+ (void *)(vmf->address & PAGE_MASK));
return -ENOMEM;
}
diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index 702040fe2001..1334aaf9016d 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -602,7 +602,7 @@ static int privcmd_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
printk(KERN_DEBUG "privcmd_fault: vma=%p %lx-%lx, pgoff=%lx, uv=%p\n",
vma, vma->vm_start, vma->vm_end,
- vmf->pgoff, vmf->virtual_address);
+ vmf->pgoff, (void *)(vmf->address & PAGE_MASK));
return VM_FAULT_SIGBUS;
}
diff --git a/fs/dax.c b/fs/dax.c
index ad131cd2605d..716a0f9c769c 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -737,7 +737,7 @@ static int dax_insert_mapping(struct address_space *mapping,
struct block_device *bdev, sector_t sector, size_t size,
void **entryp, struct vm_area_struct *vma, struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct blk_dax_ctl dax = {
.sector = sector,
.size = size,
@@ -947,7 +947,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
{
struct address_space *mapping = vma->vm_file->f_mapping;
struct inode *inode = mapping->host;
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
loff_t pos = (loff_t)vmf->pgoff << PAGE_SHIFT;
sector_t sector;
struct iomap iomap = { 0 };
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 657eb69eb87e..df3958437473 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -297,8 +297,6 @@ struct vm_fault {
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
- void __user *virtual_address; /* Faulting virtual address masked by
- * PAGE_MASK */
pmd_t *pmd; /* Pointer to pmd entry matching
* the 'address'
*/
diff --git a/mm/memory.c b/mm/memory.c
index fad45cd59ba7..c652b65469cd 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2044,7 +2044,7 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
struct vm_fault vmf;
int ret;
- vmf.virtual_address = (void __user *)(address & PAGE_MASK);
+ vmf.address = address;
vmf.pgoff = page->index;
vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
vmf.gfp_mask = __get_fault_gfp_mask(vma);
@@ -2280,8 +2280,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_fault vmf2 = {
.page = NULL,
.pgoff = linear_page_index(vma, vmf->address),
- .virtual_address =
- (void __user *)(vmf->address & PAGE_MASK),
+ .address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
@@ -2856,7 +2855,7 @@ static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct vm_fault vmf2;
int ret;
- vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.address = vmf->address;
vmf2.pgoff = pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Every single user of vmf->virtual_address typed that entry to unsigned
long before doing anything with it so the type of virtual_address does
not really provide us any additional safety. Just use masked
vmf->address which already has the appropriate type.
Signed-off-by: Jan Kara <jack@suse.cz>
---
arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
arch/x86/entry/vdso/vma.c | 4 ++--
drivers/char/agp/alpha-agp.c | 2 +-
drivers/char/mspec.c | 2 +-
drivers/dax/dax.c | 2 +-
drivers/gpu/drm/armada/armada_gem.c | 2 +-
drivers/gpu/drm/drm_vm.c | 11 ++++++-----
drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
drivers/gpu/drm/gma500/framebuffer.c | 2 +-
drivers/gpu/drm/gma500/gem.c | 5 ++---
drivers/gpu/drm/i915/i915_gem.c | 2 +-
drivers/gpu/drm/msm/msm_gem.c | 7 +++----
drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
drivers/gpu/drm/tegra/gem.c | 4 ++--
drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
drivers/gpu/drm/udl/udl_gem.c | 5 ++---
drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
drivers/misc/cxl/context.c | 2 +-
drivers/misc/sgi-gru/grumain.c | 2 +-
drivers/staging/android/ion/ion.c | 2 +-
drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
drivers/xen/privcmd.c | 2 +-
fs/dax.c | 4 ++--
include/linux/mm.h | 2 --
mm/memory.c | 7 +++----
27 files changed, 59 insertions(+), 65 deletions(-)
diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
index 06254467e4dd..e8a31fffcdda 100644
--- a/arch/powerpc/platforms/cell/spufs/file.c
+++ b/arch/powerpc/platforms/cell/spufs/file.c
@@ -236,7 +236,7 @@ static int
spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct spu_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
unsigned long pfn, offset;
offset = vmf->pgoff << PAGE_SHIFT;
@@ -355,7 +355,7 @@ static int spufs_ps_fault(struct vm_area_struct *vma,
down_read(¤t->mm->mmap_sem);
} else {
area = ctx->spu->problem_phys + ps_offs;
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK,
(area + offset) >> PAGE_SHIFT);
spu_context_trace(spufs_ps_fault__insert, ctx, ctx->spu);
}
diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c
index 23c881caabd1..e20a5cb6cd31 100644
--- a/arch/x86/entry/vdso/vma.c
+++ b/arch/x86/entry/vdso/vma.c
@@ -109,7 +109,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
return VM_FAULT_SIGBUS;
if (sym_offset == image->sym_vvar_page) {
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK,
__pa_symbol(&__vvar_page) >> PAGE_SHIFT);
} else if (sym_offset == image->sym_pvclock_page) {
struct pvclock_vsyscall_time_info *pvti =
@@ -117,7 +117,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
if (pvti && vclock_was_used(VCLOCK_PVCLOCK)) {
ret = vm_insert_pfn(
vma,
- (unsigned long)vmf->virtual_address,
+ vmf->address & PAGE_MASK,
__pa(pvti) >> PAGE_SHIFT);
}
}
diff --git a/drivers/char/agp/alpha-agp.c b/drivers/char/agp/alpha-agp.c
index 199b8e99f7d7..372d9378d997 100644
--- a/drivers/char/agp/alpha-agp.c
+++ b/drivers/char/agp/alpha-agp.c
@@ -19,7 +19,7 @@ static int alpha_core_agp_vm_fault(struct vm_area_struct *vma,
unsigned long pa;
struct page *page;
- dma_addr = (unsigned long)vmf->virtual_address - vma->vm_start
+ dma_addr = (vmf->address & PAGE_MASK) - vma->vm_start
+ agp->aperture.bus_base;
pa = agp->ops->translate(agp, dma_addr);
diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c
index f3f92d5fcda0..2b7e1bc9ac5c 100644
--- a/drivers/char/mspec.c
+++ b/drivers/char/mspec.c
@@ -227,7 +227,7 @@ mspec_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* be because another thread has installed the pte first, so it
* is no problem.
*/
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
return VM_FAULT_NOPAGE;
}
diff --git a/drivers/dax/dax.c b/drivers/dax/dax.c
index 0e499bfca41c..6100c6bb52c5 100644
--- a/drivers/dax/dax.c
+++ b/drivers/dax/dax.c
@@ -328,7 +328,7 @@ static phys_addr_t pgoff_to_phys(struct dax_dev *dax_dev, pgoff_t pgoff,
static int __dax_dev_fault(struct dax_dev *dax_dev, struct vm_area_struct *vma,
struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long) vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct device *dev = &dax_dev->dev;
struct dax_region *dax_region;
int rc = VM_FAULT_SIGBUS;
diff --git a/drivers/gpu/drm/armada/armada_gem.c b/drivers/gpu/drm/armada/armada_gem.c
index 806791897304..6ccb70dce013 100644
--- a/drivers/gpu/drm/armada/armada_gem.c
+++ b/drivers/gpu/drm/armada/armada_gem.c
@@ -17,7 +17,7 @@
static int armada_gem_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct armada_gem_object *obj = drm_to_armada_gem(vma->vm_private_data);
- unsigned long addr = (unsigned long)vmf->virtual_address;
+ unsigned long addr = vmf->address & PAGE_MASK;
unsigned long pfn = obj->phys_addr >> PAGE_SHIFT;
int ret;
diff --git a/drivers/gpu/drm/drm_vm.c b/drivers/gpu/drm/drm_vm.c
index caa4e4ca616d..7a67e6198819 100644
--- a/drivers/gpu/drm/drm_vm.c
+++ b/drivers/gpu/drm/drm_vm.c
@@ -124,8 +124,8 @@ static int drm_do_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* Using vm_pgoff as a selector forces us to use this unusual
* addressing scheme.
*/
- resource_size_t offset = (unsigned long)vmf->virtual_address -
- vma->vm_start;
+ resource_size_t offset = (vmf->address & PAGE_MASK) -
+ vma->vm_start;
resource_size_t baddr = map->offset + offset;
struct drm_agp_mem *agpmem;
struct page *page;
@@ -195,7 +195,7 @@ static int drm_do_vm_shm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!map)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
i = (unsigned long)map->handle + offset;
page = vmalloc_to_page((void *)i);
if (!page)
@@ -301,7 +301,8 @@ static int drm_do_vm_dma_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!dma->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start; /* vm_[pg]off[set] should be 0 */
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
+ /* vm_[pg]off[set] should be 0 */
page_nr = offset >> PAGE_SHIFT; /* page_nr could just be vmf->pgoff */
page = virt_to_page((void *)dma->pagelist[page_nr]);
@@ -337,7 +338,7 @@ static int drm_do_vm_sg_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!entry->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
map_offset = map->offset - (unsigned long)dev->sg->virtual;
page_offset = (offset >> PAGE_SHIFT) + (map_offset >> PAGE_SHIFT);
page = entry->pagelist[page_offset];
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
index 0370b842d9cc..9a338a91dc08 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
@@ -202,15 +202,14 @@ int etnaviv_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
page = pages[pgoff];
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
page_to_pfn(page), page_to_pfn(page) << PAGE_SHIFT);
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
out:
switch (ret) {
diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c
index f2ae72ba7d5a..2e57d5067170 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_gem.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c
@@ -455,8 +455,8 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
pgoff_t page_offset;
int ret;
- page_offset = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK) - vma->vm_start) >>
+ PAGE_SHIFT;
if (page_offset >= (exynos_gem->size >> PAGE_SHIFT)) {
DRM_ERROR("invalid page offset\n");
@@ -465,7 +465,7 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
pfn = page_to_pfn(exynos_gem->pages[page_offset]);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out:
diff --git a/drivers/gpu/drm/gma500/framebuffer.c b/drivers/gpu/drm/gma500/framebuffer.c
index 3a44e705db53..fbc336ee151d 100644
--- a/drivers/gpu/drm/gma500/framebuffer.c
+++ b/drivers/gpu/drm/gma500/framebuffer.c
@@ -125,7 +125,7 @@ static int psbfb_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
psbfb->gtt->offset;
page_num = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
- address = (unsigned long)vmf->virtual_address - (vmf->pgoff << PAGE_SHIFT);
+ address = (vmf->address & PAGE_MASK) - (vmf->pgoff << PAGE_SHIFT);
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
diff --git a/drivers/gpu/drm/gma500/gem.c b/drivers/gpu/drm/gma500/gem.c
index 6d1cb6b370b1..0064dcbcbd2b 100644
--- a/drivers/gpu/drm/gma500/gem.c
+++ b/drivers/gpu/drm/gma500/gem.c
@@ -197,15 +197,14 @@ int psb_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
/* Page relative to the VMA start - we must calculate this ourselves
because vmf->pgoff is the fake GEM offset */
- page_offset = ((unsigned long) vmf->virtual_address - vma->vm_start)
- >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
/* CPU view of the page, don't go via the GART for CPU writes */
if (r->stolen)
pfn = (dev_priv->stolen_base + r->offset) >> PAGE_SHIFT;
else
pfn = page_to_pfn(r->pages[page_offset]);
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
fail:
mutex_unlock(&dev_priv->mmap_mutex);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 947e82c2b175..9733b0274494 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1763,7 +1763,7 @@ int i915_gem_fault(struct vm_area_struct *area, struct vm_fault *vmf)
int ret;
/* We don't use vmf->pgoff since that has the fake offset */
- page_offset = ((unsigned long)vmf->virtual_address - area->vm_start) >>
+ page_offset = ((vmf->address & PAGE_MASK)- area->vm_start) >>
PAGE_SHIFT;
trace_i915_gem_object_fault(obj, page_offset, true, write);
diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c
index b6ac27e31929..84890d604fd3 100644
--- a/drivers/gpu/drm/msm/msm_gem.c
+++ b/drivers/gpu/drm/msm/msm_gem.c
@@ -225,15 +225,14 @@ int msm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
pfn = page_to_pfn(pages[pgoff]);
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out_unlock:
diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index 505dee0db973..86cdd3e9248f 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -396,8 +396,7 @@ static int fault_1d(struct drm_gem_object *obj,
pgoff_t pgoff;
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (omap_obj->pages) {
omap_gem_cpu_sync(obj, pgoff);
@@ -407,10 +406,10 @@ static int fault_1d(struct drm_gem_object *obj,
pfn = (omap_obj->paddr >> PAGE_SHIFT) + pgoff;
}
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- return vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ return vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
}
@@ -425,7 +424,7 @@ static int fault_2d(struct drm_gem_object *obj,
struct page *pages[64]; /* XXX is this too much to have on stack? */
unsigned long pfn;
pgoff_t pgoff, base_pgoff;
- void __user *vaddr;
+ unsigned long vaddr;
int i, ret, slots;
/*
@@ -445,8 +444,7 @@ static int fault_2d(struct drm_gem_object *obj,
const int m = 1 + ((omap_obj->width << fmt) / PAGE_SIZE);
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
/*
* Actual address we start mapping at is rounded down to previous slot
@@ -457,7 +455,8 @@ static int fault_2d(struct drm_gem_object *obj,
/* figure out buffer width in slots */
slots = omap_obj->width >> priv->usergart[fmt].slot_shift;
- vaddr = vmf->virtual_address - ((pgoff - base_pgoff) << PAGE_SHIFT);
+ vaddr = (vmf->address & PAGE_MASK) -
+ ((pgoff - base_pgoff) << PAGE_SHIFT);
entry = &priv->usergart[fmt].entry[priv->usergart[fmt].last];
@@ -501,12 +500,11 @@ static int fault_2d(struct drm_gem_object *obj,
pfn = entry->paddr >> PAGE_SHIFT;
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
for (i = n; i > 0; i--) {
- vm_insert_mixed(vma, (unsigned long)vaddr,
- __pfn_to_pfn_t(pfn, PFN_DEV));
+ vm_insert_mixed(vma, vaddr, __pfn_to_pfn_t(pfn, PFN_DEV));
pfn += priv->usergart[fmt].stride_pfn;
vaddr += PAGE_SIZE * m;
}
diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c
index 95e622e31931..6d1a5b467ff2 100644
--- a/drivers/gpu/drm/tegra/gem.c
+++ b/drivers/gpu/drm/tegra/gem.c
@@ -427,10 +427,10 @@ static int tegra_bo_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!bo->pages)
return VM_FAULT_SIGBUS;
- offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >> PAGE_SHIFT;
+ offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
page = bo->pages[offset];
- err = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ err = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (err) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c
index a6ed9d5e5167..4fe0bbef7119 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_vm.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c
@@ -101,7 +101,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
int ret;
int i;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
int retval = VM_FAULT_NOPAGE;
struct ttm_mem_type_manager *man =
&bdev->man[bo->mem.mem_type];
diff --git a/drivers/gpu/drm/udl/udl_gem.c b/drivers/gpu/drm/udl/udl_gem.c
index 818e70712b18..091e38a04fe6 100644
--- a/drivers/gpu/drm/udl/udl_gem.c
+++ b/drivers/gpu/drm/udl/udl_gem.c
@@ -107,14 +107,13 @@ int udl_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned int page_offset;
int ret = 0;
- page_offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >>
- PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (!obj->pages)
return VM_FAULT_SIGBUS;
page = obj->pages[page_offset];
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (ret) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/vgem/vgem_drv.c b/drivers/gpu/drm/vgem/vgem_drv.c
index f36c14729b55..46b4d16e4e17 100644
--- a/drivers/gpu/drm/vgem/vgem_drv.c
+++ b/drivers/gpu/drm/vgem/vgem_drv.c
@@ -54,7 +54,7 @@ static int vgem_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct drm_vgem_gem_object *obj = vma->vm_private_data;
/* We don't use vmf->pgoff since that has the fake offset */
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct page *page;
page = shmem_read_mapping_page(file_inode(obj->base.filp)->i_mapping,
diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c
index 1db0af6c7f94..b11e0008e436 100644
--- a/drivers/media/v4l2-core/videobuf-dma-sg.c
+++ b/drivers/media/v4l2-core/videobuf-dma-sg.c
@@ -439,13 +439,12 @@ static int videobuf_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
dprintk(3, "fault: fault @ %08lx [vma %08lx-%08lx]\n",
- (unsigned long)vmf->virtual_address,
- vma->vm_start, vma->vm_end);
+ vmf->address & PAGE_MASK, vma->vm_start, vma->vm_end);
page = alloc_page(GFP_USER | __GFP_DMA32);
if (!page)
return VM_FAULT_OOM;
- clear_user_highpage(page, (unsigned long)vmf->virtual_address);
+ clear_user_highpage(page, vmf->address & PAGE_MASK);
vmf->page = page;
return 0;
diff --git a/drivers/misc/cxl/context.c b/drivers/misc/cxl/context.c
index 5e506c19108a..0de6d1334fcd 100644
--- a/drivers/misc/cxl/context.c
+++ b/drivers/misc/cxl/context.c
@@ -117,7 +117,7 @@ int cxl_context_init(struct cxl_context *ctx, struct cxl_afu *afu, bool master,
static int cxl_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct cxl_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
u64 area, offset;
offset = vmf->pgoff << PAGE_SHIFT;
diff --git a/drivers/misc/sgi-gru/grumain.c b/drivers/misc/sgi-gru/grumain.c
index 33741ad4a74a..677c4dd39561 100644
--- a/drivers/misc/sgi-gru/grumain.c
+++ b/drivers/misc/sgi-gru/grumain.c
@@ -932,7 +932,7 @@ int gru_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned long paddr, vaddr;
unsigned long expires;
- vaddr = (unsigned long)vmf->virtual_address;
+ vaddr = vmf->address & PAGE_MASK;
gru_dbg(grudev, "vma %p, vaddr 0x%lx (0x%lx)\n",
vma, vaddr, GSEG_BASE(vaddr));
STAT(nopfn);
diff --git a/drivers/staging/android/ion/ion.c b/drivers/staging/android/ion/ion.c
index 209a8f7ef02b..5aebbb380271 100644
--- a/drivers/staging/android/ion/ion.c
+++ b/drivers/staging/android/ion/ion.c
@@ -882,7 +882,7 @@ static int ion_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
BUG_ON(!buffer->pages || !buffer->pages[vmf->pgoff]);
pfn = page_to_pfn(ion_buffer_page(buffer->pages[vmf->pgoff]));
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
mutex_unlock(&buffer->lock);
if (ret)
return VM_FAULT_ERROR;
diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index 2b7f182a15e2..1491d788dcab 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -984,7 +984,8 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
"page %p map %p index %lu flags %lx count %u priv %0lx: got addr %p type NOPAGE\n",
vmf->page, vmf->page->mapping, vmf->page->index,
(long)vmf->page->flags, page_count(vmf->page),
- page_private(vmf->page), vmf->virtual_address);
+ page_private(vmf->page),
+ (void *)(vmf->address & PAGE_MASK));
if (unlikely(!(cfio->ft_flags & VM_FAULT_LOCKED))) {
lock_page(vmf->page);
cfio->ft_flags |= VM_FAULT_LOCKED;
@@ -995,12 +996,14 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
}
if (cfio->ft_flags & (VM_FAULT_SIGBUS | VM_FAULT_SIGSEGV)) {
- CDEBUG(D_PAGE, "got addr %p - SIGBUS\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - SIGBUS\n",
+ (void *)(vmf->address & PAGE_MASK));
return -EFAULT;
}
if (cfio->ft_flags & VM_FAULT_OOM) {
- CDEBUG(D_PAGE, "got addr %p - OOM\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - OOM\n",
+ (void *)(vmf->address & PAGE_MASK));
return -ENOMEM;
}
diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index 702040fe2001..1334aaf9016d 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -602,7 +602,7 @@ static int privcmd_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
printk(KERN_DEBUG "privcmd_fault: vma=%p %lx-%lx, pgoff=%lx, uv=%p\n",
vma, vma->vm_start, vma->vm_end,
- vmf->pgoff, vmf->virtual_address);
+ vmf->pgoff, (void *)(vmf->address & PAGE_MASK));
return VM_FAULT_SIGBUS;
}
diff --git a/fs/dax.c b/fs/dax.c
index ad131cd2605d..716a0f9c769c 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -737,7 +737,7 @@ static int dax_insert_mapping(struct address_space *mapping,
struct block_device *bdev, sector_t sector, size_t size,
void **entryp, struct vm_area_struct *vma, struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct blk_dax_ctl dax = {
.sector = sector,
.size = size,
@@ -947,7 +947,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
{
struct address_space *mapping = vma->vm_file->f_mapping;
struct inode *inode = mapping->host;
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
loff_t pos = (loff_t)vmf->pgoff << PAGE_SHIFT;
sector_t sector;
struct iomap iomap = { 0 };
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 657eb69eb87e..df3958437473 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -297,8 +297,6 @@ struct vm_fault {
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
- void __user *virtual_address; /* Faulting virtual address masked by
- * PAGE_MASK */
pmd_t *pmd; /* Pointer to pmd entry matching
* the 'address'
*/
diff --git a/mm/memory.c b/mm/memory.c
index fad45cd59ba7..c652b65469cd 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2044,7 +2044,7 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
struct vm_fault vmf;
int ret;
- vmf.virtual_address = (void __user *)(address & PAGE_MASK);
+ vmf.address = address;
vmf.pgoff = page->index;
vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
vmf.gfp_mask = __get_fault_gfp_mask(vma);
@@ -2280,8 +2280,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_fault vmf2 = {
.page = NULL,
.pgoff = linear_page_index(vma, vmf->address),
- .virtual_address =
- (void __user *)(vmf->address & PAGE_MASK),
+ .address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
@@ -2856,7 +2855,7 @@ static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct vm_fault vmf2;
int ret;
- vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.address = vmf->address;
vmf2.pgoff = pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Every single user of vmf->virtual_address typed that entry to unsigned
long before doing anything with it so the type of virtual_address does
not really provide us any additional safety. Just use masked
vmf->address which already has the appropriate type.
Signed-off-by: Jan Kara <jack@suse.cz>
---
arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
arch/x86/entry/vdso/vma.c | 4 ++--
drivers/char/agp/alpha-agp.c | 2 +-
drivers/char/mspec.c | 2 +-
drivers/dax/dax.c | 2 +-
drivers/gpu/drm/armada/armada_gem.c | 2 +-
drivers/gpu/drm/drm_vm.c | 11 ++++++-----
drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
drivers/gpu/drm/gma500/framebuffer.c | 2 +-
drivers/gpu/drm/gma500/gem.c | 5 ++---
drivers/gpu/drm/i915/i915_gem.c | 2 +-
drivers/gpu/drm/msm/msm_gem.c | 7 +++----
drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
drivers/gpu/drm/tegra/gem.c | 4 ++--
drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
drivers/gpu/drm/udl/udl_gem.c | 5 ++---
drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
drivers/misc/cxl/context.c | 2 +-
drivers/misc/sgi-gru/grumain.c | 2 +-
drivers/staging/android/ion/ion.c | 2 +-
drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
drivers/xen/privcmd.c | 2 +-
fs/dax.c | 4 ++--
include/linux/mm.h | 2 --
mm/memory.c | 7 +++----
27 files changed, 59 insertions(+), 65 deletions(-)
diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
index 06254467e4dd..e8a31fffcdda 100644
--- a/arch/powerpc/platforms/cell/spufs/file.c
+++ b/arch/powerpc/platforms/cell/spufs/file.c
@@ -236,7 +236,7 @@ static int
spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct spu_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
unsigned long pfn, offset;
offset = vmf->pgoff << PAGE_SHIFT;
@@ -355,7 +355,7 @@ static int spufs_ps_fault(struct vm_area_struct *vma,
down_read(¤t->mm->mmap_sem);
} else {
area = ctx->spu->problem_phys + ps_offs;
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK,
(area + offset) >> PAGE_SHIFT);
spu_context_trace(spufs_ps_fault__insert, ctx, ctx->spu);
}
diff --git a/arch/x86/entry/vdso/vma.c b/arch/x86/entry/vdso/vma.c
index 23c881caabd1..e20a5cb6cd31 100644
--- a/arch/x86/entry/vdso/vma.c
+++ b/arch/x86/entry/vdso/vma.c
@@ -109,7 +109,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
return VM_FAULT_SIGBUS;
if (sym_offset == image->sym_vvar_page) {
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK,
__pa_symbol(&__vvar_page) >> PAGE_SHIFT);
} else if (sym_offset == image->sym_pvclock_page) {
struct pvclock_vsyscall_time_info *pvti =
@@ -117,7 +117,7 @@ static int vvar_fault(const struct vm_special_mapping *sm,
if (pvti && vclock_was_used(VCLOCK_PVCLOCK)) {
ret = vm_insert_pfn(
vma,
- (unsigned long)vmf->virtual_address,
+ vmf->address & PAGE_MASK,
__pa(pvti) >> PAGE_SHIFT);
}
}
diff --git a/drivers/char/agp/alpha-agp.c b/drivers/char/agp/alpha-agp.c
index 199b8e99f7d7..372d9378d997 100644
--- a/drivers/char/agp/alpha-agp.c
+++ b/drivers/char/agp/alpha-agp.c
@@ -19,7 +19,7 @@ static int alpha_core_agp_vm_fault(struct vm_area_struct *vma,
unsigned long pa;
struct page *page;
- dma_addr = (unsigned long)vmf->virtual_address - vma->vm_start
+ dma_addr = (vmf->address & PAGE_MASK) - vma->vm_start
+ agp->aperture.bus_base;
pa = agp->ops->translate(agp, dma_addr);
diff --git a/drivers/char/mspec.c b/drivers/char/mspec.c
index f3f92d5fcda0..2b7e1bc9ac5c 100644
--- a/drivers/char/mspec.c
+++ b/drivers/char/mspec.c
@@ -227,7 +227,7 @@ mspec_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* be because another thread has installed the pte first, so it
* is no problem.
*/
- vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
return VM_FAULT_NOPAGE;
}
diff --git a/drivers/dax/dax.c b/drivers/dax/dax.c
index 0e499bfca41c..6100c6bb52c5 100644
--- a/drivers/dax/dax.c
+++ b/drivers/dax/dax.c
@@ -328,7 +328,7 @@ static phys_addr_t pgoff_to_phys(struct dax_dev *dax_dev, pgoff_t pgoff,
static int __dax_dev_fault(struct dax_dev *dax_dev, struct vm_area_struct *vma,
struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long) vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct device *dev = &dax_dev->dev;
struct dax_region *dax_region;
int rc = VM_FAULT_SIGBUS;
diff --git a/drivers/gpu/drm/armada/armada_gem.c b/drivers/gpu/drm/armada/armada_gem.c
index 806791897304..6ccb70dce013 100644
--- a/drivers/gpu/drm/armada/armada_gem.c
+++ b/drivers/gpu/drm/armada/armada_gem.c
@@ -17,7 +17,7 @@
static int armada_gem_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct armada_gem_object *obj = drm_to_armada_gem(vma->vm_private_data);
- unsigned long addr = (unsigned long)vmf->virtual_address;
+ unsigned long addr = vmf->address & PAGE_MASK;
unsigned long pfn = obj->phys_addr >> PAGE_SHIFT;
int ret;
diff --git a/drivers/gpu/drm/drm_vm.c b/drivers/gpu/drm/drm_vm.c
index caa4e4ca616d..7a67e6198819 100644
--- a/drivers/gpu/drm/drm_vm.c
+++ b/drivers/gpu/drm/drm_vm.c
@@ -124,8 +124,8 @@ static int drm_do_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* Using vm_pgoff as a selector forces us to use this unusual
* addressing scheme.
*/
- resource_size_t offset = (unsigned long)vmf->virtual_address -
- vma->vm_start;
+ resource_size_t offset = (vmf->address & PAGE_MASK) -
+ vma->vm_start;
resource_size_t baddr = map->offset + offset;
struct drm_agp_mem *agpmem;
struct page *page;
@@ -195,7 +195,7 @@ static int drm_do_vm_shm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!map)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
i = (unsigned long)map->handle + offset;
page = vmalloc_to_page((void *)i);
if (!page)
@@ -301,7 +301,8 @@ static int drm_do_vm_dma_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!dma->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start; /* vm_[pg]off[set] should be 0 */
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
+ /* vm_[pg]off[set] should be 0 */
page_nr = offset >> PAGE_SHIFT; /* page_nr could just be vmf->pgoff */
page = virt_to_page((void *)dma->pagelist[page_nr]);
@@ -337,7 +338,7 @@ static int drm_do_vm_sg_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!entry->pagelist)
return VM_FAULT_SIGBUS; /* Nothing allocated */
- offset = (unsigned long)vmf->virtual_address - vma->vm_start;
+ offset = (vmf->address & PAGE_MASK) - vma->vm_start;
map_offset = map->offset - (unsigned long)dev->sg->virtual;
page_offset = (offset >> PAGE_SHIFT) + (map_offset >> PAGE_SHIFT);
page = entry->pagelist[page_offset];
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_gem.c b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
index 0370b842d9cc..9a338a91dc08 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_gem.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_gem.c
@@ -202,15 +202,14 @@ int etnaviv_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
page = pages[pgoff];
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
page_to_pfn(page), page_to_pfn(page) << PAGE_SHIFT);
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
out:
switch (ret) {
diff --git a/drivers/gpu/drm/exynos/exynos_drm_gem.c b/drivers/gpu/drm/exynos/exynos_drm_gem.c
index f2ae72ba7d5a..2e57d5067170 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_gem.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_gem.c
@@ -455,8 +455,8 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
pgoff_t page_offset;
int ret;
- page_offset = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK) - vma->vm_start) >>
+ PAGE_SHIFT;
if (page_offset >= (exynos_gem->size >> PAGE_SHIFT)) {
DRM_ERROR("invalid page offset\n");
@@ -465,7 +465,7 @@ int exynos_drm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
pfn = page_to_pfn(exynos_gem->pages[page_offset]);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out:
diff --git a/drivers/gpu/drm/gma500/framebuffer.c b/drivers/gpu/drm/gma500/framebuffer.c
index 3a44e705db53..fbc336ee151d 100644
--- a/drivers/gpu/drm/gma500/framebuffer.c
+++ b/drivers/gpu/drm/gma500/framebuffer.c
@@ -125,7 +125,7 @@ static int psbfb_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
psbfb->gtt->offset;
page_num = (vma->vm_end - vma->vm_start) >> PAGE_SHIFT;
- address = (unsigned long)vmf->virtual_address - (vmf->pgoff << PAGE_SHIFT);
+ address = (vmf->address & PAGE_MASK) - (vmf->pgoff << PAGE_SHIFT);
vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
diff --git a/drivers/gpu/drm/gma500/gem.c b/drivers/gpu/drm/gma500/gem.c
index 6d1cb6b370b1..0064dcbcbd2b 100644
--- a/drivers/gpu/drm/gma500/gem.c
+++ b/drivers/gpu/drm/gma500/gem.c
@@ -197,15 +197,14 @@ int psb_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
/* Page relative to the VMA start - we must calculate this ourselves
because vmf->pgoff is the fake GEM offset */
- page_offset = ((unsigned long) vmf->virtual_address - vma->vm_start)
- >> PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
/* CPU view of the page, don't go via the GART for CPU writes */
if (r->stolen)
pfn = (dev_priv->stolen_base + r->offset) >> PAGE_SHIFT;
else
pfn = page_to_pfn(r->pages[page_offset]);
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
fail:
mutex_unlock(&dev_priv->mmap_mutex);
diff --git a/drivers/gpu/drm/i915/i915_gem.c b/drivers/gpu/drm/i915/i915_gem.c
index 947e82c2b175..9733b0274494 100644
--- a/drivers/gpu/drm/i915/i915_gem.c
+++ b/drivers/gpu/drm/i915/i915_gem.c
@@ -1763,7 +1763,7 @@ int i915_gem_fault(struct vm_area_struct *area, struct vm_fault *vmf)
int ret;
/* We don't use vmf->pgoff since that has the fake offset */
- page_offset = ((unsigned long)vmf->virtual_address - area->vm_start) >>
+ page_offset = ((vmf->address & PAGE_MASK)- area->vm_start) >>
PAGE_SHIFT;
trace_i915_gem_object_fault(obj, page_offset, true, write);
diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c
index b6ac27e31929..84890d604fd3 100644
--- a/drivers/gpu/drm/msm/msm_gem.c
+++ b/drivers/gpu/drm/msm/msm_gem.c
@@ -225,15 +225,14 @@ int msm_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
}
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
pfn = page_to_pfn(pages[pgoff]);
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- ret = vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ ret = vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
out_unlock:
diff --git a/drivers/gpu/drm/omapdrm/omap_gem.c b/drivers/gpu/drm/omapdrm/omap_gem.c
index 505dee0db973..86cdd3e9248f 100644
--- a/drivers/gpu/drm/omapdrm/omap_gem.c
+++ b/drivers/gpu/drm/omapdrm/omap_gem.c
@@ -396,8 +396,7 @@ static int fault_1d(struct drm_gem_object *obj,
pgoff_t pgoff;
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (omap_obj->pages) {
omap_gem_cpu_sync(obj, pgoff);
@@ -407,10 +406,10 @@ static int fault_1d(struct drm_gem_object *obj,
pfn = (omap_obj->paddr >> PAGE_SHIFT) + pgoff;
}
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
- return vm_insert_mixed(vma, (unsigned long)vmf->virtual_address,
+ return vm_insert_mixed(vma, vmf->address & PAGE_MASK,
__pfn_to_pfn_t(pfn, PFN_DEV));
}
@@ -425,7 +424,7 @@ static int fault_2d(struct drm_gem_object *obj,
struct page *pages[64]; /* XXX is this too much to have on stack? */
unsigned long pfn;
pgoff_t pgoff, base_pgoff;
- void __user *vaddr;
+ unsigned long vaddr;
int i, ret, slots;
/*
@@ -445,8 +444,7 @@ static int fault_2d(struct drm_gem_object *obj,
const int m = 1 + ((omap_obj->width << fmt) / PAGE_SIZE);
/* We don't use vmf->pgoff since that has the fake offset: */
- pgoff = ((unsigned long)vmf->virtual_address -
- vma->vm_start) >> PAGE_SHIFT;
+ pgoff = ((vmf->address & PAGE_MASK) - vma->vm_start) >> PAGE_SHIFT;
/*
* Actual address we start mapping at is rounded down to previous slot
@@ -457,7 +455,8 @@ static int fault_2d(struct drm_gem_object *obj,
/* figure out buffer width in slots */
slots = omap_obj->width >> priv->usergart[fmt].slot_shift;
- vaddr = vmf->virtual_address - ((pgoff - base_pgoff) << PAGE_SHIFT);
+ vaddr = (vmf->address & PAGE_MASK) -
+ ((pgoff - base_pgoff) << PAGE_SHIFT);
entry = &priv->usergart[fmt].entry[priv->usergart[fmt].last];
@@ -501,12 +500,11 @@ static int fault_2d(struct drm_gem_object *obj,
pfn = entry->paddr >> PAGE_SHIFT;
- VERB("Inserting %p pfn %lx, pa %lx", vmf->virtual_address,
+ VERB("Inserting %p pfn %lx, pa %lx", (void *)(vmf->address & PAGE_MASK),
pfn, pfn << PAGE_SHIFT);
for (i = n; i > 0; i--) {
- vm_insert_mixed(vma, (unsigned long)vaddr,
- __pfn_to_pfn_t(pfn, PFN_DEV));
+ vm_insert_mixed(vma, vaddr, __pfn_to_pfn_t(pfn, PFN_DEV));
pfn += priv->usergart[fmt].stride_pfn;
vaddr += PAGE_SIZE * m;
}
diff --git a/drivers/gpu/drm/tegra/gem.c b/drivers/gpu/drm/tegra/gem.c
index 95e622e31931..6d1a5b467ff2 100644
--- a/drivers/gpu/drm/tegra/gem.c
+++ b/drivers/gpu/drm/tegra/gem.c
@@ -427,10 +427,10 @@ static int tegra_bo_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
if (!bo->pages)
return VM_FAULT_SIGBUS;
- offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >> PAGE_SHIFT;
+ offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
page = bo->pages[offset];
- err = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ err = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (err) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c
index a6ed9d5e5167..4fe0bbef7119 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_vm.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c
@@ -101,7 +101,7 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
int ret;
int i;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
int retval = VM_FAULT_NOPAGE;
struct ttm_mem_type_manager *man =
&bdev->man[bo->mem.mem_type];
diff --git a/drivers/gpu/drm/udl/udl_gem.c b/drivers/gpu/drm/udl/udl_gem.c
index 818e70712b18..091e38a04fe6 100644
--- a/drivers/gpu/drm/udl/udl_gem.c
+++ b/drivers/gpu/drm/udl/udl_gem.c
@@ -107,14 +107,13 @@ int udl_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned int page_offset;
int ret = 0;
- page_offset = ((unsigned long)vmf->virtual_address - vma->vm_start) >>
- PAGE_SHIFT;
+ page_offset = ((vmf->address & PAGE_MASK)- vma->vm_start) >> PAGE_SHIFT;
if (!obj->pages)
return VM_FAULT_SIGBUS;
page = obj->pages[page_offset];
- ret = vm_insert_page(vma, (unsigned long)vmf->virtual_address, page);
+ ret = vm_insert_page(vma, vmf->address & PAGE_MASK, page);
switch (ret) {
case -EAGAIN:
case 0:
diff --git a/drivers/gpu/drm/vgem/vgem_drv.c b/drivers/gpu/drm/vgem/vgem_drv.c
index f36c14729b55..46b4d16e4e17 100644
--- a/drivers/gpu/drm/vgem/vgem_drv.c
+++ b/drivers/gpu/drm/vgem/vgem_drv.c
@@ -54,7 +54,7 @@ static int vgem_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct drm_vgem_gem_object *obj = vma->vm_private_data;
/* We don't use vmf->pgoff since that has the fake offset */
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct page *page;
page = shmem_read_mapping_page(file_inode(obj->base.filp)->i_mapping,
diff --git a/drivers/media/v4l2-core/videobuf-dma-sg.c b/drivers/media/v4l2-core/videobuf-dma-sg.c
index 1db0af6c7f94..b11e0008e436 100644
--- a/drivers/media/v4l2-core/videobuf-dma-sg.c
+++ b/drivers/media/v4l2-core/videobuf-dma-sg.c
@@ -439,13 +439,12 @@ static int videobuf_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
struct page *page;
dprintk(3, "fault: fault @ %08lx [vma %08lx-%08lx]\n",
- (unsigned long)vmf->virtual_address,
- vma->vm_start, vma->vm_end);
+ vmf->address & PAGE_MASK, vma->vm_start, vma->vm_end);
page = alloc_page(GFP_USER | __GFP_DMA32);
if (!page)
return VM_FAULT_OOM;
- clear_user_highpage(page, (unsigned long)vmf->virtual_address);
+ clear_user_highpage(page, vmf->address & PAGE_MASK);
vmf->page = page;
return 0;
diff --git a/drivers/misc/cxl/context.c b/drivers/misc/cxl/context.c
index 5e506c19108a..0de6d1334fcd 100644
--- a/drivers/misc/cxl/context.c
+++ b/drivers/misc/cxl/context.c
@@ -117,7 +117,7 @@ int cxl_context_init(struct cxl_context *ctx, struct cxl_afu *afu, bool master,
static int cxl_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct cxl_context *ctx = vma->vm_file->private_data;
- unsigned long address = (unsigned long)vmf->virtual_address;
+ unsigned long address = vmf->address & PAGE_MASK;
u64 area, offset;
offset = vmf->pgoff << PAGE_SHIFT;
diff --git a/drivers/misc/sgi-gru/grumain.c b/drivers/misc/sgi-gru/grumain.c
index 33741ad4a74a..677c4dd39561 100644
--- a/drivers/misc/sgi-gru/grumain.c
+++ b/drivers/misc/sgi-gru/grumain.c
@@ -932,7 +932,7 @@ int gru_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
unsigned long paddr, vaddr;
unsigned long expires;
- vaddr = (unsigned long)vmf->virtual_address;
+ vaddr = vmf->address & PAGE_MASK;
gru_dbg(grudev, "vma %p, vaddr 0x%lx (0x%lx)\n",
vma, vaddr, GSEG_BASE(vaddr));
STAT(nopfn);
diff --git a/drivers/staging/android/ion/ion.c b/drivers/staging/android/ion/ion.c
index 209a8f7ef02b..5aebbb380271 100644
--- a/drivers/staging/android/ion/ion.c
+++ b/drivers/staging/android/ion/ion.c
@@ -882,7 +882,7 @@ static int ion_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
BUG_ON(!buffer->pages || !buffer->pages[vmf->pgoff]);
pfn = page_to_pfn(ion_buffer_page(buffer->pages[vmf->pgoff]));
- ret = vm_insert_pfn(vma, (unsigned long)vmf->virtual_address, pfn);
+ ret = vm_insert_pfn(vma, vmf->address & PAGE_MASK, pfn);
mutex_unlock(&buffer->lock);
if (ret)
return VM_FAULT_ERROR;
diff --git a/drivers/staging/lustre/lustre/llite/vvp_io.c b/drivers/staging/lustre/lustre/llite/vvp_io.c
index 2b7f182a15e2..1491d788dcab 100644
--- a/drivers/staging/lustre/lustre/llite/vvp_io.c
+++ b/drivers/staging/lustre/lustre/llite/vvp_io.c
@@ -984,7 +984,8 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
"page %p map %p index %lu flags %lx count %u priv %0lx: got addr %p type NOPAGE\n",
vmf->page, vmf->page->mapping, vmf->page->index,
(long)vmf->page->flags, page_count(vmf->page),
- page_private(vmf->page), vmf->virtual_address);
+ page_private(vmf->page),
+ (void *)(vmf->address & PAGE_MASK));
if (unlikely(!(cfio->ft_flags & VM_FAULT_LOCKED))) {
lock_page(vmf->page);
cfio->ft_flags |= VM_FAULT_LOCKED;
@@ -995,12 +996,14 @@ static int vvp_io_kernel_fault(struct vvp_fault_io *cfio)
}
if (cfio->ft_flags & (VM_FAULT_SIGBUS | VM_FAULT_SIGSEGV)) {
- CDEBUG(D_PAGE, "got addr %p - SIGBUS\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - SIGBUS\n",
+ (void *)(vmf->address & PAGE_MASK));
return -EFAULT;
}
if (cfio->ft_flags & VM_FAULT_OOM) {
- CDEBUG(D_PAGE, "got addr %p - OOM\n", vmf->virtual_address);
+ CDEBUG(D_PAGE, "got addr %p - OOM\n",
+ (void *)(vmf->address & PAGE_MASK));
return -ENOMEM;
}
diff --git a/drivers/xen/privcmd.c b/drivers/xen/privcmd.c
index 702040fe2001..1334aaf9016d 100644
--- a/drivers/xen/privcmd.c
+++ b/drivers/xen/privcmd.c
@@ -602,7 +602,7 @@ static int privcmd_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
printk(KERN_DEBUG "privcmd_fault: vma=%p %lx-%lx, pgoff=%lx, uv=%p\n",
vma, vma->vm_start, vma->vm_end,
- vmf->pgoff, vmf->virtual_address);
+ vmf->pgoff, (void *)(vmf->address & PAGE_MASK));
return VM_FAULT_SIGBUS;
}
diff --git a/fs/dax.c b/fs/dax.c
index ad131cd2605d..716a0f9c769c 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -737,7 +737,7 @@ static int dax_insert_mapping(struct address_space *mapping,
struct block_device *bdev, sector_t sector, size_t size,
void **entryp, struct vm_area_struct *vma, struct vm_fault *vmf)
{
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
struct blk_dax_ctl dax = {
.sector = sector,
.size = size,
@@ -947,7 +947,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
{
struct address_space *mapping = vma->vm_file->f_mapping;
struct inode *inode = mapping->host;
- unsigned long vaddr = (unsigned long)vmf->virtual_address;
+ unsigned long vaddr = vmf->address & PAGE_MASK;
loff_t pos = (loff_t)vmf->pgoff << PAGE_SHIFT;
sector_t sector;
struct iomap iomap = { 0 };
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 657eb69eb87e..df3958437473 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -297,8 +297,6 @@ struct vm_fault {
gfp_t gfp_mask; /* gfp mask to be used for allocations */
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
- void __user *virtual_address; /* Faulting virtual address masked by
- * PAGE_MASK */
pmd_t *pmd; /* Pointer to pmd entry matching
* the 'address'
*/
diff --git a/mm/memory.c b/mm/memory.c
index fad45cd59ba7..c652b65469cd 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2044,7 +2044,7 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
struct vm_fault vmf;
int ret;
- vmf.virtual_address = (void __user *)(address & PAGE_MASK);
+ vmf.address = address;
vmf.pgoff = page->index;
vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
vmf.gfp_mask = __get_fault_gfp_mask(vma);
@@ -2280,8 +2280,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_fault vmf2 = {
.page = NULL,
.pgoff = linear_page_index(vma, vmf->address),
- .virtual_address =
- (void __user *)(vmf->address & PAGE_MASK),
+ .address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
int ret;
@@ -2856,7 +2855,7 @@ static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
struct vm_fault vmf2;
int ret;
- vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
+ vmf2.address = vmf->address;
vmf2.pgoff = pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:24 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
struct vm_fault has already pgoff entry. Use it instead of passing pgoff
as a separate argument and then assigning it later.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/khugepaged.c | 1 +
mm/memory.c | 35 ++++++++++++++++++-----------------
2 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f88b2d3810a7..d7df06383b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -880,6 +880,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
.pmd = pmd,
+ .pgoff = linear_page_index(vma, address),
};
/* we only decide to swapin, if there is enough young ptes */
diff --git a/mm/memory.c b/mm/memory.c
index c652b65469cd..3b79eace8d23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2279,7 +2279,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, vmf->address),
+ .pgoff = vmf->pgoff,
.address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
@@ -2848,15 +2848,15 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
- struct page *cow_page, struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
+ struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
struct vm_fault vmf2;
int ret;
vmf2.address = vmf->address;
- vmf2.pgoff = pgoff;
+ vmf2.pgoff = vmf->pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
vmf2.gfp_mask = __get_fault_gfp_mask(vma);
@@ -3115,9 +3115,10 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf)
{
unsigned long address = vmf->address, nr_pages, mask;
+ pgoff_t start_pgoff = vmf->pgoff;
pgoff_t end_pgoff;
int off, ret = 0;
@@ -3175,7 +3176,7 @@ static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
return ret;
}
-static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3187,12 +3188,12 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(vmf, pgoff);
+ ret = do_fault_around(vmf);
if (ret)
return ret;
}
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3205,7 +3206,7 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
@@ -3226,7 +3227,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3241,7 +3242,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
unlock_page(fault_page);
put_page(fault_page);
} else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, pgoff);
+ dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3252,7 +3253,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3260,7 +3261,7 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3321,16 +3322,15 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
static int do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
if (!(vmf->flags & FAULT_FLAG_WRITE))
- return do_read_fault(vmf, pgoff);
+ return do_read_fault(vmf);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(vmf, pgoff);
- return do_shared_fault(vmf, pgoff);
+ return do_cow_fault(vmf);
+ return do_shared_fault(vmf);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3578,6 +3578,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.vma = vma,
.address = address,
.flags = flags,
+ .pgoff = linear_page_index(vma, address),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
struct vm_fault has already pgoff entry. Use it instead of passing pgoff
as a separate argument and then assigning it later.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/khugepaged.c | 1 +
mm/memory.c | 35 ++++++++++++++++++-----------------
2 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f88b2d3810a7..d7df06383b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -880,6 +880,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
.pmd = pmd,
+ .pgoff = linear_page_index(vma, address),
};
/* we only decide to swapin, if there is enough young ptes */
diff --git a/mm/memory.c b/mm/memory.c
index c652b65469cd..3b79eace8d23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2279,7 +2279,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, vmf->address),
+ .pgoff = vmf->pgoff,
.address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
@@ -2848,15 +2848,15 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
- struct page *cow_page, struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
+ struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
struct vm_fault vmf2;
int ret;
vmf2.address = vmf->address;
- vmf2.pgoff = pgoff;
+ vmf2.pgoff = vmf->pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
vmf2.gfp_mask = __get_fault_gfp_mask(vma);
@@ -3115,9 +3115,10 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf)
{
unsigned long address = vmf->address, nr_pages, mask;
+ pgoff_t start_pgoff = vmf->pgoff;
pgoff_t end_pgoff;
int off, ret = 0;
@@ -3175,7 +3176,7 @@ static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
return ret;
}
-static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3187,12 +3188,12 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(vmf, pgoff);
+ ret = do_fault_around(vmf);
if (ret)
return ret;
}
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3205,7 +3206,7 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
@@ -3226,7 +3227,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3241,7 +3242,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
unlock_page(fault_page);
put_page(fault_page);
} else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, pgoff);
+ dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3252,7 +3253,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3260,7 +3261,7 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3321,16 +3322,15 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
static int do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
if (!(vmf->flags & FAULT_FLAG_WRITE))
- return do_read_fault(vmf, pgoff);
+ return do_read_fault(vmf);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(vmf, pgoff);
- return do_shared_fault(vmf, pgoff);
+ return do_cow_fault(vmf);
+ return do_shared_fault(vmf);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3578,6 +3578,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.vma = vma,
.address = address,
.flags = flags,
+ .pgoff = linear_page_index(vma, address),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately
@ 2016-11-04 4:24 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:24 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
struct vm_fault has already pgoff entry. Use it instead of passing pgoff
as a separate argument and then assigning it later.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/khugepaged.c | 1 +
mm/memory.c | 35 ++++++++++++++++++-----------------
2 files changed, 19 insertions(+), 17 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index f88b2d3810a7..d7df06383b10 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -880,6 +880,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
.address = address,
.flags = FAULT_FLAG_ALLOW_RETRY,
.pmd = pmd,
+ .pgoff = linear_page_index(vma, address),
};
/* we only decide to swapin, if there is enough young ptes */
diff --git a/mm/memory.c b/mm/memory.c
index c652b65469cd..3b79eace8d23 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2279,7 +2279,7 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
struct vm_fault vmf2 = {
.page = NULL,
- .pgoff = linear_page_index(vma, vmf->address),
+ .pgoff = vmf->pgoff,
.address = vmf->address,
.flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
};
@@ -2848,15 +2848,15 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
- struct page *cow_page, struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
+ struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
struct vm_fault vmf2;
int ret;
vmf2.address = vmf->address;
- vmf2.pgoff = pgoff;
+ vmf2.pgoff = vmf->pgoff;
vmf2.flags = vmf->flags;
vmf2.page = NULL;
vmf2.gfp_mask = __get_fault_gfp_mask(vma);
@@ -3115,9 +3115,10 @@ late_initcall(fault_around_debugfs);
* fault_around_pages() value (and therefore to page order). This way it's
* easier to guarantee that we don't cross page table boundaries.
*/
-static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
+static int do_fault_around(struct vm_fault *vmf)
{
unsigned long address = vmf->address, nr_pages, mask;
+ pgoff_t start_pgoff = vmf->pgoff;
pgoff_t end_pgoff;
int off, ret = 0;
@@ -3175,7 +3176,7 @@ static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
return ret;
}
-static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3187,12 +3188,12 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
* something).
*/
if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
- ret = do_fault_around(vmf, pgoff);
+ ret = do_fault_around(vmf);
if (ret)
return ret;
}
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3205,7 +3206,7 @@ static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page, *new_page;
@@ -3226,7 +3227,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
+ ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3241,7 +3242,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
unlock_page(fault_page);
put_page(fault_page);
} else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, pgoff);
+ dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
@@ -3252,7 +3253,7 @@ static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
return ret;
}
-static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
+static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *fault_page;
@@ -3260,7 +3261,7 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf, NULL, &fault_page, NULL);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3321,16 +3322,15 @@ static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
static int do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- pgoff_t pgoff = linear_page_index(vma, vmf->address);
/* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
if (!vma->vm_ops->fault)
return VM_FAULT_SIGBUS;
if (!(vmf->flags & FAULT_FLAG_WRITE))
- return do_read_fault(vmf, pgoff);
+ return do_read_fault(vmf);
if (!(vma->vm_flags & VM_SHARED))
- return do_cow_fault(vmf, pgoff);
- return do_shared_fault(vmf, pgoff);
+ return do_cow_fault(vmf);
+ return do_shared_fault(vmf);
}
static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
@@ -3578,6 +3578,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.vma = vma,
.address = address,
.flags = flags,
+ .pgoff = linear_page_index(vma, address),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault()
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Instead of creating another vm_fault structure, use the one passed to
__do_fault() for passing arguments into fault handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 3b79eace8d23..8145dadb2645 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2852,37 +2852,31 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
- struct vm_fault vmf2;
int ret;
- vmf2.address = vmf->address;
- vmf2.pgoff = vmf->pgoff;
- vmf2.flags = vmf->flags;
- vmf2.page = NULL;
- vmf2.gfp_mask = __get_fault_gfp_mask(vma);
- vmf2.cow_page = cow_page;
+ vmf->cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf2);
+ ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf2.entry;
+ *entry = vmf->entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf2.page))) {
+ if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf2.page);
- put_page(vmf2.page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf2.page);
+ lock_page(vmf->page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf2.page;
+ *page = vmf->page;
return ret;
}
@@ -3579,6 +3573,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.address = address,
.flags = flags,
.pgoff = linear_page_index(vma, address),
+ .gfp_mask = __get_fault_gfp_mask(vma),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Instead of creating another vm_fault structure, use the one passed to
__do_fault() for passing arguments into fault handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 3b79eace8d23..8145dadb2645 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2852,37 +2852,31 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
- struct vm_fault vmf2;
int ret;
- vmf2.address = vmf->address;
- vmf2.pgoff = vmf->pgoff;
- vmf2.flags = vmf->flags;
- vmf2.page = NULL;
- vmf2.gfp_mask = __get_fault_gfp_mask(vma);
- vmf2.cow_page = cow_page;
+ vmf->cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf2);
+ ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf2.entry;
+ *entry = vmf->entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf2.page))) {
+ if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf2.page);
- put_page(vmf2.page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf2.page);
+ lock_page(vmf->page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf2.page;
+ *page = vmf->page;
return ret;
}
@@ -3579,6 +3573,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.address = address,
.flags = flags,
.pgoff = linear_page_index(vma, address),
+ .gfp_mask = __get_fault_gfp_mask(vma),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Instead of creating another vm_fault structure, use the one passed to
__do_fault() for passing arguments into fault handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 3b79eace8d23..8145dadb2645 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2852,37 +2852,31 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
struct page **page, void **entry)
{
struct vm_area_struct *vma = vmf->vma;
- struct vm_fault vmf2;
int ret;
- vmf2.address = vmf->address;
- vmf2.pgoff = vmf->pgoff;
- vmf2.flags = vmf->flags;
- vmf2.page = NULL;
- vmf2.gfp_mask = __get_fault_gfp_mask(vma);
- vmf2.cow_page = cow_page;
+ vmf->cow_page = cow_page;
- ret = vma->vm_ops->fault(vma, &vmf2);
+ ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf2.entry;
+ *entry = vmf->entry;
return ret;
}
- if (unlikely(PageHWPoison(vmf2.page))) {
+ if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
- unlock_page(vmf2.page);
- put_page(vmf2.page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return VM_FAULT_HWPOISON;
}
if (unlikely(!(ret & VM_FAULT_LOCKED)))
- lock_page(vmf2.page);
+ lock_page(vmf->page);
else
- VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
+ VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf2.page;
+ *page = vmf->page;
return ret;
}
@@ -3579,6 +3573,7 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
.address = address,
.flags = flags,
.pgoff = linear_page_index(vma, address),
+ .gfp_mask = __get_fault_gfp_mask(vma),
};
struct mm_struct *mm = vma->vm_mm;
pgd_t *pgd;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 05/21] mm: Trim __do_fault() arguments
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Use vm_fault structure to pass cow_page, page, and entry in and out of
the function. That reduces number of __do_fault() arguments from 4 to 1.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 53 +++++++++++++++++++++++------------------------------
1 file changed, 23 insertions(+), 30 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 8145dadb2645..f5ef7b8a30c5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,26 +2848,22 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
- struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
int ret;
- vmf->cow_page = cow_page;
-
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf->entry;
+ if (ret & VM_FAULT_DAX_LOCKED)
return ret;
- }
if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
unlock_page(vmf->page);
put_page(vmf->page);
+ vmf->page = NULL;
return VM_FAULT_HWPOISON;
}
@@ -2876,7 +2872,6 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
else
VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf->page;
return ret;
}
@@ -3173,7 +3168,6 @@ static int do_fault_around(struct vm_fault *vmf)
static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
int ret = 0;
/*
@@ -3187,24 +3181,23 @@ static int do_read_fault(struct vm_fault *vmf)
return ret;
}
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
- unlock_page(fault_page);
+ unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- put_page(fault_page);
+ put_page(vmf->page);
return ret;
}
static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page, *new_page;
- void *fault_entry;
+ struct page *new_page;
struct mem_cgroup *memcg;
int ret;
@@ -3221,20 +3214,21 @@ static int do_cow_fault(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
+ vmf->cow_page = new_page;
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
ret |= alloc_set_pte(vmf, memcg, new_page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
} else {
dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
@@ -3250,12 +3244,11 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3264,26 +3257,26 @@ static int do_shared_fault(struct vm_fault *vmf)
* about to become writable
*/
if (vma->vm_ops->page_mkwrite) {
- unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, vmf->address);
+ unlock_page(vmf->page);
+ tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(fault_page);
+ put_page(vmf->page);
return tmp;
}
}
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return ret;
}
- if (set_page_dirty(fault_page))
+ if (set_page_dirty(vmf->page))
dirtied = 1;
/*
* Take a local copy of the address_space - page.mapping may be zeroed
@@ -3291,8 +3284,8 @@ static int do_shared_fault(struct vm_fault *vmf)
* pinned by vma->vm_file's reference. We rely on unlock_page()'s
* release semantics to prevent the compiler from undoing this copying.
*/
- mapping = page_rmapping(fault_page);
- unlock_page(fault_page);
+ mapping = page_rmapping(vmf->page);
+ unlock_page(vmf->page);
if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
/*
* Some device drivers do not set page.mapping but still
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 05/21] mm: Trim __do_fault() arguments
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Use vm_fault structure to pass cow_page, page, and entry in and out of
the function. That reduces number of __do_fault() arguments from 4 to 1.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 53 +++++++++++++++++++++++------------------------------
1 file changed, 23 insertions(+), 30 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 8145dadb2645..f5ef7b8a30c5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,26 +2848,22 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
- struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
int ret;
- vmf->cow_page = cow_page;
-
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf->entry;
+ if (ret & VM_FAULT_DAX_LOCKED)
return ret;
- }
if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
unlock_page(vmf->page);
put_page(vmf->page);
+ vmf->page = NULL;
return VM_FAULT_HWPOISON;
}
@@ -2876,7 +2872,6 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
else
VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf->page;
return ret;
}
@@ -3173,7 +3168,6 @@ static int do_fault_around(struct vm_fault *vmf)
static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
int ret = 0;
/*
@@ -3187,24 +3181,23 @@ static int do_read_fault(struct vm_fault *vmf)
return ret;
}
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
- unlock_page(fault_page);
+ unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- put_page(fault_page);
+ put_page(vmf->page);
return ret;
}
static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page, *new_page;
- void *fault_entry;
+ struct page *new_page;
struct mem_cgroup *memcg;
int ret;
@@ -3221,20 +3214,21 @@ static int do_cow_fault(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
+ vmf->cow_page = new_page;
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
ret |= alloc_set_pte(vmf, memcg, new_page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
} else {
dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
@@ -3250,12 +3244,11 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3264,26 +3257,26 @@ static int do_shared_fault(struct vm_fault *vmf)
* about to become writable
*/
if (vma->vm_ops->page_mkwrite) {
- unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, vmf->address);
+ unlock_page(vmf->page);
+ tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(fault_page);
+ put_page(vmf->page);
return tmp;
}
}
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return ret;
}
- if (set_page_dirty(fault_page))
+ if (set_page_dirty(vmf->page))
dirtied = 1;
/*
* Take a local copy of the address_space - page.mapping may be zeroed
@@ -3291,8 +3284,8 @@ static int do_shared_fault(struct vm_fault *vmf)
* pinned by vma->vm_file's reference. We rely on unlock_page()'s
* release semantics to prevent the compiler from undoing this copying.
*/
- mapping = page_rmapping(fault_page);
- unlock_page(fault_page);
+ mapping = page_rmapping(vmf->page);
+ unlock_page(vmf->page);
if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
/*
* Some device drivers do not set page.mapping but still
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 05/21] mm: Trim __do_fault() arguments
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Use vm_fault structure to pass cow_page, page, and entry in and out of
the function. That reduces number of __do_fault() arguments from 4 to 1.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 53 +++++++++++++++++++++++------------------------------
1 file changed, 23 insertions(+), 30 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 8145dadb2645..f5ef7b8a30c5 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,26 +2848,22 @@ static int do_anonymous_page(struct vm_fault *vmf)
* released depending on flags and vma->vm_ops->fault() return value.
* See filemap_fault() and __lock_page_retry().
*/
-static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
- struct page **page, void **entry)
+static int __do_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
int ret;
- vmf->cow_page = cow_page;
-
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- if (ret & VM_FAULT_DAX_LOCKED) {
- *entry = vmf->entry;
+ if (ret & VM_FAULT_DAX_LOCKED)
return ret;
- }
if (unlikely(PageHWPoison(vmf->page))) {
if (ret & VM_FAULT_LOCKED)
unlock_page(vmf->page);
put_page(vmf->page);
+ vmf->page = NULL;
return VM_FAULT_HWPOISON;
}
@@ -2876,7 +2872,6 @@ static int __do_fault(struct vm_fault *vmf, struct page *cow_page,
else
VM_BUG_ON_PAGE(!PageLocked(vmf->page), vmf->page);
- *page = vmf->page;
return ret;
}
@@ -3173,7 +3168,6 @@ static int do_fault_around(struct vm_fault *vmf)
static int do_read_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
int ret = 0;
/*
@@ -3187,24 +3181,23 @@ static int do_read_fault(struct vm_fault *vmf)
return ret;
}
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
- unlock_page(fault_page);
+ unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- put_page(fault_page);
+ put_page(vmf->page);
return ret;
}
static int do_cow_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page, *new_page;
- void *fault_entry;
+ struct page *new_page;
struct mem_cgroup *memcg;
int ret;
@@ -3221,20 +3214,21 @@ static int do_cow_fault(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
- ret = __do_fault(vmf, new_page, &fault_page, &fault_entry);
+ vmf->cow_page = new_page;
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, fault_page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
ret |= alloc_set_pte(vmf, memcg, new_page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
} else {
dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
}
@@ -3250,12 +3244,11 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *fault_page;
struct address_space *mapping;
int dirtied = 0;
int ret, tmp;
- ret = __do_fault(vmf, NULL, &fault_page, NULL);
+ ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
@@ -3264,26 +3257,26 @@ static int do_shared_fault(struct vm_fault *vmf)
* about to become writable
*/
if (vma->vm_ops->page_mkwrite) {
- unlock_page(fault_page);
- tmp = do_page_mkwrite(vma, fault_page, vmf->address);
+ unlock_page(vmf->page);
+ tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(fault_page);
+ put_page(vmf->page);
return tmp;
}
}
- ret |= alloc_set_pte(vmf, NULL, fault_page);
+ ret |= alloc_set_pte(vmf, NULL, vmf->page);
if (vmf->pte)
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
- unlock_page(fault_page);
- put_page(fault_page);
+ unlock_page(vmf->page);
+ put_page(vmf->page);
return ret;
}
- if (set_page_dirty(fault_page))
+ if (set_page_dirty(vmf->page))
dirtied = 1;
/*
* Take a local copy of the address_space - page.mapping may be zeroed
@@ -3291,8 +3284,8 @@ static int do_shared_fault(struct vm_fault *vmf)
* pinned by vma->vm_file's reference. We rely on unlock_page()'s
* release semantics to prevent the compiler from undoing this copying.
*/
- mapping = page_rmapping(fault_page);
- unlock_page(fault_page);
+ mapping = page_rmapping(vmf->page);
+ unlock_page(vmf->page);
if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
/*
* Some device drivers do not set page.mapping but still
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared()
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Instead of creating another vm_fault structure, use the one passed to
wp_pfn_shared() for passing arguments into pfn_mkwrite handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index f5ef7b8a30c5..5f6bc9028a88 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2277,16 +2277,11 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf2 = {
- .page = NULL,
- .pgoff = vmf->pgoff,
- .address = vmf->address,
- .flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
- };
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
+ vmf->flags |= FAULT_FLAG_MKWRITE;
+ ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Instead of creating another vm_fault structure, use the one passed to
wp_pfn_shared() for passing arguments into pfn_mkwrite handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index f5ef7b8a30c5..5f6bc9028a88 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2277,16 +2277,11 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf2 = {
- .page = NULL,
- .pgoff = vmf->pgoff,
- .address = vmf->address,
- .flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
- };
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
+ vmf->flags |= FAULT_FLAG_MKWRITE;
+ ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Instead of creating another vm_fault structure, use the one passed to
wp_pfn_shared() for passing arguments into pfn_mkwrite handler.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 9 ++-------
1 file changed, 2 insertions(+), 7 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index f5ef7b8a30c5..5f6bc9028a88 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2277,16 +2277,11 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
struct vm_area_struct *vma = vmf->vma;
if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
- struct vm_fault vmf2 = {
- .page = NULL,
- .pgoff = vmf->pgoff,
- .address = vmf->address,
- .flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
- };
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
+ vmf->flags |= FAULT_FLAG_MKWRITE;
+ ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index df3958437473..5cc679b874eb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index df3958437473..5cc679b874eb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index df3958437473..5cc679b874eb 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
To allow full handling of COW faults add memcg field to struct vm_fault
and a return value of ->fault() handler meaning that COW fault is fully
handled and memcg charge must not be canceled. This will allow us to
remove knowledge about special DAX locking from the generic fault code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +++-
mm/memory.c | 8 +++++---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5cc679b874eb..34d2891e9195 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -301,7 +301,8 @@ struct vm_fault {
* the 'address' */
pte_t orig_pte; /* Value of PTE at the time of fault */
- struct page *cow_page; /* Handler may choose to COW */
+ struct page *cow_page; /* Page handler may use for COW fault */
+ struct mem_cgroup *memcg; /* Cgroup cow_page belongs to */
struct page *page; /* ->fault handlers should return a
* page here, unless VM_FAULT_NOPAGE
* is set (which is also implied by
@@ -1103,6 +1104,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
+#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index 25028422a578..ac901bb02398 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,9 +2848,8 @@ static int __do_fault(struct vm_fault *vmf)
int ret;
ret = vma->vm_ops->fault(vma, vmf);
- if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- return ret;
- if (ret & VM_FAULT_DAX_LOCKED)
+ if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
+ VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3209,9 +3208,12 @@ static int do_cow_fault(struct vm_fault *vmf)
}
vmf->cow_page = new_page;
+ vmf->memcg = memcg;
ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
+ if (ret & VM_FAULT_DONE_COW)
+ return ret;
if (!(ret & VM_FAULT_DAX_LOCKED))
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
To allow full handling of COW faults add memcg field to struct vm_fault
and a return value of ->fault() handler meaning that COW fault is fully
handled and memcg charge must not be canceled. This will allow us to
remove knowledge about special DAX locking from the generic fault code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +++-
mm/memory.c | 8 +++++---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5cc679b874eb..34d2891e9195 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -301,7 +301,8 @@ struct vm_fault {
* the 'address' */
pte_t orig_pte; /* Value of PTE at the time of fault */
- struct page *cow_page; /* Handler may choose to COW */
+ struct page *cow_page; /* Page handler may use for COW fault */
+ struct mem_cgroup *memcg; /* Cgroup cow_page belongs to */
struct page *page; /* ->fault handlers should return a
* page here, unless VM_FAULT_NOPAGE
* is set (which is also implied by
@@ -1103,6 +1104,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
+#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index 25028422a578..ac901bb02398 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,9 +2848,8 @@ static int __do_fault(struct vm_fault *vmf)
int ret;
ret = vma->vm_ops->fault(vma, vmf);
- if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- return ret;
- if (ret & VM_FAULT_DAX_LOCKED)
+ if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
+ VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3209,9 +3208,12 @@ static int do_cow_fault(struct vm_fault *vmf)
}
vmf->cow_page = new_page;
+ vmf->memcg = memcg;
ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
+ if (ret & VM_FAULT_DONE_COW)
+ return ret;
if (!(ret & VM_FAULT_DAX_LOCKED))
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
To allow full handling of COW faults add memcg field to struct vm_fault
and a return value of ->fault() handler meaning that COW fault is fully
handled and memcg charge must not be canceled. This will allow us to
remove knowledge about special DAX locking from the generic fault code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +++-
mm/memory.c | 8 +++++---
2 files changed, 8 insertions(+), 4 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 5cc679b874eb..34d2891e9195 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -301,7 +301,8 @@ struct vm_fault {
* the 'address' */
pte_t orig_pte; /* Value of PTE at the time of fault */
- struct page *cow_page; /* Handler may choose to COW */
+ struct page *cow_page; /* Page handler may use for COW fault */
+ struct mem_cgroup *memcg; /* Cgroup cow_page belongs to */
struct page *page; /* ->fault handlers should return a
* page here, unless VM_FAULT_NOPAGE
* is set (which is also implied by
@@ -1103,6 +1104,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
+#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index 25028422a578..ac901bb02398 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2848,9 +2848,8 @@ static int __do_fault(struct vm_fault *vmf)
int ret;
ret = vma->vm_ops->fault(vma, vmf);
- if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
- return ret;
- if (ret & VM_FAULT_DAX_LOCKED)
+ if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
+ VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3209,9 +3208,12 @@ static int do_cow_fault(struct vm_fault *vmf)
}
vmf->cow_page = new_page;
+ vmf->memcg = memcg;
ret = __do_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
+ if (ret & VM_FAULT_DONE_COW)
+ return ret;
if (!(ret & VM_FAULT_DAX_LOCKED))
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 09/21] mm: Factor out functionality to finish page faults
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Introduce function finish_fault() as a helper function for finishing
page faults. It is rather thin wrapper around alloc_set_pte() but since
we'd want to call this from DAX code or filesystems, it is still useful
to avoid some boilerplate code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 44 +++++++++++++++++++++++++++++++++++---------
2 files changed, 36 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 34d2891e9195..482455952f03 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -620,6 +620,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
+int finish_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index ac901bb02398..d3fc4988f869 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3033,6 +3033,38 @@ int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
return 0;
}
+
+/**
+ * finish_fault - finish page fault once we have prepared the page to fault
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a page fault once the
+ * page to fault in is prepared. It handles locking of PTEs, inserts PTE for
+ * given page, adds reverse page mapping, handles memcg charges and LRU
+ * addition. The function returns 0 on success, VM_FAULT_ code in case of
+ * error.
+ *
+ * The function expects the page to be locked and on success it consumes a
+ * reference of a page being mapped (for the PTE which maps it).
+ */
+int finish_fault(struct vm_fault *vmf)
+{
+ struct page *page;
+ int ret;
+
+ /* Did we COW the page? */
+ if ((vmf->flags & FAULT_FLAG_WRITE) &&
+ !(vmf->vma->vm_flags & VM_SHARED))
+ page = vmf->cow_page;
+ else
+ page = vmf->page;
+ ret = alloc_set_pte(vmf, vmf->memcg, page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return ret;
+}
+
static unsigned long fault_around_bytes __read_mostly =
rounddown_pow_of_two(65536);
@@ -3178,9 +3210,7 @@ static int do_read_fault(struct vm_fault *vmf)
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(vmf->page);
@@ -3219,9 +3249,7 @@ static int do_cow_fault(struct vm_fault *vmf)
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(vmf, memcg, new_page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(vmf->page);
put_page(vmf->page);
@@ -3262,9 +3290,7 @@ static int do_shared_fault(struct vm_fault *vmf)
}
}
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(vmf->page);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 09/21] mm: Factor out functionality to finish page faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Introduce function finish_fault() as a helper function for finishing
page faults. It is rather thin wrapper around alloc_set_pte() but since
we'd want to call this from DAX code or filesystems, it is still useful
to avoid some boilerplate code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 44 +++++++++++++++++++++++++++++++++++---------
2 files changed, 36 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 34d2891e9195..482455952f03 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -620,6 +620,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
+int finish_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index ac901bb02398..d3fc4988f869 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3033,6 +3033,38 @@ int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
return 0;
}
+
+/**
+ * finish_fault - finish page fault once we have prepared the page to fault
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a page fault once the
+ * page to fault in is prepared. It handles locking of PTEs, inserts PTE for
+ * given page, adds reverse page mapping, handles memcg charges and LRU
+ * addition. The function returns 0 on success, VM_FAULT_ code in case of
+ * error.
+ *
+ * The function expects the page to be locked and on success it consumes a
+ * reference of a page being mapped (for the PTE which maps it).
+ */
+int finish_fault(struct vm_fault *vmf)
+{
+ struct page *page;
+ int ret;
+
+ /* Did we COW the page? */
+ if ((vmf->flags & FAULT_FLAG_WRITE) &&
+ !(vmf->vma->vm_flags & VM_SHARED))
+ page = vmf->cow_page;
+ else
+ page = vmf->page;
+ ret = alloc_set_pte(vmf, vmf->memcg, page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return ret;
+}
+
static unsigned long fault_around_bytes __read_mostly =
rounddown_pow_of_two(65536);
@@ -3178,9 +3210,7 @@ static int do_read_fault(struct vm_fault *vmf)
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(vmf->page);
@@ -3219,9 +3249,7 @@ static int do_cow_fault(struct vm_fault *vmf)
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(vmf, memcg, new_page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(vmf->page);
put_page(vmf->page);
@@ -3262,9 +3290,7 @@ static int do_shared_fault(struct vm_fault *vmf)
}
}
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 09/21] mm: Factor out functionality to finish page faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Introduce function finish_fault() as a helper function for finishing
page faults. It is rather thin wrapper around alloc_set_pte() but since
we'd want to call this from DAX code or filesystems, it is still useful
to avoid some boilerplate code.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 44 +++++++++++++++++++++++++++++++++++---------
2 files changed, 36 insertions(+), 9 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 34d2891e9195..482455952f03 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -620,6 +620,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
+int finish_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index ac901bb02398..d3fc4988f869 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3033,6 +3033,38 @@ int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
return 0;
}
+
+/**
+ * finish_fault - finish page fault once we have prepared the page to fault
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a page fault once the
+ * page to fault in is prepared. It handles locking of PTEs, inserts PTE for
+ * given page, adds reverse page mapping, handles memcg charges and LRU
+ * addition. The function returns 0 on success, VM_FAULT_ code in case of
+ * error.
+ *
+ * The function expects the page to be locked and on success it consumes a
+ * reference of a page being mapped (for the PTE which maps it).
+ */
+int finish_fault(struct vm_fault *vmf)
+{
+ struct page *page;
+ int ret;
+
+ /* Did we COW the page? */
+ if ((vmf->flags & FAULT_FLAG_WRITE) &&
+ !(vmf->vma->vm_flags & VM_SHARED))
+ page = vmf->cow_page;
+ else
+ page = vmf->page;
+ ret = alloc_set_pte(vmf, vmf->memcg, page);
+ if (vmf->pte)
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return ret;
+}
+
static unsigned long fault_around_bytes __read_mostly =
rounddown_pow_of_two(65536);
@@ -3178,9 +3210,7 @@ static int do_read_fault(struct vm_fault *vmf)
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
return ret;
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
unlock_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
put_page(vmf->page);
@@ -3219,9 +3249,7 @@ static int do_cow_fault(struct vm_fault *vmf)
copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
- ret |= alloc_set_pte(vmf, memcg, new_page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (!(ret & VM_FAULT_DAX_LOCKED)) {
unlock_page(vmf->page);
put_page(vmf->page);
@@ -3262,9 +3290,7 @@ static int do_shared_fault(struct vm_fault *vmf)
}
}
- ret |= alloc_set_pte(vmf, NULL, vmf->page);
- if (vmf->pte)
- pte_unmap_unlock(vmf->pte, vmf->ptl);
+ ret |= finish_fault(vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
VM_FAULT_RETRY))) {
unlock_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 10/21] mm: Move handling of COW faults into DAX code
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Move final handling of COW faults from generic code into DAX fault
handler. That way generic code doesn't have to be aware of peculiarities
of DAX locking so remove that knowledge and make locking functions
private to fs/dax.c.
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 58 +++++++++++++++++++++++++++--------------------------
include/linux/dax.h | 7 -------
include/linux/mm.h | 9 +--------
mm/memory.c | 14 ++++---------
4 files changed, 35 insertions(+), 53 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 716a0f9c769c..acba769152bf 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -240,6 +240,23 @@ static void *get_unlocked_mapping_entry(struct address_space *mapping,
}
}
+static void dax_unlock_mapping_entry(struct address_space *mapping,
+ pgoff_t index)
+{
+ void *entry, **slot;
+
+ spin_lock_irq(&mapping->tree_lock);
+ entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
+ if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
+ !slot_locked(mapping, slot))) {
+ spin_unlock_irq(&mapping->tree_lock);
+ return;
+ }
+ unlock_slot(mapping, slot);
+ spin_unlock_irq(&mapping->tree_lock);
+ dax_wake_mapping_entry_waiter(mapping, index, entry, false);
+}
+
static void put_locked_mapping_entry(struct address_space *mapping,
pgoff_t index, void *entry)
{
@@ -434,22 +451,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
__wake_up(wq, TASK_NORMAL, wake_all ? 0 : 1, &key);
}
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index)
-{
- void *entry, **slot;
-
- spin_lock_irq(&mapping->tree_lock);
- entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
- if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
- !slot_locked(mapping, slot))) {
- spin_unlock_irq(&mapping->tree_lock);
- return;
- }
- unlock_slot(mapping, slot);
- spin_unlock_irq(&mapping->tree_lock);
- dax_wake_mapping_entry_waiter(mapping, index, entry, false);
-}
-
/*
* Delete exceptional DAX entry at @index from @mapping. Wait for radix tree
* entry to get unlocked before deleting it.
@@ -953,7 +954,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
struct iomap iomap = { 0 };
unsigned flags = IOMAP_FAULT;
int error, major = 0;
- int locked_status = 0;
+ int vmf_ret = 0;
void *entry;
/*
@@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
if (error)
goto finish_iomap;
- if (!radix_tree_exceptional_entry(entry)) {
+
+ __SetPageUptodate(vmf->cow_page);
+ if (!radix_tree_exceptional_entry(entry))
vmf->page = entry;
- locked_status = VM_FAULT_LOCKED;
- } else {
- vmf->entry = entry;
- locked_status = VM_FAULT_DAX_LOCKED;
- }
+ vmf_ret = finish_fault(vmf);
+ if (!vmf_ret)
+ vmf_ret = VM_FAULT_DONE_COW;
+ vmf->page = NULL;
goto finish_iomap;
}
@@ -1029,7 +1031,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
case IOMAP_UNWRITTEN:
case IOMAP_HOLE:
if (!(vmf->flags & FAULT_FLAG_WRITE)) {
- locked_status = dax_load_hole(mapping, entry, vmf);
+ vmf_ret = dax_load_hole(mapping, entry, vmf);
break;
}
/*FALLTHRU*/
@@ -1041,7 +1043,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
finish_iomap:
if (ops->iomap_end) {
- if (error) {
+ if (error || (vmf_ret & VM_FAULT_ERROR)) {
/* keep previous error */
ops->iomap_end(inode, pos, PAGE_SIZE, 0, flags,
&iomap);
@@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
}
}
unlock_entry:
- if (!locked_status || error)
+ if (vmf_ret != VM_FAULT_LOCKED || error)
put_locked_mapping_entry(mapping, vmf->pgoff, entry);
out:
if (error == -ENOMEM)
@@ -1059,9 +1061,9 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
/* -EBUSY is fine, somebody else faulted on the same PTE */
if (error < 0 && error != -EBUSY)
return VM_FAULT_SIGBUS | major;
- if (locked_status) {
+ if (vmf_ret) {
WARN_ON_ONCE(error); /* -EBUSY from ops->iomap_end? */
- return locked_status;
+ return vmf_ret;
}
return VM_FAULT_NOPAGE | major;
}
diff --git a/include/linux/dax.h b/include/linux/dax.h
index 0afade8bd3d7..f97bcfe79472 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -46,7 +46,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
#ifdef CONFIG_FS_DAX
struct page *read_dax_sector(struct block_device *bdev, sector_t n);
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index);
int __dax_zero_page_range(struct block_device *bdev, sector_t sector,
unsigned int offset, unsigned int length);
#else
@@ -55,12 +54,6 @@ static inline struct page *read_dax_sector(struct block_device *bdev,
{
return ERR_PTR(-ENXIO);
}
-/* Shouldn't ever be called when dax is disabled. */
-static inline void dax_unlock_mapping_entry(struct address_space *mapping,
- pgoff_t index)
-{
- BUG();
-}
static inline int __dax_zero_page_range(struct block_device *bdev,
sector_t sector, unsigned int offset, unsigned int length)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 482455952f03..fb128beecdac 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -308,12 +308,6 @@ struct vm_fault {
* is set (which is also implied by
* VM_FAULT_ERROR).
*/
- void *entry; /* ->fault handler can alternatively
- * return locked DAX entry. In that
- * case handler should return
- * VM_FAULT_DAX_LOCKED and fill in
- * entry here.
- */
/* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
@@ -1104,8 +1098,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_LOCKED 0x0200 /* ->fault locked the returned page */
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
-#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
-#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
+#define VM_FAULT_DONE_COW 0x1000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index d3fc4988f869..7be96a43d5ac 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2849,7 +2849,7 @@ static int __do_fault(struct vm_fault *vmf)
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
- VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
+ VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3245,17 +3245,11 @@ static int do_cow_fault(struct vm_fault *vmf)
if (ret & VM_FAULT_DONE_COW)
return ret;
- if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, vmf->page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
-
ret |= finish_fault(vmf);
- if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(vmf->page);
- put_page(vmf->page);
- } else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
- }
+ unlock_page(vmf->page);
+ put_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
return ret;
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 10/21] mm: Move handling of COW faults into DAX code
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Move final handling of COW faults from generic code into DAX fault
handler. That way generic code doesn't have to be aware of peculiarities
of DAX locking so remove that knowledge and make locking functions
private to fs/dax.c.
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 58 +++++++++++++++++++++++++++--------------------------
include/linux/dax.h | 7 -------
include/linux/mm.h | 9 +--------
mm/memory.c | 14 ++++---------
4 files changed, 35 insertions(+), 53 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 716a0f9c769c..acba769152bf 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -240,6 +240,23 @@ static void *get_unlocked_mapping_entry(struct address_space *mapping,
}
}
+static void dax_unlock_mapping_entry(struct address_space *mapping,
+ pgoff_t index)
+{
+ void *entry, **slot;
+
+ spin_lock_irq(&mapping->tree_lock);
+ entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
+ if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
+ !slot_locked(mapping, slot))) {
+ spin_unlock_irq(&mapping->tree_lock);
+ return;
+ }
+ unlock_slot(mapping, slot);
+ spin_unlock_irq(&mapping->tree_lock);
+ dax_wake_mapping_entry_waiter(mapping, index, entry, false);
+}
+
static void put_locked_mapping_entry(struct address_space *mapping,
pgoff_t index, void *entry)
{
@@ -434,22 +451,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
__wake_up(wq, TASK_NORMAL, wake_all ? 0 : 1, &key);
}
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index)
-{
- void *entry, **slot;
-
- spin_lock_irq(&mapping->tree_lock);
- entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
- if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
- !slot_locked(mapping, slot))) {
- spin_unlock_irq(&mapping->tree_lock);
- return;
- }
- unlock_slot(mapping, slot);
- spin_unlock_irq(&mapping->tree_lock);
- dax_wake_mapping_entry_waiter(mapping, index, entry, false);
-}
-
/*
* Delete exceptional DAX entry at @index from @mapping. Wait for radix tree
* entry to get unlocked before deleting it.
@@ -953,7 +954,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
struct iomap iomap = { 0 };
unsigned flags = IOMAP_FAULT;
int error, major = 0;
- int locked_status = 0;
+ int vmf_ret = 0;
void *entry;
/*
@@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
if (error)
goto finish_iomap;
- if (!radix_tree_exceptional_entry(entry)) {
+
+ __SetPageUptodate(vmf->cow_page);
+ if (!radix_tree_exceptional_entry(entry))
vmf->page = entry;
- locked_status = VM_FAULT_LOCKED;
- } else {
- vmf->entry = entry;
- locked_status = VM_FAULT_DAX_LOCKED;
- }
+ vmf_ret = finish_fault(vmf);
+ if (!vmf_ret)
+ vmf_ret = VM_FAULT_DONE_COW;
+ vmf->page = NULL;
goto finish_iomap;
}
@@ -1029,7 +1031,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
case IOMAP_UNWRITTEN:
case IOMAP_HOLE:
if (!(vmf->flags & FAULT_FLAG_WRITE)) {
- locked_status = dax_load_hole(mapping, entry, vmf);
+ vmf_ret = dax_load_hole(mapping, entry, vmf);
break;
}
/*FALLTHRU*/
@@ -1041,7 +1043,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
finish_iomap:
if (ops->iomap_end) {
- if (error) {
+ if (error || (vmf_ret & VM_FAULT_ERROR)) {
/* keep previous error */
ops->iomap_end(inode, pos, PAGE_SIZE, 0, flags,
&iomap);
@@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
}
}
unlock_entry:
- if (!locked_status || error)
+ if (vmf_ret != VM_FAULT_LOCKED || error)
put_locked_mapping_entry(mapping, vmf->pgoff, entry);
out:
if (error == -ENOMEM)
@@ -1059,9 +1061,9 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
/* -EBUSY is fine, somebody else faulted on the same PTE */
if (error < 0 && error != -EBUSY)
return VM_FAULT_SIGBUS | major;
- if (locked_status) {
+ if (vmf_ret) {
WARN_ON_ONCE(error); /* -EBUSY from ops->iomap_end? */
- return locked_status;
+ return vmf_ret;
}
return VM_FAULT_NOPAGE | major;
}
diff --git a/include/linux/dax.h b/include/linux/dax.h
index 0afade8bd3d7..f97bcfe79472 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -46,7 +46,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
#ifdef CONFIG_FS_DAX
struct page *read_dax_sector(struct block_device *bdev, sector_t n);
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index);
int __dax_zero_page_range(struct block_device *bdev, sector_t sector,
unsigned int offset, unsigned int length);
#else
@@ -55,12 +54,6 @@ static inline struct page *read_dax_sector(struct block_device *bdev,
{
return ERR_PTR(-ENXIO);
}
-/* Shouldn't ever be called when dax is disabled. */
-static inline void dax_unlock_mapping_entry(struct address_space *mapping,
- pgoff_t index)
-{
- BUG();
-}
static inline int __dax_zero_page_range(struct block_device *bdev,
sector_t sector, unsigned int offset, unsigned int length)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 482455952f03..fb128beecdac 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -308,12 +308,6 @@ struct vm_fault {
* is set (which is also implied by
* VM_FAULT_ERROR).
*/
- void *entry; /* ->fault handler can alternatively
- * return locked DAX entry. In that
- * case handler should return
- * VM_FAULT_DAX_LOCKED and fill in
- * entry here.
- */
/* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
@@ -1104,8 +1098,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_LOCKED 0x0200 /* ->fault locked the returned page */
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
-#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
-#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
+#define VM_FAULT_DONE_COW 0x1000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index d3fc4988f869..7be96a43d5ac 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2849,7 +2849,7 @@ static int __do_fault(struct vm_fault *vmf)
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
- VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
+ VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3245,17 +3245,11 @@ static int do_cow_fault(struct vm_fault *vmf)
if (ret & VM_FAULT_DONE_COW)
return ret;
- if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, vmf->page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
-
ret |= finish_fault(vmf);
- if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(vmf->page);
- put_page(vmf->page);
- } else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
- }
+ unlock_page(vmf->page);
+ put_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
return ret;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 10/21] mm: Move handling of COW faults into DAX code
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Move final handling of COW faults from generic code into DAX fault
handler. That way generic code doesn't have to be aware of peculiarities
of DAX locking so remove that knowledge and make locking functions
private to fs/dax.c.
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 58 +++++++++++++++++++++++++++--------------------------
include/linux/dax.h | 7 -------
include/linux/mm.h | 9 +--------
mm/memory.c | 14 ++++---------
4 files changed, 35 insertions(+), 53 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 716a0f9c769c..acba769152bf 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -240,6 +240,23 @@ static void *get_unlocked_mapping_entry(struct address_space *mapping,
}
}
+static void dax_unlock_mapping_entry(struct address_space *mapping,
+ pgoff_t index)
+{
+ void *entry, **slot;
+
+ spin_lock_irq(&mapping->tree_lock);
+ entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
+ if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
+ !slot_locked(mapping, slot))) {
+ spin_unlock_irq(&mapping->tree_lock);
+ return;
+ }
+ unlock_slot(mapping, slot);
+ spin_unlock_irq(&mapping->tree_lock);
+ dax_wake_mapping_entry_waiter(mapping, index, entry, false);
+}
+
static void put_locked_mapping_entry(struct address_space *mapping,
pgoff_t index, void *entry)
{
@@ -434,22 +451,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
__wake_up(wq, TASK_NORMAL, wake_all ? 0 : 1, &key);
}
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index)
-{
- void *entry, **slot;
-
- spin_lock_irq(&mapping->tree_lock);
- entry = __radix_tree_lookup(&mapping->page_tree, index, NULL, &slot);
- if (WARN_ON_ONCE(!entry || !radix_tree_exceptional_entry(entry) ||
- !slot_locked(mapping, slot))) {
- spin_unlock_irq(&mapping->tree_lock);
- return;
- }
- unlock_slot(mapping, slot);
- spin_unlock_irq(&mapping->tree_lock);
- dax_wake_mapping_entry_waiter(mapping, index, entry, false);
-}
-
/*
* Delete exceptional DAX entry at @index from @mapping. Wait for radix tree
* entry to get unlocked before deleting it.
@@ -953,7 +954,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
struct iomap iomap = { 0 };
unsigned flags = IOMAP_FAULT;
int error, major = 0;
- int locked_status = 0;
+ int vmf_ret = 0;
void *entry;
/*
@@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
if (error)
goto finish_iomap;
- if (!radix_tree_exceptional_entry(entry)) {
+
+ __SetPageUptodate(vmf->cow_page);
+ if (!radix_tree_exceptional_entry(entry))
vmf->page = entry;
- locked_status = VM_FAULT_LOCKED;
- } else {
- vmf->entry = entry;
- locked_status = VM_FAULT_DAX_LOCKED;
- }
+ vmf_ret = finish_fault(vmf);
+ if (!vmf_ret)
+ vmf_ret = VM_FAULT_DONE_COW;
+ vmf->page = NULL;
goto finish_iomap;
}
@@ -1029,7 +1031,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
case IOMAP_UNWRITTEN:
case IOMAP_HOLE:
if (!(vmf->flags & FAULT_FLAG_WRITE)) {
- locked_status = dax_load_hole(mapping, entry, vmf);
+ vmf_ret = dax_load_hole(mapping, entry, vmf);
break;
}
/*FALLTHRU*/
@@ -1041,7 +1043,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
finish_iomap:
if (ops->iomap_end) {
- if (error) {
+ if (error || (vmf_ret & VM_FAULT_ERROR)) {
/* keep previous error */
ops->iomap_end(inode, pos, PAGE_SIZE, 0, flags,
&iomap);
@@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
}
}
unlock_entry:
- if (!locked_status || error)
+ if (vmf_ret != VM_FAULT_LOCKED || error)
put_locked_mapping_entry(mapping, vmf->pgoff, entry);
out:
if (error == -ENOMEM)
@@ -1059,9 +1061,9 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
/* -EBUSY is fine, somebody else faulted on the same PTE */
if (error < 0 && error != -EBUSY)
return VM_FAULT_SIGBUS | major;
- if (locked_status) {
+ if (vmf_ret) {
WARN_ON_ONCE(error); /* -EBUSY from ops->iomap_end? */
- return locked_status;
+ return vmf_ret;
}
return VM_FAULT_NOPAGE | major;
}
diff --git a/include/linux/dax.h b/include/linux/dax.h
index 0afade8bd3d7..f97bcfe79472 100644
--- a/include/linux/dax.h
+++ b/include/linux/dax.h
@@ -46,7 +46,6 @@ void dax_wake_mapping_entry_waiter(struct address_space *mapping,
#ifdef CONFIG_FS_DAX
struct page *read_dax_sector(struct block_device *bdev, sector_t n);
-void dax_unlock_mapping_entry(struct address_space *mapping, pgoff_t index);
int __dax_zero_page_range(struct block_device *bdev, sector_t sector,
unsigned int offset, unsigned int length);
#else
@@ -55,12 +54,6 @@ static inline struct page *read_dax_sector(struct block_device *bdev,
{
return ERR_PTR(-ENXIO);
}
-/* Shouldn't ever be called when dax is disabled. */
-static inline void dax_unlock_mapping_entry(struct address_space *mapping,
- pgoff_t index)
-{
- BUG();
-}
static inline int __dax_zero_page_range(struct block_device *bdev,
sector_t sector, unsigned int offset, unsigned int length)
{
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 482455952f03..fb128beecdac 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -308,12 +308,6 @@ struct vm_fault {
* is set (which is also implied by
* VM_FAULT_ERROR).
*/
- void *entry; /* ->fault handler can alternatively
- * return locked DAX entry. In that
- * case handler should return
- * VM_FAULT_DAX_LOCKED and fill in
- * entry here.
- */
/* These three entries are valid only while holding ptl lock */
pte_t *pte; /* Pointer to pte entry matching
* the 'address'. NULL if the page
@@ -1104,8 +1098,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_LOCKED 0x0200 /* ->fault locked the returned page */
#define VM_FAULT_RETRY 0x0400 /* ->fault blocked, must retry */
#define VM_FAULT_FALLBACK 0x0800 /* huge page fault failed, fall back to small */
-#define VM_FAULT_DAX_LOCKED 0x1000 /* ->fault has locked DAX entry */
-#define VM_FAULT_DONE_COW 0x2000 /* ->fault has fully handled COW */
+#define VM_FAULT_DONE_COW 0x1000 /* ->fault has fully handled COW */
#define VM_FAULT_HWPOISON_LARGE_MASK 0xf000 /* encodes hpage index for large hwpoison */
diff --git a/mm/memory.c b/mm/memory.c
index d3fc4988f869..7be96a43d5ac 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2849,7 +2849,7 @@ static int __do_fault(struct vm_fault *vmf)
ret = vma->vm_ops->fault(vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY |
- VM_FAULT_DAX_LOCKED | VM_FAULT_DONE_COW)))
+ VM_FAULT_DONE_COW)))
return ret;
if (unlikely(PageHWPoison(vmf->page))) {
@@ -3245,17 +3245,11 @@ static int do_cow_fault(struct vm_fault *vmf)
if (ret & VM_FAULT_DONE_COW)
return ret;
- if (!(ret & VM_FAULT_DAX_LOCKED))
- copy_user_highpage(new_page, vmf->page, vmf->address, vma);
+ copy_user_highpage(new_page, vmf->page, vmf->address, vma);
__SetPageUptodate(new_page);
-
ret |= finish_fault(vmf);
- if (!(ret & VM_FAULT_DAX_LOCKED)) {
- unlock_page(vmf->page);
- put_page(vmf->page);
- } else {
- dax_unlock_mapping_entry(vma->vm_file->f_mapping, vmf->pgoff);
- }
+ unlock_page(vmf->page);
+ put_page(vmf->page);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
goto uncharge_out;
return ret;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
We don't check whether vma->vm_ops is NULL in do_shared_fault() so
there's hardly any point in checking it in wp_page_shared() or
wp_pfn_shared() which get called only for shared file mappings as well.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 7be96a43d5ac..26b2858e6a12 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2275,7 +2275,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
+ if (vma->vm_ops->pfn_mkwrite) {
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
@@ -2305,7 +2305,7 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
get_page(old_page);
- if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
+ if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
We don't check whether vma->vm_ops is NULL in do_shared_fault() so
there's hardly any point in checking it in wp_page_shared() or
wp_pfn_shared() which get called only for shared file mappings as well.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 7be96a43d5ac..26b2858e6a12 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2275,7 +2275,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
+ if (vma->vm_ops->pfn_mkwrite) {
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
@@ -2305,7 +2305,7 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
get_page(old_page);
- if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
+ if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
We don't check whether vma->vm_ops is NULL in do_shared_fault() so
there's hardly any point in checking it in wp_page_shared() or
wp_pfn_shared() which get called only for shared file mappings as well.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 7be96a43d5ac..26b2858e6a12 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2275,7 +2275,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
+ if (vma->vm_ops->pfn_mkwrite) {
int ret;
pte_unmap_unlock(vmf->pte, vmf->ptl);
@@ -2305,7 +2305,7 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
get_page(old_page);
- if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
+ if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 12/21] mm: Factor out common parts of write fault handling
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Currently we duplicate handling of shared write faults in
wp_page_reuse() and do_shared_fault(). Factor them out into a common
function.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 78 +++++++++++++++++++++++++++++--------------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 26b2858e6a12..4da66c984c2c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2067,6 +2067,41 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
}
/*
+ * Handle dirtying of a page in shared file mapping on a write fault.
+ *
+ * The function expects the page to be locked and unlocks it.
+ */
+static void fault_dirty_shared_page(struct vm_area_struct *vma,
+ struct page *page)
+{
+ struct address_space *mapping;
+ bool dirtied;
+ bool page_mkwrite = vma->vm_ops->page_mkwrite;
+
+ dirtied = set_page_dirty(page);
+ VM_BUG_ON_PAGE(PageAnon(page), page);
+ /*
+ * Take a local copy of the address_space - page.mapping may be zeroed
+ * by truncate after unlock_page(). The address_space itself remains
+ * pinned by vma->vm_file's reference. We rely on unlock_page()'s
+ * release semantics to prevent the compiler from undoing this copying.
+ */
+ mapping = page_rmapping(page);
+ unlock_page(page);
+
+ if ((dirtied || page_mkwrite) && mapping) {
+ /*
+ * Some device drivers do not set page.mapping
+ * but still dirty their pages
+ */
+ balance_dirty_pages_ratelimited(mapping);
+ }
+
+ if (!page_mkwrite)
+ file_update_time(vma->vm_file);
+}
+
+/*
* Handle write page faults for pages that can be reused in the current vma
*
* This can happen either due to the mapping being with the VM_SHARED flag,
@@ -2096,28 +2131,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
- struct address_space *mapping;
- int dirtied;
-
if (!page_mkwrite)
lock_page(page);
- dirtied = set_page_dirty(page);
- VM_BUG_ON_PAGE(PageAnon(page), page);
- mapping = page->mapping;
- unlock_page(page);
+ fault_dirty_shared_page(vma, page);
put_page(page);
-
- if ((dirtied || page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping
- * but still dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!page_mkwrite)
- file_update_time(vma->vm_file);
}
return VM_FAULT_WRITE;
@@ -3262,8 +3280,6 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct address_space *mapping;
- int dirtied = 0;
int ret, tmp;
ret = __do_fault(vmf);
@@ -3292,27 +3308,7 @@ static int do_shared_fault(struct vm_fault *vmf)
return ret;
}
- if (set_page_dirty(vmf->page))
- dirtied = 1;
- /*
- * Take a local copy of the address_space - page.mapping may be zeroed
- * by truncate after unlock_page(). The address_space itself remains
- * pinned by vma->vm_file's reference. We rely on unlock_page()'s
- * release semantics to prevent the compiler from undoing this copying.
- */
- mapping = page_rmapping(vmf->page);
- unlock_page(vmf->page);
- if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping but still
- * dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!vma->vm_ops->page_mkwrite)
- file_update_time(vma->vm_file);
-
+ fault_dirty_shared_page(vma, vmf->page);
return ret;
}
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 12/21] mm: Factor out common parts of write fault handling
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we duplicate handling of shared write faults in
wp_page_reuse() and do_shared_fault(). Factor them out into a common
function.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 78 +++++++++++++++++++++++++++++--------------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 26b2858e6a12..4da66c984c2c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2067,6 +2067,41 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
}
/*
+ * Handle dirtying of a page in shared file mapping on a write fault.
+ *
+ * The function expects the page to be locked and unlocks it.
+ */
+static void fault_dirty_shared_page(struct vm_area_struct *vma,
+ struct page *page)
+{
+ struct address_space *mapping;
+ bool dirtied;
+ bool page_mkwrite = vma->vm_ops->page_mkwrite;
+
+ dirtied = set_page_dirty(page);
+ VM_BUG_ON_PAGE(PageAnon(page), page);
+ /*
+ * Take a local copy of the address_space - page.mapping may be zeroed
+ * by truncate after unlock_page(). The address_space itself remains
+ * pinned by vma->vm_file's reference. We rely on unlock_page()'s
+ * release semantics to prevent the compiler from undoing this copying.
+ */
+ mapping = page_rmapping(page);
+ unlock_page(page);
+
+ if ((dirtied || page_mkwrite) && mapping) {
+ /*
+ * Some device drivers do not set page.mapping
+ * but still dirty their pages
+ */
+ balance_dirty_pages_ratelimited(mapping);
+ }
+
+ if (!page_mkwrite)
+ file_update_time(vma->vm_file);
+}
+
+/*
* Handle write page faults for pages that can be reused in the current vma
*
* This can happen either due to the mapping being with the VM_SHARED flag,
@@ -2096,28 +2131,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
- struct address_space *mapping;
- int dirtied;
-
if (!page_mkwrite)
lock_page(page);
- dirtied = set_page_dirty(page);
- VM_BUG_ON_PAGE(PageAnon(page), page);
- mapping = page->mapping;
- unlock_page(page);
+ fault_dirty_shared_page(vma, page);
put_page(page);
-
- if ((dirtied || page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping
- * but still dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!page_mkwrite)
- file_update_time(vma->vm_file);
}
return VM_FAULT_WRITE;
@@ -3262,8 +3280,6 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct address_space *mapping;
- int dirtied = 0;
int ret, tmp;
ret = __do_fault(vmf);
@@ -3292,27 +3308,7 @@ static int do_shared_fault(struct vm_fault *vmf)
return ret;
}
- if (set_page_dirty(vmf->page))
- dirtied = 1;
- /*
- * Take a local copy of the address_space - page.mapping may be zeroed
- * by truncate after unlock_page(). The address_space itself remains
- * pinned by vma->vm_file's reference. We rely on unlock_page()'s
- * release semantics to prevent the compiler from undoing this copying.
- */
- mapping = page_rmapping(vmf->page);
- unlock_page(vmf->page);
- if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping but still
- * dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!vma->vm_ops->page_mkwrite)
- file_update_time(vma->vm_file);
-
+ fault_dirty_shared_page(vma, vmf->page);
return ret;
}
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 12/21] mm: Factor out common parts of write fault handling
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we duplicate handling of shared write faults in
wp_page_reuse() and do_shared_fault(). Factor them out into a common
function.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 78 +++++++++++++++++++++++++++++--------------------------------
1 file changed, 37 insertions(+), 41 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 26b2858e6a12..4da66c984c2c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2067,6 +2067,41 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
}
/*
+ * Handle dirtying of a page in shared file mapping on a write fault.
+ *
+ * The function expects the page to be locked and unlocks it.
+ */
+static void fault_dirty_shared_page(struct vm_area_struct *vma,
+ struct page *page)
+{
+ struct address_space *mapping;
+ bool dirtied;
+ bool page_mkwrite = vma->vm_ops->page_mkwrite;
+
+ dirtied = set_page_dirty(page);
+ VM_BUG_ON_PAGE(PageAnon(page), page);
+ /*
+ * Take a local copy of the address_space - page.mapping may be zeroed
+ * by truncate after unlock_page(). The address_space itself remains
+ * pinned by vma->vm_file's reference. We rely on unlock_page()'s
+ * release semantics to prevent the compiler from undoing this copying.
+ */
+ mapping = page_rmapping(page);
+ unlock_page(page);
+
+ if ((dirtied || page_mkwrite) && mapping) {
+ /*
+ * Some device drivers do not set page.mapping
+ * but still dirty their pages
+ */
+ balance_dirty_pages_ratelimited(mapping);
+ }
+
+ if (!page_mkwrite)
+ file_update_time(vma->vm_file);
+}
+
+/*
* Handle write page faults for pages that can be reused in the current vma
*
* This can happen either due to the mapping being with the VM_SHARED flag,
@@ -2096,28 +2131,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
pte_unmap_unlock(vmf->pte, vmf->ptl);
if (dirty_shared) {
- struct address_space *mapping;
- int dirtied;
-
if (!page_mkwrite)
lock_page(page);
- dirtied = set_page_dirty(page);
- VM_BUG_ON_PAGE(PageAnon(page), page);
- mapping = page->mapping;
- unlock_page(page);
+ fault_dirty_shared_page(vma, page);
put_page(page);
-
- if ((dirtied || page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping
- * but still dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!page_mkwrite)
- file_update_time(vma->vm_file);
}
return VM_FAULT_WRITE;
@@ -3262,8 +3280,6 @@ static int do_cow_fault(struct vm_fault *vmf)
static int do_shared_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
- struct address_space *mapping;
- int dirtied = 0;
int ret, tmp;
ret = __do_fault(vmf);
@@ -3292,27 +3308,7 @@ static int do_shared_fault(struct vm_fault *vmf)
return ret;
}
- if (set_page_dirty(vmf->page))
- dirtied = 1;
- /*
- * Take a local copy of the address_space - page.mapping may be zeroed
- * by truncate after unlock_page(). The address_space itself remains
- * pinned by vma->vm_file's reference. We rely on unlock_page()'s
- * release semantics to prevent the compiler from undoing this copying.
- */
- mapping = page_rmapping(vmf->page);
- unlock_page(vmf->page);
- if ((dirtied || vma->vm_ops->page_mkwrite) && mapping) {
- /*
- * Some device drivers do not set page.mapping but still
- * dirty their pages
- */
- balance_dirty_pages_ratelimited(mapping);
- }
-
- if (!vma->vm_ops->page_mkwrite)
- file_update_time(vma->vm_file);
-
+ fault_dirty_shared_page(vma, vmf->page);
return ret;
}
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
We will need more information in the ->page_mkwrite() helper for DAX to
be able to fully finish faults there. Pass vm_fault structure to
do_page_mkwrite() and use it there so that information propagates
properly from upper layers.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 4da66c984c2c..c89f99c270bc 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
*
* We do this without the lock held, so that it can sleep if it needs to.
*/
-static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
- unsigned long address)
+static int do_page_mkwrite(struct vm_fault *vmf)
{
- struct vm_fault vmf;
int ret;
+ struct page *page = vmf->page;
- vmf.address = address;
- vmf.pgoff = page->index;
- vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.page = page;
- vmf.cow_page = NULL;
+ vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- ret = vma->vm_ops->page_mkwrite(vma, &vmf);
+ ret = vmf->vma->vm_ops->page_mkwrite(vmf->vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))
return ret;
if (unlikely(!(ret & VM_FAULT_LOCKED))) {
@@ -2327,7 +2321,8 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- tmp = do_page_mkwrite(vma, old_page, vmf->address);
+ vmf->page = old_page;
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -3292,7 +3287,7 @@ static int do_shared_fault(struct vm_fault *vmf)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(vmf->page);
- tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(vmf->page);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
We will need more information in the ->page_mkwrite() helper for DAX to
be able to fully finish faults there. Pass vm_fault structure to
do_page_mkwrite() and use it there so that information propagates
properly from upper layers.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 4da66c984c2c..c89f99c270bc 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
*
* We do this without the lock held, so that it can sleep if it needs to.
*/
-static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
- unsigned long address)
+static int do_page_mkwrite(struct vm_fault *vmf)
{
- struct vm_fault vmf;
int ret;
+ struct page *page = vmf->page;
- vmf.address = address;
- vmf.pgoff = page->index;
- vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.page = page;
- vmf.cow_page = NULL;
+ vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- ret = vma->vm_ops->page_mkwrite(vma, &vmf);
+ ret = vmf->vma->vm_ops->page_mkwrite(vmf->vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))
return ret;
if (unlikely(!(ret & VM_FAULT_LOCKED))) {
@@ -2327,7 +2321,8 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- tmp = do_page_mkwrite(vma, old_page, vmf->address);
+ vmf->page = old_page;
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -3292,7 +3287,7 @@ static int do_shared_fault(struct vm_fault *vmf)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(vmf->page);
- tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
We will need more information in the ->page_mkwrite() helper for DAX to
be able to fully finish faults there. Pass vm_fault structure to
do_page_mkwrite() and use it there so that information propagates
properly from upper layers.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 4da66c984c2c..c89f99c270bc 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
*
* We do this without the lock held, so that it can sleep if it needs to.
*/
-static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
- unsigned long address)
+static int do_page_mkwrite(struct vm_fault *vmf)
{
- struct vm_fault vmf;
int ret;
+ struct page *page = vmf->page;
- vmf.address = address;
- vmf.pgoff = page->index;
- vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- vmf.gfp_mask = __get_fault_gfp_mask(vma);
- vmf.page = page;
- vmf.cow_page = NULL;
+ vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
- ret = vma->vm_ops->page_mkwrite(vma, &vmf);
+ ret = vmf->vma->vm_ops->page_mkwrite(vmf->vma, vmf);
if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))
return ret;
if (unlikely(!(ret & VM_FAULT_LOCKED))) {
@@ -2327,7 +2321,8 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- tmp = do_page_mkwrite(vma, old_page, vmf->address);
+ vmf->page = old_page;
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(old_page);
@@ -3292,7 +3287,7 @@ static int do_shared_fault(struct vm_fault *vmf)
*/
if (vma->vm_ops->page_mkwrite) {
unlock_page(vmf->page);
- tmp = do_page_mkwrite(vma, vmf->page, vmf->address);
+ tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp ||
(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
put_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 14/21] mm: Use vmf->page during WP faults
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
So far we set vmf->page during WP faults only when we needed to pass it
to the ->page_mkwrite handler. Set it in all the cases now and use that
instead of passing page pointer explicitly around.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 58 +++++++++++++++++++++++++++++-----------------------------
1 file changed, 29 insertions(+), 29 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index c89f99c270bc..e278a8a6ccc7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,11 +2103,12 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+static inline int wp_page_reuse(struct vm_fault *vmf,
int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
+ struct page *page = vmf->page;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2151,10 +2152,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
+ struct page *old_page = vmf->page;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
@@ -2306,26 +2308,25 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, NULL, 0, 0);
+ return wp_page_reuse(vmf, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
- get_page(old_page);
+ get_page(vmf->page);
if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- vmf->page = old_page;
tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(old_page);
+ put_page(vmf->page);
return tmp;
}
/*
@@ -2337,15 +2338,15 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, page_mkwrite, 1);
}
/*
@@ -2370,10 +2371,9 @@ static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
- if (!old_page) {
+ vmf->page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
+ if (!vmf->page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
* VM_PFNMAP VMA.
@@ -2386,30 +2386,30 @@ static int do_wp_page(struct vm_fault *vmf)
return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
/*
* Take out anonymous pages first, anonymous shared vmas are
* not dirty accountable.
*/
- if (PageAnon(old_page) && !PageKsm(old_page)) {
+ if (PageAnon(vmf->page) && !PageKsm(vmf->page)) {
int total_mapcount;
- if (!trylock_page(old_page)) {
- get_page(old_page);
+ if (!trylock_page(vmf->page)) {
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- lock_page(old_page);
+ lock_page(vmf->page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
- put_page(old_page);
+ put_page(vmf->page);
}
- if (reuse_swap_page(old_page, &total_mapcount)) {
+ if (reuse_swap_page(vmf->page, &total_mapcount)) {
if (total_mapcount == 1) {
/*
* The page is all ours. Move it to
@@ -2418,24 +2418,24 @@ static int do_wp_page(struct vm_fault *vmf)
* Protected against the rmap code by
* the page lock.
*/
- page_move_anon_rmap(old_page, vma);
+ page_move_anon_rmap(vmf->page, vma);
}
- unlock_page(old_page);
- return wp_page_reuse(vmf, old_page, 0, 0);
+ unlock_page(vmf->page);
+ return wp_page_reuse(vmf, 0, 0);
}
- unlock_page(old_page);
+ unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, old_page);
+ return wp_page_shared(vmf);
}
/*
* Ok, we need to copy. Oh, well..
*/
- get_page(old_page);
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 14/21] mm: Use vmf->page during WP faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
So far we set vmf->page during WP faults only when we needed to pass it
to the ->page_mkwrite handler. Set it in all the cases now and use that
instead of passing page pointer explicitly around.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 58 +++++++++++++++++++++++++++++-----------------------------
1 file changed, 29 insertions(+), 29 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index c89f99c270bc..e278a8a6ccc7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,11 +2103,12 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+static inline int wp_page_reuse(struct vm_fault *vmf,
int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
+ struct page *page = vmf->page;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2151,10 +2152,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
+ struct page *old_page = vmf->page;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
@@ -2306,26 +2308,25 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, NULL, 0, 0);
+ return wp_page_reuse(vmf, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
- get_page(old_page);
+ get_page(vmf->page);
if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- vmf->page = old_page;
tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(old_page);
+ put_page(vmf->page);
return tmp;
}
/*
@@ -2337,15 +2338,15 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, page_mkwrite, 1);
}
/*
@@ -2370,10 +2371,9 @@ static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
- if (!old_page) {
+ vmf->page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
+ if (!vmf->page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
* VM_PFNMAP VMA.
@@ -2386,30 +2386,30 @@ static int do_wp_page(struct vm_fault *vmf)
return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
/*
* Take out anonymous pages first, anonymous shared vmas are
* not dirty accountable.
*/
- if (PageAnon(old_page) && !PageKsm(old_page)) {
+ if (PageAnon(vmf->page) && !PageKsm(vmf->page)) {
int total_mapcount;
- if (!trylock_page(old_page)) {
- get_page(old_page);
+ if (!trylock_page(vmf->page)) {
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- lock_page(old_page);
+ lock_page(vmf->page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
- put_page(old_page);
+ put_page(vmf->page);
}
- if (reuse_swap_page(old_page, &total_mapcount)) {
+ if (reuse_swap_page(vmf->page, &total_mapcount)) {
if (total_mapcount == 1) {
/*
* The page is all ours. Move it to
@@ -2418,24 +2418,24 @@ static int do_wp_page(struct vm_fault *vmf)
* Protected against the rmap code by
* the page lock.
*/
- page_move_anon_rmap(old_page, vma);
+ page_move_anon_rmap(vmf->page, vma);
}
- unlock_page(old_page);
- return wp_page_reuse(vmf, old_page, 0, 0);
+ unlock_page(vmf->page);
+ return wp_page_reuse(vmf, 0, 0);
}
- unlock_page(old_page);
+ unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, old_page);
+ return wp_page_shared(vmf);
}
/*
* Ok, we need to copy. Oh, well..
*/
- get_page(old_page);
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 14/21] mm: Use vmf->page during WP faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
So far we set vmf->page during WP faults only when we needed to pass it
to the ->page_mkwrite handler. Set it in all the cases now and use that
instead of passing page pointer explicitly around.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 58 +++++++++++++++++++++++++++++-----------------------------
1 file changed, 29 insertions(+), 29 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index c89f99c270bc..e278a8a6ccc7 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,11 +2103,12 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+static inline int wp_page_reuse(struct vm_fault *vmf,
int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
+ struct page *page = vmf->page;
pte_t entry;
/*
* Clear the pages cpupid information as the existing
@@ -2151,10 +2152,11 @@ static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
+ struct page *old_page = vmf->page;
struct page *new_page = NULL;
pte_t entry;
int page_copied = 0;
@@ -2306,26 +2308,25 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, NULL, 0, 0);
+ return wp_page_reuse(vmf, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
int page_mkwrite = 0;
- get_page(old_page);
+ get_page(vmf->page);
if (vma->vm_ops->page_mkwrite) {
int tmp;
pte_unmap_unlock(vmf->pte, vmf->ptl);
- vmf->page = old_page;
tmp = do_page_mkwrite(vmf);
if (unlikely(!tmp || (tmp &
(VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
- put_page(old_page);
+ put_page(vmf->page);
return tmp;
}
/*
@@ -2337,15 +2338,15 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, page_mkwrite, 1);
}
/*
@@ -2370,10 +2371,9 @@ static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
- if (!old_page) {
+ vmf->page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
+ if (!vmf->page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
* VM_PFNMAP VMA.
@@ -2386,30 +2386,30 @@ static int do_wp_page(struct vm_fault *vmf)
return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
/*
* Take out anonymous pages first, anonymous shared vmas are
* not dirty accountable.
*/
- if (PageAnon(old_page) && !PageKsm(old_page)) {
+ if (PageAnon(vmf->page) && !PageKsm(vmf->page)) {
int total_mapcount;
- if (!trylock_page(old_page)) {
- get_page(old_page);
+ if (!trylock_page(vmf->page)) {
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- lock_page(old_page);
+ lock_page(vmf->page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- unlock_page(old_page);
+ unlock_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- put_page(old_page);
+ put_page(vmf->page);
return 0;
}
- put_page(old_page);
+ put_page(vmf->page);
}
- if (reuse_swap_page(old_page, &total_mapcount)) {
+ if (reuse_swap_page(vmf->page, &total_mapcount)) {
if (total_mapcount == 1) {
/*
* The page is all ours. Move it to
@@ -2418,24 +2418,24 @@ static int do_wp_page(struct vm_fault *vmf)
* Protected against the rmap code by
* the page lock.
*/
- page_move_anon_rmap(old_page, vma);
+ page_move_anon_rmap(vmf->page, vma);
}
- unlock_page(old_page);
- return wp_page_reuse(vmf, old_page, 0, 0);
+ unlock_page(vmf->page);
+ return wp_page_reuse(vmf, 0, 0);
}
- unlock_page(old_page);
+ unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, old_page);
+ return wp_page_shared(vmf);
}
/*
* Ok, we need to copy. Oh, well..
*/
- get_page(old_page);
+ get_page(vmf->page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, old_page);
+ return wp_page_copy(vmf);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
wp_page_reuse() handles write shared faults which is needed only in
wp_page_shared(). Move the handling only into that location to make
wp_page_reuse() simpler and avoid a strange situation when we sometimes
pass in locked page, sometimes unlocked etc.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 27 ++++++++++++---------------
1 file changed, 12 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index e278a8a6ccc7..06aba4203104 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,8 +2103,7 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf,
- int page_mkwrite, int dirty_shared)
+static inline void wp_page_reuse(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2124,16 +2123,6 @@ static inline int wp_page_reuse(struct vm_fault *vmf,
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
pte_unmap_unlock(vmf->pte, vmf->ptl);
-
- if (dirty_shared) {
- if (!page_mkwrite)
- lock_page(page);
-
- fault_dirty_shared_page(vma, page);
- put_page(page);
- }
-
- return VM_FAULT_WRITE;
}
/*
@@ -2308,7 +2297,8 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
static int wp_page_shared(struct vm_fault *vmf)
@@ -2346,7 +2336,13 @@ static int wp_page_shared(struct vm_fault *vmf)
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, page_mkwrite, 1);
+ wp_page_reuse(vmf);
+ if (!page_mkwrite)
+ lock_page(vmf->page);
+ fault_dirty_shared_page(vma, vmf->page);
+ put_page(vmf->page);
+
+ return VM_FAULT_WRITE;
}
/*
@@ -2421,7 +2417,8 @@ static int do_wp_page(struct vm_fault *vmf)
page_move_anon_rmap(vmf->page, vma);
}
unlock_page(vmf->page);
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
wp_page_reuse() handles write shared faults which is needed only in
wp_page_shared(). Move the handling only into that location to make
wp_page_reuse() simpler and avoid a strange situation when we sometimes
pass in locked page, sometimes unlocked etc.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 27 ++++++++++++---------------
1 file changed, 12 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index e278a8a6ccc7..06aba4203104 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,8 +2103,7 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf,
- int page_mkwrite, int dirty_shared)
+static inline void wp_page_reuse(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2124,16 +2123,6 @@ static inline int wp_page_reuse(struct vm_fault *vmf,
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
pte_unmap_unlock(vmf->pte, vmf->ptl);
-
- if (dirty_shared) {
- if (!page_mkwrite)
- lock_page(page);
-
- fault_dirty_shared_page(vma, page);
- put_page(page);
- }
-
- return VM_FAULT_WRITE;
}
/*
@@ -2308,7 +2297,8 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
static int wp_page_shared(struct vm_fault *vmf)
@@ -2346,7 +2336,13 @@ static int wp_page_shared(struct vm_fault *vmf)
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, page_mkwrite, 1);
+ wp_page_reuse(vmf);
+ if (!page_mkwrite)
+ lock_page(vmf->page);
+ fault_dirty_shared_page(vma, vmf->page);
+ put_page(vmf->page);
+
+ return VM_FAULT_WRITE;
}
/*
@@ -2421,7 +2417,8 @@ static int do_wp_page(struct vm_fault *vmf)
page_move_anon_rmap(vmf->page, vma);
}
unlock_page(vmf->page);
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
wp_page_reuse() handles write shared faults which is needed only in
wp_page_shared(). Move the handling only into that location to make
wp_page_reuse() simpler and avoid a strange situation when we sometimes
pass in locked page, sometimes unlocked etc.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 27 ++++++++++++---------------
1 file changed, 12 insertions(+), 15 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index e278a8a6ccc7..06aba4203104 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2103,8 +2103,7 @@ static void fault_dirty_shared_page(struct vm_area_struct *vma,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf,
- int page_mkwrite, int dirty_shared)
+static inline void wp_page_reuse(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2124,16 +2123,6 @@ static inline int wp_page_reuse(struct vm_fault *vmf,
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
pte_unmap_unlock(vmf->pte, vmf->ptl);
-
- if (dirty_shared) {
- if (!page_mkwrite)
- lock_page(page);
-
- fault_dirty_shared_page(vma, page);
- put_page(page);
- }
-
- return VM_FAULT_WRITE;
}
/*
@@ -2308,7 +2297,8 @@ static int wp_pfn_shared(struct vm_fault *vmf)
return 0;
}
}
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
static int wp_page_shared(struct vm_fault *vmf)
@@ -2346,7 +2336,13 @@ static int wp_page_shared(struct vm_fault *vmf)
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, page_mkwrite, 1);
+ wp_page_reuse(vmf);
+ if (!page_mkwrite)
+ lock_page(vmf->page);
+ fault_dirty_shared_page(vma, vmf->page);
+ put_page(vmf->page);
+
+ return VM_FAULT_WRITE;
}
/*
@@ -2421,7 +2417,8 @@ static int do_wp_page(struct vm_fault *vmf)
page_move_anon_rmap(vmf->page, vma);
}
unlock_page(vmf->page);
- return wp_page_reuse(vmf, 0, 0);
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
}
unlock_page(vmf->page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Provide a helper function for finishing write faults due to PTE being
read-only. The helper will be used by DAX to avoid the need of
complicating generic MM code with DAX locking specifics.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 67 ++++++++++++++++++++++++++++++++----------------------
2 files changed, 41 insertions(+), 27 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index fb128beecdac..685ff1c57f2b 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -615,6 +615,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
int finish_fault(struct vm_fault *vmf);
+int finish_mkwrite_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index 06aba4203104..1517ff91c743 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2270,6 +2270,38 @@ static int wp_page_copy(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
+/**
+ * finish_mkwrite_fault - finish page fault for a shared mapping, making PTE
+ * writeable once the page is prepared
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a write page fault in a
+ * shared mapping due to PTE being read-only once the mapped page is prepared.
+ * It handles locking of PTE and modifying it. The function returns
+ * VM_FAULT_WRITE on success, 0 when PTE got changed before we acquired PTE
+ * lock.
+ *
+ * The function expects the page to be locked or other protection against
+ * concurrent faults / writeback (such as DAX radix tree locks).
+ */
+int finish_mkwrite_fault(struct vm_fault *vmf)
+{
+ WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ /*
+ * We might have raced with another page fault while we released the
+ * pte_offset_map_lock.
+ */
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
+}
+
/*
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
@@ -2286,16 +2318,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- /*
- * We might have raced with another page fault while we
- * released the pte_offset_map_lock.
- */
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
- }
+ return finish_mkwrite_fault(vmf);
}
wp_page_reuse(vmf);
return VM_FAULT_WRITE;
@@ -2305,7 +2328,6 @@ static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- int page_mkwrite = 0;
get_page(vmf->page);
@@ -2319,26 +2341,17 @@ static int wp_page_shared(struct vm_fault *vmf)
put_page(vmf->page);
return tmp;
}
- /*
- * Since we dropped the lock we need to revalidate
- * the PTE as someone else may have changed it. If
- * they did, we just return, as we can count on the
- * MMU to tell us if they didn't also make it writable.
- */
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ tmp = finish_mkwrite_fault(vmf);
+ if (unlikely(!tmp || (tmp &
+ (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
unlock_page(vmf->page);
- pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(vmf->page);
- return 0;
+ return tmp;
}
- page_mkwrite = 1;
- }
-
- wp_page_reuse(vmf);
- if (!page_mkwrite)
+ } else {
+ wp_page_reuse(vmf);
lock_page(vmf->page);
+ }
fault_dirty_shared_page(vma, vmf->page);
put_page(vmf->page);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Provide a helper function for finishing write faults due to PTE being
read-only. The helper will be used by DAX to avoid the need of
complicating generic MM code with DAX locking specifics.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 67 ++++++++++++++++++++++++++++++++----------------------
2 files changed, 41 insertions(+), 27 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index fb128beecdac..685ff1c57f2b 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -615,6 +615,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
int finish_fault(struct vm_fault *vmf);
+int finish_mkwrite_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index 06aba4203104..1517ff91c743 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2270,6 +2270,38 @@ static int wp_page_copy(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
+/**
+ * finish_mkwrite_fault - finish page fault for a shared mapping, making PTE
+ * writeable once the page is prepared
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a write page fault in a
+ * shared mapping due to PTE being read-only once the mapped page is prepared.
+ * It handles locking of PTE and modifying it. The function returns
+ * VM_FAULT_WRITE on success, 0 when PTE got changed before we acquired PTE
+ * lock.
+ *
+ * The function expects the page to be locked or other protection against
+ * concurrent faults / writeback (such as DAX radix tree locks).
+ */
+int finish_mkwrite_fault(struct vm_fault *vmf)
+{
+ WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ /*
+ * We might have raced with another page fault while we released the
+ * pte_offset_map_lock.
+ */
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
+}
+
/*
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
@@ -2286,16 +2318,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- /*
- * We might have raced with another page fault while we
- * released the pte_offset_map_lock.
- */
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
- }
+ return finish_mkwrite_fault(vmf);
}
wp_page_reuse(vmf);
return VM_FAULT_WRITE;
@@ -2305,7 +2328,6 @@ static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- int page_mkwrite = 0;
get_page(vmf->page);
@@ -2319,26 +2341,17 @@ static int wp_page_shared(struct vm_fault *vmf)
put_page(vmf->page);
return tmp;
}
- /*
- * Since we dropped the lock we need to revalidate
- * the PTE as someone else may have changed it. If
- * they did, we just return, as we can count on the
- * MMU to tell us if they didn't also make it writable.
- */
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ tmp = finish_mkwrite_fault(vmf);
+ if (unlikely(!tmp || (tmp &
+ (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
unlock_page(vmf->page);
- pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(vmf->page);
- return 0;
+ return tmp;
}
- page_mkwrite = 1;
- }
-
- wp_page_reuse(vmf);
- if (!page_mkwrite)
+ } else {
+ wp_page_reuse(vmf);
lock_page(vmf->page);
+ }
fault_dirty_shared_page(vma, vmf->page);
put_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Provide a helper function for finishing write faults due to PTE being
read-only. The helper will be used by DAX to avoid the need of
complicating generic MM code with DAX locking specifics.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 1 +
mm/memory.c | 67 ++++++++++++++++++++++++++++++++----------------------
2 files changed, 41 insertions(+), 27 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index fb128beecdac..685ff1c57f2b 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -615,6 +615,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
struct page *page);
int finish_fault(struct vm_fault *vmf);
+int finish_mkwrite_fault(struct vm_fault *vmf);
#endif
/*
diff --git a/mm/memory.c b/mm/memory.c
index 06aba4203104..1517ff91c743 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2270,6 +2270,38 @@ static int wp_page_copy(struct vm_fault *vmf)
return VM_FAULT_OOM;
}
+/**
+ * finish_mkwrite_fault - finish page fault for a shared mapping, making PTE
+ * writeable once the page is prepared
+ *
+ * @vmf: structure describing the fault
+ *
+ * This function handles all that is needed to finish a write page fault in a
+ * shared mapping due to PTE being read-only once the mapped page is prepared.
+ * It handles locking of PTE and modifying it. The function returns
+ * VM_FAULT_WRITE on success, 0 when PTE got changed before we acquired PTE
+ * lock.
+ *
+ * The function expects the page to be locked or other protection against
+ * concurrent faults / writeback (such as DAX radix tree locks).
+ */
+int finish_mkwrite_fault(struct vm_fault *vmf)
+{
+ WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
+ vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
+ &vmf->ptl);
+ /*
+ * We might have raced with another page fault while we released the
+ * pte_offset_map_lock.
+ */
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ pte_unmap_unlock(vmf->pte, vmf->ptl);
+ return 0;
+ }
+ wp_page_reuse(vmf);
+ return VM_FAULT_WRITE;
+}
+
/*
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
@@ -2286,16 +2318,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
if (ret & VM_FAULT_ERROR)
return ret;
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- /*
- * We might have raced with another page fault while we
- * released the pte_offset_map_lock.
- */
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
- pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
- }
+ return finish_mkwrite_fault(vmf);
}
wp_page_reuse(vmf);
return VM_FAULT_WRITE;
@@ -2305,7 +2328,6 @@ static int wp_page_shared(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
- int page_mkwrite = 0;
get_page(vmf->page);
@@ -2319,26 +2341,17 @@ static int wp_page_shared(struct vm_fault *vmf)
put_page(vmf->page);
return tmp;
}
- /*
- * Since we dropped the lock we need to revalidate
- * the PTE as someone else may have changed it. If
- * they did, we just return, as we can count on the
- * MMU to tell us if they didn't also make it writable.
- */
- vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
- vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, vmf->orig_pte)) {
+ tmp = finish_mkwrite_fault(vmf);
+ if (unlikely(!tmp || (tmp &
+ (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
unlock_page(vmf->page);
- pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(vmf->page);
- return 0;
+ return tmp;
}
- page_mkwrite = 1;
- }
-
- wp_page_reuse(vmf);
- if (!page_mkwrite)
+ } else {
+ wp_page_reuse(vmf);
lock_page(vmf->page);
+ }
fault_dirty_shared_page(vma, vmf->page);
put_page(vmf->page);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 17/21] mm: Change return values of finish_mkwrite_fault()
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Currently finish_mkwrite_fault() returns 0 when PTE got changed before
we acquired PTE lock and VM_FAULT_WRITE when we succeeded in modifying
the PTE. This is somewhat confusing since 0 generally means success, it
is also inconsistent with finish_fault() which returns 0 on success.
Change finish_mkwrite_fault() to return 0 on success and VM_FAULT_NOPAGE
when PTE changed. Practically, there should be no behavioral difference
since we bail out from the fault the same way regardless whether we
return 0, VM_FAULT_NOPAGE, or VM_FAULT_WRITE. Also note that
VM_FAULT_WRITE has no effect for shared mappings since the only two
places that check it - KSM and GUP - care about private mappings only.
Generally the meaning of VM_FAULT_WRITE for shared mappings is not well
defined and we should probably clean that up.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 1517ff91c743..b3bd6b6c6472 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2296,10 +2296,10 @@ int finish_mkwrite_fault(struct vm_fault *vmf)
*/
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
+ return VM_FAULT_NOPAGE;
}
wp_page_reuse(vmf);
- return VM_FAULT_WRITE;
+ return 0;
}
/*
@@ -2342,8 +2342,7 @@ static int wp_page_shared(struct vm_fault *vmf)
return tmp;
}
tmp = finish_mkwrite_fault(vmf);
- if (unlikely(!tmp || (tmp &
- (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
+ if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {
unlock_page(vmf->page);
put_page(vmf->page);
return tmp;
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 17/21] mm: Change return values of finish_mkwrite_fault()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently finish_mkwrite_fault() returns 0 when PTE got changed before
we acquired PTE lock and VM_FAULT_WRITE when we succeeded in modifying
the PTE. This is somewhat confusing since 0 generally means success, it
is also inconsistent with finish_fault() which returns 0 on success.
Change finish_mkwrite_fault() to return 0 on success and VM_FAULT_NOPAGE
when PTE changed. Practically, there should be no behavioral difference
since we bail out from the fault the same way regardless whether we
return 0, VM_FAULT_NOPAGE, or VM_FAULT_WRITE. Also note that
VM_FAULT_WRITE has no effect for shared mappings since the only two
places that check it - KSM and GUP - care about private mappings only.
Generally the meaning of VM_FAULT_WRITE for shared mappings is not well
defined and we should probably clean that up.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 1517ff91c743..b3bd6b6c6472 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2296,10 +2296,10 @@ int finish_mkwrite_fault(struct vm_fault *vmf)
*/
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
+ return VM_FAULT_NOPAGE;
}
wp_page_reuse(vmf);
- return VM_FAULT_WRITE;
+ return 0;
}
/*
@@ -2342,8 +2342,7 @@ static int wp_page_shared(struct vm_fault *vmf)
return tmp;
}
tmp = finish_mkwrite_fault(vmf);
- if (unlikely(!tmp || (tmp &
- (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
+ if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {
unlock_page(vmf->page);
put_page(vmf->page);
return tmp;
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 17/21] mm: Change return values of finish_mkwrite_fault()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently finish_mkwrite_fault() returns 0 when PTE got changed before
we acquired PTE lock and VM_FAULT_WRITE when we succeeded in modifying
the PTE. This is somewhat confusing since 0 generally means success, it
is also inconsistent with finish_fault() which returns 0 on success.
Change finish_mkwrite_fault() to return 0 on success and VM_FAULT_NOPAGE
when PTE changed. Practically, there should be no behavioral difference
since we bail out from the fault the same way regardless whether we
return 0, VM_FAULT_NOPAGE, or VM_FAULT_WRITE. Also note that
VM_FAULT_WRITE has no effect for shared mappings since the only two
places that check it - KSM and GUP - care about private mappings only.
Generally the meaning of VM_FAULT_WRITE for shared mappings is not well
defined and we should probably clean that up.
Signed-off-by: Jan Kara <jack@suse.cz>
---
mm/memory.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 1517ff91c743..b3bd6b6c6472 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2296,10 +2296,10 @@ int finish_mkwrite_fault(struct vm_fault *vmf)
*/
if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return 0;
+ return VM_FAULT_NOPAGE;
}
wp_page_reuse(vmf);
- return VM_FAULT_WRITE;
+ return 0;
}
/*
@@ -2342,8 +2342,7 @@ static int wp_page_shared(struct vm_fault *vmf)
return tmp;
}
tmp = finish_mkwrite_fault(vmf);
- if (unlikely(!tmp || (tmp &
- (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
+ if (unlikely(tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))) {
unlock_page(vmf->page);
put_page(vmf->page);
return tmp;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 18/21] mm: Export follow_pte()
2016-11-04 4:24 ` Jan Kara
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
DAX will need to implement its own version of page_check_address(). To
avoid duplicating page table walking code, export follow_pte() which
does what we need.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 2 ++
mm/memory.c | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 685ff1c57f2b..a5f52c0ce61a 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1210,6 +1210,8 @@ int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
struct vm_area_struct *vma);
void unmap_mapping_range(struct address_space *mapping,
loff_t const holebegin, loff_t const holelen, int even_cows);
+int follow_pte(struct mm_struct *mm, unsigned long address, pte_t **ptepp,
+ spinlock_t **ptlp);
int follow_pfn(struct vm_area_struct *vma, unsigned long address,
unsigned long *pfn);
int follow_phys(struct vm_area_struct *vma, unsigned long address,
diff --git a/mm/memory.c b/mm/memory.c
index b3bd6b6c6472..7660f6169bee 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3782,8 +3782,8 @@ static int __follow_pte(struct mm_struct *mm, unsigned long address,
return -EINVAL;
}
-static inline int follow_pte(struct mm_struct *mm, unsigned long address,
- pte_t **ptepp, spinlock_t **ptlp)
+int follow_pte(struct mm_struct *mm, unsigned long address, pte_t **ptepp,
+ spinlock_t **ptlp)
{
int res;
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 18/21] mm: Export follow_pte()
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
DAX will need to implement its own version of page_check_address(). To
avoid duplicating page table walking code, export follow_pte() which
does what we need.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 2 ++
mm/memory.c | 4 ++--
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 685ff1c57f2b..a5f52c0ce61a 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -1210,6 +1210,8 @@ int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
struct vm_area_struct *vma);
void unmap_mapping_range(struct address_space *mapping,
loff_t const holebegin, loff_t const holelen, int even_cows);
+int follow_pte(struct mm_struct *mm, unsigned long address, pte_t **ptepp,
+ spinlock_t **ptlp);
int follow_pfn(struct vm_area_struct *vma, unsigned long address,
unsigned long *pfn);
int follow_phys(struct vm_area_struct *vma, unsigned long address,
diff --git a/mm/memory.c b/mm/memory.c
index b3bd6b6c6472..7660f6169bee 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -3782,8 +3782,8 @@ static int __follow_pte(struct mm_struct *mm, unsigned long address,
return -EINVAL;
}
-static inline int follow_pte(struct mm_struct *mm, unsigned long address,
- pte_t **ptepp, spinlock_t **ptlp)
+int follow_pte(struct mm_struct *mm, unsigned long address, pte_t **ptepp,
+ spinlock_t **ptlp)
{
int res;
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 19/21] dax: Make cache flushing protected by entry lock
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Currently, flushing of caches for DAX mappings was ignoring entry lock.
So far this was ok (modulo a bug that a difference in entry lock could
cause cache flushing to be mistakenly skipped) but in the following
patches we will write-protect PTEs on cache flushing and clear dirty
tags. For that we will need more exclusion. So do cache flushing under
an entry lock. This allows us to remove one lock-unlock pair of
mapping->tree_lock as a bonus.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 61 +++++++++++++++++++++++++++++++++++++++----------------------
1 file changed, 39 insertions(+), 22 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index acba769152bf..167583a9a324 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -619,32 +619,50 @@ static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
struct radix_tree_root *page_tree = &mapping->page_tree;
- struct radix_tree_node *node;
struct blk_dax_ctl dax;
- void **slot;
+ void *entry2, **slot;
int ret = 0;
- spin_lock_irq(&mapping->tree_lock);
/*
- * Regular page slots are stabilized by the page lock even
- * without the tree itself locked. These unlocked entries
- * need verification under the tree lock.
+ * A page got tagged dirty in DAX mapping? Something is seriously
+ * wrong.
*/
- if (!__radix_tree_lookup(page_tree, index, &node, &slot))
- goto unlock;
- if (*slot != entry)
- goto unlock;
-
- /* another fsync thread may have already written back this entry */
- if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
- goto unlock;
+ if (WARN_ON(!radix_tree_exceptional_entry(entry)))
+ return -EIO;
+ spin_lock_irq(&mapping->tree_lock);
+ entry2 = get_unlocked_mapping_entry(mapping, index, &slot);
+ /* Entry got punched out / reallocated? */
+ if (!entry2 || !radix_tree_exceptional_entry(entry2))
+ goto put_unlocked;
+ /*
+ * Entry got reallocated elsewhere? No need to writeback. We have to
+ * compare sectors as we must not bail out due to difference in lockbit
+ * or entry type.
+ */
+ if (dax_radix_sector(entry2) != dax_radix_sector(entry))
+ goto put_unlocked;
if (WARN_ON_ONCE(dax_is_empty_entry(entry) ||
dax_is_zero_entry(entry))) {
ret = -EIO;
- goto unlock;
+ goto put_unlocked;
}
+ /* Another fsync thread may have already written back this entry */
+ if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
+ goto put_unlocked;
+ /* Lock the entry to serialize with page faults */
+ entry = lock_slot(mapping, slot);
+ /*
+ * We can clear the tag now but we have to be careful so that concurrent
+ * dax_writeback_one() calls for the same index cannot finish before we
+ * actually flush the caches. This is achieved as the calls will look
+ * at the entry only under tree_lock and once they do that they will
+ * see the entry locked and wait for it to unlock.
+ */
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
+ spin_unlock_irq(&mapping->tree_lock);
+
/*
* Even if dax_writeback_mapping_range() was given a wbc->range_start
* in the middle of a PMD, the 'index' we are given will be aligned to
@@ -654,15 +672,16 @@ static int dax_writeback_one(struct block_device *bdev,
*/
dax.sector = dax_radix_sector(entry);
dax.size = PAGE_SIZE << dax_radix_order(entry);
- spin_unlock_irq(&mapping->tree_lock);
/*
* We cannot hold tree_lock while calling dax_map_atomic() because it
* eventually calls cond_resched().
*/
ret = dax_map_atomic(bdev, &dax);
- if (ret < 0)
+ if (ret < 0) {
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
+ }
if (WARN_ON_ONCE(ret < dax.size)) {
ret = -EIO;
@@ -670,15 +689,13 @@ static int dax_writeback_one(struct block_device *bdev,
}
wb_cache_pmem(dax.addr, dax.size);
-
- spin_lock_irq(&mapping->tree_lock);
- radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
- spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
- unlock:
+ put_unlocked:
+ put_unlocked_mapping_entry(mapping, index, entry2);
spin_unlock_irq(&mapping->tree_lock);
return ret;
}
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 19/21] dax: Make cache flushing protected by entry lock
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently, flushing of caches for DAX mappings was ignoring entry lock.
So far this was ok (modulo a bug that a difference in entry lock could
cause cache flushing to be mistakenly skipped) but in the following
patches we will write-protect PTEs on cache flushing and clear dirty
tags. For that we will need more exclusion. So do cache flushing under
an entry lock. This allows us to remove one lock-unlock pair of
mapping->tree_lock as a bonus.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 61 +++++++++++++++++++++++++++++++++++++++----------------------
1 file changed, 39 insertions(+), 22 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index acba769152bf..167583a9a324 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -619,32 +619,50 @@ static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
struct radix_tree_root *page_tree = &mapping->page_tree;
- struct radix_tree_node *node;
struct blk_dax_ctl dax;
- void **slot;
+ void *entry2, **slot;
int ret = 0;
- spin_lock_irq(&mapping->tree_lock);
/*
- * Regular page slots are stabilized by the page lock even
- * without the tree itself locked. These unlocked entries
- * need verification under the tree lock.
+ * A page got tagged dirty in DAX mapping? Something is seriously
+ * wrong.
*/
- if (!__radix_tree_lookup(page_tree, index, &node, &slot))
- goto unlock;
- if (*slot != entry)
- goto unlock;
-
- /* another fsync thread may have already written back this entry */
- if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
- goto unlock;
+ if (WARN_ON(!radix_tree_exceptional_entry(entry)))
+ return -EIO;
+ spin_lock_irq(&mapping->tree_lock);
+ entry2 = get_unlocked_mapping_entry(mapping, index, &slot);
+ /* Entry got punched out / reallocated? */
+ if (!entry2 || !radix_tree_exceptional_entry(entry2))
+ goto put_unlocked;
+ /*
+ * Entry got reallocated elsewhere? No need to writeback. We have to
+ * compare sectors as we must not bail out due to difference in lockbit
+ * or entry type.
+ */
+ if (dax_radix_sector(entry2) != dax_radix_sector(entry))
+ goto put_unlocked;
if (WARN_ON_ONCE(dax_is_empty_entry(entry) ||
dax_is_zero_entry(entry))) {
ret = -EIO;
- goto unlock;
+ goto put_unlocked;
}
+ /* Another fsync thread may have already written back this entry */
+ if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
+ goto put_unlocked;
+ /* Lock the entry to serialize with page faults */
+ entry = lock_slot(mapping, slot);
+ /*
+ * We can clear the tag now but we have to be careful so that concurrent
+ * dax_writeback_one() calls for the same index cannot finish before we
+ * actually flush the caches. This is achieved as the calls will look
+ * at the entry only under tree_lock and once they do that they will
+ * see the entry locked and wait for it to unlock.
+ */
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
+ spin_unlock_irq(&mapping->tree_lock);
+
/*
* Even if dax_writeback_mapping_range() was given a wbc->range_start
* in the middle of a PMD, the 'index' we are given will be aligned to
@@ -654,15 +672,16 @@ static int dax_writeback_one(struct block_device *bdev,
*/
dax.sector = dax_radix_sector(entry);
dax.size = PAGE_SIZE << dax_radix_order(entry);
- spin_unlock_irq(&mapping->tree_lock);
/*
* We cannot hold tree_lock while calling dax_map_atomic() because it
* eventually calls cond_resched().
*/
ret = dax_map_atomic(bdev, &dax);
- if (ret < 0)
+ if (ret < 0) {
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
+ }
if (WARN_ON_ONCE(ret < dax.size)) {
ret = -EIO;
@@ -670,15 +689,13 @@ static int dax_writeback_one(struct block_device *bdev,
}
wb_cache_pmem(dax.addr, dax.size);
-
- spin_lock_irq(&mapping->tree_lock);
- radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
- spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
- unlock:
+ put_unlocked:
+ put_unlocked_mapping_entry(mapping, index, entry2);
spin_unlock_irq(&mapping->tree_lock);
return ret;
}
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 19/21] dax: Make cache flushing protected by entry lock
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently, flushing of caches for DAX mappings was ignoring entry lock.
So far this was ok (modulo a bug that a difference in entry lock could
cause cache flushing to be mistakenly skipped) but in the following
patches we will write-protect PTEs on cache flushing and clear dirty
tags. For that we will need more exclusion. So do cache flushing under
an entry lock. This allows us to remove one lock-unlock pair of
mapping->tree_lock as a bonus.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 61 +++++++++++++++++++++++++++++++++++++++----------------------
1 file changed, 39 insertions(+), 22 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index acba769152bf..167583a9a324 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -619,32 +619,50 @@ static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
struct radix_tree_root *page_tree = &mapping->page_tree;
- struct radix_tree_node *node;
struct blk_dax_ctl dax;
- void **slot;
+ void *entry2, **slot;
int ret = 0;
- spin_lock_irq(&mapping->tree_lock);
/*
- * Regular page slots are stabilized by the page lock even
- * without the tree itself locked. These unlocked entries
- * need verification under the tree lock.
+ * A page got tagged dirty in DAX mapping? Something is seriously
+ * wrong.
*/
- if (!__radix_tree_lookup(page_tree, index, &node, &slot))
- goto unlock;
- if (*slot != entry)
- goto unlock;
-
- /* another fsync thread may have already written back this entry */
- if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
- goto unlock;
+ if (WARN_ON(!radix_tree_exceptional_entry(entry)))
+ return -EIO;
+ spin_lock_irq(&mapping->tree_lock);
+ entry2 = get_unlocked_mapping_entry(mapping, index, &slot);
+ /* Entry got punched out / reallocated? */
+ if (!entry2 || !radix_tree_exceptional_entry(entry2))
+ goto put_unlocked;
+ /*
+ * Entry got reallocated elsewhere? No need to writeback. We have to
+ * compare sectors as we must not bail out due to difference in lockbit
+ * or entry type.
+ */
+ if (dax_radix_sector(entry2) != dax_radix_sector(entry))
+ goto put_unlocked;
if (WARN_ON_ONCE(dax_is_empty_entry(entry) ||
dax_is_zero_entry(entry))) {
ret = -EIO;
- goto unlock;
+ goto put_unlocked;
}
+ /* Another fsync thread may have already written back this entry */
+ if (!radix_tree_tag_get(page_tree, index, PAGECACHE_TAG_TOWRITE))
+ goto put_unlocked;
+ /* Lock the entry to serialize with page faults */
+ entry = lock_slot(mapping, slot);
+ /*
+ * We can clear the tag now but we have to be careful so that concurrent
+ * dax_writeback_one() calls for the same index cannot finish before we
+ * actually flush the caches. This is achieved as the calls will look
+ * at the entry only under tree_lock and once they do that they will
+ * see the entry locked and wait for it to unlock.
+ */
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
+ spin_unlock_irq(&mapping->tree_lock);
+
/*
* Even if dax_writeback_mapping_range() was given a wbc->range_start
* in the middle of a PMD, the 'index' we are given will be aligned to
@@ -654,15 +672,16 @@ static int dax_writeback_one(struct block_device *bdev,
*/
dax.sector = dax_radix_sector(entry);
dax.size = PAGE_SIZE << dax_radix_order(entry);
- spin_unlock_irq(&mapping->tree_lock);
/*
* We cannot hold tree_lock while calling dax_map_atomic() because it
* eventually calls cond_resched().
*/
ret = dax_map_atomic(bdev, &dax);
- if (ret < 0)
+ if (ret < 0) {
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
+ }
if (WARN_ON_ONCE(ret < dax.size)) {
ret = -EIO;
@@ -670,15 +689,13 @@ static int dax_writeback_one(struct block_device *bdev,
}
wb_cache_pmem(dax.addr, dax.size);
-
- spin_lock_irq(&mapping->tree_lock);
- radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_TOWRITE);
- spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
+ put_locked_mapping_entry(mapping, index, entry);
return ret;
- unlock:
+ put_unlocked:
+ put_unlocked_mapping_entry(mapping, index, entry2);
spin_unlock_irq(&mapping->tree_lock);
return ret;
}
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 20/21] dax: Protect PTE modification on WP fault by radix tree entry lock
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: Jan Kara, linux-nvdimm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
Currently PTE gets updated in wp_pfn_shared() after dax_pfn_mkwrite()
has released corresponding radix tree entry lock. When we want to
writeprotect PTE on cache flush, we need PTE modification to happen
under radix tree entry lock to ensure consistent updates of PTE and radix
tree (standard faults use page lock to ensure this consistency). So move
update of PTE bit into dax_pfn_mkwrite().
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 22 ++++++++++++++++------
mm/memory.c | 2 +-
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 167583a9a324..7b00316ebf49 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -784,17 +784,27 @@ int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct file *file = vma->vm_file;
struct address_space *mapping = file->f_mapping;
- void *entry;
+ void *entry, **slot;
pgoff_t index = vmf->pgoff;
spin_lock_irq(&mapping->tree_lock);
- entry = get_unlocked_mapping_entry(mapping, index, NULL);
- if (!entry || !radix_tree_exceptional_entry(entry))
- goto out;
+ entry = get_unlocked_mapping_entry(mapping, index, &slot);
+ if (!entry || !radix_tree_exceptional_entry(entry)) {
+ if (entry)
+ put_unlocked_mapping_entry(mapping, index, entry);
+ spin_unlock_irq(&mapping->tree_lock);
+ return VM_FAULT_NOPAGE;
+ }
radix_tree_tag_set(&mapping->page_tree, index, PAGECACHE_TAG_DIRTY);
- put_unlocked_mapping_entry(mapping, index, entry);
-out:
+ entry = lock_slot(mapping, slot);
spin_unlock_irq(&mapping->tree_lock);
+ /*
+ * If we race with somebody updating the PTE and finish_mkwrite_fault()
+ * fails, we don't care. We need to return VM_FAULT_NOPAGE and retry
+ * the fault in either case.
+ */
+ finish_mkwrite_fault(vmf);
+ put_locked_mapping_entry(mapping, index, entry);
return VM_FAULT_NOPAGE;
}
EXPORT_SYMBOL_GPL(dax_pfn_mkwrite);
diff --git a/mm/memory.c b/mm/memory.c
index 7660f6169bee..2683e18d6d55 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2316,7 +2316,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
pte_unmap_unlock(vmf->pte, vmf->ptl);
vmf->flags |= FAULT_FLAG_MKWRITE;
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
- if (ret & VM_FAULT_ERROR)
+ if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))
return ret;
return finish_mkwrite_fault(vmf);
}
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 20/21] dax: Protect PTE modification on WP fault by radix tree entry lock
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently PTE gets updated in wp_pfn_shared() after dax_pfn_mkwrite()
has released corresponding radix tree entry lock. When we want to
writeprotect PTE on cache flush, we need PTE modification to happen
under radix tree entry lock to ensure consistent updates of PTE and radix
tree (standard faults use page lock to ensure this consistency). So move
update of PTE bit into dax_pfn_mkwrite().
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 22 ++++++++++++++++------
mm/memory.c | 2 +-
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 167583a9a324..7b00316ebf49 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -784,17 +784,27 @@ int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct file *file = vma->vm_file;
struct address_space *mapping = file->f_mapping;
- void *entry;
+ void *entry, **slot;
pgoff_t index = vmf->pgoff;
spin_lock_irq(&mapping->tree_lock);
- entry = get_unlocked_mapping_entry(mapping, index, NULL);
- if (!entry || !radix_tree_exceptional_entry(entry))
- goto out;
+ entry = get_unlocked_mapping_entry(mapping, index, &slot);
+ if (!entry || !radix_tree_exceptional_entry(entry)) {
+ if (entry)
+ put_unlocked_mapping_entry(mapping, index, entry);
+ spin_unlock_irq(&mapping->tree_lock);
+ return VM_FAULT_NOPAGE;
+ }
radix_tree_tag_set(&mapping->page_tree, index, PAGECACHE_TAG_DIRTY);
- put_unlocked_mapping_entry(mapping, index, entry);
-out:
+ entry = lock_slot(mapping, slot);
spin_unlock_irq(&mapping->tree_lock);
+ /*
+ * If we race with somebody updating the PTE and finish_mkwrite_fault()
+ * fails, we don't care. We need to return VM_FAULT_NOPAGE and retry
+ * the fault in either case.
+ */
+ finish_mkwrite_fault(vmf);
+ put_locked_mapping_entry(mapping, index, entry);
return VM_FAULT_NOPAGE;
}
EXPORT_SYMBOL_GPL(dax_pfn_mkwrite);
diff --git a/mm/memory.c b/mm/memory.c
index 7660f6169bee..2683e18d6d55 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2316,7 +2316,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
pte_unmap_unlock(vmf->pte, vmf->ptl);
vmf->flags |= FAULT_FLAG_MKWRITE;
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
- if (ret & VM_FAULT_ERROR)
+ if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))
return ret;
return finish_mkwrite_fault(vmf);
}
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 20/21] dax: Protect PTE modification on WP fault by radix tree entry lock
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently PTE gets updated in wp_pfn_shared() after dax_pfn_mkwrite()
has released corresponding radix tree entry lock. When we want to
writeprotect PTE on cache flush, we need PTE modification to happen
under radix tree entry lock to ensure consistent updates of PTE and radix
tree (standard faults use page lock to ensure this consistency). So move
update of PTE bit into dax_pfn_mkwrite().
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 22 ++++++++++++++++------
mm/memory.c | 2 +-
2 files changed, 17 insertions(+), 7 deletions(-)
diff --git a/fs/dax.c b/fs/dax.c
index 167583a9a324..7b00316ebf49 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -784,17 +784,27 @@ int dax_pfn_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf)
{
struct file *file = vma->vm_file;
struct address_space *mapping = file->f_mapping;
- void *entry;
+ void *entry, **slot;
pgoff_t index = vmf->pgoff;
spin_lock_irq(&mapping->tree_lock);
- entry = get_unlocked_mapping_entry(mapping, index, NULL);
- if (!entry || !radix_tree_exceptional_entry(entry))
- goto out;
+ entry = get_unlocked_mapping_entry(mapping, index, &slot);
+ if (!entry || !radix_tree_exceptional_entry(entry)) {
+ if (entry)
+ put_unlocked_mapping_entry(mapping, index, entry);
+ spin_unlock_irq(&mapping->tree_lock);
+ return VM_FAULT_NOPAGE;
+ }
radix_tree_tag_set(&mapping->page_tree, index, PAGECACHE_TAG_DIRTY);
- put_unlocked_mapping_entry(mapping, index, entry);
-out:
+ entry = lock_slot(mapping, slot);
spin_unlock_irq(&mapping->tree_lock);
+ /*
+ * If we race with somebody updating the PTE and finish_mkwrite_fault()
+ * fails, we don't care. We need to return VM_FAULT_NOPAGE and retry
+ * the fault in either case.
+ */
+ finish_mkwrite_fault(vmf);
+ put_locked_mapping_entry(mapping, index, entry);
return VM_FAULT_NOPAGE;
}
EXPORT_SYMBOL_GPL(dax_pfn_mkwrite);
diff --git a/mm/memory.c b/mm/memory.c
index 7660f6169bee..2683e18d6d55 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2316,7 +2316,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
pte_unmap_unlock(vmf->pte, vmf->ptl);
vmf->flags |= FAULT_FLAG_MKWRITE;
ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
- if (ret & VM_FAULT_ERROR)
+ if (ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE))
return ret;
return finish_mkwrite_fault(vmf);
}
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 21/21] dax: Clear dirty entry tags on cache flush
2016-11-04 4:24 ` Jan Kara
(?)
@ 2016-11-04 4:25 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we never clear dirty tags in DAX mappings and thus address
ranges to flush accumulate. Now that we have locking of radix tree
entries, we have all the locking necessary to reliably clear the radix
tree dirty tag when flushing caches for corresponding address range.
Similarly to page_mkclean() we also have to write-protect pages to get a
page fault when the page is next written to so that we can mark the
entry dirty again.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
diff --git a/fs/dax.c b/fs/dax.c
index 7b00316ebf49..263f4e1671ca 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -31,6 +31,7 @@
#include <linux/vmstat.h>
#include <linux/pfn_t.h>
#include <linux/sizes.h>
+#include <linux/mmu_notifier.h>
#include <linux/iomap.h>
#include "internal.h"
@@ -615,6 +616,59 @@ static void *dax_insert_mapping_entry(struct address_space *mapping,
return new_entry;
}
+static inline unsigned long
+pgoff_address(pgoff_t pgoff, struct vm_area_struct *vma)
+{
+ unsigned long address;
+
+ address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+ return address;
+}
+
+/* Walk all mappings of a given index of a file and writeprotect them */
+static void dax_mapping_entry_mkclean(struct address_space *mapping,
+ pgoff_t index, unsigned long pfn)
+{
+ struct vm_area_struct *vma;
+ pte_t *ptep;
+ pte_t pte;
+ spinlock_t *ptl;
+ bool changed;
+
+ i_mmap_lock_read(mapping);
+ vma_interval_tree_foreach(vma, &mapping->i_mmap, index, index) {
+ unsigned long address;
+
+ cond_resched();
+
+ if (!(vma->vm_flags & VM_SHARED))
+ continue;
+
+ address = pgoff_address(index, vma);
+ changed = false;
+ if (follow_pte(vma->vm_mm, address, &ptep, &ptl))
+ continue;
+ if (pfn != pte_pfn(*ptep))
+ goto unlock;
+ if (!pte_dirty(*ptep) && !pte_write(*ptep))
+ goto unlock;
+
+ flush_cache_page(vma, address, pfn);
+ pte = ptep_clear_flush(vma, address, ptep);
+ pte = pte_wrprotect(pte);
+ pte = pte_mkclean(pte);
+ set_pte_at(vma->vm_mm, address, ptep, pte);
+ changed = true;
+unlock:
+ pte_unmap_unlock(ptep, ptl);
+
+ if (changed)
+ mmu_notifier_invalidate_page(vma->vm_mm, address);
+ }
+ i_mmap_unlock_read(mapping);
+}
+
static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
@@ -688,7 +742,17 @@ static int dax_writeback_one(struct block_device *bdev,
goto unmap;
}
+ dax_mapping_entry_mkclean(mapping, index, pfn_t_to_pfn(dax.pfn));
wb_cache_pmem(dax.addr, dax.size);
+ /*
+ * After we have flushed the cache, we can clear the dirty tag. There
+ * cannot be new dirty data in the pfn after the flush has completed as
+ * the pfn mappings are writeprotected and fault waits for mapping
+ * entry lock.
+ */
+ spin_lock_irq(&mapping->tree_lock);
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_DIRTY);
+ spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
put_locked_mapping_entry(mapping, index, entry);
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 21/21] dax: Clear dirty entry tags on cache flush
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we never clear dirty tags in DAX mappings and thus address
ranges to flush accumulate. Now that we have locking of radix tree
entries, we have all the locking necessary to reliably clear the radix
tree dirty tag when flushing caches for corresponding address range.
Similarly to page_mkclean() we also have to write-protect pages to get a
page fault when the page is next written to so that we can mark the
entry dirty again.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
diff --git a/fs/dax.c b/fs/dax.c
index 7b00316ebf49..263f4e1671ca 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -31,6 +31,7 @@
#include <linux/vmstat.h>
#include <linux/pfn_t.h>
#include <linux/sizes.h>
+#include <linux/mmu_notifier.h>
#include <linux/iomap.h>
#include "internal.h"
@@ -615,6 +616,59 @@ static void *dax_insert_mapping_entry(struct address_space *mapping,
return new_entry;
}
+static inline unsigned long
+pgoff_address(pgoff_t pgoff, struct vm_area_struct *vma)
+{
+ unsigned long address;
+
+ address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+ return address;
+}
+
+/* Walk all mappings of a given index of a file and writeprotect them */
+static void dax_mapping_entry_mkclean(struct address_space *mapping,
+ pgoff_t index, unsigned long pfn)
+{
+ struct vm_area_struct *vma;
+ pte_t *ptep;
+ pte_t pte;
+ spinlock_t *ptl;
+ bool changed;
+
+ i_mmap_lock_read(mapping);
+ vma_interval_tree_foreach(vma, &mapping->i_mmap, index, index) {
+ unsigned long address;
+
+ cond_resched();
+
+ if (!(vma->vm_flags & VM_SHARED))
+ continue;
+
+ address = pgoff_address(index, vma);
+ changed = false;
+ if (follow_pte(vma->vm_mm, address, &ptep, &ptl))
+ continue;
+ if (pfn != pte_pfn(*ptep))
+ goto unlock;
+ if (!pte_dirty(*ptep) && !pte_write(*ptep))
+ goto unlock;
+
+ flush_cache_page(vma, address, pfn);
+ pte = ptep_clear_flush(vma, address, ptep);
+ pte = pte_wrprotect(pte);
+ pte = pte_mkclean(pte);
+ set_pte_at(vma->vm_mm, address, ptep, pte);
+ changed = true;
+unlock:
+ pte_unmap_unlock(ptep, ptl);
+
+ if (changed)
+ mmu_notifier_invalidate_page(vma->vm_mm, address);
+ }
+ i_mmap_unlock_read(mapping);
+}
+
static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
@@ -688,7 +742,17 @@ static int dax_writeback_one(struct block_device *bdev,
goto unmap;
}
+ dax_mapping_entry_mkclean(mapping, index, pfn_t_to_pfn(dax.pfn));
wb_cache_pmem(dax.addr, dax.size);
+ /*
+ * After we have flushed the cache, we can clear the dirty tag. There
+ * cannot be new dirty data in the pfn after the flush has completed as
+ * the pfn mappings are writeprotected and fault waits for mapping
+ * entry lock.
+ */
+ spin_lock_irq(&mapping->tree_lock);
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_DIRTY);
+ spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
put_locked_mapping_entry(mapping, index, entry);
--
2.6.6
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 21/21] dax: Clear dirty entry tags on cache flush
@ 2016-11-04 4:25 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-04 4:25 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler,
Kirill A. Shutemov, Jan Kara
Currently we never clear dirty tags in DAX mappings and thus address
ranges to flush accumulate. Now that we have locking of radix tree
entries, we have all the locking necessary to reliably clear the radix
tree dirty tag when flushing caches for corresponding address range.
Similarly to page_mkclean() we also have to write-protect pages to get a
page fault when the page is next written to so that we can mark the
entry dirty again.
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Jan Kara <jack@suse.cz>
---
fs/dax.c | 64 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 64 insertions(+)
diff --git a/fs/dax.c b/fs/dax.c
index 7b00316ebf49..263f4e1671ca 100644
--- a/fs/dax.c
+++ b/fs/dax.c
@@ -31,6 +31,7 @@
#include <linux/vmstat.h>
#include <linux/pfn_t.h>
#include <linux/sizes.h>
+#include <linux/mmu_notifier.h>
#include <linux/iomap.h>
#include "internal.h"
@@ -615,6 +616,59 @@ static void *dax_insert_mapping_entry(struct address_space *mapping,
return new_entry;
}
+static inline unsigned long
+pgoff_address(pgoff_t pgoff, struct vm_area_struct *vma)
+{
+ unsigned long address;
+
+ address = vma->vm_start + ((pgoff - vma->vm_pgoff) << PAGE_SHIFT);
+ VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
+ return address;
+}
+
+/* Walk all mappings of a given index of a file and writeprotect them */
+static void dax_mapping_entry_mkclean(struct address_space *mapping,
+ pgoff_t index, unsigned long pfn)
+{
+ struct vm_area_struct *vma;
+ pte_t *ptep;
+ pte_t pte;
+ spinlock_t *ptl;
+ bool changed;
+
+ i_mmap_lock_read(mapping);
+ vma_interval_tree_foreach(vma, &mapping->i_mmap, index, index) {
+ unsigned long address;
+
+ cond_resched();
+
+ if (!(vma->vm_flags & VM_SHARED))
+ continue;
+
+ address = pgoff_address(index, vma);
+ changed = false;
+ if (follow_pte(vma->vm_mm, address, &ptep, &ptl))
+ continue;
+ if (pfn != pte_pfn(*ptep))
+ goto unlock;
+ if (!pte_dirty(*ptep) && !pte_write(*ptep))
+ goto unlock;
+
+ flush_cache_page(vma, address, pfn);
+ pte = ptep_clear_flush(vma, address, ptep);
+ pte = pte_wrprotect(pte);
+ pte = pte_mkclean(pte);
+ set_pte_at(vma->vm_mm, address, ptep, pte);
+ changed = true;
+unlock:
+ pte_unmap_unlock(ptep, ptl);
+
+ if (changed)
+ mmu_notifier_invalidate_page(vma->vm_mm, address);
+ }
+ i_mmap_unlock_read(mapping);
+}
+
static int dax_writeback_one(struct block_device *bdev,
struct address_space *mapping, pgoff_t index, void *entry)
{
@@ -688,7 +742,17 @@ static int dax_writeback_one(struct block_device *bdev,
goto unmap;
}
+ dax_mapping_entry_mkclean(mapping, index, pfn_t_to_pfn(dax.pfn));
wb_cache_pmem(dax.addr, dax.size);
+ /*
+ * After we have flushed the cache, we can clear the dirty tag. There
+ * cannot be new dirty data in the pfn after the flush has completed as
+ * the pfn mappings are writeprotected and fault waits for mapping
+ * entry lock.
+ */
+ spin_lock_irq(&mapping->tree_lock);
+ radix_tree_tag_clear(page_tree, index, PAGECACHE_TAG_DIRTY);
+ spin_unlock_irq(&mapping->tree_lock);
unmap:
dax_unmap_atomic(bdev, &dax);
put_locked_mapping_entry(mapping, index, entry);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-04 4:24 ` Jan Kara
@ 2016-11-15 21:50 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 21:50 UTC (permalink / raw)
To: Jan Kara, Peter Zijlstra
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> Currently we have two different structures for passing fault information
> around - struct vm_fault and struct fault_env. DAX will need more
> information in struct vm_fault to handle its faults so the content of
> that structure would become event closer to fault_env. Furthermore it
> would need to generate struct fault_env to be able to call some of the
> generic functions. So at this point I don't think there's much use in
> keeping these two structures separate. Just embed into struct vm_fault
> all that is needed to use it for both purposes.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
I'm not necessary dislike this, but I remember Peter had objections before
when I proposed something similar.
Peter?
> ---
> Documentation/filesystems/Locking | 2 +-
> fs/userfaultfd.c | 22 +-
> include/linux/huge_mm.h | 10 +-
> include/linux/mm.h | 28 +-
> include/linux/userfaultfd_k.h | 4 +-
> mm/filemap.c | 14 +-
> mm/huge_memory.c | 173 ++++++------
> mm/internal.h | 2 +-
> mm/khugepaged.c | 20 +-
> mm/memory.c | 549 +++++++++++++++++++-------------------
> mm/nommu.c | 2 +-
> 11 files changed, 414 insertions(+), 412 deletions(-)
>
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index 14cdc101d165..ac3d080eabaa 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -557,7 +557,7 @@ till "end_pgoff". ->map_pages() is called with page table locked and must
> not block. If it's not possible to reach a page without blocking,
> filesystem should skip it. Filesystem should use do_set_pte() to setup
> page table entry. Pointer to entry associated with the page is passed in
> -"pte" field in fault_env structure. Pointers to entries for other offsets
> +"pte" field in vm_fault structure. Pointers to entries for other offsets
> should be calculated relative to "pte".
>
> ->page_mkwrite() is called when a previously read-only pte is
> diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
> index 85959d8324df..d96e2f30084b 100644
> --- a/fs/userfaultfd.c
> +++ b/fs/userfaultfd.c
> @@ -257,9 +257,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
> * fatal_signal_pending()s, and the mmap_sem must be released before
> * returning it.
> */
> -int handle_userfault(struct fault_env *fe, unsigned long reason)
> +int handle_userfault(struct vm_fault *vmf, unsigned long reason)
> {
> - struct mm_struct *mm = fe->vma->vm_mm;
> + struct mm_struct *mm = vmf->vma->vm_mm;
> struct userfaultfd_ctx *ctx;
> struct userfaultfd_wait_queue uwq;
> int ret;
> @@ -268,7 +268,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
>
> ret = VM_FAULT_SIGBUS;
> - ctx = fe->vma->vm_userfaultfd_ctx.ctx;
> + ctx = vmf->vma->vm_userfaultfd_ctx.ctx;
> if (!ctx)
> goto out;
>
> @@ -301,17 +301,18 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> * without first stopping userland access to the memory. For
> * VM_UFFD_MISSING userfaults this is enough for now.
> */
> - if (unlikely(!(fe->flags & FAULT_FLAG_ALLOW_RETRY))) {
> + if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
> /*
> * Validate the invariant that nowait must allow retry
> * to be sure not to return SIGBUS erroneously on
> * nowait invocations.
> */
> - BUG_ON(fe->flags & FAULT_FLAG_RETRY_NOWAIT);
> + BUG_ON(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
> #ifdef CONFIG_DEBUG_VM
> if (printk_ratelimit()) {
> printk(KERN_WARNING
> - "FAULT_FLAG_ALLOW_RETRY missing %x\n", fe->flags);
> + "FAULT_FLAG_ALLOW_RETRY missing %x\n",
> + vmf->flags);
> dump_stack();
> }
> #endif
> @@ -323,7 +324,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> * and wait.
> */
> ret = VM_FAULT_RETRY;
> - if (fe->flags & FAULT_FLAG_RETRY_NOWAIT)
> + if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
> goto out;
>
> /* take the reference before dropping the mmap_sem */
> @@ -331,11 +332,11 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
>
> init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
> uwq.wq.private = current;
> - uwq.msg = userfault_msg(fe->address, fe->flags, reason);
> + uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
> uwq.ctx = ctx;
>
> return_to_userland =
> - (fe->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
> + (vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
> (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
>
> spin_lock(&ctx->fault_pending_wqh.lock);
> @@ -353,7 +354,8 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> TASK_KILLABLE);
> spin_unlock(&ctx->fault_pending_wqh.lock);
>
> - must_wait = userfaultfd_must_wait(ctx, fe->address, fe->flags, reason);
> + must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
> + reason);
> up_read(&mm->mmap_sem);
>
> if (likely(must_wait && !ACCESS_ONCE(ctx->released) &&
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index 9b9f65d99873..3237a39f94a4 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -1,12 +1,12 @@
> #ifndef _LINUX_HUGE_MM_H
> #define _LINUX_HUGE_MM_H
>
> -extern int do_huge_pmd_anonymous_page(struct fault_env *fe);
> +extern int do_huge_pmd_anonymous_page(struct vm_fault *vmf);
> extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
> struct vm_area_struct *vma);
> -extern void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd);
> -extern int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd);
> +extern void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
> +extern int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd);
> extern struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
> unsigned long addr,
> pmd_t *pmd,
> @@ -142,7 +142,7 @@ static inline int hpage_nr_pages(struct page *page)
> return 1;
> }
>
> -extern int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd);
> +extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
>
> extern struct page *huge_zero_page;
>
> @@ -210,7 +210,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
> return NULL;
> }
>
> -static inline int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd)
> +static inline int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> return 0;
> }
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a92c8d73aeaf..657eb69eb87e 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> * pgoff should be used in favour of virtual_address, if possible.
> */
> struct vm_fault {
> + struct vm_area_struct *vma; /* Target VMA */
> unsigned int flags; /* FAULT_FLAG_xxx flags */
> gfp_t gfp_mask; /* gfp mask to be used for allocations */
> pgoff_t pgoff; /* Logical page offset based on vma */
> - void __user *virtual_address; /* Faulting virtual address */
> + unsigned long address; /* Faulting virtual address */
> + void __user *virtual_address; /* Faulting virtual address masked by
> + * PAGE_MASK */
> + pmd_t *pmd; /* Pointer to pmd entry matching
> + * the 'address'
> + */
>
> struct page *cow_page; /* Handler may choose to COW */
> struct page *page; /* ->fault handlers should return a
> @@ -309,19 +315,7 @@ struct vm_fault {
> * VM_FAULT_DAX_LOCKED and fill in
> * entry here.
> */
> -};
> -
> -/*
> - * Page fault context: passes though page fault handler instead of endless list
> - * of function arguments.
> - */
> -struct fault_env {
> - struct vm_area_struct *vma; /* Target VMA */
> - unsigned long address; /* Faulting virtual address */
> - unsigned int flags; /* FAULT_FLAG_xxx flags */
> - pmd_t *pmd; /* Pointer to pmd entry matching
> - * the 'address'
> - */
> + /* These three entries are valid only while holding ptl lock */
> pte_t *pte; /* Pointer to pte entry matching
> * the 'address'. NULL if the page
> * table hasn't been allocated.
> @@ -351,7 +345,7 @@ struct vm_operations_struct {
> int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf);
> int (*pmd_fault)(struct vm_area_struct *, unsigned long address,
> pmd_t *, unsigned int flags);
> - void (*map_pages)(struct fault_env *fe,
> + void (*map_pages)(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff);
>
> /* notification that a previously read-only page is about to become
> @@ -625,7 +619,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
> return pte;
> }
>
> -int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> +int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page);
> #endif
>
> @@ -2097,7 +2091,7 @@ extern void truncate_inode_pages_final(struct address_space *);
>
> /* generic vm_area_ops exported for stackable file systems */
> extern int filemap_fault(struct vm_area_struct *, struct vm_fault *);
> -extern void filemap_map_pages(struct fault_env *fe,
> +extern void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff);
> extern int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf);
>
> diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
> index dd66a952e8cd..11b92b047a1e 100644
> --- a/include/linux/userfaultfd_k.h
> +++ b/include/linux/userfaultfd_k.h
> @@ -27,7 +27,7 @@
> #define UFFD_SHARED_FCNTL_FLAGS (O_CLOEXEC | O_NONBLOCK)
> #define UFFD_FLAGS_SET (EFD_SHARED_FCNTL_FLAGS)
>
> -extern int handle_userfault(struct fault_env *fe, unsigned long reason);
> +extern int handle_userfault(struct vm_fault *vmf, unsigned long reason);
>
> extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,
> unsigned long src_start, unsigned long len);
> @@ -55,7 +55,7 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
> #else /* CONFIG_USERFAULTFD */
>
> /* mm helpers */
> -static inline int handle_userfault(struct fault_env *fe, unsigned long reason)
> +static inline int handle_userfault(struct vm_fault *vmf, unsigned long reason)
> {
> return VM_FAULT_SIGBUS;
> }
> diff --git a/mm/filemap.c b/mm/filemap.c
> index db26ebc6c62f..1426fb1a99b3 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -2209,12 +2209,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> }
> EXPORT_SYMBOL(filemap_fault);
>
> -void filemap_map_pages(struct fault_env *fe,
> +void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff)
> {
> struct radix_tree_iter iter;
> void **slot;
> - struct file *file = fe->vma->vm_file;
> + struct file *file = vmf->vma->vm_file;
> struct address_space *mapping = file->f_mapping;
> pgoff_t last_pgoff = start_pgoff;
> loff_t size;
> @@ -2270,11 +2270,11 @@ void filemap_map_pages(struct fault_env *fe,
> if (file->f_ra.mmap_miss > 0)
> file->f_ra.mmap_miss--;
>
> - fe->address += (iter.index - last_pgoff) << PAGE_SHIFT;
> - if (fe->pte)
> - fe->pte += iter.index - last_pgoff;
> + vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
> + if (vmf->pte)
> + vmf->pte += iter.index - last_pgoff;
> last_pgoff = iter.index;
> - if (alloc_set_pte(fe, NULL, page))
> + if (alloc_set_pte(vmf, NULL, page))
> goto unlock;
> unlock_page(page);
> goto next;
> @@ -2284,7 +2284,7 @@ void filemap_map_pages(struct fault_env *fe,
> put_page(page);
> next:
> /* Huge page is mapped? No need to proceed. */
> - if (pmd_trans_huge(*fe->pmd))
> + if (pmd_trans_huge(*vmf->pmd))
> break;
> if (iter.index == end_pgoff)
> break;
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index cdcd25cb30fe..e286b09f9d24 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -532,13 +532,13 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
> }
> EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
>
> -static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> +static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page,
> gfp_t gfp)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mem_cgroup *memcg;
> pgtable_t pgtable;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
>
> VM_BUG_ON_PAGE(!PageCompound(page), page);
>
> @@ -563,9 +563,9 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> */
> __SetPageUptodate(page);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd))) {
> - spin_unlock(fe->ptl);
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd))) {
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, true);
> put_page(page);
> pte_free(vma->vm_mm, pgtable);
> @@ -576,11 +576,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> if (userfaultfd_missing(vma)) {
> int ret;
>
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, true);
> put_page(page);
> pte_free(vma->vm_mm, pgtable);
> - ret = handle_userfault(fe, VM_UFFD_MISSING);
> + ret = handle_userfault(vmf, VM_UFFD_MISSING);
> VM_BUG_ON(ret & VM_FAULT_FALLBACK);
> return ret;
> }
> @@ -590,11 +590,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> page_add_new_anon_rmap(page, vma, haddr, true);
> mem_cgroup_commit_charge(page, memcg, false, true);
> lru_cache_add_active_or_unevictable(page, vma);
> - pgtable_trans_huge_deposit(vma->vm_mm, fe->pmd, pgtable);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> + pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
> add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
> atomic_long_inc(&vma->vm_mm->nr_ptes);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> count_vm_event(THP_FAULT_ALLOC);
> }
>
> @@ -641,12 +641,12 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
> return true;
> }
>
> -int do_huge_pmd_anonymous_page(struct fault_env *fe)
> +int do_huge_pmd_anonymous_page(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> gfp_t gfp;
> struct page *page;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
>
> if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
> return VM_FAULT_FALLBACK;
> @@ -654,7 +654,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> return VM_FAULT_OOM;
> if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
> return VM_FAULT_OOM;
> - if (!(fe->flags & FAULT_FLAG_WRITE) &&
> + if (!(vmf->flags & FAULT_FLAG_WRITE) &&
> !mm_forbids_zeropage(vma->vm_mm) &&
> transparent_hugepage_use_zero_page()) {
> pgtable_t pgtable;
> @@ -670,22 +670,22 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> count_vm_event(THP_FAULT_FALLBACK);
> return VM_FAULT_FALLBACK;
> }
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> ret = 0;
> set = false;
> - if (pmd_none(*fe->pmd)) {
> + if (pmd_none(*vmf->pmd)) {
> if (userfaultfd_missing(vma)) {
> - spin_unlock(fe->ptl);
> - ret = handle_userfault(fe, VM_UFFD_MISSING);
> + spin_unlock(vmf->ptl);
> + ret = handle_userfault(vmf, VM_UFFD_MISSING);
> VM_BUG_ON(ret & VM_FAULT_FALLBACK);
> } else {
> set_huge_zero_page(pgtable, vma->vm_mm, vma,
> - haddr, fe->pmd, zero_page);
> - spin_unlock(fe->ptl);
> + haddr, vmf->pmd, zero_page);
> + spin_unlock(vmf->ptl);
> set = true;
> }
> } else
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> if (!set)
> pte_free(vma->vm_mm, pgtable);
> return ret;
> @@ -697,7 +697,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> return VM_FAULT_FALLBACK;
> }
> prep_transhuge_page(page);
> - return __do_huge_pmd_anonymous_page(fe, page, gfp);
> + return __do_huge_pmd_anonymous_page(vmf, page, gfp);
> }
>
> static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
> @@ -868,30 +868,30 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> return ret;
> }
>
> -void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd)
> +void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> pmd_t entry;
> unsigned long haddr;
>
> - fe->ptl = pmd_lock(fe->vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto unlock;
>
> entry = pmd_mkyoung(orig_pmd);
> - haddr = fe->address & HPAGE_PMD_MASK;
> - if (pmdp_set_access_flags(fe->vma, haddr, fe->pmd, entry,
> - fe->flags & FAULT_FLAG_WRITE))
> - update_mmu_cache_pmd(fe->vma, fe->address, fe->pmd);
> + haddr = vmf->address & HPAGE_PMD_MASK;
> + if (pmdp_set_access_flags(vmf->vma, haddr, vmf->pmd, entry,
> + vmf->flags & FAULT_FLAG_WRITE))
> + update_mmu_cache_pmd(vmf->vma, vmf->address, vmf->pmd);
>
> unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> }
>
> -static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> +static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd,
> struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + struct vm_area_struct *vma = vmf->vma;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> struct mem_cgroup *memcg;
> pgtable_t pgtable;
> pmd_t _pmd;
> @@ -910,7 +910,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> for (i = 0; i < HPAGE_PMD_NR; i++) {
> pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE |
> __GFP_OTHER_NODE, vma,
> - fe->address, page_to_nid(page));
> + vmf->address, page_to_nid(page));
> if (unlikely(!pages[i] ||
> mem_cgroup_try_charge(pages[i], vma->vm_mm,
> GFP_KERNEL, &memcg, false))) {
> @@ -941,15 +941,15 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> mmun_end = haddr + HPAGE_PMD_SIZE;
> mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto out_free_pages;
> VM_BUG_ON_PAGE(!PageHead(page), page);
>
> - pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
> + pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
> /* leave pmd empty until pte is filled */
>
> - pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, fe->pmd);
> + pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, vmf->pmd);
> pmd_populate(vma->vm_mm, &_pmd, pgtable);
>
> for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
> @@ -958,20 +958,20 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> memcg = (void *)page_private(pages[i]);
> set_page_private(pages[i], 0);
> - page_add_new_anon_rmap(pages[i], fe->vma, haddr, false);
> + page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false);
> mem_cgroup_commit_charge(pages[i], memcg, false, false);
> lru_cache_add_active_or_unevictable(pages[i], vma);
> - fe->pte = pte_offset_map(&_pmd, haddr);
> - VM_BUG_ON(!pte_none(*fe->pte));
> - set_pte_at(vma->vm_mm, haddr, fe->pte, entry);
> - pte_unmap(fe->pte);
> + vmf->pte = pte_offset_map(&_pmd, haddr);
> + VM_BUG_ON(!pte_none(*vmf->pte));
> + set_pte_at(vma->vm_mm, haddr, vmf->pte, entry);
> + pte_unmap(vmf->pte);
> }
> kfree(pages);
>
> smp_wmb(); /* make pte visible before pmd */
> - pmd_populate(vma->vm_mm, fe->pmd, pgtable);
> + pmd_populate(vma->vm_mm, vmf->pmd, pgtable);
> page_remove_rmap(page, true);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
>
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
>
> @@ -982,7 +982,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> return ret;
>
> out_free_pages:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
> for (i = 0; i < HPAGE_PMD_NR; i++) {
> memcg = (void *)page_private(pages[i]);
> @@ -994,23 +994,23 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> goto out;
> }
>
> -int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> +int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page = NULL, *new_page;
> struct mem_cgroup *memcg;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> unsigned long mmun_start; /* For mmu_notifiers */
> unsigned long mmun_end; /* For mmu_notifiers */
> gfp_t huge_gfp; /* for allocation and charge */
> int ret = 0;
>
> - fe->ptl = pmd_lockptr(vma->vm_mm, fe->pmd);
> + vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd);
> VM_BUG_ON_VMA(!vma->anon_vma, vma);
> if (is_huge_zero_pmd(orig_pmd))
> goto alloc;
> - spin_lock(fe->ptl);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + spin_lock(vmf->ptl);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto out_unlock;
>
> page = pmd_page(orig_pmd);
> @@ -1023,13 +1023,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> pmd_t entry;
> entry = pmd_mkyoung(orig_pmd);
> entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
> - if (pmdp_set_access_flags(vma, haddr, fe->pmd, entry, 1))
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1))
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> ret |= VM_FAULT_WRITE;
> goto out_unlock;
> }
> get_page(page);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> alloc:
> if (transparent_hugepage_enabled(vma) &&
> !transparent_hugepage_debug_cow()) {
> @@ -1042,12 +1042,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> prep_transhuge_page(new_page);
> } else {
> if (!page) {
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> ret |= VM_FAULT_FALLBACK;
> } else {
> - ret = do_huge_pmd_wp_page_fallback(fe, orig_pmd, page);
> + ret = do_huge_pmd_wp_page_fallback(vmf, orig_pmd, page);
> if (ret & VM_FAULT_OOM) {
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> ret |= VM_FAULT_FALLBACK;
> }
> put_page(page);
> @@ -1059,7 +1059,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
> huge_gfp, &memcg, true))) {
> put_page(new_page);
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> if (page)
> put_page(page);
> ret |= VM_FAULT_FALLBACK;
> @@ -1079,11 +1079,11 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> mmun_end = haddr + HPAGE_PMD_SIZE;
> mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
>
> - spin_lock(fe->ptl);
> + spin_lock(vmf->ptl);
> if (page)
> put_page(page);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd))) {
> - spin_unlock(fe->ptl);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) {
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(new_page, memcg, true);
> put_page(new_page);
> goto out_mn;
> @@ -1091,12 +1091,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> pmd_t entry;
> entry = mk_huge_pmd(new_page, vma->vm_page_prot);
> entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
> - pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
> + pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
> page_add_new_anon_rmap(new_page, vma, haddr, true);
> mem_cgroup_commit_charge(new_page, memcg, false, true);
> lru_cache_add_active_or_unevictable(new_page, vma);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> if (!page) {
> add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
> } else {
> @@ -1106,13 +1106,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> }
> ret |= VM_FAULT_WRITE;
> }
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> out_mn:
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
> out:
> return ret;
> out_unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> return ret;
> }
>
> @@ -1185,12 +1185,12 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
> }
>
> /* NUMA hinting page fault entry point for trans huge pmds */
> -int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> +int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct anon_vma *anon_vma = NULL;
> struct page *page;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> int page_nid = -1, this_nid = numa_node_id();
> int target_nid, last_cpupid = -1;
> bool page_locked;
> @@ -1198,8 +1198,8 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> bool was_writable;
> int flags = 0;
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(pmd, *fe->pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(pmd, *vmf->pmd)))
> goto out_unlock;
>
> /*
> @@ -1207,9 +1207,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * without disrupting NUMA hinting information. Do not relock and
> * check_same as the page may no longer be mapped.
> */
> - if (unlikely(pmd_trans_migrating(*fe->pmd))) {
> - page = pmd_page(*fe->pmd);
> - spin_unlock(fe->ptl);
> + if (unlikely(pmd_trans_migrating(*vmf->pmd))) {
> + page = pmd_page(*vmf->pmd);
> + spin_unlock(vmf->ptl);
> wait_on_page_locked(page);
> goto out;
> }
> @@ -1242,7 +1242,7 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
>
> /* Migration could have started since the pmd_trans_migrating check */
> if (!page_locked) {
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> wait_on_page_locked(page);
> page_nid = -1;
> goto out;
> @@ -1253,12 +1253,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * to serialises splits
> */
> get_page(page);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> anon_vma = page_lock_anon_vma_read(page);
>
> /* Confirm the PMD did not change while page_table_lock was released */
> - spin_lock(fe->ptl);
> - if (unlikely(!pmd_same(pmd, *fe->pmd))) {
> + spin_lock(vmf->ptl);
> + if (unlikely(!pmd_same(pmd, *vmf->pmd))) {
> unlock_page(page);
> put_page(page);
> page_nid = -1;
> @@ -1276,9 +1276,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * Migrate the THP to the requested node, returns with page unlocked
> * and access rights restored.
> */
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma,
> - fe->pmd, pmd, fe->address, page, target_nid);
> + vmf->pmd, pmd, vmf->address, page, target_nid);
> if (migrated) {
> flags |= TNF_MIGRATED;
> page_nid = target_nid;
> @@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> pmd = pmd_mkyoung(pmd);
> if (was_writable)
> pmd = pmd_mkwrite(pmd);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, pmd);
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> unlock_page(page);
> out_unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
>
> out:
> if (anon_vma)
> page_unlock_anon_vma_read(anon_vma);
>
> if (page_nid != -1)
> - task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, fe->flags);
> + task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
> + vmf->flags);
>
> return 0;
> }
> diff --git a/mm/internal.h b/mm/internal.h
> index 537ac9951f5f..093b1eacc91b 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -36,7 +36,7 @@
> /* Do not use these with a slab allocator */
> #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
>
> -int do_swap_page(struct fault_env *fe, pte_t orig_pte);
> +int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
>
> void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> unsigned long floor, unsigned long ceiling);
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 728d7790dc2d..f88b2d3810a7 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -875,7 +875,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> {
> pte_t pteval;
> int swapped_in = 0, ret = 0;
> - struct fault_env fe = {
> + struct vm_fault vmf = {
> .vma = vma,
> .address = address,
> .flags = FAULT_FLAG_ALLOW_RETRY,
> @@ -887,19 +887,19 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
> return false;
> }
> - fe.pte = pte_offset_map(pmd, address);
> - for (; fe.address < address + HPAGE_PMD_NR*PAGE_SIZE;
> - fe.pte++, fe.address += PAGE_SIZE) {
> - pteval = *fe.pte;
> + vmf.pte = pte_offset_map(pmd, address);
> + for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
> + vmf.pte++, vmf.address += PAGE_SIZE) {
> + pteval = *vmf.pte;
> if (!is_swap_pte(pteval))
> continue;
> swapped_in++;
> - ret = do_swap_page(&fe, pteval);
> + ret = do_swap_page(&vmf, pteval);
>
> /* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
> if (ret & VM_FAULT_RETRY) {
> down_read(&mm->mmap_sem);
> - if (hugepage_vma_revalidate(mm, address, &fe.vma)) {
> + if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
> /* vma is no longer available, don't continue to swapin */
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
> return false;
> @@ -913,10 +913,10 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> return false;
> }
> /* pte is unmapped now, we need to map it */
> - fe.pte = pte_offset_map(pmd, fe.address);
> + vmf.pte = pte_offset_map(pmd, vmf.address);
> }
> - fe.pte--;
> - pte_unmap(fe.pte);
> + vmf.pte--;
> + pte_unmap(vmf.pte);
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
> return true;
> }
> diff --git a/mm/memory.c b/mm/memory.c
> index e18c57bdc75c..fad45cd59ba7 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2074,11 +2074,11 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
> * case, all we need to do here is to mark the page as writable and update
> * any related book-keeping.
> */
> -static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> +static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
> struct page *page, int page_mkwrite, int dirty_shared)
> - __releases(fe->ptl)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> pte_t entry;
> /*
> * Clear the pages cpupid information as the existing
> @@ -2088,12 +2088,12 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> if (page)
> page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
>
> - flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
> + flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
> entry = pte_mkyoung(orig_pte);
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> - if (ptep_set_access_flags(vma, fe->address, fe->pte, entry, 1))
> - update_mmu_cache(vma, fe->address, fe->pte);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>
> if (dirty_shared) {
> struct address_space *mapping;
> @@ -2139,15 +2139,15 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> * held to the old page, as well as updating the rmap.
> * - In any case, unlock the PTL and drop the reference we took to the old page.
> */
> -static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> +static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
> struct page *old_page)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mm_struct *mm = vma->vm_mm;
> struct page *new_page = NULL;
> pte_t entry;
> int page_copied = 0;
> - const unsigned long mmun_start = fe->address & PAGE_MASK;
> + const unsigned long mmun_start = vmf->address & PAGE_MASK;
> const unsigned long mmun_end = mmun_start + PAGE_SIZE;
> struct mem_cgroup *memcg;
>
> @@ -2155,15 +2155,16 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> goto oom;
>
> if (is_zero_pfn(pte_pfn(orig_pte))) {
> - new_page = alloc_zeroed_user_highpage_movable(vma, fe->address);
> + new_page = alloc_zeroed_user_highpage_movable(vma,
> + vmf->address);
> if (!new_page)
> goto oom;
> } else {
> new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
> - fe->address);
> + vmf->address);
> if (!new_page)
> goto oom;
> - cow_user_page(new_page, old_page, fe->address, vma);
> + cow_user_page(new_page, old_page, vmf->address, vma);
> }
>
> if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false))
> @@ -2176,8 +2177,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> /*
> * Re-check the pte - we dropped the lock
> */
> - fe->pte = pte_offset_map_lock(mm, fe->pmd, fe->address, &fe->ptl);
> - if (likely(pte_same(*fe->pte, orig_pte))) {
> + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
> + if (likely(pte_same(*vmf->pte, orig_pte))) {
> if (old_page) {
> if (!PageAnon(old_page)) {
> dec_mm_counter_fast(mm,
> @@ -2187,7 +2188,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> } else {
> inc_mm_counter_fast(mm, MM_ANONPAGES);
> }
> - flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
> + flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
> entry = mk_pte(new_page, vma->vm_page_prot);
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> /*
> @@ -2196,8 +2197,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * seen in the presence of one thread doing SMC and another
> * thread doing COW.
> */
> - ptep_clear_flush_notify(vma, fe->address, fe->pte);
> - page_add_new_anon_rmap(new_page, vma, fe->address, false);
> + ptep_clear_flush_notify(vma, vmf->address, vmf->pte);
> + page_add_new_anon_rmap(new_page, vma, vmf->address, false);
> mem_cgroup_commit_charge(new_page, memcg, false, false);
> lru_cache_add_active_or_unevictable(new_page, vma);
> /*
> @@ -2205,8 +2206,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * mmu page tables (such as kvm shadow page tables), we want the
> * new page to be mapped directly into the secondary page table.
> */
> - set_pte_at_notify(mm, fe->address, fe->pte, entry);
> - update_mmu_cache(vma, fe->address, fe->pte);
> + set_pte_at_notify(mm, vmf->address, vmf->pte, entry);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> if (old_page) {
> /*
> * Only after switching the pte to the new page may
> @@ -2243,7 +2244,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> if (new_page)
> put_page(new_page);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
> if (old_page) {
> /*
> @@ -2271,43 +2272,43 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
> * mapping
> */
> -static int wp_pfn_shared(struct fault_env *fe, pte_t orig_pte)
> +static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
>
> if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
> - struct vm_fault vmf = {
> + struct vm_fault vmf2 = {
> .page = NULL,
> - .pgoff = linear_page_index(vma, fe->address),
> + .pgoff = linear_page_index(vma, vmf->address),
> .virtual_address =
> - (void __user *)(fe->address & PAGE_MASK),
> + (void __user *)(vmf->address & PAGE_MASK),
> .flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
> };
> int ret;
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - ret = vma->vm_ops->pfn_mkwrite(vma, &vmf);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
> if (ret & VM_FAULT_ERROR)
> return ret;
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> /*
> * We might have raced with another page fault while we
> * released the pte_offset_map_lock.
> */
> - if (!pte_same(*fe->pte, orig_pte)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
> }
> - return wp_page_reuse(fe, orig_pte, NULL, 0, 0);
> + return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
> }
>
> -static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> +static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
> struct page *old_page)
> - __releases(fe->ptl)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> int page_mkwrite = 0;
>
> get_page(old_page);
> @@ -2315,8 +2316,8 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
> int tmp;
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - tmp = do_page_mkwrite(vma, old_page, fe->address);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + tmp = do_page_mkwrite(vma, old_page, vmf->address);
> if (unlikely(!tmp || (tmp &
> (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
> put_page(old_page);
> @@ -2328,18 +2329,18 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> * they did, we just return, as we can count on the
> * MMU to tell us if they didn't also make it writable.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_same(*fe->pte, orig_pte)) {
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> unlock_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> put_page(old_page);
> return 0;
> }
> page_mkwrite = 1;
> }
>
> - return wp_page_reuse(fe, orig_pte, old_page, page_mkwrite, 1);
> + return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
> }
>
> /*
> @@ -2360,13 +2361,13 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> * but allow concurrent faults), with pte both mapped and locked.
> * We return with mmap_sem still held, but pte unmapped and unlocked.
> */
> -static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> - __releases(fe->ptl)
> +static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *old_page;
>
> - old_page = vm_normal_page(vma, fe->address, orig_pte);
> + old_page = vm_normal_page(vma, vmf->address, orig_pte);
> if (!old_page) {
> /*
> * VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
> @@ -2377,10 +2378,10 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> */
> if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
> (VM_WRITE|VM_SHARED))
> - return wp_pfn_shared(fe, orig_pte);
> + return wp_pfn_shared(vmf, orig_pte);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return wp_page_copy(fe, orig_pte, old_page);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return wp_page_copy(vmf, orig_pte, old_page);
> }
>
> /*
> @@ -2391,13 +2392,13 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> int total_mapcount;
> if (!trylock_page(old_page)) {
> get_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> lock_page(old_page);
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
> - fe->address, &fe->ptl);
> - if (!pte_same(*fe->pte, orig_pte)) {
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> unlock_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> put_page(old_page);
> return 0;
> }
> @@ -2415,12 +2416,12 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> page_move_anon_rmap(old_page, vma);
> }
> unlock_page(old_page);
> - return wp_page_reuse(fe, orig_pte, old_page, 0, 0);
> + return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
> }
> unlock_page(old_page);
> } else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
> (VM_WRITE|VM_SHARED))) {
> - return wp_page_shared(fe, orig_pte, old_page);
> + return wp_page_shared(vmf, orig_pte, old_page);
> }
>
> /*
> @@ -2428,8 +2429,8 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> */
> get_page(old_page);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return wp_page_copy(fe, orig_pte, old_page);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return wp_page_copy(vmf, orig_pte, old_page);
> }
>
> static void unmap_mapping_range_vma(struct vm_area_struct *vma,
> @@ -2517,9 +2518,9 @@ EXPORT_SYMBOL(unmap_mapping_range);
> * We return with the mmap_sem locked or unlocked in the same cases
> * as does filemap_fault().
> */
> -int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> +int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page, *swapcache;
> struct mem_cgroup *memcg;
> swp_entry_t entry;
> @@ -2528,17 +2529,18 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> int exclusive = 0;
> int ret = 0;
>
> - if (!pte_unmap_same(vma->vm_mm, fe->pmd, fe->pte, orig_pte))
> + if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
> goto out;
>
> entry = pte_to_swp_entry(orig_pte);
> if (unlikely(non_swap_entry(entry))) {
> if (is_migration_entry(entry)) {
> - migration_entry_wait(vma->vm_mm, fe->pmd, fe->address);
> + migration_entry_wait(vma->vm_mm, vmf->pmd,
> + vmf->address);
> } else if (is_hwpoison_entry(entry)) {
> ret = VM_FAULT_HWPOISON;
> } else {
> - print_bad_pte(vma, fe->address, orig_pte, NULL);
> + print_bad_pte(vma, vmf->address, orig_pte, NULL);
> ret = VM_FAULT_SIGBUS;
> }
> goto out;
> @@ -2546,16 +2548,16 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> delayacct_set_flag(DELAYACCT_PF_SWAPIN);
> page = lookup_swap_cache(entry);
> if (!page) {
> - page = swapin_readahead(entry,
> - GFP_HIGHUSER_MOVABLE, vma, fe->address);
> + page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vma,
> + vmf->address);
> if (!page) {
> /*
> * Back out if somebody else faulted in this pte
> * while we released the pte lock.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
> - fe->address, &fe->ptl);
> - if (likely(pte_same(*fe->pte, orig_pte)))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (likely(pte_same(*vmf->pte, orig_pte)))
> ret = VM_FAULT_OOM;
> delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
> goto unlock;
> @@ -2577,7 +2579,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> }
>
> swapcache = page;
> - locked = lock_page_or_retry(page, vma->vm_mm, fe->flags);
> + locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
>
> delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
> if (!locked) {
> @@ -2594,7 +2596,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> if (unlikely(!PageSwapCache(page) || page_private(page) != entry.val))
> goto out_page;
>
> - page = ksm_might_need_to_copy(page, vma, fe->address);
> + page = ksm_might_need_to_copy(page, vma, vmf->address);
> if (unlikely(!page)) {
> ret = VM_FAULT_OOM;
> page = swapcache;
> @@ -2610,9 +2612,9 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> /*
> * Back out if somebody else already faulted in this pte.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, orig_pte)))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, orig_pte)))
> goto out_nomap;
>
> if (unlikely(!PageUptodate(page))) {
> @@ -2633,22 +2635,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
> pte = mk_pte(page, vma->vm_page_prot);
> - if ((fe->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
> + if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
> pte = maybe_mkwrite(pte_mkdirty(pte), vma);
> - fe->flags &= ~FAULT_FLAG_WRITE;
> + vmf->flags &= ~FAULT_FLAG_WRITE;
> ret |= VM_FAULT_WRITE;
> exclusive = RMAP_EXCLUSIVE;
> }
> flush_icache_page(vma, page);
> if (pte_swp_soft_dirty(orig_pte))
> pte = pte_mksoft_dirty(pte);
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
> if (page == swapcache) {
> - do_page_add_anon_rmap(page, vma, fe->address, exclusive);
> + do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
> mem_cgroup_commit_charge(page, memcg, true, false);
> activate_page(page);
> } else { /* ksm created a completely new copy */
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> }
> @@ -2671,22 +2673,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> put_page(swapcache);
> }
>
> - if (fe->flags & FAULT_FLAG_WRITE) {
> - ret |= do_wp_page(fe, pte);
> + if (vmf->flags & FAULT_FLAG_WRITE) {
> + ret |= do_wp_page(vmf, pte);
> if (ret & VM_FAULT_ERROR)
> ret &= VM_FAULT_ERROR;
> goto out;
> }
>
> /* No need to invalidate - it was non-present before */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out:
> return ret;
> out_nomap:
> mem_cgroup_cancel_charge(page, memcg, false);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out_page:
> unlock_page(page);
> out_release:
> @@ -2737,9 +2739,9 @@ static inline int check_stack_guard_page(struct vm_area_struct *vma, unsigned lo
> * but allow concurrent faults), and pte mapped but not yet locked.
> * We return with mmap_sem still held, but pte unmapped and unlocked.
> */
> -static int do_anonymous_page(struct fault_env *fe)
> +static int do_anonymous_page(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mem_cgroup *memcg;
> struct page *page;
> pte_t entry;
> @@ -2749,7 +2751,7 @@ static int do_anonymous_page(struct fault_env *fe)
> return VM_FAULT_SIGBUS;
>
> /* Check if we need to add a guard page to the stack */
> - if (check_stack_guard_page(vma, fe->address) < 0)
> + if (check_stack_guard_page(vma, vmf->address) < 0)
> return VM_FAULT_SIGSEGV;
>
> /*
> @@ -2762,26 +2764,26 @@ static int do_anonymous_page(struct fault_env *fe)
> *
> * Here we only have down_read(mmap_sem).
> */
> - if (pte_alloc(vma->vm_mm, fe->pmd, fe->address))
> + if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
> return VM_FAULT_OOM;
>
> /* See the comment in pte_alloc_one_map() */
> - if (unlikely(pmd_trans_unstable(fe->pmd)))
> + if (unlikely(pmd_trans_unstable(vmf->pmd)))
> return 0;
>
> /* Use the zero-page for reads */
> - if (!(fe->flags & FAULT_FLAG_WRITE) &&
> + if (!(vmf->flags & FAULT_FLAG_WRITE) &&
> !mm_forbids_zeropage(vma->vm_mm)) {
> - entry = pte_mkspecial(pfn_pte(my_zero_pfn(fe->address),
> + entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
> vma->vm_page_prot));
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_none(*fe->pte))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_none(*vmf->pte))
> goto unlock;
> /* Deliver the page fault to userland, check inside PT lock */
> if (userfaultfd_missing(vma)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return handle_userfault(fe, VM_UFFD_MISSING);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return handle_userfault(vmf, VM_UFFD_MISSING);
> }
> goto setpte;
> }
> @@ -2789,7 +2791,7 @@ static int do_anonymous_page(struct fault_env *fe)
> /* Allocate our own private page. */
> if (unlikely(anon_vma_prepare(vma)))
> goto oom;
> - page = alloc_zeroed_user_highpage_movable(vma, fe->address);
> + page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
> if (!page)
> goto oom;
>
> @@ -2807,30 +2809,30 @@ static int do_anonymous_page(struct fault_env *fe)
> if (vma->vm_flags & VM_WRITE)
> entry = pte_mkwrite(pte_mkdirty(entry));
>
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_none(*fe->pte))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + if (!pte_none(*vmf->pte))
> goto release;
>
> /* Deliver the page fault to userland, check inside PT lock */
> if (userfaultfd_missing(vma)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, false);
> put_page(page);
> - return handle_userfault(fe, VM_UFFD_MISSING);
> + return handle_userfault(vmf, VM_UFFD_MISSING);
> }
>
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> setpte:
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
>
> /* No need to invalidate - it was non-present before */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> release:
> mem_cgroup_cancel_charge(page, memcg, false);
> @@ -2847,62 +2849,62 @@ static int do_anonymous_page(struct fault_env *fe)
> * released depending on flags and vma->vm_ops->fault() return value.
> * See filemap_fault() and __lock_page_retry().
> */
> -static int __do_fault(struct fault_env *fe, pgoff_t pgoff,
> +static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
> struct page *cow_page, struct page **page, void **entry)
> {
> - struct vm_area_struct *vma = fe->vma;
> - struct vm_fault vmf;
> + struct vm_area_struct *vma = vmf->vma;
> + struct vm_fault vmf2;
> int ret;
>
> - vmf.virtual_address = (void __user *)(fe->address & PAGE_MASK);
> - vmf.pgoff = pgoff;
> - vmf.flags = fe->flags;
> - vmf.page = NULL;
> - vmf.gfp_mask = __get_fault_gfp_mask(vma);
> - vmf.cow_page = cow_page;
> + vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
> + vmf2.pgoff = pgoff;
> + vmf2.flags = vmf->flags;
> + vmf2.page = NULL;
> + vmf2.gfp_mask = __get_fault_gfp_mask(vma);
> + vmf2.cow_page = cow_page;
>
> - ret = vma->vm_ops->fault(vma, &vmf);
> + ret = vma->vm_ops->fault(vma, &vmf2);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
> if (ret & VM_FAULT_DAX_LOCKED) {
> - *entry = vmf.entry;
> + *entry = vmf2.entry;
> return ret;
> }
>
> - if (unlikely(PageHWPoison(vmf.page))) {
> + if (unlikely(PageHWPoison(vmf2.page))) {
> if (ret & VM_FAULT_LOCKED)
> - unlock_page(vmf.page);
> - put_page(vmf.page);
> + unlock_page(vmf2.page);
> + put_page(vmf2.page);
> return VM_FAULT_HWPOISON;
> }
>
> if (unlikely(!(ret & VM_FAULT_LOCKED)))
> - lock_page(vmf.page);
> + lock_page(vmf2.page);
> else
> - VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
> + VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
>
> - *page = vmf.page;
> + *page = vmf2.page;
> return ret;
> }
>
> -static int pte_alloc_one_map(struct fault_env *fe)
> +static int pte_alloc_one_map(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
>
> - if (!pmd_none(*fe->pmd))
> + if (!pmd_none(*vmf->pmd))
> goto map_pte;
> - if (fe->prealloc_pte) {
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd))) {
> - spin_unlock(fe->ptl);
> + if (vmf->prealloc_pte) {
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd))) {
> + spin_unlock(vmf->ptl);
> goto map_pte;
> }
>
> atomic_long_inc(&vma->vm_mm->nr_ptes);
> - pmd_populate(vma->vm_mm, fe->pmd, fe->prealloc_pte);
> - spin_unlock(fe->ptl);
> - fe->prealloc_pte = 0;
> - } else if (unlikely(pte_alloc(vma->vm_mm, fe->pmd, fe->address))) {
> + pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
> + spin_unlock(vmf->ptl);
> + vmf->prealloc_pte = 0;
> + } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
> return VM_FAULT_OOM;
> }
> map_pte:
> @@ -2917,11 +2919,11 @@ static int pte_alloc_one_map(struct fault_env *fe)
> * through an atomic read in C, which is what pmd_trans_unstable()
> * provides.
> */
> - if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
> + if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
> return VM_FAULT_NOPAGE;
>
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> return 0;
> }
>
> @@ -2939,11 +2941,11 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
> return true;
> }
>
> -static int do_set_pmd(struct fault_env *fe, struct page *page)
> +static int do_set_pmd(struct vm_fault *vmf, struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - bool write = fe->flags & FAULT_FLAG_WRITE;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + struct vm_area_struct *vma = vmf->vma;
> + bool write = vmf->flags & FAULT_FLAG_WRITE;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> pmd_t entry;
> int i, ret;
>
> @@ -2953,8 +2955,8 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> ret = VM_FAULT_FALLBACK;
> page = compound_head(page);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd)))
> goto out;
>
> for (i = 0; i < HPAGE_PMD_NR; i++)
> @@ -2967,19 +2969,19 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> add_mm_counter(vma->vm_mm, MM_FILEPAGES, HPAGE_PMD_NR);
> page_add_file_rmap(page, true);
>
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
>
> - update_mmu_cache_pmd(vma, haddr, fe->pmd);
> + update_mmu_cache_pmd(vma, haddr, vmf->pmd);
>
> /* fault is handled */
> ret = 0;
> count_vm_event(THP_FILE_MAPPED);
> out:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> return ret;
> }
> #else
> -static int do_set_pmd(struct fault_env *fe, struct page *page)
> +static int do_set_pmd(struct vm_fault *vmf, struct page *page)
> {
> BUILD_BUG();
> return 0;
> @@ -2990,41 +2992,42 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> * alloc_set_pte - setup new PTE entry for given page and add reverse page
> * mapping. If needed, the fucntion allocates page table or use pre-allocated.
> *
> - * @fe: fault environment
> + * @vmf: fault environment
> * @memcg: memcg to charge page (only for private mappings)
> * @page: page to map
> *
> - * Caller must take care of unlocking fe->ptl, if fe->pte is non-NULL on return.
> + * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
> + * return.
> *
> * Target users are page handler itself and implementations of
> * vm_ops->map_pages.
> */
> -int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> +int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - bool write = fe->flags & FAULT_FLAG_WRITE;
> + struct vm_area_struct *vma = vmf->vma;
> + bool write = vmf->flags & FAULT_FLAG_WRITE;
> pte_t entry;
> int ret;
>
> - if (pmd_none(*fe->pmd) && PageTransCompound(page) &&
> + if (pmd_none(*vmf->pmd) && PageTransCompound(page) &&
> IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE)) {
> /* THP on COW? */
> VM_BUG_ON_PAGE(memcg, page);
>
> - ret = do_set_pmd(fe, page);
> + ret = do_set_pmd(vmf, page);
> if (ret != VM_FAULT_FALLBACK)
> return ret;
> }
>
> - if (!fe->pte) {
> - ret = pte_alloc_one_map(fe);
> + if (!vmf->pte) {
> + ret = pte_alloc_one_map(vmf);
> if (ret)
> return ret;
> }
>
> /* Re-check under ptl */
> - if (unlikely(!pte_none(*fe->pte)))
> + if (unlikely(!pte_none(*vmf->pte)))
> return VM_FAULT_NOPAGE;
>
> flush_icache_page(vma, page);
> @@ -3034,17 +3037,17 @@ int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> /* copy-on-write page */
> if (write && !(vma->vm_flags & VM_SHARED)) {
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> } else {
> inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page));
> page_add_file_rmap(page, false);
> }
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
>
> /* no need to invalidate: a not-present page won't be cached */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
>
> return 0;
> }
> @@ -3113,17 +3116,17 @@ late_initcall(fault_around_debugfs);
> * fault_around_pages() value (and therefore to page order). This way it's
> * easier to guarantee that we don't cross page table boundaries.
> */
> -static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
> +static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
> {
> - unsigned long address = fe->address, nr_pages, mask;
> + unsigned long address = vmf->address, nr_pages, mask;
> pgoff_t end_pgoff;
> int off, ret = 0;
>
> nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
> mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
>
> - fe->address = max(address & mask, fe->vma->vm_start);
> - off = ((address - fe->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
> + vmf->address = max(address & mask, vmf->vma->vm_start);
> + off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
> start_pgoff -= off;
>
> /*
> @@ -3131,49 +3134,51 @@ static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
> * or fault_around_pages() from start_pgoff, depending what is nearest.
> */
> end_pgoff = start_pgoff -
> - ((fe->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
> + ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
> PTRS_PER_PTE - 1;
> - end_pgoff = min3(end_pgoff, vma_pages(fe->vma) + fe->vma->vm_pgoff - 1,
> + end_pgoff = min3(end_pgoff,
> + vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
> start_pgoff + nr_pages - 1);
>
> - if (pmd_none(*fe->pmd)) {
> - fe->prealloc_pte = pte_alloc_one(fe->vma->vm_mm, fe->address);
> - if (!fe->prealloc_pte)
> + if (pmd_none(*vmf->pmd)) {
> + vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
> + vmf->address);
> + if (!vmf->prealloc_pte)
> goto out;
> smp_wmb(); /* See comment in __pte_alloc() */
> }
>
> - fe->vma->vm_ops->map_pages(fe, start_pgoff, end_pgoff);
> + vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
>
> /* preallocated pagetable is unused: free it */
> - if (fe->prealloc_pte) {
> - pte_free(fe->vma->vm_mm, fe->prealloc_pte);
> - fe->prealloc_pte = 0;
> + if (vmf->prealloc_pte) {
> + pte_free(vmf->vma->vm_mm, vmf->prealloc_pte);
> + vmf->prealloc_pte = 0;
> }
> /* Huge page is mapped? Page fault is solved */
> - if (pmd_trans_huge(*fe->pmd)) {
> + if (pmd_trans_huge(*vmf->pmd)) {
> ret = VM_FAULT_NOPAGE;
> goto out;
> }
>
> /* ->map_pages() haven't done anything useful. Cold page cache? */
> - if (!fe->pte)
> + if (!vmf->pte)
> goto out;
>
> /* check if the page fault is solved */
> - fe->pte -= (fe->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
> - if (!pte_none(*fe->pte))
> + vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
> + if (!pte_none(*vmf->pte))
> ret = VM_FAULT_NOPAGE;
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out:
> - fe->address = address;
> - fe->pte = NULL;
> + vmf->address = address;
> + vmf->pte = NULL;
> return ret;
> }
>
> -static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page;
> int ret = 0;
>
> @@ -3183,27 +3188,27 @@ static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
> * something).
> */
> if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
> - ret = do_fault_around(fe, pgoff);
> + ret = do_fault_around(vmf, pgoff);
> if (ret)
> return ret;
> }
>
> - ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
> + ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
>
> - ret |= alloc_set_pte(fe, NULL, fault_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, NULL, fault_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> unlock_page(fault_page);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> put_page(fault_page);
> return ret;
> }
>
> -static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page, *new_page;
> void *fault_entry;
> struct mem_cgroup *memcg;
> @@ -3212,7 +3217,7 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> if (unlikely(anon_vma_prepare(vma)))
> return VM_FAULT_OOM;
>
> - new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, fe->address);
> + new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
> if (!new_page)
> return VM_FAULT_OOM;
>
> @@ -3222,17 +3227,17 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> return VM_FAULT_OOM;
> }
>
> - ret = __do_fault(fe, pgoff, new_page, &fault_page, &fault_entry);
> + ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> goto uncharge_out;
>
> if (!(ret & VM_FAULT_DAX_LOCKED))
> - copy_user_highpage(new_page, fault_page, fe->address, vma);
> + copy_user_highpage(new_page, fault_page, vmf->address, vma);
> __SetPageUptodate(new_page);
>
> - ret |= alloc_set_pte(fe, memcg, new_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, memcg, new_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (!(ret & VM_FAULT_DAX_LOCKED)) {
> unlock_page(fault_page);
> put_page(fault_page);
> @@ -3248,15 +3253,15 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> return ret;
> }
>
> -static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page;
> struct address_space *mapping;
> int dirtied = 0;
> int ret, tmp;
>
> - ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
> + ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
>
> @@ -3266,7 +3271,7 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> */
> if (vma->vm_ops->page_mkwrite) {
> unlock_page(fault_page);
> - tmp = do_page_mkwrite(vma, fault_page, fe->address);
> + tmp = do_page_mkwrite(vma, fault_page, vmf->address);
> if (unlikely(!tmp ||
> (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
> put_page(fault_page);
> @@ -3274,9 +3279,9 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> }
> }
>
> - ret |= alloc_set_pte(fe, NULL, fault_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, NULL, fault_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
> VM_FAULT_RETRY))) {
> unlock_page(fault_page);
> @@ -3314,19 +3319,19 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> * The mmap_sem may have been released depending on flags and our
> * return value. See filemap_fault() and __lock_page_or_retry().
> */
> -static int do_fault(struct fault_env *fe)
> +static int do_fault(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> - pgoff_t pgoff = linear_page_index(vma, fe->address);
> + struct vm_area_struct *vma = vmf->vma;
> + pgoff_t pgoff = linear_page_index(vma, vmf->address);
>
> /* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
> if (!vma->vm_ops->fault)
> return VM_FAULT_SIGBUS;
> - if (!(fe->flags & FAULT_FLAG_WRITE))
> - return do_read_fault(fe, pgoff);
> + if (!(vmf->flags & FAULT_FLAG_WRITE))
> + return do_read_fault(vmf, pgoff);
> if (!(vma->vm_flags & VM_SHARED))
> - return do_cow_fault(fe, pgoff);
> - return do_shared_fault(fe, pgoff);
> + return do_cow_fault(vmf, pgoff);
> + return do_shared_fault(vmf, pgoff);
> }
>
> static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
> @@ -3344,9 +3349,9 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
> return mpol_misplaced(page, vma, addr);
> }
>
> -static int do_numa_page(struct fault_env *fe, pte_t pte)
> +static int do_numa_page(struct vm_fault *vmf, pte_t pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page = NULL;
> int page_nid = -1;
> int last_cpupid;
> @@ -3364,10 +3369,10 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> * page table entry is not accessible, so there would be no
> * concurrent hardware modifications to the PTE.
> */
> - fe->ptl = pte_lockptr(vma->vm_mm, fe->pmd);
> - spin_lock(fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, pte))) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
> + spin_lock(vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, pte))) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> goto out;
> }
>
> @@ -3376,18 +3381,18 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> pte = pte_mkyoung(pte);
> if (was_writable)
> pte = pte_mkwrite(pte);
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
> - update_mmu_cache(vma, fe->address, fe->pte);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
>
> - page = vm_normal_page(vma, fe->address, pte);
> + page = vm_normal_page(vma, vmf->address, pte);
> if (!page) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> /* TODO: handle PTE-mapped THP */
> if (PageCompound(page)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> @@ -3411,9 +3416,9 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
>
> last_cpupid = page_cpupid_last(page);
> page_nid = page_to_nid(page);
> - target_nid = numa_migrate_prep(page, vma, fe->address, page_nid,
> + target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
> &flags);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (target_nid == -1) {
> put_page(page);
> goto out;
> @@ -3433,28 +3438,28 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> return 0;
> }
>
> -static int create_huge_pmd(struct fault_env *fe)
> +static int create_huge_pmd(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> if (vma_is_anonymous(vma))
> - return do_huge_pmd_anonymous_page(fe);
> + return do_huge_pmd_anonymous_page(vmf);
> if (vma->vm_ops->pmd_fault)
> - return vma->vm_ops->pmd_fault(vma, fe->address, fe->pmd,
> - fe->flags);
> + return vma->vm_ops->pmd_fault(vma, vmf->address, vmf->pmd,
> + vmf->flags);
> return VM_FAULT_FALLBACK;
> }
>
> -static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
> +static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> - if (vma_is_anonymous(fe->vma))
> - return do_huge_pmd_wp_page(fe, orig_pmd);
> - if (fe->vma->vm_ops->pmd_fault)
> - return fe->vma->vm_ops->pmd_fault(fe->vma, fe->address, fe->pmd,
> - fe->flags);
> + if (vma_is_anonymous(vmf->vma))
> + return do_huge_pmd_wp_page(vmf, orig_pmd);
> + if (vmf->vma->vm_ops->pmd_fault)
> + return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
> + vmf->pmd, vmf->flags);
>
> /* COW handled on pte level: split pmd */
> - VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
> - split_huge_pmd(fe->vma, fe->pmd, fe->address);
> + VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
> + split_huge_pmd(vmf->vma, vmf->pmd, vmf->address);
>
> return VM_FAULT_FALLBACK;
> }
> @@ -3479,21 +3484,21 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
> * The mmap_sem may have been released depending on flags and our return value.
> * See filemap_fault() and __lock_page_or_retry().
> */
> -static int handle_pte_fault(struct fault_env *fe)
> +static int handle_pte_fault(struct vm_fault *vmf)
> {
> pte_t entry;
>
> - if (unlikely(pmd_none(*fe->pmd))) {
> + if (unlikely(pmd_none(*vmf->pmd))) {
> /*
> * Leave __pte_alloc() until later: because vm_ops->fault may
> * want to allocate huge page, and if we expose page table
> * for an instant, it will be difficult to retract from
> * concurrent faults and from rmap lookups.
> */
> - fe->pte = NULL;
> + vmf->pte = NULL;
> } else {
> /* See comment in pte_alloc_one_map() */
> - if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
> + if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
> return 0;
> /*
> * A regular pmd is established and it can't morph into a huge
> @@ -3501,9 +3506,9 @@ static int handle_pte_fault(struct fault_env *fe)
> * mmap_sem read mode and khugepaged takes it in write mode.
> * So now it's safe to run pte_offset_map().
> */
> - fe->pte = pte_offset_map(fe->pmd, fe->address);
> + vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
>
> - entry = *fe->pte;
> + entry = *vmf->pte;
>
> /*
> * some architectures can have larger ptes than wordsize,
> @@ -3515,37 +3520,37 @@ static int handle_pte_fault(struct fault_env *fe)
> */
> barrier();
> if (pte_none(entry)) {
> - pte_unmap(fe->pte);
> - fe->pte = NULL;
> + pte_unmap(vmf->pte);
> + vmf->pte = NULL;
> }
> }
>
> - if (!fe->pte) {
> - if (vma_is_anonymous(fe->vma))
> - return do_anonymous_page(fe);
> + if (!vmf->pte) {
> + if (vma_is_anonymous(vmf->vma))
> + return do_anonymous_page(vmf);
> else
> - return do_fault(fe);
> + return do_fault(vmf);
> }
>
> if (!pte_present(entry))
> - return do_swap_page(fe, entry);
> + return do_swap_page(vmf, entry);
>
> - if (pte_protnone(entry) && vma_is_accessible(fe->vma))
> - return do_numa_page(fe, entry);
> + if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
> + return do_numa_page(vmf, entry);
>
> - fe->ptl = pte_lockptr(fe->vma->vm_mm, fe->pmd);
> - spin_lock(fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, entry)))
> + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
> + spin_lock(vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, entry)))
> goto unlock;
> - if (fe->flags & FAULT_FLAG_WRITE) {
> + if (vmf->flags & FAULT_FLAG_WRITE) {
> if (!pte_write(entry))
> - return do_wp_page(fe, entry);
> + return do_wp_page(vmf, entry);
> entry = pte_mkdirty(entry);
> }
> entry = pte_mkyoung(entry);
> - if (ptep_set_access_flags(fe->vma, fe->address, fe->pte, entry,
> - fe->flags & FAULT_FLAG_WRITE)) {
> - update_mmu_cache(fe->vma, fe->address, fe->pte);
> + if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
> + vmf->flags & FAULT_FLAG_WRITE)) {
> + update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
> } else {
> /*
> * This is needed only for protection faults but the arch code
> @@ -3553,11 +3558,11 @@ static int handle_pte_fault(struct fault_env *fe)
> * This still avoids useless tlb flushes for .text page faults
> * with threads.
> */
> - if (fe->flags & FAULT_FLAG_WRITE)
> - flush_tlb_fix_spurious_fault(fe->vma, fe->address);
> + if (vmf->flags & FAULT_FLAG_WRITE)
> + flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
> }
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> @@ -3570,7 +3575,7 @@ static int handle_pte_fault(struct fault_env *fe)
> static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
> unsigned int flags)
> {
> - struct fault_env fe = {
> + struct vm_fault vmf = {
> .vma = vma,
> .address = address,
> .flags = flags,
> @@ -3583,35 +3588,35 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
> pud = pud_alloc(mm, pgd, address);
> if (!pud)
> return VM_FAULT_OOM;
> - fe.pmd = pmd_alloc(mm, pud, address);
> - if (!fe.pmd)
> + vmf.pmd = pmd_alloc(mm, pud, address);
> + if (!vmf.pmd)
> return VM_FAULT_OOM;
> - if (pmd_none(*fe.pmd) && transparent_hugepage_enabled(vma)) {
> - int ret = create_huge_pmd(&fe);
> + if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
> + int ret = create_huge_pmd(&vmf);
> if (!(ret & VM_FAULT_FALLBACK))
> return ret;
> } else {
> - pmd_t orig_pmd = *fe.pmd;
> + pmd_t orig_pmd = *vmf.pmd;
> int ret;
>
> barrier();
> if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
> if (pmd_protnone(orig_pmd) && vma_is_accessible(vma))
> - return do_huge_pmd_numa_page(&fe, orig_pmd);
> + return do_huge_pmd_numa_page(&vmf, orig_pmd);
>
> - if ((fe.flags & FAULT_FLAG_WRITE) &&
> + if ((vmf.flags & FAULT_FLAG_WRITE) &&
> !pmd_write(orig_pmd)) {
> - ret = wp_huge_pmd(&fe, orig_pmd);
> + ret = wp_huge_pmd(&vmf, orig_pmd);
> if (!(ret & VM_FAULT_FALLBACK))
> return ret;
> } else {
> - huge_pmd_set_accessed(&fe, orig_pmd);
> + huge_pmd_set_accessed(&vmf, orig_pmd);
> return 0;
> }
> }
> }
>
> - return handle_pte_fault(&fe);
> + return handle_pte_fault(&vmf);
> }
>
> /*
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 8b8faaf2a9e9..077d0dbe4c28 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -1801,7 +1801,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> }
> EXPORT_SYMBOL(filemap_fault);
>
> -void filemap_map_pages(struct fault_env *fe,
> +void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff)
> {
> BUG();
> --
> 2.6.6
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-15 21:50 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 21:50 UTC (permalink / raw)
To: Jan Kara, Peter Zijlstra
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> Currently we have two different structures for passing fault information
> around - struct vm_fault and struct fault_env. DAX will need more
> information in struct vm_fault to handle its faults so the content of
> that structure would become event closer to fault_env. Furthermore it
> would need to generate struct fault_env to be able to call some of the
> generic functions. So at this point I don't think there's much use in
> keeping these two structures separate. Just embed into struct vm_fault
> all that is needed to use it for both purposes.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
I'm not necessary dislike this, but I remember Peter had objections before
when I proposed something similar.
Peter?
> ---
> Documentation/filesystems/Locking | 2 +-
> fs/userfaultfd.c | 22 +-
> include/linux/huge_mm.h | 10 +-
> include/linux/mm.h | 28 +-
> include/linux/userfaultfd_k.h | 4 +-
> mm/filemap.c | 14 +-
> mm/huge_memory.c | 173 ++++++------
> mm/internal.h | 2 +-
> mm/khugepaged.c | 20 +-
> mm/memory.c | 549 +++++++++++++++++++-------------------
> mm/nommu.c | 2 +-
> 11 files changed, 414 insertions(+), 412 deletions(-)
>
> diff --git a/Documentation/filesystems/Locking b/Documentation/filesystems/Locking
> index 14cdc101d165..ac3d080eabaa 100644
> --- a/Documentation/filesystems/Locking
> +++ b/Documentation/filesystems/Locking
> @@ -557,7 +557,7 @@ till "end_pgoff". ->map_pages() is called with page table locked and must
> not block. If it's not possible to reach a page without blocking,
> filesystem should skip it. Filesystem should use do_set_pte() to setup
> page table entry. Pointer to entry associated with the page is passed in
> -"pte" field in fault_env structure. Pointers to entries for other offsets
> +"pte" field in vm_fault structure. Pointers to entries for other offsets
> should be calculated relative to "pte".
>
> ->page_mkwrite() is called when a previously read-only pte is
> diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
> index 85959d8324df..d96e2f30084b 100644
> --- a/fs/userfaultfd.c
> +++ b/fs/userfaultfd.c
> @@ -257,9 +257,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
> * fatal_signal_pending()s, and the mmap_sem must be released before
> * returning it.
> */
> -int handle_userfault(struct fault_env *fe, unsigned long reason)
> +int handle_userfault(struct vm_fault *vmf, unsigned long reason)
> {
> - struct mm_struct *mm = fe->vma->vm_mm;
> + struct mm_struct *mm = vmf->vma->vm_mm;
> struct userfaultfd_ctx *ctx;
> struct userfaultfd_wait_queue uwq;
> int ret;
> @@ -268,7 +268,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> BUG_ON(!rwsem_is_locked(&mm->mmap_sem));
>
> ret = VM_FAULT_SIGBUS;
> - ctx = fe->vma->vm_userfaultfd_ctx.ctx;
> + ctx = vmf->vma->vm_userfaultfd_ctx.ctx;
> if (!ctx)
> goto out;
>
> @@ -301,17 +301,18 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> * without first stopping userland access to the memory. For
> * VM_UFFD_MISSING userfaults this is enough for now.
> */
> - if (unlikely(!(fe->flags & FAULT_FLAG_ALLOW_RETRY))) {
> + if (unlikely(!(vmf->flags & FAULT_FLAG_ALLOW_RETRY))) {
> /*
> * Validate the invariant that nowait must allow retry
> * to be sure not to return SIGBUS erroneously on
> * nowait invocations.
> */
> - BUG_ON(fe->flags & FAULT_FLAG_RETRY_NOWAIT);
> + BUG_ON(vmf->flags & FAULT_FLAG_RETRY_NOWAIT);
> #ifdef CONFIG_DEBUG_VM
> if (printk_ratelimit()) {
> printk(KERN_WARNING
> - "FAULT_FLAG_ALLOW_RETRY missing %x\n", fe->flags);
> + "FAULT_FLAG_ALLOW_RETRY missing %x\n",
> + vmf->flags);
> dump_stack();
> }
> #endif
> @@ -323,7 +324,7 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> * and wait.
> */
> ret = VM_FAULT_RETRY;
> - if (fe->flags & FAULT_FLAG_RETRY_NOWAIT)
> + if (vmf->flags & FAULT_FLAG_RETRY_NOWAIT)
> goto out;
>
> /* take the reference before dropping the mmap_sem */
> @@ -331,11 +332,11 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
>
> init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
> uwq.wq.private = current;
> - uwq.msg = userfault_msg(fe->address, fe->flags, reason);
> + uwq.msg = userfault_msg(vmf->address, vmf->flags, reason);
> uwq.ctx = ctx;
>
> return_to_userland =
> - (fe->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
> + (vmf->flags & (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE)) ==
> (FAULT_FLAG_USER|FAULT_FLAG_KILLABLE);
>
> spin_lock(&ctx->fault_pending_wqh.lock);
> @@ -353,7 +354,8 @@ int handle_userfault(struct fault_env *fe, unsigned long reason)
> TASK_KILLABLE);
> spin_unlock(&ctx->fault_pending_wqh.lock);
>
> - must_wait = userfaultfd_must_wait(ctx, fe->address, fe->flags, reason);
> + must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags,
> + reason);
> up_read(&mm->mmap_sem);
>
> if (likely(must_wait && !ACCESS_ONCE(ctx->released) &&
> diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
> index 9b9f65d99873..3237a39f94a4 100644
> --- a/include/linux/huge_mm.h
> +++ b/include/linux/huge_mm.h
> @@ -1,12 +1,12 @@
> #ifndef _LINUX_HUGE_MM_H
> #define _LINUX_HUGE_MM_H
>
> -extern int do_huge_pmd_anonymous_page(struct fault_env *fe);
> +extern int do_huge_pmd_anonymous_page(struct vm_fault *vmf);
> extern int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr,
> struct vm_area_struct *vma);
> -extern void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd);
> -extern int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd);
> +extern void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd);
> +extern int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd);
> extern struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
> unsigned long addr,
> pmd_t *pmd,
> @@ -142,7 +142,7 @@ static inline int hpage_nr_pages(struct page *page)
> return 1;
> }
>
> -extern int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd);
> +extern int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd);
>
> extern struct page *huge_zero_page;
>
> @@ -210,7 +210,7 @@ static inline spinlock_t *pmd_trans_huge_lock(pmd_t *pmd,
> return NULL;
> }
>
> -static inline int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t orig_pmd)
> +static inline int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> return 0;
> }
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index a92c8d73aeaf..657eb69eb87e 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> * pgoff should be used in favour of virtual_address, if possible.
> */
> struct vm_fault {
> + struct vm_area_struct *vma; /* Target VMA */
> unsigned int flags; /* FAULT_FLAG_xxx flags */
> gfp_t gfp_mask; /* gfp mask to be used for allocations */
> pgoff_t pgoff; /* Logical page offset based on vma */
> - void __user *virtual_address; /* Faulting virtual address */
> + unsigned long address; /* Faulting virtual address */
> + void __user *virtual_address; /* Faulting virtual address masked by
> + * PAGE_MASK */
> + pmd_t *pmd; /* Pointer to pmd entry matching
> + * the 'address'
> + */
>
> struct page *cow_page; /* Handler may choose to COW */
> struct page *page; /* ->fault handlers should return a
> @@ -309,19 +315,7 @@ struct vm_fault {
> * VM_FAULT_DAX_LOCKED and fill in
> * entry here.
> */
> -};
> -
> -/*
> - * Page fault context: passes though page fault handler instead of endless list
> - * of function arguments.
> - */
> -struct fault_env {
> - struct vm_area_struct *vma; /* Target VMA */
> - unsigned long address; /* Faulting virtual address */
> - unsigned int flags; /* FAULT_FLAG_xxx flags */
> - pmd_t *pmd; /* Pointer to pmd entry matching
> - * the 'address'
> - */
> + /* These three entries are valid only while holding ptl lock */
> pte_t *pte; /* Pointer to pte entry matching
> * the 'address'. NULL if the page
> * table hasn't been allocated.
> @@ -351,7 +345,7 @@ struct vm_operations_struct {
> int (*fault)(struct vm_area_struct *vma, struct vm_fault *vmf);
> int (*pmd_fault)(struct vm_area_struct *, unsigned long address,
> pmd_t *, unsigned int flags);
> - void (*map_pages)(struct fault_env *fe,
> + void (*map_pages)(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff);
>
> /* notification that a previously read-only page is about to become
> @@ -625,7 +619,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
> return pte;
> }
>
> -int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> +int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page);
> #endif
>
> @@ -2097,7 +2091,7 @@ extern void truncate_inode_pages_final(struct address_space *);
>
> /* generic vm_area_ops exported for stackable file systems */
> extern int filemap_fault(struct vm_area_struct *, struct vm_fault *);
> -extern void filemap_map_pages(struct fault_env *fe,
> +extern void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff);
> extern int filemap_page_mkwrite(struct vm_area_struct *vma, struct vm_fault *vmf);
>
> diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
> index dd66a952e8cd..11b92b047a1e 100644
> --- a/include/linux/userfaultfd_k.h
> +++ b/include/linux/userfaultfd_k.h
> @@ -27,7 +27,7 @@
> #define UFFD_SHARED_FCNTL_FLAGS (O_CLOEXEC | O_NONBLOCK)
> #define UFFD_FLAGS_SET (EFD_SHARED_FCNTL_FLAGS)
>
> -extern int handle_userfault(struct fault_env *fe, unsigned long reason);
> +extern int handle_userfault(struct vm_fault *vmf, unsigned long reason);
>
> extern ssize_t mcopy_atomic(struct mm_struct *dst_mm, unsigned long dst_start,
> unsigned long src_start, unsigned long len);
> @@ -55,7 +55,7 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
> #else /* CONFIG_USERFAULTFD */
>
> /* mm helpers */
> -static inline int handle_userfault(struct fault_env *fe, unsigned long reason)
> +static inline int handle_userfault(struct vm_fault *vmf, unsigned long reason)
> {
> return VM_FAULT_SIGBUS;
> }
> diff --git a/mm/filemap.c b/mm/filemap.c
> index db26ebc6c62f..1426fb1a99b3 100644
> --- a/mm/filemap.c
> +++ b/mm/filemap.c
> @@ -2209,12 +2209,12 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> }
> EXPORT_SYMBOL(filemap_fault);
>
> -void filemap_map_pages(struct fault_env *fe,
> +void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff)
> {
> struct radix_tree_iter iter;
> void **slot;
> - struct file *file = fe->vma->vm_file;
> + struct file *file = vmf->vma->vm_file;
> struct address_space *mapping = file->f_mapping;
> pgoff_t last_pgoff = start_pgoff;
> loff_t size;
> @@ -2270,11 +2270,11 @@ void filemap_map_pages(struct fault_env *fe,
> if (file->f_ra.mmap_miss > 0)
> file->f_ra.mmap_miss--;
>
> - fe->address += (iter.index - last_pgoff) << PAGE_SHIFT;
> - if (fe->pte)
> - fe->pte += iter.index - last_pgoff;
> + vmf->address += (iter.index - last_pgoff) << PAGE_SHIFT;
> + if (vmf->pte)
> + vmf->pte += iter.index - last_pgoff;
> last_pgoff = iter.index;
> - if (alloc_set_pte(fe, NULL, page))
> + if (alloc_set_pte(vmf, NULL, page))
> goto unlock;
> unlock_page(page);
> goto next;
> @@ -2284,7 +2284,7 @@ void filemap_map_pages(struct fault_env *fe,
> put_page(page);
> next:
> /* Huge page is mapped? No need to proceed. */
> - if (pmd_trans_huge(*fe->pmd))
> + if (pmd_trans_huge(*vmf->pmd))
> break;
> if (iter.index == end_pgoff)
> break;
> diff --git a/mm/huge_memory.c b/mm/huge_memory.c
> index cdcd25cb30fe..e286b09f9d24 100644
> --- a/mm/huge_memory.c
> +++ b/mm/huge_memory.c
> @@ -532,13 +532,13 @@ unsigned long thp_get_unmapped_area(struct file *filp, unsigned long addr,
> }
> EXPORT_SYMBOL_GPL(thp_get_unmapped_area);
>
> -static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> +static int __do_huge_pmd_anonymous_page(struct vm_fault *vmf, struct page *page,
> gfp_t gfp)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mem_cgroup *memcg;
> pgtable_t pgtable;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
>
> VM_BUG_ON_PAGE(!PageCompound(page), page);
>
> @@ -563,9 +563,9 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> */
> __SetPageUptodate(page);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd))) {
> - spin_unlock(fe->ptl);
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd))) {
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, true);
> put_page(page);
> pte_free(vma->vm_mm, pgtable);
> @@ -576,11 +576,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> if (userfaultfd_missing(vma)) {
> int ret;
>
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, true);
> put_page(page);
> pte_free(vma->vm_mm, pgtable);
> - ret = handle_userfault(fe, VM_UFFD_MISSING);
> + ret = handle_userfault(vmf, VM_UFFD_MISSING);
> VM_BUG_ON(ret & VM_FAULT_FALLBACK);
> return ret;
> }
> @@ -590,11 +590,11 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
> page_add_new_anon_rmap(page, vma, haddr, true);
> mem_cgroup_commit_charge(page, memcg, false, true);
> lru_cache_add_active_or_unevictable(page, vma);
> - pgtable_trans_huge_deposit(vma->vm_mm, fe->pmd, pgtable);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> + pgtable_trans_huge_deposit(vma->vm_mm, vmf->pmd, pgtable);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
> add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
> atomic_long_inc(&vma->vm_mm->nr_ptes);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> count_vm_event(THP_FAULT_ALLOC);
> }
>
> @@ -641,12 +641,12 @@ static bool set_huge_zero_page(pgtable_t pgtable, struct mm_struct *mm,
> return true;
> }
>
> -int do_huge_pmd_anonymous_page(struct fault_env *fe)
> +int do_huge_pmd_anonymous_page(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> gfp_t gfp;
> struct page *page;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
>
> if (haddr < vma->vm_start || haddr + HPAGE_PMD_SIZE > vma->vm_end)
> return VM_FAULT_FALLBACK;
> @@ -654,7 +654,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> return VM_FAULT_OOM;
> if (unlikely(khugepaged_enter(vma, vma->vm_flags)))
> return VM_FAULT_OOM;
> - if (!(fe->flags & FAULT_FLAG_WRITE) &&
> + if (!(vmf->flags & FAULT_FLAG_WRITE) &&
> !mm_forbids_zeropage(vma->vm_mm) &&
> transparent_hugepage_use_zero_page()) {
> pgtable_t pgtable;
> @@ -670,22 +670,22 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> count_vm_event(THP_FAULT_FALLBACK);
> return VM_FAULT_FALLBACK;
> }
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> ret = 0;
> set = false;
> - if (pmd_none(*fe->pmd)) {
> + if (pmd_none(*vmf->pmd)) {
> if (userfaultfd_missing(vma)) {
> - spin_unlock(fe->ptl);
> - ret = handle_userfault(fe, VM_UFFD_MISSING);
> + spin_unlock(vmf->ptl);
> + ret = handle_userfault(vmf, VM_UFFD_MISSING);
> VM_BUG_ON(ret & VM_FAULT_FALLBACK);
> } else {
> set_huge_zero_page(pgtable, vma->vm_mm, vma,
> - haddr, fe->pmd, zero_page);
> - spin_unlock(fe->ptl);
> + haddr, vmf->pmd, zero_page);
> + spin_unlock(vmf->ptl);
> set = true;
> }
> } else
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> if (!set)
> pte_free(vma->vm_mm, pgtable);
> return ret;
> @@ -697,7 +697,7 @@ int do_huge_pmd_anonymous_page(struct fault_env *fe)
> return VM_FAULT_FALLBACK;
> }
> prep_transhuge_page(page);
> - return __do_huge_pmd_anonymous_page(fe, page, gfp);
> + return __do_huge_pmd_anonymous_page(vmf, page, gfp);
> }
>
> static void insert_pfn_pmd(struct vm_area_struct *vma, unsigned long addr,
> @@ -868,30 +868,30 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm,
> return ret;
> }
>
> -void huge_pmd_set_accessed(struct fault_env *fe, pmd_t orig_pmd)
> +void huge_pmd_set_accessed(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> pmd_t entry;
> unsigned long haddr;
>
> - fe->ptl = pmd_lock(fe->vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + vmf->ptl = pmd_lock(vmf->vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto unlock;
>
> entry = pmd_mkyoung(orig_pmd);
> - haddr = fe->address & HPAGE_PMD_MASK;
> - if (pmdp_set_access_flags(fe->vma, haddr, fe->pmd, entry,
> - fe->flags & FAULT_FLAG_WRITE))
> - update_mmu_cache_pmd(fe->vma, fe->address, fe->pmd);
> + haddr = vmf->address & HPAGE_PMD_MASK;
> + if (pmdp_set_access_flags(vmf->vma, haddr, vmf->pmd, entry,
> + vmf->flags & FAULT_FLAG_WRITE))
> + update_mmu_cache_pmd(vmf->vma, vmf->address, vmf->pmd);
>
> unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> }
>
> -static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> +static int do_huge_pmd_wp_page_fallback(struct vm_fault *vmf, pmd_t orig_pmd,
> struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + struct vm_area_struct *vma = vmf->vma;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> struct mem_cgroup *memcg;
> pgtable_t pgtable;
> pmd_t _pmd;
> @@ -910,7 +910,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> for (i = 0; i < HPAGE_PMD_NR; i++) {
> pages[i] = alloc_page_vma_node(GFP_HIGHUSER_MOVABLE |
> __GFP_OTHER_NODE, vma,
> - fe->address, page_to_nid(page));
> + vmf->address, page_to_nid(page));
> if (unlikely(!pages[i] ||
> mem_cgroup_try_charge(pages[i], vma->vm_mm,
> GFP_KERNEL, &memcg, false))) {
> @@ -941,15 +941,15 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> mmun_end = haddr + HPAGE_PMD_SIZE;
> mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto out_free_pages;
> VM_BUG_ON_PAGE(!PageHead(page), page);
>
> - pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
> + pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
> /* leave pmd empty until pte is filled */
>
> - pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, fe->pmd);
> + pgtable = pgtable_trans_huge_withdraw(vma->vm_mm, vmf->pmd);
> pmd_populate(vma->vm_mm, &_pmd, pgtable);
>
> for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) {
> @@ -958,20 +958,20 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> memcg = (void *)page_private(pages[i]);
> set_page_private(pages[i], 0);
> - page_add_new_anon_rmap(pages[i], fe->vma, haddr, false);
> + page_add_new_anon_rmap(pages[i], vmf->vma, haddr, false);
> mem_cgroup_commit_charge(pages[i], memcg, false, false);
> lru_cache_add_active_or_unevictable(pages[i], vma);
> - fe->pte = pte_offset_map(&_pmd, haddr);
> - VM_BUG_ON(!pte_none(*fe->pte));
> - set_pte_at(vma->vm_mm, haddr, fe->pte, entry);
> - pte_unmap(fe->pte);
> + vmf->pte = pte_offset_map(&_pmd, haddr);
> + VM_BUG_ON(!pte_none(*vmf->pte));
> + set_pte_at(vma->vm_mm, haddr, vmf->pte, entry);
> + pte_unmap(vmf->pte);
> }
> kfree(pages);
>
> smp_wmb(); /* make pte visible before pmd */
> - pmd_populate(vma->vm_mm, fe->pmd, pgtable);
> + pmd_populate(vma->vm_mm, vmf->pmd, pgtable);
> page_remove_rmap(page, true);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
>
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
>
> @@ -982,7 +982,7 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> return ret;
>
> out_free_pages:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
> for (i = 0; i < HPAGE_PMD_NR; i++) {
> memcg = (void *)page_private(pages[i]);
> @@ -994,23 +994,23 @@ static int do_huge_pmd_wp_page_fallback(struct fault_env *fe, pmd_t orig_pmd,
> goto out;
> }
>
> -int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> +int do_huge_pmd_wp_page(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page = NULL, *new_page;
> struct mem_cgroup *memcg;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> unsigned long mmun_start; /* For mmu_notifiers */
> unsigned long mmun_end; /* For mmu_notifiers */
> gfp_t huge_gfp; /* for allocation and charge */
> int ret = 0;
>
> - fe->ptl = pmd_lockptr(vma->vm_mm, fe->pmd);
> + vmf->ptl = pmd_lockptr(vma->vm_mm, vmf->pmd);
> VM_BUG_ON_VMA(!vma->anon_vma, vma);
> if (is_huge_zero_pmd(orig_pmd))
> goto alloc;
> - spin_lock(fe->ptl);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd)))
> + spin_lock(vmf->ptl);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd)))
> goto out_unlock;
>
> page = pmd_page(orig_pmd);
> @@ -1023,13 +1023,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> pmd_t entry;
> entry = pmd_mkyoung(orig_pmd);
> entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
> - if (pmdp_set_access_flags(vma, haddr, fe->pmd, entry, 1))
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + if (pmdp_set_access_flags(vma, haddr, vmf->pmd, entry, 1))
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> ret |= VM_FAULT_WRITE;
> goto out_unlock;
> }
> get_page(page);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> alloc:
> if (transparent_hugepage_enabled(vma) &&
> !transparent_hugepage_debug_cow()) {
> @@ -1042,12 +1042,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> prep_transhuge_page(new_page);
> } else {
> if (!page) {
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> ret |= VM_FAULT_FALLBACK;
> } else {
> - ret = do_huge_pmd_wp_page_fallback(fe, orig_pmd, page);
> + ret = do_huge_pmd_wp_page_fallback(vmf, orig_pmd, page);
> if (ret & VM_FAULT_OOM) {
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> ret |= VM_FAULT_FALLBACK;
> }
> put_page(page);
> @@ -1059,7 +1059,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
> huge_gfp, &memcg, true))) {
> put_page(new_page);
> - split_huge_pmd(vma, fe->pmd, fe->address);
> + split_huge_pmd(vma, vmf->pmd, vmf->address);
> if (page)
> put_page(page);
> ret |= VM_FAULT_FALLBACK;
> @@ -1079,11 +1079,11 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> mmun_end = haddr + HPAGE_PMD_SIZE;
> mmu_notifier_invalidate_range_start(vma->vm_mm, mmun_start, mmun_end);
>
> - spin_lock(fe->ptl);
> + spin_lock(vmf->ptl);
> if (page)
> put_page(page);
> - if (unlikely(!pmd_same(*fe->pmd, orig_pmd))) {
> - spin_unlock(fe->ptl);
> + if (unlikely(!pmd_same(*vmf->pmd, orig_pmd))) {
> + spin_unlock(vmf->ptl);
> mem_cgroup_cancel_charge(new_page, memcg, true);
> put_page(new_page);
> goto out_mn;
> @@ -1091,12 +1091,12 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> pmd_t entry;
> entry = mk_huge_pmd(new_page, vma->vm_page_prot);
> entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma);
> - pmdp_huge_clear_flush_notify(vma, haddr, fe->pmd);
> + pmdp_huge_clear_flush_notify(vma, haddr, vmf->pmd);
> page_add_new_anon_rmap(new_page, vma, haddr, true);
> mem_cgroup_commit_charge(new_page, memcg, false, true);
> lru_cache_add_active_or_unevictable(new_page, vma);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> if (!page) {
> add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
> } else {
> @@ -1106,13 +1106,13 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
> }
> ret |= VM_FAULT_WRITE;
> }
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> out_mn:
> mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
> out:
> return ret;
> out_unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> return ret;
> }
>
> @@ -1185,12 +1185,12 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
> }
>
> /* NUMA hinting page fault entry point for trans huge pmds */
> -int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> +int do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct anon_vma *anon_vma = NULL;
> struct page *page;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> int page_nid = -1, this_nid = numa_node_id();
> int target_nid, last_cpupid = -1;
> bool page_locked;
> @@ -1198,8 +1198,8 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> bool was_writable;
> int flags = 0;
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_same(pmd, *fe->pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_same(pmd, *vmf->pmd)))
> goto out_unlock;
>
> /*
> @@ -1207,9 +1207,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * without disrupting NUMA hinting information. Do not relock and
> * check_same as the page may no longer be mapped.
> */
> - if (unlikely(pmd_trans_migrating(*fe->pmd))) {
> - page = pmd_page(*fe->pmd);
> - spin_unlock(fe->ptl);
> + if (unlikely(pmd_trans_migrating(*vmf->pmd))) {
> + page = pmd_page(*vmf->pmd);
> + spin_unlock(vmf->ptl);
> wait_on_page_locked(page);
> goto out;
> }
> @@ -1242,7 +1242,7 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
>
> /* Migration could have started since the pmd_trans_migrating check */
> if (!page_locked) {
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> wait_on_page_locked(page);
> page_nid = -1;
> goto out;
> @@ -1253,12 +1253,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * to serialises splits
> */
> get_page(page);
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> anon_vma = page_lock_anon_vma_read(page);
>
> /* Confirm the PMD did not change while page_table_lock was released */
> - spin_lock(fe->ptl);
> - if (unlikely(!pmd_same(pmd, *fe->pmd))) {
> + spin_lock(vmf->ptl);
> + if (unlikely(!pmd_same(pmd, *vmf->pmd))) {
> unlock_page(page);
> put_page(page);
> page_nid = -1;
> @@ -1276,9 +1276,9 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> * Migrate the THP to the requested node, returns with page unlocked
> * and access rights restored.
> */
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> migrated = migrate_misplaced_transhuge_page(vma->vm_mm, vma,
> - fe->pmd, pmd, fe->address, page, target_nid);
> + vmf->pmd, pmd, vmf->address, page, target_nid);
> if (migrated) {
> flags |= TNF_MIGRATED;
> page_nid = target_nid;
> @@ -1293,18 +1293,19 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
> pmd = pmd_mkyoung(pmd);
> if (was_writable)
> pmd = pmd_mkwrite(pmd);
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, pmd);
> - update_mmu_cache_pmd(vma, fe->address, fe->pmd);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, pmd);
> + update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
> unlock_page(page);
> out_unlock:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
>
> out:
> if (anon_vma)
> page_unlock_anon_vma_read(anon_vma);
>
> if (page_nid != -1)
> - task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR, fe->flags);
> + task_numa_fault(last_cpupid, page_nid, HPAGE_PMD_NR,
> + vmf->flags);
>
> return 0;
> }
> diff --git a/mm/internal.h b/mm/internal.h
> index 537ac9951f5f..093b1eacc91b 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -36,7 +36,7 @@
> /* Do not use these with a slab allocator */
> #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
>
> -int do_swap_page(struct fault_env *fe, pte_t orig_pte);
> +int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
>
> void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
> unsigned long floor, unsigned long ceiling);
> diff --git a/mm/khugepaged.c b/mm/khugepaged.c
> index 728d7790dc2d..f88b2d3810a7 100644
> --- a/mm/khugepaged.c
> +++ b/mm/khugepaged.c
> @@ -875,7 +875,7 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> {
> pte_t pteval;
> int swapped_in = 0, ret = 0;
> - struct fault_env fe = {
> + struct vm_fault vmf = {
> .vma = vma,
> .address = address,
> .flags = FAULT_FLAG_ALLOW_RETRY,
> @@ -887,19 +887,19 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
> return false;
> }
> - fe.pte = pte_offset_map(pmd, address);
> - for (; fe.address < address + HPAGE_PMD_NR*PAGE_SIZE;
> - fe.pte++, fe.address += PAGE_SIZE) {
> - pteval = *fe.pte;
> + vmf.pte = pte_offset_map(pmd, address);
> + for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
> + vmf.pte++, vmf.address += PAGE_SIZE) {
> + pteval = *vmf.pte;
> if (!is_swap_pte(pteval))
> continue;
> swapped_in++;
> - ret = do_swap_page(&fe, pteval);
> + ret = do_swap_page(&vmf, pteval);
>
> /* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
> if (ret & VM_FAULT_RETRY) {
> down_read(&mm->mmap_sem);
> - if (hugepage_vma_revalidate(mm, address, &fe.vma)) {
> + if (hugepage_vma_revalidate(mm, address, &vmf.vma)) {
> /* vma is no longer available, don't continue to swapin */
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0);
> return false;
> @@ -913,10 +913,10 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
> return false;
> }
> /* pte is unmapped now, we need to map it */
> - fe.pte = pte_offset_map(pmd, fe.address);
> + vmf.pte = pte_offset_map(pmd, vmf.address);
> }
> - fe.pte--;
> - pte_unmap(fe.pte);
> + vmf.pte--;
> + pte_unmap(vmf.pte);
> trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1);
> return true;
> }
> diff --git a/mm/memory.c b/mm/memory.c
> index e18c57bdc75c..fad45cd59ba7 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2074,11 +2074,11 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
> * case, all we need to do here is to mark the page as writable and update
> * any related book-keeping.
> */
> -static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> +static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
> struct page *page, int page_mkwrite, int dirty_shared)
> - __releases(fe->ptl)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> pte_t entry;
> /*
> * Clear the pages cpupid information as the existing
> @@ -2088,12 +2088,12 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> if (page)
> page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
>
> - flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
> + flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
> entry = pte_mkyoung(orig_pte);
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> - if (ptep_set_access_flags(vma, fe->address, fe->pte, entry, 1))
> - update_mmu_cache(vma, fe->address, fe->pte);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
>
> if (dirty_shared) {
> struct address_space *mapping;
> @@ -2139,15 +2139,15 @@ static inline int wp_page_reuse(struct fault_env *fe, pte_t orig_pte,
> * held to the old page, as well as updating the rmap.
> * - In any case, unlock the PTL and drop the reference we took to the old page.
> */
> -static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> +static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
> struct page *old_page)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mm_struct *mm = vma->vm_mm;
> struct page *new_page = NULL;
> pte_t entry;
> int page_copied = 0;
> - const unsigned long mmun_start = fe->address & PAGE_MASK;
> + const unsigned long mmun_start = vmf->address & PAGE_MASK;
> const unsigned long mmun_end = mmun_start + PAGE_SIZE;
> struct mem_cgroup *memcg;
>
> @@ -2155,15 +2155,16 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> goto oom;
>
> if (is_zero_pfn(pte_pfn(orig_pte))) {
> - new_page = alloc_zeroed_user_highpage_movable(vma, fe->address);
> + new_page = alloc_zeroed_user_highpage_movable(vma,
> + vmf->address);
> if (!new_page)
> goto oom;
> } else {
> new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma,
> - fe->address);
> + vmf->address);
> if (!new_page)
> goto oom;
> - cow_user_page(new_page, old_page, fe->address, vma);
> + cow_user_page(new_page, old_page, vmf->address, vma);
> }
>
> if (mem_cgroup_try_charge(new_page, mm, GFP_KERNEL, &memcg, false))
> @@ -2176,8 +2177,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> /*
> * Re-check the pte - we dropped the lock
> */
> - fe->pte = pte_offset_map_lock(mm, fe->pmd, fe->address, &fe->ptl);
> - if (likely(pte_same(*fe->pte, orig_pte))) {
> + vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
> + if (likely(pte_same(*vmf->pte, orig_pte))) {
> if (old_page) {
> if (!PageAnon(old_page)) {
> dec_mm_counter_fast(mm,
> @@ -2187,7 +2188,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> } else {
> inc_mm_counter_fast(mm, MM_ANONPAGES);
> }
> - flush_cache_page(vma, fe->address, pte_pfn(orig_pte));
> + flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
> entry = mk_pte(new_page, vma->vm_page_prot);
> entry = maybe_mkwrite(pte_mkdirty(entry), vma);
> /*
> @@ -2196,8 +2197,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * seen in the presence of one thread doing SMC and another
> * thread doing COW.
> */
> - ptep_clear_flush_notify(vma, fe->address, fe->pte);
> - page_add_new_anon_rmap(new_page, vma, fe->address, false);
> + ptep_clear_flush_notify(vma, vmf->address, vmf->pte);
> + page_add_new_anon_rmap(new_page, vma, vmf->address, false);
> mem_cgroup_commit_charge(new_page, memcg, false, false);
> lru_cache_add_active_or_unevictable(new_page, vma);
> /*
> @@ -2205,8 +2206,8 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * mmu page tables (such as kvm shadow page tables), we want the
> * new page to be mapped directly into the secondary page table.
> */
> - set_pte_at_notify(mm, fe->address, fe->pte, entry);
> - update_mmu_cache(vma, fe->address, fe->pte);
> + set_pte_at_notify(mm, vmf->address, vmf->pte, entry);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> if (old_page) {
> /*
> * Only after switching the pte to the new page may
> @@ -2243,7 +2244,7 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> if (new_page)
> put_page(new_page);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
> if (old_page) {
> /*
> @@ -2271,43 +2272,43 @@ static int wp_page_copy(struct fault_env *fe, pte_t orig_pte,
> * Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
> * mapping
> */
> -static int wp_pfn_shared(struct fault_env *fe, pte_t orig_pte)
> +static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
>
> if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
> - struct vm_fault vmf = {
> + struct vm_fault vmf2 = {
> .page = NULL,
> - .pgoff = linear_page_index(vma, fe->address),
> + .pgoff = linear_page_index(vma, vmf->address),
> .virtual_address =
> - (void __user *)(fe->address & PAGE_MASK),
> + (void __user *)(vmf->address & PAGE_MASK),
> .flags = FAULT_FLAG_WRITE | FAULT_FLAG_MKWRITE,
> };
> int ret;
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - ret = vma->vm_ops->pfn_mkwrite(vma, &vmf);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + ret = vma->vm_ops->pfn_mkwrite(vma, &vmf2);
> if (ret & VM_FAULT_ERROR)
> return ret;
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> /*
> * We might have raced with another page fault while we
> * released the pte_offset_map_lock.
> */
> - if (!pte_same(*fe->pte, orig_pte)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
> }
> - return wp_page_reuse(fe, orig_pte, NULL, 0, 0);
> + return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
> }
>
> -static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> +static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
> struct page *old_page)
> - __releases(fe->ptl)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> int page_mkwrite = 0;
>
> get_page(old_page);
> @@ -2315,8 +2316,8 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
> int tmp;
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - tmp = do_page_mkwrite(vma, old_page, fe->address);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + tmp = do_page_mkwrite(vma, old_page, vmf->address);
> if (unlikely(!tmp || (tmp &
> (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
> put_page(old_page);
> @@ -2328,18 +2329,18 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> * they did, we just return, as we can count on the
> * MMU to tell us if they didn't also make it writable.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_same(*fe->pte, orig_pte)) {
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> unlock_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> put_page(old_page);
> return 0;
> }
> page_mkwrite = 1;
> }
>
> - return wp_page_reuse(fe, orig_pte, old_page, page_mkwrite, 1);
> + return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
> }
>
> /*
> @@ -2360,13 +2361,13 @@ static int wp_page_shared(struct fault_env *fe, pte_t orig_pte,
> * but allow concurrent faults), with pte both mapped and locked.
> * We return with mmap_sem still held, but pte unmapped and unlocked.
> */
> -static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> - __releases(fe->ptl)
> +static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
> + __releases(vmf->ptl)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *old_page;
>
> - old_page = vm_normal_page(vma, fe->address, orig_pte);
> + old_page = vm_normal_page(vma, vmf->address, orig_pte);
> if (!old_page) {
> /*
> * VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
> @@ -2377,10 +2378,10 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> */
> if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
> (VM_WRITE|VM_SHARED))
> - return wp_pfn_shared(fe, orig_pte);
> + return wp_pfn_shared(vmf, orig_pte);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return wp_page_copy(fe, orig_pte, old_page);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return wp_page_copy(vmf, orig_pte, old_page);
> }
>
> /*
> @@ -2391,13 +2392,13 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> int total_mapcount;
> if (!trylock_page(old_page)) {
> get_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> lock_page(old_page);
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
> - fe->address, &fe->ptl);
> - if (!pte_same(*fe->pte, orig_pte)) {
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_same(*vmf->pte, orig_pte)) {
> unlock_page(old_page);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> put_page(old_page);
> return 0;
> }
> @@ -2415,12 +2416,12 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> page_move_anon_rmap(old_page, vma);
> }
> unlock_page(old_page);
> - return wp_page_reuse(fe, orig_pte, old_page, 0, 0);
> + return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
> }
> unlock_page(old_page);
> } else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
> (VM_WRITE|VM_SHARED))) {
> - return wp_page_shared(fe, orig_pte, old_page);
> + return wp_page_shared(vmf, orig_pte, old_page);
> }
>
> /*
> @@ -2428,8 +2429,8 @@ static int do_wp_page(struct fault_env *fe, pte_t orig_pte)
> */
> get_page(old_page);
>
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return wp_page_copy(fe, orig_pte, old_page);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return wp_page_copy(vmf, orig_pte, old_page);
> }
>
> static void unmap_mapping_range_vma(struct vm_area_struct *vma,
> @@ -2517,9 +2518,9 @@ EXPORT_SYMBOL(unmap_mapping_range);
> * We return with the mmap_sem locked or unlocked in the same cases
> * as does filemap_fault().
> */
> -int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> +int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page, *swapcache;
> struct mem_cgroup *memcg;
> swp_entry_t entry;
> @@ -2528,17 +2529,18 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> int exclusive = 0;
> int ret = 0;
>
> - if (!pte_unmap_same(vma->vm_mm, fe->pmd, fe->pte, orig_pte))
> + if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
> goto out;
>
> entry = pte_to_swp_entry(orig_pte);
> if (unlikely(non_swap_entry(entry))) {
> if (is_migration_entry(entry)) {
> - migration_entry_wait(vma->vm_mm, fe->pmd, fe->address);
> + migration_entry_wait(vma->vm_mm, vmf->pmd,
> + vmf->address);
> } else if (is_hwpoison_entry(entry)) {
> ret = VM_FAULT_HWPOISON;
> } else {
> - print_bad_pte(vma, fe->address, orig_pte, NULL);
> + print_bad_pte(vma, vmf->address, orig_pte, NULL);
> ret = VM_FAULT_SIGBUS;
> }
> goto out;
> @@ -2546,16 +2548,16 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> delayacct_set_flag(DELAYACCT_PF_SWAPIN);
> page = lookup_swap_cache(entry);
> if (!page) {
> - page = swapin_readahead(entry,
> - GFP_HIGHUSER_MOVABLE, vma, fe->address);
> + page = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vma,
> + vmf->address);
> if (!page) {
> /*
> * Back out if somebody else faulted in this pte
> * while we released the pte lock.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd,
> - fe->address, &fe->ptl);
> - if (likely(pte_same(*fe->pte, orig_pte)))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (likely(pte_same(*vmf->pte, orig_pte)))
> ret = VM_FAULT_OOM;
> delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
> goto unlock;
> @@ -2577,7 +2579,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> }
>
> swapcache = page;
> - locked = lock_page_or_retry(page, vma->vm_mm, fe->flags);
> + locked = lock_page_or_retry(page, vma->vm_mm, vmf->flags);
>
> delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
> if (!locked) {
> @@ -2594,7 +2596,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> if (unlikely(!PageSwapCache(page) || page_private(page) != entry.val))
> goto out_page;
>
> - page = ksm_might_need_to_copy(page, vma, fe->address);
> + page = ksm_might_need_to_copy(page, vma, vmf->address);
> if (unlikely(!page)) {
> ret = VM_FAULT_OOM;
> page = swapcache;
> @@ -2610,9 +2612,9 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> /*
> * Back out if somebody else already faulted in this pte.
> */
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, orig_pte)))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, orig_pte)))
> goto out_nomap;
>
> if (unlikely(!PageUptodate(page))) {
> @@ -2633,22 +2635,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> dec_mm_counter_fast(vma->vm_mm, MM_SWAPENTS);
> pte = mk_pte(page, vma->vm_page_prot);
> - if ((fe->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
> + if ((vmf->flags & FAULT_FLAG_WRITE) && reuse_swap_page(page, NULL)) {
> pte = maybe_mkwrite(pte_mkdirty(pte), vma);
> - fe->flags &= ~FAULT_FLAG_WRITE;
> + vmf->flags &= ~FAULT_FLAG_WRITE;
> ret |= VM_FAULT_WRITE;
> exclusive = RMAP_EXCLUSIVE;
> }
> flush_icache_page(vma, page);
> if (pte_swp_soft_dirty(orig_pte))
> pte = pte_mksoft_dirty(pte);
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
> if (page == swapcache) {
> - do_page_add_anon_rmap(page, vma, fe->address, exclusive);
> + do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
> mem_cgroup_commit_charge(page, memcg, true, false);
> activate_page(page);
> } else { /* ksm created a completely new copy */
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> }
> @@ -2671,22 +2673,22 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
> put_page(swapcache);
> }
>
> - if (fe->flags & FAULT_FLAG_WRITE) {
> - ret |= do_wp_page(fe, pte);
> + if (vmf->flags & FAULT_FLAG_WRITE) {
> + ret |= do_wp_page(vmf, pte);
> if (ret & VM_FAULT_ERROR)
> ret &= VM_FAULT_ERROR;
> goto out;
> }
>
> /* No need to invalidate - it was non-present before */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out:
> return ret;
> out_nomap:
> mem_cgroup_cancel_charge(page, memcg, false);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out_page:
> unlock_page(page);
> out_release:
> @@ -2737,9 +2739,9 @@ static inline int check_stack_guard_page(struct vm_area_struct *vma, unsigned lo
> * but allow concurrent faults), and pte mapped but not yet locked.
> * We return with mmap_sem still held, but pte unmapped and unlocked.
> */
> -static int do_anonymous_page(struct fault_env *fe)
> +static int do_anonymous_page(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct mem_cgroup *memcg;
> struct page *page;
> pte_t entry;
> @@ -2749,7 +2751,7 @@ static int do_anonymous_page(struct fault_env *fe)
> return VM_FAULT_SIGBUS;
>
> /* Check if we need to add a guard page to the stack */
> - if (check_stack_guard_page(vma, fe->address) < 0)
> + if (check_stack_guard_page(vma, vmf->address) < 0)
> return VM_FAULT_SIGSEGV;
>
> /*
> @@ -2762,26 +2764,26 @@ static int do_anonymous_page(struct fault_env *fe)
> *
> * Here we only have down_read(mmap_sem).
> */
> - if (pte_alloc(vma->vm_mm, fe->pmd, fe->address))
> + if (pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))
> return VM_FAULT_OOM;
>
> /* See the comment in pte_alloc_one_map() */
> - if (unlikely(pmd_trans_unstable(fe->pmd)))
> + if (unlikely(pmd_trans_unstable(vmf->pmd)))
> return 0;
>
> /* Use the zero-page for reads */
> - if (!(fe->flags & FAULT_FLAG_WRITE) &&
> + if (!(vmf->flags & FAULT_FLAG_WRITE) &&
> !mm_forbids_zeropage(vma->vm_mm)) {
> - entry = pte_mkspecial(pfn_pte(my_zero_pfn(fe->address),
> + entry = pte_mkspecial(pfn_pte(my_zero_pfn(vmf->address),
> vma->vm_page_prot));
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_none(*fe->pte))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> + vmf->address, &vmf->ptl);
> + if (!pte_none(*vmf->pte))
> goto unlock;
> /* Deliver the page fault to userland, check inside PT lock */
> if (userfaultfd_missing(vma)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> - return handle_userfault(fe, VM_UFFD_MISSING);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return handle_userfault(vmf, VM_UFFD_MISSING);
> }
> goto setpte;
> }
> @@ -2789,7 +2791,7 @@ static int do_anonymous_page(struct fault_env *fe)
> /* Allocate our own private page. */
> if (unlikely(anon_vma_prepare(vma)))
> goto oom;
> - page = alloc_zeroed_user_highpage_movable(vma, fe->address);
> + page = alloc_zeroed_user_highpage_movable(vma, vmf->address);
> if (!page)
> goto oom;
>
> @@ -2807,30 +2809,30 @@ static int do_anonymous_page(struct fault_env *fe)
> if (vma->vm_flags & VM_WRITE)
> entry = pte_mkwrite(pte_mkdirty(entry));
>
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> - if (!pte_none(*fe->pte))
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + if (!pte_none(*vmf->pte))
> goto release;
>
> /* Deliver the page fault to userland, check inside PT lock */
> if (userfaultfd_missing(vma)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> mem_cgroup_cancel_charge(page, memcg, false);
> put_page(page);
> - return handle_userfault(fe, VM_UFFD_MISSING);
> + return handle_userfault(vmf, VM_UFFD_MISSING);
> }
>
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> setpte:
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
>
> /* No need to invalidate - it was non-present before */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> release:
> mem_cgroup_cancel_charge(page, memcg, false);
> @@ -2847,62 +2849,62 @@ static int do_anonymous_page(struct fault_env *fe)
> * released depending on flags and vma->vm_ops->fault() return value.
> * See filemap_fault() and __lock_page_retry().
> */
> -static int __do_fault(struct fault_env *fe, pgoff_t pgoff,
> +static int __do_fault(struct vm_fault *vmf, pgoff_t pgoff,
> struct page *cow_page, struct page **page, void **entry)
> {
> - struct vm_area_struct *vma = fe->vma;
> - struct vm_fault vmf;
> + struct vm_area_struct *vma = vmf->vma;
> + struct vm_fault vmf2;
> int ret;
>
> - vmf.virtual_address = (void __user *)(fe->address & PAGE_MASK);
> - vmf.pgoff = pgoff;
> - vmf.flags = fe->flags;
> - vmf.page = NULL;
> - vmf.gfp_mask = __get_fault_gfp_mask(vma);
> - vmf.cow_page = cow_page;
> + vmf2.virtual_address = (void __user *)(vmf->address & PAGE_MASK);
> + vmf2.pgoff = pgoff;
> + vmf2.flags = vmf->flags;
> + vmf2.page = NULL;
> + vmf2.gfp_mask = __get_fault_gfp_mask(vma);
> + vmf2.cow_page = cow_page;
>
> - ret = vma->vm_ops->fault(vma, &vmf);
> + ret = vma->vm_ops->fault(vma, &vmf2);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
> if (ret & VM_FAULT_DAX_LOCKED) {
> - *entry = vmf.entry;
> + *entry = vmf2.entry;
> return ret;
> }
>
> - if (unlikely(PageHWPoison(vmf.page))) {
> + if (unlikely(PageHWPoison(vmf2.page))) {
> if (ret & VM_FAULT_LOCKED)
> - unlock_page(vmf.page);
> - put_page(vmf.page);
> + unlock_page(vmf2.page);
> + put_page(vmf2.page);
> return VM_FAULT_HWPOISON;
> }
>
> if (unlikely(!(ret & VM_FAULT_LOCKED)))
> - lock_page(vmf.page);
> + lock_page(vmf2.page);
> else
> - VM_BUG_ON_PAGE(!PageLocked(vmf.page), vmf.page);
> + VM_BUG_ON_PAGE(!PageLocked(vmf2.page), vmf2.page);
>
> - *page = vmf.page;
> + *page = vmf2.page;
> return ret;
> }
>
> -static int pte_alloc_one_map(struct fault_env *fe)
> +static int pte_alloc_one_map(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
>
> - if (!pmd_none(*fe->pmd))
> + if (!pmd_none(*vmf->pmd))
> goto map_pte;
> - if (fe->prealloc_pte) {
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd))) {
> - spin_unlock(fe->ptl);
> + if (vmf->prealloc_pte) {
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd))) {
> + spin_unlock(vmf->ptl);
> goto map_pte;
> }
>
> atomic_long_inc(&vma->vm_mm->nr_ptes);
> - pmd_populate(vma->vm_mm, fe->pmd, fe->prealloc_pte);
> - spin_unlock(fe->ptl);
> - fe->prealloc_pte = 0;
> - } else if (unlikely(pte_alloc(vma->vm_mm, fe->pmd, fe->address))) {
> + pmd_populate(vma->vm_mm, vmf->pmd, vmf->prealloc_pte);
> + spin_unlock(vmf->ptl);
> + vmf->prealloc_pte = 0;
> + } else if (unlikely(pte_alloc(vma->vm_mm, vmf->pmd, vmf->address))) {
> return VM_FAULT_OOM;
> }
> map_pte:
> @@ -2917,11 +2919,11 @@ static int pte_alloc_one_map(struct fault_env *fe)
> * through an atomic read in C, which is what pmd_trans_unstable()
> * provides.
> */
> - if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
> + if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
> return VM_FAULT_NOPAGE;
>
> - fe->pte = pte_offset_map_lock(vma->vm_mm, fe->pmd, fe->address,
> - &fe->ptl);
> + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> return 0;
> }
>
> @@ -2939,11 +2941,11 @@ static inline bool transhuge_vma_suitable(struct vm_area_struct *vma,
> return true;
> }
>
> -static int do_set_pmd(struct fault_env *fe, struct page *page)
> +static int do_set_pmd(struct vm_fault *vmf, struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - bool write = fe->flags & FAULT_FLAG_WRITE;
> - unsigned long haddr = fe->address & HPAGE_PMD_MASK;
> + struct vm_area_struct *vma = vmf->vma;
> + bool write = vmf->flags & FAULT_FLAG_WRITE;
> + unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
> pmd_t entry;
> int i, ret;
>
> @@ -2953,8 +2955,8 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> ret = VM_FAULT_FALLBACK;
> page = compound_head(page);
>
> - fe->ptl = pmd_lock(vma->vm_mm, fe->pmd);
> - if (unlikely(!pmd_none(*fe->pmd)))
> + vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
> + if (unlikely(!pmd_none(*vmf->pmd)))
> goto out;
>
> for (i = 0; i < HPAGE_PMD_NR; i++)
> @@ -2967,19 +2969,19 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> add_mm_counter(vma->vm_mm, MM_FILEPAGES, HPAGE_PMD_NR);
> page_add_file_rmap(page, true);
>
> - set_pmd_at(vma->vm_mm, haddr, fe->pmd, entry);
> + set_pmd_at(vma->vm_mm, haddr, vmf->pmd, entry);
>
> - update_mmu_cache_pmd(vma, haddr, fe->pmd);
> + update_mmu_cache_pmd(vma, haddr, vmf->pmd);
>
> /* fault is handled */
> ret = 0;
> count_vm_event(THP_FILE_MAPPED);
> out:
> - spin_unlock(fe->ptl);
> + spin_unlock(vmf->ptl);
> return ret;
> }
> #else
> -static int do_set_pmd(struct fault_env *fe, struct page *page)
> +static int do_set_pmd(struct vm_fault *vmf, struct page *page)
> {
> BUILD_BUG();
> return 0;
> @@ -2990,41 +2992,42 @@ static int do_set_pmd(struct fault_env *fe, struct page *page)
> * alloc_set_pte - setup new PTE entry for given page and add reverse page
> * mapping. If needed, the fucntion allocates page table or use pre-allocated.
> *
> - * @fe: fault environment
> + * @vmf: fault environment
> * @memcg: memcg to charge page (only for private mappings)
> * @page: page to map
> *
> - * Caller must take care of unlocking fe->ptl, if fe->pte is non-NULL on return.
> + * Caller must take care of unlocking vmf->ptl, if vmf->pte is non-NULL on
> + * return.
> *
> * Target users are page handler itself and implementations of
> * vm_ops->map_pages.
> */
> -int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> +int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page)
> {
> - struct vm_area_struct *vma = fe->vma;
> - bool write = fe->flags & FAULT_FLAG_WRITE;
> + struct vm_area_struct *vma = vmf->vma;
> + bool write = vmf->flags & FAULT_FLAG_WRITE;
> pte_t entry;
> int ret;
>
> - if (pmd_none(*fe->pmd) && PageTransCompound(page) &&
> + if (pmd_none(*vmf->pmd) && PageTransCompound(page) &&
> IS_ENABLED(CONFIG_TRANSPARENT_HUGE_PAGECACHE)) {
> /* THP on COW? */
> VM_BUG_ON_PAGE(memcg, page);
>
> - ret = do_set_pmd(fe, page);
> + ret = do_set_pmd(vmf, page);
> if (ret != VM_FAULT_FALLBACK)
> return ret;
> }
>
> - if (!fe->pte) {
> - ret = pte_alloc_one_map(fe);
> + if (!vmf->pte) {
> + ret = pte_alloc_one_map(vmf);
> if (ret)
> return ret;
> }
>
> /* Re-check under ptl */
> - if (unlikely(!pte_none(*fe->pte)))
> + if (unlikely(!pte_none(*vmf->pte)))
> return VM_FAULT_NOPAGE;
>
> flush_icache_page(vma, page);
> @@ -3034,17 +3037,17 @@ int alloc_set_pte(struct fault_env *fe, struct mem_cgroup *memcg,
> /* copy-on-write page */
> if (write && !(vma->vm_flags & VM_SHARED)) {
> inc_mm_counter_fast(vma->vm_mm, MM_ANONPAGES);
> - page_add_new_anon_rmap(page, vma, fe->address, false);
> + page_add_new_anon_rmap(page, vma, vmf->address, false);
> mem_cgroup_commit_charge(page, memcg, false, false);
> lru_cache_add_active_or_unevictable(page, vma);
> } else {
> inc_mm_counter_fast(vma->vm_mm, mm_counter_file(page));
> page_add_file_rmap(page, false);
> }
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, entry);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, entry);
>
> /* no need to invalidate: a not-present page won't be cached */
> - update_mmu_cache(vma, fe->address, fe->pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
>
> return 0;
> }
> @@ -3113,17 +3116,17 @@ late_initcall(fault_around_debugfs);
> * fault_around_pages() value (and therefore to page order). This way it's
> * easier to guarantee that we don't cross page table boundaries.
> */
> -static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
> +static int do_fault_around(struct vm_fault *vmf, pgoff_t start_pgoff)
> {
> - unsigned long address = fe->address, nr_pages, mask;
> + unsigned long address = vmf->address, nr_pages, mask;
> pgoff_t end_pgoff;
> int off, ret = 0;
>
> nr_pages = READ_ONCE(fault_around_bytes) >> PAGE_SHIFT;
> mask = ~(nr_pages * PAGE_SIZE - 1) & PAGE_MASK;
>
> - fe->address = max(address & mask, fe->vma->vm_start);
> - off = ((address - fe->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
> + vmf->address = max(address & mask, vmf->vma->vm_start);
> + off = ((address - vmf->address) >> PAGE_SHIFT) & (PTRS_PER_PTE - 1);
> start_pgoff -= off;
>
> /*
> @@ -3131,49 +3134,51 @@ static int do_fault_around(struct fault_env *fe, pgoff_t start_pgoff)
> * or fault_around_pages() from start_pgoff, depending what is nearest.
> */
> end_pgoff = start_pgoff -
> - ((fe->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
> + ((vmf->address >> PAGE_SHIFT) & (PTRS_PER_PTE - 1)) +
> PTRS_PER_PTE - 1;
> - end_pgoff = min3(end_pgoff, vma_pages(fe->vma) + fe->vma->vm_pgoff - 1,
> + end_pgoff = min3(end_pgoff,
> + vma_pages(vmf->vma) + vmf->vma->vm_pgoff - 1,
> start_pgoff + nr_pages - 1);
>
> - if (pmd_none(*fe->pmd)) {
> - fe->prealloc_pte = pte_alloc_one(fe->vma->vm_mm, fe->address);
> - if (!fe->prealloc_pte)
> + if (pmd_none(*vmf->pmd)) {
> + vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm,
> + vmf->address);
> + if (!vmf->prealloc_pte)
> goto out;
> smp_wmb(); /* See comment in __pte_alloc() */
> }
>
> - fe->vma->vm_ops->map_pages(fe, start_pgoff, end_pgoff);
> + vmf->vma->vm_ops->map_pages(vmf, start_pgoff, end_pgoff);
>
> /* preallocated pagetable is unused: free it */
> - if (fe->prealloc_pte) {
> - pte_free(fe->vma->vm_mm, fe->prealloc_pte);
> - fe->prealloc_pte = 0;
> + if (vmf->prealloc_pte) {
> + pte_free(vmf->vma->vm_mm, vmf->prealloc_pte);
> + vmf->prealloc_pte = 0;
> }
> /* Huge page is mapped? Page fault is solved */
> - if (pmd_trans_huge(*fe->pmd)) {
> + if (pmd_trans_huge(*vmf->pmd)) {
> ret = VM_FAULT_NOPAGE;
> goto out;
> }
>
> /* ->map_pages() haven't done anything useful. Cold page cache? */
> - if (!fe->pte)
> + if (!vmf->pte)
> goto out;
>
> /* check if the page fault is solved */
> - fe->pte -= (fe->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
> - if (!pte_none(*fe->pte))
> + vmf->pte -= (vmf->address >> PAGE_SHIFT) - (address >> PAGE_SHIFT);
> + if (!pte_none(*vmf->pte))
> ret = VM_FAULT_NOPAGE;
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> out:
> - fe->address = address;
> - fe->pte = NULL;
> + vmf->address = address;
> + vmf->pte = NULL;
> return ret;
> }
>
> -static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_read_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page;
> int ret = 0;
>
> @@ -3183,27 +3188,27 @@ static int do_read_fault(struct fault_env *fe, pgoff_t pgoff)
> * something).
> */
> if (vma->vm_ops->map_pages && fault_around_bytes >> PAGE_SHIFT > 1) {
> - ret = do_fault_around(fe, pgoff);
> + ret = do_fault_around(vmf, pgoff);
> if (ret)
> return ret;
> }
>
> - ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
> + ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
>
> - ret |= alloc_set_pte(fe, NULL, fault_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, NULL, fault_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> unlock_page(fault_page);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> put_page(fault_page);
> return ret;
> }
>
> -static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_cow_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page, *new_page;
> void *fault_entry;
> struct mem_cgroup *memcg;
> @@ -3212,7 +3217,7 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> if (unlikely(anon_vma_prepare(vma)))
> return VM_FAULT_OOM;
>
> - new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, fe->address);
> + new_page = alloc_page_vma(GFP_HIGHUSER_MOVABLE, vma, vmf->address);
> if (!new_page)
> return VM_FAULT_OOM;
>
> @@ -3222,17 +3227,17 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> return VM_FAULT_OOM;
> }
>
> - ret = __do_fault(fe, pgoff, new_page, &fault_page, &fault_entry);
> + ret = __do_fault(vmf, pgoff, new_page, &fault_page, &fault_entry);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> goto uncharge_out;
>
> if (!(ret & VM_FAULT_DAX_LOCKED))
> - copy_user_highpage(new_page, fault_page, fe->address, vma);
> + copy_user_highpage(new_page, fault_page, vmf->address, vma);
> __SetPageUptodate(new_page);
>
> - ret |= alloc_set_pte(fe, memcg, new_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, memcg, new_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (!(ret & VM_FAULT_DAX_LOCKED)) {
> unlock_page(fault_page);
> put_page(fault_page);
> @@ -3248,15 +3253,15 @@ static int do_cow_fault(struct fault_env *fe, pgoff_t pgoff)
> return ret;
> }
>
> -static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> +static int do_shared_fault(struct vm_fault *vmf, pgoff_t pgoff)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *fault_page;
> struct address_space *mapping;
> int dirtied = 0;
> int ret, tmp;
>
> - ret = __do_fault(fe, pgoff, NULL, &fault_page, NULL);
> + ret = __do_fault(vmf, pgoff, NULL, &fault_page, NULL);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE | VM_FAULT_RETRY)))
> return ret;
>
> @@ -3266,7 +3271,7 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> */
> if (vma->vm_ops->page_mkwrite) {
> unlock_page(fault_page);
> - tmp = do_page_mkwrite(vma, fault_page, fe->address);
> + tmp = do_page_mkwrite(vma, fault_page, vmf->address);
> if (unlikely(!tmp ||
> (tmp & (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
> put_page(fault_page);
> @@ -3274,9 +3279,9 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> }
> }
>
> - ret |= alloc_set_pte(fe, NULL, fault_page);
> - if (fe->pte)
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + ret |= alloc_set_pte(vmf, NULL, fault_page);
> + if (vmf->pte)
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (unlikely(ret & (VM_FAULT_ERROR | VM_FAULT_NOPAGE |
> VM_FAULT_RETRY))) {
> unlock_page(fault_page);
> @@ -3314,19 +3319,19 @@ static int do_shared_fault(struct fault_env *fe, pgoff_t pgoff)
> * The mmap_sem may have been released depending on flags and our
> * return value. See filemap_fault() and __lock_page_or_retry().
> */
> -static int do_fault(struct fault_env *fe)
> +static int do_fault(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> - pgoff_t pgoff = linear_page_index(vma, fe->address);
> + struct vm_area_struct *vma = vmf->vma;
> + pgoff_t pgoff = linear_page_index(vma, vmf->address);
>
> /* The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */
> if (!vma->vm_ops->fault)
> return VM_FAULT_SIGBUS;
> - if (!(fe->flags & FAULT_FLAG_WRITE))
> - return do_read_fault(fe, pgoff);
> + if (!(vmf->flags & FAULT_FLAG_WRITE))
> + return do_read_fault(vmf, pgoff);
> if (!(vma->vm_flags & VM_SHARED))
> - return do_cow_fault(fe, pgoff);
> - return do_shared_fault(fe, pgoff);
> + return do_cow_fault(vmf, pgoff);
> + return do_shared_fault(vmf, pgoff);
> }
>
> static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
> @@ -3344,9 +3349,9 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
> return mpol_misplaced(page, vma, addr);
> }
>
> -static int do_numa_page(struct fault_env *fe, pte_t pte)
> +static int do_numa_page(struct vm_fault *vmf, pte_t pte)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> struct page *page = NULL;
> int page_nid = -1;
> int last_cpupid;
> @@ -3364,10 +3369,10 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> * page table entry is not accessible, so there would be no
> * concurrent hardware modifications to the PTE.
> */
> - fe->ptl = pte_lockptr(vma->vm_mm, fe->pmd);
> - spin_lock(fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, pte))) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd);
> + spin_lock(vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, pte))) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> goto out;
> }
>
> @@ -3376,18 +3381,18 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> pte = pte_mkyoung(pte);
> if (was_writable)
> pte = pte_mkwrite(pte);
> - set_pte_at(vma->vm_mm, fe->address, fe->pte, pte);
> - update_mmu_cache(vma, fe->address, fe->pte);
> + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
> + update_mmu_cache(vma, vmf->address, vmf->pte);
>
> - page = vm_normal_page(vma, fe->address, pte);
> + page = vm_normal_page(vma, vmf->address, pte);
> if (!page) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> /* TODO: handle PTE-mapped THP */
> if (PageCompound(page)) {
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> @@ -3411,9 +3416,9 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
>
> last_cpupid = page_cpupid_last(page);
> page_nid = page_to_nid(page);
> - target_nid = numa_migrate_prep(page, vma, fe->address, page_nid,
> + target_nid = numa_migrate_prep(page, vma, vmf->address, page_nid,
> &flags);
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> if (target_nid == -1) {
> put_page(page);
> goto out;
> @@ -3433,28 +3438,28 @@ static int do_numa_page(struct fault_env *fe, pte_t pte)
> return 0;
> }
>
> -static int create_huge_pmd(struct fault_env *fe)
> +static int create_huge_pmd(struct vm_fault *vmf)
> {
> - struct vm_area_struct *vma = fe->vma;
> + struct vm_area_struct *vma = vmf->vma;
> if (vma_is_anonymous(vma))
> - return do_huge_pmd_anonymous_page(fe);
> + return do_huge_pmd_anonymous_page(vmf);
> if (vma->vm_ops->pmd_fault)
> - return vma->vm_ops->pmd_fault(vma, fe->address, fe->pmd,
> - fe->flags);
> + return vma->vm_ops->pmd_fault(vma, vmf->address, vmf->pmd,
> + vmf->flags);
> return VM_FAULT_FALLBACK;
> }
>
> -static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
> +static int wp_huge_pmd(struct vm_fault *vmf, pmd_t orig_pmd)
> {
> - if (vma_is_anonymous(fe->vma))
> - return do_huge_pmd_wp_page(fe, orig_pmd);
> - if (fe->vma->vm_ops->pmd_fault)
> - return fe->vma->vm_ops->pmd_fault(fe->vma, fe->address, fe->pmd,
> - fe->flags);
> + if (vma_is_anonymous(vmf->vma))
> + return do_huge_pmd_wp_page(vmf, orig_pmd);
> + if (vmf->vma->vm_ops->pmd_fault)
> + return vmf->vma->vm_ops->pmd_fault(vmf->vma, vmf->address,
> + vmf->pmd, vmf->flags);
>
> /* COW handled on pte level: split pmd */
> - VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
> - split_huge_pmd(fe->vma, fe->pmd, fe->address);
> + VM_BUG_ON_VMA(vmf->vma->vm_flags & VM_SHARED, vmf->vma);
> + split_huge_pmd(vmf->vma, vmf->pmd, vmf->address);
>
> return VM_FAULT_FALLBACK;
> }
> @@ -3479,21 +3484,21 @@ static inline bool vma_is_accessible(struct vm_area_struct *vma)
> * The mmap_sem may have been released depending on flags and our return value.
> * See filemap_fault() and __lock_page_or_retry().
> */
> -static int handle_pte_fault(struct fault_env *fe)
> +static int handle_pte_fault(struct vm_fault *vmf)
> {
> pte_t entry;
>
> - if (unlikely(pmd_none(*fe->pmd))) {
> + if (unlikely(pmd_none(*vmf->pmd))) {
> /*
> * Leave __pte_alloc() until later: because vm_ops->fault may
> * want to allocate huge page, and if we expose page table
> * for an instant, it will be difficult to retract from
> * concurrent faults and from rmap lookups.
> */
> - fe->pte = NULL;
> + vmf->pte = NULL;
> } else {
> /* See comment in pte_alloc_one_map() */
> - if (pmd_trans_unstable(fe->pmd) || pmd_devmap(*fe->pmd))
> + if (pmd_trans_unstable(vmf->pmd) || pmd_devmap(*vmf->pmd))
> return 0;
> /*
> * A regular pmd is established and it can't morph into a huge
> @@ -3501,9 +3506,9 @@ static int handle_pte_fault(struct fault_env *fe)
> * mmap_sem read mode and khugepaged takes it in write mode.
> * So now it's safe to run pte_offset_map().
> */
> - fe->pte = pte_offset_map(fe->pmd, fe->address);
> + vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
>
> - entry = *fe->pte;
> + entry = *vmf->pte;
>
> /*
> * some architectures can have larger ptes than wordsize,
> @@ -3515,37 +3520,37 @@ static int handle_pte_fault(struct fault_env *fe)
> */
> barrier();
> if (pte_none(entry)) {
> - pte_unmap(fe->pte);
> - fe->pte = NULL;
> + pte_unmap(vmf->pte);
> + vmf->pte = NULL;
> }
> }
>
> - if (!fe->pte) {
> - if (vma_is_anonymous(fe->vma))
> - return do_anonymous_page(fe);
> + if (!vmf->pte) {
> + if (vma_is_anonymous(vmf->vma))
> + return do_anonymous_page(vmf);
> else
> - return do_fault(fe);
> + return do_fault(vmf);
> }
>
> if (!pte_present(entry))
> - return do_swap_page(fe, entry);
> + return do_swap_page(vmf, entry);
>
> - if (pte_protnone(entry) && vma_is_accessible(fe->vma))
> - return do_numa_page(fe, entry);
> + if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
> + return do_numa_page(vmf, entry);
>
> - fe->ptl = pte_lockptr(fe->vma->vm_mm, fe->pmd);
> - spin_lock(fe->ptl);
> - if (unlikely(!pte_same(*fe->pte, entry)))
> + vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
> + spin_lock(vmf->ptl);
> + if (unlikely(!pte_same(*vmf->pte, entry)))
> goto unlock;
> - if (fe->flags & FAULT_FLAG_WRITE) {
> + if (vmf->flags & FAULT_FLAG_WRITE) {
> if (!pte_write(entry))
> - return do_wp_page(fe, entry);
> + return do_wp_page(vmf, entry);
> entry = pte_mkdirty(entry);
> }
> entry = pte_mkyoung(entry);
> - if (ptep_set_access_flags(fe->vma, fe->address, fe->pte, entry,
> - fe->flags & FAULT_FLAG_WRITE)) {
> - update_mmu_cache(fe->vma, fe->address, fe->pte);
> + if (ptep_set_access_flags(vmf->vma, vmf->address, vmf->pte, entry,
> + vmf->flags & FAULT_FLAG_WRITE)) {
> + update_mmu_cache(vmf->vma, vmf->address, vmf->pte);
> } else {
> /*
> * This is needed only for protection faults but the arch code
> @@ -3553,11 +3558,11 @@ static int handle_pte_fault(struct fault_env *fe)
> * This still avoids useless tlb flushes for .text page faults
> * with threads.
> */
> - if (fe->flags & FAULT_FLAG_WRITE)
> - flush_tlb_fix_spurious_fault(fe->vma, fe->address);
> + if (vmf->flags & FAULT_FLAG_WRITE)
> + flush_tlb_fix_spurious_fault(vmf->vma, vmf->address);
> }
> unlock:
> - pte_unmap_unlock(fe->pte, fe->ptl);
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> return 0;
> }
>
> @@ -3570,7 +3575,7 @@ static int handle_pte_fault(struct fault_env *fe)
> static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
> unsigned int flags)
> {
> - struct fault_env fe = {
> + struct vm_fault vmf = {
> .vma = vma,
> .address = address,
> .flags = flags,
> @@ -3583,35 +3588,35 @@ static int __handle_mm_fault(struct vm_area_struct *vma, unsigned long address,
> pud = pud_alloc(mm, pgd, address);
> if (!pud)
> return VM_FAULT_OOM;
> - fe.pmd = pmd_alloc(mm, pud, address);
> - if (!fe.pmd)
> + vmf.pmd = pmd_alloc(mm, pud, address);
> + if (!vmf.pmd)
> return VM_FAULT_OOM;
> - if (pmd_none(*fe.pmd) && transparent_hugepage_enabled(vma)) {
> - int ret = create_huge_pmd(&fe);
> + if (pmd_none(*vmf.pmd) && transparent_hugepage_enabled(vma)) {
> + int ret = create_huge_pmd(&vmf);
> if (!(ret & VM_FAULT_FALLBACK))
> return ret;
> } else {
> - pmd_t orig_pmd = *fe.pmd;
> + pmd_t orig_pmd = *vmf.pmd;
> int ret;
>
> barrier();
> if (pmd_trans_huge(orig_pmd) || pmd_devmap(orig_pmd)) {
> if (pmd_protnone(orig_pmd) && vma_is_accessible(vma))
> - return do_huge_pmd_numa_page(&fe, orig_pmd);
> + return do_huge_pmd_numa_page(&vmf, orig_pmd);
>
> - if ((fe.flags & FAULT_FLAG_WRITE) &&
> + if ((vmf.flags & FAULT_FLAG_WRITE) &&
> !pmd_write(orig_pmd)) {
> - ret = wp_huge_pmd(&fe, orig_pmd);
> + ret = wp_huge_pmd(&vmf, orig_pmd);
> if (!(ret & VM_FAULT_FALLBACK))
> return ret;
> } else {
> - huge_pmd_set_accessed(&fe, orig_pmd);
> + huge_pmd_set_accessed(&vmf, orig_pmd);
> return 0;
> }
> }
> }
>
> - return handle_pte_fault(&fe);
> + return handle_pte_fault(&vmf);
> }
>
> /*
> diff --git a/mm/nommu.c b/mm/nommu.c
> index 8b8faaf2a9e9..077d0dbe4c28 100644
> --- a/mm/nommu.c
> +++ b/mm/nommu.c
> @@ -1801,7 +1801,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> }
> EXPORT_SYMBOL(filemap_fault);
>
> -void filemap_map_pages(struct fault_env *fe,
> +void filemap_map_pages(struct vm_fault *vmf,
> pgoff_t start_pgoff, pgoff_t end_pgoff)
> {
> BUG();
> --
> 2.6.6
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
2016-11-04 4:24 ` Jan Kara
@ 2016-11-15 21:55 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 21:55 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> Every single user of vmf->virtual_address typed that entry to unsigned
> long before doing anything with it so the type of virtual_address does
> not really provide us any additional safety. Just use masked
> vmf->address which already has the appropriate type.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> arch/x86/entry/vdso/vma.c | 4 ++--
> drivers/char/agp/alpha-agp.c | 2 +-
> drivers/char/mspec.c | 2 +-
> drivers/dax/dax.c | 2 +-
> drivers/gpu/drm/armada/armada_gem.c | 2 +-
> drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> drivers/gpu/drm/gma500/gem.c | 5 ++---
> drivers/gpu/drm/i915/i915_gem.c | 2 +-
> drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> drivers/gpu/drm/tegra/gem.c | 4 ++--
> drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> drivers/misc/cxl/context.c | 2 +-
> drivers/misc/sgi-gru/grumain.c | 2 +-
> drivers/staging/android/ion/ion.c | 2 +-
> drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> drivers/xen/privcmd.c | 2 +-
> fs/dax.c | 4 ++--
> include/linux/mm.h | 2 --
> mm/memory.c | 7 +++----
> 27 files changed, 59 insertions(+), 65 deletions(-)
>
> diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> index 06254467e4dd..e8a31fffcdda 100644
> --- a/arch/powerpc/platforms/cell/spufs/file.c
> +++ b/arch/powerpc/platforms/cell/spufs/file.c
> @@ -236,7 +236,7 @@ static int
> spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> {
> struct spu_context *ctx = vma->vm_file->private_data;
> - unsigned long address = (unsigned long)vmf->virtual_address;
> + unsigned long address = vmf->address & PAGE_MASK;
These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
need sub-page address, do we?
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
@ 2016-11-15 21:55 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 21:55 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> Every single user of vmf->virtual_address typed that entry to unsigned
> long before doing anything with it so the type of virtual_address does
> not really provide us any additional safety. Just use masked
> vmf->address which already has the appropriate type.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> arch/x86/entry/vdso/vma.c | 4 ++--
> drivers/char/agp/alpha-agp.c | 2 +-
> drivers/char/mspec.c | 2 +-
> drivers/dax/dax.c | 2 +-
> drivers/gpu/drm/armada/armada_gem.c | 2 +-
> drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> drivers/gpu/drm/gma500/gem.c | 5 ++---
> drivers/gpu/drm/i915/i915_gem.c | 2 +-
> drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> drivers/gpu/drm/tegra/gem.c | 4 ++--
> drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> drivers/misc/cxl/context.c | 2 +-
> drivers/misc/sgi-gru/grumain.c | 2 +-
> drivers/staging/android/ion/ion.c | 2 +-
> drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> drivers/xen/privcmd.c | 2 +-
> fs/dax.c | 4 ++--
> include/linux/mm.h | 2 --
> mm/memory.c | 7 +++----
> 27 files changed, 59 insertions(+), 65 deletions(-)
>
> diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> index 06254467e4dd..e8a31fffcdda 100644
> --- a/arch/powerpc/platforms/cell/spufs/file.c
> +++ b/arch/powerpc/platforms/cell/spufs/file.c
> @@ -236,7 +236,7 @@ static int
> spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> {
> struct spu_context *ctx = vma->vm_file->private_data;
> - unsigned long address = (unsigned long)vmf->virtual_address;
> + unsigned long address = vmf->address & PAGE_MASK;
These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
need sub-page address, do we?
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately
2016-11-04 4:24 ` Jan Kara
@ 2016-11-15 22:01 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:01 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:59AM +0100, Jan Kara wrote:
> struct vm_fault has already pgoff entry. Use it instead of passing pgoff
> as a separate argument and then assigning it later.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Okay, makes sense.
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately
@ 2016-11-15 22:01 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:01 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:24:59AM +0100, Jan Kara wrote:
> struct vm_fault has already pgoff entry. Use it instead of passing pgoff
> as a separate argument and then assigning it later.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Okay, makes sense.
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault()
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:05 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:05 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:00AM +0100, Jan Kara wrote:
> Instead of creating another vm_fault structure, use the one passed to
> __do_fault() for passing arguments into fault handler.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault()
@ 2016-11-15 22:05 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:05 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:00AM +0100, Jan Kara wrote:
> Instead of creating another vm_fault structure, use the one passed to
> __do_fault() for passing arguments into fault handler.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 05/21] mm: Trim __do_fault() arguments
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:10 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:10 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:01AM +0100, Jan Kara wrote:
> static int do_cow_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> - struct page *fault_page, *new_page;
> - void *fault_entry;
> + struct page *new_page;
Why not get rid of new_page too?
Otherwise makes sense:
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 05/21] mm: Trim __do_fault() arguments
@ 2016-11-15 22:10 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:10 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:01AM +0100, Jan Kara wrote:
> static int do_cow_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> - struct page *fault_page, *new_page;
> - void *fault_entry;
> + struct page *new_page;
Why not get rid of new_page too?
Otherwise makes sense:
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared()
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:10 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:10 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:02AM +0100, Jan Kara wrote:
> Instead of creating another vm_fault structure, use the one passed to
> wp_pfn_shared() for passing arguments into pfn_mkwrite handler.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared()
@ 2016-11-15 22:10 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:10 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:02AM +0100, Jan Kara wrote:
> Instead of creating another vm_fault structure, use the one passed to
> wp_pfn_shared() for passing arguments into pfn_mkwrite handler.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 07/21] mm: Add orig_pte field into vm_fault
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:14 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:14 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:03AM +0100, Jan Kara wrote:
> Add orig_pte field to vm_fault structure to allow ->page_mkwrite
> handlers to fully handle the fault. This also allows us to save some
> passing of extra arguments around.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-15 22:14 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:14 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:03AM +0100, Jan Kara wrote:
> Add orig_pte field to vm_fault structure to allow ->page_mkwrite
> handlers to fully handle the fault. This also allows us to save some
> passing of extra arguments around.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:20 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:20 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:04AM +0100, Jan Kara wrote:
> To allow full handling of COW faults add memcg field to struct vm_fault
> and a return value of ->fault() handler meaning that COW fault is fully
> handled and memcg charge must not be canceled. This will allow us to
> remove knowledge about special DAX locking from the generic fault code.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers
@ 2016-11-15 22:20 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:20 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:04AM +0100, Jan Kara wrote:
> To allow full handling of COW faults add memcg field to struct vm_fault
> and a return value of ->fault() handler meaning that COW fault is fully
> handled and memcg charge must not be canceled. This will allow us to
> remove knowledge about special DAX locking from the generic fault code.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 09/21] mm: Factor out functionality to finish page faults
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:21 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:21 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:05AM +0100, Jan Kara wrote:
> Introduce function finish_fault() as a helper function for finishing
> page faults. It is rather thin wrapper around alloc_set_pte() but since
> we'd want to call this from DAX code or filesystems, it is still useful
> to avoid some boilerplate code.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 09/21] mm: Factor out functionality to finish page faults
@ 2016-11-15 22:21 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:21 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:05AM +0100, Jan Kara wrote:
> Introduce function finish_fault() as a helper function for finishing
> page faults. It is rather thin wrapper around alloc_set_pte() but since
> we'd want to call this from DAX code or filesystems, it is still useful
> to avoid some boilerplate code.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 10/21] mm: Move handling of COW faults into DAX code
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:22 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:22 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:06AM +0100, Jan Kara wrote:
> Move final handling of COW faults from generic code into DAX fault
> handler. That way generic code doesn't have to be aware of peculiarities
> of DAX locking so remove that knowledge and make locking functions
> private to fs/dax.c.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
On core-mm bits:
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 10/21] mm: Move handling of COW faults into DAX code
@ 2016-11-15 22:22 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:22 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:06AM +0100, Jan Kara wrote:
> Move final handling of COW faults from generic code into DAX fault
> handler. That way generic code doesn't have to be aware of peculiarities
> of DAX locking so remove that knowledge and make locking functions
> private to fs/dax.c.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
On core-mm bits:
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
2016-11-04 4:25 ` Jan Kara
(?)
(?)
@ 2016-11-15 22:28 ` Kirill A. Shutemov
2016-11-16 13:29 ` Jan Kara
-1 siblings, 1 reply; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:28 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:07AM +0100, Jan Kara wrote:
> We don't check whether vma->vm_ops is NULL in do_shared_fault() so
> there's hardly any point in checking it in wp_page_shared() or
> wp_pfn_shared() which get called only for shared file mappings as well.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Well, I'm not sure about this.
do_shared_fault() doesn't have the check since we checked it upper by
stack: see vma_is_anonymous() in handle_pte_fault().
In principal, it should be fine. But random crappy driver has potential to
blow it up.
> ---
> mm/memory.c | 4 ++--
> 1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 7be96a43d5ac..26b2858e6a12 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2275,7 +2275,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
>
> - if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
> + if (vma->vm_ops->pfn_mkwrite) {
> int ret;
>
> pte_unmap_unlock(vmf->pte, vmf->ptl);
> @@ -2305,7 +2305,7 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
>
> get_page(old_page);
>
> - if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
> + if (vma->vm_ops->page_mkwrite) {
> int tmp;
>
> pte_unmap_unlock(vmf->pte, vmf->ptl);
> --
> 2.6.6
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org. For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 12/21] mm: Factor out common parts of write fault handling
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:30 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:30 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:08AM +0100, Jan Kara wrote:
> Currently we duplicate handling of shared write faults in
> wp_page_reuse() and do_shared_fault(). Factor them out into a common
> function.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 12/21] mm: Factor out common parts of write fault handling
@ 2016-11-15 22:30 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:30 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:08AM +0100, Jan Kara wrote:
> Currently we duplicate handling of shared write faults in
> wp_page_reuse() and do_shared_fault(). Factor them out into a common
> function.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:40 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:40 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:09AM +0100, Jan Kara wrote:
> We will need more information in the ->page_mkwrite() helper for DAX to
> be able to fully finish faults there. Pass vm_fault structure to
> do_page_mkwrite() and use it there so that information propagates
> properly from upper layers.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> mm/memory.c | 19 +++++++------------
> 1 file changed, 7 insertions(+), 12 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 4da66c984c2c..c89f99c270bc 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
> *
> * We do this without the lock held, so that it can sleep if it needs to.
> */
> -static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
> - unsigned long address)
> +static int do_page_mkwrite(struct vm_fault *vmf)
> {
> - struct vm_fault vmf;
> int ret;
> + struct page *page = vmf->page;
>
> - vmf.address = address;
> - vmf.pgoff = page->index;
> - vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
> - vmf.gfp_mask = __get_fault_gfp_mask(vma);
> - vmf.page = page;
> - vmf.cow_page = NULL;
> + vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
This can be destructive: we loose rest of the flags here. It's probably
okay in current state of the code, but may be should restore them before
return from do_page_mkwrite()?
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
@ 2016-11-15 22:40 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:40 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:09AM +0100, Jan Kara wrote:
> We will need more information in the ->page_mkwrite() helper for DAX to
> be able to fully finish faults there. Pass vm_fault structure to
> do_page_mkwrite() and use it there so that information propagates
> properly from upper layers.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> mm/memory.c | 19 +++++++------------
> 1 file changed, 7 insertions(+), 12 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index 4da66c984c2c..c89f99c270bc 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
> *
> * We do this without the lock held, so that it can sleep if it needs to.
> */
> -static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
> - unsigned long address)
> +static int do_page_mkwrite(struct vm_fault *vmf)
> {
> - struct vm_fault vmf;
> int ret;
> + struct page *page = vmf->page;
>
> - vmf.address = address;
> - vmf.pgoff = page->index;
> - vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
> - vmf.gfp_mask = __get_fault_gfp_mask(vma);
> - vmf.page = page;
> - vmf.cow_page = NULL;
> + vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
This can be destructive: we loose rest of the flags here. It's probably
okay in current state of the code, but may be should restore them before
return from do_page_mkwrite()?
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 14/21] mm: Use vmf->page during WP faults
2016-11-04 4:25 ` Jan Kara
(?)
(?)
@ 2016-11-15 22:42 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:42 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:10AM +0100, Jan Kara wrote:
> So far we set vmf->page during WP faults only when we needed to pass it
> to the ->page_mkwrite handler. Set it in all the cases now and use that
> instead of passing page pointer explicitly around.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:44 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:44 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:11AM +0100, Jan Kara wrote:
> wp_page_reuse() handles write shared faults which is needed only in
> wp_page_shared(). Move the handling only into that location to make
> wp_page_reuse() simpler and avoid a strange situation when we sometimes
> pass in locked page, sometimes unlocked etc.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site
@ 2016-11-15 22:44 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:44 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:11AM +0100, Jan Kara wrote:
> wp_page_reuse() handles write shared faults which is needed only in
> wp_page_shared(). Move the handling only into that location to make
> wp_page_reuse() simpler and avoid a strange situation when we sometimes
> pass in locked page, sometimes unlocked etc.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
2016-11-04 4:25 ` Jan Kara
@ 2016-11-15 22:52 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:52 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:12AM +0100, Jan Kara wrote:
> Provide a helper function for finishing write faults due to PTE being
> read-only. The helper will be used by DAX to avoid the need of
> complicating generic MM code with DAX locking specifics.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> include/linux/mm.h | 1 +
> mm/memory.c | 67 ++++++++++++++++++++++++++++++++----------------------
> 2 files changed, 41 insertions(+), 27 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index fb128beecdac..685ff1c57f2b 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -615,6 +615,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
> int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page);
> int finish_fault(struct vm_fault *vmf);
> +int finish_mkwrite_fault(struct vm_fault *vmf);
> #endif
>
> /*
> diff --git a/mm/memory.c b/mm/memory.c
> index 06aba4203104..1517ff91c743 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2270,6 +2270,38 @@ static int wp_page_copy(struct vm_fault *vmf)
> return VM_FAULT_OOM;
> }
>
> +/**
> + * finish_mkwrite_fault - finish page fault for a shared mapping, making PTE
> + * writeable once the page is prepared
> + *
> + * @vmf: structure describing the fault
> + *
> + * This function handles all that is needed to finish a write page fault in a
> + * shared mapping due to PTE being read-only once the mapped page is prepared.
> + * It handles locking of PTE and modifying it. The function returns
> + * VM_FAULT_WRITE on success, 0 when PTE got changed before we acquired PTE
> + * lock.
> + *
> + * The function expects the page to be locked or other protection against
> + * concurrent faults / writeback (such as DAX radix tree locks).
> + */
> +int finish_mkwrite_fault(struct vm_fault *vmf)
> +{
> + WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
> + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + /*
> + * We might have raced with another page fault while we released the
> + * pte_offset_map_lock.
> + */
> + if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return 0;
> + }
> + wp_page_reuse(vmf);
> + return VM_FAULT_WRITE;
> +}
> +
> /*
> * Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
> * mapping
> @@ -2286,16 +2318,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
> ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
> if (ret & VM_FAULT_ERROR)
> return ret;
> - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> - vmf->address, &vmf->ptl);
> - /*
> - * We might have raced with another page fault while we
> - * released the pte_offset_map_lock.
> - */
> - if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> - pte_unmap_unlock(vmf->pte, vmf->ptl);
> - return 0;
> - }
> + return finish_mkwrite_fault(vmf);
> }
> wp_page_reuse(vmf);
> return VM_FAULT_WRITE;
> @@ -2305,7 +2328,6 @@ static int wp_page_shared(struct vm_fault *vmf)
> __releases(vmf->ptl)
> {
> struct vm_area_struct *vma = vmf->vma;
> - int page_mkwrite = 0;
>
> get_page(vmf->page);
>
> @@ -2319,26 +2341,17 @@ static int wp_page_shared(struct vm_fault *vmf)
> put_page(vmf->page);
> return tmp;
> }
> - /*
> - * Since we dropped the lock we need to revalidate
> - * the PTE as someone else may have changed it. If
> - * they did, we just return, as we can count on the
> - * MMU to tell us if they didn't also make it writable.
> - */
> - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> - vmf->address, &vmf->ptl);
> - if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> + tmp = finish_mkwrite_fault(vmf);
> + if (unlikely(!tmp || (tmp &
> + (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
Looks like the second part of condition is never true here, right? Not
that it would matter, having the next patch in the queue.
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
@ 2016-11-15 22:52 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:52 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:12AM +0100, Jan Kara wrote:
> Provide a helper function for finishing write faults due to PTE being
> read-only. The helper will be used by DAX to avoid the need of
> complicating generic MM code with DAX locking specifics.
>
> Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> include/linux/mm.h | 1 +
> mm/memory.c | 67 ++++++++++++++++++++++++++++++++----------------------
> 2 files changed, 41 insertions(+), 27 deletions(-)
>
> diff --git a/include/linux/mm.h b/include/linux/mm.h
> index fb128beecdac..685ff1c57f2b 100644
> --- a/include/linux/mm.h
> +++ b/include/linux/mm.h
> @@ -615,6 +615,7 @@ static inline pte_t maybe_mkwrite(pte_t pte, struct vm_area_struct *vma)
> int alloc_set_pte(struct vm_fault *vmf, struct mem_cgroup *memcg,
> struct page *page);
> int finish_fault(struct vm_fault *vmf);
> +int finish_mkwrite_fault(struct vm_fault *vmf);
> #endif
>
> /*
> diff --git a/mm/memory.c b/mm/memory.c
> index 06aba4203104..1517ff91c743 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -2270,6 +2270,38 @@ static int wp_page_copy(struct vm_fault *vmf)
> return VM_FAULT_OOM;
> }
>
> +/**
> + * finish_mkwrite_fault - finish page fault for a shared mapping, making PTE
> + * writeable once the page is prepared
> + *
> + * @vmf: structure describing the fault
> + *
> + * This function handles all that is needed to finish a write page fault in a
> + * shared mapping due to PTE being read-only once the mapped page is prepared.
> + * It handles locking of PTE and modifying it. The function returns
> + * VM_FAULT_WRITE on success, 0 when PTE got changed before we acquired PTE
> + * lock.
> + *
> + * The function expects the page to be locked or other protection against
> + * concurrent faults / writeback (such as DAX radix tree locks).
> + */
> +int finish_mkwrite_fault(struct vm_fault *vmf)
> +{
> + WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED));
> + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address,
> + &vmf->ptl);
> + /*
> + * We might have raced with another page fault while we released the
> + * pte_offset_map_lock.
> + */
> + if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> + pte_unmap_unlock(vmf->pte, vmf->ptl);
> + return 0;
> + }
> + wp_page_reuse(vmf);
> + return VM_FAULT_WRITE;
> +}
> +
> /*
> * Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
> * mapping
> @@ -2286,16 +2318,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
> ret = vma->vm_ops->pfn_mkwrite(vma, vmf);
> if (ret & VM_FAULT_ERROR)
> return ret;
> - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> - vmf->address, &vmf->ptl);
> - /*
> - * We might have raced with another page fault while we
> - * released the pte_offset_map_lock.
> - */
> - if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> - pte_unmap_unlock(vmf->pte, vmf->ptl);
> - return 0;
> - }
> + return finish_mkwrite_fault(vmf);
> }
> wp_page_reuse(vmf);
> return VM_FAULT_WRITE;
> @@ -2305,7 +2328,6 @@ static int wp_page_shared(struct vm_fault *vmf)
> __releases(vmf->ptl)
> {
> struct vm_area_struct *vma = vmf->vma;
> - int page_mkwrite = 0;
>
> get_page(vmf->page);
>
> @@ -2319,26 +2341,17 @@ static int wp_page_shared(struct vm_fault *vmf)
> put_page(vmf->page);
> return tmp;
> }
> - /*
> - * Since we dropped the lock we need to revalidate
> - * the PTE as someone else may have changed it. If
> - * they did, we just return, as we can count on the
> - * MMU to tell us if they didn't also make it writable.
> - */
> - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> - vmf->address, &vmf->ptl);
> - if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> + tmp = finish_mkwrite_fault(vmf);
> + if (unlikely(!tmp || (tmp &
> + (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
Looks like the second part of condition is never true here, right? Not
that it would matter, having the next patch in the queue.
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 17/21] mm: Change return values of finish_mkwrite_fault()
2016-11-04 4:25 ` Jan Kara
(?)
(?)
@ 2016-11-15 22:57 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-15 22:57 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:13AM +0100, Jan Kara wrote:
> Currently finish_mkwrite_fault() returns 0 when PTE got changed before
> we acquired PTE lock and VM_FAULT_WRITE when we succeeded in modifying
> the PTE. This is somewhat confusing since 0 generally means success, it
> is also inconsistent with finish_fault() which returns 0 on success.
> Change finish_mkwrite_fault() to return 0 on success and VM_FAULT_NOPAGE
> when PTE changed. Practically, there should be no behavioral difference
> since we bail out from the fault the same way regardless whether we
> return 0, VM_FAULT_NOPAGE, or VM_FAULT_WRITE. Also note that
> VM_FAULT_WRITE has no effect for shared mappings since the only two
> places that check it - KSM and GUP - care about private mappings only.
> Generally the meaning of VM_FAULT_WRITE for shared mappings is not well
> defined and we should probably clean that up.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Sounds right.
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-15 21:50 ` Kirill A. Shutemov
@ 2016-11-16 10:51 ` Peter Zijlstra
-1 siblings, 0 replies; 124+ messages in thread
From: Peter Zijlstra @ 2016-11-16 10:51 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, Andrew Morton,
Kirill A. Shutemov
On Wed, Nov 16, 2016 at 12:50:21AM +0300, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> > Currently we have two different structures for passing fault information
> > around - struct vm_fault and struct fault_env. DAX will need more
> > information in struct vm_fault to handle its faults so the content of
> > that structure would become event closer to fault_env. Furthermore it
> > would need to generate struct fault_env to be able to call some of the
> > generic functions. So at this point I don't think there's much use in
> > keeping these two structures separate. Just embed into struct vm_fault
> > all that is needed to use it for both purposes.
> >
> > Signed-off-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
>
> I'm not necessary dislike this, but I remember Peter had objections before
> when I proposed something similar.
>
> Peter?
My objection was that it would be a layering violation. The 'filesystem'
shouldn't know about page-tables, all it should do is return a page
matching a specific offset.
So fault_env manages the core vm parts and has the page-table bits in,
vm_fault manages the filesystem interface and gets us a page given an
offset.
Now, I'm entirely out of touch wrt DAX, so I've not idea what that
needs/wants.
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index a92c8d73aeaf..657eb69eb87e 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> > * pgoff should be used in favour of virtual_address, if possible.
> > */
> > struct vm_fault {
> > + struct vm_area_struct *vma; /* Target VMA */
> > unsigned int flags; /* FAULT_FLAG_xxx flags */
> > gfp_t gfp_mask; /* gfp mask to be used for allocations */
> > pgoff_t pgoff; /* Logical page offset based on vma */
> > + unsigned long address; /* Faulting virtual address */
> > + void __user *virtual_address; /* Faulting virtual address masked by
> > + * PAGE_MASK */
> > + pmd_t *pmd; /* Pointer to pmd entry matching
> > + * the 'address'
> > + */
> >
> > struct page *cow_page; /* Handler may choose to COW */
> > struct page *page; /* ->fault handlers should return a
Egads, horrific commenting style that :-)
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-16 10:51 ` Peter Zijlstra
0 siblings, 0 replies; 124+ messages in thread
From: Peter Zijlstra @ 2016-11-16 10:51 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 12:50:21AM +0300, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> > Currently we have two different structures for passing fault information
> > around - struct vm_fault and struct fault_env. DAX will need more
> > information in struct vm_fault to handle its faults so the content of
> > that structure would become event closer to fault_env. Furthermore it
> > would need to generate struct fault_env to be able to call some of the
> > generic functions. So at this point I don't think there's much use in
> > keeping these two structures separate. Just embed into struct vm_fault
> > all that is needed to use it for both purposes.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
>
> I'm not necessary dislike this, but I remember Peter had objections before
> when I proposed something similar.
>
> Peter?
My objection was that it would be a layering violation. The 'filesystem'
shouldn't know about page-tables, all it should do is return a page
matching a specific offset.
So fault_env manages the core vm parts and has the page-table bits in,
vm_fault manages the filesystem interface and gets us a page given an
offset.
Now, I'm entirely out of touch wrt DAX, so I've not idea what that
needs/wants.
> > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > index a92c8d73aeaf..657eb69eb87e 100644
> > --- a/include/linux/mm.h
> > +++ b/include/linux/mm.h
> > @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> > * pgoff should be used in favour of virtual_address, if possible.
> > */
> > struct vm_fault {
> > + struct vm_area_struct *vma; /* Target VMA */
> > unsigned int flags; /* FAULT_FLAG_xxx flags */
> > gfp_t gfp_mask; /* gfp mask to be used for allocations */
> > pgoff_t pgoff; /* Logical page offset based on vma */
> > + unsigned long address; /* Faulting virtual address */
> > + void __user *virtual_address; /* Faulting virtual address masked by
> > + * PAGE_MASK */
> > + pmd_t *pmd; /* Pointer to pmd entry matching
> > + * the 'address'
> > + */
> >
> > struct page *cow_page; /* Handler may choose to COW */
> > struct page *page; /* ->fault handlers should return a
Egads, horrific commenting style that :-)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-16 10:51 ` Peter Zijlstra
@ 2016-11-16 11:01 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:01 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Jan Kara, linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, Kirill A. Shutemov,
Andrew Morton, Kirill A. Shutemov
On Wed 16-11-16 11:51:32, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 12:50:21AM +0300, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> > > Currently we have two different structures for passing fault information
> > > around - struct vm_fault and struct fault_env. DAX will need more
> > > information in struct vm_fault to handle its faults so the content of
> > > that structure would become event closer to fault_env. Furthermore it
> > > would need to generate struct fault_env to be able to call some of the
> > > generic functions. So at this point I don't think there's much use in
> > > keeping these two structures separate. Just embed into struct vm_fault
> > > all that is needed to use it for both purposes.
> > >
> > > Signed-off-by: Jan Kara <jack-AlSwsSmVLrQ@public.gmane.org>
> >
> > I'm not necessary dislike this, but I remember Peter had objections before
> > when I proposed something similar.
> >
> > Peter?
>
> My objection was that it would be a layering violation. The 'filesystem'
> shouldn't know about page-tables, all it should do is return a page
> matching a specific offset.
>
> So fault_env manages the core vm parts and has the page-table bits in,
> vm_fault manages the filesystem interface and gets us a page given an
> offset.
>
> Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> needs/wants.
Yeah, DAX does not have 'struct page' for its pages so it directly installs
PFNs in the page tables. As a result it needs to know about page tables and
stuff. Now I've abstracted knowledge about that into helper functions back
in mm/ but still we need to pass the information through the ->fault handler
into those helpers and vm_fault structure is simply natural for that.
So far we have tried to avoid that but the result was not pretty (special
return codes from DAX ->fault handlers essentially leaking information
about DAX internal locking into mm/ code to direct generic mm code to do
the right thing for DAX).
Honza
> > > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > > index a92c8d73aeaf..657eb69eb87e 100644
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> > > * pgoff should be used in favour of virtual_address, if possible.
> > > */
> > > struct vm_fault {
> > > + struct vm_area_struct *vma; /* Target VMA */
> > > unsigned int flags; /* FAULT_FLAG_xxx flags */
> > > gfp_t gfp_mask; /* gfp mask to be used for allocations */
> > > pgoff_t pgoff; /* Logical page offset based on vma */
> > > + unsigned long address; /* Faulting virtual address */
> > > + void __user *virtual_address; /* Faulting virtual address masked by
> > > + * PAGE_MASK */
> > > + pmd_t *pmd; /* Pointer to pmd entry matching
> > > + * the 'address'
> > > + */
> > >
> > > struct page *cow_page; /* Handler may choose to COW */
> > > struct page *page; /* ->fault handlers should return a
>
> Egads, horrific commenting style that :-)
--
Jan Kara <jack-IBi9RG/b67k@public.gmane.org>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-16 11:01 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:01 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Kirill A. Shutemov, Jan Kara, linux-mm, linux-fsdevel,
linux-nvdimm, Andrew Morton, Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 11:51:32, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 12:50:21AM +0300, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> > > Currently we have two different structures for passing fault information
> > > around - struct vm_fault and struct fault_env. DAX will need more
> > > information in struct vm_fault to handle its faults so the content of
> > > that structure would become event closer to fault_env. Furthermore it
> > > would need to generate struct fault_env to be able to call some of the
> > > generic functions. So at this point I don't think there's much use in
> > > keeping these two structures separate. Just embed into struct vm_fault
> > > all that is needed to use it for both purposes.
> > >
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> >
> > I'm not necessary dislike this, but I remember Peter had objections before
> > when I proposed something similar.
> >
> > Peter?
>
> My objection was that it would be a layering violation. The 'filesystem'
> shouldn't know about page-tables, all it should do is return a page
> matching a specific offset.
>
> So fault_env manages the core vm parts and has the page-table bits in,
> vm_fault manages the filesystem interface and gets us a page given an
> offset.
>
> Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> needs/wants.
Yeah, DAX does not have 'struct page' for its pages so it directly installs
PFNs in the page tables. As a result it needs to know about page tables and
stuff. Now I've abstracted knowledge about that into helper functions back
in mm/ but still we need to pass the information through the ->fault handler
into those helpers and vm_fault structure is simply natural for that.
So far we have tried to avoid that but the result was not pretty (special
return codes from DAX ->fault handlers essentially leaking information
about DAX internal locking into mm/ code to direct generic mm code to do
the right thing for DAX).
Honza
> > > diff --git a/include/linux/mm.h b/include/linux/mm.h
> > > index a92c8d73aeaf..657eb69eb87e 100644
> > > --- a/include/linux/mm.h
> > > +++ b/include/linux/mm.h
> > > @@ -292,10 +292,16 @@ extern pgprot_t protection_map[16];
> > > * pgoff should be used in favour of virtual_address, if possible.
> > > */
> > > struct vm_fault {
> > > + struct vm_area_struct *vma; /* Target VMA */
> > > unsigned int flags; /* FAULT_FLAG_xxx flags */
> > > gfp_t gfp_mask; /* gfp mask to be used for allocations */
> > > pgoff_t pgoff; /* Logical page offset based on vma */
> > > + unsigned long address; /* Faulting virtual address */
> > > + void __user *virtual_address; /* Faulting virtual address masked by
> > > + * PAGE_MASK */
> > > + pmd_t *pmd; /* Pointer to pmd entry matching
> > > + * the 'address'
> > > + */
> > >
> > > struct page *cow_page; /* Handler may choose to COW */
> > > struct page *page; /* ->fault handlers should return a
>
> Egads, horrific commenting style that :-)
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
2016-11-15 21:55 ` Kirill A. Shutemov
@ 2016-11-16 11:05 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:05 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton,
Kirill A. Shutemov
On Wed 16-11-16 00:55:31, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> > Every single user of vmf->virtual_address typed that entry to unsigned
> > long before doing anything with it so the type of virtual_address does
> > not really provide us any additional safety. Just use masked
> > vmf->address which already has the appropriate type.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> > arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> > arch/x86/entry/vdso/vma.c | 4 ++--
> > drivers/char/agp/alpha-agp.c | 2 +-
> > drivers/char/mspec.c | 2 +-
> > drivers/dax/dax.c | 2 +-
> > drivers/gpu/drm/armada/armada_gem.c | 2 +-
> > drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> > drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> > drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> > drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> > drivers/gpu/drm/gma500/gem.c | 5 ++---
> > drivers/gpu/drm/i915/i915_gem.c | 2 +-
> > drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> > drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> > drivers/gpu/drm/tegra/gem.c | 4 ++--
> > drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> > drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> > drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> > drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> > drivers/misc/cxl/context.c | 2 +-
> > drivers/misc/sgi-gru/grumain.c | 2 +-
> > drivers/staging/android/ion/ion.c | 2 +-
> > drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> > drivers/xen/privcmd.c | 2 +-
> > fs/dax.c | 4 ++--
> > include/linux/mm.h | 2 --
> > mm/memory.c | 7 +++----
> > 27 files changed, 59 insertions(+), 65 deletions(-)
> >
> > diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> > index 06254467e4dd..e8a31fffcdda 100644
> > --- a/arch/powerpc/platforms/cell/spufs/file.c
> > +++ b/arch/powerpc/platforms/cell/spufs/file.c
> > @@ -236,7 +236,7 @@ static int
> > spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> > {
> > struct spu_context *ctx = vma->vm_file->private_data;
> > - unsigned long address = (unsigned long)vmf->virtual_address;
> > + unsigned long address = vmf->address & PAGE_MASK;
>
> These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
> need sub-page address, do we?
Usually not AFAICT but I was not really sure in some cases (e.g. is
vm_insert_pfn() and friends really safe to call with unaligned address) so
I've decided to do a simple search-and-replace for such a wide-scale patch.
Later we can remove unnecessary masking from places where we are sure it is
not needed.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
@ 2016-11-16 11:05 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:05 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 00:55:31, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> > Every single user of vmf->virtual_address typed that entry to unsigned
> > long before doing anything with it so the type of virtual_address does
> > not really provide us any additional safety. Just use masked
> > vmf->address which already has the appropriate type.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> > arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> > arch/x86/entry/vdso/vma.c | 4 ++--
> > drivers/char/agp/alpha-agp.c | 2 +-
> > drivers/char/mspec.c | 2 +-
> > drivers/dax/dax.c | 2 +-
> > drivers/gpu/drm/armada/armada_gem.c | 2 +-
> > drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> > drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> > drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> > drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> > drivers/gpu/drm/gma500/gem.c | 5 ++---
> > drivers/gpu/drm/i915/i915_gem.c | 2 +-
> > drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> > drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> > drivers/gpu/drm/tegra/gem.c | 4 ++--
> > drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> > drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> > drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> > drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> > drivers/misc/cxl/context.c | 2 +-
> > drivers/misc/sgi-gru/grumain.c | 2 +-
> > drivers/staging/android/ion/ion.c | 2 +-
> > drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> > drivers/xen/privcmd.c | 2 +-
> > fs/dax.c | 4 ++--
> > include/linux/mm.h | 2 --
> > mm/memory.c | 7 +++----
> > 27 files changed, 59 insertions(+), 65 deletions(-)
> >
> > diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> > index 06254467e4dd..e8a31fffcdda 100644
> > --- a/arch/powerpc/platforms/cell/spufs/file.c
> > +++ b/arch/powerpc/platforms/cell/spufs/file.c
> > @@ -236,7 +236,7 @@ static int
> > spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> > {
> > struct spu_context *ctx = vma->vm_file->private_data;
> > - unsigned long address = (unsigned long)vmf->virtual_address;
> > + unsigned long address = vmf->address & PAGE_MASK;
>
> These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
> need sub-page address, do we?
Usually not AFAICT but I was not really sure in some cases (e.g. is
vm_insert_pfn() and friends really safe to call with unaligned address) so
I've decided to do a simple search-and-replace for such a wide-scale patch.
Later we can remove unnecessary masking from places where we are sure it is
not needed.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-16 10:51 ` Peter Zijlstra
(?)
(?)
@ 2016-11-16 11:13 ` Kirill A. Shutemov
-1 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-16 11:13 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 11:51:32AM +0100, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 12:50:21AM +0300, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:24:57AM +0100, Jan Kara wrote:
> > > Currently we have two different structures for passing fault information
> > > around - struct vm_fault and struct fault_env. DAX will need more
> > > information in struct vm_fault to handle its faults so the content of
> > > that structure would become event closer to fault_env. Furthermore it
> > > would need to generate struct fault_env to be able to call some of the
> > > generic functions. So at this point I don't think there's much use in
> > > keeping these two structures separate. Just embed into struct vm_fault
> > > all that is needed to use it for both purposes.
> > >
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> >
> > I'm not necessary dislike this, but I remember Peter had objections before
> > when I proposed something similar.
> >
> > Peter?
>
> My objection was that it would be a layering violation. The 'filesystem'
> shouldn't know about page-tables, all it should do is return a page
> matching a specific offset.
Well, this layering violation is already there (blame me): see
vm_ops->map_pages(). :P
> So fault_env manages the core vm parts and has the page-table bits in,
> vm_fault manages the filesystem interface and gets us a page given an
> offset.
>
> Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> needs/wants.
I think we are better off with one structure. It streamlines code in quite
a few places.
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
2016-11-16 11:05 ` Jan Kara
(?)
@ 2016-11-16 11:32 ` Kirill A. Shutemov
2016-11-16 11:55 ` Jan Kara
-1 siblings, 1 reply; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-16 11:32 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 12:05:05PM +0100, Jan Kara wrote:
> On Wed 16-11-16 00:55:31, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> > > Every single user of vmf->virtual_address typed that entry to unsigned
> > > long before doing anything with it so the type of virtual_address does
> > > not really provide us any additional safety. Just use masked
> > > vmf->address which already has the appropriate type.
> > >
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> > > ---
> > > arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> > > arch/x86/entry/vdso/vma.c | 4 ++--
> > > drivers/char/agp/alpha-agp.c | 2 +-
> > > drivers/char/mspec.c | 2 +-
> > > drivers/dax/dax.c | 2 +-
> > > drivers/gpu/drm/armada/armada_gem.c | 2 +-
> > > drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> > > drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> > > drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> > > drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> > > drivers/gpu/drm/gma500/gem.c | 5 ++---
> > > drivers/gpu/drm/i915/i915_gem.c | 2 +-
> > > drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> > > drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> > > drivers/gpu/drm/tegra/gem.c | 4 ++--
> > > drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> > > drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> > > drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> > > drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> > > drivers/misc/cxl/context.c | 2 +-
> > > drivers/misc/sgi-gru/grumain.c | 2 +-
> > > drivers/staging/android/ion/ion.c | 2 +-
> > > drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> > > drivers/xen/privcmd.c | 2 +-
> > > fs/dax.c | 4 ++--
> > > include/linux/mm.h | 2 --
> > > mm/memory.c | 7 +++----
> > > 27 files changed, 59 insertions(+), 65 deletions(-)
> > >
> > > diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> > > index 06254467e4dd..e8a31fffcdda 100644
> > > --- a/arch/powerpc/platforms/cell/spufs/file.c
> > > +++ b/arch/powerpc/platforms/cell/spufs/file.c
> > > @@ -236,7 +236,7 @@ static int
> > > spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> > > {
> > > struct spu_context *ctx = vma->vm_file->private_data;
> > > - unsigned long address = (unsigned long)vmf->virtual_address;
> > > + unsigned long address = vmf->address & PAGE_MASK;
> >
> > These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
> > need sub-page address, do we?
>
> Usually not AFAICT but I was not really sure in some cases (e.g. is
> vm_insert_pfn() and friends really safe to call with unaligned address) so
> I've decided to do a simple search-and-replace for such a wide-scale patch.
> Later we can remove unnecessary masking from places where we are sure it is
> not needed.
I think it should be fine just to zero out ~PAGE_MASK bits of the address
on entrance into handle_mm_fault() or where vmf is initialized.
I believe we only care about these bits on SIGBUS/SEGV/etc. which is
handled by caller.
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
2016-11-16 11:32 ` Kirill A. Shutemov
@ 2016-11-16 11:55 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:55 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton,
Kirill A. Shutemov
On Wed 16-11-16 14:32:48, Kirill A. Shutemov wrote:
> On Wed, Nov 16, 2016 at 12:05:05PM +0100, Jan Kara wrote:
> > On Wed 16-11-16 00:55:31, Kirill A. Shutemov wrote:
> > > On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> > > > Every single user of vmf->virtual_address typed that entry to unsigned
> > > > long before doing anything with it so the type of virtual_address does
> > > > not really provide us any additional safety. Just use masked
> > > > vmf->address which already has the appropriate type.
> > > >
> > > > Signed-off-by: Jan Kara <jack@suse.cz>
> > > > ---
> > > > arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> > > > arch/x86/entry/vdso/vma.c | 4 ++--
> > > > drivers/char/agp/alpha-agp.c | 2 +-
> > > > drivers/char/mspec.c | 2 +-
> > > > drivers/dax/dax.c | 2 +-
> > > > drivers/gpu/drm/armada/armada_gem.c | 2 +-
> > > > drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> > > > drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> > > > drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> > > > drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> > > > drivers/gpu/drm/gma500/gem.c | 5 ++---
> > > > drivers/gpu/drm/i915/i915_gem.c | 2 +-
> > > > drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> > > > drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> > > > drivers/gpu/drm/tegra/gem.c | 4 ++--
> > > > drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> > > > drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> > > > drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> > > > drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> > > > drivers/misc/cxl/context.c | 2 +-
> > > > drivers/misc/sgi-gru/grumain.c | 2 +-
> > > > drivers/staging/android/ion/ion.c | 2 +-
> > > > drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> > > > drivers/xen/privcmd.c | 2 +-
> > > > fs/dax.c | 4 ++--
> > > > include/linux/mm.h | 2 --
> > > > mm/memory.c | 7 +++----
> > > > 27 files changed, 59 insertions(+), 65 deletions(-)
> > > >
> > > > diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> > > > index 06254467e4dd..e8a31fffcdda 100644
> > > > --- a/arch/powerpc/platforms/cell/spufs/file.c
> > > > +++ b/arch/powerpc/platforms/cell/spufs/file.c
> > > > @@ -236,7 +236,7 @@ static int
> > > > spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> > > > {
> > > > struct spu_context *ctx = vma->vm_file->private_data;
> > > > - unsigned long address = (unsigned long)vmf->virtual_address;
> > > > + unsigned long address = vmf->address & PAGE_MASK;
> > >
> > > These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
> > > need sub-page address, do we?
> >
> > Usually not AFAICT but I was not really sure in some cases (e.g. is
> > vm_insert_pfn() and friends really safe to call with unaligned address) so
> > I've decided to do a simple search-and-replace for such a wide-scale patch.
> > Later we can remove unnecessary masking from places where we are sure it is
> > not needed.
>
> I think it should be fine just to zero out ~PAGE_MASK bits of the address
> on entrance into handle_mm_fault() or where vmf is initialized.
OK, I wasn't really sure whether something in mm does not rely on low bits
being accurate. But if you also don't think anything relies on those bits,
I can try it we'll see if something breaks :)
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address
@ 2016-11-16 11:55 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 11:55 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 14:32:48, Kirill A. Shutemov wrote:
> On Wed, Nov 16, 2016 at 12:05:05PM +0100, Jan Kara wrote:
> > On Wed 16-11-16 00:55:31, Kirill A. Shutemov wrote:
> > > On Fri, Nov 04, 2016 at 05:24:58AM +0100, Jan Kara wrote:
> > > > Every single user of vmf->virtual_address typed that entry to unsigned
> > > > long before doing anything with it so the type of virtual_address does
> > > > not really provide us any additional safety. Just use masked
> > > > vmf->address which already has the appropriate type.
> > > >
> > > > Signed-off-by: Jan Kara <jack@suse.cz>
> > > > ---
> > > > arch/powerpc/platforms/cell/spufs/file.c | 4 ++--
> > > > arch/x86/entry/vdso/vma.c | 4 ++--
> > > > drivers/char/agp/alpha-agp.c | 2 +-
> > > > drivers/char/mspec.c | 2 +-
> > > > drivers/dax/dax.c | 2 +-
> > > > drivers/gpu/drm/armada/armada_gem.c | 2 +-
> > > > drivers/gpu/drm/drm_vm.c | 11 ++++++-----
> > > > drivers/gpu/drm/etnaviv/etnaviv_gem.c | 7 +++----
> > > > drivers/gpu/drm/exynos/exynos_drm_gem.c | 6 +++---
> > > > drivers/gpu/drm/gma500/framebuffer.c | 2 +-
> > > > drivers/gpu/drm/gma500/gem.c | 5 ++---
> > > > drivers/gpu/drm/i915/i915_gem.c | 2 +-
> > > > drivers/gpu/drm/msm/msm_gem.c | 7 +++----
> > > > drivers/gpu/drm/omapdrm/omap_gem.c | 20 +++++++++-----------
> > > > drivers/gpu/drm/tegra/gem.c | 4 ++--
> > > > drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +-
> > > > drivers/gpu/drm/udl/udl_gem.c | 5 ++---
> > > > drivers/gpu/drm/vgem/vgem_drv.c | 2 +-
> > > > drivers/media/v4l2-core/videobuf-dma-sg.c | 5 ++---
> > > > drivers/misc/cxl/context.c | 2 +-
> > > > drivers/misc/sgi-gru/grumain.c | 2 +-
> > > > drivers/staging/android/ion/ion.c | 2 +-
> > > > drivers/staging/lustre/lustre/llite/vvp_io.c | 9 ++++++---
> > > > drivers/xen/privcmd.c | 2 +-
> > > > fs/dax.c | 4 ++--
> > > > include/linux/mm.h | 2 --
> > > > mm/memory.c | 7 +++----
> > > > 27 files changed, 59 insertions(+), 65 deletions(-)
> > > >
> > > > diff --git a/arch/powerpc/platforms/cell/spufs/file.c b/arch/powerpc/platforms/cell/spufs/file.c
> > > > index 06254467e4dd..e8a31fffcdda 100644
> > > > --- a/arch/powerpc/platforms/cell/spufs/file.c
> > > > +++ b/arch/powerpc/platforms/cell/spufs/file.c
> > > > @@ -236,7 +236,7 @@ static int
> > > > spufs_mem_mmap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
> > > > {
> > > > struct spu_context *ctx = vma->vm_file->private_data;
> > > > - unsigned long address = (unsigned long)vmf->virtual_address;
> > > > + unsigned long address = vmf->address & PAGE_MASK;
> > >
> > > These "& PAGE_MASK" everewhere look unnecesary. I don't think we ever
> > > need sub-page address, do we?
> >
> > Usually not AFAICT but I was not really sure in some cases (e.g. is
> > vm_insert_pfn() and friends really safe to call with unaligned address) so
> > I've decided to do a simple search-and-replace for such a wide-scale patch.
> > Later we can remove unnecessary masking from places where we are sure it is
> > not needed.
>
> I think it should be fine just to zero out ~PAGE_MASK bits of the address
> on entrance into handle_mm_fault() or where vmf is initialized.
OK, I wasn't really sure whether something in mm does not rely on low bits
being accurate. But if you also don't think anything relies on those bits,
I can try it we'll see if something breaks :)
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 05/21] mm: Trim __do_fault() arguments
2016-11-15 22:10 ` Kirill A. Shutemov
(?)
@ 2016-11-16 13:12 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 13:12 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 01:10:01, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:25:01AM +0100, Jan Kara wrote:
> > static int do_cow_fault(struct vm_fault *vmf)
> > {
> > struct vm_area_struct *vma = vmf->vma;
> > - struct page *fault_page, *new_page;
> > - void *fault_entry;
> > + struct page *new_page;
>
> Why not get rid of new_page too?
OK, I did that as well.
> Otherwise makes sense:
>
> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Thanks!
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
2016-11-15 22:28 ` Kirill A. Shutemov
@ 2016-11-16 13:29 ` Jan Kara
2016-11-16 14:27 ` Kirill A. Shutemov
0 siblings, 1 reply; 124+ messages in thread
From: Jan Kara @ 2016-11-16 13:29 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 01:28:19, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:25:07AM +0100, Jan Kara wrote:
> > We don't check whether vma->vm_ops is NULL in do_shared_fault() so
> > there's hardly any point in checking it in wp_page_shared() or
> > wp_pfn_shared() which get called only for shared file mappings as well.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
>
> Well, I'm not sure about this.
>
> do_shared_fault() doesn't have the check since we checked it upper by
> stack: see vma_is_anonymous() in handle_pte_fault().
>
> In principal, it should be fine. But random crappy driver has potential to
> blow it up.
Ok, so do you prefer me to keep this patch or discard it? Either is fine with
me. It was just a cleanup I wrote when factoring out the functionality.
Honza
>
> > ---
> > mm/memory.c | 4 ++--
> > 1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/mm/memory.c b/mm/memory.c
> > index 7be96a43d5ac..26b2858e6a12 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -2275,7 +2275,7 @@ static int wp_pfn_shared(struct vm_fault *vmf)
> > {
> > struct vm_area_struct *vma = vmf->vma;
> >
> > - if (vma->vm_ops && vma->vm_ops->pfn_mkwrite) {
> > + if (vma->vm_ops->pfn_mkwrite) {
> > int ret;
> >
> > pte_unmap_unlock(vmf->pte, vmf->ptl);
> > @@ -2305,7 +2305,7 @@ static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
> >
> > get_page(old_page);
> >
> > - if (vma->vm_ops && vma->vm_ops->page_mkwrite) {
> > + if (vma->vm_ops->page_mkwrite) {
> > int tmp;
> >
> > pte_unmap_unlock(vmf->pte, vmf->ptl);
> > --
> > 2.6.6
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org. For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>
> --
> Kirill A. Shutemov
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite()
2016-11-15 22:40 ` Kirill A. Shutemov
(?)
@ 2016-11-16 13:34 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 13:34 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 01:40:23, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:25:09AM +0100, Jan Kara wrote:
> > We will need more information in the ->page_mkwrite() helper for DAX to
> > be able to fully finish faults there. Pass vm_fault structure to
> > do_page_mkwrite() and use it there so that information propagates
> > properly from upper layers.
> >
> > Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> > mm/memory.c | 19 +++++++------------
> > 1 file changed, 7 insertions(+), 12 deletions(-)
> >
> > diff --git a/mm/memory.c b/mm/memory.c
> > index 4da66c984c2c..c89f99c270bc 100644
> > --- a/mm/memory.c
> > +++ b/mm/memory.c
> > @@ -2038,20 +2038,14 @@ static gfp_t __get_fault_gfp_mask(struct vm_area_struct *vma)
> > *
> > * We do this without the lock held, so that it can sleep if it needs to.
> > */
> > -static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
> > - unsigned long address)
> > +static int do_page_mkwrite(struct vm_fault *vmf)
> > {
> > - struct vm_fault vmf;
> > int ret;
> > + struct page *page = vmf->page;
> >
> > - vmf.address = address;
> > - vmf.pgoff = page->index;
> > - vmf.flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
> > - vmf.gfp_mask = __get_fault_gfp_mask(vma);
> > - vmf.page = page;
> > - vmf.cow_page = NULL;
> > + vmf->flags = FAULT_FLAG_WRITE|FAULT_FLAG_MKWRITE;
>
> This can be destructive: we loose rest of the flags here. It's probably
> okay in current state of the code, but may be should restore them before
> return from do_page_mkwrite()?
Yeah, probably makes sense as future-proofing. Changed.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 16/21] mm: Provide helper for finishing mkwrite faults
2016-11-15 22:52 ` Kirill A. Shutemov
(?)
@ 2016-11-16 13:39 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 13:39 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 01:52:10, Kirill A. Shutemov wrote:
> On Fri, Nov 04, 2016 at 05:25:12AM +0100, Jan Kara wrote:
> > Provide a helper function for finishing write faults due to PTE being
> > read-only. The helper will be used by DAX to avoid the need of
> > complicating generic MM code with DAX locking specifics.
> >
> > Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
> > Signed-off-by: Jan Kara <jack@suse.cz>
...
> > - /*
> > - * Since we dropped the lock we need to revalidate
> > - * the PTE as someone else may have changed it. If
> > - * they did, we just return, as we can count on the
> > - * MMU to tell us if they didn't also make it writable.
> > - */
> > - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
> > - vmf->address, &vmf->ptl);
> > - if (!pte_same(*vmf->pte, vmf->orig_pte)) {
> > + tmp = finish_mkwrite_fault(vmf);
> > + if (unlikely(!tmp || (tmp &
> > + (VM_FAULT_ERROR | VM_FAULT_NOPAGE)))) {
>
> Looks like the second part of condition is never true here, right? Not
> that it would matter, having the next patch in the queue.
Yeah, I had the condition like this to handle the standard set of return
values. And as you say, the next patch even makes this condition have a
real effect.
> Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Thanks.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
2016-11-16 13:29 ` Jan Kara
@ 2016-11-16 14:27 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-16 14:27 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 02:29:18PM +0100, Jan Kara wrote:
> On Wed 16-11-16 01:28:19, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:25:07AM +0100, Jan Kara wrote:
> > > We don't check whether vma->vm_ops is NULL in do_shared_fault() so
> > > there's hardly any point in checking it in wp_page_shared() or
> > > wp_pfn_shared() which get called only for shared file mappings as well.
> > >
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> >
> > Well, I'm not sure about this.
> >
> > do_shared_fault() doesn't have the check since we checked it upper by
> > stack: see vma_is_anonymous() in handle_pte_fault().
> >
> > In principal, it should be fine. But random crappy driver has potential to
> > blow it up.
>
> Ok, so do you prefer me to keep this patch or discard it? Either is fine with
> me. It was just a cleanup I wrote when factoring out the functionality.
I would rather drop it.
Eventually, we need to make sure that all file-backed vma has vm_ops.
I tried to do this once, but that back-fired...
--
Kirill A. Shutemov
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
@ 2016-11-16 14:27 ` Kirill A. Shutemov
0 siblings, 0 replies; 124+ messages in thread
From: Kirill A. Shutemov @ 2016-11-16 14:27 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 02:29:18PM +0100, Jan Kara wrote:
> On Wed 16-11-16 01:28:19, Kirill A. Shutemov wrote:
> > On Fri, Nov 04, 2016 at 05:25:07AM +0100, Jan Kara wrote:
> > > We don't check whether vma->vm_ops is NULL in do_shared_fault() so
> > > there's hardly any point in checking it in wp_page_shared() or
> > > wp_pfn_shared() which get called only for shared file mappings as well.
> > >
> > > Signed-off-by: Jan Kara <jack@suse.cz>
> >
> > Well, I'm not sure about this.
> >
> > do_shared_fault() doesn't have the check since we checked it upper by
> > stack: see vma_is_anonymous() in handle_pte_fault().
> >
> > In principal, it should be fine. But random crappy driver has potential to
> > blow it up.
>
> Ok, so do you prefer me to keep this patch or discard it? Either is fine with
> me. It was just a cleanup I wrote when factoring out the functionality.
I would rather drop it.
Eventually, we need to make sure that all file-backed vma has vm_ops.
I tried to do this once, but that back-fired...
--
Kirill A. Shutemov
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check
2016-11-16 14:27 ` Kirill A. Shutemov
(?)
@ 2016-11-16 14:43 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-16 14:43 UTC (permalink / raw)
To: Kirill A. Shutemov
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 17:27:55, Kirill A. Shutemov wrote:
> On Wed, Nov 16, 2016 at 02:29:18PM +0100, Jan Kara wrote:
> > On Wed 16-11-16 01:28:19, Kirill A. Shutemov wrote:
> > > On Fri, Nov 04, 2016 at 05:25:07AM +0100, Jan Kara wrote:
> > > > We don't check whether vma->vm_ops is NULL in do_shared_fault() so
> > > > there's hardly any point in checking it in wp_page_shared() or
> > > > wp_pfn_shared() which get called only for shared file mappings as well.
> > > >
> > > > Signed-off-by: Jan Kara <jack@suse.cz>
> > >
> > > Well, I'm not sure about this.
> > >
> > > do_shared_fault() doesn't have the check since we checked it upper by
> > > stack: see vma_is_anonymous() in handle_pte_fault().
> > >
> > > In principal, it should be fine. But random crappy driver has potential to
> > > blow it up.
> >
> > Ok, so do you prefer me to keep this patch or discard it? Either is fine with
> > me. It was just a cleanup I wrote when factoring out the functionality.
>
> I would rather drop it.
>
> Eventually, we need to make sure that all file-backed vma has vm_ops.
> I tried to do this once, but that back-fired...
OK, will do.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-16 11:01 ` Jan Kara
@ 2016-11-16 17:21 ` Peter Zijlstra
-1 siblings, 0 replies; 124+ messages in thread
From: Peter Zijlstra @ 2016-11-16 17:21 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm-y27Ovi1pjclAfugRpC6u6w,
linux-mm-Bw31MaZKKs3YtjvyW6yDsg,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA, Kirill A. Shutemov,
Andrew Morton, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 12:01:01PM +0100, Jan Kara wrote:
> On Wed 16-11-16 11:51:32, Peter Zijlstra wrote:
> > Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> > needs/wants.
>
> Yeah, DAX does not have 'struct page' for its pages so it directly installs
> PFNs in the page tables. As a result it needs to know about page tables and
> stuff.
Not convinced, a physical address should then be the equivalent of a
struct page. You still don't need access to the actual pages tables. The
VM core can then convert the physical address to a PFN and stuff it in
the PTE entry.
> Now I've abstracted knowledge about that into helper functions back
> in mm/ but still we need to pass the information through the ->fault handler
> into those helpers and vm_fault structure is simply natural for that.
> So far we have tried to avoid that but the result was not pretty (special
> return codes from DAX ->fault handlers essentially leaking information
> about DAX internal locking into mm/ code to direct generic mm code to do
> the right thing for DAX).
Its probably the DAX locking bit I'm missing, because I cannot see why
VM_FAULT_DAX_LOCKED is 'broken' -- also, I'd have called that
VM_FAULT_PFN or similar and not have used the full entry but only the
PFN bits from it.
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
@ 2016-11-16 17:21 ` Peter Zijlstra
0 siblings, 0 replies; 124+ messages in thread
From: Peter Zijlstra @ 2016-11-16 17:21 UTC (permalink / raw)
To: Jan Kara
Cc: Kirill A. Shutemov, linux-mm, linux-fsdevel, linux-nvdimm,
Andrew Morton, Ross Zwisler, Kirill A. Shutemov
On Wed, Nov 16, 2016 at 12:01:01PM +0100, Jan Kara wrote:
> On Wed 16-11-16 11:51:32, Peter Zijlstra wrote:
> > Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> > needs/wants.
>
> Yeah, DAX does not have 'struct page' for its pages so it directly installs
> PFNs in the page tables. As a result it needs to know about page tables and
> stuff.
Not convinced, a physical address should then be the equivalent of a
struct page. You still don't need access to the actual pages tables. The
VM core can then convert the physical address to a PFN and stuff it in
the PTE entry.
> Now I've abstracted knowledge about that into helper functions back
> in mm/ but still we need to pass the information through the ->fault handler
> into those helpers and vm_fault structure is simply natural for that.
> So far we have tried to avoid that but the result was not pretty (special
> return codes from DAX ->fault handlers essentially leaking information
> about DAX internal locking into mm/ code to direct generic mm code to do
> the right thing for DAX).
Its probably the DAX locking bit I'm missing, because I cannot see why
VM_FAULT_DAX_LOCKED is 'broken' -- also, I'd have called that
VM_FAULT_PFN or similar and not have used the full entry but only the
PFN bits from it.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 07/21] mm: Add orig_pte field into vm_fault
2016-11-04 4:25 ` Jan Kara
@ 2016-11-16 20:00 ` Ross Zwisler
-1 siblings, 0 replies; 124+ messages in thread
From: Ross Zwisler @ 2016-11-16 20:00 UTC (permalink / raw)
To: Jan Kara
Cc: linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:03AM +0100, Jan Kara wrote:
> Add orig_pte field to vm_fault structure to allow ->page_mkwrite
> handlers to fully handle the fault. This also allows us to save some
> passing of extra arguments around.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-16 20:00 ` Ross Zwisler
0 siblings, 0 replies; 124+ messages in thread
From: Ross Zwisler @ 2016-11-16 20:00 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:03AM +0100, Jan Kara wrote:
> Add orig_pte field to vm_fault structure to allow ->page_mkwrite
> handlers to fully handle the fault. This also allows us to save some
> passing of extra arguments around.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
Reviewed-by: Ross Zwisler <ross.zwisler@linux.intel.com>
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 10/21] mm: Move handling of COW faults into DAX code
2016-11-04 4:25 ` Jan Kara
` (2 preceding siblings ...)
(?)
@ 2016-11-16 21:28 ` Ross Zwisler
2016-11-17 9:36 ` Jan Kara
-1 siblings, 1 reply; 124+ messages in thread
From: Ross Zwisler @ 2016-11-16 21:28 UTC (permalink / raw)
To: Jan Kara
Cc: linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Ross Zwisler, Kirill A. Shutemov
On Fri, Nov 04, 2016 at 05:25:06AM +0100, Jan Kara wrote:
> Move final handling of COW faults from generic code into DAX fault
> handler. That way generic code doesn't have to be aware of peculiarities
> of DAX locking so remove that knowledge and make locking functions
> private to fs/dax.c.
>
> Signed-off-by: Jan Kara <jack@suse.cz>
> ---
> @@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
>
> if (error)
> goto finish_iomap;
> - if (!radix_tree_exceptional_entry(entry)) {
> +
> + __SetPageUptodate(vmf->cow_page);
> + if (!radix_tree_exceptional_entry(entry))
> vmf->page = entry;
I don't think we need to set vmf->page anymore. We would clear it to NULL in
a few lines anyway, and the only call in between is finish_fault(), which
only cares about vmf->cow_page(). This allows us to remove the vmf->page =
NULL line a few lines below as well.
> @@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
> }
> }
> unlock_entry:
> - if (!locked_status || error)
> + if (vmf_ret != VM_FAULT_LOCKED || error)
> put_locked_mapping_entry(mapping, vmf->pgoff, entry);
I don't think this is quite right. For example, for dax_load_hole(), if we
can't get a page we put_locked_mapping_entry() and return VM_FAULT_OOM.
Previously this logic would have skipped the second call to
put_locked_mapping_entry(), but now with the strict check against
VM_FAULT_LOCKED put the entry twice.
Maybe the right thing to do is just fix dax_load_hole() so it never calls
put_locked_mapping_entry(), and leave this check as you have it?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 01/21] mm: Join struct fault_env and vm_fault
2016-11-16 17:21 ` Peter Zijlstra
(?)
@ 2016-11-17 9:07 ` Jan Kara
-1 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-17 9:07 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Jan Kara, Kirill A. Shutemov, linux-mm, linux-fsdevel,
linux-nvdimm, Andrew Morton, Ross Zwisler, Kirill A. Shutemov
On Wed 16-11-16 18:21:08, Peter Zijlstra wrote:
> On Wed, Nov 16, 2016 at 12:01:01PM +0100, Jan Kara wrote:
> > On Wed 16-11-16 11:51:32, Peter Zijlstra wrote:
>
> > > Now, I'm entirely out of touch wrt DAX, so I've not idea what that
> > > needs/wants.
> >
> > Yeah, DAX does not have 'struct page' for its pages so it directly installs
> > PFNs in the page tables. As a result it needs to know about page tables and
> > stuff.
>
> Not convinced, a physical address should then be the equivalent of a
> struct page. You still don't need access to the actual pages tables. The
> VM core can then convert the physical address to a PFN and stuff it in
> the PTE entry.
>
> > Now I've abstracted knowledge about that into helper functions back
> > in mm/ but still we need to pass the information through the ->fault handler
> > into those helpers and vm_fault structure is simply natural for that.
> > So far we have tried to avoid that but the result was not pretty (special
> > return codes from DAX ->fault handlers essentially leaking information
> > about DAX internal locking into mm/ code to direct generic mm code to do
> > the right thing for DAX).
>
> Its probably the DAX locking bit I'm missing, because I cannot see why
> VM_FAULT_DAX_LOCKED is 'broken' -- also, I'd have called that
> VM_FAULT_PFN or similar and not have used the full entry but only the
> PFN bits from it.
The problem really is in the locking. Currently fault code expects to get a
locked page from the ->fault() handler (or it locks the returned page on
its own). When we don't have struct page, we don't have a page lock. So we
use different locks to synchronize against truncate, or other page faults
of the same page by a different process. And modification of page tables
needs to be protected by these locks in the same way as it needs to be
protected by the page lock for normal pages.
So I find it to be bigger layering violation for generic mm layer to know
how DAX decided to serialize racing page faults, than for DAX layer to call
helpers from mm layer to update page tables when it holds appropriate
locks.
Essentially we just switch from a model "Return data from a fault handler
so that generic layer can finish the page fault" to a model "Call helper
from the fault handler passing it necessary information to finish the page
fault". That model is more flexible (I agree it offers more opportunities
for abuse as well) but overall result looks IMHO cleaner.
And I would add that modifying page tables from a fault handler is nothing
new - basically every video driver does it. They just generally don't have
that comprehensive support for mmap so they get away with less exported
functionality.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 10/21] mm: Move handling of COW faults into DAX code
2016-11-16 21:28 ` Ross Zwisler
@ 2016-11-17 9:36 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-17 9:36 UTC (permalink / raw)
To: Ross Zwisler
Cc: Jan Kara, linux-nvdimm, linux-mm, linux-fsdevel, Andrew Morton,
Kirill A. Shutemov
On Wed 16-11-16 14:28:20, Ross Zwisler wrote:
> On Fri, Nov 04, 2016 at 05:25:06AM +0100, Jan Kara wrote:
> > Move final handling of COW faults from generic code into DAX fault
> > handler. That way generic code doesn't have to be aware of peculiarities
> > of DAX locking so remove that knowledge and make locking functions
> > private to fs/dax.c.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
>
> > @@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
> >
> > if (error)
> > goto finish_iomap;
> > - if (!radix_tree_exceptional_entry(entry)) {
> > +
> > + __SetPageUptodate(vmf->cow_page);
> > + if (!radix_tree_exceptional_entry(entry))
> > vmf->page = entry;
>
> I don't think we need to set vmf->page anymore. We would clear it to NULL in
> a few lines anyway, and the only call in between is finish_fault(), which
> only cares about vmf->cow_page(). This allows us to remove the vmf->page =
> NULL line a few lines below as well.
Well, I would not like to depend too much on which fields of vm_fault
finish_fault() actually uses - we should fill in as much as we have
available. But the truth is we sometime have page to fill in into vmf->page
and sometimes we don't so in this case I agree filling it in is pointless.
Changed.
> > @@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
> > }
> > }
> > unlock_entry:
> > - if (!locked_status || error)
> > + if (vmf_ret != VM_FAULT_LOCKED || error)
> > put_locked_mapping_entry(mapping, vmf->pgoff, entry);
>
> I don't think this is quite right. For example, for dax_load_hole(), if we
> can't get a page we put_locked_mapping_entry() and return VM_FAULT_OOM.
> Previously this logic would have skipped the second call to
> put_locked_mapping_entry(), but now with the strict check against
> VM_FAULT_LOCKED put the entry twice.
>
> Maybe the right thing to do is just fix dax_load_hole() so it never calls
> put_locked_mapping_entry(), and leave this check as you have it?
Yeah, good catch. Actually I have follow up patches which somewhat clean up
dax_iomap_fault() so that page fault is fully completed within
dax_iomap_fault() even when instantiating a hole page which makes error
handling simpler. But I didn't want to complicate this series with it. So
for now I'll do what you suggest. Thanks.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply [flat|nested] 124+ messages in thread
* Re: [PATCH 10/21] mm: Move handling of COW faults into DAX code
@ 2016-11-17 9:36 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-17 9:36 UTC (permalink / raw)
To: Ross Zwisler
Cc: Jan Kara, linux-mm, linux-fsdevel, linux-nvdimm, Andrew Morton,
Kirill A. Shutemov
On Wed 16-11-16 14:28:20, Ross Zwisler wrote:
> On Fri, Nov 04, 2016 at 05:25:06AM +0100, Jan Kara wrote:
> > Move final handling of COW faults from generic code into DAX fault
> > handler. That way generic code doesn't have to be aware of peculiarities
> > of DAX locking so remove that knowledge and make locking functions
> > private to fs/dax.c.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
>
> > @@ -1006,13 +1007,14 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
> >
> > if (error)
> > goto finish_iomap;
> > - if (!radix_tree_exceptional_entry(entry)) {
> > +
> > + __SetPageUptodate(vmf->cow_page);
> > + if (!radix_tree_exceptional_entry(entry))
> > vmf->page = entry;
>
> I don't think we need to set vmf->page anymore. We would clear it to NULL in
> a few lines anyway, and the only call in between is finish_fault(), which
> only cares about vmf->cow_page(). This allows us to remove the vmf->page =
> NULL line a few lines below as well.
Well, I would not like to depend too much on which fields of vm_fault
finish_fault() actually uses - we should fill in as much as we have
available. But the truth is we sometime have page to fill in into vmf->page
and sometimes we don't so in this case I agree filling it in is pointless.
Changed.
> > @@ -1051,7 +1053,7 @@ int dax_iomap_fault(struct vm_area_struct *vma, struct vm_fault *vmf,
> > }
> > }
> > unlock_entry:
> > - if (!locked_status || error)
> > + if (vmf_ret != VM_FAULT_LOCKED || error)
> > put_locked_mapping_entry(mapping, vmf->pgoff, entry);
>
> I don't think this is quite right. For example, for dax_load_hole(), if we
> can't get a page we put_locked_mapping_entry() and return VM_FAULT_OOM.
> Previously this logic would have skipped the second call to
> put_locked_mapping_entry(), but now with the strict check against
> VM_FAULT_LOCKED put the entry twice.
>
> Maybe the right thing to do is just fix dax_load_hole() so it never calls
> put_locked_mapping_entry(), and leave this check as you have it?
Yeah, good catch. Actually I have follow up patches which somewhat clean up
dax_iomap_fault() so that page fault is fully completed within
dax_iomap_fault() even when instantiating a hole page which makes error
handling simpler. But I didn't want to complicate this series with it. So
for now I'll do what you suggest. Thanks.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
2016-11-01 22:36 [PATCH 0/21 v4] dax: Clear dirty bits after flushing caches Jan Kara
2016-11-01 22:36 ` Jan Kara
@ 2016-11-01 22:36 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-01 22:36 UTC (permalink / raw)
To: linux-mm; +Cc: linux-fsdevel, Andrew Morton, Jan Kara, linux-nvdimm
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 2a4ebe3c67c6..f8e758060851 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
_______________________________________________
Linux-nvdimm mailing list
Linux-nvdimm@lists.01.org
https://lists.01.org/mailman/listinfo/linux-nvdimm
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-01 22:36 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-01 22:36 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler, Jan Kara
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 2a4ebe3c67c6..f8e758060851 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
* [PATCH 07/21] mm: Add orig_pte field into vm_fault
@ 2016-11-01 22:36 ` Jan Kara
0 siblings, 0 replies; 124+ messages in thread
From: Jan Kara @ 2016-11-01 22:36 UTC (permalink / raw)
To: linux-mm
Cc: linux-fsdevel, linux-nvdimm, Andrew Morton, Ross Zwisler, Jan Kara
Add orig_pte field to vm_fault structure to allow ->page_mkwrite
handlers to fully handle the fault. This also allows us to save some
passing of extra arguments around.
Signed-off-by: Jan Kara <jack@suse.cz>
---
include/linux/mm.h | 4 +--
mm/internal.h | 2 +-
mm/khugepaged.c | 7 ++---
mm/memory.c | 82 +++++++++++++++++++++++++++---------------------------
4 files changed, 47 insertions(+), 48 deletions(-)
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 2a4ebe3c67c6..f8e758060851 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -298,8 +298,8 @@ struct vm_fault {
pgoff_t pgoff; /* Logical page offset based on vma */
unsigned long address; /* Faulting virtual address */
pmd_t *pmd; /* Pointer to pmd entry matching
- * the 'address'
- */
+ * the 'address' */
+ pte_t orig_pte; /* Value of PTE at the time of fault */
struct page *cow_page; /* Handler may choose to COW */
struct page *page; /* ->fault handlers should return a
diff --git a/mm/internal.h b/mm/internal.h
index 093b1eacc91b..44d68895a9b9 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,7 +36,7 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte);
+int do_swap_page(struct vm_fault *vmf);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index d7df06383b10..1f20f25fe029 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -873,7 +873,6 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
unsigned long address, pmd_t *pmd,
int referenced)
{
- pte_t pteval;
int swapped_in = 0, ret = 0;
struct vm_fault vmf = {
.vma = vma,
@@ -891,11 +890,11 @@ static bool __collapse_huge_page_swapin(struct mm_struct *mm,
vmf.pte = pte_offset_map(pmd, address);
for (; vmf.address < address + HPAGE_PMD_NR*PAGE_SIZE;
vmf.pte++, vmf.address += PAGE_SIZE) {
- pteval = *vmf.pte;
- if (!is_swap_pte(pteval))
+ vmf.orig_pte = *vmf.pte;
+ if (!is_swap_pte(vmf.orig_pte))
continue;
swapped_in++;
- ret = do_swap_page(&vmf, pteval);
+ ret = do_swap_page(&vmf);
/* do_swap_page returns VM_FAULT_RETRY with released mmap_sem */
if (ret & VM_FAULT_RETRY) {
diff --git a/mm/memory.c b/mm/memory.c
index 5f6bc9028a88..25028422a578 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2074,8 +2074,8 @@ static int do_page_mkwrite(struct vm_area_struct *vma, struct page *page,
* case, all we need to do here is to mark the page as writable and update
* any related book-keeping.
*/
-static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
- struct page *page, int page_mkwrite, int dirty_shared)
+static inline int wp_page_reuse(struct vm_fault *vmf, struct page *page,
+ int page_mkwrite, int dirty_shared)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2088,8 +2088,8 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
if (page)
page_cpupid_xchg_last(page, (1 << LAST_CPUPID_SHIFT) - 1);
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
- entry = pte_mkyoung(orig_pte);
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
+ entry = pte_mkyoung(vmf->orig_pte);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
if (ptep_set_access_flags(vma, vmf->address, vmf->pte, entry, 1))
update_mmu_cache(vma, vmf->address, vmf->pte);
@@ -2139,8 +2139,7 @@ static inline int wp_page_reuse(struct vm_fault *vmf, pte_t orig_pte,
* held to the old page, as well as updating the rmap.
* - In any case, unlock the PTL and drop the reference we took to the old page.
*/
-static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_copy(struct vm_fault *vmf, struct page *old_page)
{
struct vm_area_struct *vma = vmf->vma;
struct mm_struct *mm = vma->vm_mm;
@@ -2154,7 +2153,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
if (unlikely(anon_vma_prepare(vma)))
goto oom;
- if (is_zero_pfn(pte_pfn(orig_pte))) {
+ if (is_zero_pfn(pte_pfn(vmf->orig_pte))) {
new_page = alloc_zeroed_user_highpage_movable(vma,
vmf->address);
if (!new_page)
@@ -2178,7 +2177,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Re-check the pte - we dropped the lock
*/
vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte))) {
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte))) {
if (old_page) {
if (!PageAnon(old_page)) {
dec_mm_counter_fast(mm,
@@ -2188,7 +2187,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
} else {
inc_mm_counter_fast(mm, MM_ANONPAGES);
}
- flush_cache_page(vma, vmf->address, pte_pfn(orig_pte));
+ flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte));
entry = mk_pte(new_page, vma->vm_page_prot);
entry = maybe_mkwrite(pte_mkdirty(entry), vma);
/*
@@ -2272,7 +2271,7 @@ static int wp_page_copy(struct vm_fault *vmf, pte_t orig_pte,
* Handle write page faults for VM_MIXEDMAP or VM_PFNMAP for a VM_SHARED
* mapping
*/
-static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
+static int wp_pfn_shared(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2290,16 +2289,15 @@ static int wp_pfn_shared(struct vm_fault *vmf, pte_t orig_pte)
* We might have raced with another page fault while we
* released the pte_offset_map_lock.
*/
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
pte_unmap_unlock(vmf->pte, vmf->ptl);
return 0;
}
}
- return wp_page_reuse(vmf, orig_pte, NULL, 0, 0);
+ return wp_page_reuse(vmf, NULL, 0, 0);
}
-static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
- struct page *old_page)
+static int wp_page_shared(struct vm_fault *vmf, struct page *old_page)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
@@ -2325,7 +2323,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2334,7 +2332,7 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
page_mkwrite = 1;
}
- return wp_page_reuse(vmf, orig_pte, old_page, page_mkwrite, 1);
+ return wp_page_reuse(vmf, old_page, page_mkwrite, 1);
}
/*
@@ -2355,13 +2353,13 @@ static int wp_page_shared(struct vm_fault *vmf, pte_t orig_pte,
* but allow concurrent faults), with pte both mapped and locked.
* We return with mmap_sem still held, but pte unmapped and unlocked.
*/
-static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
+static int do_wp_page(struct vm_fault *vmf)
__releases(vmf->ptl)
{
struct vm_area_struct *vma = vmf->vma;
struct page *old_page;
- old_page = vm_normal_page(vma, vmf->address, orig_pte);
+ old_page = vm_normal_page(vma, vmf->address, vmf->orig_pte);
if (!old_page) {
/*
* VM_MIXEDMAP !pfn_valid() case, or VM_SOFTDIRTY clear on a
@@ -2372,10 +2370,10 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
*/
if ((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))
- return wp_pfn_shared(vmf, orig_pte);
+ return wp_pfn_shared(vmf);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
/*
@@ -2390,7 +2388,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
lock_page(old_page);
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (!pte_same(*vmf->pte, orig_pte)) {
+ if (!pte_same(*vmf->pte, vmf->orig_pte)) {
unlock_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
put_page(old_page);
@@ -2410,12 +2408,12 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
page_move_anon_rmap(old_page, vma);
}
unlock_page(old_page);
- return wp_page_reuse(vmf, orig_pte, old_page, 0, 0);
+ return wp_page_reuse(vmf, old_page, 0, 0);
}
unlock_page(old_page);
} else if (unlikely((vma->vm_flags & (VM_WRITE|VM_SHARED)) ==
(VM_WRITE|VM_SHARED))) {
- return wp_page_shared(vmf, orig_pte, old_page);
+ return wp_page_shared(vmf, old_page);
}
/*
@@ -2424,7 +2422,7 @@ static int do_wp_page(struct vm_fault *vmf, pte_t orig_pte)
get_page(old_page);
pte_unmap_unlock(vmf->pte, vmf->ptl);
- return wp_page_copy(vmf, orig_pte, old_page);
+ return wp_page_copy(vmf, old_page);
}
static void unmap_mapping_range_vma(struct vm_area_struct *vma,
@@ -2512,7 +2510,7 @@ EXPORT_SYMBOL(unmap_mapping_range);
* We return with the mmap_sem locked or unlocked in the same cases
* as does filemap_fault().
*/
-int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
+int do_swap_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page, *swapcache;
@@ -2523,10 +2521,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
int exclusive = 0;
int ret = 0;
- if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, orig_pte))
+ if (!pte_unmap_same(vma->vm_mm, vmf->pmd, vmf->pte, vmf->orig_pte))
goto out;
- entry = pte_to_swp_entry(orig_pte);
+ entry = pte_to_swp_entry(vmf->orig_pte);
if (unlikely(non_swap_entry(entry))) {
if (is_migration_entry(entry)) {
migration_entry_wait(vma->vm_mm, vmf->pmd,
@@ -2534,7 +2532,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
} else if (is_hwpoison_entry(entry)) {
ret = VM_FAULT_HWPOISON;
} else {
- print_bad_pte(vma, vmf->address, orig_pte, NULL);
+ print_bad_pte(vma, vmf->address, vmf->orig_pte, NULL);
ret = VM_FAULT_SIGBUS;
}
goto out;
@@ -2551,7 +2549,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
vmf->address, &vmf->ptl);
- if (likely(pte_same(*vmf->pte, orig_pte)))
+ if (likely(pte_same(*vmf->pte, vmf->orig_pte)))
ret = VM_FAULT_OOM;
delayacct_clear_flag(DELAYACCT_PF_SWAPIN);
goto unlock;
@@ -2608,7 +2606,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
*/
vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address,
&vmf->ptl);
- if (unlikely(!pte_same(*vmf->pte, orig_pte)))
+ if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
goto out_nomap;
if (unlikely(!PageUptodate(page))) {
@@ -2636,9 +2634,10 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
exclusive = RMAP_EXCLUSIVE;
}
flush_icache_page(vma, page);
- if (pte_swp_soft_dirty(orig_pte))
+ if (pte_swp_soft_dirty(vmf->orig_pte))
pte = pte_mksoft_dirty(pte);
set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte);
+ vmf->orig_pte = pte;
if (page == swapcache) {
do_page_add_anon_rmap(page, vma, vmf->address, exclusive);
mem_cgroup_commit_charge(page, memcg, true, false);
@@ -2668,7 +2667,7 @@ int do_swap_page(struct vm_fault *vmf, pte_t orig_pte)
}
if (vmf->flags & FAULT_FLAG_WRITE) {
- ret |= do_wp_page(vmf, pte);
+ ret |= do_wp_page(vmf);
if (ret & VM_FAULT_ERROR)
ret &= VM_FAULT_ERROR;
goto out;
@@ -3330,7 +3329,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma,
return mpol_misplaced(page, vma, addr);
}
-static int do_numa_page(struct vm_fault *vmf, pte_t pte)
+static int do_numa_page(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
struct page *page = NULL;
@@ -3338,6 +3337,7 @@ static int do_numa_page(struct vm_fault *vmf, pte_t pte)
int last_cpupid;
int target_nid;
bool migrated = false;
+ pte_t pte = vmf->orig_pte;
bool was_writable = pte_write(pte);
int flags = 0;
@@ -3488,8 +3488,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* So now it's safe to run pte_offset_map().
*/
vmf->pte = pte_offset_map(vmf->pmd, vmf->address);
-
- entry = *vmf->pte;
+ vmf->orig_pte = *vmf->pte;
/*
* some architectures can have larger ptes than wordsize,
@@ -3500,7 +3499,7 @@ static int handle_pte_fault(struct vm_fault *vmf)
* ptl lock held. So here a barrier will do.
*/
barrier();
- if (pte_none(entry)) {
+ if (pte_none(vmf->orig_pte)) {
pte_unmap(vmf->pte);
vmf->pte = NULL;
}
@@ -3513,19 +3512,20 @@ static int handle_pte_fault(struct vm_fault *vmf)
return do_fault(vmf);
}
- if (!pte_present(entry))
- return do_swap_page(vmf, entry);
+ if (!pte_present(vmf->orig_pte))
+ return do_swap_page(vmf);
- if (pte_protnone(entry) && vma_is_accessible(vmf->vma))
- return do_numa_page(vmf, entry);
+ if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma))
+ return do_numa_page(vmf);
vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd);
spin_lock(vmf->ptl);
+ entry = vmf->orig_pte;
if (unlikely(!pte_same(*vmf->pte, entry)))
goto unlock;
if (vmf->flags & FAULT_FLAG_WRITE) {
if (!pte_write(entry))
- return do_wp_page(vmf, entry);
+ return do_wp_page(vmf);
entry = pte_mkdirty(entry);
}
entry = pte_mkyoung(entry);
--
2.6.6
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply related [flat|nested] 124+ messages in thread
end of thread, other threads:[~2016-11-17 9:36 UTC | newest]
Thread overview: 124+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-11-04 4:24 [PATCH 0/21 v4 RESEND] dax: Clear dirty bits after flushing caches Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-04 4:24 ` [PATCH 01/21] mm: Join struct fault_env and vm_fault Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-15 21:50 ` Kirill A. Shutemov
2016-11-15 21:50 ` Kirill A. Shutemov
2016-11-16 10:51 ` Peter Zijlstra
2016-11-16 10:51 ` Peter Zijlstra
[not found] ` <20161116105132.GR3142-ndre7Fmf5hadTX5a5knrm8zTDFooKrT+cvkQGrU6aU0@public.gmane.org>
2016-11-16 11:01 ` Jan Kara
2016-11-16 11:01 ` Jan Kara
[not found] ` <20161116110101.GE21785-4I4JzKEfoa/jFM9bn6wA6Q@public.gmane.org>
2016-11-16 17:21 ` Peter Zijlstra
2016-11-16 17:21 ` Peter Zijlstra
2016-11-17 9:07 ` Jan Kara
2016-11-16 11:13 ` Kirill A. Shutemov
2016-11-04 4:24 ` [PATCH 02/21] mm: Use vmf->address instead of of vmf->virtual_address Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-15 21:55 ` Kirill A. Shutemov
2016-11-15 21:55 ` Kirill A. Shutemov
2016-11-16 11:05 ` Jan Kara
2016-11-16 11:05 ` Jan Kara
2016-11-16 11:32 ` Kirill A. Shutemov
2016-11-16 11:55 ` Jan Kara
2016-11-16 11:55 ` Jan Kara
2016-11-04 4:24 ` [PATCH 03/21] mm: Use pgoff in struct vm_fault instead of passing it separately Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-04 4:24 ` Jan Kara
2016-11-15 22:01 ` Kirill A. Shutemov
2016-11-15 22:01 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 04/21] mm: Use passed vm_fault structure in __do_fault() Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:05 ` Kirill A. Shutemov
2016-11-15 22:05 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 05/21] mm: Trim __do_fault() arguments Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:10 ` Kirill A. Shutemov
2016-11-15 22:10 ` Kirill A. Shutemov
2016-11-16 13:12 ` Jan Kara
2016-11-04 4:25 ` [PATCH 06/21] mm: Use passed vm_fault structure for in wp_pfn_shared() Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:10 ` Kirill A. Shutemov
2016-11-15 22:10 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 07/21] mm: Add orig_pte field into vm_fault Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:14 ` Kirill A. Shutemov
2016-11-15 22:14 ` Kirill A. Shutemov
2016-11-16 20:00 ` Ross Zwisler
2016-11-16 20:00 ` Ross Zwisler
2016-11-04 4:25 ` [PATCH 08/21] mm: Allow full handling of COW faults in ->fault handlers Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:20 ` Kirill A. Shutemov
2016-11-15 22:20 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 09/21] mm: Factor out functionality to finish page faults Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:21 ` Kirill A. Shutemov
2016-11-15 22:21 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 10/21] mm: Move handling of COW faults into DAX code Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:22 ` Kirill A. Shutemov
2016-11-15 22:22 ` Kirill A. Shutemov
2016-11-16 21:28 ` Ross Zwisler
2016-11-17 9:36 ` Jan Kara
2016-11-17 9:36 ` Jan Kara
2016-11-04 4:25 ` [PATCH 11/21] mm: Remove unnecessary vma->vm_ops check Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:28 ` Kirill A. Shutemov
2016-11-16 13:29 ` Jan Kara
2016-11-16 14:27 ` Kirill A. Shutemov
2016-11-16 14:27 ` Kirill A. Shutemov
2016-11-16 14:43 ` Jan Kara
2016-11-04 4:25 ` [PATCH 12/21] mm: Factor out common parts of write fault handling Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:30 ` Kirill A. Shutemov
2016-11-15 22:30 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 13/21] mm: Pass vm_fault structure into do_page_mkwrite() Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:40 ` Kirill A. Shutemov
2016-11-15 22:40 ` Kirill A. Shutemov
2016-11-16 13:34 ` Jan Kara
2016-11-04 4:25 ` [PATCH 14/21] mm: Use vmf->page during WP faults Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:42 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 15/21] mm: Move part of wp_page_reuse() into the single call site Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:44 ` Kirill A. Shutemov
2016-11-15 22:44 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 16/21] mm: Provide helper for finishing mkwrite faults Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:52 ` Kirill A. Shutemov
2016-11-15 22:52 ` Kirill A. Shutemov
2016-11-16 13:39 ` Jan Kara
2016-11-04 4:25 ` [PATCH 17/21] mm: Change return values of finish_mkwrite_fault() Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-15 22:57 ` Kirill A. Shutemov
2016-11-04 4:25 ` [PATCH 18/21] mm: Export follow_pte() Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` [PATCH 19/21] dax: Make cache flushing protected by entry lock Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` [PATCH 20/21] dax: Protect PTE modification on WP fault by radix tree " Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` [PATCH 21/21] dax: Clear dirty entry tags on cache flush Jan Kara
2016-11-04 4:25 ` Jan Kara
2016-11-04 4:25 ` Jan Kara
-- strict thread matches above, loose matches on Subject: below --
2016-11-01 22:36 [PATCH 0/21 v4] dax: Clear dirty bits after flushing caches Jan Kara
2016-11-01 22:36 ` [PATCH 07/21] mm: Add orig_pte field into vm_fault Jan Kara
2016-11-01 22:36 ` Jan Kara
2016-11-01 22:36 ` Jan Kara
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.