From: Rik van Riel <riel@redhat.com>
To: linux-kernel@vger.kernel.org
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>,
Lee Schermerhorn <Lee.Schermerhorn@hp.com>,
linux-mm@kvack.org, Lee Schermerhorn <lee.schermerhorn@hp.com>
Subject: [patch 18/21] mlock vma pages under mmap_sem held for read
Date: Thu, 28 Feb 2008 14:29:26 -0500 [thread overview]
Message-ID: <20080228192929.537996441@redhat.com> (raw)
In-Reply-To: 20080228192908.126720629@redhat.com
[-- Attachment #1: noreclaim-04.1a-lock-vma-pages-under-read-lock.patch --]
[-- Type: text/plain, Size: 6681 bytes --]
V2 -> V3:
+ rebase to 23-mm1 atop RvR's split lru series [no change]
+ fix function return types [void -> int] to fix build when
not configured.
New in V2.
We need to hold the mmap_sem for write to initiatate mlock()/munlock()
because we may need to merge/split vmas. However, this can lead to
very long lock hold times attempting to fault in a large memory region
to mlock it into memory. This can hold off other faults against the
mm [multithreaded tasks] and other scans of the mm, such as via /proc.
To alleviate this, downgrade the mmap_sem to read mode during the
population of the region for locking. This is especially the case
if we need to reclaim memory to lock down the region. We [probably?]
don't need to do this for unlocking as all of the pages should be
resident--they're already mlocked.
Now, the caller's of the mlock functions [mlock_fixup() and
mlock_vma_pages_range()] expect the mmap_sem to be returned in write
mode. Changing all callers appears to be way too much effort at this
point. So, restore write mode before returning. Note that this opens
a window where the mmap list could change in a multithreaded process.
So, at least for mlock_fixup(), where we could be called in a loop over
multiple vmas, we check that a vma still exists at the start address
and that vma still covers the page range [start,end). If not, we return
an error, -EAGAIN, and let the caller deal with it.
Return -EAGAIN from mlock_vma_pages_range() function and mlock_fixup()
if the vma at 'start' disappears or changes so that the page range
[start,end) is no longer contained in the vma. Again, let the caller
deal with it. Looks like only sys_remap_file_pages() [via mmap_region()]
should actually care.
With this patch, I no longer see processes like ps(1) blocked for seconds
or minutes at a time waiting for a large [multiple gigabyte] region to be
locked down.
Signed-off-by: Lee Schermerhorn <lee.schermerhorn@hp.com>
Signed-off-by: Rik van Riel <riel@redhat.com>
Index: linux-2.6.25-rc2-mm1/mm/mlock.c
===================================================================
--- linux-2.6.25-rc2-mm1.orig/mm/mlock.c 2008-02-28 12:49:02.000000000 -0500
+++ linux-2.6.25-rc2-mm1/mm/mlock.c 2008-02-28 12:49:09.000000000 -0500
@@ -200,6 +200,37 @@ int __mlock_vma_pages_range(struct vm_ar
return ret;
}
+/**
+ * mlock_vma_pages_range
+ * @vma - vm area to mlock into memory
+ * @start - start address in @vma of range to mlock,
+ * @end - end address in @vma of range
+ *
+ * Called with current->mm->mmap_sem held write locked. Downgrade to read
+ * for faulting in pages. This can take a looong time for large segments.
+ *
+ * We need to restore the mmap_sem to write locked because our callers'
+ * callers expect this. However, because the mmap could have changed
+ * [in a multi-threaded process], we need to recheck.
+ */
+int mlock_vma_pages_range(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end)
+{
+ struct mm_struct *mm = vma->vm_mm;
+
+ downgrade_write(&mm->mmap_sem);
+ __mlock_vma_pages_range(vma, start, end, 1);
+
+ up_read(&mm->mmap_sem);
+ /* vma can change or disappear */
+ down_write(&mm->mmap_sem);
+ vma = find_vma(mm, start);
+ /* non-NULL vma must contain @start, but need to check @end */
+ if (!vma || end > vma->vm_end)
+ return -EAGAIN;
+ return 0;
+}
+
#else /* CONFIG_NORECLAIM_MLOCK */
/*
@@ -266,14 +297,38 @@ success:
mm->locked_vm += nr_pages;
/*
- * vm_flags is protected by the mmap_sem held in write mode.
+ * vm_flags is protected by the mmap_sem held for write.
* It's okay if try_to_unmap_one unmaps a page just after we
* set VM_LOCKED, __mlock_vma_pages_range will bring it back.
*/
vma->vm_flags = newflags;
+ /*
+ * mmap_sem is currently held for write. If we're locking pages,
+ * downgrade the write lock to a read lock so that other faults,
+ * mmap scans, ... while we fault in all pages.
+ */
+ if (lock)
+ downgrade_write(&mm->mmap_sem);
+
__mlock_vma_pages_range(vma, start, end, lock);
+ if (lock) {
+ /*
+ * Need to reacquire mmap sem in write mode, as our callers
+ * expect this. We have no support for atomically upgrading
+ * a sem to write, so we need to check for changes while sem
+ * is unlocked.
+ */
+ up_read(&mm->mmap_sem);
+ /* vma can change or disappear */
+ down_write(&mm->mmap_sem);
+ *prev = find_vma(mm, start);
+ /* non-NULL *prev must contain @start, but need to check @end */
+ if (!(*prev) || end > (*prev)->vm_end)
+ ret = -EAGAIN;
+ }
+
out:
if (ret == -ENOMEM)
ret = -EAGAIN;
Index: linux-2.6.25-rc2-mm1/mm/internal.h
===================================================================
--- linux-2.6.25-rc2-mm1.orig/mm/internal.h 2008-02-28 12:49:02.000000000 -0500
+++ linux-2.6.25-rc2-mm1/mm/internal.h 2008-02-28 12:49:09.000000000 -0500
@@ -61,24 +61,21 @@ extern int __mlock_vma_pages_range(struc
/*
* mlock all pages in this vma range. For mmap()/mremap()/...
*/
-static inline void mlock_vma_pages_range(struct vm_area_struct *vma,
- unsigned long start, unsigned long end)
-{
- __mlock_vma_pages_range(vma, start, end, 1);
-}
+extern int mlock_vma_pages_range(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end);
/*
* munlock range of pages. For munmap() and exit().
* Always called to operate on a full vma that is being unmapped.
*/
-static inline void munlock_vma_pages_range(struct vm_area_struct *vma,
+static inline int munlock_vma_pages_range(struct vm_area_struct *vma,
unsigned long start, unsigned long end)
{
// TODO: verify my assumption. Should we just drop the start/end args?
VM_BUG_ON(start != vma->vm_start || end != vma->vm_end);
vma->vm_flags &= ~VM_LOCKED; /* try_to_unlock() needs this */
- __mlock_vma_pages_range(vma, start, end, 0);
+ return __mlock_vma_pages_range(vma, start, end, 0);
}
extern void clear_page_mlock(struct page *page);
@@ -90,10 +87,10 @@ static inline int is_mlocked_vma(struct
}
static inline void clear_page_mlock(struct page *page) { }
static inline void mlock_vma_page(struct page *page) { }
-static inline void mlock_vma_pages_range(struct vm_area_struct *vma,
- unsigned long start, unsigned long end) { }
-static inline void munlock_vma_pages_range(struct vm_area_struct *vma,
- unsigned long start, unsigned long end) { }
+static inline int mlock_vma_pages_range(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end) { return 0; }
+static inline int munlock_vma_pages_range(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end) { return 0; }
#endif /* CONFIG_NORECLAIM_MLOCK */
--
All Rights Reversed
next prev parent reply other threads:[~2008-02-28 19:51 UTC|newest]
Thread overview: 60+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-02-28 19:29 [patch 00/21] VM pageout scalability improvements Rik van Riel
2008-02-28 19:29 ` [patch 01/21] move isolate_lru_page() to vmscan.c Rik van Riel
2008-02-29 2:29 ` KOSAKI Motohiro
2008-02-29 2:41 ` Rik van Riel
2008-02-29 2:47 ` KOSAKI Motohiro
2008-02-28 19:29 ` [patch 02/21] Use an indexed array for LRU variables Rik van Riel
2008-02-29 16:03 ` Andy Whitcroft
2008-03-03 18:57 ` Rik van Riel
2008-02-28 19:29 ` [patch 03/21] use an array for the LRU pagevecs Rik van Riel
2008-02-29 15:40 ` Andy Whitcroft
2008-03-01 7:02 ` KOSAKI Motohiro
2008-03-04 11:04 ` KOSAKI Motohiro
2008-03-04 20:38 ` Rik van Riel
2008-03-05 1:38 ` KOSAKI Motohiro
2008-02-28 19:29 ` [patch 04/21] free swap space on swap-in/activation Rik van Riel
2008-02-28 20:05 ` Lee Schermerhorn
2008-02-28 20:20 ` Rik van Riel
2008-02-28 19:29 ` [patch 05/21] define page_file_cache() function Rik van Riel
2008-02-29 11:53 ` KOSAKI Motohiro
2008-02-28 19:29 ` [patch 06/21] split LRU lists into anon & file sets Rik van Riel
2008-03-01 12:13 ` KOSAKI Motohiro
2008-03-01 12:46 ` KOSAKI Motohiro
2008-02-28 19:29 ` [patch 07/21] SEQ replacement for anonymous pages Rik van Riel
2008-03-03 10:50 ` barrioskmc@gmail
2008-02-28 19:29 ` [patch 08/21] (NEW) add some sanity checks to get_scan_ratio Rik van Riel
2008-03-04 10:40 ` minchan Kim
2008-02-28 19:29 ` [patch 09/21] (NEW) improve reclaim balancing Rik van Riel
2008-03-01 13:35 ` KOSAKI Motohiro
2008-03-03 19:26 ` Rik van Riel
2008-02-28 19:29 ` [patch 10/21] add newly swapped in pages to the inactive list Rik van Riel
2008-02-28 19:29 ` [patch 11/21] (NEW) more aggressively use lumpy reclaim Rik van Riel
2008-03-02 10:35 ` KOSAKI Motohiro
2008-03-02 14:23 ` Rik van Riel
2008-02-28 19:29 ` [patch 12/21] No Reclaim LRU Infrastructure Rik van Riel
[not found] ` <44c63dc40802282058h67f7597bvb614575f06c62e2c@mail.gmail.com>
2008-02-29 14:48 ` Lee Schermerhorn
[not found] ` <44c63dc40803021904n5de681datba400e08079c152d@mail.gmail.com>
2008-03-03 3:06 ` minchan Kim
2008-03-03 18:46 ` Rik van Riel
2008-03-03 23:38 ` barrioskmc@gmail
2008-03-04 1:55 ` Rik van Riel
2008-03-04 10:46 ` KOSAKI Motohiro
2008-03-04 15:05 ` Lee Schermerhorn
2008-03-04 21:21 ` Rik van Riel
2008-03-05 1:42 ` KOSAKI Motohiro
2008-02-28 19:29 ` [patch 13/21] Non-reclaimable page statistics Rik van Riel
2008-02-28 19:29 ` [patch 14/21] scan noreclaim list for reclaimable pages Rik van Riel
2008-02-28 23:41 ` Randy Dunlap
2008-02-29 14:38 ` Lee Schermerhorn
2008-02-28 19:29 ` [patch 15/21] ramfs pages are non-reclaimable Rik van Riel
2008-02-28 19:29 ` [patch 16/21] SHM_LOCKED pages are nonreclaimable Rik van Riel
2008-02-28 19:29 ` [patch 17/21] non-reclaimable mlocked pages Rik van Riel
[not found] ` <44c63dc40802282055q508af6ccsb0e8ac3fb5e67d24@mail.gmail.com>
2008-02-29 14:47 ` Lee Schermerhorn
2008-02-28 19:29 ` Rik van Riel [this message]
2008-02-28 19:29 ` [patch 19/21] handle mlocked pages during map/unmap and truncate Rik van Riel
2008-02-28 19:29 ` [patch 20/21] account mlocked pages Rik van Riel
2008-02-28 19:29 ` [patch 21/21] cull non-reclaimable anon pages from the LRU at fault time Rik van Riel
2008-02-28 20:19 ` Lee Schermerhorn
2008-02-28 22:27 ` Rik van Riel
2008-02-28 19:49 ` [patch 00/21] VM pageout scalability improvements Rik van Riel
2008-02-28 20:14 ` John Stoffel
2008-02-28 20:23 ` Rik van Riel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20080228192929.537996441@redhat.com \
--to=riel@redhat.com \
--cc=Lee.Schermerhorn@hp.com \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).