From: Christoph Lameter <clameter@sgi.com>
To: Andrea Arcangeli <andrea@qumranet.com>
Cc: Robin Holt <holt@sgi.com>, Avi Kivity <avi@qumranet.com>,
Izik Eidus <izike@qumranet.com>, Andrew Morton <akpm@osdl.org>,
Nick Piggin <npiggin@suse.de>,
kvm-devel@lists.sourceforge.net,
Benjamin Herrenschmidt <benh@kernel.crashing.org>,
steiner@sgi.com, linux-kernel@vger.kernel.org,
linux-mm@kvack.org, daniel.blueman@quadrics.com,
Hugh Dickins <hugh@veritas.com>
Subject: Re: [kvm-devel] [PATCH] export notifier #1
Date: Wed, 23 Jan 2008 12:27:47 -0800 (PST) [thread overview]
Message-ID: <Pine.LNX.4.64.0801231220590.13547@schroedinger.engr.sgi.com> (raw)
In-Reply-To: <20080123173325.GG7141@v2.random>
On Wed, 23 Jan 2008, Andrea Arcangeli wrote:
> You want to be able to tell the mmu_notifier that you want the flush
> repeated without locks later? Sorry but then if you're always going to
> set the bitflag unconditionally, why don't you simply implement a
> second notifier in addition of my current ->invalidate_page (like
> ->after_invalidate_page).
Because there is no mm_struct available at that point. So we cannot do a
callback based on the mmu_ops in that fasion. We would have to build a
list of notifiers while scanning the reverse maps.
> We can then implement a method in rmap.c for you to call to do the
> final freeing of the page (pagecache/swapcache won't be collected
> unless it's a truncate, as long as you keep it pinned and you
> certainly don't want to wait a second round of lru scan before freeing
> the page after you release the external reference, so you may need to
> call this method before returning from the
The page count is elevated because of the remote pte so the page is
effectively pinned.
> ->after_invalidate_page). Infact I can call that method for you in the
> notifier implementation itself after all ->after_invalidate_pages have
> been called. (of course only if at least one of them was implemented
> and not-null)
Ok.
> > As an example of thousands, we currently have one customer job that
> > has 16880 processors all with the same physical page faulted into their
> > address space. The way XPMEM is currently structured, there is fan-out of
> > that PFN information so we would not need to queue up that many messages,
> > but it would still be considerable. Our upcoming version of the hardware
> > will potentially make this fanout worse because we are going to allow
> > even more fine-grained divisions of the machine to help with memory
> > error containment.
>
> Well as long as you send these messages somewhat serially and you
> don't pretend to allocate all packets at once it should be ok. Perhaps
> you should preallocate all packets statically and serialize the access
> to the pool with a lock.
>
> What I'd like to stress to be sure it's crystal clear, is that in the
> mm/rmap.c path GFP_KERNEL==GFP_ATOMIC, infact both are = PF_MEMALLOC =
> TIF_MEMDIE = if mempool is empty it will crash. The argument that you
> need to sleep to allocate memory with GFP_KERNEL is totally bogus. If
> that's the only reason, you don't need to sleep at all. alloc_pages
> will not invoke the VM when called inside the VM, it will grab ram
> from PF_MEMALLOC instead. At most it will schedule so the only benefit
> would be lower -rt latency in the end.
If you are holding a lock then you have to use GFP_ATOMIC and the number
of GFP_ATOMIC allocs is limited. PF_MEMALLOC does not do reclaim so we are
in trouble if too many allocs occur.
> > We have a counter associated with a pfn that indicates when the pfn is no
> > longer referenced by other partitions. This counter triggers changing of
> > memory protections so any subsequent access to this page will result in
> > a memory error on the remote partition (this should be an illegal case).
>
> As long as you keep a reference on the page too, you don't risk
> any corruption by flushing after.
There are still dirty bit issues.
> The window that you must close with that bitflag is the request coming
> from the remote node to map the page after the linux pte has been
> cleared. If you map the page in a remote node after the linux pte has
> been cleared ->invalidate_page won't be called again because the page
> will look unmapped in the linux VM. Now invalidate_page will clear the
> bitflag, so the map requests will block. But where exactly you know
> that the linux pte has been cleared so you can "unblock" the map
> requests? If a page is not mapped by some linux pte, mm/rmap.c will
> never be called and this is why any notification in mm/rmap.c should
> track the "address space" and not the "physical page".
The subsystem needs to establish proper locking for that case.
> In effect you don't care less about the address space of the task in
> the master node, so IMHO you're hooking your ->invalidate_page(page)
> (instead of my ->invalidate_page(mm, address)) in the very wrong
> place. You should hook it in mm/vmscan.c shrink-list so it will be
> invoked regardless if the pte is mapped or not. Then your model that
Then page migration and other uses of try_to_unmap wont get there. Also
the page lock is an item that helps with serialization of new faults.
> If you work the "pages" you should stick to pages and to stay away
> from mm/rmap.c and ignore whatever is mapped in the master address
> space of the task. mm/rmap.c only deals with ptes/sptes and other
> _virtual-tracked_ mappings.
It also deals f.e. with page dirty status.
next prev parent reply other threads:[~2008-01-23 20:28 UTC|newest]
Thread overview: 67+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-01-13 16:24 [PATCH] mmu notifiers #v2 Andrea Arcangeli
2008-01-13 21:11 ` Benjamin Herrenschmidt
2008-01-14 20:02 ` Christoph Lameter
2008-01-15 4:28 ` Benjamin Herrenschmidt
2008-01-15 12:44 ` Andrea Arcangeli
2008-01-15 20:18 ` Benjamin Herrenschmidt
2008-01-16 1:06 ` Andrea Arcangeli
2008-01-16 9:01 ` Brice Goglin
2008-01-16 10:19 ` Andrea Arcangeli
2008-01-16 17:42 ` Rik van Riel
2008-01-16 17:48 ` Izik Eidus
2008-01-17 16:23 ` Andrea Arcangeli
2008-01-17 18:21 ` Izik Eidus
2008-01-17 19:32 ` Andrea Arcangeli
2008-01-21 12:52 ` [PATCH] mmu notifiers #v3 Andrea Arcangeli
2008-01-22 2:21 ` Rik van Riel
2008-01-22 14:12 ` [kvm-devel] " Avi Kivity
2008-01-22 14:43 ` Andrea Arcangeli
2008-01-22 20:08 ` [kvm-devel] [PATCH] mmu notifiers #v4 Andrea Arcangeli
2008-01-22 20:34 ` [kvm-devel] [PATCH] export notifier #1 Christoph Lameter
2008-01-22 22:31 ` Andrea Arcangeli
2008-01-22 22:53 ` Christoph Lameter
2008-01-23 10:27 ` Avi Kivity
2008-01-23 10:52 ` Robin Holt
2008-01-23 12:04 ` Andrea Arcangeli
2008-01-23 12:34 ` Robin Holt
2008-01-23 19:48 ` Christoph Lameter
2008-01-23 19:58 ` Robin Holt
2008-01-23 19:47 ` Christoph Lameter
2008-01-24 5:56 ` Avi Kivity
2008-01-24 12:26 ` Andrea Arcangeli
2008-01-24 12:34 ` Avi Kivity
2008-01-23 11:41 ` Andrea Arcangeli
2008-01-23 12:32 ` Robin Holt
2008-01-23 17:33 ` Andrea Arcangeli
2008-01-23 20:27 ` Christoph Lameter [this message]
2008-01-24 15:42 ` Andrea Arcangeli
2008-01-24 20:07 ` Christoph Lameter
2008-01-25 6:35 ` Avi Kivity
2008-01-23 20:18 ` Christoph Lameter
2008-01-24 14:34 ` Andrea Arcangeli
2008-01-24 14:41 ` Andrea Arcangeli
2008-01-24 15:15 ` Avi Kivity
2008-01-24 15:18 ` Avi Kivity
2008-01-24 20:01 ` Christoph Lameter
2008-01-22 23:36 ` Benjamin Herrenschmidt
2008-01-23 0:40 ` Christoph Lameter
2008-01-23 1:21 ` Robin Holt
2008-01-23 12:51 ` Gerd Hoffmann
2008-01-23 13:19 ` Robin Holt
2008-01-23 14:12 ` Gerd Hoffmann
2008-01-23 14:18 ` Robin Holt
2008-01-23 14:35 ` Gerd Hoffmann
2008-01-23 15:48 ` Robin Holt
2008-01-23 14:17 ` Avi Kivity
2008-01-24 4:03 ` Benjamin Herrenschmidt
2008-01-23 15:41 ` Andrea Arcangeli
2008-01-23 17:47 ` Gerd Hoffmann
2008-01-24 6:01 ` Avi Kivity
2008-01-24 6:45 ` Jeremy Fitzhardinge
2008-01-23 20:40 ` Christoph Lameter
2008-01-24 2:00 ` Enhance mmu notifiers to accomplish a lockless implementation (incomplete) Robin Holt
2008-01-24 4:05 ` Robin Holt
2008-01-22 19:28 ` [PATCH] mmu notifiers #v3 Peter Zijlstra
2008-01-22 20:31 ` Christoph Lameter
2008-01-22 20:31 ` Andrea Arcangeli
2008-01-22 22:10 ` Hugh Dickins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Pine.LNX.4.64.0801231220590.13547@schroedinger.engr.sgi.com \
--to=clameter@sgi.com \
--cc=akpm@osdl.org \
--cc=andrea@qumranet.com \
--cc=avi@qumranet.com \
--cc=benh@kernel.crashing.org \
--cc=daniel.blueman@quadrics.com \
--cc=holt@sgi.com \
--cc=hugh@veritas.com \
--cc=izike@qumranet.com \
--cc=kvm-devel@lists.sourceforge.net \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=npiggin@suse.de \
--cc=steiner@sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).