From: Ingo Molnar <mingo@kernel.org> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andy Lutomirski <luto@amacapital.net>, Andrew Morton <akpm@linux-foundation.org>, Denys Vlasenko <dvlasenk@redhat.com>, Brian Gerst <brgerst@gmail.com>, Peter Zijlstra <peterz@infradead.org>, Borislav Petkov <bp@alien8.de>, "H. Peter Anvin" <hpa@zytor.com>, Linus Torvalds <torvalds@linux-foundation.org>, Oleg Nesterov <oleg@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, Waiman Long <Waiman.Long@hp.com> Subject: [PATCH 02/12] x86/mm/hotplug: Remove pgd_list use from the memory hotplug code Date: Sat, 13 Jun 2015 11:49:05 +0200 [thread overview] Message-ID: <1434188955-31397-3-git-send-email-mingo@kernel.org> (raw) In-Reply-To: <1434188955-31397-1-git-send-email-mingo@kernel.org> The memory hotplug code uses sync_global_pgds() to synchronize updates to the global (&init_mm) kernel PGD and the task PGDs. It does this by iterating over the pgd_list - which list closely tracks task creation/destruction via fork/clone. But we want to remove this list, so that it does not have to be maintained from fork()/exit(), so convert the memory hotplug code to use the task list to iterate over all pgds in the system. Also improve the comments a bit, to make this function easier to understand. Only lightly tested, as I don't have a memory hotplug setup. Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Borislav Petkov <bp@alien8.de> Cc: Brian Gerst <brgerst@gmail.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Waiman Long <Waiman.Long@hp.com> Cc: linux-mm@kvack.org Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/mm/init_64.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 3fba623e3ba5..527d5d4d020c 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -160,8 +160,8 @@ static int __init nonx32_setup(char *str) __setup("noexec32=", nonx32_setup); /* - * When memory was added/removed make sure all the processes MM have - * suitable PGD entries in the local PGD level page. + * When memory was added/removed make sure all the process MMs have + * matching PGD entries in the local PGD level page as well. */ void sync_global_pgds(unsigned long start, unsigned long end, int removed) { @@ -169,29 +169,40 @@ void sync_global_pgds(unsigned long start, unsigned long end, int removed) for (address = start; address <= end; address += PGDIR_SIZE) { const pgd_t *pgd_ref = pgd_offset_k(address); - struct page *page; + struct task_struct *g, *p; /* - * When it is called after memory hot remove, pgd_none() - * returns true. In this case (removed == 1), we must clear - * the PGD entries in the local PGD level page. + * When this function is called after memory hot remove, + * pgd_none() already returns true, but only the reference + * kernel PGD has been cleared, not the process PGDs. + * + * So clear the affected entries in every process PGD as well: */ if (pgd_none(*pgd_ref) && !removed) continue; - spin_lock(&pgd_lock); - list_for_each_entry(page, &pgd_list, lru) { + spin_lock(&pgd_lock); /* Implies rcu_read_lock() for the task list iteration: */ + + for_each_process_thread(g, p) { + struct mm_struct *mm; pgd_t *pgd; spinlock_t *pgt_lock; - pgd = (pgd_t *)page_address(page) + pgd_index(address); - /* the pgt_lock only for Xen */ - pgt_lock = &pgd_page_get_mm(page)->page_table_lock; + task_lock(p); + mm = p->mm; + if (!mm) { + task_unlock(p); + continue; + } + + pgd = mm->pgd; + + /* The pgt_lock is only used by Xen: */ + pgt_lock = &mm->page_table_lock; spin_lock(pgt_lock); if (!pgd_none(*pgd_ref) && !pgd_none(*pgd)) - BUG_ON(pgd_page_vaddr(*pgd) - != pgd_page_vaddr(*pgd_ref)); + BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref)); if (removed) { if (pgd_none(*pgd_ref) && !pgd_none(*pgd)) @@ -202,6 +213,7 @@ void sync_global_pgds(unsigned long start, unsigned long end, int removed) } spin_unlock(pgt_lock); + task_unlock(p); } spin_unlock(&pgd_lock); } -- 2.1.4
WARNING: multiple messages have this Message-ID (diff)
From: Ingo Molnar <mingo@kernel.org> To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, Andy Lutomirski <luto@amacapital.net>, Andrew Morton <akpm@linux-foundation.org>, Denys Vlasenko <dvlasenk@redhat.com>, Brian Gerst <brgerst@gmail.com>, Peter Zijlstra <peterz@infradead.org>, Borislav Petkov <bp@alien8.de>, "H. Peter Anvin" <hpa@zytor.com>, Linus Torvalds <torvalds@linux-foundation.org>, Oleg Nesterov <oleg@redhat.com>, Thomas Gleixner <tglx@linutronix.de>, Waiman Long <Waiman.Long@hp.com> Subject: [PATCH 02/12] x86/mm/hotplug: Remove pgd_list use from the memory hotplug code Date: Sat, 13 Jun 2015 11:49:05 +0200 [thread overview] Message-ID: <1434188955-31397-3-git-send-email-mingo@kernel.org> (raw) In-Reply-To: <1434188955-31397-1-git-send-email-mingo@kernel.org> The memory hotplug code uses sync_global_pgds() to synchronize updates to the global (&init_mm) kernel PGD and the task PGDs. It does this by iterating over the pgd_list - which list closely tracks task creation/destruction via fork/clone. But we want to remove this list, so that it does not have to be maintained from fork()/exit(), so convert the memory hotplug code to use the task list to iterate over all pgds in the system. Also improve the comments a bit, to make this function easier to understand. Only lightly tested, as I don't have a memory hotplug setup. Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Borislav Petkov <bp@alien8.de> Cc: Brian Gerst <brgerst@gmail.com> Cc: Denys Vlasenko <dvlasenk@redhat.com> Cc: H. Peter Anvin <hpa@zytor.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Oleg Nesterov <oleg@redhat.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Waiman Long <Waiman.Long@hp.com> Cc: linux-mm@kvack.org Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/mm/init_64.c | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 3fba623e3ba5..527d5d4d020c 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -160,8 +160,8 @@ static int __init nonx32_setup(char *str) __setup("noexec32=", nonx32_setup); /* - * When memory was added/removed make sure all the processes MM have - * suitable PGD entries in the local PGD level page. + * When memory was added/removed make sure all the process MMs have + * matching PGD entries in the local PGD level page as well. */ void sync_global_pgds(unsigned long start, unsigned long end, int removed) { @@ -169,29 +169,40 @@ void sync_global_pgds(unsigned long start, unsigned long end, int removed) for (address = start; address <= end; address += PGDIR_SIZE) { const pgd_t *pgd_ref = pgd_offset_k(address); - struct page *page; + struct task_struct *g, *p; /* - * When it is called after memory hot remove, pgd_none() - * returns true. In this case (removed == 1), we must clear - * the PGD entries in the local PGD level page. + * When this function is called after memory hot remove, + * pgd_none() already returns true, but only the reference + * kernel PGD has been cleared, not the process PGDs. + * + * So clear the affected entries in every process PGD as well: */ if (pgd_none(*pgd_ref) && !removed) continue; - spin_lock(&pgd_lock); - list_for_each_entry(page, &pgd_list, lru) { + spin_lock(&pgd_lock); /* Implies rcu_read_lock() for the task list iteration: */ + + for_each_process_thread(g, p) { + struct mm_struct *mm; pgd_t *pgd; spinlock_t *pgt_lock; - pgd = (pgd_t *)page_address(page) + pgd_index(address); - /* the pgt_lock only for Xen */ - pgt_lock = &pgd_page_get_mm(page)->page_table_lock; + task_lock(p); + mm = p->mm; + if (!mm) { + task_unlock(p); + continue; + } + + pgd = mm->pgd; + + /* The pgt_lock is only used by Xen: */ + pgt_lock = &mm->page_table_lock; spin_lock(pgt_lock); if (!pgd_none(*pgd_ref) && !pgd_none(*pgd)) - BUG_ON(pgd_page_vaddr(*pgd) - != pgd_page_vaddr(*pgd_ref)); + BUG_ON(pgd_page_vaddr(*pgd) != pgd_page_vaddr(*pgd_ref)); if (removed) { if (pgd_none(*pgd_ref) && !pgd_none(*pgd)) @@ -202,6 +213,7 @@ void sync_global_pgds(unsigned long start, unsigned long end, int removed) } spin_unlock(pgt_lock); + task_unlock(p); } spin_unlock(&pgd_lock); } -- 2.1.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2015-06-13 9:49 UTC|newest] Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top 2015-06-13 9:49 [PATCH 00/12, v2] x86/mm: Implement lockless pgd_alloc()/pgd_free() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 01/12] x86/mm/pat: Don't free PGD entries on memory unmap Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar [this message] 2015-06-13 9:49 ` [PATCH 02/12] x86/mm/hotplug: Remove pgd_list use from the memory hotplug code Ingo Molnar 2015-06-13 19:24 ` Oleg Nesterov 2015-06-13 19:24 ` Oleg Nesterov 2015-06-14 7:36 ` Ingo Molnar 2015-06-14 7:36 ` Ingo Molnar 2015-06-14 19:24 ` Oleg Nesterov 2015-06-14 19:24 ` Oleg Nesterov 2015-06-14 19:38 ` Oleg Nesterov 2015-06-14 19:38 ` Oleg Nesterov 2015-06-15 0:40 ` Paul E. McKenney 2015-06-15 0:40 ` Paul E. McKenney 2015-06-15 20:33 ` Ingo Molnar 2015-06-15 20:33 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 03/12] x86/mm/hotplug: Don't remove PGD entries in remove_pagetable() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 04/12] x86/mm/hotplug: Simplify sync_global_pgds() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 05/12] mm: Introduce arch_pgd_init_late() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 06/12] x86/mm: Enable and use the arch_pgd_init_late() method Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 07/12] x86/virt/guest/xen: Remove use of pgd_list from the Xen guest code Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-14 8:26 ` Ingo Molnar 2015-06-14 8:26 ` Ingo Molnar 2015-06-15 9:05 ` Ian Campbell 2015-06-15 9:05 ` Ian Campbell 2015-06-15 9:05 ` Ian Campbell 2015-06-15 10:30 ` David Vrabel 2015-06-15 10:30 ` David Vrabel 2015-06-15 20:35 ` Ingo Molnar 2015-06-15 20:35 ` Ingo Molnar 2015-06-15 20:35 ` Ingo Molnar 2015-06-16 14:15 ` David Vrabel 2015-06-16 14:15 ` David Vrabel 2015-06-16 14:15 ` David Vrabel 2015-06-16 14:19 ` Boris Ostrovsky 2015-06-16 14:19 ` Boris Ostrovsky 2015-06-16 14:19 ` Boris Ostrovsky 2015-06-16 14:27 ` David Vrabel 2015-06-16 14:27 ` David Vrabel 2015-06-16 14:27 ` David Vrabel 2015-06-15 10:30 ` David Vrabel 2015-06-13 9:49 ` [PATCH 08/12] x86/mm: Remove pgd_list use from vmalloc_sync_all() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 09/12] x86/mm/pat/32: Remove pgd_list use from the PAT code Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 10/12] x86/mm: Make pgd_alloc()/pgd_free() lockless Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 11/12] x86/mm: Remove pgd_list leftovers Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 9:49 ` [PATCH 12/12] x86/mm: Simplify pgd_alloc() Ingo Molnar 2015-06-13 9:49 ` Ingo Molnar 2015-06-13 18:58 ` why do we need vmalloc_sync_all? Oleg Nesterov 2015-06-13 18:58 ` Oleg Nesterov 2015-06-14 7:59 ` Ingo Molnar 2015-06-14 7:59 ` Ingo Molnar 2015-06-14 20:06 ` Oleg Nesterov 2015-06-14 20:06 ` Oleg Nesterov 2015-06-15 2:47 ` Andi Kleen 2015-06-15 2:47 ` Andi Kleen 2015-06-15 2:57 ` Andy Lutomirski 2015-06-15 2:57 ` Andy Lutomirski 2015-06-15 20:28 ` Ingo Molnar 2015-06-15 20:28 ` Ingo Molnar 2015-06-15 20:48 ` Andy Lutomirski 2015-06-15 20:48 ` Andy Lutomirski -- strict thread matches above, loose matches on Subject: below -- 2015-06-11 14:07 [RFC PATCH 00/12] x86/mm: Implement lockless pgd_alloc()/pgd_free() Ingo Molnar 2015-06-11 14:07 ` [PATCH 02/12] x86/mm/hotplug: Remove pgd_list use from the memory hotplug code Ingo Molnar 2015-06-12 22:22 ` Oleg Nesterov 2015-06-12 22:48 ` Waiman Long 2015-06-13 7:46 ` Ingo Molnar 2015-06-12 23:15 ` Oleg Nesterov 2015-06-13 7:48 ` Ingo Molnar
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1434188955-31397-3-git-send-email-mingo@kernel.org \ --to=mingo@kernel.org \ --cc=Waiman.Long@hp.com \ --cc=akpm@linux-foundation.org \ --cc=bp@alien8.de \ --cc=brgerst@gmail.com \ --cc=dvlasenk@redhat.com \ --cc=hpa@zytor.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=luto@amacapital.net \ --cc=oleg@redhat.com \ --cc=peterz@infradead.org \ --cc=tglx@linutronix.de \ --cc=torvalds@linux-foundation.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.