From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> To: Alex Thorlton <athorlton@sgi.com>, Ingo Molnar <mingo@redhat.com>, Andrew Morton <akpm@linux-foundation.org>, Naoya Horiguchi <n-horiguchi@ah.jp.nec.com> Cc: "Eric W . Biederman" <ebiederm@xmission.com>, "Paul E . McKenney" <paulmck@linux.vnet.ibm.com>, Al Viro <viro@zeniv.linux.org.uk>, Andi Kleen <ak@linux.intel.com>, Andrea Arcangeli <aarcange@redhat.com>, Dave Hansen <dave.hansen@intel.com>, Dave Jones <davej@redhat.com>, David Howells <dhowells@redhat.com>, Frederic Weisbecker <fweisbec@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>, Kees Cook <keescook@chromium.org>, Mel Gorman <mgorman@suse.de>, Michael Kerrisk <mtk.manpages@gmail.com>, Oleg Nesterov <oleg@redhat.com>, Peter Zijlstra <peterz@infradead.org>, Rik van Riel <riel@redhat.com>, Robin Holt <robinmholt@gmail.com>, Sedat Dilek <sedat.dilek@gmail.com>, Srikar Dronamraju <srikar@linux.vnet.ibm.com>, Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> Subject: [PATCH 6/9] mm, thp: do not access mm->pmd_huge_pte directly Date: Fri, 13 Sep 2013 16:06:13 +0300 [thread overview] Message-ID: <1379077576-2472-7-git-send-email-kirill.shutemov@linux.intel.com> (raw) In-Reply-To: <1379077576-2472-1-git-send-email-kirill.shutemov@linux.intel.com> Currently mm->pmd_huge_pte protected by page table lock. It will not work with split lock. We have to have per-pmd pmd_huge_pte for proper access serialization. For now, let's just introduce wrapper to access mm->pmd_huge_pte. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> --- arch/s390/mm/pgtable.c | 12 ++++++------ arch/sparc/mm/tlb.c | 12 ++++++------ include/linux/mm.h | 1 + mm/pgtable-generic.c | 12 ++++++------ 4 files changed, 19 insertions(+), 18 deletions(-) diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index de8cbc3..c463e5c 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -1225,11 +1225,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(lh); else - list_add(lh, (struct list_head *) mm->pmd_huge_pte); - mm->pmd_huge_pte = pgtable; + list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp)); + pmd_huge_pte(mm, pmdp) = pgtable; } pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) @@ -1241,12 +1241,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); lh = (struct list_head *) pgtable; if (list_empty(lh)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = (pgtable_t) lh->next; + pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next; list_del(lh); } ptep = (pte_t *) pgtable; diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c index 7a91f28..656cc46 100644 --- a/arch/sparc/mm/tlb.c +++ b/arch/sparc/mm/tlb.c @@ -196,11 +196,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(lh); else - list_add(lh, (struct list_head *) mm->pmd_huge_pte); - mm->pmd_huge_pte = pgtable; + list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp)); + pmd_huge_pte(mm, pmdp) = pgtable; } pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) @@ -211,12 +211,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); lh = (struct list_head *) pgtable; if (list_empty(lh)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = (pgtable_t) lh->next; + pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next; list_del(lh); } pte_val(pgtable[0]) = 0; diff --git a/include/linux/mm.h b/include/linux/mm.h index d4361e7..d2f8a50 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1299,6 +1299,7 @@ static inline spinlock_t *huge_pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) return &mm->page_table_lock; } +#define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) static inline spinlock_t *huge_pmd_lock(struct mm_struct *mm, pmd_t *pmd) { diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 3929a40..41fee3e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -154,11 +154,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(&pgtable->lru); else - list_add(&pgtable->lru, &mm->pmd_huge_pte->lru); - mm->pmd_huge_pte = pgtable; + list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->lru); + pmd_huge_pte(mm, pmdp) = pgtable; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif @@ -173,11 +173,11 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); if (list_empty(&pgtable->lru)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = list_entry(pgtable->lru.next, + pmd_huge_pte(mm, pmdp) = list_entry(pgtable->lru.next, struct page, lru); list_del(&pgtable->lru); } -- 1.8.4.rc3
WARNING: multiple messages have this Message-ID (diff)
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> To: Alex Thorlton <athorlton@sgi.com>, Ingo Molnar <mingo@redhat.com>, Andrew Morton <akpm@linux-foundation.org>, Naoya Horiguchi <n-horiguchi@ah.jp.nec.com> Cc: "Eric W . Biederman" <ebiederm@xmission.com>, "Paul E . McKenney" <paulmck@linux.vnet.ibm.com>, Al Viro <viro@zeniv.linux.org.uk>, Andi Kleen <ak@linux.intel.com>, Andrea Arcangeli <aarcange@redhat.com>, Dave Hansen <dave.hansen@intel.com>, Dave Jones <davej@redhat.com>, David Howells <dhowells@redhat.com>, Frederic Weisbecker <fweisbec@gmail.com>, Johannes Weiner <hannes@cmpxchg.org>, Kees Cook <keescook@chromium.org>, Mel Gorman <mgorman@suse.de>, Michael Kerrisk <mtk.manpages@gmail.com>, Oleg Nesterov <oleg@redhat.com>, Peter Zijlstra <peterz@infradead.org>, Rik van Riel <riel@redhat.com>, Robin Holt <robinmholt@gmail.com>, Sedat Dilek <sedat.dilek@gmail.com>, Srikar Dronamraju <srikar@linux.vnet.ibm.com>, Thomas Gleixner <tglx@linutronix.de>, linux-kernel@vger.kernel.org, linux-mm@kvack.org, "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> Subject: [PATCH 6/9] mm, thp: do not access mm->pmd_huge_pte directly Date: Fri, 13 Sep 2013 16:06:13 +0300 [thread overview] Message-ID: <1379077576-2472-7-git-send-email-kirill.shutemov@linux.intel.com> (raw) In-Reply-To: <1379077576-2472-1-git-send-email-kirill.shutemov@linux.intel.com> Currently mm->pmd_huge_pte protected by page table lock. It will not work with split lock. We have to have per-pmd pmd_huge_pte for proper access serialization. For now, let's just introduce wrapper to access mm->pmd_huge_pte. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> --- arch/s390/mm/pgtable.c | 12 ++++++------ arch/sparc/mm/tlb.c | 12 ++++++------ include/linux/mm.h | 1 + mm/pgtable-generic.c | 12 ++++++------ 4 files changed, 19 insertions(+), 18 deletions(-) diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index de8cbc3..c463e5c 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -1225,11 +1225,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(lh); else - list_add(lh, (struct list_head *) mm->pmd_huge_pte); - mm->pmd_huge_pte = pgtable; + list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp)); + pmd_huge_pte(mm, pmdp) = pgtable; } pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) @@ -1241,12 +1241,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); lh = (struct list_head *) pgtable; if (list_empty(lh)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = (pgtable_t) lh->next; + pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next; list_del(lh); } ptep = (pte_t *) pgtable; diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c index 7a91f28..656cc46 100644 --- a/arch/sparc/mm/tlb.c +++ b/arch/sparc/mm/tlb.c @@ -196,11 +196,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(lh); else - list_add(lh, (struct list_head *) mm->pmd_huge_pte); - mm->pmd_huge_pte = pgtable; + list_add(lh, (struct list_head *) pmd_huge_pte(mm, pmdp)); + pmd_huge_pte(mm, pmdp) = pgtable; } pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) @@ -211,12 +211,12 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); lh = (struct list_head *) pgtable; if (list_empty(lh)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = (pgtable_t) lh->next; + pmd_huge_pte(mm, pmdp) = (pgtable_t) lh->next; list_del(lh); } pte_val(pgtable[0]) = 0; diff --git a/include/linux/mm.h b/include/linux/mm.h index d4361e7..d2f8a50 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1299,6 +1299,7 @@ static inline spinlock_t *huge_pmd_lockptr(struct mm_struct *mm, pmd_t *pmd) return &mm->page_table_lock; } +#define pmd_huge_pte(mm, pmd) ((mm)->pmd_huge_pte) static inline spinlock_t *huge_pmd_lock(struct mm_struct *mm, pmd_t *pmd) { diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 3929a40..41fee3e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -154,11 +154,11 @@ void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, assert_spin_locked(&mm->page_table_lock); /* FIFO */ - if (!mm->pmd_huge_pte) + if (!pmd_huge_pte(mm, pmdp)) INIT_LIST_HEAD(&pgtable->lru); else - list_add(&pgtable->lru, &mm->pmd_huge_pte->lru); - mm->pmd_huge_pte = pgtable; + list_add(&pgtable->lru, &pmd_huge_pte(mm, pmdp)->lru); + pmd_huge_pte(mm, pmdp) = pgtable; } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif @@ -173,11 +173,11 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) assert_spin_locked(&mm->page_table_lock); /* FIFO */ - pgtable = mm->pmd_huge_pte; + pgtable = pmd_huge_pte(mm, pmdp); if (list_empty(&pgtable->lru)) - mm->pmd_huge_pte = NULL; + pmd_huge_pte(mm, pmdp) = NULL; else { - mm->pmd_huge_pte = list_entry(pgtable->lru.next, + pmd_huge_pte(mm, pmdp) = list_entry(pgtable->lru.next, struct page, lru); list_del(&pgtable->lru); } -- 1.8.4.rc3 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-09-13 13:06 UTC|newest] Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top 2013-08-02 19:46 [PATCH] Add per-process flag to control thp Alex Thorlton 2013-08-02 19:53 ` Dave Jones 2013-08-02 20:00 ` Alex Thorlton 2013-08-02 20:13 ` Kirill A. Shutemov 2013-08-02 20:34 ` Alex Thorlton 2013-08-02 23:59 ` Kirill A. Shutemov 2013-08-03 19:35 ` Kees Cook 2013-08-04 14:19 ` Oleg Nesterov 2013-08-05 2:36 ` Andi Kleen 2013-08-05 15:07 ` Alex Thorlton 2013-08-16 14:33 ` [PATCH 0/8] " Alex Thorlton 2013-08-16 14:33 ` [PATCH 1/8] THP: Use real address for NUMA policy Alex Thorlton 2013-08-16 17:53 ` Dave Hansen 2013-08-16 18:17 ` Alex Thorlton 2013-08-16 18:52 ` Kirill A. Shutemov 2013-08-27 16:50 ` Alex Thorlton 2013-08-27 17:01 ` Robin Holt 2013-09-04 15:43 ` Alex Thorlton 2013-09-04 17:15 ` Alex Thorlton 2013-09-05 11:15 ` Ingo Molnar 2013-09-09 16:48 ` Alex Thorlton 2013-09-10 7:47 ` [benchmark] THP performance testcase Ingo Molnar 2013-09-13 13:06 ` [PATCH 0/9] split page table lock for PMD tables Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 1/9] mm: rename SPLIT_PTLOCKS to SPLIT_PTE_PTLOCKS Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 15:20 ` Dave Hansen 2013-09-13 15:20 ` Dave Hansen 2013-09-13 13:06 ` [PATCH 2/9] mm: convert mm->nr_ptes to atomic_t Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 3/9] mm: introduce api for split page table lock for PMD level Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:19 ` Peter Zijlstra 2013-09-13 13:19 ` Peter Zijlstra 2013-09-13 14:22 ` Kirill A. Shutemov 2013-09-13 14:22 ` Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 4/9] mm, thp: change pmd_trans_huge_lock() to return taken lock Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 5/9] mm, thp: move ptl taking inside page_check_address_pmd() Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov [this message] 2013-09-13 13:06 ` [PATCH 6/9] mm, thp: do not access mm->pmd_huge_pte directly Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 7/9] mm: convent the rest to new page table lock api Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:06 ` [PATCH 8/9] mm: implement split page table lock for PMD level Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov 2013-09-13 13:24 ` Peter Zijlstra 2013-09-13 13:24 ` Peter Zijlstra 2013-09-13 14:25 ` Kirill A. Shutemov 2013-09-13 14:25 ` Kirill A. Shutemov 2013-09-13 14:52 ` Peter Zijlstra 2013-09-13 14:52 ` Peter Zijlstra 2013-09-13 13:36 ` Peter Zijlstra 2013-09-13 13:36 ` Peter Zijlstra 2013-09-13 14:25 ` Kirill A. Shutemov 2013-09-13 14:25 ` Kirill A. Shutemov 2013-09-13 15:45 ` Naoya Horiguchi 2013-09-13 15:45 ` Naoya Horiguchi 2013-09-13 19:57 ` Dave Hansen 2013-09-13 19:57 ` Dave Hansen 2013-09-13 13:06 ` [PATCH 9/9] x86, mm: enable " Kirill A. Shutemov 2013-09-13 13:06 ` Kirill A. Shutemov [not found] ` <20130828091814.GA13681@gmail.com> 2013-08-28 9:32 ` [PATCH 1/8] THP: Use real address for NUMA policy Peter Zijlstra 2013-08-16 19:46 ` Peter Zijlstra 2013-08-16 19:49 ` Alex Thorlton 2013-08-16 14:33 ` [PATCH 2/8] mm: make clear_huge_page tolerate non aligned address Alex Thorlton 2013-08-16 14:33 ` [PATCH 3/8] THP: Pass real, not rounded, address to clear_huge_page Alex Thorlton 2013-08-16 14:34 ` [PATCH 4/8] x86: Add clear_page_nocache Alex Thorlton 2013-08-16 14:34 ` [PATCH 5/8] mm: make clear_huge_page cache clear only around the fault address Alex Thorlton 2013-08-16 18:02 ` Dave Hansen 2013-08-16 14:34 ` [PATCH 6/8] x86: switch the 64bit uncached page clear to SSE/AVX v2 Alex Thorlton 2013-08-16 14:34 ` [PATCH 7/8] remove KM_USER0 from kmap_atomic call Alex Thorlton 2013-08-16 14:34 ` [PATCH 8/8] fix up references to kernel_fpu_begin/end Alex Thorlton 2013-08-16 14:47 ` [PATCH 0/8] Re: [PATCH] Add per-process flag to control thp Peter Zijlstra 2013-08-16 15:04 ` Alex Thorlton 2013-08-04 14:44 ` Rasmus Villemoes 2013-08-28 13:56 ` Andrea Arcangeli
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=1379077576-2472-7-git-send-email-kirill.shutemov@linux.intel.com \ --to=kirill.shutemov@linux.intel.com \ --cc=aarcange@redhat.com \ --cc=ak@linux.intel.com \ --cc=akpm@linux-foundation.org \ --cc=athorlton@sgi.com \ --cc=dave.hansen@intel.com \ --cc=davej@redhat.com \ --cc=dhowells@redhat.com \ --cc=ebiederm@xmission.com \ --cc=fweisbec@gmail.com \ --cc=hannes@cmpxchg.org \ --cc=keescook@chromium.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mgorman@suse.de \ --cc=mingo@redhat.com \ --cc=mtk.manpages@gmail.com \ --cc=n-horiguchi@ah.jp.nec.com \ --cc=oleg@redhat.com \ --cc=paulmck@linux.vnet.ibm.com \ --cc=peterz@infradead.org \ --cc=riel@redhat.com \ --cc=robinmholt@gmail.com \ --cc=sedat.dilek@gmail.com \ --cc=srikar@linux.vnet.ibm.com \ --cc=tglx@linutronix.de \ --cc=viro@zeniv.linux.org.uk \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.