* [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work
@ 2007-08-07 7:19 Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 1/12] remove frv usage of flush_tlb_pgtables() Benjamin Herrenschmidt
` (11 more replies)
0 siblings, 12 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This is a snapshot of my current work on PTE accessors and
mmu_gather. It's not complete but it should show the direction
I'm heading toward.
The main goals are:
- Make mmu_gather used for all page table walk operations that
also need to invalidate TLB entries, thus obsoleting flush_tlb_range()
and flush_tlb_mm() as generic APIs to the TLB flushing.
- Make mmu_gather stack based. The cleans quite a bit of stuff up,
and once fully done, should allow to reduce latencies caused by the
need to use get_cpu() for a long time.
- Make mmu_gather more flexible so that archs who need to do more than
what the standard implementation does don't have to copy all of it and
do their own implementation.
- Make mmu_gather suitable for batching on powerpc :-) This involves
mostly adding a hook before PTE pages are unlocked and some work on
the interaction between PTE accessors and tlb_remove_tlb_entry()
- Remove other remaings of flush_tlb_*, keeping only for now
flush_tlb_kernel_range() which will be harder to "fix" (provided we
want to do it at all)
- Go through all remaining page table accessors, remove all the unused
ones (there's still a few), there should be only a handful left and redo
the documentation accordingly.
These goals are _NOT_ yet met by this patch serie and some of the bits
in there may want to be done a bit differently. As I did the patches,
and hacked various archs, I got a better visibility on what is done and
why and thus some of my initial directions end up not looking so good.
Most notably, the MMF_DEAD flag doesn't sound like such a good idea
anymore and I'm considering instead replacing ptep_get_and_clear()
and tlb_remove_tlb_entry() with a version that takes the batch as an
argument.
Also, I haven't fully moved the batch off the per-cpu, only "part 1" is
there at this stage.
However, I'll be travelling for a while and won't have much time to work
on it until I'm back mid september, so I decided now was a good time to
post what I have for comments and discussions on the approach taken.
Cheers,
Ben.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 1/12] remove frv usage of flush_tlb_pgtables()
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 2/12] remove flush_tlb_pgtables Benjamin Herrenschmidt
` (10 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
frv is the last user in the tree of that dubious hook, and it's my
understanding that it's not even needed. It's only called by memory.c
free_pgd_range() which is always called within an mmu_gather, and
tlb_flush() on frv will do a flush_tlb_mm(), which from my reading
of the code, seems to do what flush_tlb_ptables() does, which is
to clear the cached PGE.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
include/asm-frv/tlbflush.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
Index: linux-work/include/asm-frv/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-frv/tlbflush.h 2007-07-27 09:40:38.000000000 +1000
+++ linux-work/include/asm-frv/tlbflush.h 2007-07-27 09:43:17.000000000 +1000
@@ -57,8 +57,7 @@ do { \
#define __flush_tlb_global() flush_tlb_all()
#define flush_tlb() flush_tlb_all()
#define flush_tlb_kernel_range(start, end) flush_tlb_all()
-#define flush_tlb_pgtables(mm,start,end) \
- asm volatile("movgs %0,scr0 ! movgs %0,scr1" :: "r"(ULONG_MAX) : "memory");
+#define flush_tlb_pgtables(mm,start,end) do { } while(0)
#else
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 3/12] Add MMF_DEAD flag to mm_struct being torn down
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 1/12] remove frv usage of flush_tlb_pgtables() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 2/12] remove flush_tlb_pgtables Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 4/12] disconnect sparc64 tlb flush from mmu_gather Benjamin Herrenschmidt
` (8 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This adds a flag to mm_struct that is set when the mm is being
torn down. It's needed by my next patch disconnecting sparc64
tlb flushing from the mmu_gather in order to let the later
become a stack based structure. In general, that flag provides
the information that is in tlb->fullmm but more easily accessible
to all page table accessors.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
include/linux/sched.h | 2 ++
kernel/fork.c | 1 +
mm/mmap.c | 3 +++
3 files changed, 6 insertions(+)
Index: linux-work/include/linux/sched.h
===================================================================
--- linux-work.orig/include/linux/sched.h 2007-08-02 11:25:34.000000000 +1000
+++ linux-work/include/linux/sched.h 2007-08-02 11:39:44.000000000 +1000
@@ -366,6 +366,8 @@ extern int get_dumpable(struct mm_struct
#define MMF_DUMP_FILTER_DEFAULT \
((1 << MMF_DUMP_ANON_PRIVATE) | (1 << MMF_DUMP_ANON_SHARED))
+#define MMF_DEAD 6 /* mm is being destroyed */
+
struct mm_struct {
struct vm_area_struct * mmap; /* list of VMAs */
struct rb_root mm_rb;
Index: linux-work/mm/mmap.c
===================================================================
--- linux-work.orig/mm/mmap.c 2007-08-02 11:25:35.000000000 +1000
+++ linux-work/mm/mmap.c 2007-08-02 11:39:44.000000000 +1000
@@ -2025,6 +2025,9 @@ void exit_mmap(struct mm_struct *mm)
unsigned long nr_accounted = 0;
unsigned long end;
+ /* Mark the MM as dead */
+ __set_bit(MMF_DEAD, &mm->flags);
+
/* mm's last user has gone, and its about to be pulled down */
arch_exit_mmap(mm);
Index: linux-work/kernel/fork.c
===================================================================
--- linux-work.orig/kernel/fork.c 2007-07-22 10:26:48.000000000 +1000
+++ linux-work/kernel/fork.c 2007-08-02 11:39:44.000000000 +1000
@@ -336,6 +336,7 @@ static struct mm_struct * mm_init(struct
INIT_LIST_HEAD(&mm->mmlist);
mm->flags = (current->mm) ? current->mm->flags
: MMF_DUMP_FILTER_DEFAULT;
+ __clear_bit(MMF_DEAD, &mm->flags);
mm->core_waiters = 0;
mm->nr_ptes = 0;
set_mm_counter(mm, file_rss, 0);
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 2/12] remove flush_tlb_pgtables
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 1/12] remove frv usage of flush_tlb_pgtables() Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 3/12] Add MMF_DEAD flag to mm_struct being torn down Benjamin Herrenschmidt
` (9 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
After my frv patch, nobody uses flush_tlb_pgtables anymore, this patch
removes all remaining traces of it from all archs.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
Documentation/cachetlb.txt | 29 +++--------------------------
include/asm-alpha/tlbflush.h | 11 -----------
include/asm-arm/tlbflush.h | 5 -----
include/asm-avr32/tlbflush.h | 7 -------
include/asm-blackfin/tlbflush.h | 6 ------
include/asm-cris/tlbflush.h | 7 -------
include/asm-frv/tlbflush.h | 2 --
include/asm-h8300/tlbflush.h | 6 ------
include/asm-i386/tlbflush.h | 7 -------
include/asm-ia64/tlbflush.h | 13 -------------
include/asm-m32r/tlbflush.h | 3 ---
include/asm-m68k/tlbflush.h | 10 ----------
include/asm-m68knommu/tlbflush.h | 6 ------
include/asm-mips/tlbflush.h | 7 -------
include/asm-parisc/tlbflush.h | 4 ----
include/asm-powerpc/tlbflush.h | 11 -----------
include/asm-s390/tlbflush.h | 7 -------
include/asm-sh/tlbflush.h | 6 ------
include/asm-sh64/tlbflush.h | 4 ----
include/asm-sparc/tlbflush.h | 6 ------
include/asm-sparc64/tlbflush.h | 7 -------
include/asm-um/tlbflush.h | 6 ------
include/asm-v850/tlbflush.h | 6 ------
include/asm-x86_64/tlbflush.h | 9 ---------
include/asm-xtensa/tlbflush.h | 11 -----------
mm/memory.c | 3 ---
26 files changed, 3 insertions(+), 196 deletions(-)
Index: linux-work/include/asm-alpha/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-alpha/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-alpha/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -92,17 +92,6 @@ flush_tlb_other(struct mm_struct *mm)
if (*mmc) *mmc = 0;
}
-/* Flush a specified range of user mapping page tables from TLB.
- Although Alpha uses VPTE caches, this can be a nop, as Alpha does
- not have finegrained tlb flushing, so it will flush VPTE stuff
- during next flush_tlb_range. */
-
-static inline void
-flush_tlb_pgtables(struct mm_struct *mm, unsigned long start,
- unsigned long end)
-{
-}
-
#ifndef CONFIG_SMP
/* Flush everything (kernel mapping may also have changed
due to vmalloc/vfree). */
Index: linux-work/include/asm-arm/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-arm/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-arm/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -463,11 +463,6 @@ extern void flush_tlb_kernel_range(unsig
*/
extern void update_mmu_cache(struct vm_area_struct *vma, unsigned long addr, pte_t pte);
-/*
- * ARM processors do not cache TLB tables in RAM.
- */
-#define flush_tlb_pgtables(mm,start,end) do { } while (0)
-
#endif
#endif /* CONFIG_MMU */
Index: linux-work/include/asm-avr32/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-avr32/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-avr32/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -19,7 +19,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
extern void flush_tlb(void);
extern void flush_tlb_all(void);
@@ -29,12 +28,6 @@ extern void flush_tlb_range(struct vm_ar
extern void flush_tlb_page(struct vm_area_struct *vma, unsigned long page);
extern void __flush_tlb_page(unsigned long asid, unsigned long page);
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* Nothing to do */
-}
-
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
#endif /* __ASM_AVR32_TLBFLUSH_H */
Index: linux-work/include/asm-blackfin/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-blackfin/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-blackfin/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -53,10 +53,4 @@ static inline void flush_tlb_kernel_page
BUG();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- BUG();
-}
-
#endif
Index: linux-work/include/asm-cris/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-cris/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-cris/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -38,13 +38,6 @@ static inline void flush_tlb_range(struc
flush_tlb_mm(vma->vm_mm);
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* CRIS does not keep any page table caches in TLB */
-}
-
-
static inline void flush_tlb(void)
{
flush_tlb_mm(current->mm);
Index: linux-work/include/asm-frv/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-frv/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-frv/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -57,7 +57,6 @@ do { \
#define __flush_tlb_global() flush_tlb_all()
#define flush_tlb() flush_tlb_all()
#define flush_tlb_kernel_range(start, end) flush_tlb_all()
-#define flush_tlb_pgtables(mm,start,end) do { } while(0)
#else
@@ -66,7 +65,6 @@ do { \
#define flush_tlb_mm(mm) BUG()
#define flush_tlb_page(vma,addr) BUG()
#define flush_tlb_range(mm,start,end) BUG()
-#define flush_tlb_pgtables(mm,start,end) BUG()
#define flush_tlb_kernel_range(start, end) BUG()
#endif
Index: linux-work/include/asm-h8300/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-h8300/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-h8300/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -52,10 +52,4 @@ static inline void flush_tlb_kernel_page
BUG();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- BUG();
-}
-
#endif /* _H8300_TLBFLUSH_H */
Index: linux-work/include/asm-i386/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-i386/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-i386/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -78,7 +78,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
* - flush_tlb_others(cpumask, mm, va) flushes a TLBs on other cpus
*
* ..but the i386 has somewhat limited tlb flushing capabilities,
@@ -166,10 +165,4 @@ static inline void flush_tlb_kernel_rang
flush_tlb_all();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* i386 does not keep any page table caches in TLB */
-}
-
#endif /* _I386_TLBFLUSH_H */
Index: linux-work/include/asm-ia64/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-ia64/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-ia64/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -84,19 +84,6 @@ flush_tlb_page (struct vm_area_struct *v
}
/*
- * Flush the TLB entries mapping the virtually mapped linear page
- * table corresponding to address range [START-END).
- */
-static inline void
-flush_tlb_pgtables (struct mm_struct *mm, unsigned long start, unsigned long end)
-{
- /*
- * Deprecated. The virtual page table is now flushed via the normal gather/flush
- * interface (see tlb.h).
- */
-}
-
-/*
* Flush the local TLB. Invoked from another cpu using an IPI.
*/
#ifdef CONFIG_SMP
Index: linux-work/include/asm-m32r/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-m32r/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-m32r/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -12,7 +12,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
extern void local_flush_tlb_all(void);
@@ -93,8 +92,6 @@ static __inline__ void __flush_tlb_all(v
);
}
-#define flush_tlb_pgtables(mm, start, end) do { } while (0)
-
extern void update_mmu_cache(struct vm_area_struct *, unsigned long, pte_t);
#endif /* _ASM_M32R_TLBFLUSH_H */
Index: linux-work/include/asm-m68k/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-m68k/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-m68k/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -92,11 +92,6 @@ static inline void flush_tlb_kernel_rang
flush_tlb_all();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
-
#else
@@ -219,11 +214,6 @@ static inline void flush_tlb_kernel_page
sun3_put_segmap (addr & ~(SUN3_PMEG_SIZE - 1), SUN3_INVALID_PMEG);
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
-
#endif
#endif /* _M68K_TLBFLUSH_H */
Index: linux-work/include/asm-m68knommu/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-m68knommu/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-m68knommu/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -52,10 +52,4 @@ static inline void flush_tlb_kernel_page
BUG();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- BUG();
-}
-
#endif /* _M68KNOMMU_TLBFLUSH_H */
Index: linux-work/include/asm-mips/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-mips/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-mips/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -11,7 +11,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
extern void local_flush_tlb_all(void);
extern void local_flush_tlb_mm(struct mm_struct *mm);
@@ -45,10 +44,4 @@ extern void flush_tlb_one(unsigned long
#endif /* CONFIG_SMP */
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* Nothing to do on MIPS. */
-}
-
#endif /* __ASM_TLBFLUSH_H */
Index: linux-work/include/asm-parisc/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-parisc/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-parisc/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -57,10 +57,6 @@ static inline void flush_tlb_mm(struct m
#endif
}
-extern __inline__ void flush_tlb_pgtables(struct mm_struct *mm, unsigned long start, unsigned long end)
-{
-}
-
static inline void flush_tlb_page(struct vm_area_struct *vma,
unsigned long addr)
{
Index: linux-work/include/asm-powerpc/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-powerpc/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-powerpc/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -8,7 +8,6 @@
* - flush_tlb_page_nohash(vma, vmaddr) flushes one page if SW loaded TLB
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
@@ -173,15 +172,5 @@ extern void __flush_hash_table_range(str
*/
extern void update_mmu_cache(struct vm_area_struct *, unsigned long, pte_t);
-/*
- * This is called in munmap when we have freed up some page-table
- * pages. We don't need to do anything here, there's nothing special
- * about our page-table pages. -- paulus
- */
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
-
#endif /*__KERNEL__ */
#endif /* _ASM_POWERPC_TLBFLUSH_H */
Index: linux-work/include/asm-s390/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-s390/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-s390/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -14,7 +14,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
/*
@@ -152,10 +151,4 @@ static inline void flush_tlb_range(struc
#endif
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* S/390 does not keep any page table caches in TLB */
-}
-
#endif /* _S390_TLBFLUSH_H */
Index: linux-work/include/asm-sh/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-sh/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-sh/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -9,7 +9,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
extern void local_flush_tlb_all(void);
extern void local_flush_tlb_mm(struct mm_struct *mm);
@@ -47,9 +46,4 @@ extern void flush_tlb_one(unsigned long
#endif /* CONFIG_SMP */
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* Nothing to do */
-}
#endif /* __ASM_SH_TLBFLUSH_H */
Index: linux-work/include/asm-sh64/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-sh64/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-sh64/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -20,10 +20,6 @@ extern void flush_tlb_mm(struct mm_struc
extern void flush_tlb_range(struct vm_area_struct *vma, unsigned long start,
unsigned long end);
extern void flush_tlb_page(struct vm_area_struct *vma, unsigned long page);
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
Index: linux-work/include/asm-sparc/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-sparc/tlbflush.h 2007-07-27 10:37:48.000000000 +1000
+++ linux-work/include/asm-sparc/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -13,7 +13,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
#ifdef CONFIG_SMP
@@ -42,11 +41,6 @@ BTFIXUPDEF_CALL(void, flush_tlb_mm, stru
BTFIXUPDEF_CALL(void, flush_tlb_range, struct vm_area_struct *, unsigned long, unsigned long)
BTFIXUPDEF_CALL(void, flush_tlb_page, struct vm_area_struct *, unsigned long)
-// Thanks to Anton Blanchard, our pagetables became uncached in 2.4. Wee!
-// extern void flush_tlb_pgtables(struct mm_struct *mm,
-// unsigned long start, unsigned long end);
-#define flush_tlb_pgtables(mm, start, end) do{ }while(0)
-
#define flush_tlb_all() BTFIXUP_CALL(flush_tlb_all)()
#define flush_tlb_mm(mm) BTFIXUP_CALL(flush_tlb_mm)(mm)
#define flush_tlb_range(vma,start,end) BTFIXUP_CALL(flush_tlb_range)(vma,start,end)
Index: linux-work/include/asm-sparc64/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlbflush.h 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/include/asm-sparc64/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -41,11 +41,4 @@ do { flush_tsb_kernel_range(start,end);
#endif /* ! CONFIG_SMP */
-static inline void flush_tlb_pgtables(struct mm_struct *mm, unsigned long start, unsigned long end)
-{
- /* We don't use virtual page tables for TLB miss processing
- * any more. Nowadays we use the TSB.
- */
-}
-
#endif /* _SPARC64_TLBFLUSH_H */
Index: linux-work/include/asm-um/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-um/tlbflush.h 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/include/asm-um/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -18,7 +18,6 @@
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_kernel_vm() flushes the kernel vm area
* - flush_tlb_range(vma, start, end) flushes a range of pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*/
extern void flush_tlb_all(void);
@@ -42,9 +41,4 @@ extern void flush_tlb_kernel_vm(void);
extern void flush_tlb_kernel_range(unsigned long start, unsigned long end);
extern void __flush_tlb_one(unsigned long addr);
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
-
#endif
Index: linux-work/include/asm-v850/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-v850/tlbflush.h 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/include/asm-v850/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -61,10 +61,4 @@ static inline void flush_tlb_kernel_page
BUG ();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- BUG ();
-}
-
#endif /* __V850_TLBFLUSH_H__ */
Index: linux-work/include/asm-x86_64/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-x86_64/tlbflush.h 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/include/asm-x86_64/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -31,7 +31,6 @@ static inline void __flush_tlb_all(void)
* - flush_tlb_page(vma, vmaddr) flushes one page
* - flush_tlb_range(vma, start, end) flushes a range of pages
* - flush_tlb_kernel_range(start, end) flushes a range of kernel pages
- * - flush_tlb_pgtables(mm, start, end) flushes a range of page tables
*
* x86-64 can only flush individual pages or full VMs. For a range flush
* we always do the full VM. Might be worth trying if for a small
@@ -98,12 +97,4 @@ static inline void flush_tlb_kernel_rang
flush_tlb_all();
}
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
- /* x86_64 does not keep any page table caches in a software TLB.
- The CPUs do in their hardware TLBs, but they are handled
- by the normal TLB flushing algorithms. */
-}
-
#endif /* _X8664_TLBFLUSH_H */
Index: linux-work/include/asm-xtensa/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-xtensa/tlbflush.h 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/include/asm-xtensa/tlbflush.h 2007-07-27 10:38:23.000000000 +1000
@@ -41,17 +41,6 @@ extern void flush_tlb_range(struct vm_ar
#define flush_tlb_kernel_range(start,end) flush_tlb_all()
-
-/* This is calld in munmap when we have freed up some page-table pages.
- * We don't need to do anything here, there's nothing special about our
- * page-table pages.
- */
-
-static inline void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-{
-}
-
/* TLB operations. */
static inline unsigned long itlb_probe(unsigned long addr)
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-07-27 10:37:49.000000000 +1000
+++ linux-work/mm/memory.c 2007-07-27 10:38:23.000000000 +1000
@@ -259,9 +259,6 @@ void free_pgd_range(struct mmu_gather **
continue;
free_pud_range(*tlb, pgd, addr, next, floor, ceiling);
} while (pgd++, addr = next, addr != end);
-
- if (!(*tlb)->fullmm)
- flush_tlb_pgtables((*tlb)->mm, start, end);
}
void free_pgtables(struct mmu_gather **tlb, struct vm_area_struct *vma,
Index: linux-work/Documentation/cachetlb.txt
===================================================================
--- linux-work.orig/Documentation/cachetlb.txt 2007-07-27 10:38:44.000000000 +1000
+++ linux-work/Documentation/cachetlb.txt 2007-07-27 10:39:00.000000000 +1000
@@ -87,30 +87,7 @@ changes occur:
This is used primarily during fault processing.
-5) void flush_tlb_pgtables(struct mm_struct *mm,
- unsigned long start, unsigned long end)
-
- The software page tables for address space 'mm' for virtual
- addresses in the range 'start' to 'end-1' are being torn down.
-
- Some platforms cache the lowest level of the software page tables
- in a linear virtually mapped array, to make TLB miss processing
- more efficient. On such platforms, since the TLB is caching the
- software page table structure, it needs to be flushed when parts
- of the software page table tree are unlinked/freed.
-
- Sparc64 is one example of a platform which does this.
-
- Usually, when munmap()'ing an area of user virtual address
- space, the kernel leaves the page table parts around and just
- marks the individual pte's as invalid. However, if very large
- portions of the address space are unmapped, the kernel frees up
- those portions of the software page tables to prevent potential
- excessive kernel memory usage caused by erratic mmap/mmunmap
- sequences. It is at these times that flush_tlb_pgtables will
- be invoked.
-
-6) void update_mmu_cache(struct vm_area_struct *vma,
+5) void update_mmu_cache(struct vm_area_struct *vma,
unsigned long address, pte_t pte)
At the end of every page fault, this routine is invoked to
@@ -123,7 +100,7 @@ changes occur:
translations for software managed TLB configurations.
The sparc64 port currently does this.
-7) void tlb_migrate_finish(struct mm_struct *mm)
+6) void tlb_migrate_finish(struct mm_struct *mm)
This interface is called at the end of an explicit
process migration. This interface provides a hook
@@ -133,7 +110,7 @@ changes occur:
The ia64 sn2 platform is one example of a platform
that uses this interface.
-8) void lazy_mmu_prot_update(pte_t pte)
+7) void lazy_mmu_prot_update(pte_t pte)
This interface is called whenever the protection on
any user PTEs change. This interface provides a notification
to architecture specific code to take appropriate action.
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 4/12] disconnect sparc64 tlb flush from mmu_gather
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (2 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 3/12] Add MMF_DEAD flag to mm_struct being torn down Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 5/12] Add mm argument to pte/pmd/pud/pgd_free Benjamin Herrenschmidt
` (7 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
sparc64 current tlb flush needs it's per-cpu list of vaddr's. This
patch disconnect that from the mmu_gather, allowing sparc64 to use
the generic implementation of the later for now.
This is somewhat temporary. It allows to keep sparc64 working while
I'm doing surgery on mmu_gather. Once the later goes fully stack
based and provide a non-percpu way of storing the page list along
with arch specific informations, I'll be able to reconnect sparc64
to it (and get rid of the get_cpu/put_cpu and associated preempt
disabling).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/sparc64/kernel/smp.c | 2
arch/sparc64/mm/tlb.c | 43 ++++++++++--------
arch/sparc64/mm/tsb.c | 6 +-
arch/sparc64/mm/ultra.S | 12 ++---
include/asm-sparc64/system.h | 2
include/asm-sparc64/tlb.h | 95 +++++++++--------------------------------
include/asm-sparc64/tlbflush.h | 4 -
7 files changed, 59 insertions(+), 105 deletions(-)
Index: linux-work/arch/sparc64/mm/tlb.c
===================================================================
--- linux-work.orig/arch/sparc64/mm/tlb.c 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/arch/sparc64/mm/tlb.c 2007-07-27 18:06:44.000000000 +1000
@@ -19,35 +19,38 @@
/* Heavily inspired by the ppc64 code. */
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers) = { 0, };
+DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct tlb_batch, tlb_batch);
-void flush_tlb_pending(void)
+void __flush_tlb_pending(struct tlb_batch *mp)
{
- struct mmu_gather *mp = &__get_cpu_var(mmu_gathers);
-
- preempt_disable();
-
- if (mp->tlb_nr) {
+ if (mp->nr) {
flush_tsb_user(mp);
if (CTX_VALID(mp->mm->context)) {
#ifdef CONFIG_SMP
- smp_flush_tlb_pending(mp->mm, mp->tlb_nr,
+ smp_flush_tlb_pending(mp->mm, mp->nr,
&mp->vaddrs[0]);
#else
- __flush_tlb_pending(CTX_HWBITS(mp->mm->context),
- mp->tlb_nr, &mp->vaddrs[0]);
+ local_flush_tlb_pending(CTX_HWBITS(mp->mm->context),
+ mp->nr, &mp->vaddrs[0]);
#endif
}
- mp->tlb_nr = 0;
+ mp->nr = 0;
}
+}
+
+void flush_tlb_pending(void)
+{
+ struct tlb_batch *mp = &get_cpu_var(tlb_batch);
- preempt_enable();
+ __flush_tlb_pending(mp);
+ put_cpu_var(tlb_batch);
}
void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr, pte_t *ptep, pte_t orig)
{
- struct mmu_gather *mp = &__get_cpu_var(mmu_gathers);
+ struct tlb_batch *mp;
unsigned long nr;
vaddr &= PAGE_MASK;
@@ -79,13 +82,17 @@ void tlb_batch_add(struct mm_struct *mm,
no_cache_flush:
- if (mp->fullmm)
+ if (test_bit(MMF_DEAD, &mm->flags))
return;
- nr = mp->tlb_nr;
+ /* we are called with enough spinlocks held to make
+ * preempt disable useless here
+ */
+ mp = &__get_cpu_var(tlb_batch);
+ nr = mp->nr;
if (unlikely(nr != 0 && mm != mp->mm)) {
- flush_tlb_pending();
+ __flush_tlb_pending(mp);
nr = 0;
}
@@ -93,7 +100,7 @@ no_cache_flush:
mp->mm = mm;
mp->vaddrs[nr] = vaddr;
- mp->tlb_nr = ++nr;
+ mp->nr = ++nr;
if (nr >= TLB_BATCH_NR)
- flush_tlb_pending();
+ __flush_tlb_pending(mp);
}
Index: linux-work/arch/sparc64/mm/tsb.c
===================================================================
--- linux-work.orig/arch/sparc64/mm/tsb.c 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/arch/sparc64/mm/tsb.c 2007-07-27 18:06:44.000000000 +1000
@@ -47,11 +47,11 @@ void flush_tsb_kernel_range(unsigned lon
}
}
-static void __flush_tsb_one(struct mmu_gather *mp, unsigned long hash_shift, unsigned long tsb, unsigned long nentries)
+static void __flush_tsb_one(struct tlb_batch *mp, unsigned long hash_shift, unsigned long tsb, unsigned long nentries)
{
unsigned long i;
- for (i = 0; i < mp->tlb_nr; i++) {
+ for (i = 0; i < mp->nr; i++) {
unsigned long v = mp->vaddrs[i];
unsigned long tag, ent, hash;
@@ -65,7 +65,7 @@ static void __flush_tsb_one(struct mmu_g
}
}
-void flush_tsb_user(struct mmu_gather *mp)
+void flush_tsb_user(struct tlb_batch *mp)
{
struct mm_struct *mm = mp->mm;
unsigned long nentries, base, flags;
Index: linux-work/arch/sparc64/mm/ultra.S
===================================================================
--- linux-work.orig/arch/sparc64/mm/ultra.S 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/arch/sparc64/mm/ultra.S 2007-07-27 18:06:44.000000000 +1000
@@ -52,8 +52,8 @@ __flush_tlb_mm: /* 18 insns */
nop
.align 32
- .globl __flush_tlb_pending
-__flush_tlb_pending: /* 26 insns */
+ .globl local_flush_tlb_pending
+local_flush_tlb_pending: /* 26 insns */
/* %o0 = context, %o1 = nr, %o2 = vaddrs[] */
rdpr %pstate, %g7
sllx %o1, 3, %o1
@@ -346,8 +346,8 @@ cheetah_patch_cachetlbops:
call tlb_patch_one
mov 19, %o2
- sethi %hi(__flush_tlb_pending), %o0
- or %o0, %lo(__flush_tlb_pending), %o0
+ sethi %hi(local_flush_tlb_pending), %o0
+ or %o0, %lo(local_flush_tlb_pending), %o0
sethi %hi(__cheetah_flush_tlb_pending), %o1
or %o1, %lo(__cheetah_flush_tlb_pending), %o1
call tlb_patch_one
@@ -699,8 +699,8 @@ hypervisor_patch_cachetlbops:
call tlb_patch_one
mov 10, %o2
- sethi %hi(__flush_tlb_pending), %o0
- or %o0, %lo(__flush_tlb_pending), %o0
+ sethi %hi(local_flush_tlb_pending), %o0
+ or %o0, %lo(local_flush_tlb_pending), %o0
sethi %hi(__hypervisor_flush_tlb_pending), %o1
or %o1, %lo(__hypervisor_flush_tlb_pending), %o1
call tlb_patch_one
Index: linux-work/include/asm-sparc64/tlb.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlb.h 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/include/asm-sparc64/tlb.h 2007-07-27 18:06:44.000000000 +1000
@@ -7,62 +7,38 @@
#include <asm/tlbflush.h>
#include <asm/mmu_context.h>
-#define TLB_BATCH_NR 192
+#define tlb_flush(mp) \
+do { \
+ if (!test_bit(MMF_DEAD, &mp->mm->flags)) \
+ flush_tlb_pending(); \
+} while(0)
-/*
- * For UP we don't need to worry about TLB flush
- * and page free order so much..
- */
-#ifdef CONFIG_SMP
- #define FREE_PTE_NR 506
- #define tlb_fast_mode(bp) ((bp)->pages_nr == ~0U)
-#else
- #define FREE_PTE_NR 1
- #define tlb_fast_mode(bp) 1
-#endif
+#include <asm-generic/tlb.h>
+
+#define TLB_BATCH_NR 192
-struct mmu_gather {
+struct tlb_batch {
struct mm_struct *mm;
- unsigned int pages_nr;
- unsigned int need_flush;
- unsigned int fullmm;
- unsigned int tlb_nr;
unsigned long vaddrs[TLB_BATCH_NR];
- struct page *pages[FREE_PTE_NR];
+ unsigned int nr;
};
-
-DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
+DECLARE_PER_CPU(struct tlb_batch, tlb_batch);
#ifdef CONFIG_SMP
extern void smp_flush_tlb_pending(struct mm_struct *,
unsigned long, unsigned long *);
#endif
-extern void __flush_tlb_pending(unsigned long, unsigned long, unsigned long *);
+extern void local_flush_tlb_pending(unsigned long, unsigned long, unsigned long *);
+extern void __flush_tlb_pending(struct tlb_batch *mp);
extern void flush_tlb_pending(void);
-static inline struct mmu_gather *tlb_gather_mmu(struct mm_struct *mm, unsigned int full_mm_flush)
-{
- struct mmu_gather *mp = &get_cpu_var(mmu_gathers);
-
- BUG_ON(mp->tlb_nr);
-
- mp->mm = mm;
- mp->pages_nr = num_online_cpus() > 1 ? 0U : ~0U;
- mp->fullmm = full_mm_flush;
-
- return mp;
-}
-
-
-static inline void tlb_flush_mmu(struct mmu_gather *mp)
+/* for use by _switch() */
+static inline void check_flush_tlb_pending(void)
{
- if (mp->need_flush) {
- free_pages_and_swap_cache(mp->pages, mp->pages_nr);
- mp->pages_nr = 0;
- mp->need_flush = 0;
- }
-
+ struct tlb_batch *mp = &__get_cpu_var(tlb_batch);
+ if (mp->nr)
+ __flush_tlb_pending(mp);
}
#ifdef CONFIG_SMP
@@ -72,39 +48,10 @@ extern void smp_flush_tlb_mm(struct mm_s
#define do_flush_tlb_mm(mm) __flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT)
#endif
-static inline void tlb_finish_mmu(struct mmu_gather *mp, unsigned long start, unsigned long end)
-{
- tlb_flush_mmu(mp);
-
- if (mp->fullmm)
- mp->fullmm = 0;
- else
- flush_tlb_pending();
-
- /* keep the page table cache within bounds */
- check_pgt_cache();
-
- put_cpu_var(mmu_gathers);
-}
-
-static inline void tlb_remove_page(struct mmu_gather *mp, struct page *page)
-{
- if (tlb_fast_mode(mp)) {
- free_page_and_swap_cache(page);
- return;
- }
- mp->need_flush = 1;
- mp->pages[mp->pages_nr++] = page;
- if (mp->pages_nr >= FREE_PTE_NR)
- tlb_flush_mmu(mp);
-}
-
-#define tlb_remove_tlb_entry(mp,ptep,addr) do { } while (0)
-#define pte_free_tlb(mp,ptepage) pte_free(ptepage)
-#define pmd_free_tlb(mp,pmdp) pmd_free(pmdp)
-#define pud_free_tlb(tlb,pudp) __pud_free_tlb(tlb,pudp)
+#define __tlb_remove_tlb_entry(mp,ptep,addr) do { } while (0)
+#define __pte_free_tlb(mp,ptepage) pte_free(ptepage)
+#define __pmd_free_tlb(mp,pmdp) pmd_free(pmdp)
-#define tlb_migrate_finish(mm) do { } while (0)
#define tlb_start_vma(tlb, vma) do { } while (0)
#define tlb_end_vma(tlb, vma) do { } while (0)
Index: linux-work/include/asm-sparc64/tlbflush.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlbflush.h 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/include/asm-sparc64/tlbflush.h 2007-07-27 18:06:44.000000000 +1000
@@ -5,9 +5,9 @@
#include <asm/mmu_context.h>
/* TSB flush operations. */
-struct mmu_gather;
+struct tlb_batch;
extern void flush_tsb_kernel_range(unsigned long start, unsigned long end);
-extern void flush_tsb_user(struct mmu_gather *mp);
+extern void flush_tsb_user(struct tlb_batch *mp);
/* TLB flush operations. */
Index: linux-work/arch/sparc64/kernel/smp.c
===================================================================
--- linux-work.orig/arch/sparc64/kernel/smp.c 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/arch/sparc64/kernel/smp.c 2007-07-27 18:06:44.000000000 +1000
@@ -1142,7 +1142,7 @@ void smp_flush_tlb_pending(struct mm_str
ctx, nr, (unsigned long) vaddrs,
mm->cpu_vm_mask);
- __flush_tlb_pending(ctx, nr, vaddrs);
+ local_flush_tlb_pending(ctx, nr, vaddrs);
put_cpu();
}
Index: linux-work/include/asm-sparc64/system.h
===================================================================
--- linux-work.orig/include/asm-sparc64/system.h 2007-07-27 18:06:08.000000000 +1000
+++ linux-work/include/asm-sparc64/system.h 2007-07-27 18:06:44.000000000 +1000
@@ -151,7 +151,7 @@ do { if (test_thread_flag(TIF_PERFCTR))
current_thread_info()->kernel_cntd0 += (unsigned int)(__tmp);\
current_thread_info()->kernel_cntd1 += ((__tmp) >> 32); \
} \
- flush_tlb_pending(); \
+ check_flush_tlb_pending(); \
save_and_clear_fpu(); \
/* If you are tempted to conditionalize the following */ \
/* so that ASI is only written if it changes, think again. */ \
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 5/12] Add mm argument to pte/pmd/pud/pgd_free.
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (3 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 4/12] disconnect sparc64 tlb flush from mmu_gather Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 6/12] Add "address" argument to pte/pmd/pud_free_tlb() Benjamin Herrenschmidt
` (6 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
Based on a patch from: Martin Schwidefsky <schwidefsky@de.ibm.com>
pte_alloc_one/pte_alloc_one_kernel get the mm as argument. If the page
table allocation depends on the mm pte/pmd/pud/pgd_free should get
the mm argument as well. That way it will be possible to have per-mm
lists of page table pages. I haven't changed pte_free_kernel at this
stage though.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/arm/kernel/smp.c | 2 +-
arch/arm/mm/pgd.c | 6 +++---
arch/frv/mm/pgalloc.c | 2 +-
arch/i386/mm/pgtable.c | 2 +-
arch/powerpc/mm/pgtable_32.c | 4 ++--
arch/ppc/mm/pgtable.c | 4 ++--
arch/um/kernel/mem.c | 2 +-
arch/um/kernel/skas/mmu.c | 6 +++---
include/asm-alpha/pgalloc.h | 6 +++---
include/asm-alpha/tlb.h | 4 ++--
include/asm-arm/pgalloc.h | 6 +++---
include/asm-arm/tlb.h | 4 ++--
include/asm-avr32/pgalloc.h | 4 ++--
include/asm-cris/pgalloc.h | 4 ++--
include/asm-frv/pgalloc.h | 6 +++---
include/asm-generic/4level-fixup.h | 2 +-
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/asm-i386/pgalloc.h | 6 +++---
include/asm-ia64/pgalloc.h | 14 +++++++-------
include/asm-m32r/pgalloc.h | 8 ++++----
include/asm-m68k/motorola_pgalloc.h | 8 ++++----
include/asm-m68k/sun3_pgalloc.h | 6 +++---
include/asm-mips/pgalloc.h | 10 +++++-----
include/asm-parisc/pgalloc.h | 8 ++++----
include/asm-parisc/tlb.h | 4 ++--
include/asm-powerpc/pgalloc-32.h | 8 ++++----
include/asm-powerpc/pgalloc-64.h | 8 ++++----
include/asm-ppc/pgalloc.h | 8 ++++----
include/asm-s390/pgalloc.h | 10 +++++-----
include/asm-sh/pgalloc.h | 6 +++---
include/asm-sh64/pgalloc.h | 10 +++++-----
include/asm-sparc/pgalloc.h | 12 ++++++------
include/asm-sparc64/pgalloc.h | 6 +++---
include/asm-sparc64/tlb.h | 4 ++--
include/asm-um/pgalloc.h | 6 +++---
include/asm-x86_64/pgalloc.h | 8 ++++----
include/asm-xtensa/pgalloc.h | 4 ++--
include/asm-xtensa/tlb.h | 2 +-
kernel/fork.c | 2 +-
mm/memory.c | 8 ++++----
41 files changed, 117 insertions(+), 117 deletions(-)
Index: linux-work/arch/powerpc/mm/pgtable_32.c
===================================================================
--- linux-work.orig/arch/powerpc/mm/pgtable_32.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/powerpc/mm/pgtable_32.c 2007-08-06 16:15:18.000000000 +1000
@@ -86,7 +86,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
return ret;
}
-void pgd_free(pgd_t *pgd)
+void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_pages((unsigned long)pgd, PGDIR_ORDER);
}
@@ -131,7 +131,7 @@ void pte_free_kernel(pte_t *pte)
free_page((unsigned long)pte);
}
-void pte_free(struct page *ptepage)
+void pte_free(struct mm_struct *mm, struct page *ptepage)
{
#ifdef CONFIG_SMP
hash_page_sync();
Index: linux-work/arch/ppc/mm/pgtable.c
===================================================================
--- linux-work.orig/arch/ppc/mm/pgtable.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/ppc/mm/pgtable.c 2007-08-06 16:15:18.000000000 +1000
@@ -87,7 +87,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
return ret;
}
-void pgd_free(pgd_t *pgd)
+void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_pages((unsigned long)pgd, PGDIR_ORDER);
}
@@ -132,7 +132,7 @@ void pte_free_kernel(pte_t *pte)
free_page((unsigned long)pte);
}
-void pte_free(struct page *ptepage)
+void pte_free(struct mm_struct *mm, struct page *ptepage)
{
#ifdef CONFIG_SMP
hash_page_sync();
Index: linux-work/include/asm-alpha/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-alpha/pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-alpha/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -31,7 +31,7 @@ pgd_populate(struct mm_struct *mm, pgd_t
extern pgd_t *pgd_alloc(struct mm_struct *mm);
static inline void
-pgd_free(pgd_t *pgd)
+pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_page((unsigned long)pgd);
}
@@ -44,7 +44,7 @@ pmd_alloc_one(struct mm_struct *mm, unsi
}
static inline void
-pmd_free(pmd_t *pmd)
+pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
free_page((unsigned long)pmd);
}
@@ -67,7 +67,7 @@ pte_alloc_one(struct mm_struct *mm, unsi
}
static inline void
-pte_free(struct page *page)
+pte_free(struct mm_struct *mm, struct page *page)
{
__free_page(page);
}
Index: linux-work/include/asm-alpha/tlb.h
===================================================================
--- linux-work.orig/include/asm-alpha/tlb.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-alpha/tlb.h 2007-08-06 16:26:04.000000000 +1000
@@ -9,7 +9,7 @@
#include <asm-generic/tlb.h>
-#define __pte_free_tlb(tlb,pte) pte_free(pte)
-#define __pmd_free_tlb(tlb,pmd) pmd_free(pmd)
+#define __pte_free_tlb(tlb,pte) pte_free((tlb)->mm, pte)
+#define __pmd_free_tlb(tlb,pmd) pmd_free((tlb)->mm, pmd)
#endif
Index: linux-work/include/asm-arm/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-arm/pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-arm/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -27,14 +27,14 @@
* Since we have only two-level page tables, these are trivial
*/
#define pmd_alloc_one(mm,addr) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(pmd) do { } while (0)
+#define pmd_free(mm,pmd) do { } while (0)
#define pgd_populate(mm,pmd,pte) BUG()
extern pgd_t *get_pgd_slow(struct mm_struct *mm);
extern void free_pgd_slow(pgd_t *pgd);
#define pgd_alloc(mm) get_pgd_slow(mm)
-#define pgd_free(pgd) free_pgd_slow(pgd)
+#define pgd_free(mm,pgd) free_pgd_slow(pgd)
/*
* Allocate one PTE table.
@@ -91,7 +91,7 @@ static inline void pte_free_kernel(pte_t
}
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
Index: linux-work/include/asm-arm/tlb.h
===================================================================
--- linux-work.orig/include/asm-arm/tlb.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-arm/tlb.h 2007-08-06 16:26:04.000000000 +1000
@@ -85,8 +85,8 @@ tlb_end_vma(struct mmu_gather *tlb, stru
}
#define tlb_remove_page(tlb,page) free_page_and_swap_cache(page)
-#define pte_free_tlb(tlb,ptep) pte_free(ptep)
-#define pmd_free_tlb(tlb,pmdp) pmd_free(pmdp)
+#define pte_free_tlb(tlb,ptep) pte_free((tlb)->mm, ptep)
+#define pmd_free_tlb(tlb,pmdp) pmd_free((tlb)->mm, pmdp)
#define tlb_migrate_finish(mm) do { } while (0)
Index: linux-work/include/asm-avr32/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-avr32/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-avr32/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -36,7 +36,7 @@ static __inline__ pgd_t *pgd_alloc(struc
return pgd;
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
kfree(pgd);
}
@@ -84,7 +84,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
Index: linux-work/include/asm-cris/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-cris/pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-cris/pgalloc.h 2007-08-06 16:26:04.000000000 +1000
@@ -16,7 +16,7 @@ static inline pgd_t *pgd_alloc (struct m
return (pgd_t *)get_zeroed_page(GFP_KERNEL);
}
-static inline void pgd_free (pgd_t *pgd)
+static inline void pgd_free (struct mm_struct *mm, pgd_t *pgd)
{
free_page((unsigned long)pgd);
}
@@ -39,7 +39,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
Index: linux-work/include/asm-frv/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-frv/pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-frv/pgalloc.h 2007-08-06 16:26:04.000000000 +1000
@@ -31,7 +31,7 @@ do { \
*/
extern pgd_t *pgd_alloc(struct mm_struct *);
-extern void pgd_free(pgd_t *);
+extern void pgd_free(struct mm_struct *mm, pgd_t *);
extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
@@ -42,7 +42,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
@@ -55,7 +55,7 @@ static inline void pte_free(struct page
* (In the PAE case we free the pmds as part of the pgd.)
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *) 2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
#endif /* CONFIG_MMU */
Index: linux-work/include/asm-i386/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-i386/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-i386/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -33,7 +33,7 @@ do { \
* Allocate and free page tables.
*/
extern pgd_t *pgd_alloc(struct mm_struct *);
-extern void pgd_free(pgd_t *pgd);
+extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
extern struct page *pte_alloc_one(struct mm_struct *, unsigned long);
@@ -43,7 +43,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
@@ -60,7 +60,7 @@ do { \
* In the PAE case we free the pmds as part of the pgd.
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm,x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
#define pud_populate(mm, pmd, pte) BUG()
#endif
Index: linux-work/include/asm-ia64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-ia64/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-ia64/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -27,7 +27,7 @@ static inline pgd_t *pgd_alloc(struct mm
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pgd_free(pgd_t * pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t * pgd)
{
quicklist_free(0, NULL, pgd);
}
@@ -44,11 +44,11 @@ static inline pud_t *pud_alloc_one(struc
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pud_free(pud_t * pud)
+static inline void pud_free(struct mm_struct *mm, pud_t * pud)
{
quicklist_free(0, NULL, pud);
}
-#define __pud_free_tlb(tlb, pud) pud_free(pud)
+#define __pud_free_tlb(tlb, pud) pud_free((tlb)->mm, pud)
#endif /* CONFIG_PGTABLE_4 */
static inline void
@@ -62,12 +62,12 @@ static inline pmd_t *pmd_alloc_one(struc
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pmd_free(pmd_t * pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t * pmd)
{
quicklist_free(0, NULL, pmd);
}
-#define __pmd_free_tlb(tlb, pmd) pmd_free(pmd)
+#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
static inline void
pmd_populate(struct mm_struct *mm, pmd_t * pmd_entry, struct page *pte)
@@ -94,7 +94,7 @@ static inline pte_t *pte_alloc_one_kerne
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
quicklist_free_page(0, NULL, pte);
}
@@ -109,6 +109,6 @@ static inline void check_pgt_cache(void)
quicklist_trim(0, NULL, 25, 16);
}
-#define __pte_free_tlb(tlb, pte) pte_free(pte)
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
#endif /* _ASM_IA64_PGALLOC_H */
Index: linux-work/include/asm-m32r/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m32r/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-m32r/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -24,7 +24,7 @@ static __inline__ pgd_t *pgd_alloc(struc
return pgd;
}
-static __inline__ void pgd_free(pgd_t *pgd)
+static __inline__ void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_page((unsigned long)pgd);
}
@@ -51,12 +51,12 @@ static __inline__ void pte_free_kernel(p
free_page((unsigned long)pte);
}
-static __inline__ void pte_free(struct page *pte)
+static __inline__ void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
-#define __pte_free_tlb(tlb, pte) pte_free((pte))
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
/*
* allocating and freeing a pmd is trivial: the 1-entry pmd is
@@ -65,7 +65,7 @@ static __inline__ void pte_free(struct p
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm,x) do { } while (0)
#define __pmd_free_tlb(tlb, x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
Index: linux-work/include/asm-m68k/motorola_pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m68k/motorola_pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-m68k/motorola_pgalloc.h 2007-08-06 16:26:04.000000000 +1000
@@ -47,7 +47,7 @@ static inline struct page *pte_alloc_one
return page;
}
-static inline void pte_free(struct page *page)
+static inline void pte_free(struct mm_struct *mm, struct page *page)
{
cache_page(kmap(page));
kunmap(page);
@@ -67,7 +67,7 @@ static inline pmd_t *pmd_alloc_one(struc
return get_pointer_table();
}
-static inline int pmd_free(pmd_t *pmd)
+static inline int pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
return free_pointer_table(pmd);
}
@@ -78,9 +78,9 @@ static inline int __pmd_free_tlb(struct
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
- pmd_free((pmd_t *)pgd);
+ pmd_free(mm, (pmd_t *)pgd);
}
static inline pgd_t *pgd_alloc(struct mm_struct *mm)
Index: linux-work/include/asm-m68k/sun3_pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m68k/sun3_pgalloc.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-m68k/sun3_pgalloc.h 2007-08-06 16:26:04.000000000 +1000
@@ -26,7 +26,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long) pte);
}
-static inline void pte_free(struct page *page)
+static inline void pte_free(struct mm_struct *mm, struct page *page)
{
__free_page(page);
}
@@ -72,10 +72,10 @@ static inline void pmd_populate(struct m
* allocating and freeing a pmd is trivial: the 1-entry pmd is
* inside the pgd, so has no extra memory associated with it.
*/
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define __pmd_free_tlb(tlb, x) do { } while (0)
-static inline void pgd_free(pgd_t * pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t * pgd)
{
free_page((unsigned long) pgd);
}
Index: linux-work/include/asm-mips/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-mips/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-mips/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -58,7 +58,7 @@ static inline pgd_t *pgd_alloc(struct mm
return ret;
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_pages((unsigned long)pgd, PGD_ORDER);
}
@@ -90,7 +90,7 @@ static inline void pte_free_kernel(pte_t
free_pages((unsigned long)pte, PTE_ORDER);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_pages(pte, PTE_ORDER);
}
@@ -103,7 +103,7 @@ static inline void pte_free(struct page
* allocating and freeing a pmd is trivial: the 1-entry pmd is
* inside the pgd, so has no extra memory associated with it.
*/
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
#endif
@@ -120,12 +120,12 @@ static inline pmd_t *pmd_alloc_one(struc
return pmd;
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
free_pages((unsigned long)pmd, PMD_ORDER);
}
-#define __pmd_free_tlb(tlb,x) pmd_free(x)
+#define __pmd_free_tlb(tlb,x) pmd_free((tlb)->mm, x)
#endif
Index: linux-work/include/asm-parisc/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-parisc/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-parisc/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -43,7 +43,7 @@ static inline pgd_t *pgd_alloc(struct mm
return actual_pgd;
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
#ifdef CONFIG_64BIT
pgd -= PTRS_PER_PGD;
@@ -70,7 +70,7 @@ static inline pmd_t *pmd_alloc_one(struc
return pmd;
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
#ifdef CONFIG_64BIT
if(pmd_flag(*pmd) & PxD_FLAG_ATTACHED)
@@ -91,7 +91,7 @@ static inline void pmd_free(pmd_t *pmd)
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#endif
@@ -135,7 +135,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-#define pte_free(page) pte_free_kernel(page_address(page))
+#define pte_free(mm,page) pte_free_kernel(page_address(page))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-parisc/tlb.h
===================================================================
--- linux-work.orig/include/asm-parisc/tlb.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-parisc/tlb.h 2007-08-06 16:26:05.000000000 +1000
@@ -21,7 +21,7 @@ do { if (!(tlb)->fullmm) \
#include <asm-generic/tlb.h>
-#define __pmd_free_tlb(tlb, pmd) pmd_free(pmd)
-#define __pte_free_tlb(tlb, pte) pte_free(pte)
+#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
#endif
Index: linux-work/include/asm-powerpc/pgalloc-32.h
===================================================================
--- linux-work.orig/include/asm-powerpc/pgalloc-32.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-powerpc/pgalloc-32.h 2007-08-06 16:26:05.000000000 +1000
@@ -6,14 +6,14 @@
extern void __bad_pte(pmd_t *pmd);
extern pgd_t *pgd_alloc(struct mm_struct *mm);
-extern void pgd_free(pgd_t *pgd);
+extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
/*
* We don't have any real pmd's, and this code never triggers because
* the pgd will always be present..
*/
/* #define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); }) */
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
/* #define pgd_populate(mm, pmd, pte) BUG() */
@@ -32,9 +32,9 @@ extern void pgd_free(pgd_t *pgd);
extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
extern struct page *pte_alloc_one(struct mm_struct *mm, unsigned long addr);
extern void pte_free_kernel(pte_t *pte);
-extern void pte_free(struct page *pte);
+extern void pte_free(struct mm_struct *mm, struct page *pte);
-#define __pte_free_tlb(tlb, pte) pte_free((pte))
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-powerpc/pgalloc-64.h
===================================================================
--- linux-work.orig/include/asm-powerpc/pgalloc-64.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-powerpc/pgalloc-64.h 2007-08-06 16:26:05.000000000 +1000
@@ -25,7 +25,7 @@ static inline pgd_t *pgd_alloc(struct mm
return kmem_cache_alloc(pgtable_cache[PGD_CACHE_NUM], GFP_KERNEL);
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
kmem_cache_free(pgtable_cache[PGD_CACHE_NUM], pgd);
}
@@ -40,7 +40,7 @@ static inline pud_t *pud_alloc_one(struc
GFP_KERNEL|__GFP_REPEAT);
}
-static inline void pud_free(pud_t *pud)
+static inline void pud_free(struct mm_struct *mm, pud_t *pud)
{
kmem_cache_free(pgtable_cache[PUD_CACHE_NUM], pud);
}
@@ -76,7 +76,7 @@ static inline pmd_t *pmd_alloc_one(struc
GFP_KERNEL|__GFP_REPEAT);
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
kmem_cache_free(pgtable_cache[PMD_CACHE_NUM], pmd);
}
@@ -99,7 +99,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long)pte);
}
-static inline void pte_free(struct page *ptepage)
+static inline void pte_free(struct mm_struct *mm, struct page *ptepage)
{
__free_page(ptepage);
}
Index: linux-work/include/asm-ppc/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-ppc/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-ppc/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -7,14 +7,14 @@
extern void __bad_pte(pmd_t *pmd);
extern pgd_t *pgd_alloc(struct mm_struct *mm);
-extern void pgd_free(pgd_t *pgd);
+extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
/*
* We don't have any real pmd's, and this code never triggers because
* the pgd will always be present..
*/
#define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm,x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
@@ -33,9 +33,9 @@ extern void pgd_free(pgd_t *pgd);
extern pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long addr);
extern struct page *pte_alloc_one(struct mm_struct *mm, unsigned long addr);
extern void pte_free_kernel(pte_t *pte);
-extern void pte_free(struct page *pte);
+extern void pte_free(struct mm_struct *mm, struct page *pte);
-#define __pte_free_tlb(tlb, pte) pte_free((pte))
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-s390/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-s390/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-s390/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -67,7 +67,7 @@ static inline pgd_t *pgd_alloc(struct mm
return pgd;
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
pgd_t *shadow_pgd = get_shadow_pgd(pgd);
@@ -83,7 +83,7 @@ static inline void pgd_free(pgd_t *pgd)
* code never triggers because the pgd will always be present.
*/
#define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#define pgd_populate_kernel(mm, pmd, pte) BUG()
@@ -111,7 +111,7 @@ static inline pmd_t * pmd_alloc_one(stru
return pmd;
}
-static inline void pmd_free (pmd_t *pmd)
+static inline void pmd_free (struct mm_struct *mm, pmd_t *pmd)
{
pmd_t *shadow_pmd = get_shadow_pmd(pmd);
@@ -123,7 +123,7 @@ static inline void pmd_free (pmd_t *pmd)
#define __pmd_free_tlb(tlb,pmd) \
do { \
tlb_flush_mmu(tlb, 0, 0); \
- pmd_free(pmd); \
+ pmd_free((tlb)->mm, pmd); \
} while (0)
static inline void
@@ -217,7 +217,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long) pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
struct page *shadow_page = get_shadow_page(pte);
Index: linux-work/include/asm-sh/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sh/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-sh/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -36,7 +36,7 @@ static inline pgd_t *pgd_alloc(struct mm
return quicklist_alloc(QUICK_PGD, GFP_KERNEL | __GFP_REPEAT, pgd_ctor);
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
quicklist_free(QUICK_PGD, NULL, pgd);
}
@@ -59,7 +59,7 @@ static inline void pte_free_kernel(pte_t
quicklist_free(QUICK_PT, NULL, pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
quicklist_free_page(QUICK_PT, NULL, pte);
}
@@ -71,7 +71,7 @@ static inline void pte_free(struct page
* inside the pgd, so has no extra memory associated with it.
*/
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm,x) do { } while (0)
#define __pmd_free_tlb(tlb,x) do { } while (0)
static inline void check_pgt_cache(void)
Index: linux-work/include/asm-sh64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sh64/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-sh64/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -46,7 +46,7 @@ static inline pgd_t *pgd_alloc(struct mm
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
quicklist_free(0, NULL, pgd);
}
@@ -63,7 +63,7 @@ static inline void pte_free_kernel(pte_t
quicklist_free(0, NULL, pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
quicklist_free_page(0, NULL, pte);
}
@@ -84,7 +84,7 @@ static inline pte_t *pte_alloc_one_kerne
#if defined(CONFIG_SH64_PGTABLE_2_LEVEL)
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm, x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
#define __pmd_free_tlb(tlb,pmd) do { } while (0)
@@ -96,13 +96,13 @@ static inline pmd_t *pmd_alloc_one(struc
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
quicklist_free(0, NULL, pmd);
}
#define pgd_populate(mm, pgd, pmd) pgd_set(pgd, pmd)
-#define __pmd_free_tlb(tlb,pmd) pmd_free(pmd)
+#define __pmd_free_tlb(tlb,pmd) pmd_free((tlb)->mm, pmd)
#else
#error "No defined page table size"
Index: linux-work/include/asm-sparc/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sparc/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-sparc/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -32,8 +32,8 @@ BTFIXUPDEF_CALL(pgd_t *, get_pgd_fast, v
BTFIXUPDEF_CALL(void, free_pgd_fast, pgd_t *)
#define free_pgd_fast(pgd) BTFIXUP_CALL(free_pgd_fast)(pgd)
-#define pgd_free(pgd) free_pgd_fast(pgd)
-#define pgd_alloc(mm) get_pgd_fast()
+#define pgd_free(mm,pgd) free_pgd_fast(pgd)
+#define pgd_alloc(mm) get_pgd_fast()
BTFIXUPDEF_CALL(void, pgd_set, pgd_t *, pmd_t *)
#define pgd_set(pgdp,pmdp) BTFIXUP_CALL(pgd_set)(pgdp,pmdp)
@@ -45,8 +45,8 @@ BTFIXUPDEF_CALL(pmd_t *, pmd_alloc_one,
BTFIXUPDEF_CALL(void, free_pmd_fast, pmd_t *)
#define free_pmd_fast(pmd) BTFIXUP_CALL(free_pmd_fast)(pmd)
-#define pmd_free(pmd) free_pmd_fast(pmd)
-#define __pmd_free_tlb(tlb, pmd) pmd_free(pmd)
+#define pmd_free(mm,pmd) free_pmd_fast(pmd)
+#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
BTFIXUPDEF_CALL(void, pmd_populate, pmd_t *, struct page *)
#define pmd_populate(MM, PMD, PTE) BTFIXUP_CALL(pmd_populate)(PMD, PTE)
@@ -62,7 +62,7 @@ BTFIXUPDEF_CALL(void, free_pte_fast, pte
#define pte_free_kernel(pte) BTFIXUP_CALL(free_pte_fast)(pte)
BTFIXUPDEF_CALL(void, pte_free, struct page *)
-#define pte_free(pte) BTFIXUP_CALL(pte_free)(pte)
-#define __pte_free_tlb(tlb, pte) pte_free(pte)
+#define pte_free(mm,pte) BTFIXUP_CALL(pte_free)(pte)
+#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
#endif /* _SPARC_PGALLOC_H */
Index: linux-work/include/asm-sparc64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sparc64/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-sparc64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -20,7 +20,7 @@ static inline pgd_t *pgd_alloc(struct mm
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
quicklist_free(0, NULL, pgd);
}
@@ -32,7 +32,7 @@ static inline pmd_t *pmd_alloc_one(struc
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
quicklist_free(0, NULL, pmd);
}
@@ -55,7 +55,7 @@ static inline void pte_free_kernel(pte_t
quicklist_free(0, NULL, pte);
}
-static inline void pte_free(struct page *ptepage)
+static inline void pte_free(struct mm_struct *mm, struct page *ptepage)
{
quicklist_free_page(0, NULL, ptepage);
}
Index: linux-work/include/asm-sparc64/tlb.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-sparc64/tlb.h 2007-08-06 16:26:05.000000000 +1000
@@ -49,8 +49,8 @@ extern void smp_flush_tlb_mm(struct mm_s
#endif
#define __tlb_remove_tlb_entry(mp,ptep,addr) do { } while (0)
-#define __pte_free_tlb(mp,ptepage) pte_free(ptepage)
-#define __pmd_free_tlb(mp,pmdp) pmd_free(pmdp)
+#define __pte_free_tlb(mp,ptepage) pte_free((mp)->mm,ptepage)
+#define __pmd_free_tlb(mp,pmdp) pmd_free((mp)->mm,pmdp)
#define tlb_start_vma(tlb, vma) do { } while (0)
#define tlb_end_vma(tlb, vma) do { } while (0)
Index: linux-work/include/asm-um/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-um/pgalloc.h 2007-07-27 13:44:46.000000000 +1000
+++ linux-work/include/asm-um/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -23,7 +23,7 @@
* Allocate and free page tables.
*/
extern pgd_t *pgd_alloc(struct mm_struct *);
-extern void pgd_free(pgd_t *pgd);
+extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
extern pte_t *pte_alloc_one_kernel(struct mm_struct *, unsigned long);
extern struct page *pte_alloc_one(struct mm_struct *, unsigned long);
@@ -33,7 +33,7 @@ static inline void pte_free_kernel(pte_t
free_page((unsigned long) pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
__free_page(pte);
}
@@ -42,7 +42,7 @@ static inline void pte_free(struct page
#ifdef CONFIG_3_LEVEL_PGTABLES
-extern __inline__ void pmd_free(pmd_t *pmd)
+extern __inline__ void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
free_page((unsigned long)pmd);
}
Index: linux-work/include/asm-x86_64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-x86_64/pgalloc.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-x86_64/pgalloc.h 2007-08-06 16:26:05.000000000 +1000
@@ -21,7 +21,7 @@ static inline void pmd_populate(struct m
set_pmd(pmd, __pmd(_PAGE_TABLE | (page_to_pfn(pte) << PAGE_SHIFT)));
}
-static inline void pmd_free(pmd_t *pmd)
+static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd)
{
BUG_ON((unsigned long)pmd & (PAGE_SIZE-1));
quicklist_free(QUICK_PT, NULL, pmd);
@@ -37,7 +37,7 @@ static inline pud_t *pud_alloc_one(struc
return (pud_t *)quicklist_alloc(QUICK_PT, GFP_KERNEL|__GFP_REPEAT, NULL);
}
-static inline void pud_free (pud_t *pud)
+static inline void pud_free (struct mm_struct *mm, pud_t *pud)
{
BUG_ON((unsigned long)pud & (PAGE_SIZE-1));
quicklist_free(QUICK_PT, NULL, pud);
@@ -97,7 +97,7 @@ static inline pgd_t *pgd_alloc(struct mm
return pgd;
}
-static inline void pgd_free(pgd_t *pgd)
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
BUG_ON((unsigned long)pgd & (PAGE_SIZE-1));
quicklist_free(QUICK_PGD, pgd_dtor, pgd);
@@ -126,7 +126,7 @@ static inline void pte_free_kernel(pte_t
quicklist_free(QUICK_PT, NULL, pte);
}
-static inline void pte_free(struct page *pte)
+static inline void pte_free(struct mm_struct *mm, struct page *pte)
{
quicklist_free_page(QUICK_PT, NULL, pte);
}
Index: linux-work/include/asm-xtensa/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-xtensa/pgalloc.h 2007-07-27 13:44:46.000000000 +1000
+++ linux-work/include/asm-xtensa/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -63,7 +63,7 @@
* inside the pgd, so has no extra memory associated with it.
*/
-#define pgd_free(pgd) free_page((unsigned long)(pgd))
+#define pgd_free(mm,pgd) free_page((unsigned long)(pgd))
#if (DCACHE_WAY_SIZE > PAGE_SIZE) && XCHAL_DCACHE_IS_WRITEBACK
@@ -109,7 +109,7 @@ extern pte_t* pte_alloc_one_kernel(struc
extern struct page* pte_alloc_one(struct mm_struct* mm, unsigned long addr);
#define pte_free_kernel(pte) free_page((unsigned long)pte)
-#define pte_free(pte) __free_page(pte)
+#define pte_free(mm,pte) __free_page(pte)
#endif /* __KERNEL__ */
#endif /* _XTENSA_PGALLOC_H */
Index: linux-work/include/asm-xtensa/tlb.h
===================================================================
--- linux-work.orig/include/asm-xtensa/tlb.h 2007-07-27 13:44:46.000000000 +1000
+++ linux-work/include/asm-xtensa/tlb.h 2007-08-06 16:26:05.000000000 +1000
@@ -20,6 +20,6 @@
#include <asm-generic/tlb.h>
#include <asm/page.h>
-#define __pte_free_tlb(tlb,pte) pte_free(pte)
+#define __pte_free_tlb(tlb,pte) pte_free((tlb)->mm, pte)
#endif /* _XTENSA_TLB_H */
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-06 16:26:05.000000000 +1000
@@ -305,7 +305,7 @@ int __pte_alloc(struct mm_struct *mm, pm
spin_lock(&mm->page_table_lock);
if (pmd_present(*pmd)) { /* Another has populated it */
pte_lock_deinit(new);
- pte_free(new);
+ pte_free(mm, new);
} else {
mm->nr_ptes++;
inc_zone_page_state(new, NR_PAGETABLE);
@@ -2669,7 +2669,7 @@ int __pud_alloc(struct mm_struct *mm, pg
spin_lock(&mm->page_table_lock);
if (pgd_present(*pgd)) /* Another has populated it */
- pud_free(new);
+ pud_free(mm, new);
else
pgd_populate(mm, pgd, new);
spin_unlock(&mm->page_table_lock);
@@ -2691,12 +2691,12 @@ int __pmd_alloc(struct mm_struct *mm, pu
spin_lock(&mm->page_table_lock);
#ifndef __ARCH_HAS_4LEVEL_HACK
if (pud_present(*pud)) /* Another has populated it */
- pmd_free(new);
+ pmd_free(mm, new);
else
pud_populate(mm, pud, new);
#else
if (pgd_present(*pud)) /* Another has populated it */
- pmd_free(new);
+ pmd_free(mm, new);
else
pgd_populate(mm, pud, new);
#endif /* __ARCH_HAS_4LEVEL_HACK */
Index: linux-work/include/asm-generic/4level-fixup.h
===================================================================
--- linux-work.orig/include/asm-generic/4level-fixup.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-generic/4level-fixup.h 2007-08-06 16:26:04.000000000 +1000
@@ -28,7 +28,7 @@
#undef pud_free_tlb
#define pud_free_tlb(tlb, x) do { } while (0)
-#define pud_free(x) do { } while (0)
+#define pud_free(mm,x) do { } while (0)
#define __pud_free_tlb(tlb, x) do { } while (0)
#undef pud_addr_end
Index: linux-work/include/asm-generic/pgtable-nopmd.h
===================================================================
--- linux-work.orig/include/asm-generic/pgtable-nopmd.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-generic/pgtable-nopmd.h 2007-08-06 16:26:04.000000000 +1000
@@ -54,7 +54,7 @@ static inline pmd_t * pmd_offset(pud_t *
* inside the pud, so has no extra memory associated with it.
*/
#define pmd_alloc_one(mm, address) NULL
-#define pmd_free(x) do { } while (0)
+#define pmd_free(mm,x) do { } while (0)
#define __pmd_free_tlb(tlb, x) do { } while (0)
#undef pmd_addr_end
Index: linux-work/include/asm-generic/pgtable-nopud.h
===================================================================
--- linux-work.orig/include/asm-generic/pgtable-nopud.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-generic/pgtable-nopud.h 2007-08-06 16:26:04.000000000 +1000
@@ -51,7 +51,7 @@ static inline pud_t * pud_offset(pgd_t *
* inside the pgd, so has no extra memory associated with it.
*/
#define pud_alloc_one(mm, address) NULL
-#define pud_free(x) do { } while (0)
+#define pud_free(mm,x) do { } while (0)
#define __pud_free_tlb(tlb, x) do { } while (0)
#undef pud_addr_end
Index: linux-work/arch/arm/kernel/smp.c
===================================================================
--- linux-work.orig/arch/arm/kernel/smp.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/arm/kernel/smp.c 2007-08-06 16:15:18.000000000 +1000
@@ -150,7 +150,7 @@ int __cpuinit __cpu_up(unsigned int cpu)
secondary_data.pgdir = 0;
*pmd_offset(pgd, PHYS_OFFSET) = __pmd(0);
- pgd_free(pgd);
+ pgd_free(&init_mm, pgd);
if (ret) {
printk(KERN_CRIT "CPU%u: processor failed to boot\n", cpu);
Index: linux-work/arch/arm/mm/pgd.c
===================================================================
--- linux-work.orig/arch/arm/mm/pgd.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/arm/mm/pgd.c 2007-08-06 16:15:18.000000000 +1000
@@ -65,7 +65,7 @@ pgd_t *get_pgd_slow(struct mm_struct *mm
return new_pgd;
no_pte:
- pmd_free(new_pmd);
+ pmd_free(mm,new_pmd);
no_pmd:
free_pages((unsigned long)new_pgd, 2);
no_pgd:
@@ -94,8 +94,8 @@ void free_pgd_slow(pgd_t *pgd)
pmd_clear(pmd);
dec_zone_page_state(virt_to_page((unsigned long *)pgd), NR_PAGETABLE);
pte_lock_deinit(pte);
- pte_free(pte);
- pmd_free(pmd);
+ pte_free(NULL, pte); /* our impl. doesn't use the mm arg */
+ pmd_free(NULL, pmd); /* ditto */
free:
free_pages((unsigned long) pgd, 2);
}
Index: linux-work/arch/frv/mm/pgalloc.c
===================================================================
--- linux-work.orig/arch/frv/mm/pgalloc.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/frv/mm/pgalloc.c 2007-08-06 16:15:18.000000000 +1000
@@ -140,7 +140,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
return pgd;
}
-void pgd_free(pgd_t *pgd)
+void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
/* in the non-PAE case, clear_page_tables() clears user pgd entries */
quicklist_free(0, pgd_dtor, pgd);
Index: linux-work/arch/i386/mm/pgtable.c
===================================================================
--- linux-work.orig/arch/i386/mm/pgtable.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/i386/mm/pgtable.c 2007-08-06 16:15:18.000000000 +1000
@@ -350,7 +350,7 @@ out_oom:
return NULL;
}
-void pgd_free(pgd_t *pgd)
+void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
int i;
Index: linux-work/arch/um/kernel/mem.c
===================================================================
--- linux-work.orig/arch/um/kernel/mem.c 2007-08-06 13:48:28.000000000 +1000
+++ linux-work/arch/um/kernel/mem.c 2007-08-06 16:15:18.000000000 +1000
@@ -348,7 +348,7 @@ pgd_t *pgd_alloc(struct mm_struct *mm)
return pgd;
}
-void pgd_free(pgd_t *pgd)
+void pgd_free(struct mm_struct *mm, pgd_t *pgd)
{
free_page((unsigned long) pgd);
}
Index: linux-work/arch/um/kernel/skas/mmu.c
===================================================================
--- linux-work.orig/arch/um/kernel/skas/mmu.c 2007-08-06 13:48:28.000000000 +1000
+++ linux-work/arch/um/kernel/skas/mmu.c 2007-08-06 16:15:18.000000000 +1000
@@ -64,9 +64,9 @@ static int init_stub_pte(struct mm_struc
return(0);
out_pmd:
- pud_free(pud);
+ pud_free(mm, pud);
out_pte:
- pmd_free(pmd);
+ pmd_free(mm, pmd);
out:
return(-ENOMEM);
}
@@ -152,7 +152,7 @@ void destroy_context_skas(struct mm_stru
pte_free_kernel((pte_t *) mmu->last_page_table);
dec_zone_page_state(virt_to_page(mmu->last_page_table), NR_PAGETABLE);
#ifdef CONFIG_3_LEVEL_PGTABLES
- pmd_free((pmd_t *) mmu->last_pmd);
+ pmd_free(mm, (pmd_t *) mmu->last_pmd);
#endif
}
}
Index: linux-work/kernel/fork.c
===================================================================
--- linux-work.orig/kernel/fork.c 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/kernel/fork.c 2007-08-06 16:26:17.000000000 +1000
@@ -313,7 +313,7 @@ static inline int mm_alloc_pgd(struct mm
static inline void mm_free_pgd(struct mm_struct * mm)
{
- pgd_free(mm->pgd);
+ pgd_free(mm, mm->pgd);
}
#else
#define dup_mmap(mm, oldmm) (0)
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 6/12] Add "address" argument to pte/pmd/pud_free_tlb()
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (4 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 5/12] Add mm argument to pte/pmd/pud/pgd_free Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 7/12] ia64 tracks freed page tables addresses Benjamin Herrenschmidt
` (5 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
Architectures like ia64 who use a virtual page table can benefit
from knowing the virtual address range affected by a page table
being removed. We already pass that information to the alloc
functions, let's pass it to the free ones. I've only changed
the tlb_* versions for simplicity.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/mm/hugetlbpage.c | 4 ++--
include/asm-alpha/tlb.h | 4 ++--
include/asm-arm/tlb.h | 4 ++--
include/asm-cris/pgalloc.h | 2 +-
include/asm-frv/pgalloc.h | 4 ++--
include/asm-generic/4level-fixup.h | 4 ++--
include/asm-generic/pgtable-nopmd.h | 2 +-
include/asm-generic/pgtable-nopud.h | 2 +-
include/asm-generic/tlb.h | 12 ++++++------
include/asm-i386/pgalloc.h | 4 ++--
include/asm-ia64/pgalloc.h | 6 +++---
include/asm-ia64/tlb.h | 12 ++++++------
include/asm-m32r/pgalloc.h | 4 ++--
include/asm-m68k/motorola_pgalloc.h | 6 ++++--
include/asm-m68k/sun3_pgalloc.h | 4 ++--
include/asm-mips/pgalloc.h | 6 +++---
include/asm-parisc/tlb.h | 4 ++--
include/asm-powerpc/pgalloc-32.h | 4 ++--
include/asm-powerpc/pgalloc-64.h | 6 +++---
include/asm-ppc/pgalloc.h | 4 ++--
include/asm-s390/pgalloc.h | 6 +++---
include/asm-sh/pgalloc.h | 4 ++--
include/asm-sh64/pgalloc.h | 7 +++----
include/asm-sparc/pgalloc.h | 4 ++--
include/asm-sparc64/tlb.h | 6 +++---
include/asm-x86_64/pgalloc.h | 6 +++---
include/asm-xtensa/tlb.h | 2 +-
mm/memory.c | 11 ++++++-----
28 files changed, 73 insertions(+), 71 deletions(-)
Index: linux-work/include/asm-m68k/sun3_pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m68k/sun3_pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-m68k/sun3_pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -31,7 +31,7 @@ static inline void pte_free(struct mm_st
__free_page(page);
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm,
unsigned long address)
@@ -73,7 +73,7 @@ static inline void pmd_populate(struct m
* inside the pgd, so has no extra memory associated with it.
*/
#define pmd_free(mm, x) do { } while (0)
-#define __pmd_free_tlb(tlb, x) do { } while (0)
+#define __pmd_free_tlb(tlb, x, a) do { } while (0)
static inline void pgd_free(struct mm_struct *mm, pgd_t * pgd)
{
Index: linux-work/include/asm-generic/tlb.h
===================================================================
--- linux-work.orig/include/asm-generic/tlb.h 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/include/asm-generic/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -123,24 +123,24 @@ static inline void tlb_remove_page(struc
__tlb_remove_tlb_entry(tlb, ptep, address); \
} while (0)
-#define pte_free_tlb(tlb, ptep) \
+#define pte_free_tlb(tlb, ptep, address) \
do { \
tlb->need_flush = 1; \
- __pte_free_tlb(tlb, ptep); \
+ __pte_free_tlb(tlb, ptep, address); \
} while (0)
#ifndef __ARCH_HAS_4LEVEL_HACK
-#define pud_free_tlb(tlb, pudp) \
+#define pud_free_tlb(tlb, pudp, address) \
do { \
tlb->need_flush = 1; \
- __pud_free_tlb(tlb, pudp); \
+ __pud_free_tlb(tlb, pudp, address); \
} while (0)
#endif
-#define pmd_free_tlb(tlb, pmdp) \
+#define pmd_free_tlb(tlb, pmdp, address) \
do { \
tlb->need_flush = 1; \
- __pmd_free_tlb(tlb, pmdp); \
+ __pmd_free_tlb(tlb, pmdp, address); \
} while (0)
#define tlb_migrate_finish(mm) do {} while (0)
Index: linux-work/include/asm-alpha/tlb.h
===================================================================
--- linux-work.orig/include/asm-alpha/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-alpha/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -9,7 +9,7 @@
#include <asm-generic/tlb.h>
-#define __pte_free_tlb(tlb,pte) pte_free((tlb)->mm, pte)
-#define __pmd_free_tlb(tlb,pmd) pmd_free((tlb)->mm, pmd)
+#define __pte_free_tlb(tlb,pte,address) pte_free((tlb)->mm, pte)
+#define __pmd_free_tlb(tlb,pmd,address) pmd_free((tlb)->mm, pmd)
#endif
Index: linux-work/include/asm-arm/tlb.h
===================================================================
--- linux-work.orig/include/asm-arm/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-arm/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -85,8 +85,8 @@ tlb_end_vma(struct mmu_gather *tlb, stru
}
#define tlb_remove_page(tlb,page) free_page_and_swap_cache(page)
-#define pte_free_tlb(tlb,ptep) pte_free((tlb)->mm, ptep)
-#define pmd_free_tlb(tlb,pmdp) pmd_free((tlb)->mm, pmdp)
+#define pte_free_tlb(tlb,ptep,address) pte_free((tlb)->mm, ptep)
+#define pmd_free_tlb(tlb,pmdp,address) pmd_free((tlb)->mm, pmdp)
#define tlb_migrate_finish(mm) do { } while (0)
Index: linux-work/include/asm-cris/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-cris/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-cris/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -45,7 +45,7 @@ static inline void pte_free(struct mm_st
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-frv/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-frv/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-frv/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -47,7 +47,7 @@ static inline void pte_free(struct mm_st
__free_page(pte);
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
/*
* allocating and freeing a pmd is trivial: the 1-entry pmd is
@@ -56,7 +56,7 @@ static inline void pte_free(struct mm_st
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *) 2); })
#define pmd_free(mm, x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
#endif /* CONFIG_MMU */
Index: linux-work/include/asm-i386/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-i386/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-i386/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -49,7 +49,7 @@ static inline void pte_free(struct mm_st
}
-#define __pte_free_tlb(tlb,pte) \
+#define __pte_free_tlb(tlb,pte,address) \
do { \
paravirt_release_pt(page_to_pfn(pte)); \
tlb_remove_page((tlb),(pte)); \
@@ -61,7 +61,7 @@ do { \
*/
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
#define pmd_free(mm,x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
#define pud_populate(mm, pmd, pte) BUG()
#endif
Index: linux-work/include/asm-ia64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-ia64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-ia64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -48,7 +48,7 @@ static inline void pud_free(struct mm_st
{
quicklist_free(0, NULL, pud);
}
-#define __pud_free_tlb(tlb, pud) pud_free((tlb)->mm, pud)
+#define __pud_free_tlb(tlb, pud, address) pud_free((tlb)->mm, pud)
#endif /* CONFIG_PGTABLE_4 */
static inline void
@@ -67,7 +67,7 @@ static inline void pmd_free(struct mm_st
quicklist_free(0, NULL, pmd);
}
-#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)
static inline void
pmd_populate(struct mm_struct *mm, pmd_t * pmd_entry, struct page *pte)
@@ -109,6 +109,6 @@ static inline void check_pgt_cache(void)
quicklist_trim(0, NULL, 25, 16);
}
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
+#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte)
#endif /* _ASM_IA64_PGALLOC_H */
Index: linux-work/include/asm-m32r/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m32r/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-m32r/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -56,7 +56,7 @@ static __inline__ void pte_free(struct m
__free_page(pte);
}
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
+#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, (pte))
/*
* allocating and freeing a pmd is trivial: the 1-entry pmd is
@@ -66,7 +66,7 @@ static __inline__ void pte_free(struct m
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
#define pmd_free(mm,x) do { } while (0)
-#define __pmd_free_tlb(tlb, x) do { } while (0)
+#define __pmd_free_tlb(tlb, x, a) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-m68k/motorola_pgalloc.h
===================================================================
--- linux-work.orig/include/asm-m68k/motorola_pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-m68k/motorola_pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -54,7 +54,8 @@ static inline void pte_free(struct mm_st
__free_page(page);
}
-static inline void __pte_free_tlb(struct mmu_gather *tlb, struct page *page)
+static inline void __pte_free_tlb(struct mmu_gather *tlb, struct page *page,
+ unsigned long address)
{
cache_page(kmap(page));
kunmap(page);
@@ -72,7 +73,8 @@ static inline int pmd_free(struct mm_str
return free_pointer_table(pmd);
}
-static inline int __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd)
+static inline int __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd,
+ unsigned long address)
{
return free_pointer_table(pmd);
}
Index: linux-work/include/asm-mips/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-mips/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-mips/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -95,7 +95,7 @@ static inline void pte_free(struct mm_st
__free_pages(pte, PTE_ORDER);
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
#ifdef CONFIG_32BIT
@@ -104,7 +104,7 @@ static inline void pte_free(struct mm_st
* inside the pgd, so has no extra memory associated with it.
*/
#define pmd_free(mm, x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
#endif
@@ -125,7 +125,7 @@ static inline void pmd_free(struct mm_st
free_pages((unsigned long)pmd, PMD_ORDER);
}
-#define __pmd_free_tlb(tlb,x) pmd_free((tlb)->mm, x)
+#define __pmd_free_tlb(tlb,x,a) pmd_free((tlb)->mm, x)
#endif
Index: linux-work/include/asm-parisc/tlb.h
===================================================================
--- linux-work.orig/include/asm-parisc/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-parisc/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -21,7 +21,7 @@ do { if (!(tlb)->fullmm) \
#include <asm-generic/tlb.h>
-#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
+#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)
+#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte)
#endif
Index: linux-work/include/asm-powerpc/pgalloc-32.h
===================================================================
--- linux-work.orig/include/asm-powerpc/pgalloc-32.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-powerpc/pgalloc-32.h 2007-08-06 16:15:18.000000000 +1000
@@ -14,7 +14,7 @@ extern void pgd_free(struct mm_struct *m
*/
/* #define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); }) */
#define pmd_free(mm, x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
/* #define pgd_populate(mm, pmd, pte) BUG() */
#ifndef CONFIG_BOOKE
@@ -34,7 +34,7 @@ extern struct page *pte_alloc_one(struct
extern void pte_free_kernel(pte_t *pte);
extern void pte_free(struct mm_struct *mm, struct page *pte);
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
+#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, (pte))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-powerpc/pgalloc-64.h
===================================================================
--- linux-work.orig/include/asm-powerpc/pgalloc-64.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-powerpc/pgalloc-64.h 2007-08-06 16:15:18.000000000 +1000
@@ -131,14 +131,14 @@ static inline void pgtable_free(pgtable_
extern void pgtable_free_tlb(struct mmu_gather *tlb, pgtable_free_t pgf);
-#define __pte_free_tlb(tlb, ptepage) \
+#define __pte_free_tlb(tlb, ptepage, address) \
pgtable_free_tlb(tlb, pgtable_free_cache(page_address(ptepage), \
PTE_NONCACHE_NUM, PTE_TABLE_SIZE-1))
-#define __pmd_free_tlb(tlb, pmd) \
+#define __pmd_free_tlb(tlb, pmd, address) \
pgtable_free_tlb(tlb, pgtable_free_cache(pmd, \
PMD_CACHE_NUM, PMD_TABLE_SIZE-1))
#ifndef CONFIG_PPC_64K_PAGES
-#define __pud_free_tlb(tlb, pud) \
+#define __pud_free_tlb(tlb, pud, address) \
pgtable_free_tlb(tlb, pgtable_free_cache(pud, \
PUD_CACHE_NUM, PUD_TABLE_SIZE-1))
#endif /* CONFIG_PPC_64K_PAGES */
Index: linux-work/include/asm-ppc/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-ppc/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-ppc/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -15,7 +15,7 @@ extern void pgd_free(struct mm_struct *m
*/
#define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); })
#define pmd_free(mm,x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#ifndef CONFIG_BOOKE
@@ -35,7 +35,7 @@ extern struct page *pte_alloc_one(struct
extern void pte_free_kernel(pte_t *pte);
extern void pte_free(struct mm_struct *mm, struct page *pte);
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, (pte))
+#define __pte_free_tlb(tlb, pte, addr) pte_free((tlb)->mm, (pte))
#define check_pgt_cache() do { } while (0)
Index: linux-work/include/asm-s390/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-s390/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-s390/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -84,7 +84,7 @@ static inline void pgd_free(struct mm_st
*/
#define pmd_alloc_one(mm,address) ({ BUG(); ((pmd_t *)2); })
#define pmd_free(mm, x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
#define pgd_populate_kernel(mm, pmd, pte) BUG()
#else /* __s390x__ */
@@ -120,7 +120,7 @@ static inline void pmd_free (struct mm_s
free_pages((unsigned long) pmd, PMD_ALLOC_ORDER);
}
-#define __pmd_free_tlb(tlb,pmd) \
+#define __pmd_free_tlb(tlb,pmd,address) \
do { \
tlb_flush_mmu(tlb, 0, 0); \
pmd_free((tlb)->mm, pmd); \
@@ -226,7 +226,7 @@ static inline void pte_free(struct mm_st
__free_page(pte);
}
-#define __pte_free_tlb(tlb, pte) \
+#define __pte_free_tlb(tlb, pte, address) \
({ \
struct mmu_gather *__tlb = (tlb); \
struct page *__pte = (pte); \
Index: linux-work/include/asm-sh/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sh/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-sh/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -64,7 +64,7 @@ static inline void pte_free(struct mm_st
quicklist_free_page(QUICK_PT, NULL, pte);
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
/*
* allocating and freeing a pmd is trivial: the 1-entry pmd is
@@ -72,7 +72,7 @@ static inline void pte_free(struct mm_st
*/
#define pmd_free(mm,x) do { } while (0)
-#define __pmd_free_tlb(tlb,x) do { } while (0)
+#define __pmd_free_tlb(tlb,x,a) do { } while (0)
static inline void check_pgt_cache(void)
{
Index: linux-work/include/asm-sh64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sh64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-sh64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -74,7 +74,7 @@ static inline pte_t *pte_alloc_one_kerne
return quicklist_alloc(0, GFP_KERNEL, NULL);
}
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
+#define __pte_free_tlb(tlb,pte,address) tlb_remove_page((tlb),(pte))
/*
* allocating and freeing a pmd is trivial: the 1-entry pmd is
@@ -86,8 +86,7 @@ static inline pte_t *pte_alloc_one_kerne
#define pmd_alloc_one(mm, addr) ({ BUG(); ((pmd_t *)2); })
#define pmd_free(mm, x) do { } while (0)
#define pgd_populate(mm, pmd, pte) BUG()
-#define __pte_free_tlb(tlb,pte) tlb_remove_page((tlb),(pte))
-#define __pmd_free_tlb(tlb,pmd) do { } while (0)
+#define __pmd_free_tlb(tlb,pmd,addr) do { } while (0)
#elif defined(CONFIG_SH64_PGTABLE_3_LEVEL)
@@ -102,7 +101,7 @@ static inline void pmd_free(struct mm_st
}
#define pgd_populate(mm, pgd, pmd) pgd_set(pgd, pmd)
-#define __pmd_free_tlb(tlb,pmd) pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb,pmd,addr) pmd_free((tlb)->mm, pmd)
#else
#error "No defined page table size"
Index: linux-work/include/asm-sparc/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-sparc/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-sparc/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -46,7 +46,7 @@ BTFIXUPDEF_CALL(void, free_pmd_fast, pmd
#define free_pmd_fast(pmd) BTFIXUP_CALL(free_pmd_fast)(pmd)
#define pmd_free(mm,pmd) free_pmd_fast(pmd)
-#define __pmd_free_tlb(tlb, pmd) pmd_free((tlb)->mm, pmd)
+#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)
BTFIXUPDEF_CALL(void, pmd_populate, pmd_t *, struct page *)
#define pmd_populate(MM, PMD, PTE) BTFIXUP_CALL(pmd_populate)(PMD, PTE)
@@ -63,6 +63,6 @@ BTFIXUPDEF_CALL(void, free_pte_fast, pte
BTFIXUPDEF_CALL(void, pte_free, struct page *)
#define pte_free(mm,pte) BTFIXUP_CALL(pte_free)(pte)
-#define __pte_free_tlb(tlb, pte) pte_free((tlb)->mm, pte)
+#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte)
#endif /* _SPARC_PGALLOC_H */
Index: linux-work/include/asm-sparc64/tlb.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-sparc64/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -48,9 +48,9 @@ extern void smp_flush_tlb_mm(struct mm_s
#define do_flush_tlb_mm(mm) __flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT)
#endif
-#define __tlb_remove_tlb_entry(mp,ptep,addr) do { } while (0)
-#define __pte_free_tlb(mp,ptepage) pte_free((mp)->mm,ptepage)
-#define __pmd_free_tlb(mp,pmdp) pmd_free((mp)->mm,pmdp)
+#define __tlb_remove_tlb_entry(mp,ptep,addr) do { } while (0)
+#define __pte_free_tlb(mp,ptepage,address) pte_free((mp)->mm,ptepage)
+#define __pmd_free_tlb(mp,pmdp,address) pmd_free((mp)->mm,pmdp)
#define tlb_start_vma(tlb, vma) do { } while (0)
#define tlb_end_vma(tlb, vma) do { } while (0)
Index: linux-work/include/asm-x86_64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-x86_64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-x86_64/pgalloc.h 2007-08-06 16:15:18.000000000 +1000
@@ -131,10 +131,10 @@ static inline void pte_free(struct mm_st
quicklist_free_page(QUICK_PT, NULL, pte);
}
-#define __pte_free_tlb(tlb,pte) quicklist_free_page(QUICK_PT, NULL,(pte))
+#define __pte_free_tlb(tlb,pte,addr) quicklist_free_page(QUICK_PT, NULL,(pte))
-#define __pmd_free_tlb(tlb,x) quicklist_free(QUICK_PT, NULL, (x))
-#define __pud_free_tlb(tlb,x) quicklist_free(QUICK_PT, NULL, (x))
+#define __pmd_free_tlb(tlb,x,addr) quicklist_free(QUICK_PT, NULL, (x))
+#define __pud_free_tlb(tlb,x,addr) quicklist_free(QUICK_PT, NULL, (x))
static inline void check_pgt_cache(void)
{
Index: linux-work/include/asm-xtensa/tlb.h
===================================================================
--- linux-work.orig/include/asm-xtensa/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-xtensa/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -20,6 +20,6 @@
#include <asm-generic/tlb.h>
#include <asm/page.h>
-#define __pte_free_tlb(tlb,pte) pte_free((tlb)->mm, pte)
+#define __pte_free_tlb(tlb,pte,address) pte_free((tlb)->mm, pte)
#endif /* _XTENSA_TLB_H */
Index: linux-work/arch/powerpc/mm/hugetlbpage.c
===================================================================
--- linux-work.orig/arch/powerpc/mm/hugetlbpage.c 2007-08-06 13:48:27.000000000 +1000
+++ linux-work/arch/powerpc/mm/hugetlbpage.c 2007-08-06 16:15:18.000000000 +1000
@@ -192,7 +192,7 @@ static void hugetlb_free_pmd_range(struc
pmd = pmd_offset(pud, start);
pud_clear(pud);
- pmd_free_tlb(tlb, pmd);
+ pmd_free_tlb(tlb, pmd, addr);
}
#endif
@@ -232,7 +232,7 @@ static void hugetlb_free_pud_range(struc
pud = pud_offset(pgd, start);
pgd_clear(pgd);
- pud_free_tlb(tlb, pud);
+ pud_free_tlb(tlb, pud, addr);
}
/*
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-06 16:15:18.000000000 +1000
@@ -120,12 +120,13 @@ void pmd_clear_bad(pmd_t *pmd)
* Note: this doesn't free the actual pages themselves. That
* has been handled earlier when unmapping all the memory regions.
*/
-static void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd)
+static void free_pte_range(struct mmu_gather *tlb, pmd_t *pmd,
+ unsigned long addr)
{
struct page *page = pmd_page(*pmd);
pmd_clear(pmd);
pte_lock_deinit(page);
- pte_free_tlb(tlb, page);
+ pte_free_tlb(tlb, page, addr);
dec_zone_page_state(page, NR_PAGETABLE);
tlb->mm->nr_ptes--;
}
@@ -144,7 +145,7 @@ static inline void free_pmd_range(struct
next = pmd_addr_end(addr, end);
if (pmd_none_or_clear_bad(pmd))
continue;
- free_pte_range(tlb, pmd);
+ free_pte_range(tlb, pmd, addr);
} while (pmd++, addr = next, addr != end);
start &= PUD_MASK;
@@ -160,7 +161,7 @@ static inline void free_pmd_range(struct
pmd = pmd_offset(pud, start);
pud_clear(pud);
- pmd_free_tlb(tlb, pmd);
+ pmd_free_tlb(tlb, pmd, addr);
}
static inline void free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,
@@ -193,7 +194,7 @@ static inline void free_pud_range(struct
pud = pud_offset(pgd, start);
pgd_clear(pgd);
- pud_free_tlb(tlb, pud);
+ pud_free_tlb(tlb, pud, addr);
}
/*
Index: linux-work/include/asm-ia64/tlb.h
===================================================================
--- linux-work.orig/include/asm-ia64/tlb.h 2007-07-27 13:44:45.000000000 +1000
+++ linux-work/include/asm-ia64/tlb.h 2007-08-06 16:24:42.000000000 +1000
@@ -210,22 +210,22 @@ do { \
__tlb_remove_tlb_entry(tlb, ptep, addr); \
} while (0)
-#define pte_free_tlb(tlb, ptep) \
+#define pte_free_tlb(tlb, ptep, addr) \
do { \
tlb->need_flush = 1; \
- __pte_free_tlb(tlb, ptep); \
+ __pte_free_tlb(tlb, ptep, addr); \
} while (0)
-#define pmd_free_tlb(tlb, ptep) \
+#define pmd_free_tlb(tlb, ptep, addr) \
do { \
tlb->need_flush = 1; \
- __pmd_free_tlb(tlb, ptep); \
+ __pmd_free_tlb(tlb, ptep, addr); \
} while (0)
-#define pud_free_tlb(tlb, pudp) \
+#define pud_free_tlb(tlb, pudp, addr) \
do { \
tlb->need_flush = 1; \
- __pud_free_tlb(tlb, pudp); \
+ __pud_free_tlb(tlb, pudp, addr); \
} while (0)
#endif /* _ASM_IA64_TLB_H */
Index: linux-work/include/asm-generic/4level-fixup.h
===================================================================
--- linux-work.orig/include/asm-generic/4level-fixup.h 2007-08-06 16:25:33.000000000 +1000
+++ linux-work/include/asm-generic/4level-fixup.h 2007-08-06 16:25:49.000000000 +1000
@@ -27,9 +27,9 @@
#define pud_page_vaddr(pud) pgd_page_vaddr(pud)
#undef pud_free_tlb
-#define pud_free_tlb(tlb, x) do { } while (0)
+#define pud_free_tlb(tlb, x, a) do { } while (0)
#define pud_free(mm,x) do { } while (0)
-#define __pud_free_tlb(tlb, x) do { } while (0)
+#define __pud_free_tlb(tlb, x, a) do { } while (0)
#undef pud_addr_end
#define pud_addr_end(addr, end) (end)
Index: linux-work/include/asm-generic/pgtable-nopmd.h
===================================================================
--- linux-work.orig/include/asm-generic/pgtable-nopmd.h 2007-08-06 16:24:54.000000000 +1000
+++ linux-work/include/asm-generic/pgtable-nopmd.h 2007-08-06 16:25:06.000000000 +1000
@@ -55,7 +55,7 @@ static inline pmd_t * pmd_offset(pud_t *
*/
#define pmd_alloc_one(mm, address) NULL
#define pmd_free(mm,x) do { } while (0)
-#define __pmd_free_tlb(tlb, x) do { } while (0)
+#define __pmd_free_tlb(tlb, x, a) do { } while (0)
#undef pmd_addr_end
#define pmd_addr_end(addr, end) (end)
Index: linux-work/include/asm-generic/pgtable-nopud.h
===================================================================
--- linux-work.orig/include/asm-generic/pgtable-nopud.h 2007-08-06 16:25:21.000000000 +1000
+++ linux-work/include/asm-generic/pgtable-nopud.h 2007-08-06 16:25:25.000000000 +1000
@@ -52,7 +52,7 @@ static inline pud_t * pud_offset(pgd_t *
*/
#define pud_alloc_one(mm, address) NULL
#define pud_free(mm,x) do { } while (0)
-#define __pud_free_tlb(tlb, x) do { } while (0)
+#define __pud_free_tlb(tlb, x, a) do { } while (0)
#undef pud_addr_end
#define pud_addr_end(addr, end) (end)
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 7/12] ia64 tracks freed page tables addresses
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (5 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 6/12] Add "address" argument to pte/pmd/pud_free_tlb() Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 8/12] remove ptep_get_and_clear_full Benjamin Herrenschmidt
` (4 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
Until now, ia64 pretty much relied on the start/end arguments
passed to tlb_finish_mmu() to flush the virtual page tables.
Not only these tend to provide larger ranges than necessary,
but keeping track in the callers is a pain and I intend to remove
those from my mmu_gather rework.
This patch uses the newly added "address" arguemnt to pte_free_tlb()
to track the actual range covered by freed page tables and uses
that to perform the actual freeing.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
include/asm-ia64/tlb.h | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
Index: linux-work/include/asm-ia64/tlb.h
===================================================================
--- linux-work.orig/include/asm-ia64/tlb.h 2007-08-06 16:15:18.000000000 +1000
+++ linux-work/include/asm-ia64/tlb.h 2007-08-06 16:15:18.000000000 +1000
@@ -61,6 +61,8 @@ struct mmu_gather {
unsigned char need_flush; /* really unmapped some PTEs? */
unsigned long start_addr;
unsigned long end_addr;
+ unsigned long start_pgtable;
+ unsigned long end_pgtable;
struct page *pages[FREE_PTE_NR];
};
@@ -72,8 +74,10 @@ DECLARE_PER_CPU(struct mmu_gather, mmu_g
* freed pages that where gathered up to this point.
*/
static inline void
-ia64_tlb_flush_mmu (struct mmu_gather *tlb, unsigned long start, unsigned long end)
+ia64_tlb_flush_mmu (struct mmu_gather *tlb)
{
+ unsigned long start = tlb->start_addr;
+ unsigned long end = tlb->end_addr;
unsigned int nr;
if (!tlb->need_flush)
@@ -107,7 +111,10 @@ ia64_tlb_flush_mmu (struct mmu_gather *t
/* flush the address range from the tlb: */
flush_tlb_range(&vma, start, end);
/* now flush the virt. page-table area mapping the address range: */
- flush_tlb_range(&vma, ia64_thash(start), ia64_thash(end));
+ if (tlb->start_pgtable < tlb->end_pgtable)
+ flush_tlb_range(&vma,
+ ia64_thash(tlb->start_pgtable),
+ ia64_thash(tlb->end_pgtable));
}
/* lastly, release the freed pages */
@@ -115,7 +122,7 @@ ia64_tlb_flush_mmu (struct mmu_gather *t
if (!tlb_fast_mode(tlb)) {
unsigned long i;
tlb->nr = 0;
- tlb->start_addr = ~0UL;
+ tlb->start_addr = tlb->start_pgtable = ~0UL;
for (i = 0; i < nr; ++i)
free_page_and_swap_cache(tlb->pages[i]);
}
@@ -145,7 +152,7 @@ tlb_gather_mmu (struct mm_struct *mm, un
*/
tlb->nr = (num_online_cpus() == 1) ? ~0U : 0;
tlb->fullmm = full_mm_flush;
- tlb->start_addr = ~0UL;
+ tlb->start_addr = tlb->start_pgtable = ~0UL;
return tlb;
}
@@ -160,7 +167,7 @@ tlb_finish_mmu (struct mmu_gather *tlb,
* Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and
* tlb->end_addr.
*/
- ia64_tlb_flush_mmu(tlb, start, end);
+ ia64_tlb_flush_mmu(tlb);
/* keep the page table cache within bounds */
check_pgt_cache();
@@ -184,7 +191,7 @@ tlb_remove_page (struct mmu_gather *tlb,
}
tlb->pages[tlb->nr++] = page;
if (tlb->nr >= FREE_PTE_NR)
- ia64_tlb_flush_mmu(tlb, tlb->start_addr, tlb->end_addr);
+ ia64_tlb_flush_mmu(tlb);
}
/*
@@ -194,7 +201,7 @@ tlb_remove_page (struct mmu_gather *tlb,
static inline void
__tlb_remove_tlb_entry (struct mmu_gather *tlb, pte_t *ptep, unsigned long address)
{
- if (tlb->start_addr == ~0UL)
+ if (tlb->start_addr > address)
tlb->start_addr = address;
tlb->end_addr = address + PAGE_SIZE;
}
@@ -213,6 +220,9 @@ do { \
#define pte_free_tlb(tlb, ptep, addr) \
do { \
tlb->need_flush = 1; \
+ if (tlb->start_pgtable > addr) \
+ tlb->start_pgtable = addr; \
+ tlb->end_pgtable = (addr + PMD_SIZE) & PMD_MASK;\
__pte_free_tlb(tlb, ptep, addr); \
} while (0)
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 8/12] remove ptep_get_and_clear_full
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (6 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 7/12] ia64 tracks freed page tables addresses Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 9/12] mmu_gather on stack, part 1 Benjamin Herrenschmidt
` (3 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This patch removes it, instead, the arch implementations that
care use the new MMF_DEAD flag in the mm_struct.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
NOTE: I'm not necessarily convinced yet this is the right approach,
but I needed to get rid of it because my current new batch structure
doesn't have this fullmm field anymore.
However, I'm thinking about putting it back in and replacing
ptep_get_and_clear with a tlb_get_and_clear_pte() which would
take a batch as an argument instead once I'm done moving all
page table walkers to use the batch for TLB operations.
include/asm-generic/pgtable.h | 9 ---------
include/asm-i386/pgtable.h | 17 +++++++----------
include/asm-x86_64/pgtable.h | 8 ++++----
mm/memory.c | 3 +--
4 files changed, 12 insertions(+), 25 deletions(-)
Index: linux-work/include/asm-generic/pgtable.h
===================================================================
--- linux-work.orig/include/asm-generic/pgtable.h 2007-08-07 16:19:14.000000000 +1000
+++ linux-work/include/asm-generic/pgtable.h 2007-08-07 16:19:19.000000000 +1000
@@ -58,15 +58,6 @@
})
#endif
-#ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL
-#define ptep_get_and_clear_full(__mm, __address, __ptep, __full) \
-({ \
- pte_t __pte; \
- __pte = ptep_get_and_clear((__mm), (__address), (__ptep)); \
- __pte; \
-})
-#endif
-
/*
* Some architectures may be able to avoid expensive synchronization
* primitives when modifications are made to PTE's which are already
Index: linux-work/include/asm-i386/pgtable.h
===================================================================
--- linux-work.orig/include/asm-i386/pgtable.h 2007-08-07 16:20:05.000000000 +1000
+++ linux-work/include/asm-i386/pgtable.h 2007-08-07 16:22:05.000000000 +1000
@@ -311,15 +311,9 @@ static inline pte_t native_local_ptep_ge
})
#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
-static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
-{
- pte_t pte = native_ptep_get_and_clear(ptep);
- pte_update(mm, addr, ptep);
- return pte;
-}
-
-#define __HAVE_ARCH_PTEP_GET_AND_CLEAR_FULL
-static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, unsigned long addr, pte_t *ptep, int full)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long addr, pte_t *ptep,
+ int full)
{
pte_t pte;
if (full) {
@@ -329,10 +323,13 @@ static inline pte_t ptep_get_and_clear_f
*/
pte = native_local_ptep_get_and_clear(ptep);
} else {
- pte = ptep_get_and_clear(mm, addr, ptep);
+ pte = native_ptep_get_and_clear(ptep);
+ pte_update(mm, addr, ptep);
}
return pte;
}
+#define ptep_get_and_clear(mm, addr, ptep) \
+ __ptep_get_and_clear(mm, addr, ptep, test_bit(MMF_DEAD, &mm->flags))
#define __HAVE_ARCH_PTEP_SET_WRPROTECT
static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
Index: linux-work/include/asm-x86_64/pgtable.h
===================================================================
--- linux-work.orig/include/asm-x86_64/pgtable.h 2007-08-07 16:22:23.000000000 +1000
+++ linux-work/include/asm-x86_64/pgtable.h 2007-08-07 16:23:14.000000000 +1000
@@ -102,21 +102,21 @@ static inline void pgd_clear (pgd_t * pg
set_pgd(pgd, __pgd(0));
}
-#define ptep_get_and_clear(mm,addr,xp) __pte(xchg(&(xp)->pte, 0))
-
struct mm_struct;
-static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, unsigned long addr, pte_t *ptep, int full)
+static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, int full)
{
pte_t pte;
if (full) {
pte = *ptep;
*ptep = __pte(0);
} else {
- pte = ptep_get_and_clear(mm, addr, ptep);
+ __pte(xchg(&(xp)->pte, 0))
}
return pte;
}
+#define ptep_get_and_clear(mm, addr, ptep) \
+ __ptep_get_and_clear(mm, addr, ptep, test_bit(MMF_DEAD, &mm->flags))
#define pte_same(a, b) ((a).pte == (b).pte)
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-07 16:18:43.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-07 16:18:48.000000000 +1000
@@ -658,8 +658,7 @@ static unsigned long zap_pte_range(struc
page->index > details->last_index))
continue;
}
- ptent = ptep_get_and_clear_full(mm, addr, pte,
- tlb->fullmm);
+ ptent = ptep_get_and_clear(mm, addr, pte);
tlb_remove_tlb_entry(tlb, pte, addr);
if (unlikely(!page))
continue;
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 9/12] mmu_gather on stack, part 1
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (7 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 8/12] remove ptep_get_and_clear_full Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 11/12] Use mmu_gather for fork() instead of flush_tlb_mm() Benjamin Herrenschmidt
` (2 subsequent siblings)
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This is the first step of moving the mmu_gather to a stack based
data structure and removing the per-cpu usage.
This patch reworks the mmu_gather such that it's made of two parts,
one is a stack based data structure, which optionally points to a
list of page pointers used when freeing pages. That list is for now
still kept per-cpu.
It also massages the mmu_gather APIs a bit, to avoid having archs
re-implementing it, but instead, having hooks for archs to use.
With that patch, platforms that don't use the batch for freeing page
tables (though that could be considered a bug...) will now have
free_pgtables() run without preemption disabling.
NOTE: This is still a WIP, arm hasn't been adapted yet among others
(I need to understand why it's not batching page freeing at all in
the first place).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/avr32/mm/init.c | 2
arch/i386/mm/init.c | 2
arch/ia64/mm/hugetlbpage.c | 2
arch/powerpc/mm/hugetlbpage.c | 8 -
arch/powerpc/mm/init_32.c | 2
arch/powerpc/mm/tlb_64.c | 2
arch/sparc/mm/init.c | 2
arch/sparc64/mm/tlb.c | 2
arch/um/kernel/smp.c | 2
arch/x86_64/mm/init.c | 2
arch/xtensa/mm/init.c | 2
fs/exec.c | 6 -
include/asm-generic/tlb.h | 83 ++++++++++++++-----
include/asm-i386/tlb.h | 5 -
include/asm-ia64/pgalloc.h | 3
include/asm-ia64/tlb.h | 180 ++++++++++--------------------------------
include/asm-parisc/tlb.h | 10 +-
include/asm-powerpc/tlb.h | 3
include/asm-sparc64/tlb.h | 3
include/asm-x86_64/tlb.h | 2
include/linux/hugetlb.h | 2
include/linux/mm.h | 6 -
mm/memory.c | 52 +++++-------
mm/mmap.c | 14 +--
24 files changed, 166 insertions(+), 231 deletions(-)
Index: linux-work/include/asm-generic/tlb.h
===================================================================
--- linux-work.orig/include/asm-generic/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-generic/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -33,48 +33,62 @@
#define tlb_fast_mode(tlb) 1
#endif
+/* arch may add fields to mmu_gather */
+#ifndef mmu_gather_arch
+struct mmu_gather_arch { };
+#define tlb_arch_init(tlb) do { } while(0)
+#define tlb_arch_finish(tlb) do { } while(0)
+#endif
+
/* struct mmu_gather is an opaque type used by the mm code for passing around
* any data needed by arch specific code for tlb_remove_page.
*/
struct mmu_gather {
struct mm_struct *mm;
+ unsigned int need_flush;/* Really changed some ptes? */
unsigned int nr; /* set to ~0U means fast mode */
- unsigned int need_flush;/* Really unmapped some ptes? */
- unsigned int fullmm; /* non-zero means full mm flush */
+ struct mmu_gather_arch archdata;
+ struct page ** pages;
+};
+
+/* per-cpu page list storage for an mmu_gather */
+struct mmu_gather_store {
struct page * pages[FREE_PTE_NR];
};
/* Users of the generic TLB shootdown code must declare this storage space. */
-DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
+DECLARE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
+
/* tlb_gather_mmu
* Return a pointer to an initialized struct mmu_gather.
*/
-static inline struct mmu_gather *
-tlb_gather_mmu(struct mm_struct *mm, unsigned int full_mm_flush)
+static inline void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm)
{
- struct mmu_gather *tlb = &get_cpu_var(mmu_gathers);
-
tlb->mm = mm;
+ tlb->need_flush = 0;
+ tlb->pages = NULL;
/* Use fast mode if only one CPU is online */
tlb->nr = num_online_cpus() > 1 ? 0U : ~0U;
- tlb->fullmm = full_mm_flush;
-
- return tlb;
+ tlb_arch_init(tlb);
}
-static inline void
-tlb_flush_mmu(struct mmu_gather *tlb, unsigned long start, unsigned long end)
+/* tlb_flush_mmu
+ * Call at any time the pending TLB needs to be flushed
+ */
+static inline void tlb_flush_mmu(struct mmu_gather *tlb)
{
if (!tlb->need_flush)
return;
tlb->need_flush = 0;
tlb_flush(tlb);
- if (!tlb_fast_mode(tlb)) {
+ if (!tlb_fast_mode(tlb) && tlb->pages) {
free_pages_and_swap_cache(tlb->pages, tlb->nr);
+ put_cpu_var(mmu_gather_store);
tlb->nr = 0;
+ tlb->pages = NULL;
}
}
@@ -82,17 +96,42 @@ tlb_flush_mmu(struct mmu_gather *tlb, un
* Called at the end of the shootdown operation to free up any resources
* that were required.
*/
-static inline void
-tlb_finish_mmu(struct mmu_gather *tlb, unsigned long start, unsigned long end)
+static inline void tlb_finish_mmu(struct mmu_gather *tlb)
{
- tlb_flush_mmu(tlb, start, end);
+ tlb_flush_mmu(tlb);
/* keep the page table cache within bounds */
check_pgt_cache();
- put_cpu_var(mmu_gathers);
+ tlb_arch_finish(tlb);
}
+/* tlb_pte_lock_break
+ * To be implemented by architectures that need to do something special
+ * before the PTE lock is released
+ */
+#ifndef tlb_pte_lock_break
+static inline void tlb_pte_lock_break(struct mmu_gather *tlb) { }
+#endif
+
+/* tlb_start_vma
+ * To be implemented by architectures that need to do something special
+ * before starting to flush a VMA
+ */
+#ifndef tlb_start_vma
+static inline void tlb_start_vma(struct mmu_gather *tlb,
+ struct vm_area_struct *vma) { }
+#endif
+
+/* tlb_end_vma
+ * To be implemented by architectures that need to do something special
+ * after finishing to flush a VMA
+ */
+#ifndef tlb_end_vma
+static inline void tlb_end_vma(struct mmu_gather *tlb,
+ struct vm_area_struct *vma) { }
+#endif
+
/* tlb_remove_page
* Must perform the equivalent to __free_pte(pte_get_and_clear(ptep)), while
* handling the additional races in SMP caused by other CPUs caching valid
@@ -105,11 +144,18 @@ static inline void tlb_remove_page(struc
free_page_and_swap_cache(page);
return;
}
+ /* Need to get pages ? */
+ if (!tlb->pages)
+ tlb->pages = get_cpu_var(mmu_gather_store).pages;
tlb->pages[tlb->nr++] = page;
if (tlb->nr >= FREE_PTE_NR)
- tlb_flush_mmu(tlb, 0, 0);
+ tlb_flush_mmu(tlb);
}
+#ifndef tlb_migrate_finish
+#define tlb_migrate_finish(mm) do {} while (0)
+#endif
+
/**
* tlb_remove_tlb_entry - remember a pte unmapping for later tlb invalidation.
*
@@ -143,6 +189,5 @@ static inline void tlb_remove_page(struc
__pmd_free_tlb(tlb, pmdp, address); \
} while (0)
-#define tlb_migrate_finish(mm) do {} while (0)
#endif /* _ASM_GENERIC__TLB_H */
Index: linux-work/arch/powerpc/mm/tlb_64.c
===================================================================
--- linux-work.orig/arch/powerpc/mm/tlb_64.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/powerpc/mm/tlb_64.c 2007-08-07 16:23:53.000000000 +1000
@@ -36,7 +36,7 @@ DEFINE_PER_CPU(struct ppc64_tlb_batch, p
/* This is declared as we are using the more or less generic
* include/asm-powerpc/tlb.h file -- tgall
*/
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
DEFINE_PER_CPU(struct pte_freelist_batch *, pte_freelist_cur);
unsigned long pte_freelist_forced_free;
Index: linux-work/include/asm-powerpc/tlb.h
===================================================================
--- linux-work.orig/include/asm-powerpc/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-powerpc/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -25,9 +25,6 @@
struct mmu_gather;
-#define tlb_start_vma(tlb, vma) do { } while (0)
-#define tlb_end_vma(tlb, vma) do { } while (0)
-
#if !defined(CONFIG_PPC_STD_MMU)
#define tlb_flush(tlb) flush_tlb_mm((tlb)->mm)
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-07 16:18:48.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-07 16:23:53.000000000 +1000
@@ -202,9 +202,9 @@ static inline void free_pud_range(struct
*
* Must be called with pagetable lock held.
*/
-void free_pgd_range(struct mmu_gather **tlb,
- unsigned long addr, unsigned long end,
- unsigned long floor, unsigned long ceiling)
+void free_pgd_range(struct mmu_gather *tlb,
+ unsigned long addr, unsigned long end,
+ unsigned long floor, unsigned long ceiling)
{
pgd_t *pgd;
unsigned long next;
@@ -253,16 +253,16 @@ void free_pgd_range(struct mmu_gather **
return;
start = addr;
- pgd = pgd_offset((*tlb)->mm, addr);
+ pgd = pgd_offset(tlb->mm, addr);
do {
next = pgd_addr_end(addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
- free_pud_range(*tlb, pgd, addr, next, floor, ceiling);
+ free_pud_range(tlb, pgd, addr, next, floor, ceiling);
} while (pgd++, addr = next, addr != end);
}
-void free_pgtables(struct mmu_gather **tlb, struct vm_area_struct *vma,
+void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,
unsigned long floor, unsigned long ceiling)
{
while (vma) {
@@ -275,6 +275,14 @@ void free_pgtables(struct mmu_gather **t
anon_vma_unlink(vma);
unlink_file_vma(vma);
+ /*
+ * Check if there's a need_resched here, flush the batch. That
+ * will drop the preempt block.
+ */
+ if (need_resched()) {
+ tlb_flush_mmu(tlb);
+ cond_resched();
+ }
if (is_vm_hugetlb_page(vma)) {
hugetlb_free_pgd_range(tlb, addr, vma->vm_end,
floor, next? next->vm_start: ceiling);
@@ -292,6 +300,7 @@ void free_pgtables(struct mmu_gather **t
free_pgd_range(tlb, addr, vma->vm_end,
floor, next? next->vm_start: ceiling);
}
+
vma = next;
}
}
@@ -693,6 +702,7 @@ static unsigned long zap_pte_range(struc
add_mm_rss(mm, file_rss, anon_rss);
arch_leave_lazy_mmu_mode();
+ tlb_pte_lock_break(tlb);
pte_unmap_unlock(pte - 1, ptl);
return addr;
@@ -803,17 +813,14 @@ static unsigned long unmap_page_range(st
* ensure that any thus-far unmapped pages are flushed before unmap_vmas()
* drops the lock and schedules.
*/
-unsigned long unmap_vmas(struct mmu_gather **tlbp,
+unsigned long unmap_vmas(struct mmu_gather *tlb,
struct vm_area_struct *vma, unsigned long start_addr,
unsigned long end_addr, unsigned long *nr_accounted,
struct zap_details *details)
{
long zap_work = ZAP_BLOCK_SIZE;
- unsigned long tlb_start = 0; /* For tlb_finish_mmu */
- int tlb_start_valid = 0;
unsigned long start = start_addr;
spinlock_t *i_mmap_lock = details? details->i_mmap_lock: NULL;
- int fullmm = (*tlbp)->fullmm;
for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) {
unsigned long end;
@@ -829,18 +836,13 @@ unsigned long unmap_vmas(struct mmu_gath
*nr_accounted += (end - start) >> PAGE_SHIFT;
while (start != end) {
- if (!tlb_start_valid) {
- tlb_start = start;
- tlb_start_valid = 1;
- }
-
if (unlikely(is_vm_hugetlb_page(vma))) {
unmap_hugepage_range(vma, start, end);
zap_work -= (end - start) /
(HPAGE_SIZE / PAGE_SIZE);
start = end;
} else
- start = unmap_page_range(*tlbp, vma,
+ start = unmap_page_range(tlb, vma,
start, end, &zap_work, details);
if (zap_work > 0) {
@@ -848,23 +850,18 @@ unsigned long unmap_vmas(struct mmu_gath
break;
}
- tlb_finish_mmu(*tlbp, tlb_start, start);
-
if (need_resched() ||
(i_mmap_lock && need_lockbreak(i_mmap_lock))) {
- if (i_mmap_lock) {
- *tlbp = NULL;
+ if (i_mmap_lock)
goto out;
- }
+ tlb_flush_mmu(tlb);
cond_resched();
}
-
- *tlbp = tlb_gather_mmu(vma->vm_mm, fullmm);
- tlb_start_valid = 0;
zap_work = ZAP_BLOCK_SIZE;
}
}
out:
+ tlb_flush_mmu(tlb);
return start; /* which is now the end (or restart) address */
}
@@ -879,16 +876,15 @@ unsigned long zap_page_range(struct vm_a
unsigned long size, struct zap_details *details)
{
struct mm_struct *mm = vma->vm_mm;
- struct mmu_gather *tlb;
+ struct mmu_gather tlb;
unsigned long end = address + size;
unsigned long nr_accounted = 0;
lru_add_drain();
- tlb = tlb_gather_mmu(mm, 0);
+ tlb_gather_mmu(&tlb, mm);
update_hiwater_rss(mm);
end = unmap_vmas(&tlb, vma, address, end, &nr_accounted, details);
- if (tlb)
- tlb_finish_mmu(tlb, address, end);
+ tlb_finish_mmu(&tlb);
return end;
}
Index: linux-work/mm/mmap.c
===================================================================
--- linux-work.orig/mm/mmap.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/mm/mmap.c 2007-08-07 16:23:53.000000000 +1000
@@ -1733,17 +1733,17 @@ static void unmap_region(struct mm_struc
unsigned long start, unsigned long end)
{
struct vm_area_struct *next = prev? prev->vm_next: mm->mmap;
- struct mmu_gather *tlb;
+ struct mmu_gather tlb;
unsigned long nr_accounted = 0;
lru_add_drain();
- tlb = tlb_gather_mmu(mm, 0);
+ tlb_gather_mmu(&tlb, mm);
update_hiwater_rss(mm);
unmap_vmas(&tlb, vma, start, end, &nr_accounted, NULL);
vm_unacct_memory(nr_accounted);
free_pgtables(&tlb, vma, prev? prev->vm_end: FIRST_USER_ADDRESS,
next? next->vm_start: 0);
- tlb_finish_mmu(tlb, start, end);
+ tlb_finish_mmu(&tlb);
}
/*
@@ -2020,7 +2020,7 @@ EXPORT_SYMBOL(do_brk);
/* Release all mmaps. */
void exit_mmap(struct mm_struct *mm)
{
- struct mmu_gather *tlb;
+ struct mmu_gather tlb;
struct vm_area_struct *vma = mm->mmap;
unsigned long nr_accounted = 0;
unsigned long end;
@@ -2031,15 +2031,17 @@ void exit_mmap(struct mm_struct *mm)
/* mm's last user has gone, and its about to be pulled down */
arch_exit_mmap(mm);
+ __set_bit(MMF_DEAD, &mm->flags);
lru_add_drain();
flush_cache_mm(mm);
- tlb = tlb_gather_mmu(mm, 1);
+ tlb_gather_mmu(&tlb, mm);
+
/* Don't update_hiwater_rss(mm) here, do_exit already did */
/* Use -1 here to ensure all VMAs in the mm are unmapped */
end = unmap_vmas(&tlb, vma, 0, -1, &nr_accounted, NULL);
vm_unacct_memory(nr_accounted);
free_pgtables(&tlb, vma, FIRST_USER_ADDRESS, 0);
- tlb_finish_mmu(tlb, 0, end);
+ tlb_finish_mmu(&tlb);
/*
* Walk the list again, actually closing and freeing it,
Index: linux-work/arch/powerpc/mm/hugetlbpage.c
===================================================================
--- linux-work.orig/arch/powerpc/mm/hugetlbpage.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/powerpc/mm/hugetlbpage.c 2007-08-07 16:23:53.000000000 +1000
@@ -240,7 +240,7 @@ static void hugetlb_free_pud_range(struc
*
* Must be called with pagetable lock held.
*/
-void hugetlb_free_pgd_range(struct mmu_gather **tlb,
+void hugetlb_free_pgd_range(struct mmu_gather *tlb,
unsigned long addr, unsigned long end,
unsigned long floor, unsigned long ceiling)
{
@@ -300,13 +300,13 @@ void hugetlb_free_pgd_range(struct mmu_g
return;
start = addr;
- pgd = pgd_offset((*tlb)->mm, addr);
+ pgd = pgd_offset(tlb->mm, addr);
do {
- BUG_ON(get_slice_psize((*tlb)->mm, addr) != mmu_huge_psize);
+ BUG_ON(get_slice_psize(tlb->mm, addr) != mmu_huge_psize);
next = pgd_addr_end(addr, end);
if (pgd_none_or_clear_bad(pgd))
continue;
- hugetlb_free_pud_range(*tlb, pgd, addr, next, floor, ceiling);
+ hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling);
} while (pgd++, addr = next, addr != end);
}
Index: linux-work/fs/exec.c
===================================================================
--- linux-work.orig/fs/exec.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/fs/exec.c 2007-08-07 16:23:53.000000000 +1000
@@ -525,7 +525,7 @@ static int shift_arg_pages(struct vm_are
unsigned long length = old_end - old_start;
unsigned long new_start = old_start - shift;
unsigned long new_end = old_end - shift;
- struct mmu_gather *tlb;
+ struct mmu_gather tlb;
BUG_ON(new_start > new_end);
@@ -550,7 +550,7 @@ static int shift_arg_pages(struct vm_are
return -ENOMEM;
lru_add_drain();
- tlb = tlb_gather_mmu(mm, 0);
+ tlb_gather_mmu(&tlb, mm);
if (new_end > old_start) {
/*
* when the old and new regions overlap clear from new_end.
@@ -567,7 +567,7 @@ static int shift_arg_pages(struct vm_are
free_pgd_range(&tlb, old_start, old_end, new_end,
vma->vm_next ? vma->vm_next->vm_start : 0);
}
- tlb_finish_mmu(tlb, new_end, old_end);
+ tlb_finish_mmu(&tlb);
/*
* shrink the vma to just the new range.
Index: linux-work/include/linux/hugetlb.h
===================================================================
--- linux-work.orig/include/linux/hugetlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/linux/hugetlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -56,7 +56,7 @@ void hugetlb_change_protection(struct vm
#ifndef ARCH_HAS_HUGETLB_FREE_PGD_RANGE
#define hugetlb_free_pgd_range free_pgd_range
#else
-void hugetlb_free_pgd_range(struct mmu_gather **tlb, unsigned long addr,
+void hugetlb_free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
unsigned long end, unsigned long floor,
unsigned long ceiling);
#endif
Index: linux-work/include/linux/mm.h
===================================================================
--- linux-work.orig/include/linux/mm.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/linux/mm.h 2007-08-07 16:23:53.000000000 +1000
@@ -769,13 +769,13 @@ struct zap_details {
struct page *vm_normal_page(struct vm_area_struct *, unsigned long, pte_t);
unsigned long zap_page_range(struct vm_area_struct *vma, unsigned long address,
unsigned long size, struct zap_details *);
-unsigned long unmap_vmas(struct mmu_gather **tlb,
+unsigned long unmap_vmas(struct mmu_gather *tlb,
struct vm_area_struct *start_vma, unsigned long start_addr,
unsigned long end_addr, unsigned long *nr_accounted,
struct zap_details *);
-void free_pgd_range(struct mmu_gather **tlb, unsigned long addr,
+void free_pgd_range(struct mmu_gather *tlb, unsigned long addr,
unsigned long end, unsigned long floor, unsigned long ceiling);
-void free_pgtables(struct mmu_gather **tlb, struct vm_area_struct *start_vma,
+void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
struct vm_area_struct *vma);
Index: linux-work/arch/i386/mm/init.c
===================================================================
--- linux-work.orig/arch/i386/mm/init.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/i386/mm/init.c 2007-08-07 16:23:53.000000000 +1000
@@ -47,7 +47,7 @@
unsigned int __VMALLOC_RESERVE = 128 << 20;
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
unsigned long highstart_pfn, highend_pfn;
static int noinline do_test_wp_bit(void);
Index: linux-work/arch/powerpc/mm/init_32.c
===================================================================
--- linux-work.orig/arch/powerpc/mm/init_32.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/powerpc/mm/init_32.c 2007-08-07 16:23:53.000000000 +1000
@@ -55,7 +55,7 @@
#endif
#define MAX_LOW_MEM CONFIG_LOWMEM_SIZE
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
unsigned long total_memory;
unsigned long total_lowmem;
Index: linux-work/arch/x86_64/mm/init.c
===================================================================
--- linux-work.orig/arch/x86_64/mm/init.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/x86_64/mm/init.c 2007-08-07 16:23:53.000000000 +1000
@@ -53,7 +53,7 @@ EXPORT_SYMBOL(dma_ops);
static unsigned long dma_reserve __initdata;
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
/*
* NOTE: pagetable_init alloc all the fixmap pagetables contiguous on the
Index: linux-work/include/asm-i386/tlb.h
===================================================================
--- linux-work.orig/include/asm-i386/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-i386/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -2,11 +2,8 @@
#define _I386_TLB_H
/*
- * x86 doesn't need any special per-pte or
- * per-vma handling..
+ * x86 doesn't need any special per-pte batch handling..
*/
-#define tlb_start_vma(tlb, vma) do { } while (0)
-#define tlb_end_vma(tlb, vma) do { } while (0)
#define __tlb_remove_tlb_entry(tlb, ptep, address) do { } while (0)
/*
Index: linux-work/arch/avr32/mm/init.c
===================================================================
--- linux-work.orig/arch/avr32/mm/init.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/avr32/mm/init.c 2007-08-07 16:23:53.000000000 +1000
@@ -23,7 +23,7 @@
#include <asm/setup.h>
#include <asm/sections.h>
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
pgd_t swapper_pg_dir[PTRS_PER_PGD];
Index: linux-work/arch/sparc/mm/init.c
===================================================================
--- linux-work.orig/arch/sparc/mm/init.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/sparc/mm/init.c 2007-08-07 16:23:53.000000000 +1000
@@ -32,7 +32,7 @@
#include <asm/tlb.h>
#include <asm/prom.h>
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
unsigned long *sparc_valid_addr_bitmap;
Index: linux-work/arch/sparc64/mm/tlb.c
===================================================================
--- linux-work.orig/arch/sparc64/mm/tlb.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/sparc64/mm/tlb.c 2007-08-07 16:23:53.000000000 +1000
@@ -19,7 +19,7 @@
/* Heavily inspired by the ppc64 code. */
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
DEFINE_PER_CPU(struct tlb_batch, tlb_batch);
void __flush_tlb_pending(struct tlb_batch *mp)
Index: linux-work/arch/um/kernel/smp.c
===================================================================
--- linux-work.orig/arch/um/kernel/smp.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/um/kernel/smp.c 2007-08-07 16:23:53.000000000 +1000
@@ -8,7 +8,7 @@
#include "asm/tlb.h"
/* For some reason, mmu_gathers are referenced when CONFIG_SMP is off. */
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
#ifdef CONFIG_SMP
Index: linux-work/arch/xtensa/mm/init.c
===================================================================
--- linux-work.orig/arch/xtensa/mm/init.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/xtensa/mm/init.c 2007-08-07 16:23:53.000000000 +1000
@@ -38,7 +38,7 @@
#define DEBUG 0
-DEFINE_PER_CPU(struct mmu_gather, mmu_gathers);
+DEFINE_PER_CPU(struct mmu_gather_store, mmu_gather_store);
//static DEFINE_SPINLOCK(tlb_lock);
/*
Index: linux-work/include/asm-sparc64/tlb.h
===================================================================
--- linux-work.orig/include/asm-sparc64/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-sparc64/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -52,7 +52,4 @@ extern void smp_flush_tlb_mm(struct mm_s
#define __pte_free_tlb(mp,ptepage,address) pte_free((mp)->mm,ptepage)
#define __pmd_free_tlb(mp,pmdp,address) pmd_free((mp)->mm,pmdp)
-#define tlb_start_vma(tlb, vma) do { } while (0)
-#define tlb_end_vma(tlb, vma) do { } while (0)
-
#endif /* _SPARC64_TLB_H */
Index: linux-work/include/asm-x86_64/tlb.h
===================================================================
--- linux-work.orig/include/asm-x86_64/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-x86_64/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -2,8 +2,6 @@
#define TLB_H 1
-#define tlb_start_vma(tlb, vma) do { } while (0)
-#define tlb_end_vma(tlb, vma) do { } while (0)
#define __tlb_remove_tlb_entry(tlb, ptep, address) do { } while (0)
#define tlb_flush(tlb) flush_tlb_mm((tlb)->mm)
Index: linux-work/include/asm-ia64/pgalloc.h
===================================================================
--- linux-work.orig/include/asm-ia64/pgalloc.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-ia64/pgalloc.h 2007-08-07 16:23:53.000000000 +1000
@@ -48,7 +48,6 @@ static inline void pud_free(struct mm_st
{
quicklist_free(0, NULL, pud);
}
-#define __pud_free_tlb(tlb, pud, address) pud_free((tlb)->mm, pud)
#endif /* CONFIG_PGTABLE_4 */
static inline void
@@ -67,7 +66,6 @@ static inline void pmd_free(struct mm_st
quicklist_free(0, NULL, pmd);
}
-#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)
static inline void
pmd_populate(struct mm_struct *mm, pmd_t * pmd_entry, struct page *pte)
@@ -109,6 +107,5 @@ static inline void check_pgt_cache(void)
quicklist_trim(0, NULL, 25, 16);
}
-#define __pte_free_tlb(tlb, pte, address) pte_free((tlb)->mm, pte)
#endif /* _ASM_IA64_PGALLOC_H */
Index: linux-work/include/asm-ia64/tlb.h
===================================================================
--- linux-work.orig/include/asm-ia64/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-ia64/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -46,51 +46,30 @@
#include <asm/tlbflush.h>
#include <asm/machvec.h>
-#ifdef CONFIG_SMP
-# define FREE_PTE_NR 2048
-# define tlb_fast_mode(tlb) ((tlb)->nr == ~0U)
-#else
-# define FREE_PTE_NR 0
-# define tlb_fast_mode(tlb) (1)
-#endif
-
-struct mmu_gather {
- struct mm_struct *mm;
- unsigned int nr; /* == ~0U => fast mode */
- unsigned char fullmm; /* non-zero means full mm flush */
- unsigned char need_flush; /* really unmapped some PTEs? */
+struct mmu_gather_arch {
unsigned long start_addr;
unsigned long end_addr;
unsigned long start_pgtable;
unsigned long end_pgtable;
- struct page *pages[FREE_PTE_NR];
};
+#define mmu_gather_arch mmu_gather_arch
-/* Users of the generic TLB shootdown code must declare this storage space. */
-DECLARE_PER_CPU(struct mmu_gather, mmu_gathers);
-/*
- * Flush the TLB for address range START to END and, if not in fast mode, release the
+/* Flush the TLB for address range START to END and, if not in fast mode, release the
* freed pages that where gathered up to this point.
*/
-static inline void
-ia64_tlb_flush_mmu (struct mmu_gather *tlb)
+static inline void __tlb_flush(struct mmu_gather_arch *tlba, struct mm_struct *mm)
{
- unsigned long start = tlb->start_addr;
- unsigned long end = tlb->end_addr;
- unsigned int nr;
-
- if (!tlb->need_flush)
- return;
- tlb->need_flush = 0;
+ unsigned long start = tlba->start_addr;
+ unsigned long end = tlba->end_addr;
- if (tlb->fullmm) {
+ if (test_bit(MMF_DEAD, &mm->flags)) {
/*
* Tearing down the entire address space. This happens both as a result
* of exit() and execve(). The latter case necessitates the call to
* flush_tlb_mm() here.
*/
- flush_tlb_mm(tlb->mm);
+ flush_tlb_mm(mm);
} else if (unlikely (end - start >= 1024*1024*1024*1024UL
|| REGION_NUMBER(start) != REGION_NUMBER(end - 1)))
{
@@ -104,138 +83,65 @@ ia64_tlb_flush_mmu (struct mmu_gather *t
/*
* XXX fix me: flush_tlb_range() should take an mm pointer instead of a
* vma pointer.
+ *
+ * Will fix that once flush_tlb_range() is no more a generic hook, as
+ * soon as the batch has been generalized. --BenH.
*/
struct vm_area_struct vma;
- vma.vm_mm = tlb->mm;
+ vma.vm_mm = mm;
+
/* flush the address range from the tlb: */
flush_tlb_range(&vma, start, end);
+
/* now flush the virt. page-table area mapping the address range: */
- if (tlb->start_pgtable < tlb->end_pgtable)
+ if (tlba->start_pgtable < tlba->end_pgtable)
flush_tlb_range(&vma,
- ia64_thash(tlb->start_pgtable),
- ia64_thash(tlb->end_pgtable));
+ ia64_thash(tlba->start_pgtable),
+ ia64_thash(tlba->end_pgtable));
}
- /* lastly, release the freed pages */
- nr = tlb->nr;
- if (!tlb_fast_mode(tlb)) {
- unsigned long i;
- tlb->nr = 0;
- tlb->start_addr = tlb->start_pgtable = ~0UL;
- for (i = 0; i < nr; ++i)
- free_page_and_swap_cache(tlb->pages[i]);
- }
+ tlba->start_addr = tlba->start_pgtable = ~0UL;
}
-/*
- * Return a pointer to an initialized struct mmu_gather.
- */
-static inline struct mmu_gather *
-tlb_gather_mmu (struct mm_struct *mm, unsigned int full_mm_flush)
+static inline void __tlb_arch_init(struct mmu_gather_arch *tlba)
{
- struct mmu_gather *tlb = &get_cpu_var(mmu_gathers);
-
- tlb->mm = mm;
- /*
- * Use fast mode if only 1 CPU is online.
- *
- * It would be tempting to turn on fast-mode for full_mm_flush as well. But this
- * doesn't work because of speculative accesses and software prefetching: the page
- * table of "mm" may (and usually is) the currently active page table and even
- * though the kernel won't do any user-space accesses during the TLB shoot down, a
- * compiler might use speculation or lfetch.fault on what happens to be a valid
- * user-space address. This in turn could trigger a TLB miss fault (or a VHPT
- * walk) and re-insert a TLB entry we just removed. Slow mode avoids such
- * problems. (We could make fast-mode work by switching the current task to a
- * different "mm" during the shootdown.) --davidm 08/02/2002
- */
- tlb->nr = (num_online_cpus() == 1) ? ~0U : 0;
- tlb->fullmm = full_mm_flush;
- tlb->start_addr = tlb->start_pgtable = ~0UL;
- return tlb;
+ tlba->start_addr = tlba->start_pgtable = ~0UL;
}
-/*
- * Called at the end of the shootdown operation to free up any resources that were
- * collected.
- */
-static inline void
-tlb_finish_mmu (struct mmu_gather *tlb, unsigned long start, unsigned long end)
-{
- /*
- * Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and
- * tlb->end_addr.
- */
- ia64_tlb_flush_mmu(tlb);
-
- /* keep the page table cache within bounds */
- check_pgt_cache();
-
- put_cpu_var(mmu_gathers);
-}
-
-/*
- * Logically, this routine frees PAGE. On MP machines, the actual freeing of the page
- * must be delayed until after the TLB has been flushed (see comments at the beginning of
- * this file).
- */
-static inline void
-tlb_remove_page (struct mmu_gather *tlb, struct page *page)
-{
- tlb->need_flush = 1;
+#define tlb_flush(tlb) __tlb_flush(&tlb->archdata, tlb->mm)
+#define tlb_arch_init(tlb) __tlb_arch_init(&tlb->archdata)
+#define tlb_arch_finish(tlb) do { } while(0)
+#define tlb_migrate_finish(mm) platform_tlb_migrate_finish(mm)
- if (tlb_fast_mode(tlb)) {
- free_page_and_swap_cache(page);
- return;
- }
- tlb->pages[tlb->nr++] = page;
- if (tlb->nr >= FREE_PTE_NR)
- ia64_tlb_flush_mmu(tlb);
-}
+#include <asm-generic/tlb.h>
/*
- * Remove TLB entry for PTE mapped at virtual address ADDRESS. This is called for any
- * PTE, not just those pointing to (normal) physical memory.
+ * Remove TLB entry for PTE mapped at virtual address ADDRESS.
+ * This is called for any PTE, not just those pointing to (normal)
+ * physical memory.
*/
-static inline void
-__tlb_remove_tlb_entry (struct mmu_gather *tlb, pte_t *ptep, unsigned long address)
+static inline void __tlb_remove_tlb_entry (struct mmu_gather *tlb, pte_t *ptep,
+ unsigned long address)
{
- if (tlb->start_addr > address)
- tlb->start_addr = address;
- tlb->end_addr = address + PAGE_SIZE;
+ if (tlb->archdata.start_addr > address)
+ tlb->archdata.start_addr = address;
+ tlb->archdata.end_addr = address + PAGE_SIZE;
}
-#define tlb_migrate_finish(mm) platform_tlb_migrate_finish(mm)
-
-#define tlb_start_vma(tlb, vma) do { } while (0)
-#define tlb_end_vma(tlb, vma) do { } while (0)
-#define tlb_remove_tlb_entry(tlb, ptep, addr) \
-do { \
- tlb->need_flush = 1; \
- __tlb_remove_tlb_entry(tlb, ptep, addr); \
+#define __pte_free_tlb(tlb, ptep, addr) \
+do { \
+ if (tlb->archdata.start_pgtable > addr) \
+ tlb->archdata.start_pgtable = addr; \
+ tlb->archdata.end_pgtable = (addr + PMD_SIZE) & PMD_MASK; \
+ pte_free((tlb)->mm, ptep); \
} while (0)
-#define pte_free_tlb(tlb, ptep, addr) \
-do { \
- tlb->need_flush = 1; \
- if (tlb->start_pgtable > addr) \
- tlb->start_pgtable = addr; \
- tlb->end_pgtable = (addr + PMD_SIZE) & PMD_MASK;\
- __pte_free_tlb(tlb, ptep, addr); \
-} while (0)
+#define __pmd_free_tlb(tlb, pmd, address) pmd_free((tlb)->mm, pmd)
-#define pmd_free_tlb(tlb, ptep, addr) \
-do { \
- tlb->need_flush = 1; \
- __pmd_free_tlb(tlb, ptep, addr); \
-} while (0)
-
-#define pud_free_tlb(tlb, pudp, addr) \
-do { \
- tlb->need_flush = 1; \
- __pud_free_tlb(tlb, pudp, addr); \
-} while (0)
+#ifdef CONFIG_PGTABLE_4
+#define __pud_free_tlb(tlb, pud, address) pud_free((tlb)->mm, pud)
+#endif
#endif /* _ASM_IA64_TLB_H */
Index: linux-work/include/asm-parisc/tlb.h
===================================================================
--- linux-work.orig/include/asm-parisc/tlb.h 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/include/asm-parisc/tlb.h 2007-08-07 16:23:53.000000000 +1000
@@ -1,18 +1,18 @@
#ifndef _PARISC_TLB_H
#define _PARISC_TLB_H
-#define tlb_flush(tlb) \
-do { if ((tlb)->fullmm) \
- flush_tlb_mm((tlb)->mm);\
+#define tlb_flush(tlb) \
+do { if (test_bit(MMF_DEAD, &(tlb)->mm->flags)) \
+ flush_tlb_mm((tlb)->mm); \
} while (0)
#define tlb_start_vma(tlb, vma) \
-do { if (!(tlb)->fullmm) \
+do { if (!test_bit(MMF_DEAD, &(tlb)->mm->flags)) \
flush_cache_range(vma, vma->vm_start, vma->vm_end); \
} while (0)
#define tlb_end_vma(tlb, vma) \
-do { if (!(tlb)->fullmm) \
+do { if (!test_bit(MMF_DEAD, &(tlb)->mm->flags)) \
flush_tlb_range(vma, vma->vm_start, vma->vm_end); \
} while (0)
Index: linux-work/arch/ia64/mm/hugetlbpage.c
===================================================================
--- linux-work.orig/arch/ia64/mm/hugetlbpage.c 2007-08-07 16:18:13.000000000 +1000
+++ linux-work/arch/ia64/mm/hugetlbpage.c 2007-08-07 16:23:53.000000000 +1000
@@ -114,7 +114,7 @@ follow_huge_pmd(struct mm_struct *mm, un
return NULL;
}
-void hugetlb_free_pgd_range(struct mmu_gather **tlb,
+void hugetlb_free_pgd_range(struct mmu_gather *tlb,
unsigned long addr, unsigned long end,
unsigned long floor, unsigned long ceiling)
{
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 10/12] remove call to flush_tlb_page() from handle_pte_fault()
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (9 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 11/12] Use mmu_gather for fork() instead of flush_tlb_mm() Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 12/12] Use mmu_gather for fs/proc/task_mmu.c Benjamin Herrenschmidt
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
I've asked several times what archs need that, got not reply, so
let's remove it. If an arch needs it, it should be done by that
arch implementation of ptep_set_access_flags() anyway.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
mm/memory.c | 9 ---------
1 file changed, 9 deletions(-)
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-07 16:23:53.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-07 16:27:30.000000000 +1000
@@ -2605,15 +2605,6 @@ static inline int handle_pte_fault(struc
if (ptep_set_access_flags(vma, address, pte, entry, write_access)) {
update_mmu_cache(vma, address, entry);
lazy_mmu_prot_update(entry);
- } else {
- /*
- * This is needed only for protection faults but the arch code
- * is not yet telling us if this is a protection fault or not.
- * This still avoids useless tlb flushes for .text page faults
- * with threads.
- */
- if (write_access)
- flush_tlb_page(vma, address);
}
unlock:
pte_unmap_unlock(pte, ptl);
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 11/12] Use mmu_gather for fork() instead of flush_tlb_mm()
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (8 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 9/12] mmu_gather on stack, part 1 Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 10/12] remove call to flush_tlb_page() from handle_pte_fault() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 12/12] Use mmu_gather for fs/proc/task_mmu.c Benjamin Herrenschmidt
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This patch uses an mmu_gather for copying page tables instead of
flush_tlb_mm(). This allows archs like ppc32 with hash table to
avoid walking the page tables a second time to invalidate hash
entries, and to only flush PTEs that have actually been changed
from RW to RO.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
include/linux/hugetlb.h | 4 ++--
include/linux/mm.h | 4 ++--
kernel/fork.c | 10 ++++++++--
mm/hugetlb.c | 11 ++++++++---
mm/memory.c | 45 ++++++++++++++++++++++++++++-----------------
5 files changed, 48 insertions(+), 26 deletions(-)
Index: linux-work/include/linux/hugetlb.h
===================================================================
--- linux-work.orig/include/linux/hugetlb.h 2007-08-07 16:23:53.000000000 +1000
+++ linux-work/include/linux/hugetlb.h 2007-08-07 16:51:37.000000000 +1000
@@ -18,7 +18,7 @@ static inline int is_vm_hugetlb_page(str
int hugetlb_sysctl_handler(struct ctl_table *, int, struct file *, void __user *, size_t *, loff_t *);
int hugetlb_treat_movable_handler(struct ctl_table *, int, struct file *, void __user *, size_t *, loff_t *);
-int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *);
+int copy_hugetlb_page_range(struct mmu_gather *tlb, struct mm_struct *, struct mm_struct *, struct vm_area_struct *);
int follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, struct vm_area_struct **, unsigned long *, int *, int);
void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
void __unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long);
@@ -111,7 +111,7 @@ static inline unsigned long hugetlb_tota
#define follow_hugetlb_page(m,v,p,vs,a,b,i) ({ BUG(); 0; })
#define follow_huge_addr(mm, addr, write) ERR_PTR(-EINVAL)
-#define copy_hugetlb_page_range(src, dst, vma) ({ BUG(); 0; })
+#define copy_hugetlb_page_range(tlb, src, dst, vma) ({ BUG(); 0; })
#define hugetlb_prefault(mapping, vma) ({ BUG(); 0; })
#define unmap_hugepage_range(vma, start, end) BUG()
#define hugetlb_report_meminfo(buf) 0
Index: linux-work/include/linux/mm.h
===================================================================
--- linux-work.orig/include/linux/mm.h 2007-08-07 16:23:53.000000000 +1000
+++ linux-work/include/linux/mm.h 2007-08-07 16:48:03.000000000 +1000
@@ -777,8 +777,8 @@ void free_pgd_range(struct mmu_gather *t
unsigned long end, unsigned long floor, unsigned long ceiling);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
unsigned long floor, unsigned long ceiling);
-int copy_page_range(struct mm_struct *dst, struct mm_struct *src,
- struct vm_area_struct *vma);
+int copy_page_range(struct mmu_gather *tlb, struct mm_struct *dst,
+ struct mm_struct *src, struct vm_area_struct *vma);
int zeromap_page_range(struct vm_area_struct *vma, unsigned long from,
unsigned long size, pgprot_t prot);
void unmap_mapping_range(struct address_space *mapping,
Index: linux-work/kernel/fork.c
===================================================================
--- linux-work.orig/kernel/fork.c 2007-08-07 16:07:08.000000000 +1000
+++ linux-work/kernel/fork.c 2007-08-07 16:52:32.000000000 +1000
@@ -57,6 +57,7 @@
#include <asm/mmu_context.h>
#include <asm/cacheflush.h>
#include <asm/tlbflush.h>
+#include <asm/tlb.h>
/*
* Protected counters by write_lock_irq(&tasklist_lock)
@@ -202,6 +203,7 @@ static inline int dup_mmap(struct mm_str
int retval;
unsigned long charge;
struct mempolicy *pol;
+ struct mmu_gather tlb;
down_write(&oldmm->mmap_sem);
flush_cache_dup_mm(oldmm);
@@ -222,6 +224,8 @@ static inline int dup_mmap(struct mm_str
rb_parent = NULL;
pprev = &mm->mmap;
+ tlb_gather_mmu(&tlb, oldmm);
+
for (mpnt = oldmm->mmap; mpnt; mpnt = mpnt->vm_next) {
struct file *file;
@@ -243,6 +247,7 @@ static inline int dup_mmap(struct mm_str
if (!tmp)
goto fail_nomem;
*tmp = *mpnt;
+
pol = mpol_copy(vma_policy(mpnt));
retval = PTR_ERR(pol);
if (IS_ERR(pol))
@@ -279,7 +284,7 @@ static inline int dup_mmap(struct mm_str
rb_parent = &tmp->vm_rb;
mm->map_count++;
- retval = copy_page_range(mm, oldmm, mpnt);
+ retval = copy_page_range(&tlb, mm, oldmm, mpnt);
if (tmp->vm_ops && tmp->vm_ops->open)
tmp->vm_ops->open(tmp);
@@ -292,12 +297,13 @@ static inline int dup_mmap(struct mm_str
retval = 0;
out:
up_write(&mm->mmap_sem);
- flush_tlb_mm(oldmm);
+ tlb_finish_mmu(&tlb);
up_write(&oldmm->mmap_sem);
return retval;
fail_nomem_policy:
kmem_cache_free(vm_area_cachep, tmp);
fail_nomem:
+ tlb_finish_mmu(&tlb);
retval = -ENOMEM;
vm_unacct_memory(charge);
goto out;
Index: linux-work/mm/hugetlb.c
===================================================================
--- linux-work.orig/mm/hugetlb.c 2007-08-06 13:48:31.000000000 +1000
+++ linux-work/mm/hugetlb.c 2007-08-07 16:53:49.000000000 +1000
@@ -17,6 +17,8 @@
#include <asm/page.h>
#include <asm/pgtable.h>
+#include <asm/tlbflush.h>
+#include <asm/tlb.h>
#include <linux/hugetlb.h>
#include "internal.h"
@@ -358,8 +360,8 @@ static void set_huge_ptep_writable(struc
}
-int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
- struct vm_area_struct *vma)
+int copy_hugetlb_page_range(struct mmu_gather *tlb, struct mm_struct *dst,
+ struct mm_struct *src, struct vm_area_struct *vma)
{
pte_t *src_pte, *dst_pte, entry;
struct page *ptepage;
@@ -378,8 +380,10 @@ int copy_hugetlb_page_range(struct mm_st
spin_lock(&dst->page_table_lock);
spin_lock(&src->page_table_lock);
if (!pte_none(*src_pte)) {
- if (cow)
+ if (cow) {
ptep_set_wrprotect(src, addr, src_pte);
+ tlb_remove_tlb_entry(tlb, src_pte, addr);
+ }
entry = *src_pte;
ptepage = pte_page(entry);
get_page(ptepage);
@@ -388,6 +392,7 @@ int copy_hugetlb_page_range(struct mm_st
spin_unlock(&src->page_table_lock);
spin_unlock(&dst->page_table_lock);
}
+
return 0;
nomem:
Index: linux-work/mm/memory.c
===================================================================
--- linux-work.orig/mm/memory.c 2007-08-07 16:27:30.000000000 +1000
+++ linux-work/mm/memory.c 2007-08-07 16:50:03.000000000 +1000
@@ -430,9 +430,9 @@ struct page *vm_normal_page(struct vm_ar
*/
static inline void
-copy_one_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
- pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *vma,
- unsigned long addr, int *rss)
+copy_one_pte(struct mmu_gather *tlb, struct mm_struct *dst_mm,
+ struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte,
+ struct vm_area_struct *vma, unsigned long addr, int *rss)
{
unsigned long vm_flags = vma->vm_flags;
pte_t pte = *src_pte;
@@ -471,8 +471,11 @@ copy_one_pte(struct mm_struct *dst_mm, s
* in the parent and the child
*/
if (is_cow_mapping(vm_flags)) {
+ pte_t old = *src_pte;
ptep_set_wrprotect(src_mm, addr, src_pte);
pte = pte_wrprotect(pte);
+ if (tlb && !pte_same(old, *src_pte))
+ tlb_remove_tlb_entry(tlb, src_pte, addr);
}
/*
@@ -494,12 +497,14 @@ out_set_pte:
set_pte_at(dst_mm, addr, dst_pte, pte);
}
-static int copy_pte_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+static int copy_pte_range(struct mmu_gather *tlb,
+ struct mm_struct *dst_mm, struct mm_struct *src_mm,
pmd_t *dst_pmd, pmd_t *src_pmd, struct vm_area_struct *vma,
unsigned long addr, unsigned long end)
{
pte_t *src_pte, *dst_pte;
spinlock_t *src_ptl, *dst_ptl;
+ unsigned long start_addr = addr;
int progress = 0;
int rss[2];
@@ -529,22 +534,27 @@ again:
progress++;
continue;
}
- copy_one_pte(dst_mm, src_mm, dst_pte, src_pte, vma, addr, rss);
+ copy_one_pte(tlb, dst_mm, src_mm, dst_pte, src_pte,
+ vma, addr, rss);
progress += 8;
} while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end);
arch_leave_lazy_mmu_mode();
+ tlb_pte_lock_break(tlb);
spin_unlock(src_ptl);
pte_unmap_nested(src_pte - 1);
add_mm_rss(dst_mm, rss[0], rss[1]);
pte_unmap_unlock(dst_pte - 1, dst_ptl);
cond_resched();
- if (addr != end)
+ if (addr != end) {
+ start_addr = addr;
goto again;
+ }
return 0;
}
-static inline int copy_pmd_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+static inline int copy_pmd_range(struct mmu_gather *tlb,
+ struct mm_struct *dst_mm, struct mm_struct *src_mm,
pud_t *dst_pud, pud_t *src_pud, struct vm_area_struct *vma,
unsigned long addr, unsigned long end)
{
@@ -559,14 +569,15 @@ static inline int copy_pmd_range(struct
next = pmd_addr_end(addr, end);
if (pmd_none_or_clear_bad(src_pmd))
continue;
- if (copy_pte_range(dst_mm, src_mm, dst_pmd, src_pmd,
- vma, addr, next))
+ if (copy_pte_range(tlb, dst_mm, src_mm, dst_pmd, src_pmd,
+ vma, addr, next))
return -ENOMEM;
} while (dst_pmd++, src_pmd++, addr = next, addr != end);
return 0;
}
-static inline int copy_pud_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
+static inline int copy_pud_range(struct mmu_gather *tlb,
+ struct mm_struct *dst_mm, struct mm_struct *src_mm,
pgd_t *dst_pgd, pgd_t *src_pgd, struct vm_area_struct *vma,
unsigned long addr, unsigned long end)
{
@@ -581,15 +592,15 @@ static inline int copy_pud_range(struct
next = pud_addr_end(addr, end);
if (pud_none_or_clear_bad(src_pud))
continue;
- if (copy_pmd_range(dst_mm, src_mm, dst_pud, src_pud,
- vma, addr, next))
+ if (copy_pmd_range(tlb, dst_mm, src_mm, dst_pud, src_pud,
+ vma, addr, next))
return -ENOMEM;
} while (dst_pud++, src_pud++, addr = next, addr != end);
return 0;
}
-int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
- struct vm_area_struct *vma)
+int copy_page_range(struct mmu_gather *tlb, struct mm_struct *dst_mm,
+ struct mm_struct *src_mm, struct vm_area_struct *vma)
{
pgd_t *src_pgd, *dst_pgd;
unsigned long next;
@@ -608,7 +619,7 @@ int copy_page_range(struct mm_struct *ds
}
if (is_vm_hugetlb_page(vma))
- return copy_hugetlb_page_range(dst_mm, src_mm, vma);
+ return copy_hugetlb_page_range(tlb, dst_mm, src_mm, vma);
dst_pgd = pgd_offset(dst_mm, addr);
src_pgd = pgd_offset(src_mm, addr);
@@ -616,8 +627,8 @@ int copy_page_range(struct mm_struct *ds
next = pgd_addr_end(addr, end);
if (pgd_none_or_clear_bad(src_pgd))
continue;
- if (copy_pud_range(dst_mm, src_mm, dst_pgd, src_pgd,
- vma, addr, next))
+ if (copy_pud_range(tlb, dst_mm, src_mm, dst_pgd, src_pgd,
+ vma, addr, next))
return -ENOMEM;
} while (dst_pgd++, src_pgd++, addr = next, addr != end);
return 0;
^ permalink raw reply [flat|nested] 13+ messages in thread
* [RFC/PATCH 12/12] Use mmu_gather for fs/proc/task_mmu.c
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
` (10 preceding siblings ...)
2007-08-07 7:19 ` [RFC/PATCH 10/12] remove call to flush_tlb_page() from handle_pte_fault() Benjamin Herrenschmidt
@ 2007-08-07 7:19 ` Benjamin Herrenschmidt
11 siblings, 0 replies; 13+ messages in thread
From: Benjamin Herrenschmidt @ 2007-08-07 7:19 UTC (permalink / raw)
To: Linux Memory Management; +Cc: linux-kernel
This removes the use of flush_tlb_mm() from that proc file, using
an mmu_gather instead.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
fs/proc/task_mmu.c | 40 +++++++++++++++++++++++++---------------
1 file changed, 25 insertions(+), 15 deletions(-)
Index: linux-work/fs/proc/task_mmu.c
===================================================================
--- linux-work.orig/fs/proc/task_mmu.c 2007-08-06 13:48:30.000000000 +1000
+++ linux-work/fs/proc/task_mmu.c 2007-08-07 17:01:41.000000000 +1000
@@ -9,7 +9,7 @@
#include <asm/elf.h>
#include <asm/uaccess.h>
-#include <asm/tlbflush.h>
+#include <asm/tlb.h>
#include "internal.h"
char *task_mem(struct mm_struct *mm, char *buffer)
@@ -124,11 +124,13 @@ struct mem_size_stats
unsigned long referenced;
};
+typedef void (*pmd_action_t)(struct mmu_gather *tlb, struct vm_area_struct *,
+ pmd_t *, unsigned long, unsigned long, void *);
struct pmd_walker {
+ struct mmu_gather *tlb;
struct vm_area_struct *vma;
void *private;
- void (*action)(struct vm_area_struct *, pmd_t *, unsigned long,
- unsigned long, void *);
+ pmd_action_t action;
};
static int show_map_internal(struct seq_file *m, void *v, struct mem_size_stats *mss)
@@ -218,7 +220,8 @@ static int show_map(struct seq_file *m,
return show_map_internal(m, v, NULL);
}
-static void smaps_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
+static void smaps_pte_range(struct mmu_gather *tlb,
+ struct vm_area_struct *vma, pmd_t *pmd,
unsigned long addr, unsigned long end,
void *private)
{
@@ -258,7 +261,8 @@ static void smaps_pte_range(struct vm_ar
cond_resched();
}
-static void clear_refs_pte_range(struct vm_area_struct *vma, pmd_t *pmd,
+static void clear_refs_pte_range(struct mmu_gather *tlb,
+ struct vm_area_struct *vma, pmd_t *pmd,
unsigned long addr, unsigned long end,
void *private)
{
@@ -279,6 +283,7 @@ static void clear_refs_pte_range(struct
/* Clear accessed and referenced bits. */
ptep_test_and_clear_young(vma, addr, pte);
ClearPageReferenced(page);
+ tlb_remove_tlb_entry(tlb, pte, addr);
}
pte_unmap_unlock(pte - 1, ptl);
cond_resched();
@@ -295,7 +300,8 @@ static inline void walk_pmd_range(struct
next = pmd_addr_end(addr, end);
if (pmd_none_or_clear_bad(pmd))
continue;
- walker->action(walker->vma, pmd, addr, next, walker->private);
+ walker->action(walker->tlb, walker->vma, pmd, addr, next,
+ walker->private);
}
}
@@ -323,11 +329,9 @@ static inline void walk_pud_range(struct
* Recursively walk the page table for the memory area in a VMA, calling
* a callback for every bottom-level (PTE) page table.
*/
-static inline void walk_page_range(struct vm_area_struct *vma,
- void (*action)(struct vm_area_struct *,
- pmd_t *, unsigned long,
- unsigned long, void *),
- void *private)
+static inline void walk_page_range(struct mmu_gather *tlb,
+ struct vm_area_struct *vma,
+ pmd_action_t action, void *private)
{
unsigned long addr = vma->vm_start;
unsigned long end = vma->vm_end;
@@ -335,6 +339,7 @@ static inline void walk_page_range(struc
.vma = vma,
.private = private,
.action = action,
+ .tlb = tlb,
};
pgd_t *pgd;
unsigned long next;
@@ -355,19 +360,24 @@ static int show_smap(struct seq_file *m,
memset(&mss, 0, sizeof mss);
if (vma->vm_mm && !is_vm_hugetlb_page(vma))
- walk_page_range(vma, smaps_pte_range, &mss);
+ walk_page_range(NULL, vma, smaps_pte_range, &mss);
return show_map_internal(m, v, &mss);
}
void clear_refs_smap(struct mm_struct *mm)
{
struct vm_area_struct *vma;
+ struct mmu_gather tlb;
+ unsigned long end_addr = 0;
down_read(&mm->mmap_sem);
+ tlb_gather_mmu(&tlb, mm);
for (vma = mm->mmap; vma; vma = vma->vm_next)
- if (vma->vm_mm && !is_vm_hugetlb_page(vma))
- walk_page_range(vma, clear_refs_pte_range, NULL);
- flush_tlb_mm(mm);
+ if (vma->vm_mm && !is_vm_hugetlb_page(vma)) {
+ end_addr = max(vma->vm_end, end_addr);
+ walk_page_range(&tlb, vma, clear_refs_pte_range, NULL);
+ }
+ tlb_finish_mmu(&tlb);
up_read(&mm->mmap_sem);
}
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2007-08-07 7:24 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2007-08-07 7:19 [RFC/PATCH 0/12] WIP mmu_gather and PTE accessors work Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 1/12] remove frv usage of flush_tlb_pgtables() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 2/12] remove flush_tlb_pgtables Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 3/12] Add MMF_DEAD flag to mm_struct being torn down Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 4/12] disconnect sparc64 tlb flush from mmu_gather Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 5/12] Add mm argument to pte/pmd/pud/pgd_free Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 6/12] Add "address" argument to pte/pmd/pud_free_tlb() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 7/12] ia64 tracks freed page tables addresses Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 8/12] remove ptep_get_and_clear_full Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 9/12] mmu_gather on stack, part 1 Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 11/12] Use mmu_gather for fork() instead of flush_tlb_mm() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 10/12] remove call to flush_tlb_page() from handle_pte_fault() Benjamin Herrenschmidt
2007-08-07 7:19 ` [RFC/PATCH 12/12] Use mmu_gather for fs/proc/task_mmu.c Benjamin Herrenschmidt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).