[PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Catalin Marinas]

Hi,

This is version 3 of the set of patches removing
__ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
ARM LPAE patches. There are no functional changes from v2. I plan to
push this to -next and get it ready for 3.4-rc1.

Question for Peter/Ingo - how do we merge the first patch that
introduces finish_arch_post_lock_switch? Do you pick it up or I can
merge it via rmk (with your ack)?

Thanks,

Catalin


Catalin Marinas (4):
  sched: Introduce the finish_arch_post_lock_switch() scheduler hook
  ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on ASID-capable CPUs
  ARM: Remove current_mm per-cpu variable
  ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on pre-ARMv6 CPUs

Will Deacon (2):
  ARM: Use TTBR1 instead of reserved context ID
  ARM: Allow ASID 0 to be allocated to tasks

 arch/arm/include/asm/mmu_context.h |  104 +++++++++++++++++++++++++++---------
 arch/arm/include/asm/system.h      |    7 ---
 arch/arm/include/asm/thread_info.h |    1 +
 arch/arm/mm/context.c              |   63 +++++++++++-----------
 arch/arm/mm/proc-v7-2level.S       |    9 +---
 kernel/sched/core.c                |    1 +
 kernel/sched/sched.h               |    3 +
 7 files changed, 117 insertions(+), 71 deletions(-)

[PATCH v3 1/6] sched: Introduce the finish_arch_post_lock_switch() scheduler hook

[Catalin Marinas]

This hook is called by the scheduler after rq->lock has been released
and interrupts enabled. It will be used in subsequent patches on the ARM
architecture.

Reviewed-by: Will Deacon <will.deacon@arm.com>
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 kernel/sched/core.c  |    1 +
 kernel/sched/sched.h |    3 +++
 2 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index df00cb0..f1db6d8 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1937,6 +1937,7 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
 	local_irq_enable();
 #endif /* __ARCH_WANT_INTERRUPTS_ON_CTXSW */
 	finish_lock_switch(rq, prev);
+	finish_arch_post_lock_switch();
 	trace_sched_stat_sleeptime(current, rq->clock);
 
 	fire_sched_in_preempt_notifiers(current);
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index 98c0c26..d72483d 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -692,6 +692,9 @@ static inline int task_running(struct rq *rq, struct task_struct *p)
 #ifndef finish_arch_switch
 # define finish_arch_switch(prev)	do { } while (0)
 #endif
+#ifndef finish_arch_post_lock_switch
+# define finish_arch_post_lock_switch()	do { } while (0)
+#endif
 
 #ifndef __ARCH_WANT_UNLOCKED_CTXSW
 static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)

[PATCH v3 2/6] ARM: Use TTBR1 instead of reserved context ID

[Catalin Marinas]

From: Will Deacon <will.deacon@arm.com>

On ARMv7 CPUs that cache first level page table entries (like the
Cortex-A15), using a reserved ASID while changing the TTBR or flushing
the TLB is unsafe.

This is because the CPU may cache the first level entry as the result of
a speculative memory access while the reserved ASID is assigned. After
the process owning the page tables dies, the memory will be reallocated
and may be written with junk values which can be interpreted as global,
valid PTEs by the processor. This will result in the TLB being populated
with bogus global entries.

This patch avoids the use of a reserved context ID in the v7 switch_mm
and ASID rollover code by temporarily using the swapper_pg_dir pointed
at by TTBR1, which contains only global entries that are not tagged
with ASIDs.

Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/mm/context.c        |   45 +++++++++++++++++++++++++----------------
 arch/arm/mm/proc-v7-2level.S |   10 +++-----
 2 files changed, 31 insertions(+), 24 deletions(-)

diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c
index ee9bb36..97842a0 100644
--- a/arch/arm/mm/context.c
+++ b/arch/arm/mm/context.c
@@ -23,25 +23,37 @@ DEFINE_PER_CPU(struct mm_struct *, current_mm);
 #endif
 
 #ifdef CONFIG_ARM_LPAE
-#define cpu_set_asid(asid) {						\
-	unsigned long ttbl, ttbh;					\
-	asm volatile(							\
-	"	mrrc	p15, 0, %0, %1, c2		@ read TTBR0\n"	\
-	"	mov	%1, %2, lsl #(48 - 32)		@ set ASID\n"	\
-	"	mcrr	p15, 0, %0, %1, c2		@ set TTBR0\n"	\
-	: "=&r" (ttbl), "=&r" (ttbh)					\
-	: "r" (asid & ~ASID_MASK));					\
+static void cpu_set_reserved_ttbr0(void)
+{
+	unsigned long ttbl = __pa(swapper_pg_dir);
+	unsigned long ttbh = 0;
+
+	/*
+	 * Set TTBR0 to swapper_pg_dir. Note that swapper_pg_dir only contains
+	 * global entries so the ASID value is not relevant.
+	 */
+	asm volatile(
+	"	mcrr	p15, 0, %0, %1, c2		@ set TTBR0\n"
+	:
+	: "r" (ttbl), "r" (ttbh));
+	isb();
 }
 #else
-#define cpu_set_asid(asid) \
-	asm("	mcr	p15, 0, %0, c13, c0, 1\n" : : "r" (asid))
+static void cpu_set_reserved_ttbr0(void)
+{
+	u32 ttb;
+	/* Copy TTBR1 into TTBR0 */
+	asm volatile(
+	"	mrc	p15, 0, %0, c2, c0, 1		@ read TTBR1\n"
+	"	mcr	p15, 0, %0, c2, c0, 0		@ set TTBR0\n"
+	: "=r" (ttb));
+	isb();
+}
 #endif
 
 /*
  * We fork()ed a process, and we need a new context for the child
- * to run in.  We reserve version 0 for initial tasks so we will
- * always allocate an ASID. The ASID 0 is reserved for the TTBR
- * register changing sequence.
+ * to run in.
  */
 void __init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 {
@@ -51,9 +63,7 @@ void __init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 
 static void flush_context(void)
 {
-	/* set the reserved ASID before flushing the TLB */
-	cpu_set_asid(0);
-	isb();
+	cpu_set_reserved_ttbr0();
 	local_flush_tlb_all();
 	if (icache_is_vivt_asid_tagged()) {
 		__flush_icache_all();
@@ -114,8 +124,7 @@ static void reset_context(void *info)
 	set_mm_context(mm, asid);
 
 	/* set the new ASID */
-	cpu_set_asid(mm->context.id);
-	isb();
+	cpu_switch_mm(mm->pgd, mm);
 }
 
 #else
diff --git a/arch/arm/mm/proc-v7-2level.S b/arch/arm/mm/proc-v7-2level.S
index 3a4b3e7..7227048 100644
--- a/arch/arm/mm/proc-v7-2level.S
+++ b/arch/arm/mm/proc-v7-2level.S
@@ -46,18 +46,16 @@ ENTRY(cpu_v7_switch_mm)
 #ifdef CONFIG_ARM_ERRATA_430973
 	mcr	p15, 0, r2, c7, c5, 6		@ flush BTAC/BTB
 #endif
-#ifdef CONFIG_ARM_ERRATA_754322
-	dsb
-#endif
-	mcr	p15, 0, r2, c13, c0, 1		@ set reserved context ID
-	isb
-1:	mcr	p15, 0, r0, c2, c0, 0		@ set TTB 0
+	mrc	p15, 0, r2, c2, c0, 1		@ load TTB 1
+	mcr	p15, 0, r2, c2, c0, 0		@ into TTB 0
 	isb
 #ifdef CONFIG_ARM_ERRATA_754322
 	dsb
 #endif
 	mcr	p15, 0, r1, c13, c0, 1		@ set context ID
 	isb
+	mcr	p15, 0, r0, c2, c0, 0		@ set TTB 0
+	isb
 #endif
 	mov	pc, lr
 ENDPROC(cpu_v7_switch_mm)

[PATCH v3 3/6] ARM: Allow ASID 0 to be allocated to tasks

[Catalin Marinas]

From: Will Deacon <will.deacon@arm.com>

Now that ASID 0 is no longer used as a reserved value, allow it to be
allocated to tasks.

Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/mm/context.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c
index 97842a0..efa413a 100644
--- a/arch/arm/mm/context.c
+++ b/arch/arm/mm/context.c
@@ -118,7 +118,7 @@ static void reset_context(void *info)
 		return;
 
 	smp_rmb();
-	asid = cpu_last_asid + cpu + 1;
+	asid = cpu_last_asid + cpu;
 
 	flush_context();
 	set_mm_context(mm, asid);
@@ -167,13 +167,13 @@ void __new_context(struct mm_struct *mm)
 	 * to start a new version and flush the TLB.
 	 */
 	if (unlikely((asid & ~ASID_MASK) == 0)) {
-		asid = cpu_last_asid + smp_processor_id() + 1;
+		asid = cpu_last_asid + smp_processor_id();
 		flush_context();
 #ifdef CONFIG_SMP
 		smp_wmb();
 		smp_call_function(reset_context, NULL, 1);
 #endif
-		cpu_last_asid += NR_CPUS;
+		cpu_last_asid += NR_CPUS - 1;
 	}
 
 	set_mm_context(mm, asid);

[PATCH v3 4/6] ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on ASID-capable CPUs

[Catalin Marinas]

Since the ASIDs must be unique to an mm across all the CPUs in a system,
the __new_context() function needs to broadcast a context reset event to
all the CPUs during ASID allocation if a roll-over occurred. Such IPIs
cannot be issued with interrupts disabled and ARM had to define
__ARCH_WANT_INTERRUPTS_ON_CTXSW.

This patch changes the check_context() function to
check_and_switch_context() called from switch_mm(). In case of
ASID-capable CPUs (ARMv6 onwards), if a new ASID is needed and the
interrupts are disabled, it defers the __new_context() and
cpu_switch_mm() calls to the post-lock switch hook where the interrupts
are enabled. Setting the reserved TTBR0 was also moved to
check_and_switch_context() from cpu_v7_switch_mm().

Reviewed-by: Will Deacon <will.deacon@arm.com>
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/include/asm/mmu_context.h |   72 ++++++++++++++++++++++++++++--------
 arch/arm/include/asm/system.h      |    2 +
 arch/arm/include/asm/thread_info.h |    1 +
 arch/arm/mm/context.c              |    4 +-
 arch/arm/mm/proc-v7-2level.S       |    3 -
 5 files changed, 61 insertions(+), 21 deletions(-)

diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 71605d9..35483a8 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -48,39 +48,80 @@ DECLARE_PER_CPU(struct mm_struct *, current_mm);
 
 void __init_new_context(struct task_struct *tsk, struct mm_struct *mm);
 void __new_context(struct mm_struct *mm);
+void cpu_set_reserved_ttbr0(void);
 
-static inline void check_context(struct mm_struct *mm)
+static void switch_new_context(struct mm_struct *mm)
 {
-	/*
-	 * This code is executed with interrupts enabled. Therefore,
-	 * mm->context.id cannot be updated to the latest ASID version
-	 * on a different CPU (and condition below not triggered)
-	 * without first getting an IPI to reset the context. The
-	 * alternative is to take a read_lock on mm->context.id_lock
-	 * (after changing its type to rwlock_t).
-	 */
-	if (unlikely((mm->context.id ^ cpu_last_asid) >> ASID_BITS))
-		__new_context(mm);
+	unsigned long flags;
 
+	__new_context(mm);
+
+	local_irq_save(flags);
+	cpu_switch_mm(mm->pgd, mm);
+	local_irq_restore(flags);
+}
+
+static inline void check_and_switch_context(struct mm_struct *mm,
+					    struct task_struct *tsk)
+{
 	if (unlikely(mm->context.kvm_seq != init_mm.context.kvm_seq))
 		__check_kvm_seq(mm);
+
+	/*
+	 * Required during context switch to avoid speculative page table
+	 * walking with the wrong TTBR.
+	 */
+	cpu_set_reserved_ttbr0();
+
+	if (!((mm->context.id ^ cpu_last_asid) >> ASID_BITS))
+		/*
+		 * The ASID is from the current generation, just switch to the
+		 * new pgd. This condition is only true for calls from
+		 * context_switch() and interrupts are already disabled.
+		 */
+		cpu_switch_mm(mm->pgd, mm);
+	else if (irqs_disabled())
+		/*
+		 * Defer the new ASID allocation until after the context
+		 * switch critical region since __new_context() cannot be
+		 * called with interrupts disabled (it sends IPIs).
+		 */
+		set_ti_thread_flag(task_thread_info(tsk), TIF_SWITCH_MM);
+	else
+		/*
+		 * That is a direct call to switch_mm() or activate_mm() with
+		 * interrupts enabled and a new context.
+		 */
+		switch_new_context(mm);
 }
 
 #define init_new_context(tsk,mm)	(__init_new_context(tsk,mm),0)
 
-#else
+#define finish_arch_post_lock_switch \
+	finish_arch_post_lock_switch
+static inline void finish_arch_post_lock_switch(void)
+{
+	if (test_and_clear_thread_flag(TIF_SWITCH_MM))
+		switch_new_context(current->mm);
+}
 
-static inline void check_context(struct mm_struct *mm)
+#else	/* !CONFIG_CPU_HAS_ASID */
+
+static inline void check_and_switch_context(struct mm_struct *mm,
+					    struct task_struct *tsk)
 {
 #ifdef CONFIG_MMU
 	if (unlikely(mm->context.kvm_seq != init_mm.context.kvm_seq))
 		__check_kvm_seq(mm);
+	cpu_switch_mm(mm->pgd, mm);
 #endif
 }
 
 #define init_new_context(tsk,mm)	0
 
-#endif
+#define finish_arch_post_lock_switch()	do { } while (0)
+
+#endif	/* CONFIG_CPU_HAS_ASID */
 
 #define destroy_context(mm)		do { } while(0)
 
@@ -122,8 +163,7 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 		struct mm_struct **crt_mm = &per_cpu(current_mm, cpu);
 		*crt_mm = next;
 #endif
-		check_context(next);
-		cpu_switch_mm(next->pgd, next);
+		check_and_switch_context(next, tsk);
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
 	}
diff --git a/arch/arm/include/asm/system.h b/arch/arm/include/asm/system.h
index e4c96cc..e4b41dd 100644
--- a/arch/arm/include/asm/system.h
+++ b/arch/arm/include/asm/system.h
@@ -230,7 +230,9 @@ static inline void set_copro_access(unsigned int val)
  * so enable interrupts over the context switch to avoid high
  * latency.
  */
+#ifndef CONFIG_CPU_HAS_ASID
 #define __ARCH_WANT_INTERRUPTS_ON_CTXSW
+#endif
 
 /*
  * switch_to(prev, next) should switch from task `prev' to `next'
diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h
index d4c24d4..9e13e33 100644
--- a/arch/arm/include/asm/thread_info.h
+++ b/arch/arm/include/asm/thread_info.h
@@ -146,6 +146,7 @@ extern void vfp_flush_hwstate(struct thread_info *);
 #define TIF_MEMDIE		18	/* is terminating due to OOM killer */
 #define TIF_RESTORE_SIGMASK	20
 #define TIF_SECCOMP		21
+#define TIF_SWITCH_MM		22	/* deferred switch_mm */
 
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c
index efa413a..ab35a86 100644
--- a/arch/arm/mm/context.c
+++ b/arch/arm/mm/context.c
@@ -23,7 +23,7 @@ DEFINE_PER_CPU(struct mm_struct *, current_mm);
 #endif
 
 #ifdef CONFIG_ARM_LPAE
-static void cpu_set_reserved_ttbr0(void)
+void cpu_set_reserved_ttbr0(void)
 {
 	unsigned long ttbl = __pa(swapper_pg_dir);
 	unsigned long ttbh = 0;
@@ -39,7 +39,7 @@ static void cpu_set_reserved_ttbr0(void)
 	isb();
 }
 #else
-static void cpu_set_reserved_ttbr0(void)
+void cpu_set_reserved_ttbr0(void)
 {
 	u32 ttb;
 	/* Copy TTBR1 into TTBR0 */
diff --git a/arch/arm/mm/proc-v7-2level.S b/arch/arm/mm/proc-v7-2level.S
index 7227048..42ac069 100644
--- a/arch/arm/mm/proc-v7-2level.S
+++ b/arch/arm/mm/proc-v7-2level.S
@@ -46,9 +46,6 @@ ENTRY(cpu_v7_switch_mm)
 #ifdef CONFIG_ARM_ERRATA_430973
 	mcr	p15, 0, r2, c7, c5, 6		@ flush BTAC/BTB
 #endif
-	mrc	p15, 0, r2, c2, c0, 1		@ load TTB 1
-	mcr	p15, 0, r2, c2, c0, 0		@ into TTB 0
-	isb
 #ifdef CONFIG_ARM_ERRATA_754322
 	dsb
 #endif

[PATCH v3 5/6] ARM: Remove current_mm per-cpu variable

[Catalin Marinas]

The current_mm variable was used to store the new mm between the
switch_mm() and switch_to() calls where an IPI to reset the context
could have set the wrong mm. Since the interrupts are disabled during
context switch, there is no need for this variable, current->active_mm
already points to the current mm when interrupts are re-enabled.

Reviewed-by: Will Deacon <will.deacon@arm.com>
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/include/asm/mmu_context.h |    7 -------
 arch/arm/mm/context.c              |   12 +-----------
 2 files changed, 1 insertions(+), 18 deletions(-)

diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 35483a8..6c42d51 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -42,9 +42,6 @@ void __check_kvm_seq(struct mm_struct *mm);
 #define ASID_FIRST_VERSION	(1 << ASID_BITS)
 
 extern unsigned int cpu_last_asid;
-#ifdef CONFIG_SMP
-DECLARE_PER_CPU(struct mm_struct *, current_mm);
-#endif
 
 void __init_new_context(struct task_struct *tsk, struct mm_struct *mm);
 void __new_context(struct mm_struct *mm);
@@ -159,10 +156,6 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
 		__flush_icache_all();
 #endif
 	if (!cpumask_test_and_set_cpu(cpu, mm_cpumask(next)) || prev != next) {
-#ifdef CONFIG_SMP
-		struct mm_struct **crt_mm = &per_cpu(current_mm, cpu);
-		*crt_mm = next;
-#endif
 		check_and_switch_context(next, tsk);
 		if (cache_is_vivt())
 			cpumask_clear_cpu(cpu, mm_cpumask(prev));
diff --git a/arch/arm/mm/context.c b/arch/arm/mm/context.c
index ab35a86..1be5a55 100644
--- a/arch/arm/mm/context.c
+++ b/arch/arm/mm/context.c
@@ -18,9 +18,6 @@
 
 static DEFINE_RAW_SPINLOCK(cpu_asid_lock);
 unsigned int cpu_last_asid = ASID_FIRST_VERSION;
-#ifdef CONFIG_SMP
-DEFINE_PER_CPU(struct mm_struct *, current_mm);
-#endif
 
 #ifdef CONFIG_ARM_LPAE
 void cpu_set_reserved_ttbr0(void)
@@ -108,14 +105,7 @@ static void reset_context(void *info)
 {
 	unsigned int asid;
 	unsigned int cpu = smp_processor_id();
-	struct mm_struct *mm = per_cpu(current_mm, cpu);
-
-	/*
-	 * Check if a current_mm was set on this CPU as it might still
-	 * be in the early booting stages and using the reserved ASID.
-	 */
-	if (!mm)
-		return;
+	struct mm_struct *mm = current->active_mm;
 
 	smp_rmb();
 	asid = cpu_last_asid + cpu;

[PATCH v3 6/6] ARM: Remove __ARCH_WANT_INTERRUPTS_ON_CTXSW on pre-ARMv6 CPUs

[Catalin Marinas]

This patch removes the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition for
ARMv5 and earlier processors. On such processors, the context switch
requires a full cache flush. To avoid high interrupt latencies, this
patch defers the mm switching to the post-lock switch hook if the
interrupts are disabled.

Reviewed-by: Will Deacon <will.deacon@arm.com>
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
Cc: Russell King <linux@arm.linux.org.uk>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
---
 arch/arm/include/asm/mmu_context.h |   31 ++++++++++++++++++++++++++-----
 arch/arm/include/asm/system.h      |    9 ---------
 2 files changed, 26 insertions(+), 14 deletions(-)

diff --git a/arch/arm/include/asm/mmu_context.h b/arch/arm/include/asm/mmu_context.h
index 6c42d51..a8b5e37 100644
--- a/arch/arm/include/asm/mmu_context.h
+++ b/arch/arm/include/asm/mmu_context.h
@@ -104,19 +104,40 @@ static inline void finish_arch_post_lock_switch(void)
 
 #else	/* !CONFIG_CPU_HAS_ASID */
 
+#ifdef CONFIG_MMU
+
 static inline void check_and_switch_context(struct mm_struct *mm,
 					    struct task_struct *tsk)
 {
-#ifdef CONFIG_MMU
 	if (unlikely(mm->context.kvm_seq != init_mm.context.kvm_seq))
 		__check_kvm_seq(mm);
-	cpu_switch_mm(mm->pgd, mm);
-#endif
+
+	if (irqs_disabled())
+		/*
+		 * cpu_switch_mm() needs to flush the VIVT caches. To avoid
+		 * high interrupt latencies, defer the call and continue
+		 * running with the old mm. Since we only support UP systems
+		 * on non-ASID CPUs, the old mm will remain valid until the
+		 * finish_arch_post_lock_switch() call.
+		 */
+		set_ti_thread_flag(task_thread_info(tsk), TIF_SWITCH_MM);
+	else
+		cpu_switch_mm(mm->pgd, mm);
 }
 
-#define init_new_context(tsk,mm)	0
+#define finish_arch_post_lock_switch \
+	finish_arch_post_lock_switch
+static inline void finish_arch_post_lock_switch(void)
+{
+	if (test_and_clear_thread_flag(TIF_SWITCH_MM)) {
+		struct mm_struct *mm = current->mm;
+		cpu_switch_mm(mm->pgd, mm);
+	}
+}
 
-#define finish_arch_post_lock_switch()	do { } while (0)
+#endif	/* CONFIG_MMU */
+
+#define init_new_context(tsk,mm)	0
 
 #endif	/* CONFIG_CPU_HAS_ASID */
 
diff --git a/arch/arm/include/asm/system.h b/arch/arm/include/asm/system.h
index e4b41dd..4f85ced 100644
--- a/arch/arm/include/asm/system.h
+++ b/arch/arm/include/asm/system.h
@@ -226,15 +226,6 @@ static inline void set_copro_access(unsigned int val)
 }
 
 /*
- * switch_mm() may do a full cache flush over the context switch,
- * so enable interrupts over the context switch to avoid high
- * latency.
- */
-#ifndef CONFIG_CPU_HAS_ASID
-#define __ARCH_WANT_INTERRUPTS_ON_CTXSW
-#endif
-
-/*
  * switch_to(prev, next) should switch from task `prev' to `next'
  * `prev' will never be the same as `next'.  schedule() itself
  * contains the memory barrier to tell GCC not to cache `current'.

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Peter Zijlstra]

On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> Hi,
> 
> This is version 3 of the set of patches removing
> __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> ARM LPAE patches. There are no functional changes from v2. I plan to
> push this to -next and get it ready for 3.4-rc1.
> 
> Question for Peter/Ingo - how do we merge the first patch that
> introduces finish_arch_post_lock_switch? Do you pick it up or I can
> merge it via rmk (with your ack)?

I'm fine either way, I'll probably ask Ingo to pull your tree so that I
can stack some other patches on top.

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Catalin Marinas]

On Mon, Jan 23, 2012 at 09:15:31AM +0000, Peter Zijlstra wrote:
> On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> > This is version 3 of the set of patches removing
> > __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> > 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> > ARM LPAE patches. There are no functional changes from v2. I plan to
> > push this to -next and get it ready for 3.4-rc1.
> > 
> > Question for Peter/Ingo - how do we merge the first patch that
> > introduces finish_arch_post_lock_switch? Do you pick it up or I can
> > merge it via rmk (with your ack)?
> 
> I'm fine either way, I'll probably ask Ingo to pull your tree so that I
> can stack some other patches on top.

In which case I would need Russell's acked-by.

Thanks.

-- 
Catalin

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Russell King - ARM Linux]

On Mon, Jan 23, 2012 at 09:26:24AM +0000, Catalin Marinas wrote:
> On Mon, Jan 23, 2012 at 09:15:31AM +0000, Peter Zijlstra wrote:
> > On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> > > This is version 3 of the set of patches removing
> > > __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> > > 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> > > ARM LPAE patches. There are no functional changes from v2. I plan to
> > > push this to -next and get it ready for 3.4-rc1.
> > > 
> > > Question for Peter/Ingo - how do we merge the first patch that
> > > introduces finish_arch_post_lock_switch? Do you pick it up or I can
> > > merge it via rmk (with your ack)?
> > 
> > I'm fine either way, I'll probably ask Ingo to pull your tree so that I
> > can stack some other patches on top.
> 
> In which case I would need Russell's acked-by.

That depends on knowing what CPU architectures this has been tested on,
and whether anyone external has tested it.  It's definitely a change
which needs some tested-by tags on it.

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Catalin Marinas]

On Mon, Jan 23, 2012 at 09:47:59AM +0000, Russell King - ARM Linux wrote:
> On Mon, Jan 23, 2012 at 09:26:24AM +0000, Catalin Marinas wrote:
> > On Mon, Jan 23, 2012 at 09:15:31AM +0000, Peter Zijlstra wrote:
> > > On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> > > > This is version 3 of the set of patches removing
> > > > __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> > > > 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> > > > ARM LPAE patches. There are no functional changes from v2. I plan to
> > > > push this to -next and get it ready for 3.4-rc1.
> > > > 
> > > > Question for Peter/Ingo - how do we merge the first patch that
> > > > introduces finish_arch_post_lock_switch? Do you pick it up or I can
> > > > merge it via rmk (with your ack)?
> > > 
> > > I'm fine either way, I'll probably ask Ingo to pull your tree so that I
> > > can stack some other patches on top.
> > 
> > In which case I would need Russell's acked-by.
> 
> That depends on knowing what CPU architectures this has been tested on,
> and whether anyone external has tested it.  It's definitely a change
> which needs some tested-by tags on it.

I agree. On my side, I tested it on:

Versatile Express + Cortex-A9 (SMP configuration, ASIDs)
Versatile PB926 (UP configuration, no ASIDs)

-- 
Catalin

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Will Deacon]

On Mon, Jan 23, 2012 at 09:53:38AM +0000, Catalin Marinas wrote:
> On Mon, Jan 23, 2012 at 09:47:59AM +0000, Russell King - ARM Linux wrote:
> > On Mon, Jan 23, 2012 at 09:26:24AM +0000, Catalin Marinas wrote:
> > > On Mon, Jan 23, 2012 at 09:15:31AM +0000, Peter Zijlstra wrote:
> > > > On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> > > > > This is version 3 of the set of patches removing
> > > > > __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> > > > > 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> > > > > ARM LPAE patches. There are no functional changes from v2. I plan to
> > > > > push this to -next and get it ready for 3.4-rc1.
> > > > > 
> > > > > Question for Peter/Ingo - how do we merge the first patch that
> > > > > introduces finish_arch_post_lock_switch? Do you pick it up or I can
> > > > > merge it via rmk (with your ack)?
> > > > 
> > > > I'm fine either way, I'll probably ask Ingo to pull your tree so that I
> > > > can stack some other patches on top.
> > > 
> > > In which case I would need Russell's acked-by.
> > 
> > That depends on knowing what CPU architectures this has been tested on,
> > and whether anyone external has tested it.  It's definitely a change
> > which needs some tested-by tags on it.
> 
> I agree. On my side, I tested it on:
> 
> Versatile Express + Cortex-A9 (SMP configuration, ASIDs)
> Versatile PB926 (UP configuration, no ASIDs)

For what it's worth, I've also tested this on:

Realview PB1176 (UP, ASIDs)
Cortex-A5 (SMP, ASIDs)
Cortex-A7 (SMP, ASIDs)

and I haven't seen any problems with native, parallel kernel builds. I don't
have anything prior to ARMv5 available, but at least we seem to have covered
v5-v7.

Will

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Marc Zyngier]

On 24/01/12 15:14, Will Deacon wrote:
> On Mon, Jan 23, 2012 at 09:53:38AM +0000, Catalin Marinas wrote:
>> On Mon, Jan 23, 2012 at 09:47:59AM +0000, Russell King - ARM Linux wrote:
>>> On Mon, Jan 23, 2012 at 09:26:24AM +0000, Catalin Marinas wrote:
>>>> On Mon, Jan 23, 2012 at 09:15:31AM +0000, Peter Zijlstra wrote:
>>>>> On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
>>>>>> This is version 3 of the set of patches removing
>>>>>> __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
>>>>>> 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
>>>>>> ARM LPAE patches. There are no functional changes from v2. I plan to
>>>>>> push this to -next and get it ready for 3.4-rc1.
>>>>>>
>>>>>> Question for Peter/Ingo - how do we merge the first patch that
>>>>>> introduces finish_arch_post_lock_switch? Do you pick it up or I can
>>>>>> merge it via rmk (with your ack)?
>>>>>
>>>>> I'm fine either way, I'll probably ask Ingo to pull your tree so that I
>>>>> can stack some other patches on top.
>>>>
>>>> In which case I would need Russell's acked-by.
>>>
>>> That depends on knowing what CPU architectures this has been tested on,
>>> and whether anyone external has tested it.  It's definitely a change
>>> which needs some tested-by tags on it.
>>
>> I agree. On my side, I tested it on:
>>
>> Versatile Express + Cortex-A9 (SMP configuration, ASIDs)
>> Versatile PB926 (UP configuration, no ASIDs)
> 
> For what it's worth, I've also tested this on:
> 
> Realview PB1176 (UP, ASIDs)
> Cortex-A5 (SMP, ASIDs)
> Cortex-A7 (SMP, ASIDs)
> 
> and I haven't seen any problems with native, parallel kernel builds. I don't
> have anything prior to ARMv5 available, but at least we seem to have covered
> v5-v7.

Probably redundant with the above, but nonetheless tested on:
Realview PB11MP (SMP, ASIDs)
Realview PBA8 (UP, ASIDs)
Panda OMAP4 (SMP, ASIDs)

using parallel kernel builds.

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH v3 0/6] ARM: Remove the __ARCH_WANT_INTERRUPTS_ON_CTXSW definition

[Peter Zijlstra]

On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> Hi,
> 
> This is version 3 of the set of patches removing
> __ARCH_WANT_INTERRUPTS_ON_CTXSW on ARM. The series was rebased on top of
> 3.3-rc1 and fixed the conflicts with the kernel/sched/ changes and the
> ARM LPAE patches. There are no functional changes from v2. I plan to
> push this to -next and get it ready for 3.4-rc1.
> 
> Question for Peter/Ingo - how do we merge the first patch that
> introduces finish_arch_post_lock_switch? Do you pick it up or I can
> merge it via rmk (with your ack)?

Russell, what's the status of these patches? I'd like to see them land
in 3.4 if possible.

If you take them in your arm tree, I can ask Ingo to pull from there and
then I can build on them from there..

Re: [PATCH v3 1/6] sched: Introduce the finish_arch_post_lock_switch() scheduler hook

[Peter Zijlstra]

On Fri, 2012-01-20 at 17:42 +0000, Catalin Marinas wrote:
> This hook is called by the scheduler after rq->lock has been released
> and interrupts enabled. It will be used in subsequent patches on the ARM
> architecture.
> 
> Reviewed-by: Will Deacon <will.deacon@arm.com>
> Reviewed-by: Frank Rowand <frank.rowand@am.sony.com>
> Cc: Russell King <linux@arm.linux.org.uk>
> Cc: Ingo Molnar <mingo@elte.hu>

Acked-by: Peter Zijlstra <peterz@infradead.org>

> Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
> ---
>  kernel/sched/core.c  |    1 +
>  kernel/sched/sched.h |    3 +++
>  2 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/kernel/sched/core.c b/kernel/sched/core.c
> index df00cb0..f1db6d8 100644
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -1937,6 +1937,7 @@ static void finish_task_switch(struct rq *rq, struct task_struct *prev)
>  	local_irq_enable();
>  #endif /* __ARCH_WANT_INTERRUPTS_ON_CTXSW */
>  	finish_lock_switch(rq, prev);
> +	finish_arch_post_lock_switch();
>  	trace_sched_stat_sleeptime(current, rq->clock);
>  
>  	fire_sched_in_preempt_notifiers(current);
> diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
> index 98c0c26..d72483d 100644
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -692,6 +692,9 @@ static inline int task_running(struct rq *rq, struct task_struct *p)
>  #ifndef finish_arch_switch
>  # define finish_arch_switch(prev)	do { } while (0)
>  #endif
> +#ifndef finish_arch_post_lock_switch
> +# define finish_arch_post_lock_switch()	do { } while (0)
> +#endif
>  
>  #ifndef __ARCH_WANT_UNLOCKED_CTXSW
>  static inline void prepare_lock_switch(struct rq *rq, struct task_struct *next)
>