[PATCH 0/4] PREEMPT_RT support for sparc64

[PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

Have tested it on UltraSparc T4 (Niagara4). 

Allen Pais (4):
  sparc64: use generic rwsem spinlocks rt
  sparc64: allow forced irq threading
  sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
  sparc64: convert ctx_alloc_lock raw_spinlock_t

 arch/sparc/Kconfig                      |    7 +++----
 arch/sparc/include/asm/mmu_64.h         |    2 +-
 arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
 arch/sparc/kernel/smp_64.c              |    4 ++--
 arch/sparc/mm/init_64.c                 |   14 +++++++-------
 arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
 6 files changed, 28 insertions(+), 29 deletions(-)

-- 
1.7.10.4

[PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

Have tested it on UltraSparc T4 (Niagara4). 

Allen Pais (4):
  sparc64: use generic rwsem spinlocks rt
  sparc64: allow forced irq threading
  sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
  sparc64: convert ctx_alloc_lock raw_spinlock_t

 arch/sparc/Kconfig                      |    7 +++----
 arch/sparc/include/asm/mmu_64.h         |    2 +-
 arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
 arch/sparc/kernel/smp_64.c              |    4 ++--
 arch/sparc/mm/init_64.c                 |   14 +++++++-------
 arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
 6 files changed, 28 insertions(+), 29 deletions(-)

-- 
1.7.10.4

[PATCH 1/4] sparc64: use generic rwsem spinlocks rt

[Allen Pais]

This patch enables the use of generic rwsem spinlocks
to support RT on sparc64

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/Kconfig |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 6787bd3..554995d 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -179,12 +179,10 @@ config NR_CPUS
 source kernel/Kconfig.hz
 
 config RWSEM_GENERIC_SPINLOCK
-	bool
-	default y if SPARC32
+	def_bool PREEMPT_RT_FULL
 
 config RWSEM_XCHGADD_ALGORITHM
-	bool
-	default y if SPARC64
+	def_bool !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT_FULL
 
 config GENERIC_HWEIGHT
 	bool
-- 
1.7.10.4

[PATCH 1/4] sparc64: use generic rwsem spinlocks rt

[Allen Pais]

This patch enables the use of generic rwsem spinlocks
to support RT on sparc64

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/Kconfig |    6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 6787bd3..554995d 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -179,12 +179,10 @@ config NR_CPUS
 source kernel/Kconfig.hz
 
 config RWSEM_GENERIC_SPINLOCK
-	bool
-	default y if SPARC32
+	def_bool PREEMPT_RT_FULL
 
 config RWSEM_XCHGADD_ALGORITHM
-	bool
-	default y if SPARC64
+	def_bool !RWSEM_GENERIC_SPINLOCK && !PREEMPT_RT_FULL
 
 config GENERIC_HWEIGHT
 	bool
-- 
1.7.10.4

[PATCH 2/4] sparc64: allow forced irq threading

[Allen Pais]

Forced irq threading is a prerequisite for RT.
The following patch enables this on the sparc architecture.

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/Kconfig |    1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 554995d..aae5aa9 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -27,6 +27,7 @@ config SPARC
 	select HAVE_DMA_API_DEBUG
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_GENERIC_HARDIRQS
+	select IRQ_FORCED_THREADING
 	select GENERIC_IRQ_SHOW
 	select ARCH_WANT_IPC_PARSE_VERSION
 	select USE_GENERIC_SMP_HELPERS if SMP
-- 
1.7.10.4

[PATCH 2/4] sparc64: allow forced irq threading

[Allen Pais]

Forced irq threading is a prerequisite for RT.
The following patch enables this on the sparc architecture.

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/Kconfig |    1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 554995d..aae5aa9 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -27,6 +27,7 @@ config SPARC
 	select HAVE_DMA_API_DEBUG
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_GENERIC_HARDIRQS
+	select IRQ_FORCED_THREADING
 	select GENERIC_IRQ_SHOW
 	select ARCH_WANT_IPC_PARSE_VERSION
 	select USE_GENERIC_SMP_HELPERS if SMP
-- 
1.7.10.4

[PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

In the attempt of get PREEMPT_RT working on sparc64 using
linux-stable-rt version 3.10.22-rt19+, the kernel crash
with the following trace:

[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
[ 1487.027885] Call Trace:
[ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
[ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
[ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
[ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
[ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
[ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
[ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
[ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
[ 1487.027922]  [0000000000000000]           (null)

Thomas debugged this issue and pointed to switch_mm

        spin_lock_irqsave(&mm->context.lock, flags);

context.lock needs to be a raw_spinlock.

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/include/asm/mmu_64.h         |    2 +-
 arch/sparc/include/asm/mmu_context_64.h |    8 ++++----
 arch/sparc/kernel/smp_64.c              |    4 ++--
 arch/sparc/mm/init_64.c                 |    4 ++--
 arch/sparc/mm/tsb.c                     |   16 ++++++++--------
 5 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/arch/sparc/include/asm/mmu_64.h b/arch/sparc/include/asm/mmu_64.h
index 76092c4..e945ddb 100644
--- a/arch/sparc/include/asm/mmu_64.h
+++ b/arch/sparc/include/asm/mmu_64.h
@@ -90,7 +90,7 @@ struct tsb_config {
 #endif
 
 typedef struct {
-	spinlock_t		lock;
+	raw_spinlock_t		lock;
 	unsigned long		sparc64_ctx_val;
 	unsigned long		huge_pte_count;
 	struct page		*pgtable_page;
diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
index 3d528f0..3a85624 100644
--- a/arch/sparc/include/asm/mmu_context_64.h
+++ b/arch/sparc/include/asm/mmu_context_64.h
@@ -77,7 +77,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
 	if (unlikely(mm == &init_mm))
 		return;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 	ctx_valid = CTX_VALID(mm->context);
 	if (!ctx_valid)
 		get_new_mmu_context(mm);
@@ -125,7 +125,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
 		__flush_tlb_mm(CTX_HWBITS(mm->context),
 			       SECONDARY_CONTEXT);
 	}
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #define deactivate_mm(tsk,mm)	do { } while (0)
@@ -136,7 +136,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
 	unsigned long flags;
 	int cpu;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 	if (!CTX_VALID(mm->context))
 		get_new_mmu_context(mm);
 	cpu = smp_processor_id();
@@ -146,7 +146,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
 	load_secondary_context(mm);
 	__flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT);
 	tsb_context_switch(mm);
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #endif /* !(__ASSEMBLY__) */
diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
index 77539ed..f42e1a7 100644
--- a/arch/sparc/kernel/smp_64.c
+++ b/arch/sparc/kernel/smp_64.c
@@ -975,12 +975,12 @@ void __irq_entry smp_new_mmu_context_version_client(int irq, struct pt_regs *reg
 	if (unlikely(!mm || (mm == &init_mm)))
 		return;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	if (unlikely(!CTX_VALID(mm->context)))
 		get_new_mmu_context(mm);
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 	load_secondary_context(mm);
 	__flush_tlb_mm(CTX_HWBITS(mm->context),
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 04fd55a..bd5253d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -350,7 +350,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
 
 	mm = vma->vm_mm;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 #if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)
 	if (mm->context.huge_pte_count && is_hugetlb_pte(pte))
@@ -361,7 +361,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
 		__update_mmu_tsb_insert(mm, MM_TSB_BASE, PAGE_SHIFT,
 					address, pte_val(pte));
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 void flush_dcache_page(struct page *page)
diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index 2cc3bce..d84d4ea 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -73,7 +73,7 @@ void flush_tsb_user(struct tlb_batch *tb)
 	struct mm_struct *mm = tb->mm;
 	unsigned long nentries, base, flags;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
 	nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -90,14 +90,14 @@ void flush_tsb_user(struct tlb_batch *tb)
 		__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
 	}
 #endif
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
 {
 	unsigned long nentries, base, flags;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
 	nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -114,7 +114,7 @@ void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
 		__flush_tsb_one_entry(base, vaddr, HPAGE_SHIFT, nentries);
 	}
 #endif
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #define HV_PGSZ_IDX_BASE	HV_PGSZ_IDX_8K
@@ -392,7 +392,7 @@ retry_tsb_alloc:
 	 * the lock and ask all other cpus running this address space
 	 * to run tsb_context_switch() to see the new TSB table.
 	 */
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	old_tsb = mm->context.tsb_block[tsb_index].tsb;
 	old_cache_index =
@@ -407,7 +407,7 @@ retry_tsb_alloc:
 	 */
 	if (unlikely(old_tsb &&
 		     (rss < mm->context.tsb_block[tsb_index].tsb_rss_limit))) {
-		spin_unlock_irqrestore(&mm->context.lock, flags);
+		raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 		kmem_cache_free(tsb_caches[new_cache_index], new_tsb);
 		return;
@@ -433,7 +433,7 @@ retry_tsb_alloc:
 	mm->context.tsb_block[tsb_index].tsb = new_tsb;
 	setup_tsb_params(mm, tsb_index, new_size);
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 	/* If old_tsb is NULL, we're being invoked for the first time
 	 * from init_new_context().
@@ -459,7 +459,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 #endif
 	unsigned int i;
 
-	spin_lock_init(&mm->context.lock);
+	raw_spin_lock_init(&mm->context.lock);
 
 	mm->context.sparc64_ctx_val = 0UL;
 
-- 
1.7.10.4

[PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

In the attempt of get PREEMPT_RT working on sparc64 using
linux-stable-rt version 3.10.22-rt19+, the kernel crash
with the following trace:

[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
[ 1487.027885] Call Trace:
[ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
[ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
[ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
[ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
[ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
[ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
[ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
[ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
[ 1487.027922]  [0000000000000000]           (null)

Thomas debugged this issue and pointed to switch_mm

        spin_lock_irqsave(&mm->context.lock, flags);

context.lock needs to be a raw_spinlock.

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/include/asm/mmu_64.h         |    2 +-
 arch/sparc/include/asm/mmu_context_64.h |    8 ++++----
 arch/sparc/kernel/smp_64.c              |    4 ++--
 arch/sparc/mm/init_64.c                 |    4 ++--
 arch/sparc/mm/tsb.c                     |   16 ++++++++--------
 5 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/arch/sparc/include/asm/mmu_64.h b/arch/sparc/include/asm/mmu_64.h
index 76092c4..e945ddb 100644
--- a/arch/sparc/include/asm/mmu_64.h
+++ b/arch/sparc/include/asm/mmu_64.h
@@ -90,7 +90,7 @@ struct tsb_config {
 #endif
 
 typedef struct {
-	spinlock_t		lock;
+	raw_spinlock_t		lock;
 	unsigned long		sparc64_ctx_val;
 	unsigned long		huge_pte_count;
 	struct page		*pgtable_page;
diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
index 3d528f0..3a85624 100644
--- a/arch/sparc/include/asm/mmu_context_64.h
+++ b/arch/sparc/include/asm/mmu_context_64.h
@@ -77,7 +77,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
 	if (unlikely(mm = &init_mm))
 		return;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 	ctx_valid = CTX_VALID(mm->context);
 	if (!ctx_valid)
 		get_new_mmu_context(mm);
@@ -125,7 +125,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
 		__flush_tlb_mm(CTX_HWBITS(mm->context),
 			       SECONDARY_CONTEXT);
 	}
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #define deactivate_mm(tsk,mm)	do { } while (0)
@@ -136,7 +136,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
 	unsigned long flags;
 	int cpu;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 	if (!CTX_VALID(mm->context))
 		get_new_mmu_context(mm);
 	cpu = smp_processor_id();
@@ -146,7 +146,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
 	load_secondary_context(mm);
 	__flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT);
 	tsb_context_switch(mm);
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #endif /* !(__ASSEMBLY__) */
diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
index 77539ed..f42e1a7 100644
--- a/arch/sparc/kernel/smp_64.c
+++ b/arch/sparc/kernel/smp_64.c
@@ -975,12 +975,12 @@ void __irq_entry smp_new_mmu_context_version_client(int irq, struct pt_regs *reg
 	if (unlikely(!mm || (mm = &init_mm)))
 		return;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	if (unlikely(!CTX_VALID(mm->context)))
 		get_new_mmu_context(mm);
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 	load_secondary_context(mm);
 	__flush_tlb_mm(CTX_HWBITS(mm->context),
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index 04fd55a..bd5253d 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -350,7 +350,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
 
 	mm = vma->vm_mm;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 #if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)
 	if (mm->context.huge_pte_count && is_hugetlb_pte(pte))
@@ -361,7 +361,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
 		__update_mmu_tsb_insert(mm, MM_TSB_BASE, PAGE_SHIFT,
 					address, pte_val(pte));
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 void flush_dcache_page(struct page *page)
diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index 2cc3bce..d84d4ea 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -73,7 +73,7 @@ void flush_tsb_user(struct tlb_batch *tb)
 	struct mm_struct *mm = tb->mm;
 	unsigned long nentries, base, flags;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
 	nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -90,14 +90,14 @@ void flush_tsb_user(struct tlb_batch *tb)
 		__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
 	}
 #endif
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
 {
 	unsigned long nentries, base, flags;
 
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
 	nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -114,7 +114,7 @@ void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
 		__flush_tsb_one_entry(base, vaddr, HPAGE_SHIFT, nentries);
 	}
 #endif
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 }
 
 #define HV_PGSZ_IDX_BASE	HV_PGSZ_IDX_8K
@@ -392,7 +392,7 @@ retry_tsb_alloc:
 	 * the lock and ask all other cpus running this address space
 	 * to run tsb_context_switch() to see the new TSB table.
 	 */
-	spin_lock_irqsave(&mm->context.lock, flags);
+	raw_spin_lock_irqsave(&mm->context.lock, flags);
 
 	old_tsb = mm->context.tsb_block[tsb_index].tsb;
 	old_cache_index @@ -407,7 +407,7 @@ retry_tsb_alloc:
 	 */
 	if (unlikely(old_tsb &&
 		     (rss < mm->context.tsb_block[tsb_index].tsb_rss_limit))) {
-		spin_unlock_irqrestore(&mm->context.lock, flags);
+		raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 		kmem_cache_free(tsb_caches[new_cache_index], new_tsb);
 		return;
@@ -433,7 +433,7 @@ retry_tsb_alloc:
 	mm->context.tsb_block[tsb_index].tsb = new_tsb;
 	setup_tsb_params(mm, tsb_index, new_size);
 
-	spin_unlock_irqrestore(&mm->context.lock, flags);
+	raw_spin_unlock_irqrestore(&mm->context.lock, flags);
 
 	/* If old_tsb is NULL, we're being invoked for the first time
 	 * from init_new_context().
@@ -459,7 +459,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
 #endif
 	unsigned int i;
 
-	spin_lock_init(&mm->context.lock);
+	raw_spin_lock_init(&mm->context.lock);
 
 	mm->context.sparc64_ctx_val = 0UL;
 
-- 
1.7.10.4

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
> In the attempt of get PREEMPT_RT working on sparc64 using
> linux-stable-rt version 3.10.22-rt19+, the kernel crash
> with the following trace:
>
> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
> [ 1487.027885] Call Trace:
> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
> [ 1487.027922]  [0000000000000000]           (null)
>
> Thomas debugged this issue and pointed to switch_mm
>
>         spin_lock_irqsave(&mm->context.lock, flags);
>
> context.lock needs to be a raw_spinlock.
>
> Acked-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Allen Pais <allen.pais@oracle.com>
> ---
>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>  arch/sparc/include/asm/mmu_context_64.h |    8 ++++----
>  arch/sparc/kernel/smp_64.c              |    4 ++--
>  arch/sparc/mm/init_64.c                 |    4 ++--
>  arch/sparc/mm/tsb.c                     |   16 ++++++++--------
>  5 files changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/arch/sparc/include/asm/mmu_64.h b/arch/sparc/include/asm/mmu_64.h
> index 76092c4..e945ddb 100644
> --- a/arch/sparc/include/asm/mmu_64.h
> +++ b/arch/sparc/include/asm/mmu_64.h
> @@ -90,7 +90,7 @@ struct tsb_config {
>  #endif
>
>  typedef struct {
> - spinlock_t lock;
> + raw_spinlock_t lock;
>          unsigned long sparc64_ctx_val;
>          unsigned long huge_pte_count;
>          struct page *pgtable_page;
> diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
> index 3d528f0..3a85624 100644
> --- a/arch/sparc/include/asm/mmu_context_64.h
> +++ b/arch/sparc/include/asm/mmu_context_64.h
> @@ -77,7 +77,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
>          if (unlikely(mm == &init_mm))
>                  return;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>          ctx_valid = CTX_VALID(mm->context);
>          if (!ctx_valid)
>                  get_new_mmu_context(mm);
> @@ -125,7 +125,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
>                  __flush_tlb_mm(CTX_HWBITS(mm->context),
>                                 SECONDARY_CONTEXT);
>          }
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>  }
>
>  #define deactivate_mm(tsk,mm) do { } while (0)
> @@ -136,7 +136,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
>          unsigned long flags;
>          int cpu;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>          if (!CTX_VALID(mm->context))
>                  get_new_mmu_context(mm);
>          cpu = smp_processor_id();
> @@ -146,7 +146,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
>          load_secondary_context(mm);
>          __flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT);
>          tsb_context_switch(mm);
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>  }
>
>  #endif /* !(__ASSEMBLY__) */
> diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
> index 77539ed..f42e1a7 100644
> --- a/arch/sparc/kernel/smp_64.c
> +++ b/arch/sparc/kernel/smp_64.c
> @@ -975,12 +975,12 @@ void __irq_entry smp_new_mmu_context_version_client(int irq, struct pt_regs *reg
>          if (unlikely(!mm || (mm == &init_mm)))
>                  return;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
>          if (unlikely(!CTX_VALID(mm->context)))
>                  get_new_mmu_context(mm);
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
>          load_secondary_context(mm);
>          __flush_tlb_mm(CTX_HWBITS(mm->context),
> diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
> index 04fd55a..bd5253d 100644
> --- a/arch/sparc/mm/init_64.c
> +++ b/arch/sparc/mm/init_64.c
> @@ -350,7 +350,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
>
>          mm = vma->vm_mm;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
>  #if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)
>          if (mm->context.huge_pte_count && is_hugetlb_pte(pte))
> @@ -361,7 +361,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
>                  __update_mmu_tsb_insert(mm, MM_TSB_BASE, PAGE_SHIFT,
>                                          address, pte_val(pte));
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>  }

We also should do the same in update_mmu_cache_pmd().

>
>  void flush_dcache_page(struct page *page)
> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
> index 2cc3bce..d84d4ea 100644
> --- a/arch/sparc/mm/tsb.c
> +++ b/arch/sparc/mm/tsb.c
> @@ -73,7 +73,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>          struct mm_struct *mm = tb->mm;
>          unsigned long nentries, base, flags;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -90,14 +90,14 @@ void flush_tsb_user(struct tlb_batch *tb)
>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>          }
>  #endif
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>  }
>
>  void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
>  {
>          unsigned long nentries, base, flags;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -114,7 +114,7 @@ void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
>                  __flush_tsb_one_entry(base, vaddr, HPAGE_SHIFT, nentries);
>          }
>  #endif
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>  }
>
>  #define HV_PGSZ_IDX_BASE HV_PGSZ_IDX_8K
> @@ -392,7 +392,7 @@ retry_tsb_alloc:
>           * the lock and ask all other cpus running this address space
>           * to run tsb_context_switch() to see the new TSB table.
>           */
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
>          old_tsb = mm->context.tsb_block[tsb_index].tsb;
>          old_cache_index =
> @@ -407,7 +407,7 @@ retry_tsb_alloc:
>           */
>          if (unlikely(old_tsb &&
>                       (rss < mm->context.tsb_block[tsb_index].tsb_rss_limit))) {
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
>                  kmem_cache_free(tsb_caches[new_cache_index], new_tsb);
>                  return;
> @@ -433,7 +433,7 @@ retry_tsb_alloc:
>          mm->context.tsb_block[tsb_index].tsb = new_tsb;
>          setup_tsb_params(mm, tsb_index, new_size);
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
>          /* If old_tsb is NULL, we're being invoked for the first time
>           * from init_new_context().
> @@ -459,7 +459,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
>  #endif
>          unsigned int i;
>
> - spin_lock_init(&mm->context.lock);
> + raw_spin_lock_init(&mm->context.lock);
>
>          mm->context.sparc64_ctx_val = 0UL;
>
> --
> 1.7.10.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
> In the attempt of get PREEMPT_RT working on sparc64 using
> linux-stable-rt version 3.10.22-rt19+, the kernel crash
> with the following trace:
>
> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
> [ 1487.027885] Call Trace:
> [ 1487.027887] š[00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
> [ 1487.027892] š[00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
> [ 1487.027895] š[0000000000819114] rt_spin_lock_slowlock+0x94/0x300
> [ 1487.027897] š[0000000000817ebc] __schedule+0x39c/0x53c
> [ 1487.027899] š[00000000008185fc] schedule+0x1c/0xc0
> [ 1487.027908] š[000000000048fff4] smpboot_thread_fn+0x154/0x2e0
> [ 1487.027913] š[000000000048753c] kthread+0x7c/0xa0
> [ 1487.027920] š[00000000004060c4] ret_from_syscall+0x1c/0x2c
> [ 1487.027922] š[0000000000000000] šššššššššš(null)
>
> Thomas debugged this issue and pointed to switch_mm
>
> ššššššššspin_lock_irqsave(&mm->context.lock, flags);
>
> context.lock needs to be a raw_spinlock.
>
> Acked-by: David S. Miller <davem@davemloft.net>
> Signed-off-by: Allen Pais <allen.pais@oracle.com>
> ---
> šarch/sparc/include/asm/mmu_64.h šššššššš| ššš2 +-
> šarch/sparc/include/asm/mmu_context_64.h | ššš8 ++++----
> šarch/sparc/kernel/smp_64.c ššššššššššššš| ššš4 ++--
> šarch/sparc/mm/init_64.c šššššššššššššššš| ššš4 ++--
> šarch/sparc/mm/tsb.c šššššššššššššššššššš| šš16 ++++++++--------
> š5 files changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/arch/sparc/include/asm/mmu_64.h b/arch/sparc/include/asm/mmu_64.h
> index 76092c4..e945ddb 100644
> --- a/arch/sparc/include/asm/mmu_64.h
> +++ b/arch/sparc/include/asm/mmu_64.h
> @@ -90,7 +90,7 @@ struct tsb_config {
> š#endif
>
> štypedef struct {
> - spinlock_t lock;
> + raw_spinlock_t lock;
> šššššššššunsigned long sparc64_ctx_val;
> šššššššššunsigned long huge_pte_count;
> šššššššššstruct page *pgtable_page;
> diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
> index 3d528f0..3a85624 100644
> --- a/arch/sparc/include/asm/mmu_context_64.h
> +++ b/arch/sparc/include/asm/mmu_context_64.h
> @@ -77,7 +77,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
> šššššššššif (unlikely(mm = &init_mm))
> šššššššššššššššššreturn;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
> šššššššššctx_valid = CTX_VALID(mm->context);
> šššššššššif (!ctx_valid)
> šššššššššššššššššget_new_mmu_context(mm);
> @@ -125,7 +125,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, str
> ššššššššššššššššš__flush_tlb_mm(CTX_HWBITS(mm->context),
> ššššššššššššššššššššššššššššššššSECONDARY_CONTEXT);
> ššššššššš}
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> š}
>
> š#define deactivate_mm(tsk,mm) do { } while (0)
> @@ -136,7 +136,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
> šššššššššunsigned long flags;
> šššššššššint cpu;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
> šššššššššif (!CTX_VALID(mm->context))
> šššššššššššššššššget_new_mmu_context(mm);
> šššššššššcpu = smp_processor_id();
> @@ -146,7 +146,7 @@ static inline void activate_mm(struct mm_struct *active_mm, struct mm_struct *mm
> šššššššššload_secondary_context(mm);
> ššššššššš__flush_tlb_mm(CTX_HWBITS(mm->context), SECONDARY_CONTEXT);
> ššššššššštsb_context_switch(mm);
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> š}
>
> š#endif /* !(__ASSEMBLY__) */
> diff --git a/arch/sparc/kernel/smp_64.c b/arch/sparc/kernel/smp_64.c
> index 77539ed..f42e1a7 100644
> --- a/arch/sparc/kernel/smp_64.c
> +++ b/arch/sparc/kernel/smp_64.c
> @@ -975,12 +975,12 @@ void __irq_entry smp_new_mmu_context_version_client(int irq, struct pt_regs *reg
> šššššššššif (unlikely(!mm || (mm = &init_mm)))
> šššššššššššššššššreturn;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
> šššššššššif (unlikely(!CTX_VALID(mm->context)))
> šššššššššššššššššget_new_mmu_context(mm);
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
> šššššššššload_secondary_context(mm);
> ššššššššš__flush_tlb_mm(CTX_HWBITS(mm->context),
> diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
> index 04fd55a..bd5253d 100644
> --- a/arch/sparc/mm/init_64.c
> +++ b/arch/sparc/mm/init_64.c
> @@ -350,7 +350,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
>
> šššššššššmm = vma->vm_mm;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
> š#if defined(CONFIG_HUGETLB_PAGE) || defined(CONFIG_TRANSPARENT_HUGEPAGE)
> šššššššššif (mm->context.huge_pte_count && is_hugetlb_pte(pte))
> @@ -361,7 +361,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address, pte_t *
> ššššššššššššššššš__update_mmu_tsb_insert(mm, MM_TSB_BASE, PAGE_SHIFT,
> šššššššššššššššššššššššššššššššššššššššššaddress, pte_val(pte));
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> š}

We also should do the same in update_mmu_cache_pmd().

>
> švoid flush_dcache_page(struct page *page)
> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
> index 2cc3bce..d84d4ea 100644
> --- a/arch/sparc/mm/tsb.c
> +++ b/arch/sparc/mm/tsb.c
> @@ -73,7 +73,7 @@ void flush_tsb_user(struct tlb_batch *tb)
> šššššššššstruct mm_struct *mm = tb->mm;
> šššššššššunsigned long nentries, base, flags;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
> šššššššššbase = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
> šššššššššnentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -90,14 +90,14 @@ void flush_tsb_user(struct tlb_batch *tb)
> ššššššššššššššššš__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
> ššššššššš}
> š#endif
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> š}
>
> švoid flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
> š{
> šššššššššunsigned long nentries, base, flags;
>
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
> šššššššššbase = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
> šššššššššnentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -114,7 +114,7 @@ void flush_tsb_user_page(struct mm_struct *mm, unsigned long vaddr)
> ššššššššššššššššš__flush_tsb_one_entry(base, vaddr, HPAGE_SHIFT, nentries);
> ššššššššš}
> š#endif
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> š}
>
> š#define HV_PGSZ_IDX_BASE HV_PGSZ_IDX_8K
> @@ -392,7 +392,7 @@ retry_tsb_alloc:
> šššššššššš* the lock and ask all other cpus running this address space
> šššššššššš* to run tsb_context_switch() to see the new TSB table.
> šššššššššš*/
> - spin_lock_irqsave(&mm->context.lock, flags);
> + raw_spin_lock_irqsave(&mm->context.lock, flags);
>
> šššššššššold_tsb = mm->context.tsb_block[tsb_index].tsb;
> šššššššššold_cache_index > @@ -407,7 +407,7 @@ retry_tsb_alloc:
> šššššššššš*/
> šššššššššif (unlikely(old_tsb &&
> šššššššššššššššššššššš(rss < mm->context.tsb_block[tsb_index].tsb_rss_limit))) {
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
> ššššššššššššššššškmem_cache_free(tsb_caches[new_cache_index], new_tsb);
> šššššššššššššššššreturn;
> @@ -433,7 +433,7 @@ retry_tsb_alloc:
> šššššššššmm->context.tsb_block[tsb_index].tsb = new_tsb;
> šššššššššsetup_tsb_params(mm, tsb_index, new_size);
>
> - spin_unlock_irqrestore(&mm->context.lock, flags);
> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>
> ššššššššš/* If old_tsb is NULL, we're being invoked for the first time
> šššššššššš* from init_new_context().
> @@ -459,7 +459,7 @@ int init_new_context(struct task_struct *tsk, struct mm_struct *mm)
> š#endif
> šššššššššunsigned int i;
>
> - spin_lock_init(&mm->context.lock);
> + raw_spin_lock_init(&mm->context.lock);
>
> šššššššššmm->context.sparc64_ctx_val = 0UL;
>
> --
> 1.7.10.4
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at šhttp://vger.kernel.org/majordomo-info.html
> Please read the FAQ at šhttp://www.tux.org/lkml/

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
> 
> 
> 06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>> In the attempt of get PREEMPT_RT working on sparc64 using
>> linux-stable-rt version 3.10.22-rt19+, the kernel crash
>> with the following trace:
>>
>> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>> [ 1487.027885] Call Trace:
>> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>> [ 1487.027922]  [0000000000000000]           (null)
>> - spin_unlock_irqrestore(&mm->context.lock, flags);
>> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>  }
> 
> We also should do the same in update_mmu_cache_pmd().
> 

I have already done this. I should have updated the patch. the issue still
persists though.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
> 
> 
> 06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>> In the attempt of get PREEMPT_RT working on sparc64 using
>> linux-stable-rt version 3.10.22-rt19+, the kernel crash
>> with the following trace:
>>
>> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>> [ 1487.027885] Call Trace:
>> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>> [ 1487.027922]  [0000000000000000]           (null)
>> - spin_unlock_irqrestore(&mm->context.lock, flags);
>> + raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>  }
> 
> We also should do the same in update_mmu_cache_pmd().
> 

I have already done this. I should have updated the patch. the issue still
persists though.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
> 
> 
> 06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>> In the attempt of get PREEMPT_RT working on sparc64 using
>> linux-stable-rt version 3.10.22-rt19+, the kernel crash
>> with the following trace:
>>
>> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>> [ 1487.027885] Call Trace:
>> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>> [ 1487.027922]  [0000000000000000]           (null)
>>

Now, consistently I've been getting sun4v_data_access_exception.
Here's the trace:

[ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
[ 4673.360124]               \|/ ____ \|/
[ 4673.360124]               "@'/ .. \`@"
[ 4673.360124]               /_| \__/ |_\
[ 4673.360124]                  \__U_/
[ 4673.360128] hackbench(4183): Dax [#1]
[ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G        W    3.10.24-rt22+ #12
[ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
[ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000    Tainted: G        W   
[ 4673.360157] TPC: <tsb_flush+0x4/0x40>
[ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
[ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
[ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
[ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
[ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
[ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
[ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
[ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
[ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
[ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
[ 4673.360193] Call Trace:
[ 4673.360198]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[ 4673.360203]  [000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360210]  [000000000052e520] unmap_single_vma+0x400/0x6c0
[ 4673.360213]  [000000000052e808] unmap_vmas+0x28/0x60
[ 4673.360220]  [0000000000530cc8] exit_mmap+0x88/0x160
[ 4673.360226]  [000000000045e0d4] mmput+0x34/0xe0
[ 4673.360236]  [00000000004669fc] do_exit+0x1fc/0xa40
[ 4673.360241]  [0000000000467270] do_group_exit+0x30/0xe0
[ 4673.360245]  [000000000046733c] SyS_exit_group+0x1c/0x40
[ 4673.360256]  [0000000000406234] linux_sparc_syscall+0x34/0x44
[ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
[ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
[ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
[ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
[ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
[ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
[ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
[ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
[ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44



- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
> 
> 
> 06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>> In the attempt of get PREEMPT_RT working on sparc64 using
>> linux-stable-rt version 3.10.22-rt19+, the kernel crash
>> with the following trace:
>>
>> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>> [ 1487.027885] Call Trace:
>> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>> [ 1487.027922]  [0000000000000000]           (null)
>>

Now, consistently I've been getting sun4v_data_access_exception.
Here's the trace:

[ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
[ 4673.360124]               \|/ ____ \|/
[ 4673.360124]               "@'/ .. \`@"
[ 4673.360124]               /_| \__/ |_\
[ 4673.360124]                  \__U_/
[ 4673.360128] hackbench(4183): Dax [#1]
[ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G        W    3.10.24-rt22+ #12
[ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
[ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000    Tainted: G        W   
[ 4673.360157] TPC: <tsb_flush+0x4/0x40>
[ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
[ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
[ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
[ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
[ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
[ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
[ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
[ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
[ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
[ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
[ 4673.360193] Call Trace:
[ 4673.360198]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[ 4673.360203]  [000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360210]  [000000000052e520] unmap_single_vma+0x400/0x6c0
[ 4673.360213]  [000000000052e808] unmap_vmas+0x28/0x60
[ 4673.360220]  [0000000000530cc8] exit_mmap+0x88/0x160
[ 4673.360226]  [000000000045e0d4] mmput+0x34/0xe0
[ 4673.360236]  [00000000004669fc] do_exit+0x1fc/0xa40
[ 4673.360241]  [0000000000467270] do_group_exit+0x30/0xe0
[ 4673.360245]  [000000000046733c] SyS_exit_group+0x1c/0x40
[ 4673.360256]  [0000000000406234] linux_sparc_syscall+0x34/0x44
[ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
[ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
[ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
[ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
[ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
[ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
[ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
[ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
[ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44



- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:

>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>   with the following trace:
>>>
>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>   [ 1487.027885] Call Trace:
>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>   [ 1487.027922]  [0000000000000000]           (null)
>  Now, consistently I've been getting sun4v_data_access_exception.
>  Here's the trace:
>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.

I've never dived at sparc's tlb before, but it seems now I'm understanding.

arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
you collect flush requests before you really flush all of them.

In RT you collect them too, but you are able to be preempted in any moment.
So, you may switch to other process with unflushed tlb, which is very bad.

Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
We will look if this robust fix helps.


>  [ 4673.360124]               \|/ ____ \|/
>  [ 4673.360124]               "@'/ .. \`@"
>  [ 4673.360124]               /_| \__/ |_\
>  [ 4673.360124]                  \__U_/
>  [ 4673.360128] hackbench(4183): Dax [#1]
>  [ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G        W    3.10.24-rt22+ #12
>  [ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
>  [ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000    Tainted: G        W
>  [ 4673.360157] TPC: <tsb_flush+0x4/0x40>
>  [ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
>  [ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
>  [ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
>  [ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
>  [ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
>  [ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
>  [ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
>  [ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
>  [ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
>  [ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
>  [ 4673.360193] Call Trace:
>  [ 4673.360198]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
>  [ 4673.360203]  [000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
>  [ 4673.360210]  [000000000052e520] unmap_single_vma+0x400/0x6c0
>  [ 4673.360213]  [000000000052e808] unmap_vmas+0x28/0x60
>  [ 4673.360220]  [0000000000530cc8] exit_mmap+0x88/0x160
>  [ 4673.360226]  [000000000045e0d4] mmput+0x34/0xe0
>  [ 4673.360236]  [00000000004669fc] do_exit+0x1fc/0xa40
>  [ 4673.360241]  [0000000000467270] do_group_exit+0x30/0xe0
>  [ 4673.360245]  [000000000046733c] SyS_exit_group+0x1c/0x40
>  [ 4673.360256]  [0000000000406234] linux_sparc_syscall+0x34/0x44
>  [ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
>  [ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
>  [ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
>  [ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
>  [ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
>  [ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
>  [ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
>  [ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
>  [ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
>  [ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44
>
>  - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:

> šOn Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>> šš06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>> ššIn the attempt of get PREEMPT_RT working on sparc64 using
>>> ššlinux-stable-rt version 3.10.22-rt19+, the kernel crash
>>> ššwith the following trace:
>>>
>>> šš[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>> šš[ 1487.027885] Call Trace:
>>> šš[ 1487.027887] š[00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>> šš[ 1487.027892] š[00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>> šš[ 1487.027895] š[0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>> šš[ 1487.027897] š[0000000000817ebc] __schedule+0x39c/0x53c
>>> šš[ 1487.027899] š[00000000008185fc] schedule+0x1c/0xc0
>>> šš[ 1487.027908] š[000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>> šš[ 1487.027913] š[000000000048753c] kthread+0x7c/0xa0
>>> šš[ 1487.027920] š[00000000004060c4] ret_from_syscall+0x1c/0x2c
>>> šš[ 1487.027922] š[0000000000000000] šššššššššš(null)
> šNow, consistently I've been getting sun4v_data_access_exception.
> šHere's the trace:
> š[ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.

I've never dived at sparc's tlb before, but it seems now I'm understanding.

arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
you collect flush requests before you really flush all of them.

In RT you collect them too, but you are able to be preempted in any moment.
So, you may switch to other process with unflushed tlb, which is very bad.

Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
We will look if this robust fix helps.


> š[ 4673.360124] šššššššššššššš\|/ ____ \|/
> š[ 4673.360124] šššššššššššššš"@'/ .. \`@"
> š[ 4673.360124] šššššššššššššš/_| \__/ |_\
> š[ 4673.360124] ššššššššššššššššš\__U_/
> š[ 4673.360128] hackbench(4183): Dax [#1]
> š[ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G šššššššW ššš3.10.24-rt22+ #12
> š[ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
> š[ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000 šššTainted: G šššššššW
> š[ 4673.360157] TPC: <tsb_flush+0x4/0x40>
> š[ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
> š[ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
> š[ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
> š[ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
> š[ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
> š[ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
> š[ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
> š[ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
> š[ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
> š[ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
> š[ 4673.360193] Call Trace:
> š[ 4673.360198] š[00000000004515a8] flush_tlb_pending+0x68/0xe0
> š[ 4673.360203] š[000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
> š[ 4673.360210] š[000000000052e520] unmap_single_vma+0x400/0x6c0
> š[ 4673.360213] š[000000000052e808] unmap_vmas+0x28/0x60
> š[ 4673.360220] š[0000000000530cc8] exit_mmap+0x88/0x160
> š[ 4673.360226] š[000000000045e0d4] mmput+0x34/0xe0
> š[ 4673.360236] š[00000000004669fc] do_exit+0x1fc/0xa40
> š[ 4673.360241] š[0000000000467270] do_group_exit+0x30/0xe0
> š[ 4673.360245] š[000000000046733c] SyS_exit_group+0x1c/0x40
> š[ 4673.360256] š[0000000000406234] linux_sparc_syscall+0x34/0x44
> š[ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
> š[ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
> š[ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
> š[ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
> š[ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
> š[ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
> š[ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
> š[ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
> š[ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
> š[ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44
>
> š- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.
> 

Kirill, Well the change works. So far the machine is up and no stall or crashes
with Hackbench. I'll run it for longer period and check.

Thanks,

Allen

 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.
> 

Kirill, Well the change works. So far the machine is up and no stall or crashes
with Hackbench. I'll run it for longer period and check.

Thanks,

Allen

 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>    [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>    [ 1487.027885] Call Trace:
>>>>>    [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>    [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>    [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>    [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>    [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>    [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>    [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>    [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>    [ 1487.027922]  [0000000000000000]           (null)
>>>   Now, consistently I've been getting sun4v_data_access_exception.
>>>   Here's the trace:
>>>   [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>>  I've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>>  arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>>  you collect flush requests before you really flush all of them.
>>
>>  In RT you collect them too, but you are able to be preempted in any moment.
>>  So, you may switch to other process with unflushed tlb, which is very bad.
>>
>>  Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>>  We will look if this robust fix helps.
>
> Kirill, Well the change works. So far the machine is up and no stall or crashes
> with Hackbench. I'll run it for longer period and check.

Ok, good.

But I don't know is this the best fix. May we have to implement another optimization
for RT.

For example, collect only batches which does not require smp call function. Or the
main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The other serious thing is to know does __set_pte_at() execute in preemption disable
context on !RT kernel. Because the place is interesting.

If yes, we have to do the same for RT. If not, then no.

Kirill

>
> Thanks,
>
> Allen
>
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>> ššš[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>> ššš[ 1487.027885] Call Trace:
>>>>> ššš[ 1487.027887] š[00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>> ššš[ 1487.027892] š[00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>> ššš[ 1487.027895] š[0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>> ššš[ 1487.027897] š[0000000000817ebc] __schedule+0x39c/0x53c
>>>>> ššš[ 1487.027899] š[00000000008185fc] schedule+0x1c/0xc0
>>>>> ššš[ 1487.027908] š[000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>> ššš[ 1487.027913] š[000000000048753c] kthread+0x7c/0xa0
>>>>> ššš[ 1487.027920] š[00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>> ššš[ 1487.027922] š[0000000000000000] šššššššššš(null)
>>> ššNow, consistently I've been getting sun4v_data_access_exception.
>>> ššHere's the trace:
>>> šš[ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>> šI've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>> šarch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>> šyou collect flush requests before you really flush all of them.
>>
>> šIn RT you collect them too, but you are able to be preempted in any moment.
>> šSo, you may switch to other process with unflushed tlb, which is very bad.
>>
>> šTry to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>> šWe will look if this robust fix helps.
>
> Kirill, Well the change works. So far the machine is up and no stall or crashes
> with Hackbench. I'll run it for longer period and check.

Ok, good.

But I don't know is this the best fix. May we have to implement another optimization
for RT.

For example, collect only batches which does not require smp call function. Or the
main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The other serious thing is to know does __set_pte_at() execute in preemption disable
context on !RT kernel. Because the place is interesting.

If yes, we have to do the same for RT. If not, then no.

Kirill

>
> Thanks,
>
> Allen
>
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at šhttp://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
> 12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>    [ 1487.027885] Call Trace:
>>>>>>    [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>    [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>    [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>    [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>    [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>    [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>    [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>    [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>    [ 1487.027922]  [0000000000000000]           (null)

>>
>> Kirill, Well the change works. So far the machine is up and no stall or crashes
>> with Hackbench. I'll run it for longer period and check.
> 
> Ok, good.
> 
> But I don't know is this the best fix. May we have to implement another optimization
> for RT.

No, unfortunately, the system hit a stall on about 8 cpu's. 
CPU: 31 PID: 28675 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
[ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000    Tainted: G      D W   
TPC: <do_raw_spin_lock+0xb4/0x120>
[ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
[ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
[ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
[ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
[ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
[ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
[ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
[ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
[ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
[ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
[ 5725.097692] Call Trace:
[ 5725.097697]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5725.097702]  [0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
[ 5725.097707]  [00000000008790ac] rt_spin_lock+0xc/0x40
[ 5725.097712]  [00000000008610bc] unix_stream_sendmsg+0x15c/0x380
[ 5725.097717]  [00000000007ac114] sock_aio_write+0xf4/0x120
[ 5725.097722]  [000000000055891c] do_sync_write+0x5c/0xa0
[ 5725.097727]  [0000000000559e1c] vfs_write+0x15c/0x180
[ 5725.097732]  [0000000000559ef8] SyS_write+0x38/0x80
[ 5725.097738]  [0000000000406234] linux_sparc_syscall+0x34/0x44

This(above) on a few cpu's and this(below) on the other

BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
[ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
[ 5728.378347] irq event stamp: 0
[ 5728.378350] hardirqs last  enabled at (0): [<          (null)>]           (null)
[ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378361] softirqs last  enabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378364] softirqs last disabled at (0): [<          (null)>]           (null)
[ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
[ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000    Tainted: G      D W   
[ 5728.378378] TPC: <do_exit+0xb4/0xa40>
[ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
[ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
[ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
[ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
[ 5728.378390] RPC: <do_exit+0x9c/0xa40>
[ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
[ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
[ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
[ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
[ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
[ 5728.378406] Call Trace:
[ 5728.378410]  [000000000087a650] perfctr_irq+0x3d0/0x420
[ 5728.378415]  [00000000004209f4] tl0_irq15+0x14/0x20
[ 5728.378419]  [000000000042f608] stick_get_tick+0x8/0x20
[ 5728.378422]  [000000000042fa24] __delay+0x24/0x60
[ 5728.378426]  [0000000000671e58] do_raw_spin_lock+0xb8/0x120
[ 5728.378430]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5728.378435]  [00000000004a1978] load_balance+0x538/0x860
[ 5728.378438]  [00000000004a2154] idle_balance+0x134/0x1c0
[ 5728.378442]  [0000000000877d54] switch_to_pc+0x1f4/0x2c0
[ 5728.378445]  [0000000000877ec4] schedule+0x24/0xc0
[ 5728.378449]  [0000000000876860] schedule_timeout+0x1c0/0x2a0
[ 5728.378452]  [0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
[ 5728.378456]  [00000000007ac23c] sock_aio_read+0xfc/0x120
[ 5728.378460]  [0000000000558adc] do_sync_read+0x5c/0xa0
[ 5728.378464]  [000000000055a04c] vfs_read+0x10c/0x120
[ 5728.378467]  [000000000055a118] SyS_read+0x38/0x80

> 
> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..
> 
> The other serious thing is to know does __set_pte_at() execute in preemption disable
> context on !RT kernel. Because the place is interesting.
> 
> If yes, we have to do the same for RT. If not, then no.

I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

Thanks,

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
> 12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>    [ 1487.027885] Call Trace:
>>>>>>    [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>    [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>    [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>    [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>    [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>    [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>    [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>    [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>    [ 1487.027922]  [0000000000000000]           (null)

>>
>> Kirill, Well the change works. So far the machine is up and no stall or crashes
>> with Hackbench. I'll run it for longer period and check.
> 
> Ok, good.
> 
> But I don't know is this the best fix. May we have to implement another optimization
> for RT.

No, unfortunately, the system hit a stall on about 8 cpu's. 
CPU: 31 PID: 28675 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
[ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000    Tainted: G      D W   
TPC: <do_raw_spin_lock+0xb4/0x120>
[ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
[ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
[ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
[ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
[ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
[ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
[ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
[ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
[ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
[ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
[ 5725.097692] Call Trace:
[ 5725.097697]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5725.097702]  [0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
[ 5725.097707]  [00000000008790ac] rt_spin_lock+0xc/0x40
[ 5725.097712]  [00000000008610bc] unix_stream_sendmsg+0x15c/0x380
[ 5725.097717]  [00000000007ac114] sock_aio_write+0xf4/0x120
[ 5725.097722]  [000000000055891c] do_sync_write+0x5c/0xa0
[ 5725.097727]  [0000000000559e1c] vfs_write+0x15c/0x180
[ 5725.097732]  [0000000000559ef8] SyS_write+0x38/0x80
[ 5725.097738]  [0000000000406234] linux_sparc_syscall+0x34/0x44

This(above) on a few cpu's and this(below) on the other

BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
[ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
[ 5728.378347] irq event stamp: 0
[ 5728.378350] hardirqs last  enabled at (0): [<          (null)>]           (null)
[ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378361] softirqs last  enabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378364] softirqs last disabled at (0): [<          (null)>]           (null)
[ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
[ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000    Tainted: G      D W   
[ 5728.378378] TPC: <do_exit+0xb4/0xa40>
[ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
[ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
[ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
[ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
[ 5728.378390] RPC: <do_exit+0x9c/0xa40>
[ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
[ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
[ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
[ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
[ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
[ 5728.378406] Call Trace:
[ 5728.378410]  [000000000087a650] perfctr_irq+0x3d0/0x420
[ 5728.378415]  [00000000004209f4] tl0_irq15+0x14/0x20
[ 5728.378419]  [000000000042f608] stick_get_tick+0x8/0x20
[ 5728.378422]  [000000000042fa24] __delay+0x24/0x60
[ 5728.378426]  [0000000000671e58] do_raw_spin_lock+0xb8/0x120
[ 5728.378430]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5728.378435]  [00000000004a1978] load_balance+0x538/0x860
[ 5728.378438]  [00000000004a2154] idle_balance+0x134/0x1c0
[ 5728.378442]  [0000000000877d54] switch_to_pc+0x1f4/0x2c0
[ 5728.378445]  [0000000000877ec4] schedule+0x24/0xc0
[ 5728.378449]  [0000000000876860] schedule_timeout+0x1c0/0x2a0
[ 5728.378452]  [0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
[ 5728.378456]  [00000000007ac23c] sock_aio_read+0xfc/0x120
[ 5728.378460]  [0000000000558adc] do_sync_read+0x5c/0xa0
[ 5728.378464]  [000000000055a04c] vfs_read+0x10c/0x120
[ 5728.378467]  [000000000055a118] SyS_read+0x38/0x80

> 
> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..
> 
> The other serious thing is to know does __set_pte_at() execute in preemption disable
> context on !RT kernel. Because the place is interesting.
> 
> If yes, we have to do the same for RT. If not, then no.

I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

Thanks,

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>
>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)
>>>  Kirill, Well the change works. So far the machine is up and no stall or crashes
>>>  with Hackbench. I'll run it for longer period and check.
>>  Ok, good.
>>
>>  But I don't know is this the best fix. May we have to implement another optimization
>>  for RT.
>
> No, unfortunately, the system hit a stall on about 8 cpu's.
> CPU: 31 PID: 28675 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
> [ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
> [ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000    Tainted: G      D W
> TPC: <do_raw_spin_lock+0xb4/0x120>
> [ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
> [ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
> [ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
> [ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
> [ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
> [ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
> [ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
> [ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
> [ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
> [ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
> [ 5725.097692] Call Trace:
> [ 5725.097697]  [0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5725.097702]  [0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
> [ 5725.097707]  [00000000008790ac] rt_spin_lock+0xc/0x40
> [ 5725.097712]  [00000000008610bc] unix_stream_sendmsg+0x15c/0x380
> [ 5725.097717]  [00000000007ac114] sock_aio_write+0xf4/0x120
> [ 5725.097722]  [000000000055891c] do_sync_write+0x5c/0xa0
> [ 5725.097727]  [0000000000559e1c] vfs_write+0x15c/0x180
> [ 5725.097732]  [0000000000559ef8] SyS_write+0x38/0x80
> [ 5725.097738]  [0000000000406234] linux_sparc_syscall+0x34/0x44

No ideas right now.

> This(above) on a few cpu's and this(below) on the other
>
> BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
> [ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> [ 5728.378347] irq event stamp: 0
> [ 5728.378350] hardirqs last  enabled at (0): [<          (null)>]           (null)
> [ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378361] softirqs last  enabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378364] softirqs last disabled at (0): [<          (null)>]           (null)
> [ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
> [ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
> [ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000    Tainted: G      D W
> [ 5728.378378] TPC: <do_exit+0xb4/0xa40>
> [ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
> [ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
> [ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
> [ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
> [ 5728.378390] RPC: <do_exit+0x9c/0xa40>
> [ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
> [ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
> [ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
> [ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
> [ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
> [ 5728.378406] Call Trace:
> [ 5728.378410]  [000000000087a650] perfctr_irq+0x3d0/0x420
> [ 5728.378415]  [00000000004209f4] tl0_irq15+0x14/0x20
> [ 5728.378419]  [000000000042f608] stick_get_tick+0x8/0x20
> [ 5728.378422]  [000000000042fa24] __delay+0x24/0x60
> [ 5728.378426]  [0000000000671e58] do_raw_spin_lock+0xb8/0x120
> [ 5728.378430]  [0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5728.378435]  [00000000004a1978] load_balance+0x538/0x860
> [ 5728.378438]  [00000000004a2154] idle_balance+0x134/0x1c0
> [ 5728.378442]  [0000000000877d54] switch_to_pc+0x1f4/0x2c0
> [ 5728.378445]  [0000000000877ec4] schedule+0x24/0xc0
> [ 5728.378449]  [0000000000876860] schedule_timeout+0x1c0/0x2a0
> [ 5728.378452]  [0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
> [ 5728.378456]  [00000000007ac23c] sock_aio_read+0xfc/0x120
> [ 5728.378460]  [0000000000558adc] do_sync_read+0x5c/0xa0
> [ 5728.378464]  [000000000055a04c] vfs_read+0x10c/0x120
> [ 5728.378467]  [000000000055a118] SyS_read+0x38/0x80
>
>>  For example, collect only batches which does not require smp call function. Or the
>>  main goal of lazy tlb was to prevent smp calls?! It's good to discover this..
>>
>>  The other serious thing is to know does __set_pte_at() execute in preemption disable
>>  context on !RT kernel. Because the place is interesting.
>>
>>  If yes, we have to do the same for RT. If not, then no.
>
> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

++all above. May we have to add one more crutch... Put preempt_disable() at begining of
__set_pte_at() and enable at end...


> Thanks,
>
> Allen
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>
>> š12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>> šššš[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>> šššš[ 1487.027885] Call Trace:
>>>>>>> šššš[ 1487.027887] š[00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>> šššš[ 1487.027892] š[00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>> šššš[ 1487.027895] š[0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>> šššš[ 1487.027897] š[0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>> šššš[ 1487.027899] š[00000000008185fc] schedule+0x1c/0xc0
>>>>>>> šššš[ 1487.027908] š[000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>> šššš[ 1487.027913] š[000000000048753c] kthread+0x7c/0xa0
>>>>>>> šššš[ 1487.027920] š[00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>> šššš[ 1487.027922] š[0000000000000000] šššššššššš(null)
>>> šKirill, Well the change works. So far the machine is up and no stall or crashes
>>> šwith Hackbench. I'll run it for longer period and check.
>> šOk, good.
>>
>> šBut I don't know is this the best fix. May we have to implement another optimization
>> šfor RT.
>
> No, unfortunately, the system hit a stall on about 8 cpu's.
> CPU: 31 PID: 28675 Comm: hackbench Tainted: G šššššD W ššš3.10.24-rt22+ #13
> [ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
> [ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000 šššTainted: G šššššD W
> TPC: <do_raw_spin_lock+0xb4/0x120>
> [ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
> [ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
> [ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
> [ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
> [ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
> [ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
> [ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
> [ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
> [ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
> [ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
> [ 5725.097692] Call Trace:
> [ 5725.097697] š[0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5725.097702] š[0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
> [ 5725.097707] š[00000000008790ac] rt_spin_lock+0xc/0x40
> [ 5725.097712] š[00000000008610bc] unix_stream_sendmsg+0x15c/0x380
> [ 5725.097717] š[00000000007ac114] sock_aio_write+0xf4/0x120
> [ 5725.097722] š[000000000055891c] do_sync_write+0x5c/0xa0
> [ 5725.097727] š[0000000000559e1c] vfs_write+0x15c/0x180
> [ 5725.097732] š[0000000000559ef8] SyS_write+0x38/0x80
> [ 5725.097738] š[0000000000406234] linux_sparc_syscall+0x34/0x44

No ideas right now.

> This(above) on a few cpu's and this(below) on the other
>
> BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
> [ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> [ 5728.378347] irq event stamp: 0
> [ 5728.378350] hardirqs last šenabled at (0): [< ššššššššš(null)>] šššššššššš(null)
> [ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378361] softirqs last šenabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378364] softirqs last disabled at (0): [< ššššššššš(null)>] šššššššššš(null)
> [ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G šššššD W ššš3.10.24-rt22+ #13
> [ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
> [ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000 šššTainted: G šššššD W
> [ 5728.378378] TPC: <do_exit+0xb4/0xa40>
> [ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
> [ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
> [ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
> [ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
> [ 5728.378390] RPC: <do_exit+0x9c/0xa40>
> [ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
> [ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
> [ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
> [ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
> [ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
> [ 5728.378406] Call Trace:
> [ 5728.378410] š[000000000087a650] perfctr_irq+0x3d0/0x420
> [ 5728.378415] š[00000000004209f4] tl0_irq15+0x14/0x20
> [ 5728.378419] š[000000000042f608] stick_get_tick+0x8/0x20
> [ 5728.378422] š[000000000042fa24] __delay+0x24/0x60
> [ 5728.378426] š[0000000000671e58] do_raw_spin_lock+0xb8/0x120
> [ 5728.378430] š[0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5728.378435] š[00000000004a1978] load_balance+0x538/0x860
> [ 5728.378438] š[00000000004a2154] idle_balance+0x134/0x1c0
> [ 5728.378442] š[0000000000877d54] switch_to_pc+0x1f4/0x2c0
> [ 5728.378445] š[0000000000877ec4] schedule+0x24/0xc0
> [ 5728.378449] š[0000000000876860] schedule_timeout+0x1c0/0x2a0
> [ 5728.378452] š[0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
> [ 5728.378456] š[00000000007ac23c] sock_aio_read+0xfc/0x120
> [ 5728.378460] š[0000000000558adc] do_sync_read+0x5c/0xa0
> [ 5728.378464] š[000000000055a04c] vfs_read+0x10c/0x120
> [ 5728.378467] š[000000000055a118] SyS_read+0x38/0x80
>
>> šFor example, collect only batches which does not require smp call function. Or the
>> šmain goal of lazy tlb was to prevent smp calls?! It's good to discover this..
>>
>> šThe other serious thing is to know does __set_pte_at() execute in preemption disable
>> šcontext on !RT kernel. Because the place is interesting.
>>
>> šIf yes, we have to do the same for RT. If not, then no.
>
> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

++all above. May we have to add one more crutch... Put preempt_disable() at begining of
__set_pte_at() and enable at end...


> Thanks,
>
> Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 06:15 PM, Kirill Tkhai wrote:
> 
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)

>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...
> 
I'll look into it. Thanks again.

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 06:15 PM, Kirill Tkhai wrote:
> 
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)

>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...
> 
I'll look into it. Thanks again.

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)
>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...


 I realized locking in tsb is very tricky. My attempts to try and get hackbench run
without causing a stall failed. So here's what I tried to fix it, am not sure if it's
an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
with hackbench and dd, the system did not stall :)

diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index 9eb10b4..24dcd29 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -6,6 +6,7 @@
 #include <linux/kernel.h>
 #include <linux/preempt.h>
 #include <linux/slab.h>
+#include <linux/locallock.h>
 #include <asm/page.h>
 #include <asm/pgtable.h>
 #include <asm/mmu_context.h>
@@ -14,6 +15,7 @@
 #include <asm/oplib.h>
 
 extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
+static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
 
 static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
 {
@@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
 void flush_tsb_user(struct tlb_batch *tb)
 {
        struct mm_struct *mm = tb->mm;
-       unsigned long nentries, base, flags;
+       unsigned long nentries, base;
 
-       raw_spin_lock_irqsave(&mm->context.lock, flags);
+       local_lock(tsb_lock);
 
        base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
        nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
                __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
        }
 #endif
-       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
+       local_unlock(tsb_lock);
 }


Thanks,

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)
>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...


 I realized locking in tsb is very tricky. My attempts to try and get hackbench run
without causing a stall failed. So here's what I tried to fix it, am not sure if it's
an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
with hackbench and dd, the system did not stall :)

diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index 9eb10b4..24dcd29 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -6,6 +6,7 @@
 #include <linux/kernel.h>
 #include <linux/preempt.h>
 #include <linux/slab.h>
+#include <linux/locallock.h>
 #include <asm/page.h>
 #include <asm/pgtable.h>
 #include <asm/mmu_context.h>
@@ -14,6 +15,7 @@
 #include <asm/oplib.h>
 
 extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
+static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
 
 static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
 {
@@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
 void flush_tsb_user(struct tlb_batch *tb)
 {
        struct mm_struct *mm = tb->mm;
-       unsigned long nentries, base, flags;
+       unsigned long nentries, base;
 
-       raw_spin_lock_irqsave(&mm->context.lock, flags);
+       local_lock(tsb_lock);
 
        base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
        nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
                __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
        }
 #endif
-       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
+       local_unlock(tsb_lock);
 }


Thanks,

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 07:54, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>>  12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>>>  On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>>>   12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>>      [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>>      [ 1487.027885] Call Trace:
>>>>>>>>>      [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>>      [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>>      [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>>      [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>>      [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>>      [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>>      [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>>      [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>>      [ 1487.027922]  [0000000000000000]           (null)
>>>  I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
>>  ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
>>  __set_pte_at() and enable at end...
>
>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)
>
> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
> index 9eb10b4..24dcd29 100644
> --- a/arch/sparc/mm/tsb.c
> +++ b/arch/sparc/mm/tsb.c
> @@ -6,6 +6,7 @@
>  #include <linux/kernel.h>
>  #include <linux/preempt.h>
>  #include <linux/slab.h>
> +#include <linux/locallock.h>
>  #include <asm/page.h>
>  #include <asm/pgtable.h>
>  #include <asm/mmu_context.h>
> @@ -14,6 +15,7 @@
>  #include <asm/oplib.h>
>
>  extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>
>  static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>  {
> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>  void flush_tsb_user(struct tlb_batch *tb)
>  {
>         struct mm_struct *mm = tb->mm;
> -       unsigned long nentries, base, flags;
> +       unsigned long nentries, base;
>
> -       raw_spin_lock_irqsave(&mm->context.lock, flags);
> +       local_lock(tsb_lock);
>
>         base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>         nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>                 __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>         }
>  #endif
> -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> +       local_unlock(tsb_lock);

It seems to be not good for me. Tsb setup is in tsb_grow() and it must
be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..

Which last stack stack has you received with tb->active, permanently set to zero?

>  }
>
> Thanks,
>
> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 07:54, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>> š12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>>> šOn Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>>> šš12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>> ššššš[ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>> ššššš[ 1487.027885] Call Trace:
>>>>>>>>> ššššš[ 1487.027887] š[00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>> ššššš[ 1487.027892] š[00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>> ššššš[ 1487.027895] š[0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>> ššššš[ 1487.027897] š[0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>> ššššš[ 1487.027899] š[00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>> ššššš[ 1487.027908] š[000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>> ššššš[ 1487.027913] š[000000000048753c] kthread+0x7c/0xa0
>>>>>>>>> ššššš[ 1487.027920] š[00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>> ššššš[ 1487.027922] š[0000000000000000] šššššššššš(null)
>>> šI am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
>> š++all above. May we have to add one more crutch... Put preempt_disable() at begining of
>> š__set_pte_at() and enable at end...
>
> šI realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)
>
> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
> index 9eb10b4..24dcd29 100644
> --- a/arch/sparc/mm/tsb.c
> +++ b/arch/sparc/mm/tsb.c
> @@ -6,6 +6,7 @@
> š#include <linux/kernel.h>
> š#include <linux/preempt.h>
> š#include <linux/slab.h>
> +#include <linux/locallock.h>
> š#include <asm/page.h>
> š#include <asm/pgtable.h>
> š#include <asm/mmu_context.h>
> @@ -14,6 +15,7 @@
> š#include <asm/oplib.h>
>
> šextern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>
> šstatic inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
> š{
> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
> švoid flush_tsb_user(struct tlb_batch *tb)
> š{
> ššššššššstruct mm_struct *mm = tb->mm;
> - ššššššunsigned long nentries, base, flags;
> + ššššššunsigned long nentries, base;
>
> - ššššššraw_spin_lock_irqsave(&mm->context.lock, flags);
> + ššššššlocal_lock(tsb_lock);
>
> ššššššššbase = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
> ššššššššnentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
> šššššššššššššššš__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
> šššššššš}
> š#endif
> - ššššššraw_spin_unlock_irqrestore(&mm->context.lock, flags);
> + ššššššlocal_unlock(tsb_lock);

It seems to be not good for me. Tsb setup is in tsb_grow() and it must
be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..

Which last stack stack has you received with tb->active, permanently set to zero?

> š}
>
> Thanks,
>
> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at šhttp://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>> index 9eb10b4..24dcd29 100644
>> --- a/arch/sparc/mm/tsb.c
>> +++ b/arch/sparc/mm/tsb.c
>> @@ -6,6 +6,7 @@
>>  #include <linux/kernel.h>
>>  #include <linux/preempt.h>
>>  #include <linux/slab.h>
>> +#include <linux/locallock.h>
>>  #include <asm/page.h>
>>  #include <asm/pgtable.h>
>>  #include <asm/mmu_context.h>
>> @@ -14,6 +15,7 @@
>>  #include <asm/oplib.h>
>>
>>  extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>
>>  static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>  {
>> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>  void flush_tsb_user(struct tlb_batch *tb)
>>  {
>>         struct mm_struct *mm = tb->mm;
>> -       unsigned long nentries, base, flags;
>> +       unsigned long nentries, base;
>>
>> -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>> +       local_lock(tsb_lock);
>>
>>         base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>         nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>                 __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>         }
>>  #endif
>> -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>> +       local_unlock(tsb_lock);
> 
> It seems to be not good for me. Tsb setup is in tsb_grow() and it must
> be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
> 
> Which last stack stack has you received with tb->active, permanently set to zero?
> 

I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
a bit tricky to actually debug.

Yes, tb->active was set to zero.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>> index 9eb10b4..24dcd29 100644
>> --- a/arch/sparc/mm/tsb.c
>> +++ b/arch/sparc/mm/tsb.c
>> @@ -6,6 +6,7 @@
>>  #include <linux/kernel.h>
>>  #include <linux/preempt.h>
>>  #include <linux/slab.h>
>> +#include <linux/locallock.h>
>>  #include <asm/page.h>
>>  #include <asm/pgtable.h>
>>  #include <asm/mmu_context.h>
>> @@ -14,6 +15,7 @@
>>  #include <asm/oplib.h>
>>
>>  extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>
>>  static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>  {
>> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>  void flush_tsb_user(struct tlb_batch *tb)
>>  {
>>         struct mm_struct *mm = tb->mm;
>> -       unsigned long nentries, base, flags;
>> +       unsigned long nentries, base;
>>
>> -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>> +       local_lock(tsb_lock);
>>
>>         base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>         nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>                 __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>         }
>>  #endif
>> -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>> +       local_unlock(tsb_lock);
> 
> It seems to be not good for me. Tsb setup is in tsb_grow() and it must
> be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
> 
> Which last stack stack has you received with tb->active, permanently set to zero?
> 

I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
a bit tricky to actually debug.

Yes, tb->active was set to zero.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>  diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>  index 9eb10b4..24dcd29 100644
>>>  --- a/arch/sparc/mm/tsb.c
>>>  +++ b/arch/sparc/mm/tsb.c
>>>  @@ -6,6 +6,7 @@
>>>   #include <linux/kernel.h>
>>>   #include <linux/preempt.h>
>>>   #include <linux/slab.h>
>>>  +#include <linux/locallock.h>
>>>   #include <asm/page.h>
>>>   #include <asm/pgtable.h>
>>>   #include <asm/mmu_context.h>
>>>  @@ -14,6 +15,7 @@
>>>   #include <asm/oplib.h>
>>>
>>>   extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>  +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>
>>>   static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>   {
>>>  @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>   void flush_tsb_user(struct tlb_batch *tb)
>>>   {
>>>          struct mm_struct *mm = tb->mm;
>>>  -       unsigned long nentries, base, flags;
>>>  +       unsigned long nentries, base;
>>>
>>>  -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>  +       local_lock(tsb_lock);
>>>
>>>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>  @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>          }
>>>   #endif
>>>  -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>  +       local_unlock(tsb_lock);
>>  It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>  be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>
>>  Which last stack stack has you received with tb->active, permanently set to zero?
>
> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
> a bit tricky to actually debug.
>
> Yes, tb->active was set to zero.

If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.

> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>> šdiff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>> šindex 9eb10b4..24dcd29 100644
>>> š--- a/arch/sparc/mm/tsb.c
>>> š+++ b/arch/sparc/mm/tsb.c
>>> š@@ -6,6 +6,7 @@
>>> šš#include <linux/kernel.h>
>>> šš#include <linux/preempt.h>
>>> šš#include <linux/slab.h>
>>> š+#include <linux/locallock.h>
>>> šš#include <asm/page.h>
>>> šš#include <asm/pgtable.h>
>>> šš#include <asm/mmu_context.h>
>>> š@@ -14,6 +15,7 @@
>>> šš#include <asm/oplib.h>
>>>
>>> ššextern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>> š+static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>
>>> ššstatic inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>> šš{
>>> š@@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>> ššvoid flush_tsb_user(struct tlb_batch *tb)
>>> šš{
>>> šššššššššstruct mm_struct *mm = tb->mm;
>>> š- ššššššunsigned long nentries, base, flags;
>>> š+ ššššššunsigned long nentries, base;
>>>
>>> š- ššššššraw_spin_lock_irqsave(&mm->context.lock, flags);
>>> š+ ššššššlocal_lock(tsb_lock);
>>>
>>> šššššššššbase = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>> šššššššššnentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>> š@@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>> ššššššššššššššššš__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>> ššššššššš}
>>> šš#endif
>>> š- ššššššraw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>> š+ ššššššlocal_unlock(tsb_lock);
>> šIt seems to be not good for me. Tsb setup is in tsb_grow() and it must
>> šbe synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>
>> šWhich last stack stack has you received with tb->active, permanently set to zero?
>
> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
> a bit tricky to actually debug.
>
> Yes, tb->active was set to zero.

If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.

> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at šhttp://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
> 
> 
> 19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>  diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>  index 9eb10b4..24dcd29 100644
>>>>  --- a/arch/sparc/mm/tsb.c
>>>>  +++ b/arch/sparc/mm/tsb.c
>>>>  @@ -6,6 +6,7 @@
>>>>   #include <linux/kernel.h>
>>>>   #include <linux/preempt.h>
>>>>   #include <linux/slab.h>
>>>>  +#include <linux/locallock.h>
>>>>   #include <asm/page.h>
>>>>   #include <asm/pgtable.h>
>>>>   #include <asm/mmu_context.h>
>>>>  @@ -14,6 +15,7 @@
>>>>   #include <asm/oplib.h>
>>>>
>>>>   extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>  +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>
>>>>   static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>   {
>>>>  @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>   void flush_tsb_user(struct tlb_batch *tb)
>>>>   {
>>>>          struct mm_struct *mm = tb->mm;
>>>>  -       unsigned long nentries, base, flags;
>>>>  +       unsigned long nentries, base;
>>>>
>>>>  -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>  +       local_lock(tsb_lock);
>>>>
>>>>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>  @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>          }
>>>>   #endif
>>>>  -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>  +       local_unlock(tsb_lock);
>>>  It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>  be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>
>>>  Which last stack stack has you received with tb->active, permanently set to zero?
>>
>> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>> a bit tricky to actually debug.
>>
>> Yes, tb->active was set to zero.
> 
> If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>
Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
> 
> 
> 19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>  diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>  index 9eb10b4..24dcd29 100644
>>>>  --- a/arch/sparc/mm/tsb.c
>>>>  +++ b/arch/sparc/mm/tsb.c
>>>>  @@ -6,6 +6,7 @@
>>>>   #include <linux/kernel.h>
>>>>   #include <linux/preempt.h>
>>>>   #include <linux/slab.h>
>>>>  +#include <linux/locallock.h>
>>>>   #include <asm/page.h>
>>>>   #include <asm/pgtable.h>
>>>>   #include <asm/mmu_context.h>
>>>>  @@ -14,6 +15,7 @@
>>>>   #include <asm/oplib.h>
>>>>
>>>>   extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>  +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>
>>>>   static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>   {
>>>>  @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>   void flush_tsb_user(struct tlb_batch *tb)
>>>>   {
>>>>          struct mm_struct *mm = tb->mm;
>>>>  -       unsigned long nentries, base, flags;
>>>>  +       unsigned long nentries, base;
>>>>
>>>>  -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>  +       local_lock(tsb_lock);
>>>>
>>>>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>  @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>          }
>>>>   #endif
>>>>  -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>  +       local_unlock(tsb_lock);
>>>  It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>  be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>
>>>  Which last stack stack has you received with tb->active, permanently set to zero?
>>
>> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>> a bit tricky to actually debug.
>>
>> Yes, tb->active was set to zero.
> 
> If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>
Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:

>  On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>>   19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>>    diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>>    index 9eb10b4..24dcd29 100644
>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>    @@ -6,6 +6,7 @@
>>>>>     #include <linux/kernel.h>
>>>>>     #include <linux/preempt.h>
>>>>>     #include <linux/slab.h>
>>>>>    +#include <linux/locallock.h>
>>>>>     #include <asm/page.h>
>>>>>     #include <asm/pgtable.h>
>>>>>     #include <asm/mmu_context.h>
>>>>>    @@ -14,6 +15,7 @@
>>>>>     #include <asm/oplib.h>
>>>>>
>>>>>     extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>>    +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>
>>>>>     static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>>     {
>>>>>    @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>>     void flush_tsb_user(struct tlb_batch *tb)
>>>>>     {
>>>>>            struct mm_struct *mm = tb->mm;
>>>>>    -       unsigned long nentries, base, flags;
>>>>>    +       unsigned long nentries, base;
>>>>>
>>>>>    -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>>    +       local_lock(tsb_lock);
>>>>>
>>>>>            base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>>            nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>>    @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>>                    __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>>            }
>>>>>     #endif
>>>>>    -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>>    +       local_unlock(tsb_lock);
>>>>    It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>>    be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>
>>>>    Which last stack stack has you received with tb->active, permanently set to zero?
>>>   I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>>   a bit tricky to actually debug.
>>>
>>>   Yes, tb->active was set to zero.
>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
You may get good stack without sun4v_data_access_exception error, which was in the first or second
message.

>  - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:

> šOn Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>> šš19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>> šššdiff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>> šššindex 9eb10b4..24dcd29 100644
>>>>> ššš--- a/arch/sparc/mm/tsb.c
>>>>> ššš+++ b/arch/sparc/mm/tsb.c
>>>>> ššš@@ -6,6 +6,7 @@
>>>>> šššš#include <linux/kernel.h>
>>>>> šššš#include <linux/preempt.h>
>>>>> šššš#include <linux/slab.h>
>>>>> ššš+#include <linux/locallock.h>
>>>>> šššš#include <asm/page.h>
>>>>> šššš#include <asm/pgtable.h>
>>>>> šššš#include <asm/mmu_context.h>
>>>>> ššš@@ -14,6 +15,7 @@
>>>>> šššš#include <asm/oplib.h>
>>>>>
>>>>> ššššextern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>> ššš+static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>
>>>>> ššššstatic inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>> šššš{
>>>>> ššš@@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>> ššššvoid flush_tsb_user(struct tlb_batch *tb)
>>>>> šššš{
>>>>> šššššššššššstruct mm_struct *mm = tb->mm;
>>>>> ššš- ššššššunsigned long nentries, base, flags;
>>>>> ššš+ ššššššunsigned long nentries, base;
>>>>>
>>>>> ššš- ššššššraw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>> ššš+ ššššššlocal_lock(tsb_lock);
>>>>>
>>>>> šššššššššššbase = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>> šššššššššššnentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>> ššš@@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>> ššššššššššššššššššš__flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>> ššššššššššš}
>>>>> šššš#endif
>>>>> ššš- ššššššraw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>> ššš+ ššššššlocal_unlock(tsb_lock);
>>>> šššIt seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>> šššbe synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>
>>>> šššWhich last stack stack has you received with tb->active, permanently set to zero?
>>> ššI agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>> šša bit tricky to actually debug.
>>>
>>> ššYes, tb->active was set to zero.
>> ššIf tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
> šSorry, my bad. tb->active was set to one when I ran the test with the above patch.

It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
You may get good stack without sun4v_data_access_exception error, which was in the first or second
message.

> š- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:55 PM, Kirill Tkhai wrote:
> 19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>>>   19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>>>    index 9eb10b4..24dcd29 100644
>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>
>>>>>>
>>>>>>     extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>>>    +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>>
>>>>>>     static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>>>     {
>>>>>>    @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>>>     void flush_tsb_user(struct tlb_batch *tb)
>>>>>>     {
>>>>>>            struct mm_struct *mm = tb->mm;
>>>>>>    -       unsigned long nentries, base, flags;
>>>>>>    +       unsigned long nentries, base;
>>>>>>
>>>>>>    -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>>>    +       local_lock(tsb_lock);
>>>>>>
>>>>>>            base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>>>            nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>>>    @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>>>                    __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>>>            }
>>>>>>     #endif
>>>>>>    -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>>>    +       local_unlock(tsb_lock);
>>>>>    It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>>>    be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>>
>>>>>    Which last stack stack has you received with tb->active, permanently set to zero?
>>>>   I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>>>   a bit tricky to actually debug.
>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.
> 
> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

I agree, I hope to get more a few opinions on the patches/issues. Meanwhile, Lemme see if I could
manage a way to stick to the old protectors and fix the stalls.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:55 PM, Kirill Tkhai wrote:
> 19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>>>   19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>>>    index 9eb10b4..24dcd29 100644
>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>
>>>>>>
>>>>>>     extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>>>    +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>>
>>>>>>     static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>>>     {
>>>>>>    @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>>>     void flush_tsb_user(struct tlb_batch *tb)
>>>>>>     {
>>>>>>            struct mm_struct *mm = tb->mm;
>>>>>>    -       unsigned long nentries, base, flags;
>>>>>>    +       unsigned long nentries, base;
>>>>>>
>>>>>>    -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>>>    +       local_lock(tsb_lock);
>>>>>>
>>>>>>            base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>>>            nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>>>    @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>>>                    __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>>>            }
>>>>>>     #endif
>>>>>>    -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>>>    +       local_unlock(tsb_lock);
>>>>>    It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>>>    be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>>
>>>>>    Which last stack stack has you received with tb->active, permanently set to zero?
>>>>   I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>>>   a bit tricky to actually debug.
>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.
> 
> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

I agree, I hope to get more a few opinions on the patches/issues. Meanwhile, Lemme see if I could
manage a way to stick to the old protectors and fix the stalls.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,

>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>

>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

  The CPU now does not stall, the change I did was remove debug lockdep from the config.
Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.

1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
[53990.477387] kernel BUG at kernel/rtmutex.c:738!
[53990.477393]               \|/ ____ \|/
[53990.477393]               "@'/ .. \`@"
[53990.477393]               /_| \__/ |_\
[53990.477393]                  \__U_/
[53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
[53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
[53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000    Tainted: G      D W   
[53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
[53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
[53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
[53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
[53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
[53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
[53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
[53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
[53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
[53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
[53990.477462] I7: <rt_spin_lock+0x20/0x60>
[53990.477464] Call Trace:
[53990.477470]  [0000000000876ee0] rt_spin_lock+0x20/0x60
[53990.477476]  [000000000052ee60] unmap_single_vma+0x200/0x6c0
[53990.477482]  [000000000052f348] unmap_vmas+0x28/0x60
[53990.477488]  [0000000000531868] exit_mmap+0x88/0x160
[53990.477492]  [000000000045e0e8] mmput+0x48/0x100
[53990.477496]  [0000000000466a3c] do_exit+0x1fc/0xa40
[53990.477500]  [0000000000427f00] die_if_kernel+0x1a0/0x340
[53990.477506]  [00000000004294a8] sun4v_data_access_exception+0x108/0x120
[53990.477512]  [0000000000406c08] sun4v_dacc+0x28/0x34
[53990.477517]  [0000000000407b64] tsb_flush+0x4/0x40
[53990.477523]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53990.477528]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53990.477534]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53990.477539]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53990.477544]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53990.477548]  [000000000052cab8] handle_mm_fault+0xd8/0x160

and

2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
[53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
[53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000    Tainted: G      D W   
[53998.070236] TPC: <stick_get_tick+0x8/0x20>
[53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
[53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
[53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
[53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
[53998.070264] RPC: <tl0_irq15+0x14/0x20>
[53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
[53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
[53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
[53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
[53998.070280] I7: <__delay+0x10/0x60>
[53998.070282] Call Trace:
[53998.070286]  [000000000042fa10] __delay+0x10/0x60
[53998.070291]  [000000000066f8b8] do_raw_spin_lock+0xb8/0x120
[53998.070300]  [0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
[53998.070306]  [0000000000452074] flush_tsb_user+0x14/0x120
[53998.070309]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53998.070312]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53998.070325]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53998.070328]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53998.070331]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53998.070334]  [000000000052cab8] handle_mm_fault+0xd8/0x160
[53998.070339]  [0000000000879724] do_sparc64_fault+0x404/0x700
[53998.070342]  [0000000000407ae0] sparc64_realfault_common+0x10/0x20



But strangely, during boot-up I have more crash messages. 
Here's what I see

[  520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
tex.c:659
[  520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
[  520.570803] INFO: lockdep is turned off.
[  520.570805] irq event stamp: 4502
[  520.570806] hardirqs last  enabled at (4501): [<00000000004d68c4>] rcu_note_c
ontext_switch+0xa4/0x300
[  520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
lock_irq+0x10/0x80
[  520.570822] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  520.570828] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G        W    3.10.24-r
t22+ #25
[  520.570835] Call Trace:
[  520.570842]  [0000000000495f0c] __might_sleep+0xec/0x160
[  520.570846]  [0000000000876ed8] rt_spin_lock+0x18/0x60
[  520.570852]  [00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
[  520.570855]  [00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
[  520.570858]  [0000000000462a14] console_unlock+0x394/0x400
[  520.570861]  [0000000000463108] vprintk_emit+0x3a8/0x5a0
[  520.570863]  [0000000000874378] printk+0x38/0x4c
[  520.570874]  [000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
sas]
[  520.570882]  [0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
[  520.570893]  [000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
[  520.570900]  [0000000000686120] local_pci_probe+0x20/0x40
[  520.570903]  [000000000068680c] pci_device_probe+0xec/0x100
[  520.570907]  [00000000006ee574] driver_probe_device+0x74/0x220
[  520.570909]  [00000000006ee7a8] __driver_attach+0x88/0xa0
[  520.570913]  [00000000006eca0c] bus_for_each_dev+0x6c/0xa0
[  520.570916]  [00000000006ee39c] driver_attach+0x1c/0x40


and this one 

[  519.160755] =================================
[  519.160756] [ INFO: inconsistent lock state ]
[  519.160760] 3.10.24-rt22+ #25 Not tainted
[  519.160761] ---------------------------------
[  519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
[  519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
[  519.160778]  (&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
simple_irq+0xc/0xe0
[  519.160779] {IN-HARDIRQ-W} state was registered at:
[  519.160785]   [<00000000004ba7c0>] lock_acquire+0x60/0x100
[  519.160791]   [<0000000000877930>] _raw_spin_lock+0x30/0x80
[  519.160795]   [<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
[  519.160800]   [<00000000004cf118>] generic_handle_irq+0x38/0x60
[  519.160804]   [<000000000087bf44>] handler_irq+0xc4/0x100
[  519.160808]   [<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
[  519.160812]   [<000000000042ced4>] arch_cpu_idle+0x54/0xe0
[  519.160817]   [<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
[  519.160822]   [<0000000000870f18>] smp_callin+0x100/0x110
[  519.160825]   [<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
[  519.160827]   [<          (null)>]           (null)
[  519.160829] irq event stamp: 19
[  519.160832] hardirqs last  enabled at (19): [<0000000000877cc4>] _raw_spin_un
lock_irq+0x24/0x60
[  519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
ck_irq+0x10/0x80
[  519.160841] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  519.160843] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  519.160844] 
[  519.160844] other info that might help us debug this:
[  519.160844]  Possible unsafe locking scenario:
[  519.160844] 
[  519.160845]        CPU0
[  519.160845]        ----
[  519.160847]   lock(&irq_desc_lock_class);
[  519.160848]   <Interrupt>
[  519.160850]     lock(&irq_desc_lock_class);
[  519.160850] 
[  519.160850]  *** DEADLOCK ***
[  519.160850] 
[  519.160852] no locks held by irq/36-MSIQ/640.
[  519.160853] 
[  519.160853] stack backtrace:
[  519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
[  519.160856] Call Trace:
[  519.160860]  [00000000004b50b4] print_usage_bug+0x234/0x2e0
[  519.160862]  [00000000004b5728] mark_lock+0x5c8/0x800
[  519.160864]  [00000000004ba238] __lock_acquire+0x7b8/0xce0
[  519.160866]  [00000000004ba7c0] lock_acquire+0x60/0x100
[  519.160868]  [0000000000877930] _raw_spin_lock+0x30/0x80
[  519.160870]  [00000000004d314c] handle_simple_irq+0xc/0xe0
[  519.160872]  [00000000004cf118] generic_handle_irq+0x38/0x60
[  519.160877]  [0000000000447870] sparc64_msiq_interrupt+0x50/0x120
[  519.160880]  [00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
[  519.160883]  [00000000004d019c] irq_thread+0xdc/0x140
[  519.160888]  [0000000000489560] kthread+0x80/0xa0
[  519.160893]  [0000000000406104] ret_from_syscall+0x1c/0x2c
[  519.160894]  [0000000000000000]           (null)
[  519.160897] ------------[ cut here ]------------


what do you think?

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,

>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>

>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

  The CPU now does not stall, the change I did was remove debug lockdep from the config.
Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.

1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
[53990.477387] kernel BUG at kernel/rtmutex.c:738!
[53990.477393]               \|/ ____ \|/
[53990.477393]               "@'/ .. \`@"
[53990.477393]               /_| \__/ |_\
[53990.477393]                  \__U_/
[53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
[53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
[53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000    Tainted: G      D W   
[53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
[53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
[53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
[53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
[53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
[53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
[53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
[53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
[53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
[53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
[53990.477462] I7: <rt_spin_lock+0x20/0x60>
[53990.477464] Call Trace:
[53990.477470]  [0000000000876ee0] rt_spin_lock+0x20/0x60
[53990.477476]  [000000000052ee60] unmap_single_vma+0x200/0x6c0
[53990.477482]  [000000000052f348] unmap_vmas+0x28/0x60
[53990.477488]  [0000000000531868] exit_mmap+0x88/0x160
[53990.477492]  [000000000045e0e8] mmput+0x48/0x100
[53990.477496]  [0000000000466a3c] do_exit+0x1fc/0xa40
[53990.477500]  [0000000000427f00] die_if_kernel+0x1a0/0x340
[53990.477506]  [00000000004294a8] sun4v_data_access_exception+0x108/0x120
[53990.477512]  [0000000000406c08] sun4v_dacc+0x28/0x34
[53990.477517]  [0000000000407b64] tsb_flush+0x4/0x40
[53990.477523]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53990.477528]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53990.477534]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53990.477539]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53990.477544]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53990.477548]  [000000000052cab8] handle_mm_fault+0xd8/0x160

and

2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
[53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
[53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000    Tainted: G      D W   
[53998.070236] TPC: <stick_get_tick+0x8/0x20>
[53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
[53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
[53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
[53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
[53998.070264] RPC: <tl0_irq15+0x14/0x20>
[53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
[53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
[53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
[53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
[53998.070280] I7: <__delay+0x10/0x60>
[53998.070282] Call Trace:
[53998.070286]  [000000000042fa10] __delay+0x10/0x60
[53998.070291]  [000000000066f8b8] do_raw_spin_lock+0xb8/0x120
[53998.070300]  [0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
[53998.070306]  [0000000000452074] flush_tsb_user+0x14/0x120
[53998.070309]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53998.070312]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53998.070325]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53998.070328]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53998.070331]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53998.070334]  [000000000052cab8] handle_mm_fault+0xd8/0x160
[53998.070339]  [0000000000879724] do_sparc64_fault+0x404/0x700
[53998.070342]  [0000000000407ae0] sparc64_realfault_common+0x10/0x20



But strangely, during boot-up I have more crash messages. 
Here's what I see

[  520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
tex.c:659
[  520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
[  520.570803] INFO: lockdep is turned off.
[  520.570805] irq event stamp: 4502
[  520.570806] hardirqs last  enabled at (4501): [<00000000004d68c4>] rcu_note_c
ontext_switch+0xa4/0x300
[  520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
lock_irq+0x10/0x80
[  520.570822] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  520.570828] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G        W    3.10.24-r
t22+ #25
[  520.570835] Call Trace:
[  520.570842]  [0000000000495f0c] __might_sleep+0xec/0x160
[  520.570846]  [0000000000876ed8] rt_spin_lock+0x18/0x60
[  520.570852]  [00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
[  520.570855]  [00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
[  520.570858]  [0000000000462a14] console_unlock+0x394/0x400
[  520.570861]  [0000000000463108] vprintk_emit+0x3a8/0x5a0
[  520.570863]  [0000000000874378] printk+0x38/0x4c
[  520.570874]  [000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
sas]
[  520.570882]  [0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
[  520.570893]  [000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
[  520.570900]  [0000000000686120] local_pci_probe+0x20/0x40
[  520.570903]  [000000000068680c] pci_device_probe+0xec/0x100
[  520.570907]  [00000000006ee574] driver_probe_device+0x74/0x220
[  520.570909]  [00000000006ee7a8] __driver_attach+0x88/0xa0
[  520.570913]  [00000000006eca0c] bus_for_each_dev+0x6c/0xa0
[  520.570916]  [00000000006ee39c] driver_attach+0x1c/0x40


and this one 

[  519.160755] ================[  519.160756] [ INFO: inconsistent lock state ]
[  519.160760] 3.10.24-rt22+ #25 Not tainted
[  519.160761] ---------------------------------
[  519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
[  519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
[  519.160778]  (&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
simple_irq+0xc/0xe0
[  519.160779] {IN-HARDIRQ-W} state was registered at:
[  519.160785]   [<00000000004ba7c0>] lock_acquire+0x60/0x100
[  519.160791]   [<0000000000877930>] _raw_spin_lock+0x30/0x80
[  519.160795]   [<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
[  519.160800]   [<00000000004cf118>] generic_handle_irq+0x38/0x60
[  519.160804]   [<000000000087bf44>] handler_irq+0xc4/0x100
[  519.160808]   [<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
[  519.160812]   [<000000000042ced4>] arch_cpu_idle+0x54/0xe0
[  519.160817]   [<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
[  519.160822]   [<0000000000870f18>] smp_callin+0x100/0x110
[  519.160825]   [<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
[  519.160827]   [<          (null)>]           (null)
[  519.160829] irq event stamp: 19
[  519.160832] hardirqs last  enabled at (19): [<0000000000877cc4>] _raw_spin_un
lock_irq+0x24/0x60
[  519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
ck_irq+0x10/0x80
[  519.160841] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  519.160843] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  519.160844] 
[  519.160844] other info that might help us debug this:
[  519.160844]  Possible unsafe locking scenario:
[  519.160844] 
[  519.160845]        CPU0
[  519.160845]        ----
[  519.160847]   lock(&irq_desc_lock_class);
[  519.160848]   <Interrupt>
[  519.160850]     lock(&irq_desc_lock_class);
[  519.160850] 
[  519.160850]  *** DEADLOCK ***
[  519.160850] 
[  519.160852] no locks held by irq/36-MSIQ/640.
[  519.160853] 
[  519.160853] stack backtrace:
[  519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
[  519.160856] Call Trace:
[  519.160860]  [00000000004b50b4] print_usage_bug+0x234/0x2e0
[  519.160862]  [00000000004b5728] mark_lock+0x5c8/0x800
[  519.160864]  [00000000004ba238] __lock_acquire+0x7b8/0xce0
[  519.160866]  [00000000004ba7c0] lock_acquire+0x60/0x100
[  519.160868]  [0000000000877930] _raw_spin_lock+0x30/0x80
[  519.160870]  [00000000004d314c] handle_simple_irq+0xc/0xe0
[  519.160872]  [00000000004cf118] generic_handle_irq+0x38/0x60
[  519.160877]  [0000000000447870] sparc64_msiq_interrupt+0x50/0x120
[  519.160880]  [00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
[  519.160883]  [00000000004d019c] irq_thread+0xdc/0x140
[  519.160888]  [0000000000489560] kthread+0x80/0xa0
[  519.160893]  [0000000000406104] ret_from_syscall+0x1c/0x2c
[  519.160894]  [0000000000000000]           (null)
[  519.160897] ------------[ cut here ]------------


what do you think?

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

26.02.2014, 11:52, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>>>>>>>     --- a/arch/sparc/mm/tsb.c
>>>>>>>     +++ b/arch/sparc/mm/tsb.c
>>>>>>>     @@ -6,6 +6,7 @@
>>>>>>>      #include <linux/kernel.h>
>>>>>>>      #include <linux/preempt.h>
>>>>>>>      #include <linux/slab.h>
>>>>>>>     +#include <linux/locallock.h>
>>>>>>>      #include <asm/page.h>
>>>>>>>      #include <asm/pgtable.h>
>>>>>>>      #include <asm/mmu_context.h>
>>>>>>>     @@ -14,6 +15,7 @@
>>>>>>>      #include <asm/oplib.h>
>>>>>    Yes, tb->active was set to zero.
>>>>    If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>>   Sorry, my bad. tb->active was set to one when I ran the test with the above patch.
>
>   The CPU now does not stall, the change I did was remove debug lockdep from the config.
> Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.
>
> 1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
> [53990.477387] kernel BUG at kernel/rtmutex.c:738!
> [53990.477393]               \|/ ____ \|/
> [53990.477393]               "@'/ .. \`@"
> [53990.477393]               /_| \__/ |_\
> [53990.477393]                  \__U_/
> [53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
> [53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
> [53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
> [53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000    Tainted: G      D W
> [53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
> [53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
> [53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
> [53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
> [53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
> [53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
> [53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
> [53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
> [53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
> [53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
> [53990.477462] I7: <rt_spin_lock+0x20/0x60>
> [53990.477464] Call Trace:
> [53990.477470]  [0000000000876ee0] rt_spin_lock+0x20/0x60
> [53990.477476]  [000000000052ee60] unmap_single_vma+0x200/0x6c0
> [53990.477482]  [000000000052f348] unmap_vmas+0x28/0x60
> [53990.477488]  [0000000000531868] exit_mmap+0x88/0x160
> [53990.477492]  [000000000045e0e8] mmput+0x48/0x100
> [53990.477496]  [0000000000466a3c] do_exit+0x1fc/0xa40
> [53990.477500]  [0000000000427f00] die_if_kernel+0x1a0/0x340
> [53990.477506]  [00000000004294a8] sun4v_data_access_exception+0x108/0x120
> [53990.477512]  [0000000000406c08] sun4v_dacc+0x28/0x34
> [53990.477517]  [0000000000407b64] tsb_flush+0x4/0x40
> [53990.477523]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53990.477528]  [0000000000451800] tlb_batch_add+0x1e0/0x200
> [53990.477534]  [000000000053cad8] ptep_clear_flush+0x38/0x60
> [53990.477539]  [000000000052b47c] do_wp_page+0x1dc/0x880
> [53990.477544]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53990.477548]  [000000000052cab8] handle_mm_fault+0xd8/0x160
>
> and
>
> 2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
> [53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
> [53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
> [53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000    Tainted: G      D W
> [53998.070236] TPC: <stick_get_tick+0x8/0x20>
> [53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
> [53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
> [53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
> [53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
> [53998.070264] RPC: <tl0_irq15+0x14/0x20>
> [53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
> [53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
> [53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
> [53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
> [53998.070280] I7: <__delay+0x10/0x60>
> [53998.070282] Call Trace:
> [53998.070286]  [000000000042fa10] __delay+0x10/0x60
> [53998.070291]  [000000000066f8b8] do_raw_spin_lock+0xb8/0x120
> [53998.070300]  [0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
> [53998.070306]  [0000000000452074] flush_tsb_user+0x14/0x120
> [53998.070309]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53998.070312]  [0000000000451800] tlb_batch_add+0x1e0/0x200
> [53998.070325]  [000000000053cad8] ptep_clear_flush+0x38/0x60
> [53998.070328]  [000000000052b47c] do_wp_page+0x1dc/0x880
> [53998.070331]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53998.070334]  [000000000052cab8] handle_mm_fault+0xd8/0x160
> [53998.070339]  [0000000000879724] do_sparc64_fault+0x404/0x700
> [53998.070342]  [0000000000407ae0] sparc64_realfault_common+0x10/0x20
>
> But strangely, during boot-up I have more crash messages.
> Here's what I see
>
> [  520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
> tex.c:659
> [  520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
> [  520.570803] INFO: lockdep is turned off.
> [  520.570805] irq event stamp: 4502
> [  520.570806] hardirqs last  enabled at (4501): [<00000000004d68c4>] rcu_note_c
> ontext_switch+0xa4/0x300
> [  520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
> lock_irq+0x10/0x80
> [  520.570822] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [  520.570828] softirqs last disabled at (0): [<          (null)>]           (nu
> ll)
> [  520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G        W    3.10.24-r
> t22+ #25
> [  520.570835] Call Trace:
> [  520.570842]  [0000000000495f0c] __might_sleep+0xec/0x160
> [  520.570846]  [0000000000876ed8] rt_spin_lock+0x18/0x60
> [  520.570852]  [00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
> [  520.570855]  [00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
> [  520.570858]  [0000000000462a14] console_unlock+0x394/0x400
> [  520.570861]  [0000000000463108] vprintk_emit+0x3a8/0x5a0
> [  520.570863]  [0000000000874378] printk+0x38/0x4c
> [  520.570874]  [000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
> sas]
> [  520.570882]  [0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
> [  520.570893]  [000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
> [  520.570900]  [0000000000686120] local_pci_probe+0x20/0x40
> [  520.570903]  [000000000068680c] pci_device_probe+0xec/0x100
> [  520.570907]  [00000000006ee574] driver_probe_device+0x74/0x220
> [  520.570909]  [00000000006ee7a8] __driver_attach+0x88/0xa0
> [  520.570913]  [00000000006eca0c] bus_for_each_dev+0x6c/0xa0
> [  520.570916]  [00000000006ee39c] driver_attach+0x1c/0x40

This should be fixed like that's done for 8250
[patch drivers-serial-cleanup-locking-for-rt.patch]


> and this one
>
> [  519.160755] =================================
> [  519.160756] [ INFO: inconsistent lock state ]
> [  519.160760] 3.10.24-rt22+ #25 Not tainted
> [  519.160761] ---------------------------------
> [  519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
> [  519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [  519.160778]  (&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
> simple_irq+0xc/0xe0
> [  519.160779] {IN-HARDIRQ-W} state was registered at:
> [  519.160785]   [<00000000004ba7c0>] lock_acquire+0x60/0x100
> [  519.160791]   [<0000000000877930>] _raw_spin_lock+0x30/0x80
> [  519.160795]   [<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
> [  519.160800]   [<00000000004cf118>] generic_handle_irq+0x38/0x60
> [  519.160804]   [<000000000087bf44>] handler_irq+0xc4/0x100
> [  519.160808]   [<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
> [  519.160812]   [<000000000042ced4>] arch_cpu_idle+0x54/0xe0
> [  519.160817]   [<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
> [  519.160822]   [<0000000000870f18>] smp_callin+0x100/0x110
> [  519.160825]   [<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
> [  519.160827]   [<          (null)>]           (null)
> [  519.160829] irq event stamp: 19
> [  519.160832] hardirqs last  enabled at (19): [<0000000000877cc4>] _raw_spin_un
> lock_irq+0x24/0x60
> [  519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
> ck_irq+0x10/0x80
> [  519.160841] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [  519.160843] softirqs last disabled at (0): [<          (null)>]           (nu
> ll)
> [  519.160844]
> [  519.160844] other info that might help us debug this:
> [  519.160844]  Possible unsafe locking scenario:
> [  519.160844]
> [  519.160845]        CPU0
> [  519.160845]        ----
> [  519.160847]   lock(&irq_desc_lock_class);
> [  519.160848]   <Interrupt>
> [  519.160850]     lock(&irq_desc_lock_class);
> [  519.160850]
> [  519.160850]  *** DEADLOCK ***
> [  519.160850]
> [  519.160852] no locks held by irq/36-MSIQ/640.
> [  519.160853]
> [  519.160853] stack backtrace:
> [  519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
> [  519.160856] Call Trace:
> [  519.160860]  [00000000004b50b4] print_usage_bug+0x234/0x2e0
> [  519.160862]  [00000000004b5728] mark_lock+0x5c8/0x800
> [  519.160864]  [00000000004ba238] __lock_acquire+0x7b8/0xce0
> [  519.160866]  [00000000004ba7c0] lock_acquire+0x60/0x100
> [  519.160868]  [0000000000877930] _raw_spin_lock+0x30/0x80
> [  519.160870]  [00000000004d314c] handle_simple_irq+0xc/0xe0
> [  519.160872]  [00000000004cf118] generic_handle_irq+0x38/0x60
> [  519.160877]  [0000000000447870] sparc64_msiq_interrupt+0x50/0x120
> [  519.160880]  [00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
> [  519.160883]  [00000000004d019c] irq_thread+0xdc/0x140
> [  519.160888]  [0000000000489560] kthread+0x80/0xa0
> [  519.160893]  [0000000000406104] ret_from_syscall+0x1c/0x2c
> [  519.160894]  [0000000000000000]           (null)
> [  519.160897] ------------[ cut here ]------------
>
> what do you think?
>
> - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

26.02.2014, 11:52, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>>>>>>> šššš--- a/arch/sparc/mm/tsb.c
>>>>>>> šššš+++ b/arch/sparc/mm/tsb.c
>>>>>>> šššš@@ -6,6 +6,7 @@
>>>>>>> ššššš#include <linux/kernel.h>
>>>>>>> ššššš#include <linux/preempt.h>
>>>>>>> ššššš#include <linux/slab.h>
>>>>>>> šššš+#include <linux/locallock.h>
>>>>>>> ššššš#include <asm/page.h>
>>>>>>> ššššš#include <asm/pgtable.h>
>>>>>>> ššššš#include <asm/mmu_context.h>
>>>>>>> šššš@@ -14,6 +15,7 @@
>>>>>>> ššššš#include <asm/oplib.h>
>>>>> šššYes, tb->active was set to zero.
>>>> šššIf tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>> ššSorry, my bad. tb->active was set to one when I ran the test with the above patch.
>
> ššThe CPU now does not stall, the change I did was remove debug lockdep from the config.
> Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.
>
> 1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
> [53990.477387] kernel BUG at kernel/rtmutex.c:738!
> [53990.477393] šššššššššššššš\|/ ____ \|/
> [53990.477393] šššššššššššššš"@'/ .. \`@"
> [53990.477393] šššššššššššššš/_| \__/ |_\
> [53990.477393] ššššššššššššššššš\__U_/
> [53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
> [53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G šššššD W ššš3.10.24-rt22+ #25
> [53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
> [53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000 šššTainted: G šššššD W
> [53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
> [53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
> [53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
> [53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
> [53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
> [53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
> [53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
> [53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
> [53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
> [53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
> [53990.477462] I7: <rt_spin_lock+0x20/0x60>
> [53990.477464] Call Trace:
> [53990.477470] š[0000000000876ee0] rt_spin_lock+0x20/0x60
> [53990.477476] š[000000000052ee60] unmap_single_vma+0x200/0x6c0
> [53990.477482] š[000000000052f348] unmap_vmas+0x28/0x60
> [53990.477488] š[0000000000531868] exit_mmap+0x88/0x160
> [53990.477492] š[000000000045e0e8] mmput+0x48/0x100
> [53990.477496] š[0000000000466a3c] do_exit+0x1fc/0xa40
> [53990.477500] š[0000000000427f00] die_if_kernel+0x1a0/0x340
> [53990.477506] š[00000000004294a8] sun4v_data_access_exception+0x108/0x120
> [53990.477512] š[0000000000406c08] sun4v_dacc+0x28/0x34
> [53990.477517] š[0000000000407b64] tsb_flush+0x4/0x40
> [53990.477523] š[00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53990.477528] š[0000000000451800] tlb_batch_add+0x1e0/0x200
> [53990.477534] š[000000000053cad8] ptep_clear_flush+0x38/0x60
> [53990.477539] š[000000000052b47c] do_wp_page+0x1dc/0x880
> [53990.477544] š[000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53990.477548] š[000000000052cab8] handle_mm_fault+0xd8/0x160
>
> and
>
> 2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
> [53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G šššššD W ššš3.10.24-rt22+ #25
> [53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
> [53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000 šššTainted: G šššššD W
> [53998.070236] TPC: <stick_get_tick+0x8/0x20>
> [53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
> [53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
> [53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
> [53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
> [53998.070264] RPC: <tl0_irq15+0x14/0x20>
> [53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
> [53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
> [53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
> [53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
> [53998.070280] I7: <__delay+0x10/0x60>
> [53998.070282] Call Trace:
> [53998.070286] š[000000000042fa10] __delay+0x10/0x60
> [53998.070291] š[000000000066f8b8] do_raw_spin_lock+0xb8/0x120
> [53998.070300] š[0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
> [53998.070306] š[0000000000452074] flush_tsb_user+0x14/0x120
> [53998.070309] š[00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53998.070312] š[0000000000451800] tlb_batch_add+0x1e0/0x200
> [53998.070325] š[000000000053cad8] ptep_clear_flush+0x38/0x60
> [53998.070328] š[000000000052b47c] do_wp_page+0x1dc/0x880
> [53998.070331] š[000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53998.070334] š[000000000052cab8] handle_mm_fault+0xd8/0x160
> [53998.070339] š[0000000000879724] do_sparc64_fault+0x404/0x700
> [53998.070342] š[0000000000407ae0] sparc64_realfault_common+0x10/0x20
>
> But strangely, during boot-up I have more crash messages.
> Here's what I see
>
> [ š520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
> tex.c:659
> [ š520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
> [ š520.570803] INFO: lockdep is turned off.
> [ š520.570805] irq event stamp: 4502
> [ š520.570806] hardirqs last šenabled at (4501): [<00000000004d68c4>] rcu_note_c
> ontext_switch+0xa4/0x300
> [ š520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
> lock_irq+0x10/0x80
> [ š520.570822] softirqs last šenabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [ š520.570828] softirqs last disabled at (0): [< ššššššššš(null)>] šššššššššš(nu
> ll)
> [ š520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G šššššššW ššš3.10.24-r
> t22+ #25
> [ š520.570835] Call Trace:
> [ š520.570842] š[0000000000495f0c] __might_sleep+0xec/0x160
> [ š520.570846] š[0000000000876ed8] rt_spin_lock+0x18/0x60
> [ š520.570852] š[00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
> [ š520.570855] š[00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
> [ š520.570858] š[0000000000462a14] console_unlock+0x394/0x400
> [ š520.570861] š[0000000000463108] vprintk_emit+0x3a8/0x5a0
> [ š520.570863] š[0000000000874378] printk+0x38/0x4c
> [ š520.570874] š[000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
> sas]
> [ š520.570882] š[0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
> [ š520.570893] š[000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
> [ š520.570900] š[0000000000686120] local_pci_probe+0x20/0x40
> [ š520.570903] š[000000000068680c] pci_device_probe+0xec/0x100
> [ š520.570907] š[00000000006ee574] driver_probe_device+0x74/0x220
> [ š520.570909] š[00000000006ee7a8] __driver_attach+0x88/0xa0
> [ š520.570913] š[00000000006eca0c] bus_for_each_dev+0x6c/0xa0
> [ š520.570916] š[00000000006ee39c] driver_attach+0x1c/0x40

This should be fixed like that's done for 8250
[patch drivers-serial-cleanup-locking-for-rt.patch]


> and this one
>
> [ š519.160755] ================> [ š519.160756] [ INFO: inconsistent lock state ]
> [ š519.160760] 3.10.24-rt22+ #25 Not tainted
> [ š519.160761] ---------------------------------
> [ š519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
> [ š519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [ š519.160778] š(&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
> simple_irq+0xc/0xe0
> [ š519.160779] {IN-HARDIRQ-W} state was registered at:
> [ š519.160785] šš[<00000000004ba7c0>] lock_acquire+0x60/0x100
> [ š519.160791] šš[<0000000000877930>] _raw_spin_lock+0x30/0x80
> [ š519.160795] šš[<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
> [ š519.160800] šš[<00000000004cf118>] generic_handle_irq+0x38/0x60
> [ š519.160804] šš[<000000000087bf44>] handler_irq+0xc4/0x100
> [ š519.160808] šš[<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
> [ š519.160812] šš[<000000000042ced4>] arch_cpu_idle+0x54/0xe0
> [ š519.160817] šš[<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
> [ š519.160822] šš[<0000000000870f18>] smp_callin+0x100/0x110
> [ š519.160825] šš[<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
> [ š519.160827] šš[< ššššššššš(null)>] šššššššššš(null)
> [ š519.160829] irq event stamp: 19
> [ š519.160832] hardirqs last šenabled at (19): [<0000000000877cc4>] _raw_spin_un
> lock_irq+0x24/0x60
> [ š519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
> ck_irq+0x10/0x80
> [ š519.160841] softirqs last šenabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [ š519.160843] softirqs last disabled at (0): [< ššššššššš(null)>] šššššššššš(nu
> ll)
> [ š519.160844]
> [ š519.160844] other info that might help us debug this:
> [ š519.160844] šPossible unsafe locking scenario:
> [ š519.160844]
> [ š519.160845] šššššššCPU0
> [ š519.160845] ššššššš----
> [ š519.160847] ššlock(&irq_desc_lock_class);
> [ š519.160848] šš<Interrupt>
> [ š519.160850] ššššlock(&irq_desc_lock_class);
> [ š519.160850]
> [ š519.160850] š*** DEADLOCK ***
> [ š519.160850]
> [ š519.160852] no locks held by irq/36-MSIQ/640.
> [ š519.160853]
> [ š519.160853] stack backtrace:
> [ š519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
> [ š519.160856] Call Trace:
> [ š519.160860] š[00000000004b50b4] print_usage_bug+0x234/0x2e0
> [ š519.160862] š[00000000004b5728] mark_lock+0x5c8/0x800
> [ š519.160864] š[00000000004ba238] __lock_acquire+0x7b8/0xce0
> [ š519.160866] š[00000000004ba7c0] lock_acquire+0x60/0x100
> [ š519.160868] š[0000000000877930] _raw_spin_lock+0x30/0x80
> [ š519.160870] š[00000000004d314c] handle_simple_irq+0xc/0xe0
> [ š519.160872] š[00000000004cf118] generic_handle_irq+0x38/0x60
> [ š519.160877] š[0000000000447870] sparc64_msiq_interrupt+0x50/0x120
> [ š519.160880] š[00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
> [ š519.160883] š[00000000004d019c] irq_thread+0xdc/0x140
> [ š519.160888] š[0000000000489560] kthread+0x80/0xa0
> [ š519.160893] š[0000000000406104] ret_from_syscall+0x1c/0x2c
> [ š519.160894] š[0000000000000000] šššššššššš(null)
> [ š519.160897] ------------[ cut here ]------------
>
> what do you think?
>
> - Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Fri, 28 Feb 2014 18:51:23 +0400

> This should be fixed like that's done for 8250
> [patch drivers-serial-cleanup-locking-for-rt.patch]

Like the patch below?

Is this transformation legitimate outside of the -rt tree?  I wish the commit
messages for the 8250 change was more verbose, explaining exactly why this
is needed.

And if it's needed, all the sparc serial/console drivers need this too.

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Fri, 28 Feb 2014 18:51:23 +0400

> This should be fixed like that's done for 8250
> [patch drivers-serial-cleanup-locking-for-rt.patch]

Like the patch below?

Is this transformation legitimate outside of the -rt tree?  I wish the commit
messages for the 8250 change was more verbose, explaining exactly why this
is needed.

And if it's needed, all the sparc serial/console drivers need this too.

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)

> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Here's an updated patch that takes care of all of the Sun serial
drivers, and is compile tested:

--------------------
sparc: serial: Clean up the locking for -rt

Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {
diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
index 380fb53..5faa8e9 100644
--- a/drivers/tty/serial/sunsab.c
+++ b/drivers/tty/serial/sunsab.c
@@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, n, sunsab_console_putchar);
 	sunsab_tec_wait(up);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int sunsab_console_setup(struct console *con, char *options)
diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
index db79b76..9a0f24f 100644
--- a/drivers/tty/serial/sunsu.c
+++ b/drivers/tty/serial/sunsu.c
@@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
 	unsigned int ier;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	/*
 	 *	First save the UER then disable the interrupts
@@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
 	serial_out(up, UART_IER, ier);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 /*
diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
index 45a8c6a..a2c40ed 100644
--- a/drivers/tty/serial/sunzilog.c
+++ b/drivers/tty/serial/sunzilog.c
@@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, count, sunzilog_putchar);
 	udelay(2);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int __init sunzilog_console_setup(struct console *con, char *options)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)

> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Here's an updated patch that takes care of all of the Sun serial
drivers, and is compile tested:

--------------------
sparc: serial: Clean up the locking for -rt

Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {
diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
index 380fb53..5faa8e9 100644
--- a/drivers/tty/serial/sunsab.c
+++ b/drivers/tty/serial/sunsab.c
@@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, n, sunsab_console_putchar);
 	sunsab_tec_wait(up);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int sunsab_console_setup(struct console *con, char *options)
diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
index db79b76..9a0f24f 100644
--- a/drivers/tty/serial/sunsu.c
+++ b/drivers/tty/serial/sunsu.c
@@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
 	unsigned int ier;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	/*
 	 *	First save the UER then disable the interrupts
@@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
 	serial_out(up, UART_IER, ier);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 /*
diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
index 45a8c6a..a2c40ed 100644
--- a/drivers/tty/serial/sunzilog.c
+++ b/drivers/tty/serial/sunzilog.c
@@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, count, sunzilog_putchar);
 	udelay(2);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int __init sunzilog_console_setup(struct console *con, char *options)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
> 
>> From: Kirill Tkhai <tkhai@yandex.ru>
>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>
>>> This should be fixed like that's done for 8250
>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>
>> Like the patch below?

tested. Sounds good to me.
Tested-by: Allen Pais <allen.pais@oracle.com>

> 
> Here's an updated patch that takes care of all of the Sun serial
> drivers, and is compile tested:
> 
> --------------------
> sparc: serial: Clean up the locking for -rt
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
> index 380fb53..5faa8e9 100644
> --- a/drivers/tty/serial/sunsab.c
> +++ b/drivers/tty/serial/sunsab.c
> @@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, n, sunsab_console_putchar);
>  	sunsab_tec_wait(up);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int sunsab_console_setup(struct console *con, char *options)
> diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
> index db79b76..9a0f24f 100644
> --- a/drivers/tty/serial/sunsu.c
> +++ b/drivers/tty/serial/sunsu.c
> @@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	unsigned int ier;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	/*
>  	 *	First save the UER then disable the interrupts
> @@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	serial_out(up, UART_IER, ier);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  /*
> diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
> index 45a8c6a..a2c40ed 100644
> --- a/drivers/tty/serial/sunzilog.c
> +++ b/drivers/tty/serial/sunzilog.c
> @@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, count, sunzilog_putchar);
>  	udelay(2);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int __init sunzilog_console_setup(struct console *con, char *options)
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
> 
>> From: Kirill Tkhai <tkhai@yandex.ru>
>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>
>>> This should be fixed like that's done for 8250
>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>
>> Like the patch below?

tested. Sounds good to me.
Tested-by: Allen Pais <allen.pais@oracle.com>

> 
> Here's an updated patch that takes care of all of the Sun serial
> drivers, and is compile tested:
> 
> --------------------
> sparc: serial: Clean up the locking for -rt
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
> index 380fb53..5faa8e9 100644
> --- a/drivers/tty/serial/sunsab.c
> +++ b/drivers/tty/serial/sunsab.c
> @@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, n, sunsab_console_putchar);
>  	sunsab_tec_wait(up);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int sunsab_console_setup(struct console *con, char *options)
> diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
> index db79b76..9a0f24f 100644
> --- a/drivers/tty/serial/sunsu.c
> +++ b/drivers/tty/serial/sunsu.c
> @@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	unsigned int ier;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	/*
>  	 *	First save the UER then disable the interrupts
> @@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	serial_out(up, UART_IER, ier);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  /*
> diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
> index 45a8c6a..a2c40ed 100644
> --- a/drivers/tty/serial/sunzilog.c
> +++ b/drivers/tty/serial/sunzilog.c
> @@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, count, sunzilog_putchar);
>  	udelay(2);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int __init sunzilog_console_setup(struct console *con, char *options)
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:00:20 +0530

> On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
>> 
>>> From: Kirill Tkhai <tkhai@yandex.ru>
>>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>>
>>>> This should be fixed like that's done for 8250
>>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>>
>>> Like the patch below?
> 
> tested. Sounds good to me.
> Tested-by: Allen Pais <allen.pais@oracle.com>

Great, I've queued this up into my tree.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:00:20 +0530

> On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
>> 
>>> From: Kirill Tkhai <tkhai@yandex.ru>
>>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>>
>>>> This should be fixed like that's done for 8250
>>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>>
>>> Like the patch below?
> 
> tested. Sounds good to me.
> Tested-by: Allen Pais <allen.pais@oracle.com>

Great, I've queued this up into my tree.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-06 16:36:50 [-0500]:

>Great, I've queued this up into my tree.

And I just took that in my -RT.

Sebastian

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-06 16:36:50 [-0500]:

>Great, I've queued this up into my tree.

And I just took that in my -RT.

Sebastian

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

Hi, David!

On 04.03.2014 23:10, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Exactly this.

> Is this transformation legitimate outside of the -rt tree?  I wish the commit
> messages for the 8250 change was more verbose, explaining exactly why this
> is needed.
> 
> And if it's needed, all the sparc serial/console drivers need this too.

port->lock is mutex, so we should not disable interrupts before we are
doing spin_lock(). In RT spin_trylock_irqsave() does not disable irqs.

The patch is made to be good for both RT and !RT cases, so everything
is ok!

Really I don't know, why the original patch is not in upstream
***shrugging***. It's from 2009, everything is possible.


> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

Hi, David!

On 04.03.2014 23:10, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Exactly this.

> Is this transformation legitimate outside of the -rt tree?  I wish the commit
> messages for the 8250 change was more verbose, explaining exactly why this
> is needed.
> 
> And if it's needed, all the sparc serial/console drivers need this too.

port->lock is mutex, so we should not disable interrupts before we are
doing spin_lock(). In RT spin_trylock_irqsave() does not disable irqs.

The patch is made to be good for both RT and !RT cases, so everything
is ok!

Really I don't know, why the original patch is not in upstream
***shrugging***. It's from 2009, everything is possible.


> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:10:03 [-0500]:

>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>
>Like the patch below?

The patch looks good.

>Is this transformation legitimate outside of the -rt tree?  I wish the commit
>messages for the 8250 change was more verbose, explaining exactly why this
>is needed.

Looking at it, it looks like the transformation is legitimate in
vanila, too.

In -RT the spin_lock_irqsave() does not spin but sleep if the lock is
taken. Before that, local_irq_save() is invoked which disables
interrupts even on -RT. That is why you might_sleep() triggers a warning
here.
In the ->sysrq and oops_in_progress case it is save to trylock the lock
i.e. this is what we do now anyway except for ->sysrq where we assume
that the lock is already taken.

The spin_lock_irqsave() grabs the lock and disables the interrupts on
vanila (the same behavior) and on -RT it won't disable interrupts.

>And if it's needed, all the sparc serial/console drivers need this too.

Let me forward the 8250 to GregKH and see how he reacts. Would be nice
if it is another patch that gets removed from that queue…

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:10:03 [-0500]:

>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>
>Like the patch below?

The patch looks good.

>Is this transformation legitimate outside of the -rt tree?  I wish the commit
>messages for the 8250 change was more verbose, explaining exactly why this
>is needed.

Looking at it, it looks like the transformation is legitimate in
vanila, too.

In -RT the spin_lock_irqsave() does not spin but sleep if the lock is
taken. Before that, local_irq_save() is invoked which disables
interrupts even on -RT. That is why you might_sleep() triggers a warning
here.
In the ->sysrq and oops_in_progress case it is save to trylock the lock
i.e. this is what we do now anyway except for ->sysrq where we assume
that the lock is already taken.

The spin_lock_irqsave() grabs the lock and disables the interrupts on
vanila (the same behavior) and on -RT it won't disable interrupts.

>And if it's needed, all the sparc serial/console drivers need this too.

Let me forward the 8250 to GregKH and see how he reacts. Would be nice
if it is another patch that gets removed from that queue…

Sebastian

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 19 Feb 2014 13:25:38 +0400

> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

My suspicion is that what happens when we get the data access error is
that we sample the tlb batch count as non-zero, preempt, then come
back from preemption seeing the tlb batch in a completely different state.

And that's what leads to the crash, in the one trace I saw the TSB address
passed to tsb_flush() (register %o0) was some garbage like 0x103.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 19 Feb 2014 13:25:38 +0400

> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

My suspicion is that what happens when we get the data access error is
that we sample the tlb batch count as non-zero, preempt, then come
back from preemption seeing the tlb batch in a completely different state.

And that's what leads to the crash, in the one trace I saw the TSB address
passed to tsb_flush() (register %o0) was some garbage like 0x103.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 05.03.2014 00:03, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 19 Feb 2014 13:25:38 +0400
> 
>> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
>> You may get good stack without sun4v_data_access_exception error, which was in the first or second
>> message.
> 
> My suspicion is that what happens when we get the data access error is
> that we sample the tlb batch count as non-zero, preempt, then come
> back from preemption seeing the tlb batch in a completely different state.
> 
> And that's what leads to the crash, in the one trace I saw the TSB address
> passed to tsb_flush() (register %o0) was some garbage like 0x103.
> 

I suggested to set tb_active to zero just for experiment. This way

diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index b12cb5e..e1d1fd6 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -54,7 +54,7 @@ void arch_enter_lazy_mmu_mode(void)
 {
 	struct tlb_batch *tb = &__get_cpu_var(tlb_batch);

-	tb->active = 1;
+	tb->active = 0;
 }

 void arch_leave_lazy_mmu_mode(void)

Last Allen's stack (from 26 feb. 11:52) still contains
flush_tlb_pending(). Strange, why this is so, maybe
bad initialized per-cpu tlb_batch, and something bad is with BSS...

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 05.03.2014 00:03, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 19 Feb 2014 13:25:38 +0400
> 
>> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
>> You may get good stack without sun4v_data_access_exception error, which was in the first or second
>> message.
> 
> My suspicion is that what happens when we get the data access error is
> that we sample the tlb batch count as non-zero, preempt, then come
> back from preemption seeing the tlb batch in a completely different state.
> 
> And that's what leads to the crash, in the one trace I saw the TSB address
> passed to tsb_flush() (register %o0) was some garbage like 0x103.
> 

I suggested to set tb_active to zero just for experiment. This way

diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index b12cb5e..e1d1fd6 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -54,7 +54,7 @@ void arch_enter_lazy_mmu_mode(void)
 {
 	struct tlb_batch *tb = &__get_cpu_var(tlb_batch);

-	tb->active = 1;
+	tb->active = 0;
 }

 void arch_leave_lazy_mmu_mode(void)

Last Allen's stack (from 26 feb. 11:52) still contains
flush_tlb_pending(). Strange, why this is so, maybe
bad initialized per-cpu tlb_batch, and something bad is with BSS...

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 19 Feb 2014 09:23:46 +0530

>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)

If using a local lock fixes your problem, it just means we're being
preempted during flush_tsb_user() which is absolutely not allowed.

The batch is a per-cpu object, and must be forcibly flushed if we
preempt.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 19 Feb 2014 09:23:46 +0530

>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)

If using a local lock fixes your problem, it just means we're being
preempted during flush_tsb_user() which is absolutely not allowed.

The batch is a per-cpu object, and must be forcibly flushed if we
preempt.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
> From: Allen Pais <allen.pais@oracle.com>
> Date: Wed, 19 Feb 2014 09:23:46 +0530
> 
>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>> with hackbench and dd, the system did not stall :)
> 
> If using a local lock fixes your problem, it just means we're being
> preempted during flush_tsb_user() which is absolutely not allowed.

David, Its my bad, I realized my mistake of using local locks.
Am still looking into the tsb flush code. It's left me sleepless. :)
 

> The batch is a per-cpu object, and must be forcibly flushed if we
> preempt.
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
> From: Allen Pais <allen.pais@oracle.com>
> Date: Wed, 19 Feb 2014 09:23:46 +0530
> 
>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>> with hackbench and dd, the system did not stall :)
> 
> If using a local lock fixes your problem, it just means we're being
> preempted during flush_tsb_user() which is absolutely not allowed.

David, Its my bad, I realized my mistake of using local locks.
Am still looking into the tsb flush code. It's left me sleepless. :)
 

> The batch is a per-cpu object, and must be forcibly flushed if we
> preempt.
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:04:56 +0530

> On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
>> From: Allen Pais <allen.pais@oracle.com>
>> Date: Wed, 19 Feb 2014 09:23:46 +0530
>> 
>>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>>> with hackbench and dd, the system did not stall :)
>> 
>> If using a local lock fixes your problem, it just means we're being
>> preempted during flush_tsb_user() which is absolutely not allowed.
> 
> David, Its my bad, I realized my mistake of using local locks.
> Am still looking into the tsb flush code. It's left me sleepless. :)

So let's do a simple test, in flush_tlb_user() assert to make sure we
cannot be preempted by the -rt kernel.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:04:56 +0530

> On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
>> From: Allen Pais <allen.pais@oracle.com>
>> Date: Wed, 19 Feb 2014 09:23:46 +0530
>> 
>>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>>> with hackbench and dd, the system did not stall :)
>> 
>> If using a local lock fixes your problem, it just means we're being
>> preempted during flush_tsb_user() which is absolutely not allowed.
> 
> David, Its my bad, I realized my mistake of using local locks.
> Am still looking into the tsb flush code. It's left me sleepless. :)

So let's do a simple test, in flush_tlb_user() assert to make sure we
cannot be preempted by the -rt kernel.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 15:43:06 +0400

> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The goal of the lazy tlb stuff is to only have one (potential)
synchronization point with other cpus.

Then we sweep away the TSB entries, after which any TLB miss must
enter the full fault path and synchronize with the current thread
doing the flush.

Then we kill the TLB entries, both local and remote.

The tsb_grow() code path is quite the animal, as any of you who have
read the comment above it's implementation can attest :-) It took 6
months to get that code right in a non-RT context back when it was
originally written.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 15:43:06 +0400

> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The goal of the lazy tlb stuff is to only have one (potential)
synchronization point with other cpus.

Then we sweep away the TSB entries, after which any TLB miss must
enter the full fault path and synchronize with the current thread
doing the flush.

Then we kill the TLB entries, both local and remote.

The tsb_grow() code path is quite the animal, as any of you who have
read the comment above it's implementation can attest :-) It took 6
months to get that code right in a non-RT context back when it was
originally written.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 12:33:58 +0400

> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>   with the following trace:
>>>>
>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.

Sorry for coming into this discussion so late.

Indeed, the pending flushes are per-cpu and we must flush them out in the
event of a preemption.

PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
fact that's where I copied the logic from.  Does PowerPC not work with
-rt? :-)
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 12:33:58 +0400

> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>   with the following trace:
>>>>
>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.

Sorry for coming into this discussion so late.

Indeed, the pending flushes are per-cpu and we must flush them out in the
event of a preemption.

PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
fact that's where I copied the logic from.  Does PowerPC not work with
-rt? :-)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 04.03.2014 23:55, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 12 Feb 2014 12:33:58 +0400
> 
>> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
>>
>>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>>   with the following trace:
>>>>>
>>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>   [ 1487.027885] Call Trace:
>>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>>  Now, consistently I've been getting sun4v_data_access_exception.
>>>  Here's the trace:
>>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>>
>> I've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>> you collect flush requests before you really flush all of them.
>>
>> In RT you collect them too, but you are able to be preempted in any moment.
>> So, you may switch to other process with unflushed tlb, which is very bad.
>>
>> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>> We will look if this robust fix helps.
> 
> Sorry for coming into this discussion so late.
> 
> Indeed, the pending flushes are per-cpu and we must flush them out in the
> event of a preemption.
> 
> PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
> fact that's where I copied the logic from.  Does PowerPC not work with
> -rt? :-)
> 

It does not work, but we will :)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 04.03.2014 23:55, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 12 Feb 2014 12:33:58 +0400
> 
>> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
>>
>>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>>   with the following trace:
>>>>>
>>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>   [ 1487.027885] Call Trace:
>>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>>  Now, consistently I've been getting sun4v_data_access_exception.
>>>  Here's the trace:
>>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>>
>> I've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>> you collect flush requests before you really flush all of them.
>>
>> In RT you collect them too, but you are able to be preempted in any moment.
>> So, you may switch to other process with unflushed tlb, which is very bad.
>>
>> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>> We will look if this robust fix helps.
> 
> Sorry for coming into this discussion so late.
> 
> Indeed, the pending flushes are per-cpu and we must flush them out in the
> event of a preemption.
> 
> PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
> fact that's where I copied the logic from.  Does PowerPC not work with
> -rt? :-)
> 

It does not work, but we will :)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:55:23 [-0500]:

>PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
>fact that's where I copied the logic from.  Does PowerPC not work with
>-rt? :-)

It is only PPC64 as far as I can see and I have mostly 32bit book-e
around. That preempt-lazy stuff also touches PPC64 bits so I assume
PPC64 works.

Sebastian

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:55:23 [-0500]:

>PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
>fact that's where I copied the logic from.  Does PowerPC not work with
>-rt? :-)

It is only PPC64 as far as I can see and I have mostly 32bit book-e
around. That preempt-lazy stuff also touches PPC64 bits so I assume
PPC64 works.

Sebastian

[PATCH 4/4] sparc64: convert ctx_alloc_lock raw_spinlock_t

[Allen Pais]

This patch fixes the kernel crash faced while
trying to attempt linux-stable-rt v3.10.22-rt19
on sparc64.

[ 2317.606015]  [00000000008072f4] rt_spin_lock_slowlock+0x94/0x300
[ 2317.606020]  [0000000000451d74] get_new_mmu_context+0x14/0x160
[ 2317.606026]  [0000000000806394] switch_to_pc+0xd4/0x2a0
[ 2317.606029]  [00000000008067dc] schedule+0x1c/0xc0
[ 2317.606031]  [0000000000807364] rt_spin_lock_slowlock+0x104/0x300
[ 2317.606033]  [0000000000450284] destroy_context+0x84/0x120
[ 2317.606036]  [000000000045c788] __mmdrop+0x28/0xe0
[ 2317.606045]  [00000000004bf290] rcu_process_callbacks+0x450/0x760
[ 2317.606049]  [0000000000466d48] do_current_softirqs+0x208/0x3c0
[ 2317.606051]  [0000000000466f14] run_ksoftirqd+0x14/0x40
[ 2317.606057]  [000000000048c64c] smpboot_thread_fn+0x18c/0x2e0
[ 2317.606061]  [0000000000483b5c] kthread+0x7c/0xa0
[ 2317.606069]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
[ 2317.606070]  [0000000000000000]           (null)

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/include/asm/mmu_context_64.h |    2 +-
 arch/sparc/mm/init_64.c                 |   10 +++++-----
 arch/sparc/mm/tsb.c                     |    4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
index 3a85624..44e393b 100644
--- a/arch/sparc/include/asm/mmu_context_64.h
+++ b/arch/sparc/include/asm/mmu_context_64.h
@@ -13,7 +13,7 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
 {
 }
 
-extern spinlock_t ctx_alloc_lock;
+extern raw_spinlock_t ctx_alloc_lock;
 extern unsigned long tlb_context_cache;
 extern unsigned long mmu_context_bmap[];
 
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index bd5253d..ac5ae7a 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -661,7 +661,7 @@ void __flush_dcache_range(unsigned long start, unsigned long end)
 EXPORT_SYMBOL(__flush_dcache_range);
 
 /* get_new_mmu_context() uses "cache + 1".  */
-DEFINE_SPINLOCK(ctx_alloc_lock);
+DEFINE_RAW_SPINLOCK(ctx_alloc_lock);
 unsigned long tlb_context_cache = CTX_FIRST_VERSION - 1;
 #define MAX_CTX_NR	(1UL << CTX_NR_BITS)
 #define CTX_BMAP_SLOTS	BITS_TO_LONGS(MAX_CTX_NR)
@@ -683,7 +683,7 @@ void get_new_mmu_context(struct mm_struct *mm)
 	unsigned long orig_pgsz_bits;
 	int new_version;
 
-	spin_lock(&ctx_alloc_lock);
+	raw_spin_lock(&ctx_alloc_lock);
 	orig_pgsz_bits = (mm->context.sparc64_ctx_val & CTX_PGSZ_MASK);
 	ctx = (tlb_context_cache + 1) & CTX_NR_MASK;
 	new_ctx = find_next_zero_bit(mmu_context_bmap, 1 << CTX_NR_BITS, ctx);
@@ -719,7 +719,7 @@ void get_new_mmu_context(struct mm_struct *mm)
 out:
 	tlb_context_cache = new_ctx;
 	mm->context.sparc64_ctx_val = new_ctx | orig_pgsz_bits;
-	spin_unlock(&ctx_alloc_lock);
+	raw_spin_unlock(&ctx_alloc_lock);
 
 	if (unlikely(new_version))
 		smp_new_mmu_context_version();
@@ -2739,7 +2739,7 @@ void hugetlb_setup(struct pt_regs *regs)
 	if (tlb_type == cheetah_plus) {
 		unsigned long ctx;
 
-		spin_lock(&ctx_alloc_lock);
+		raw_spin_lock(&ctx_alloc_lock);
 		ctx = mm->context.sparc64_ctx_val;
 		ctx &= ~CTX_PGSZ_MASK;
 		ctx |= CTX_PGSZ_BASE << CTX_PGSZ0_SHIFT;
@@ -2760,7 +2760,7 @@ void hugetlb_setup(struct pt_regs *regs)
 			mm->context.sparc64_ctx_val = ctx;
 			on_each_cpu(context_reload, mm, 0);
 		}
-		spin_unlock(&ctx_alloc_lock);
+		raw_spin_unlock(&ctx_alloc_lock);
 	}
 }
 #endif
diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index d84d4ea..9eb10b4 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -523,12 +523,12 @@ void destroy_context(struct mm_struct *mm)
 		free_hot_cold_page(page, 0);
 	}
 
-	spin_lock_irqsave(&ctx_alloc_lock, flags);
+	raw_spin_lock_irqsave(&ctx_alloc_lock, flags);
 
 	if (CTX_VALID(mm->context)) {
 		unsigned long nr = CTX_NRBITS(mm->context);
 		mmu_context_bmap[nr>>6] &= ~(1UL << (nr & 63));
 	}
 
-	spin_unlock_irqrestore(&ctx_alloc_lock, flags);
+	raw_spin_unlock_irqrestore(&ctx_alloc_lock, flags);
 }
-- 
1.7.10.4

[PATCH 4/4] sparc64: convert ctx_alloc_lock raw_spinlock_t

[Allen Pais]

This patch fixes the kernel crash faced while
trying to attempt linux-stable-rt v3.10.22-rt19
on sparc64.

[ 2317.606015]  [00000000008072f4] rt_spin_lock_slowlock+0x94/0x300
[ 2317.606020]  [0000000000451d74] get_new_mmu_context+0x14/0x160
[ 2317.606026]  [0000000000806394] switch_to_pc+0xd4/0x2a0
[ 2317.606029]  [00000000008067dc] schedule+0x1c/0xc0
[ 2317.606031]  [0000000000807364] rt_spin_lock_slowlock+0x104/0x300
[ 2317.606033]  [0000000000450284] destroy_context+0x84/0x120
[ 2317.606036]  [000000000045c788] __mmdrop+0x28/0xe0
[ 2317.606045]  [00000000004bf290] rcu_process_callbacks+0x450/0x760
[ 2317.606049]  [0000000000466d48] do_current_softirqs+0x208/0x3c0
[ 2317.606051]  [0000000000466f14] run_ksoftirqd+0x14/0x40
[ 2317.606057]  [000000000048c64c] smpboot_thread_fn+0x18c/0x2e0
[ 2317.606061]  [0000000000483b5c] kthread+0x7c/0xa0
[ 2317.606069]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
[ 2317.606070]  [0000000000000000]           (null)

Acked-by: David S. Miller <davem@davemloft.net>
Signed-off-by: Allen Pais <allen.pais@oracle.com>
---
 arch/sparc/include/asm/mmu_context_64.h |    2 +-
 arch/sparc/mm/init_64.c                 |   10 +++++-----
 arch/sparc/mm/tsb.c                     |    4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h
index 3a85624..44e393b 100644
--- a/arch/sparc/include/asm/mmu_context_64.h
+++ b/arch/sparc/include/asm/mmu_context_64.h
@@ -13,7 +13,7 @@ static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
 {
 }
 
-extern spinlock_t ctx_alloc_lock;
+extern raw_spinlock_t ctx_alloc_lock;
 extern unsigned long tlb_context_cache;
 extern unsigned long mmu_context_bmap[];
 
diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c
index bd5253d..ac5ae7a 100644
--- a/arch/sparc/mm/init_64.c
+++ b/arch/sparc/mm/init_64.c
@@ -661,7 +661,7 @@ void __flush_dcache_range(unsigned long start, unsigned long end)
 EXPORT_SYMBOL(__flush_dcache_range);
 
 /* get_new_mmu_context() uses "cache + 1".  */
-DEFINE_SPINLOCK(ctx_alloc_lock);
+DEFINE_RAW_SPINLOCK(ctx_alloc_lock);
 unsigned long tlb_context_cache = CTX_FIRST_VERSION - 1;
 #define MAX_CTX_NR	(1UL << CTX_NR_BITS)
 #define CTX_BMAP_SLOTS	BITS_TO_LONGS(MAX_CTX_NR)
@@ -683,7 +683,7 @@ void get_new_mmu_context(struct mm_struct *mm)
 	unsigned long orig_pgsz_bits;
 	int new_version;
 
-	spin_lock(&ctx_alloc_lock);
+	raw_spin_lock(&ctx_alloc_lock);
 	orig_pgsz_bits = (mm->context.sparc64_ctx_val & CTX_PGSZ_MASK);
 	ctx = (tlb_context_cache + 1) & CTX_NR_MASK;
 	new_ctx = find_next_zero_bit(mmu_context_bmap, 1 << CTX_NR_BITS, ctx);
@@ -719,7 +719,7 @@ void get_new_mmu_context(struct mm_struct *mm)
 out:
 	tlb_context_cache = new_ctx;
 	mm->context.sparc64_ctx_val = new_ctx | orig_pgsz_bits;
-	spin_unlock(&ctx_alloc_lock);
+	raw_spin_unlock(&ctx_alloc_lock);
 
 	if (unlikely(new_version))
 		smp_new_mmu_context_version();
@@ -2739,7 +2739,7 @@ void hugetlb_setup(struct pt_regs *regs)
 	if (tlb_type = cheetah_plus) {
 		unsigned long ctx;
 
-		spin_lock(&ctx_alloc_lock);
+		raw_spin_lock(&ctx_alloc_lock);
 		ctx = mm->context.sparc64_ctx_val;
 		ctx &= ~CTX_PGSZ_MASK;
 		ctx |= CTX_PGSZ_BASE << CTX_PGSZ0_SHIFT;
@@ -2760,7 +2760,7 @@ void hugetlb_setup(struct pt_regs *regs)
 			mm->context.sparc64_ctx_val = ctx;
 			on_each_cpu(context_reload, mm, 0);
 		}
-		spin_unlock(&ctx_alloc_lock);
+		raw_spin_unlock(&ctx_alloc_lock);
 	}
 }
 #endif
diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index d84d4ea..9eb10b4 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -523,12 +523,12 @@ void destroy_context(struct mm_struct *mm)
 		free_hot_cold_page(page, 0);
 	}
 
-	spin_lock_irqsave(&ctx_alloc_lock, flags);
+	raw_spin_lock_irqsave(&ctx_alloc_lock, flags);
 
 	if (CTX_VALID(mm->context)) {
 		unsigned long nr = CTX_NRBITS(mm->context);
 		mmu_context_bmap[nr>>6] &= ~(1UL << (nr & 63));
 	}
 
-	spin_unlock_irqrestore(&ctx_alloc_lock, flags);
+	raw_spin_unlock_irqrestore(&ctx_alloc_lock, flags);
 }
-- 
1.7.10.4

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

> Have tested it on UltraSparc T4 (Niagara4). 
> 
> Allen Pais (4):
>   sparc64: use generic rwsem spinlocks rt
>   sparc64: allow forced irq threading
>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>   sparc64: convert ctx_alloc_lock raw_spinlock_t
> 
>  arch/sparc/Kconfig                      |    7 +++----
>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>  arch/sparc/kernel/smp_64.c              |    4 ++--
>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>  6 files changed, 28 insertions(+), 29 deletions(-)
> 

Sebastian, Any idea on the status of these patches?

Thanks,
Allen

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Sebastian Andrzej Siewior]

On 02/05/2014 04:31 AM, Allen Pais wrote:
> 
>> Have tested it on UltraSparc T4 (Niagara4). 
>>
>> Allen Pais (4):
>>   sparc64: use generic rwsem spinlocks rt
>>   sparc64: allow forced irq threading
>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>
>>  arch/sparc/Kconfig                      |    7 +++----
>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>
> 
> Sebastian, Any idea on the status of these patches?

I took that all in one I think so they are part of -RT. These individual
should go upstream via DaveM / the sparc tree.

> 
> Thanks,
> Allen

Sebastian

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

>>>
>>> Allen Pais (4):
>>>   sparc64: use generic rwsem spinlocks rt
>>>   sparc64: allow forced irq threading
>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>
>>>  arch/sparc/Kconfig                      |    7 +++----
>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>
>>
>> Sebastian, Any idea on the status of these patches?
> 
> I took that all in one I think so they are part of -RT. These individual
> should go upstream via DaveM / the sparc tree.
> 

 I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
list and check.

Thanks,
Allen

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Sebastian Andrzej Siewior]

On 02/05/2014 11:38 AM, Allen Pais wrote:
>>>>
>>>> Allen Pais (4):
>>>>   sparc64: use generic rwsem spinlocks rt
>>>>   sparc64: allow forced irq threading
>>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>>
>>>>  arch/sparc/Kconfig                      |    7 +++----
>>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>>
>>>
>>> Sebastian, Any idea on the status of these patches?
>>
>> I took that all in one I think so they are part of -RT. These individual
>> should go upstream via DaveM / the sparc tree.
>>
> 
>  I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
> list and check.

The rwsem thingy is the only -RT specific and it depends on -RT only
Kconfig symbols. The spin_lock_t to raw_spinlock_t conversations can
also go upstream.

> Thanks,
> Allen
> 
Sebastian

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

On Wednesday 05 February 2014 04:13 PM, Sebastian Andrzej Siewior wrote:
> On 02/05/2014 11:38 AM, Allen Pais wrote:
>>>>>
>>>>> Allen Pais (4):
>>>>>   sparc64: use generic rwsem spinlocks rt
>>>>>   sparc64: allow forced irq threading
>>>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>>>
>>>>>  arch/sparc/Kconfig                      |    7 +++----
>>>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>>>
>>>>
>>>> Sebastian, Any idea on the status of these patches?
>>>
>>> I took that all in one I think so they are part of -RT. These individual
>>> should go upstream via DaveM / the sparc tree.
>>>
>>
>>  I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
>> list and check.
> 
> The rwsem thingy is the only -RT specific and it depends on -RT only
> Kconfig symbols. The spin_lock_t to raw_spinlock_t conversations can
> also go upstream.

They've all been sent upstream. Will check with DaveM on the status.

Thanks again.

Allen