[ANNOUNCE] 3.14-rt1

[ANNOUNCE] 3.14-rt1

[Sebastian Andrzej Siewior]

Dear RT folks!

I'm pleased to announce the v3.14-rt1 patch set.

Changes since v3.12.15-rt25
- I dropped the sparc64 patches I had in the queue. They did not apply
  cleanly, the code in v3.14 changed in the MMU area. Here is where I
  remembered that it was not working perfectly either.

- Scott Wood pointed out that I forgot a return value in the latest
  gianfar fixup for RT. Return value added, thanks Scott.

This -RT series didn't crashed within ~4h testing on my ARM and
x86-32.
x86-64 crashed after I started hackbench. I figured out that the crash
does not happen with lazy-preempt disabled. Therefore the last but one
patch in the queue disables lazy preempt on x86-64. With this change the
test box survived ~2h without a crash. I look at this later but it looks
good now.

I decided to release it now because otherwise it would be lying around
probably the next two weeks or so. So here it is, enjoy.

Known issues:

      - bcache is disabled.

The RT patch against 3.14 can be found here:

   https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patch-3.14.0-rt1.patch.xz

The split quilt queue is available at:

   https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patches-3.14.0-rt1.tar.xz

Sebastian

Re: [ANNOUNCE] 3.14-rt1

[Pavel Vasilyev]

11.04.2014 22:57, Sebastian Andrzej Siewior пишет:
> Dear RT folks!
>
> I'm pleased to announce the v3.14-rt1 patch set.

Hooooooooooooooooray!



-- 

                                                          Pavel.

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

Hi Sebastian,

On Fri, 2014-04-11 at 20:57 +0200, Sebastian Andrzej Siewior wrote: 
> Dear RT folks!
> 
> I'm pleased to announce the v3.14-rt1 patch set.

This hunk in hotplug-light-get-online-cpus.patch looks like a bug.

@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
                /* CPU didn't die: tell everyone.  Can't complain. */
                smpboot_unpark_threads(cpu);
                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
-               goto out_release;
+               goto out_cancel;
        }
        BUG_ON(cpu_online(cpu));


> x86-64 crashed after I started hackbench. I figured out that the crash
> does not happen with lazy-preempt disabled. Therefore the last but one
> patch in the queue disables lazy preempt on x86-64. With this change the
> test box survived ~2h without a crash. I look at this later but it looks
> good now.

Ah, I had trouble there a while back too.  I'll try to scrape up cycles
for a round 2, see who begs for mercy this time, it or me again.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Sat, 2014-04-19 at 16:46 +0200, Mike Galbraith wrote: 
> Hi Sebastian,
> 
> On Fri, 2014-04-11 at 20:57 +0200, Sebastian Andrzej Siewior wrote: 
> > Dear RT folks!
> > 
> > I'm pleased to announce the v3.14-rt1 patch set.
> 
> This hunk in hotplug-light-get-online-cpus.patch looks like a bug.
> 
> @@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
>                 /* CPU didn't die: tell everyone.  Can't complain. */
>                 smpboot_unpark_threads(cpu);
>                 cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
> -               goto out_release;
> +               goto out_cancel;
>         }
>         BUG_ON(cpu_online(cpu));
> 

Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
should be while (atomic_read(&done.nr_todo)) 

@@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
        ret = multi_cpu_stop(&msdata);

        /* Busy wait for completion. */
-       while (!completion_done(&done.completion))
+       while (!atomic_read(&done.nr_todo))
                cpu_relax();

        mutex_unlock(&stop_cpus_mutex);
                                        

Re: [ANNOUNCE] 3.14-rt1

[Sebastian Andrzej Siewior]

* Mike Galbraith | 2014-04-21 05:31:18 [+0200]:

>Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
>should be while (atomic_read(&done.nr_todo)) 

Thanks, fixed up.

Sebastian

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

[-- Attachment #1: Type: text/plain, Size: 9319 bytes --]

On Sat, 2014-04-19 at 16:46 +0200, Mike Galbraith wrote: 
> Hi Sebastian,
> 
> On Fri, 2014-04-11 at 20:57 +0200, Sebastian Andrzej Siewior wrote: 
> > Dear RT folks!
> > 
> > I'm pleased to announce the v3.14-rt1 patch set.
> 
> This hunk in hotplug-light-get-online-cpus.patch looks like a bug.
> 
> @@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
>                 /* CPU didn't die: tell everyone.  Can't complain. */
>                 smpboot_unpark_threads(cpu);
>                 cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
> -               goto out_release;
> +               goto out_cancel;
>         }
>         BUG_ON(cpu_online(cpu));

...

BTW, the reason I was eyeballing this stuff is because I was highly
interested in what you were going to do here...

# XXX stomp-machine-deal-clever-with-stopper-lock.patch

...with that bloody lglock.  What I did is attached for your amusement.
(warning: viewing may induce "Medussa" syndrome:)

Hotplug can still deadlock in rt trees too, and will if you beat it
hard.  The splat below is virgin 3.12-rt (where wonderful lock doesn't
yet exist) while running Stevens stress-cpu-hotplug.sh, which is still
plenty deadly when liberally applied.

[  161.951908] CPU0 attaching NULL sched-domain.
[  161.970417] CPU2 attaching NULL sched-domain.
[  161.976594] CPU3 attaching NULL sched-domain.
[  161.981044] CPU0 attaching sched-domain:
[  161.985010]  domain 0: span 0,3 level CPU
[  161.990627]   groups: 0 (cpu_power = 997) 3 (cpu_power = 1021)
[  162.000609] CPU3 attaching sched-domain:
[  162.007723]  domain 0: span 0,3 level CPU
[  162.012756]   groups: 3 (cpu_power = 1021) 0 (cpu_power = 997)
[  162.025533] smpboot: CPU 2 is now offline
[  162.036113] 
[  162.036114] ======================================================
[  162.036115] [ INFO: possible circular locking dependency detected ]
[  162.036116] 3.12.17-rt25 #14 Not tainted
[  162.036117] -------------------------------------------------------
[  162.036118] boot.kdump/6853 is trying to acquire lock:
[  162.036126]  (&hp->lock){+.+...}, at: [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  162.036126] 
[  162.036126] but task is already holding lock:
[  162.036131]  (&mm->mmap_sem){+++++.}, at: [<ffffffff8156285c>] __do_page_fault+0x14c/0x5d0
[  162.036132] 
[  162.036132] which lock already depends on the new lock.
[  162.036132] 
[  162.036133] 
[  162.036133] the existing dependency chain (in reverse order) is:
[  162.036135] 
[  162.036135] -> #2 (&mm->mmap_sem){+++++.}:
[  162.036138]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  162.036140]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  162.036142]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  162.036143]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  162.036146]        [<ffffffff8112df03>] might_fault+0x83/0xb0
[  162.036149]        [<ffffffff81341851>] sel_loadlut+0x11/0x70
[  162.036152]        [<ffffffff8134aa1d>] tioclinux+0x23d/0x2c0
[  162.036153]        [<ffffffff8133f88c>] vt_ioctl+0x86c/0x11f0
[  162.036155]        [<ffffffff81333cf8>] tty_ioctl+0x2a8/0x940
[  162.036158]        [<ffffffff8116c161>] do_vfs_ioctl+0x81/0x340
[  162.036159]        [<ffffffff8116c46b>] SyS_ioctl+0x4b/0x90
[  162.036162]        [<ffffffff81566c22>] system_call_fastpath+0x16/0x1b
[  162.036164] 
[  162.036164] -> #1 (console_lock){+.+.+.}:
[  162.036165]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  162.036167]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  162.036169]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  162.036171]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  162.036173]        [<ffffffff810957bf>] console_lock+0x6f/0x80
[  162.036174]        [<ffffffff8109673d>] console_cpu_notify+0x1d/0x30
[  162.036176]        [<ffffffff81562d3d>] notifier_call_chain+0x4d/0x70
[  162.036179]        [<ffffffff81070b49>] __raw_notifier_call_chain+0x9/0x10
[  162.036181]        [<ffffffff8104443b>] __cpu_notify+0x1b/0x30
[  162.036182]        [<ffffffff81044650>] cpu_notify_nofail+0x10/0x20
[  162.036185]        [<ffffffff815480fd>] _cpu_down+0x20d/0x440
[  162.036186]        [<ffffffff81548360>] cpu_down+0x30/0x50
[  162.036188]        [<ffffffff8137118c>] cpu_subsys_offline+0x1c/0x30
[  162.036191]        [<ffffffff8136c285>] device_offline+0x95/0xc0
[  162.036192]        [<ffffffff8136c390>] online_store+0x40/0x80
[  162.036194]        [<ffffffff813697c3>] dev_attr_store+0x13/0x30
[  162.036197]        [<ffffffff811c8820>] sysfs_write_file+0xf0/0x170
[  162.036200]        [<ffffffff8115a068>] vfs_write+0xc8/0x1d0
[  162.036202]        [<ffffffff8115a500>] SyS_write+0x50/0xa0
[  162.036203]        [<ffffffff81566c22>] system_call_fastpath+0x16/0x1b
[  162.036205] 
[  162.036205] -> #0 (&hp->lock){+.+...}:
[  162.036207]        [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  162.036209]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  162.036210]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  162.036212]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  162.036214]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  162.036216]        [<ffffffff8155e645>] rt_spin_lock+0x55/0x70
[  162.036218]        [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  162.036220]        [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  162.036222]        [<ffffffff8112fd38>] handle_pte_fault+0xf8/0x1c0
[  162.036223]        [<ffffffff81131646>] __handle_mm_fault+0x106/0x1b0
[  162.036225]        [<ffffffff81131712>] handle_mm_fault+0x22/0x30
[  162.036227]        [<ffffffff815628c1>] __do_page_fault+0x1b1/0x5d0
[  162.036229]        [<ffffffff81562ce9>] do_page_fault+0x9/0x10
[  162.036230]        [<ffffffff8155f9d2>] page_fault+0x22/0x30
[  162.036232]        [<ffffffff81566b0f>] ret_from_fork+0xf/0xb0
[  162.036233] 
[  162.036233] other info that might help us debug this:
[  162.036233] 
[  162.036235] Chain exists of:
[  162.036235]   &hp->lock --> console_lock --> &mm->mmap_sem
[  162.036235] 
[  162.036236]  Possible unsafe locking scenario:
[  162.036236] 
[  162.036236]        CPU0                    CPU1
[  162.036237]        ----                    ----
[  162.036238]   lock(&mm->mmap_sem);
[  162.036239]                                lock(console_lock);
[  162.036241]                                lock(&mm->mmap_sem);
[  162.036242]   lock(&hp->lock);
[  162.036242] 
[  162.036242]  *** DEADLOCK ***
[  162.036242] 
[  162.036243] 1 lock held by boot.kdump/6853:
[  162.036247]  #0:  (&mm->mmap_sem){+++++.}, at: [<ffffffff8156285c>] __do_page_fault+0x14c/0x5d0
[  162.036247] 
[  162.036247] stack backtrace:
[  162.036250] CPU: 0 PID: 6853 Comm: boot.kdump Not tainted 3.12.17-rt25 #14
[  162.036251] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[  162.036253]  ffff8801fd0e6d58 ffff8800bbe85918 ffffffff8155532c 0000000000000000
[  162.036255]  0000000000000000 ffff8800bbe85968 ffffffff8154d07f ffff8800bbe85958
[  162.036257]  ffffffff82350640 ffff8801fd0e6d58 ffff8801fd0e6d20 ffff8801fd0e6d58
[  162.036258] Call Trace:
[  162.036261]  [<ffffffff8155532c>] dump_stack+0x4f/0x91
[  162.036263]  [<ffffffff8154d07f>] print_circular_bug+0xd3/0xe4
[  162.036265]  [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  162.036268]  [<ffffffff8107e1f5>] ? sched_clock_local+0x25/0x90
[  162.036270]  [<ffffffff8107e388>] ? sched_clock_cpu+0xa8/0x120
[  162.036272]  [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  162.036273]  [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  162.036275]  [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  162.036277]  [<ffffffff8155d9b1>] ? rt_spin_lock_slowlock+0x231/0x280
[  162.036279]  [<ffffffff8155d8b1>] ? rt_spin_lock_slowlock+0x131/0x280
[  162.036281]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  162.036282]  [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  162.036284]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  162.036286]  [<ffffffff8155e645>] rt_spin_lock+0x55/0x70
[  162.036288]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  162.036289]  [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  162.036291]  [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  162.036293]  [<ffffffff8112fd38>] handle_pte_fault+0xf8/0x1c0
[  162.036295]  [<ffffffff8156285c>] ? __do_page_fault+0x14c/0x5d0
[  162.036296]  [<ffffffff81131646>] __handle_mm_fault+0x106/0x1b0
[  162.036298]  [<ffffffff81131712>] handle_mm_fault+0x22/0x30
[  162.036300]  [<ffffffff815628c1>] __do_page_fault+0x1b1/0x5d0
[  162.036302]  [<ffffffff8107e1f5>] ? sched_clock_local+0x25/0x90
[  162.036304]  [<ffffffff810790d1>] ? get_parent_ip+0x11/0x50
[  162.036306]  [<ffffffff81562f7d>] ? add_preempt_count.part.93+0x5d/0xb0
[  162.036307]  [<ffffffff810aa2c2>] ? get_lock_stats+0x22/0x70
[  162.036309]  [<ffffffff810aa7ce>] ? put_lock_stats.isra.26+0xe/0x40
[  162.036311]  [<ffffffff812841ed>] ? trace_hardirqs_off_thunk+0x3a/0x3c
[  162.036313]  [<ffffffff81562ce9>] do_page_fault+0x9/0x10
[  162.036315]  [<ffffffff8155f9d2>] page_fault+0x22/0x30
[  162.036317]  [<ffffffff81284130>] ? __put_user_4+0x20/0x30
[  162.036319]  [<ffffffff81078c47>] ? schedule_tail+0x67/0xb0
[  162.036321]  [<ffffffff81566b0f>] ret_from_fork+0xf/0xb0


[-- Attachment #2: stomp-machine-create-lg_global_trylock_relax.patch --]
[-- Type: text/x-patch, Size: 2102 bytes --]

---
 include/linux/lglock.h      |    6 ++++++
 include/linux/spinlock_rt.h |    1 +
 kernel/locking/lglock.c     |   25 +++++++++++++++++++++++++
 kernel/locking/rtmutex.c    |    5 +++++
 4 files changed, 37 insertions(+)

--- a/include/linux/lglock.h
+++ b/include/linux/lglock.h
@@ -74,4 +74,10 @@ void lg_local_unlock_cpu(struct lglock *
 void lg_global_lock(struct lglock *lg);
 void lg_global_unlock(struct lglock *lg);
 
+#ifndef CONFIG_PREEMPT_RT_FULL
+#define lg_global_trylock_relax(name)	lg_global_lock(name)
+#else
+void lg_global_trylock_relax(struct lglock *lg);
+#endif
+
 #endif
--- a/include/linux/spinlock_rt.h
+++ b/include/linux/spinlock_rt.h
@@ -35,6 +35,7 @@ extern int atomic_dec_and_spin_lock(atom
  */
 extern void __lockfunc __rt_spin_lock(struct rt_mutex *lock);
 extern void __lockfunc __rt_spin_unlock(struct rt_mutex *lock);
+extern int __lockfunc __rt_spin_trylock(struct rt_mutex *lock);
 
 #define spin_lock(lock)				\
 	do {					\
--- a/kernel/locking/lglock.c
+++ b/kernel/locking/lglock.c
@@ -105,3 +105,28 @@ void lg_global_unlock(struct lglock *lg)
 	preempt_enable_nort();
 }
 EXPORT_SYMBOL(lg_global_unlock);
+
+#ifdef CONFIG_PREEMPT_RT_FULL
+/*
+ * HACK: If you use this, you get to keep the pieces.
+ * Used in queue_stop_cpus_work() when stop machinery
+ * is called from inactive CPU, so we can't schedule.
+ */
+# define lg_do_trylock_relax(l)			\
+	do {					\
+		while (!__rt_spin_trylock(l))	\
+			cpu_relax();		\
+	} while (0)
+
+void lg_global_trylock_relax(struct lglock *lg)
+{
+	int i;
+
+	lock_acquire_exclusive(&lg->lock_dep_map, 0, 0, NULL, _RET_IP_);
+	for_each_possible_cpu(i) {
+		lg_lock_ptr *lock;
+		lock = per_cpu_ptr(lg->lock, i);
+		lg_do_trylock_relax(lock);
+	}
+}
+#endif
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1001,6 +1001,11 @@ void __lockfunc rt_spin_unlock_wait(spin
 }
 EXPORT_SYMBOL(rt_spin_unlock_wait);
 
+int __lockfunc __rt_spin_trylock(struct rt_mutex *lock)
+{
+	return rt_mutex_trylock(lock);
+}
+
 int __lockfunc rt_spin_trylock(spinlock_t *lock)
 {
 	int ret = rt_mutex_trylock(&lock->lock);

[-- Attachment #3: stomp-machine-deal-clever-with-stopper-lock.patch --]
[-- Type: text/x-patch, Size: 2716 bytes --]

---
 kernel/stop_machine.c |   24 ++++++++++++++----------
 1 file changed, 14 insertions(+), 10 deletions(-)

--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -266,7 +266,7 @@ int stop_two_cpus(unsigned int cpu1, uns
 	struct irq_cpu_stop_queue_work_info call_args;
 	struct multi_stop_data msdata;
 
-	preempt_disable();
+	preempt_disable_nort();
 	msdata = (struct multi_stop_data){
 		.fn = fn,
 		.data = arg,
@@ -299,7 +299,7 @@ int stop_two_cpus(unsigned int cpu1, uns
 	 * This relies on the stopper workqueues to be FIFO.
 	 */
 	if (!cpu_active(cpu1) || !cpu_active(cpu2)) {
-		preempt_enable();
+		preempt_enable_nort();
 		return -ENOENT;
 	}
 
@@ -313,7 +313,7 @@ int stop_two_cpus(unsigned int cpu1, uns
 				 &irq_cpu_stop_queue_work,
 				 &call_args, 1);
 	lg_local_unlock(&stop_cpus_lock);
-	preempt_enable();
+	preempt_enable_nort();
 
 	wait_for_stop_done(&done);
 
@@ -346,7 +346,7 @@ static DEFINE_PER_CPU(struct cpu_stop_wo
 
 static void queue_stop_cpus_work(const struct cpumask *cpumask,
 				 cpu_stop_fn_t fn, void *arg,
-				 struct cpu_stop_done *done)
+				 struct cpu_stop_done *done, bool inactive)
 {
 	struct cpu_stop_work *work;
 	unsigned int cpu;
@@ -360,11 +360,13 @@ static void queue_stop_cpus_work(const s
 	}
 
 	/*
-	 * Disable preemption while queueing to avoid getting
-	 * preempted by a stopper which might wait for other stoppers
-	 * to enter @fn which can lead to deadlock.
+	 * Make sure that all work is queued on all cpus before
+	 * any of the cpus can execute it.
 	 */
-	lg_global_lock(&stop_cpus_lock);
+	if (!inactive)
+		lg_global_lock(&stop_cpus_lock);
+	else
+		lg_global_trylock_relax(&stop_cpus_lock);
 	for_each_cpu(cpu, cpumask)
 		cpu_stop_queue_work(cpu, &per_cpu(stop_cpus_work, cpu));
 	lg_global_unlock(&stop_cpus_lock);
@@ -376,7 +378,7 @@ static int __stop_cpus(const struct cpum
 	struct cpu_stop_done done;
 
 	cpu_stop_init_done(&done, cpumask_weight(cpumask));
-	queue_stop_cpus_work(cpumask, fn, arg, &done);
+	queue_stop_cpus_work(cpumask, fn, arg, &done, false);
 	wait_for_stop_done(&done);
 	return done.executed ? done.ret : -ENOENT;
 }
@@ -572,6 +574,8 @@ static int __init cpu_stop_init(void)
 		INIT_LIST_HEAD(&stopper->works);
 	}
 
+	lg_lock_init(&stop_cpus_lock, "stop_cpus_lock");
+
 	BUG_ON(smpboot_register_percpu_thread(&cpu_stop_threads));
 	stop_machine_initialized = true;
 	return 0;
@@ -667,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
 	set_state(&msdata, MULTI_STOP_PREPARE);
 	cpu_stop_init_done(&done, num_active_cpus());
 	queue_stop_cpus_work(cpu_active_mask, multi_cpu_stop, &msdata,
-			     &done);
+			     &done, true);
 	ret = multi_cpu_stop(&msdata);
 
 	/* Busy wait for completion. */

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Fri, 2014-04-25 at 09:40 +0200, Mike Galbraith wrote:

> Hotplug can still deadlock in rt trees too, and will if you beat it
> hard.

Box actually deadlocks like so.

                                                CPU3 boot.kdump
                                                sys_wait4
                                                do_wait
                                                   read_lock(&tasklist_lock)
                                                      rt_read_lock
                                                         __rt_spin_lock(lock)
                                                            migrate_disable()
                                                               pin_current_cpu()
                                                                  if (hp->grab_lock) {
                                                                     preempt_enable(); <== hmm
                                                                     hotplug_lock(hp); 
                                                                       hp = &__get_cpu_var(hotplug_pcp);  <== hmm
                                                                       struct hotplug_pcp {
                                                                          unplug = 0xffff8800b7d0e540,
                                                                          sync_tsk = 0x0,
                                                                          refcount = 0, <== hmm
                                                                          grab_lock = 1,
                                                                          ...
                                                                          lock = {
                                                                             ...
                                                                             owner = 0xffff8802039f0001,
                                                                             stress-cpu-hotplug_stress.sh?!?
                                                                             <=== he's way over yonder.
                                                                             Yo, dude, would you please NOT
                                                                             take percpu locks with you?

CPU0 stress-cpu-hotplug_stress.sh
sysfs_write_file
dev_attr_store
online_store
device_offline
cpu_subsys_offline
cpu_down
_cpu_down
   cpu_hotplug_begin
       mutex_lock(&cpu_hotplug.lock);
       ...
      check_for_tasks
         write_lock_irq(&tasklist_lock);
         held by CPU3 boot.kdump over there ===>

CPU0 kworker/0:0
cpuset_hotplug_workfn+0x23e/0x380
rebuild_sched_domains+0x15/0x30
rebuild_sched_domains_locked+0x17/0x80
get_online_cpus+0x35/0x50
   mutex_lock(&cpu_hotplug.lock);
   held by stress-cpu-hotplug_stress.sh

twiddle twiddle twiddle...
INFO: task kworker/0:0:4 blocked for more than 120 seconds.

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Sat, 2014-04-26 at 10:38 +0200, Mike Galbraith wrote: 
> On Fri, 2014-04-25 at 09:40 +0200, Mike Galbraith wrote:
> 
> > Hotplug can still deadlock in rt trees too, and will if you beat it
> > hard.
> 
> Box actually deadlocks like so.

...

3.12-rt looks a bit busted migrate_disable/enable() wise.

/me eyeballs 3.10-rt (looks better), confirms 3.10-rt hotplug works,
swipes working code, confirms 3.12-rt now works.  Yup, that was it.

When I fix lg_global_lock() (I think it and Medusa are both busted) I
bet a nickle 3.14-rt will work.

Hm, actually, rt_write_trylock() in swiped 3.10-rt code below (and some
others) look busted to me.  migrate_disable() _after_ grabbing a lock is
too late, no?

---
 include/linux/rwlock_rt.h |   32 ++++++++++++++++++++++++++++----
 kernel/rt.c               |   21 +++++++++++----------
 2 files changed, 39 insertions(+), 14 deletions(-)

--- a/include/linux/rwlock_rt.h
+++ b/include/linux/rwlock_rt.h
@@ -33,50 +33,72 @@ extern void __rt_rwlock_init(rwlock_t *r
 #define read_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
+		migrate_disable();			\
 		flags = rt_read_lock_irqsave(lock);	\
 	} while (0)
 
 #define write_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
+		migrate_disable();			\
 		flags = rt_write_lock_irqsave(lock);	\
 	} while (0)
 
-#define read_lock(lock)		rt_read_lock(lock)
+#define read_lock(lock)					\
+	do {						\
+		migrate_disable();			\
+		rt_read_lock(lock);			\
+	} while (0)
 
 #define read_lock_bh(lock)				\
 	do {						\
 		local_bh_disable();			\
+		migrate_disable();			\
 		rt_read_lock(lock);			\
 	} while (0)
 
 #define read_lock_irq(lock)	read_lock(lock)
 
-#define write_lock(lock)	rt_write_lock(lock)
+#define write_lock(lock)				\
+	do {						\
+		migrate_disable();			\
+		rt_write_lock(lock);			\
+	} while (0)
 
 #define write_lock_bh(lock)				\
 	do {						\
 		local_bh_disable();			\
+		migrate_disable();			\
 		rt_write_lock(lock);			\
 	} while (0)
 
 #define write_lock_irq(lock)	write_lock(lock)
 
-#define read_unlock(lock)	rt_read_unlock(lock)
+#define read_unlock(lock)				\
+	do {						\
+		rt_read_unlock(lock);			\
+		migrate_enable();			\
+	} while (0)
 
 #define read_unlock_bh(lock)				\
 	do {						\
 		rt_read_unlock(lock);			\
+		migrate_enable();			\
 		local_bh_enable();			\
 	} while (0)
 
 #define read_unlock_irq(lock)	read_unlock(lock)
 
-#define write_unlock(lock)	rt_write_unlock(lock)
+#define write_unlock(lock)				\
+	do {						\
+		rt_write_unlock(lock);			\
+		migrate_enable();			\
+	} while (0)
 
 #define write_unlock_bh(lock)				\
 	do {						\
 		rt_write_unlock(lock);			\
+		migrate_enable();			\
 		local_bh_enable();			\
 	} while (0)
 
@@ -87,6 +109,7 @@ extern void __rt_rwlock_init(rwlock_t *r
 		typecheck(unsigned long, flags);	\
 		(void) flags;				\
 		rt_read_unlock(lock);			\
+		migrate_enable();			\
 	} while (0)
 
 #define write_unlock_irqrestore(lock, flags) \
@@ -94,6 +117,7 @@ extern void __rt_rwlock_init(rwlock_t *r
 		typecheck(unsigned long, flags);	\
 		(void) flags;				\
 		rt_write_unlock(lock);			\
+		migrate_enable();			\
 	} while (0)
 
 #endif
--- a/kernel/rt.c
+++ b/kernel/rt.c
@@ -182,10 +182,11 @@ int __lockfunc rt_write_trylock(rwlock_t
 {
 	int ret = rt_mutex_trylock(&rwlock->lock);
 
-	if (ret) {
+	migrate_disable();
+	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-		migrate_disable();
-	}
+	else
+		migrate_enable();
 
 	return ret;
 }
@@ -196,7 +197,10 @@ int __lockfunc rt_write_trylock_irqsave(
 	int ret;
 
 	*flags = 0;
+	migrate_disable();
 	ret = rt_write_trylock(rwlock);
+	if (!ret)
+		migrate_enable();
 	return ret;
 }
 EXPORT_SYMBOL(rt_write_trylock_irqsave);
@@ -211,18 +215,19 @@ int __lockfunc rt_read_trylock(rwlock_t
 	 * but not when read_depth == 0 which means that the lock is
 	 * write locked.
 	 */
+	migrate_disable();
 	if (rt_mutex_owner(lock) != current) {
 		ret = rt_mutex_trylock(lock);
-		if (ret) {
+		if (ret)
 			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-			migrate_disable();
-		}
 	} else if (!rwlock->read_depth) {
 		ret = 0;
 	}
 
 	if (ret)
 		rwlock->read_depth++;
+	else
+		migrate_enable();
 
 	return ret;
 }
@@ -231,7 +236,6 @@ EXPORT_SYMBOL(rt_read_trylock);
 void __lockfunc rt_write_lock(rwlock_t *rwlock)
 {
 	rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
-	migrate_disable();
 	__rt_spin_lock(&rwlock->lock);
 }
 EXPORT_SYMBOL(rt_write_lock);
@@ -246,7 +250,6 @@ void __lockfunc rt_read_lock(rwlock_t *r
 	if (rt_mutex_owner(lock) != current) {
 		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
 		__rt_spin_lock(lock);
-		migrate_disable();
 	}
 	rwlock->read_depth++;
 }
@@ -258,7 +261,6 @@ void __lockfunc rt_write_unlock(rwlock_t
 	/* NOTE: we always pass in '1' for nested, for simplicity */
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	__rt_spin_unlock(&rwlock->lock);
-	migrate_enable();
 }
 EXPORT_SYMBOL(rt_write_unlock);
 
@@ -268,7 +270,6 @@ void __lockfunc rt_read_unlock(rwlock_t
 	if (--rwlock->read_depth == 0) {
 		rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 		__rt_spin_unlock(&rwlock->lock);
-		migrate_enable();
 	}
 }
 EXPORT_SYMBOL(rt_read_unlock);

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

Hi Nicholas,

On Sat, 2014-04-26 at 15:58 +0200, Mike Galbraith wrote: 
> On Sat, 2014-04-26 at 10:38 +0200, Mike Galbraith wrote: 
> > On Fri, 2014-04-25 at 09:40 +0200, Mike Galbraith wrote:
> > 
> > > Hotplug can still deadlock in rt trees too, and will if you beat it
> > > hard.
> > 
> > Box actually deadlocks like so.
> 
> ...
> 
> 3.12-rt looks a bit busted migrate_disable/enable() wise.
> 
> /me eyeballs 3.10-rt (looks better), confirms 3.10-rt hotplug works,
> swipes working code, confirms 3.12-rt now works.  Yup, that was it.

My boxen, including 64 core DL980 that ran hotplug stress for 3 hours
yesterday with pre-pushdown rwlocks, say the migrate_disable/enable
pushdown patches are very definitely busted.

Instead of whacking selective bits, as I did to verify that the rwlock
changes were indeed causing hotplug stress deadlock woes, I'm eyeballing
the lot, twiddling primitives to look like I think they should, after
which I'll let my boxen express their opinions of the result.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Mon, 2014-04-28 at 07:09 +0200, Mike Galbraith wrote: 
> Hi Nicholas,
> 
> On Sat, 2014-04-26 at 15:58 +0200, Mike Galbraith wrote: 
> > On Sat, 2014-04-26 at 10:38 +0200, Mike Galbraith wrote: 
> > > On Fri, 2014-04-25 at 09:40 +0200, Mike Galbraith wrote:
> > > 
> > > > Hotplug can still deadlock in rt trees too, and will if you beat it
> > > > hard.
> > > 
> > > Box actually deadlocks like so.
> > 
> > ...
> > 
> > 3.12-rt looks a bit busted migrate_disable/enable() wise.
> > 
> > /me eyeballs 3.10-rt (looks better), confirms 3.10-rt hotplug works,
> > swipes working code, confirms 3.12-rt now works.  Yup, that was it.
> 
> My boxen, including 64 core DL980 that ran hotplug stress for 3 hours
> yesterday with pre-pushdown rwlocks, say the migrate_disable/enable
> pushdown patches are very definitely busted.

migrate_disable-pushd-down-in-atomic_dec_and_spin_lo.patch

bug: migrate_disable() after blocking is too late.

@@ -1028,12 +1028,12 @@ int atomic_dec_and_spin_lock(atomic_t *a
        /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
        if (atomic_add_unless(atomic, -1, 1))
                return 0;
-       migrate_disable();
        rt_spin_lock(lock);
-       if (atomic_dec_and_test(atomic))
+       if (atomic_dec_and_test(atomic)){
+               migrate_disable();
                return 1;
+       }
        rt_spin_unlock(lock);
-       migrate_enable();
        return 0;
 }
 EXPORT_SYMBOL(atomic_dec_and_spin_lock);

read_lock-migrate_disable-pushdown-to-rt_read_lock.patch

bug: ditto.

@@ -244,8 +246,10 @@ void __lockfunc rt_read_lock(rwlock_t *r
        /*
         * recursive read locks succeed when current owns the lock
         */
-       if (rt_mutex_owner(lock) != current)
+       if (rt_mutex_owner(lock) != current) {
                __rt_spin_lock(lock);
+               migrate_disable();
+       }
        rwlock->read_depth++;
 }

Moving that migrate_disable() up will likely fix my hotplug troubles.
I'll find out when I get back from physical torture (therapy) session.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Mon, 28 Apr 2014 11:09:46 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
 
> migrate_disable-pushd-down-in-atomic_dec_and_spin_lo.patch
> 
> bug: migrate_disable() after blocking is too late.
> 
> @@ -1028,12 +1028,12 @@ int atomic_dec_and_spin_lock(atomic_t *a
>         /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
>         if (atomic_add_unless(atomic, -1, 1))
>                 return 0;
> -       migrate_disable();
>         rt_spin_lock(lock);
> -       if (atomic_dec_and_test(atomic))
> +       if (atomic_dec_and_test(atomic)){
> +               migrate_disable();

Makes sense, as the CPU can go offline right after the lock is grabbed
and before the migrate_disable() is called.

Seems that migrate_disable() must be called before taking the lock as
it is done in every other location.

-- Steve


>                 return 1;
> +       }
>         rt_spin_unlock(lock);
> -       migrate_enable();
>         return 0;
>  }
>  EXPORT_SYMBOL(atomic_dec_and_spin_lock);
> 
> read_lock-migrate_disable-pushdown-to-rt_read_lock.patch
> 
> bug: ditto.
> 
> @@ -244,8 +246,10 @@ void __lockfunc rt_read_lock(rwlock_t *r
>         /*
>          * recursive read locks succeed when current owns the lock
>          */
> -       if (rt_mutex_owner(lock) != current)
> +       if (rt_mutex_owner(lock) != current) {
>                 __rt_spin_lock(lock);
> +               migrate_disable();
> +       }
>         rwlock->read_depth++;
>  }
> 
> Moving that migrate_disable() up will likely fix my hotplug troubles.
> I'll find out when I get back from physical torture (therapy) session.
> 
> -Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Mon, 2014-04-28 at 10:18 -0400, Steven Rostedt wrote: 
> On Mon, 28 Apr 2014 11:09:46 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
>  
> > migrate_disable-pushd-down-in-atomic_dec_and_spin_lo.patch
> > 
> > bug: migrate_disable() after blocking is too late.
> > 
> > @@ -1028,12 +1028,12 @@ int atomic_dec_and_spin_lock(atomic_t *a
> >         /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
> >         if (atomic_add_unless(atomic, -1, 1))
> >                 return 0;
> > -       migrate_disable();
> >         rt_spin_lock(lock);
> > -       if (atomic_dec_and_test(atomic))
> > +       if (atomic_dec_and_test(atomic)){
> > +               migrate_disable();
> 
> Makes sense, as the CPU can go offline right after the lock is grabbed
> and before the migrate_disable() is called.
> 
> Seems that migrate_disable() must be called before taking the lock as
> it is done in every other location.

And for tasklist_lock, seems you also MUST do that prior to trylock as
well, else you'll run afoul of the hotplug beast.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Mon, 2014-04-28 at 16:37 +0200, Mike Galbraith wrote: 
> On Mon, 2014-04-28 at 10:18 -0400, Steven Rostedt wrote: 
> > On Mon, 28 Apr 2014 11:09:46 +0200
> > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> >  
> > > migrate_disable-pushd-down-in-atomic_dec_and_spin_lo.patch
> > > 
> > > bug: migrate_disable() after blocking is too late.
> > > 
> > > @@ -1028,12 +1028,12 @@ int atomic_dec_and_spin_lock(atomic_t *a
> > >         /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
> > >         if (atomic_add_unless(atomic, -1, 1))
> > >                 return 0;
> > > -       migrate_disable();
> > >         rt_spin_lock(lock);
> > > -       if (atomic_dec_and_test(atomic))
> > > +       if (atomic_dec_and_test(atomic)){
> > > +               migrate_disable();
> > 
> > Makes sense, as the CPU can go offline right after the lock is grabbed
> > and before the migrate_disable() is called.
> > 
> > Seems that migrate_disable() must be called before taking the lock as
> > it is done in every other location.
> 
> And for tasklist_lock, seems you also MUST do that prior to trylock as
> well, else you'll run afoul of the hotplug beast.

This lockdep gripe is from the deadlocked crashdump with only the
clearly busted bits patched up.

[  193.033224] ======================================================
[  193.033225] [ INFO: possible circular locking dependency detected ]
[  193.033227] 3.12.18-rt25 #19 Not tainted
[  193.033227] -------------------------------------------------------
[  193.033228] boot.kdump/5422 is trying to acquire lock:
[  193.033237]  (&hp->lock){+.+...}, at: [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033238] 
               but task is already holding lock:
[  193.033241]  (tasklist_lock){+.+...}, at: [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033242] 
               which lock already depends on the new lock.
               
[  193.033242] 
               the existing dependency chain (in reverse order) is:
[  193.033244] 
               -> #1 (tasklist_lock){+.+...}:
[  193.033248]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033250]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033252]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033253]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033257]        [<ffffffff8155e99c>] rt_write_lock+0x2c/0x40
[  193.033260]        [<ffffffff81548169>] _cpu_down+0x219/0x440
[  193.033261]        [<ffffffff815483c0>] cpu_down+0x30/0x50
[  193.033264]        [<ffffffff813711dc>] cpu_subsys_offline+0x1c/0x30
[  193.033267]        [<ffffffff8136c2d5>] device_offline+0x95/0xc0
[  193.033269]        [<ffffffff8136c3e0>] online_store+0x40/0x80
[  193.033271]        [<ffffffff81369813>] dev_attr_store+0x13/0x30
[  193.033274]        [<ffffffff811c8820>] sysfs_write_file+0xf0/0x170
[  193.033277]        [<ffffffff8115a068>] vfs_write+0xc8/0x1d0
[  193.033279]        [<ffffffff8115a500>] SyS_write+0x50/0xa0
[  193.033282]        [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033284] 
               -> #0 (&hp->lock){+.+...}:
[  193.033286]        [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  193.033287]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033289]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033291]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033293]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033295]        [<ffffffff8155e6a5>] rt_spin_lock+0x55/0x70
[  193.033296]        [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033299]        [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  193.033301]        [<ffffffff8155e947>] rt_read_lock+0x47/0x60
[  193.033303]        [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033305]        [<ffffffff8104777e>] SyS_wait4+0x9e/0x100
[  193.033307]        [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033307] 
               other info that might help us debug this:
               
[  193.033308]  Possible unsafe locking scenario:
               
[  193.033309]        CPU0                    CPU1
[  193.033309]        ----                    ----
[  193.033310]   lock(tasklist_lock);
[  193.033312]                                lock(&hp->lock);
[  193.033313]                                lock(tasklist_lock);
[  193.033314]   lock(&hp->lock);
[  193.033315] 
                *** DEADLOCK ***
               
[  193.033316] 1 lock held by boot.kdump/5422:
[  193.033319]  #0:  (tasklist_lock){+.+...}, at: [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033320] 
               stack backtrace:
[  193.033322] CPU: 0 PID: 5422 Comm: boot.kdump Not tainted 3.12.18-rt25 #19
[  193.033323] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[  193.033326]  ffff880200550818 ffff8802004e5ad8 ffffffff8155538c 0000000000000000
[  193.033328]  0000000000000000 ffff8802004e5b28 ffffffff8154d0df ffff8802004e5b18
[  193.033330]  ffff8802004e5b50 ffff880200550818 ffff8802005507e0 ffff880200550818
[  193.033331] Call Trace:
[  193.033335]  [<ffffffff8155538c>] dump_stack+0x4f/0x91
[  193.033337]  [<ffffffff8154d0df>] print_circular_bug+0xd3/0xe4
[  193.033339]  [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  193.033342]  [<ffffffff8107e1f5>] ? sched_clock_local+0x25/0x90
[  193.033344]  [<ffffffff8107e388>] ? sched_clock_cpu+0xa8/0x120
[  193.033346]  [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033348]  [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033350]  [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033352]  [<ffffffff8155da11>] ? rt_spin_lock_slowlock+0x231/0x280
[  193.033354]  [<ffffffff8155d911>] ? rt_spin_lock_slowlock+0x131/0x280
[  193.033356]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033358]  [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033360]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033362]  [<ffffffff8155e6a5>] rt_spin_lock+0x55/0x70
[  193.033363]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033365]  [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033367]  [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  193.033369]  [<ffffffff8155e947>] rt_read_lock+0x47/0x60
[  193.033371]  [<ffffffff81046a5b>] ? do_wait+0xbb/0x2a0
[  193.033373]  [<ffffffff8155cd39>] ? schedule+0x29/0x90
[  193.033374]  [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033378]  [<ffffffff8112ded6>] ? might_fault+0x56/0xb0
[  193.033380]  [<ffffffff8104777e>] SyS_wait4+0x9e/0x100
[  193.033382]  [<ffffffff81566cc7>] ? sysret_check+0x1b/0x56
[  193.033384]  [<ffffffff81045d50>] ? task_stopped_code+0xa0/0xa0
[  193.033386]  [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033845] SMP alternatives: lockdep: fixing up alternatives

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Mon, 2014-04-28 at 16:37 +0200, Mike Galbraith wrote: 

> > Seems that migrate_disable() must be called before taking the lock as
> > it is done in every other location.
> 
> And for tasklist_lock, seems you also MUST do that prior to trylock as
> well, else you'll run afoul of the hotplug beast.

Bah.  Futzing with dmesg while stress script is running is either a very
bad idea, or a very good test.  Both virgin 3.10-rt and 3.12-rt with new
bugs squashed will deadlock.

Too bad I kept on testing, I liked the notion that hotplug was solid ;-)

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Tue, 29 Apr 2014 07:21:09 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Mon, 2014-04-28 at 16:37 +0200, Mike Galbraith wrote: 
> 
> > > Seems that migrate_disable() must be called before taking the lock as
> > > it is done in every other location.
> > 
> > And for tasklist_lock, seems you also MUST do that prior to trylock as
> > well, else you'll run afoul of the hotplug beast.
> 
> Bah.  Futzing with dmesg while stress script is running is either a very
> bad idea, or a very good test.  Both virgin 3.10-rt and 3.12-rt with new
> bugs squashed will deadlock.
> 
> Too bad I kept on testing, I liked the notion that hotplug was solid ;-)

I was able to stress cpu hotplug on 3.12-rt after applying the
following patch.

If there's no complaints about it. I'm going to add this to the 3.12-rt
stable tree. As without it, it fails horribly with the cpu hotplug
stress test, and I wont release a stable kernel that does that.

-- Steve

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

diff --git a/kernel/rt.c b/kernel/rt.c
index bb72347..4f2a613 100644
--- a/kernel/rt.c
+++ b/kernel/rt.c
@@ -180,12 +180,15 @@ EXPORT_SYMBOL(_mutex_unlock);
  */
 int __lockfunc rt_write_trylock(rwlock_t *rwlock)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	int ret;
+
+	migrate_disable();
+	ret = rt_mutex_trylock(&rwlock->lock);
 
-	if (ret) {
+	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-		migrate_disable();
-	}
+	else
+		migrate_enable();
 
 	return ret;
 }
@@ -212,11 +215,12 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
 	 * write locked.
 	 */
 	if (rt_mutex_owner(lock) != current) {
+		migrate_disable();
 		ret = rt_mutex_trylock(lock);
-		if (ret) {
+		if (ret)
 			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-			migrate_disable();
-		}
+		else
+			migrate_enable();
 	} else if (!rwlock->read_depth) {
 		ret = 0;
 	}
@@ -245,8 +249,8 @@ void __lockfunc rt_read_lock(rwlock_t *rwlock)
 	 */
 	if (rt_mutex_owner(lock) != current) {
 		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
-		__rt_spin_lock(lock);
 		migrate_disable();
+		__rt_spin_lock(lock);
 	}
 	rwlock->read_depth++;
 }

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Tue, 2014-04-29 at 20:13 -0400, Steven Rostedt wrote: 
> On Tue, 29 Apr 2014 07:21:09 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > On Mon, 2014-04-28 at 16:37 +0200, Mike Galbraith wrote: 
> > 
> > > > Seems that migrate_disable() must be called before taking the lock as
> > > > it is done in every other location.
> > > 
> > > And for tasklist_lock, seems you also MUST do that prior to trylock as
> > > well, else you'll run afoul of the hotplug beast.
> > 
> > Bah.  Futzing with dmesg while stress script is running is either a very
> > bad idea, or a very good test.  Both virgin 3.10-rt and 3.12-rt with new
> > bugs squashed will deadlock.
> > 
> > Too bad I kept on testing, I liked the notion that hotplug was solid ;-)
> 
> I was able to stress cpu hotplug on 3.12-rt after applying the
> following patch.
> 
> If there's no complaints about it. I'm going to add this to the 3.12-rt
> stable tree. As without it, it fails horribly with the cpu hotplug
> stress test, and I wont release a stable kernel that does that.

My local boxen are happy, 64 core box with 14-rt seems happy as well,
though I couldn't let it burn for long.

BTW, that dmesg business went into hiding.  I didn't have time to put
virgin 10-rt back on and play around poking both kernels this that and
the other way again, but seems there's some phase-of-moon factor there.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 09:43 +0200, Mike Galbraith wrote: 
> On Tue, 2014-04-29 at 20:13 -0400, Steven Rostedt wrote: 
> > On Tue, 29 Apr 2014 07:21:09 +0200
> > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > 
> > > On Mon, 2014-04-28 at 16:37 +0200, Mike Galbraith wrote: 
> > > 
> > > > > Seems that migrate_disable() must be called before taking the lock as
> > > > > it is done in every other location.
> > > > 
> > > > And for tasklist_lock, seems you also MUST do that prior to trylock as
> > > > well, else you'll run afoul of the hotplug beast.
> > > 
> > > Bah.  Futzing with dmesg while stress script is running is either a very
> > > bad idea, or a very good test.  Both virgin 3.10-rt and 3.12-rt with new
> > > bugs squashed will deadlock.
> > > 
> > > Too bad I kept on testing, I liked the notion that hotplug was solid ;-)
> > 
> > I was able to stress cpu hotplug on 3.12-rt after applying the
> > following patch.
> > 
> > If there's no complaints about it. I'm going to add this to the 3.12-rt
> > stable tree. As without it, it fails horribly with the cpu hotplug
> > stress test, and I wont release a stable kernel that does that.
> 
> My local boxen are happy, 64 core box with 14-rt seems happy as well,
> though I couldn't let it burn for long.

And 3.12 looks stable on 64 core DL980 as well.  (If it survived a 24
hour busy+stress session I'd still likely fall outta my chair though)

My kinda sorta 3.12-rt enterprise to be kernel wasn't stable on DL980,
while appearing just fine on small boxen, which made me suspect that
there was still a big box something lurking, only raising its ugly head
in the fatter kernel.  That wasn't an rt problem after all, someone in
enterprise land just didn't stack their goody pile quite high enough
while wedging upstream into the stable base kernel, which bent rt.

The End.. I hope.  I've had enough hotplug entertainment for a while.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 30 Apr 2014 15:06:29 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:


> The End.. I hope.  I've had enough hotplug entertainment for a while.

Not for me. 3.14-rt stress-cpu-hotplug crashes quickly. But it's a
different issues than what my patch addressed. I'm still debugging it.

-- Steve

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 09:15 -0400, Steven Rostedt wrote: 
> On Wed, 30 Apr 2014 15:06:29 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> 
> > The End.. I hope.  I've had enough hotplug entertainment for a while.
> 
> Not for me. 3.14-rt stress-cpu-hotplug crashes quickly. But it's a
> different issues than what my patch addressed. I'm still debugging it.

If you didn't fix the two bugs I showed, and (wisely) didn't look at the
beautiful lglock patches I posted (no frozen shark, I'm disappointed;),
your patch won't help.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 30 Apr 2014 16:00:03 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Wed, 2014-04-30 at 09:15 -0400, Steven Rostedt wrote: 
> > On Wed, 30 Apr 2014 15:06:29 +0200
> > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > 
> > 
> > > The End.. I hope.  I've had enough hotplug entertainment for a while.
> > 
> > Not for me. 3.14-rt stress-cpu-hotplug crashes quickly. But it's a
> > different issues than what my patch addressed. I'm still debugging it.
> 
> If you didn't fix the two bugs I showed, and (wisely) didn't look at the
> beautiful lglock patches I posted (no frozen shark, I'm disappointed;),
> your patch won't help.

Mike,

I'm testing it now. But could you please post them as regular patches.
They were attachments to this thread, and were not something that stood
out.

Thanks,

-- Steve

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 30 Apr 2014 10:19:19 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> I'm testing it now. But could you please post them as regular patches.
> They were attachments to this thread, and were not something that stood
> out.

With your two patches, it still crashes exactly the same way. I
probably should remove my debug just in case, but I think this box has
another problem with it.

-- Steve

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 10:33 -0400, Steven Rostedt wrote: 
> On Wed, 30 Apr 2014 10:19:19 -0400
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > I'm testing it now. But could you please post them as regular patches.
> > They were attachments to this thread, and were not something that stood
> > out.
> 
> With your two patches, it still crashes exactly the same way. I
> probably should remove my debug just in case, but I think this box has
> another problem with it.

You killed this hunk of hotplug-light-get-online-cpus.patch

@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
                /* CPU didn't die: tell everyone.  Can't complain. */
                smpboot_unpark_threads(cpu);
                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
-               goto out_release;
+               goto out_cancel;
        }
        BUG_ON(cpu_online(cpu));

..and fixed this too?

Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
should be while (atomic_read(&done.nr_todo)) 

@@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
        ret = multi_cpu_stop(&msdata);

        /* Busy wait for completion. */
-       while (!completion_done(&done.completion))
+       while (!atomic_read(&done.nr_todo))
                cpu_relax();

        mutex_unlock(&stop_cpus_mutex);

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 30 Apr 2014 16:54:46 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Wed, 2014-04-30 at 10:33 -0400, Steven Rostedt wrote: 
> > On Wed, 30 Apr 2014 10:19:19 -0400
> > Steven Rostedt <rostedt@goodmis.org> wrote:
> > 
> > > I'm testing it now. But could you please post them as regular patches.
> > > They were attachments to this thread, and were not something that stood
> > > out.
> > 
> > With your two patches, it still crashes exactly the same way. I
> > probably should remove my debug just in case, but I think this box has
> > another problem with it.
> 
> You killed this hunk of hotplug-light-get-online-cpus.patch
> 
> @@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
>                 /* CPU didn't die: tell everyone.  Can't complain. */
>                 smpboot_unpark_threads(cpu);
>                 cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
> -               goto out_release;
> +               goto out_cancel;

I added this, but it only happens on the failed case, which I don't
think is an issue with what I'm dealing with.

>         }
>         BUG_ON(cpu_online(cpu));
> 
> ..and fixed this too?
> 
> Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> should be while (atomic_read(&done.nr_todo)) 
> 
> @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
>         ret = multi_cpu_stop(&msdata);
> 
>         /* Busy wait for completion. */
> -       while (!completion_done(&done.completion))
> +       while (!atomic_read(&done.nr_todo))

I don't see this in the code. That is, there is no "completion_done()"
in stop_machine_from_inactive_cpu(). It is already an atomic_read().

-- Steve

>                 cpu_relax();
> 
>         mutex_unlock(&stop_cpus_mutex);

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 11:11 -0400, Steven Rostedt wrote:

> > Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> > should be while (atomic_read(&done.nr_todo)) 
> > 
> > @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
> >         ret = multi_cpu_stop(&msdata);
> > 
> >         /* Busy wait for completion. */
> > -       while (!completion_done(&done.completion))
> > +       while (!atomic_read(&done.nr_todo))
                   ^--- that ! needs to go away 
> 
> I don't see this in the code. That is, there is no "completion_done()"
> in stop_machine_from_inactive_cpu(). It is already an atomic_read().

Yes, but it should read "while (atomic_read(&done.nr_todo))"

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 30 Apr 2014 17:15:57 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Wed, 2014-04-30 at 11:11 -0400, Steven Rostedt wrote:
> 
> > > Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> > > should be while (atomic_read(&done.nr_todo)) 
> > > 
> > > @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
> > >         ret = multi_cpu_stop(&msdata);
> > > 
> > >         /* Busy wait for completion. */
> > > -       while (!completion_done(&done.completion))
> > > +       while (!atomic_read(&done.nr_todo))
>                    ^--- that ! needs to go away 
> > 
> > I don't see this in the code. That is, there is no "completion_done()"
> > in stop_machine_from_inactive_cpu(). It is already an atomic_read().
> 
> Yes, but it should read "while (atomic_read(&done.nr_todo))"

Ah, this would have been better if you had sent a patch. I misread what
you talked about.

Yes, this was the culprit of my failures. After removing the '!', it
worked.

Care to send a patch :-)

-- Steve

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 11:48 -0400, Steven Rostedt wrote: 
> On Wed, 30 Apr 2014 17:15:57 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > On Wed, 2014-04-30 at 11:11 -0400, Steven Rostedt wrote:
> > 
> > > > Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> > > > should be while (atomic_read(&done.nr_todo)) 
> > > > 
> > > > @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
> > > >         ret = multi_cpu_stop(&msdata);
> > > > 
> > > >         /* Busy wait for completion. */
> > > > -       while (!completion_done(&done.completion))
> > > > +       while (!atomic_read(&done.nr_todo))
> >                    ^--- that ! needs to go away 
> > > 
> > > I don't see this in the code. That is, there is no "completion_done()"
> > > in stop_machine_from_inactive_cpu(). It is already an atomic_read().
> > 
> > Yes, but it should read "while (atomic_read(&done.nr_todo))"
> 
> Ah, this would have been better if you had sent a patch. I misread what
> you talked about.
> 
> Yes, this was the culprit of my failures. After removing the '!', it
> worked.
> 
> Care to send a patch :-)

I figured those two were just edit patch, done, but yeah, I can do that.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 11:48 -0400, Steven Rostedt wrote: 
> On Wed, 30 Apr 2014 17:15:57 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > On Wed, 2014-04-30 at 11:11 -0400, Steven Rostedt wrote:
> > 
> > > > Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> > > > should be while (atomic_read(&done.nr_todo)) 
> > > > 
> > > > @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
> > > >         ret = multi_cpu_stop(&msdata);
> > > > 
> > > >         /* Busy wait for completion. */
> > > > -       while (!completion_done(&done.completion))
> > > > +       while (!atomic_read(&done.nr_todo))
> >                    ^--- that ! needs to go away 
> > > 
> > > I don't see this in the code. That is, there is no "completion_done()"
> > > in stop_machine_from_inactive_cpu(). It is already an atomic_read().
> > 
> > Yes, but it should read "while (atomic_read(&done.nr_todo))"
> 
> Ah, this would have been better if you had sent a patch. I misread what
> you talked about.
> 
> Yes, this was the culprit of my failures. After removing the '!', it
> worked.

Hah!  I knew you were just hiding, you sneaky little SOB ;-)


[50661.070049] smpboot: Booting Node 0 Processor 15 APIC 0x36
[50661.142381] kvm: enabling virtualization on CPU15
[50661.142397] BUG: unable to handle kernel NULL pointer dereference at           (null)
[50661.142417] IP: [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142420] PGD 0
[50661.142422] Oops: 0000 [#1] PREEMPT SMP
[50661.142470] Modules linked in: nfsd(F) lockd(F) nfs_acl(F) auth_rpcgss(F) sunrpc(F) autofs4(F) binfmt_misc(F) edd(F) af_packet(F) bridge(F) stp(F) llc(F) cpufreq_conservative(F) cpufreq_ondemand(F) cpufreq_userspace(F) cpufreq_powersave(F) pcc_cpufreq(F) fuse(F) loop(F) md_mod(F) dm_mod(F) iTCO_wdt(F) iTCO_vendor_support(F) gpio_ich(F) vhost_net(F) macvtap(F) macvlan(F) vhost(F) tun(F) i7core_edac(F) netxen_nic(F) kvm_intel(F) joydev(F) shpchp(F) edac_core(F) hid_generic(F) kvm(F) ipmi_si(F) sr_mod(F) ipmi_msghandler(F) bnx2(F) cdrom(F) sg(F) hpilo(F) hpwdt(F) ehci_pci(F) lpc_ich(F) mfd_core(F) acpi_power_meter(F) pcspkr(F) button(F) ext4(F) jbd2(F) mbcache(F) crc16(F) usbhid(F) uhci_hcd(F) ehci_hcd(F) usbcore(F) sd_mod(F) usb_common(F) thermal(F) processor(F) scsi_dh_rdac(F) scsi_dh_alua(F) scsi_dh_emc(F)
[50661.142475]  scsi_dh_hp_sw(F) scsi_dh(F) ata_generic(F) ata_piix(F) libata(F) cciss(F) hpsa(F) scsi_mod(F)
[50661.142479] CPU: 39 PID: 283 Comm: migration/39 Tainted: GF            3.14.2-rt1 #667
[50661.142481] Hardware name: Hewlett-Packard ProLiant DL980 G7, BIOS P66 07/07/2010
[50661.142482] task: ffff880274515bb0 ti: ffff88027454e000 task.ti: ffff88027454e000
[50661.142486] RIP: 0010:[<ffffffff810922f1>]  [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142487] RSP: 0018:ffff88027454fda8  EFLAGS: 00010002
[50661.142488] RAX: 0000000080000001 RBX: ffff880275581eb8 RCX: 0000000000000000
[50661.142488] RDX: ffffffff81aacec0 RSI: 0000000000000100 RDI: 0000000000000000
[50661.142489] RBP: ffff8802772ee9b0 R08: 0000000000000000 R09: ffffffff81aacec0
[50661.142490] R10: 0000000000000000 R11: ffffffff8103d640 R12: ffffffff810f26c0
[50661.142490] R13: ffff880275581e88 R14: ffff8802772ee9b8 R15: ffff88027454e010
[50661.142492] FS:  0000000000000000(0000) GS:ffff8802772e0000(0000) knlGS:0000000000000000
[50661.142493] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[50661.142494] CR2: 0000000000000000 CR3: 0000000001a0f000 CR4: 00000000000007e0
[50661.142494] Stack:
[50661.142505]  ffff880275581eb8 ffffffff810f2555 ffff880274515bb0 0000000000000005
[50661.142508]  0000000000000001 0000000000000001 0140000000000000 0000000000000001
[50661.142512]  ffff880274515bb0 ffff88027454e000 ffff8802772f4020 0000000000000005
[50661.142512] Call Trace:
[50661.142526]  [<ffffffff810f2555>] ? cpu_stopper_thread+0x125/0x1a0
[50661.142530]  [<ffffffff8108ba2d>] ? smpboot_thread_fn+0x23d/0x320
[50661.142533]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
[50661.142535]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
[50661.142543]  [<ffffffff81083c32>] ? kthread+0xd2/0xe0
[50661.142545]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
[50661.142553]  [<ffffffff815337cc>] ? ret_from_fork+0x7c/0xb0
[50661.142555]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
[50661.142568] Code: fd ff ff 0f 1f 80 00 00 00 00 31 d2 e9 09 fd ff ff 66 0f 1f 84 00 00 00 00 00 ba 08 00 00 00 be 0f 00 00 00 e9 f1 fc ff ff 90 53 <48> 8b 07 48 89 fb a8 0c 75 08 48 8b 47 08 a8 0c 74 11 be ba 06
[50661.142570] RIP  [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142570]  RSP <ffff88027454fda8>
[50661.142571] CR2: 0000000000000000

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Thu, 01 May 2014 19:36:18 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Wed, 2014-04-30 at 11:48 -0400, Steven Rostedt wrote: 
> > On Wed, 30 Apr 2014 17:15:57 +0200
> > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > 
> > > On Wed, 2014-04-30 at 11:11 -0400, Steven Rostedt wrote:
> > > 
> > > > > Another little bug.  This hunk of patches/stomp-machine-raw-lock.patch
> > > > > should be while (atomic_read(&done.nr_todo)) 
> > > > > 
> > > > > @@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (
> > > > >         ret = multi_cpu_stop(&msdata);
> > > > > 
> > > > >         /* Busy wait for completion. */
> > > > > -       while (!completion_done(&done.completion))
> > > > > +       while (!atomic_read(&done.nr_todo))
> > >                    ^--- that ! needs to go away 
> > > > 
> > > > I don't see this in the code. That is, there is no "completion_done()"
> > > > in stop_machine_from_inactive_cpu(). It is already an atomic_read().
> > > 
> > > Yes, but it should read "while (atomic_read(&done.nr_todo))"
> > 
> > Ah, this would have been better if you had sent a patch. I misread what
> > you talked about.
> > 
> > Yes, this was the culprit of my failures. After removing the '!', it
> > worked.
> 
> Hah!  I knew you were just hiding, you sneaky little SOB ;-)

What's this from? A new bug that had all the patches applied? Or was
this without one of the patches?

-- Steve

> 
> 
> [50661.070049] smpboot: Booting Node 0 Processor 15 APIC 0x36
> [50661.142381] kvm: enabling virtualization on CPU15
> [50661.142397] BUG: unable to handle kernel NULL pointer dereference at           (null)
> [50661.142417] IP: [<ffffffff810922f1>] wake_up_process+0x1/0x40
> [50661.142420] PGD 0
> [50661.142422] Oops: 0000 [#1] PREEMPT SMP
> [50661.142470] Modules linked in: nfsd(F) lockd(F) nfs_acl(F) auth_rpcgss(F) sunrpc(F) autofs4(F) binfmt_misc(F) edd(F) af_packet(F) bridge(F) stp(F) llc(F) cpufreq_conservative(F) cpufreq_ondemand(F) cpufreq_userspace(F) cpufreq_powersave(F) pcc_cpufreq(F) fuse(F) loop(F) md_mod(F) dm_mod(F) iTCO_wdt(F) iTCO_vendor_support(F) gpio_ich(F) vhost_net(F) macvtap(F) macvlan(F) vhost(F) tun(F) i7core_edac(F) netxen_nic(F) kvm_intel(F) joydev(F) shpchp(F) edac_core(F) hid_generic(F) kvm(F) ipmi_si(F) sr_mod(F) ipmi_msghandler(F) bnx2(F) cdrom(F) sg(F) hpilo(F) hpwdt(F) ehci_pci(F) lpc_ich(F) mfd_core(F) acpi_power_meter(F) pcspkr(F) button(F) ext4(F) jbd2(F) mbcache(F) crc16(F) usbhid(F) uhci_hcd(F) ehci_hcd(F) usbcore(F) sd_mod(F) usb_common(F) thermal(F) processor(F) scsi_dh_rdac(F) scsi_dh_alua(F) scsi_dh_emc(F)
> [50661.142475]  scsi_dh_hp_sw(F) scsi_dh(F) ata_generic(F) ata_piix(F) libata(F) cciss(F) hpsa(F) scsi_mod(F)
> [50661.142479] CPU: 39 PID: 283 Comm: migration/39 Tainted: GF            3.14.2-rt1 #667
> [50661.142481] Hardware name: Hewlett-Packard ProLiant DL980 G7, BIOS P66 07/07/2010
> [50661.142482] task: ffff880274515bb0 ti: ffff88027454e000 task.ti: ffff88027454e000
> [50661.142486] RIP: 0010:[<ffffffff810922f1>]  [<ffffffff810922f1>] wake_up_process+0x1/0x40
> [50661.142487] RSP: 0018:ffff88027454fda8  EFLAGS: 00010002
> [50661.142488] RAX: 0000000080000001 RBX: ffff880275581eb8 RCX: 0000000000000000
> [50661.142488] RDX: ffffffff81aacec0 RSI: 0000000000000100 RDI: 0000000000000000
> [50661.142489] RBP: ffff8802772ee9b0 R08: 0000000000000000 R09: ffffffff81aacec0
> [50661.142490] R10: 0000000000000000 R11: ffffffff8103d640 R12: ffffffff810f26c0
> [50661.142490] R13: ffff880275581e88 R14: ffff8802772ee9b8 R15: ffff88027454e010
> [50661.142492] FS:  0000000000000000(0000) GS:ffff8802772e0000(0000) knlGS:0000000000000000
> [50661.142493] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [50661.142494] CR2: 0000000000000000 CR3: 0000000001a0f000 CR4: 00000000000007e0
> [50661.142494] Stack:
> [50661.142505]  ffff880275581eb8 ffffffff810f2555 ffff880274515bb0 0000000000000005
> [50661.142508]  0000000000000001 0000000000000001 0140000000000000 0000000000000001
> [50661.142512]  ffff880274515bb0 ffff88027454e000 ffff8802772f4020 0000000000000005
> [50661.142512] Call Trace:
> [50661.142526]  [<ffffffff810f2555>] ? cpu_stopper_thread+0x125/0x1a0
> [50661.142530]  [<ffffffff8108ba2d>] ? smpboot_thread_fn+0x23d/0x320
> [50661.142533]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
> [50661.142535]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
> [50661.142543]  [<ffffffff81083c32>] ? kthread+0xd2/0xe0
> [50661.142545]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
> [50661.142553]  [<ffffffff815337cc>] ? ret_from_fork+0x7c/0xb0
> [50661.142555]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
> [50661.142568] Code: fd ff ff 0f 1f 80 00 00 00 00 31 d2 e9 09 fd ff ff 66 0f 1f 84 00 00 00 00 00 ba 08 00 00 00 be 0f 00 00 00 e9 f1 fc ff ff 90 53 <48> 8b 07 48 89 fb a8 0c 75 08 48 8b 47 08 a8 0c 74 11 be ba 06
> [50661.142570] RIP  [<ffffffff810922f1>] wake_up_process+0x1/0x40
> [50661.142570]  RSP <ffff88027454fda8>
> [50661.142571] CR2: 0000000000000000
> 

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Thu, 2014-05-01 at 14:42 -0400, Steven Rostedt wrote: 
> On Thu, 01 May 2014 19:36:18 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 

> > Hah!  I knew you were just hiding, you sneaky little SOB ;-)
> 
> What's this from? A new bug that had all the patches applied? Or was
> this without one of the patches?

It's with all patches applied.  It's not new, it has muddied the water
during other hunting expeditions.  You may never see it in a box with a
sane topology, that box is kinda special.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

I fired off a 100 iteration run on 64 core box.  If it's still alive in
the morning, it should still be busy as hell.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Wed, 2014-04-30 at 10:19 -0400, Steven Rostedt wrote: 
> On Wed, 30 Apr 2014 16:00:03 +0200
> Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> 
> > On Wed, 2014-04-30 at 09:15 -0400, Steven Rostedt wrote: 
> > > On Wed, 30 Apr 2014 15:06:29 +0200
> > > Mike Galbraith <umgwanakikbuti@gmail.com> wrote:
> > > 
> > > 
> > > > The End.. I hope.  I've had enough hotplug entertainment for a while.
> > > 
> > > Not for me. 3.14-rt stress-cpu-hotplug crashes quickly. But it's a
> > > different issues than what my patch addressed. I'm still debugging it.
> > 
> > If you didn't fix the two bugs I showed, and (wisely) didn't look at the
> > beautiful lglock patches I posted (no frozen shark, I'm disappointed;),
> > your patch won't help.
> 
> Mike,
> 
> I'm testing it now. But could you please post them as regular patches.
> They were attachments to this thread, and were not something that stood
> out.

They were meant to not stick out :)  I showed what I did to deal with
that damn lglock, but showing them at all felt more akin to chumming the
waters for frozen sharks than posting patches.

'spose I could try to muster up some courage, showing them put a pretty
big dent in my supply.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Sebastian Andrzej Siewior]

* Mike Galbraith | 2014-04-19 16:46:06 [+0200]:

>Hi Sebastian,
Hi Mike,

>This hunk in hotplug-light-get-online-cpus.patch looks like a bug.
>
>@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
>                /* CPU didn't die: tell everyone.  Can't complain. */
>                smpboot_unpark_threads(cpu);
>                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
>-               goto out_release;
>+               goto out_cancel;
>        }
>        BUG_ON(cpu_online(cpu));

Yes, it looks like it. v3.12-rt did not have this…

Sebastian

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Fri, 2014-05-02 at 12:09 +0200, Sebastian Andrzej Siewior wrote: 
> * Mike Galbraith | 2014-04-19 16:46:06 [+0200]:
> 
> >Hi Sebastian,
> Hi Mike,
> 
> >This hunk in hotplug-light-get-online-cpus.patch looks like a bug.
> >
> >@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int
> >                /* CPU didn't die: tell everyone.  Can't complain. */
> >                smpboot_unpark_threads(cpu);
> >                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
> >-               goto out_release;
> >+               goto out_cancel;
> >        }
> >        BUG_ON(cpu_online(cpu));
> 
> Yes, it looks like it. v3.12-rt did not have this…

I just sent a set of dinky patches with which patch to fold them into,
along with my way of dealing with the new stopper_lock.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Fri, 2014-04-11 at 20:57 +0200, Sebastian Andrzej Siewior wrote:

> This -RT series didn't crashed within ~4h testing on my ARM and
> x86-32.
> x86-64 crashed after I started hackbench. I figured out that the crash
> does not happen with lazy-preempt disabled. Therefore the last but one
> patch in the queue disables lazy preempt on x86-64. With this change the
> test box survived ~2h without a crash. I look at this later but it looks
> good now.

I think the below fixes it (in a more or less minimalist way), but it's
not very pretty.  Methinks it would be prettier to either clone the x86
percpu + fold logic, or neutralize that optimization completely when
PREEMPT_LAZY is enabled.

x86_32 bit is completely untested, x86_64 hasn't exploded.. yet :) 

---
 include/linux/preempt.h        |    3 +--
 arch/x86/include/asm/preempt.h |    8 ++++++++
 arch/x86/kernel/asm-offsets.c  |    1 +
 arch/x86/kernel/entry_32.S     |    9 ++++++---
 arch/x86/kernel/entry_64.S     |    7 +++++--
 5 files changed, 21 insertions(+), 7 deletions(-)

--- a/include/linux/preempt.h
+++ b/include/linux/preempt.h
@@ -126,8 +126,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	if (unlikely(__preempt_count_dec_and_test() || \
-				test_thread_flag(TIF_NEED_RESCHED_LAZY))) \
+	if (unlikely(__preempt_count_dec_and_test())) \
 		__preempt_schedule_context(); \
 } while (0)
 #else
--- a/arch/x86/include/asm/preempt.h
+++ b/arch/x86/include/asm/preempt.h
@@ -94,7 +94,11 @@ static __always_inline bool __preempt_co
 {
 	if (____preempt_count_dec_and_test())
 		return true;
+#ifdef CONFIG_PREEMPT_LAZY
 	return test_thread_flag(TIF_NEED_RESCHED_LAZY);
+#else
+	return false;
+#endif
 }
 
 /*
@@ -102,8 +106,12 @@ static __always_inline bool __preempt_co
  */
 static __always_inline bool should_resched(void)
 {
+#ifdef CONFIG_PREEMPT_LAZY
 	return unlikely(!__this_cpu_read_4(__preempt_count) || \
 			test_thread_flag(TIF_NEED_RESCHED_LAZY));
+#else
+	return unlikely(!__this_cpu_read_4(__preempt_count));
+#endif
 }
 
 #ifdef CONFIG_PREEMPT
--- a/arch/x86/kernel/asm-offsets.c
+++ b/arch/x86/kernel/asm-offsets.c
@@ -72,4 +72,5 @@ void common(void) {
 
 	BLANK();
 	DEFINE(PTREGS_SIZE, sizeof(struct pt_regs));
+	DEFINE(_PREEMPT_ENABLED, PREEMPT_ENABLED);
 }
--- a/arch/x86/kernel/entry_32.S
+++ b/arch/x86/kernel/entry_32.S
@@ -365,19 +365,22 @@ ENTRY(resume_kernel)
 need_resched:
 	# preempt count == 0 + NEED_RS set?
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
+	jnz restore_all
+#else
 	jz test_int_off
 
 	# atleast preempt count == 0 ?
-	cmpl $_TIF_NEED_RESCHED,PER_CPU_VAR(__preempt_count)
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
 	jne restore_all
 
 	cmpl $0,TI_preempt_lazy_count(%ebp)	# non-zero preempt_lazy_count ?
 	jnz restore_all
 
-	testl $_TIF_NEED_RESCHED_LAZY, %ecx
+	testl $_TIF_NEED_RESCHED_LAZY, TI_flags(%ebp)
 	jz restore_all
-
 test_int_off:
+#endif
 	testl $X86_EFLAGS_IF,PT_EFLAGS(%esp)	# interrupts off (exception path) ?
 	jz restore_all
 	call preempt_schedule_irq
--- a/arch/x86/kernel/entry_64.S
+++ b/arch/x86/kernel/entry_64.S
@@ -1104,10 +1104,13 @@ ENTRY(native_iret)
 	/* rcx:	 threadinfo. interrupts off. */
 ENTRY(retint_kernel)
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
+	jnz  retint_restore_args
+#else
 	jz  check_int_off
 
 	# atleast preempt count == 0 ?
-	cmpl $_TIF_NEED_RESCHED,PER_CPU_VAR(__preempt_count)
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
 	jnz retint_restore_args
 
 	cmpl $0, TI_preempt_lazy_count(%rcx)
@@ -1115,8 +1118,8 @@ ENTRY(retint_kernel)
 
 	bt $TIF_NEED_RESCHED_LAZY,TI_flags(%rcx)
 	jnc  retint_restore_args
-
 check_int_off:
+#endif
 	bt   $9,EFLAGS-ARGOFFSET(%rsp)	/* interrupts off? */
 	jnc  retint_restore_args
 	call preempt_schedule_irq

Re: [ANNOUNCE] 3.14-rt1

[Steven Rostedt]

On Wed, 23 Apr 2014 12:37:05 +0200
Mike Galbraith <umgwanakikbuti@gmail.com> wrote:

> On Fri, 2014-04-11 at 20:57 +0200, Sebastian Andrzej Siewior wrote:
> 
> > This -RT series didn't crashed within ~4h testing on my ARM and
> > x86-32.
> > x86-64 crashed after I started hackbench. I figured out that the crash
> > does not happen with lazy-preempt disabled. Therefore the last but one
> > patch in the queue disables lazy preempt on x86-64. With this change the
> > test box survived ~2h without a crash. I look at this later but it looks
> > good now.
> 
> I think the below fixes it (in a more or less minimalist way), but it's
> not very pretty.  Methinks it would be prettier to either clone the x86
> percpu + fold logic, or neutralize that optimization completely when
> PREEMPT_LAZY is enabled.
> 
> x86_32 bit is completely untested, x86_64 hasn't exploded.. yet :) 
> 

This patch makes sense to me.

Acked-by: Steven Rostedt <rostedt@goodmis.org>

-- Steve

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

Turning lockdep on, it says it's busted.

(I'll go stare at it, maybe the beast will blink first for a change)

[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 6367 kB
[    0.000000]  per task-struct memory footprint: 2688 bytes
[    0.000000] ------------------------
[    0.000000] | Locking API testsuite:
[    0.000000] ----------------------------------------------------------------------------
[    0.000000]                                  | spin |wlock |rlock |mutex | wsem | rsem |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]                      A-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e710f>] locking_selftest+0xdf/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]                  A-B-B-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e719e>] locking_selftest+0x16e/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]              A-B-B-C-C-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e722d>] locking_selftest+0x1fd/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]              A-B-C-A-B-C deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e72bc>] locking_selftest+0x28c/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-B-C-C-D-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e734b>] locking_selftest+0x31b/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-D-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e73da>] locking_selftest+0x3aa/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-C-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e7469>] locking_selftest+0x439/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]                     double unlock:  ok  |
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:2660 migrate_disable+0xbd/0xd0()
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000a64 ffffffff81a01d38 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000000 ffffffff81a01d78 ffffffff8104f0cc ffffffff81a01d78
[    0.000000]  ffffffff81a194c0 0000000000000027 0000000000000006 0000000000000001
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff8104f0cc>] warn_slowpath_common+0x8c/0xc0
[    0.000000]  [<ffffffff8104f11a>] warn_slowpath_null+0x1a/0x20
[    0.000000]  [<ffffffff8108795d>] migrate_disable+0xbd/0xd0
[    0.000000]  [<ffffffff810b7a7f>] call_console_drivers.constprop.20+0x4f/0x140
[    0.000000]  [<ffffffff810b823f>] console_unlock.part.15+0x1df/0x2e0
[    0.000000]  [<ffffffff815ec835>] ? _raw_spin_unlock+0x35/0x60
[    0.000000]  [<ffffffff810b8358>] console_unlock+0x18/0x30
[    0.000000]  [<ffffffff810b8771>] vprintk_emit+0x231/0x400
[    0.000000]  [<ffffffff815e9aca>] ? preempt_schedule+0x4a/0x70
[    0.000000]  [<ffffffff812e1320>] ? bad_unlock_order_spin+0x40/0x40
[    0.000000]  [<ffffffff815d8519>] printk+0x4d/0x4f
[    0.000000]  [<ffffffff815f11d9>] ? preempt_count_sub+0x29/0x70
[    0.000000]  [<ffffffff812e1348>] ? double_unlock_spin+0x28/0x30
[    0.000000]  [<ffffffff815e1dc8>] dotest+0x75/0xc7
[    0.000000]  [<ffffffff812e74cf>] locking_selftest+0x49f/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] ---[ end trace 0000000000000001 ]---
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:2693 migrate_enable+0xf6/0x1a0()
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000a85 ffffffff81a01cb8 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000000 ffffffff81a01cf8 ffffffff8104f0cc ffffffff81a01d18
[    0.000000]  ffffffff81a194c0 0000000000000000 000000000000000a ffffffff8275f367
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff8104f0cc>] warn_slowpath_common+0x8c/0xc0
[    0.000000]  [<ffffffff8104f11a>] warn_slowpath_null+0x1a/0x20
[    0.000000]  [<ffffffff810877f6>] migrate_enable+0xf6/0x1a0
[    0.000000]  [<ffffffff813aab31>] vt_console_print+0x2f1/0x3d0
[    0.000000]  [<ffffffff810b7ae7>] call_console_drivers.constprop.20+0xb7/0x140
[    0.000000]  [<ffffffff810b823f>] console_unlock.part.15+0x1df/0x2e0
[    0.000000]  [<ffffffff815ec835>] ? _raw_spin_unlock+0x35/0x60
[    0.000000]  [<ffffffff810b8358>] console_unlock+0x18/0x30
[    0.000000]  [<ffffffff810b8771>] vprintk_emit+0x231/0x400
[    0.000000]  [<ffffffff815e9aca>] ? preempt_schedule+0x4a/0x70
[    0.000000]  [<ffffffff812e1320>] ? bad_unlock_order_spin+0x40/0x40
[    0.000000]  [<ffffffff815d8519>] printk+0x4d/0x4f
[    0.000000]  [<ffffffff815f11d9>] ? preempt_count_sub+0x29/0x70
[    0.000000]  [<ffffffff812e1348>] ? double_unlock_spin+0x28/0x30
[    0.000000]  [<ffffffff815e1dc8>] dotest+0x75/0xc7
[    0.000000]  [<ffffffff812e74cf>] locking_selftest+0x49f/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] ---[ end trace 0000000000000002 ]---
[    0.000000]   ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 0000000000000046
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e74f5>] locking_selftest+0x4c5/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000008 ffffffff81a01f28 ffffffff815e12a5 0000000000000046
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e752e>] locking_selftest+0x4fe/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] 
[    0.000000]                   initialize held:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]                  bad unlock order:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]               recursive read-lock:             |  ok  |             |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000008 ffffffff81a01f28 ffffffff815e12a5 0000000000000006
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e76c5>] locking_selftest+0x695/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] 
[    0.000000]            recursive read-lock #2:             |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e7703>] locking_selftest+0x6d3/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]              |  ok  |
[    0.000000]             mixed read-write-lock:             |  ok  |             |  ok  |
[    0.000000]             mixed write-read-lock:             |  ok  |             |  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]      hard-irqs-on + irq-safe-A/12:  ok  |
[    0.000000]      hard-irqs-on + irq-safe-A/21:  ok  |
[    0.000000]          hard-safe-A + irqs-on/12:  ok  |
[    0.000000]          hard-safe-A + irqs-on/21:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/123:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/132:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/213:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/231:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/312:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/321:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/123:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/132:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/213:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/231:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/312:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/321:  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]   | Wound/wait tests |
[    0.000000]   ---------------------
[    0.000000]                   ww api failures:  ok  |  ok  |  ok  |
[    0.000000]                ww contexts mixing:  ok  |  ok  |
[    0.000000]              finishing ww context:  ok  |  ok  |  ok  |  ok  |
[    0.000000]                locking mismatches:  ok  |  ok  |  ok  |
[    0.000000]                  EDEADLK handling:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]            spinlock nest unlocked:  ok  |
[    0.000000]   -----------------------------------------------------
[    0.000000]                                  |block | try  |context|
[    0.000000]   -----------------------------------------------------
[    0.000000]                           context:  ok  |  ok  |  ok  |
[    0.000000]                               try:  ok  |  ok  |  ok  |
[    0.000000]                             block:  ok  |  ok  |  ok  |
[    0.000000]                          spinlock:  ok  |  ok  |  ok  |
[    0.000000] -----------------------------------------------------------------
[    0.000000] BUG:  11 unexpected failures (out of 119) - debugging disabled! |
[    0.000000] -----------------------------------------------------------------

Re: [ANNOUNCE] 3.14-rt1

[Sebastian Andrzej Siewior]

On 04/24/2014 06:06 AM, Mike Galbraith wrote:
> Turning lockdep on, it says it's busted.

http://www.spinics.net/lists/linux-rt-users/msg11179.html

Sebastian

Re: [ANNOUNCE] 3.14-rt1

[Mike Galbraith]

On Thu, 2014-04-24 at 09:12 +0200, Sebastian Andrzej Siewior wrote: 
> On 04/24/2014 06:06 AM, Mike Galbraith wrote:
> > Turning lockdep on, it says it's busted.
> 
> http://www.spinics.net/lists/linux-rt-users/msg11179.html

I was heading toward the same conclusion while regression testing.
Guess I can stop that.

-Mike

Re: [ANNOUNCE] 3.14-rt1

[Fernando Lopez-Lezcano]

On 04/11/2014 11:57 AM, Sebastian Andrzej Siewior wrote:
> Dear RT folks!
>
> I'm pleased to announce the v3.14-rt1 patch setty).
>
> Changes since v3.12.15-rt25
> - I dropped the sparc64 patches I had in the queue. They did not apply
>    cleanly, the code in v3.14 changed in the MMU area. Here is where I
>    remembered that it was not working perfectly either.

Saw this a moment ago (3.14.1 + rt1, Fedora 19 laptop - I think I have 
seen something similar in 3.12.x-r):

Apr 26 11:16:11 localhost kernel: [   96.323248] ------------[ cut here 
]------------
Apr 26 11:16:11 localhost kernel: [   96.323262] WARNING: CPU: 0 PID: 
2051 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
Apr 26 11:16:11 localhost kernel: [   96.323264] list_del corruption. 
prev->next should be ffff8802101196a0, but was 0000000000000001
Apr 26 11:16:11 localhost kernel: [   96.323266] Modules linked in: fuse 
ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack 
ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables 
ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 
ip6table_mangle ip6table_security ip6table_raw rfcomm ip6table_filter 
bnep ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 
nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iTCO_wdt 
iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul 
crc32c_intel ghash_clmulni_intel uvcvideo videobuf2_vmalloc microcode 
videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev media 
serio_raw btusb bluetooth intel_ips i2c_i801 6lowpan_iphc 
snd_hda_codec_conexant snd_hda_codec_generic arc4 iwldvm mac80211 
iwlwifi lpc_ich sdhci_pci mfd_core sdhci cfg80211 mmc_core snd_hda_intel 
snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e snd_timer 
ptp mei_me pps_core mei shpchp thinkpad_acpi snd ppdev soundcore rfkill 
parport_pc parport acpi_cpufreq uinput firewire_ohci nouveau 
firewire_core crc_itu_t i2c_algo_bit drm_kms_helper ttm drm mxm_wmi 
i2c_core wmi video
Apr 26 11:16:11 localhost kernel: [   96.323331] CPU: 0 PID: 2051 Comm: 
cinnamon Not tainted 3.14.1-200.rt1.1.fc19.ccrma.x86_64+rt #1
Apr 26 11:16:11 localhost kernel: [   96.323332] Hardware name: LENOVO 
4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
Apr 26 11:16:11 localhost kernel: [   96.323334]  0000000000000000 
000000008a5c11dc ffff8800ae715a88 ffffffff81707fca
Apr 26 11:16:11 localhost kernel: [   96.323336]  ffff8800ae715ad0 
ffff8800ae715ac0 ffffffff8108d03d ffff8802101196a0
Apr 26 11:16:11 localhost kernel: [   96.323337]  ffff880210119b50 
ffff880210119b50 ffff880210119b40 ffff88021a615648
Apr 26 11:16:11 localhost kernel: [   96.323338] Call Trace:
Apr 26 11:16:11 localhost kernel: [   96.323345]  [<ffffffff81707fca>] 
dump_stack+0x4d/0x82
Apr 26 11:16:11 localhost kernel: [   96.323351]  [<ffffffff8108d03d>] 
warn_slowpath_common+0x7d/0xc0
Apr 26 11:16:11 localhost kernel: [   96.323352]  [<ffffffff8108d0dc>] 
warn_slowpath_fmt+0x5c/0x80
Apr 26 11:16:11 localhost kernel: [   96.323354]  [<ffffffff8137c551>] 
__list_del_entry+0xa1/0xd0
Apr 26 11:16:11 localhost kernel: [   96.323355]  [<ffffffff8137c58d>] 
list_del+0xd/0x30
Apr 26 11:16:11 localhost kernel: [   96.323393]  [<ffffffffa0135593>] 
nouveau_fence_signal+0x53/0x80 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323414]  [<ffffffffa0135678>] 
nouveau_fence_update+0x48/0xa0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323435]  [<ffffffffa0135f85>] 
nouveau_fence_sync+0x45/0x80 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323456]  [<ffffffffa013aea8>] 
validate_list+0xd8/0x2e0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323478]  [<ffffffffa013c3d3>] 
nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323500]  [<ffffffffa002ad02>] 
drm_ioctl+0x4f2/0x620 [drm]
Apr 26 11:16:11 localhost kernel: [   96.323506]  [<ffffffff810c1af4>] ? 
migrate_enable+0x94/0x1c0
Apr 26 11:16:11 localhost kernel: [   96.323527]  [<ffffffffa0132cfe>] 
nouveau_drm_ioctl+0x4e/0x90 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323530]  [<ffffffff81203480>] 
do_vfs_ioctl+0x2e0/0x4c0
Apr 26 11:16:11 localhost kernel: [   96.323533]  [<ffffffff812fd8d6>] ? 
file_has_perm+0xa6/0xb0
Apr 26 11:16:11 localhost kernel: [   96.323535]  [<ffffffff812036e1>] 
SyS_ioctl+0x81/0xa0
Apr 26 11:16:11 localhost kernel: [   96.323538]  [<ffffffff81716769>] 
system_call_fastpath+0x16/0x1b
Apr 26 11:16:11 localhost kernel: [   96.323569] ---[ end trace 
0000000000000002 ]---

-- Fernando

Re: [ANNOUNCE] 3.14-rt1

[Sebastian Andrzej Siewior]

* Fernando Lopez-Lezcano | 2014-04-26 11:29:04 [-0700]:

>Saw this a moment ago (3.14.1 + rt1, Fedora 19 laptop - I think I
>have seen something similar in 3.12.x-r):

Yes, you did: https://lkml.org/lkml/2014/3/7/163
You did not test I've sent. Care to do so?

>Apr 26 11:16:11 localhost kernel: [   96.323248] ------------[ cut
>here ]------------
>Apr 26 11:16:11 localhost kernel: [   96.323262] WARNING: CPU: 0 PID:
>2051 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
>Apr 26 11:16:11 localhost kernel: [   96.323264] list_del corruption.
>prev->next should be ffff8802101196a0, but was 0000000000000001
>Apr 26 11:16:11 localhost kernel: [   96.323266] Modules linked in:

and please send backtrace information properly formatted. This is
terrible hard to read.

Sebastian

Re: [ANNOUNCE] 3.14-rt1

[Fernando Lopez-Lezcano]

On 05/02/2014 04:37 AM, Sebastian Andrzej Siewior wrote:
> * Fernando Lopez-Lezcano | 2014-04-26 11:29:04 [-0700]:
>
>> Saw this a moment ago (3.14.1 + rt1, Fedora 19 laptop - I think I
>> have seen something similar in 3.12.x-r):
>
> Yes, you did: https://lkml.org/lkml/2014/3/7/163
> You did not test I've sent. Care to do so?

I did patch my kernel and (I think) I did not see the problem again. I 
did get some very occassional hangs that seemed to be video related but 
I think I could not see what had caused them.

>> Apr 26 11:16:11 localhost kernel: [   96.323248] ------------[ cut
>> here ]------------
>> Apr 26 11:16:11 localhost kernel: [   96.323262] WARNING: CPU: 0 PID:
>> 2051 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
>> Apr 26 11:16:11 localhost kernel: [   96.323264] list_del corruption.
>> prev->next should be ffff8802101196a0, but was 0000000000000001
>> Apr 26 11:16:11 localhost kernel: [   96.323266] Modules linked in:
>
> and please send backtrace information properly formatted. This is
> terrible hard to read.

Sorry about that, I will attach files in the future.

I re-patched 3.14.3-rt5 with a slightly tweaked version of you patch. 
Will see what happens and report back.
-- Fernando

Re: [ANNOUNCE] 3.14-rt1

[Fernando Lopez-Lezcano]

[-- Attachment #1: Type: text/plain, Size: 658 bytes --]

On 05/15/2014 10:51 AM, Fernando Lopez-Lezcano wrote:
> On 05/02/2014 04:37 AM, Sebastian Andrzej Siewior wrote:
>> * Fernando Lopez-Lezcano | 2014-04-26 11:29:04 [-0700]:
>>
>>> Saw this a moment ago (3.14.1 + rt1, Fedora 19 laptop - I think I
>>> have seen something similar in 3.12.x-r):
>>
>> Yes, you did: https://lkml.org/lkml/2014/3/7/163
>> You did not test I've sent. Care to do so?
...
> I re-patched 3.14.3-rt5 with a slightly tweaked version of you patch.
> Will see what happens and report back.

I got another freeze a moment ago. I'm attaching a text file with the 
oops, the patch I added to rt5, and the kernel configuration...

-- Fernando

[-- Attachment #2: oops.txt --]
[-- Type: text/plain, Size: 13057 bytes --]

May 15 12:51:27 localhost kernel: [ 9282.171060] ------------[ cut here ]------------
May 15 12:51:27 localhost kernel: [ 9282.171079] WARNING: CPU: 0 PID: 2049 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
May 15 12:51:27 localhost kernel: [ 9282.171082] list_del corruption. prev->next should be ffff880211174c60, but was dead000000100100
May 15 12:51:27 localhost kernel: [ 9282.171083] Modules linked in: fuse ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat rfcomm nf_conntrack iptable_mangle bnep iptable_security iptable_raw iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode uvcvideo serio_raw videobuf2_vmalloc videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev intel_ips i2c_i801 media btusb bluetooth 6lowpan_iphc arc4 iwldvm snd_hda_codec_conexant snd_hda_codec_generic mac80211 iwlwifi sdhci_pci sdhci cfg80211 mmc_core lpc_ich mfd_core snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer e1000e mei_me ptp mei thinkpad_acpi pps_core shpchpnouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - ppdev snd soundcore rfkill parport_pc parport acpi_cpufreq uinput nouveau i2c_algo_bit drm_kms_helper firewire_ohci mxm_wmi firewire_core ttm
May 15 12:51:27 localhost kernel: [ 9282.171136]  DST2D_FAULT
May 15 12:51:27 localhost kernel: [ 9282.171139]  crc_itu_t drm i2c_core
May 15 12:51:27 localhost kernel: [ 9282.171139]  - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.171141]  wmi video
May 15 12:51:27 localhost kernel: [ 9282.171144] CPU: 0 PID: 2049 Comm: cinnamon Tainted: G        W    3.14.3-200.rt5.1.fc19.ccrma.x86_64+rt #1
May 15 12:51:27 localhost kernel: [ 9282.171145] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00000011, e24: 0c030000
May 15 12:51:27 localhost kernel: [ 9282.171146] Hardware name: LENOVO 4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
May 15 12:51:27 localhost kernel: [ 9282.171149]  0000000000000000 000000006a3e8229 ffff8800ada91a88 ffffffff81703b57
May 15 12:51:27 localhost kernel: [ 9282.171151]  ffff8800ada91ad0 ffff8800ada91ac0
May 15 12:51:27 localhost kernel: [ 9282.171151] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 2 class 0x502d mthd 0x060c data 0x00000022
May 15 12:51:27 localhost kernel: [ 9282.171153]  ffffffff8108bc2d ffff880211174c60
May 15 12:51:27 localhost kernel: [ 9282.171156]  ffff880211174d50 ffff880211174d50 ffff880211174d40 ffff88021b73be48
May 15 12:51:27 localhost kernel: [ 9282.171156] Call Trace:
May 15 12:51:27 localhost kernel: [ 9282.171164]  [<ffffffff81703b57>] dump_stack+0x4d/0x6f
May 15 12:51:27 localhost kernel: [ 9282.171170]  [<ffffffff8108bc2d>] warn_slowpath_common+0x7d/0xc0
May 15 12:51:27 localhost kernel: [ 9282.171171] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] 
May 15 12:51:27 localhost kernel: [ 9282.171173]  [<ffffffff8108bccc>] warn_slowpath_fmt+0x5c/0x80
May 15 12:51:27 localhost kernel: [ 9282.171176]  [<ffffffff81376581>] __list_del_entry+0xa1/0xd0
May 15 12:51:27 localhost kernel: [ 9282.171178]  [<ffffffff813765bd>] list_del+0xd/0x30
May 15 12:51:27 localhost kernel: [ 9282.171181] PGRAPH/
May 15 12:51:27 localhost kernel: [ 9282.171181]  [<ffffffffa0145593>] nouveau_fence_signal+0x53/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171219] PROP/ [<ffffffffa0145678>] nouveau_fence_update+0x48/0xa0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171260]  [<ffffffffa0145fb5>] nouveau_fence_sync+0x45/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171282] DST2D
May 15 12:51:27 localhost kernel: [ 9282.171282]  [<ffffffffa014aed8>] validate_list+0xd8/0x2e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171304]  [<ffffffffa014c403>] nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171305]  reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.171318]  [<ffffffffa002ad02>] drm_ioctl+0x4f2/0x620 [drm]
May 15 12:51:27 localhost kernel: [ 9282.171323]  [<ffffffff810bcf44>] ? migrate_enable+0x94/0x1a0
May 15 12:51:27 localhost kernel: [ 9282.171344]  [<ffffffffa0142cfe>] nouveau_drm_ioctl+0x4e/0x90 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171347]  [<ffffffff811feb50>] do_vfs_ioctl+0x2e0/0x4c0
May 15 12:51:27 localhost kernel: [ 9282.171349]  [<ffffffff812f7da6>] ? file_has_perm+0xa6/0xb0
May 15 12:51:27 localhost kernel: [ 9282.171351]  [<ffffffff810bb701>] ? __sched_fork+0x171/0x220
May 15 12:51:27 localhost kernel: [ 9282.171352]  [<ffffffff811fedb1>] SyS_ioctl+0x81/0xa0
May 15 12:51:27 localhost kernel: [ 9282.171355]  [<ffffffff81712069>] system_call_fastpath+0x16/0x1b
May 15 12:51:27 localhost kernel: [ 9282.171356] ---[ end trace 0000000000000006 ]---
May 15 12:51:27 localhost kernel: [ 9282.172554] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - RT_FAULT - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.172561] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00002a00, e24: 00030000
May 15 12:51:27 localhost kernel: [ 9282.172566] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 7 class 0x8597 mthd 0x15e0 data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172575] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/PROP/RT0 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.172616] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - RT_FAULT - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.172621] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00002a00, e24: 00030000
May 15 12:51:27 localhost kernel: [ 9282.172625] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 7 class 0x8597 mthd 0x15e0 data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172635] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/PROP/RT0 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.172656] nouveau E[  PGRAPH][0000:01:00.0] magic set 0:
May 15 12:51:27 localhost kernel: [ 9282.172659] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408604: 0x20095203
May 15 12:51:27 localhost kernel: [ 9282.172662] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408608: 0x0020a958
May 15 12:51:27 localhost kernel: [ 9282.172665] nouveau E[  PGRAPH][0000:01:00.0] 	0x0040860c: 0x80000432
May 15 12:51:27 localhost kernel: [ 9282.172668] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408610: 0xa9200000
May 15 12:51:27 localhost kernel: [ 9282.172671] nouveau E[  PGRAPH][0000:01:00.0] TRAP_TEXTURE - TP0: Unhandled ustatus 0x00000003
May 15 12:51:27 localhost kernel: [ 9282.172674] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 2 class 0x502d mthd 0x08dc data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172685] nouveau E[     PFB][0000:01:00.0] trapped read at 0x0020a92200 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/TEXTURE/00 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.240139] BUG: unable to handle kernel NULL pointer dereference at 0000000000000009
May 15 12:51:27 localhost kernel: [ 9282.240149] IP: [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240152] PGD 227e4e067 PUD 227e4d067 PMD 0 
May 15 12:51:27 localhost kernel: [ 9282.240154] Oops: 0000 [#1] PREEMPT SMP 
May 15 12:51:27 localhost kernel: [ 9282.240188] Modules linked in: fuse ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat rfcomm nf_conntrack iptable_mangle bnep iptable_security iptable_raw iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode uvcvideo serio_raw videobuf2_vmalloc videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev intel_ips i2c_i801 media btusb bluetooth 6lowpan_iphc arc4 iwldvm snd_hda_codec_conexant snd_hda_codec_generic mac80211 iwlwifi sdhci_pci sdhci cfg80211 mmc_core lpc_ich mfd_core snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer e1000e mei_me ptp mei thinkpad_acpi pps_core shpchp ppdev snd soundcore rfkill parport_pc parport acpi_cpufreq uinput nouveau i2c_algo_bit drm_kms_helper firewire_ohci mxm_wmi firewire_core ttm crc_itu_t drm i2c_core wmi video
May 15 12:51:27 localhost kernel: [ 9282.240206] CPU: 0 PID: 2049 Comm: cinnamon Tainted: G        W    3.14.3-200.rt5.1.fc19.ccrma.x86_64+rt #1
May 15 12:51:27 localhost kernel: [ 9282.240207] Hardware name: LENOVO 4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
May 15 12:51:27 localhost kernel: [ 9282.240208] task: ffff880227cfc750 ti: ffff8800ada90000 task.ti: ffff8800ada90000
May 15 12:51:27 localhost kernel: [ 9282.240211] RIP: 0010:[<ffffffff81376511>]  [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240212] RSP: 0018:ffff8800ada91b38  EFLAGS: 00010246
May 15 12:51:27 localhost kernel: [ 9282.240213] RAX: ffff8800a79fcf50 RBX: ffff8800a79fc3a0 RCX: dead000000200200
May 15 12:51:27 localhost kernel: [ 9282.240214] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff8800a79fc3a0
May 15 12:51:27 localhost kernel: [ 9282.240214] RBP: ffff8800ada91b38 R08: ffff8800a79fc3a0 R09: 0000000000000000
May 15 12:51:27 localhost kernel: [ 9282.240215] R10: ffffea00029e2d40 R11: ffffffffa01456c8 R12: ffff8800a79fc820
May 15 12:51:27 localhost kernel: [ 9282.240216] R13: ffff8800a79fcf50 R14: ffff8800a79fcf40 R15: ffff88021b73be48
May 15 12:51:27 localhost kernel: [ 9282.240217] FS:  00007fc079062a40(0000) GS:ffff88023bc00000(0000) knlGS:0000000000000000
May 15 12:51:27 localhost kernel: [ 9282.240218] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 15 12:51:27 localhost kernel: [ 9282.240219] CR2: 0000000000000009 CR3: 000000022834f000 CR4: 00000000000007f0
May 15 12:51:27 localhost kernel: [ 9282.240220] Stack:
May 15 12:51:27 localhost kernel: [ 9282.240222]  ffff8800ada91b50 ffffffff813765bd ffff8800a79fc800 ffff8800ada91b80
May 15 12:51:27 localhost kernel: [ 9282.240223]  ffffffffa0145593 ffff8800a79fcf40 ffff8800a79fc340 ffff88021b73be00
May 15 12:51:27 localhost kernel: [ 9282.240225]  ffff88021742ab40 ffff8800ada91bb8 ffffffffa0145678 ffff8800a79fc340
May 15 12:51:27 localhost kernel: [ 9282.240225] Call Trace:
May 15 12:51:27 localhost kernel: [ 9282.240229]  [<ffffffff813765bd>] list_del+0xd/0x30
May 15 12:51:27 localhost kernel: [ 9282.240268]  [<ffffffffa0145593>] nouveau_fence_signal+0x53/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240295]  [<ffffffffa0145678>] nouveau_fence_update+0x48/0xa0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240321]  [<ffffffffa0145fb5>] nouveau_fence_sync+0x45/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240347]  [<ffffffffa014aed8>] validate_list+0xd8/0x2e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240374]  [<ffffffffa014c403>] nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240386]  [<ffffffffa002ad02>] drm_ioctl+0x4f2/0x620 [drm]
May 15 12:51:27 localhost kernel: [ 9282.240393]  [<ffffffff810bcf44>] ? migrate_enable+0x94/0x1a0
May 15 12:51:27 localhost kernel: [ 9282.240418]  [<ffffffffa0142cfe>] nouveau_drm_ioctl+0x4e/0x90 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240421]  [<ffffffff811feb50>] do_vfs_ioctl+0x2e0/0x4c0
May 15 12:51:27 localhost kernel: [ 9282.240424]  [<ffffffff812f7da6>] ? file_has_perm+0xa6/0xb0
May 15 12:51:27 localhost kernel: [ 9282.240426]  [<ffffffff811fedb1>] SyS_ioctl+0x81/0xa0
May 15 12:51:27 localhost kernel: [ 9282.240429]  [<ffffffff81129566>] ? __audit_syscall_exit+0x1f6/0x2a0
May 15 12:51:27 localhost kernel: [ 9282.240432]  [<ffffffff81712069>] system_call_fastpath+0x16/0x1b
May 15 12:51:27 localhost kernel: [ 9282.240451] Code: 00 01 10 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 20 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 <4c> 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10 5d c3 49 89 d0 
May 15 12:51:27 localhost kernel: [ 9282.240453] RIP  [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240453]  RSP <ffff8800ada91b38>
May 15 12:51:27 localhost kernel: [ 9282.240454] CR2: 0000000000000009
May 15 12:51:27 localhost kernel: [ 9282.247071] ---[ end trace 0000000000000007 ]---

[-- Attachment #3: kernel-nouveau-channel.patch --]
[-- Type: text/x-patch, Size: 695 bytes --]

--- a/drivers/gpu/drm/nouveau/nouveau_fence.c~	2014-03-30 20:40:15.000000000 -0700
+++ b/drivers/gpu/drm/nouveau/nouveau_fence.c	2014-05-14 13:39:02.115359923 -0700
@@ -178,11 +178,17 @@
 
 {
 	struct nouveau_channel *chan = fence->channel;
-	struct nouveau_fifo *pfifo = nouveau_fifo(chan->drm->device);
-	struct nouveau_fence_priv *priv = chan->drm->fence;
+	struct nouveau_fifo *pfifo;
+	struct nouveau_fence_priv *priv;
 	struct nouveau_eventh *handler;
 	int ret = 0;
 
+	if (WARN_ON_ONCE(!chan))
+		return 0;
+
+	pfifo = nouveau_fifo(chan->drm->device);
+	priv = chan->drm->fence;
+
 	ret = nouveau_event_new(pfifo->uevent, 0,
 				nouveau_fence_wait_uevent_handler,
 				priv, &handler);

[-- Attachment #4: config-3.14.3-200.rt5.1.fc19.ccrma.x86_64+rt.bz2 --]
[-- Type: application/x-bzip, Size: 32591 bytes --]