[ANNOUNCE] v4.14.29-rt25

[ANNOUNCE] v4.14.29-rt25

[Sebastian Andrzej Siewior]

Dear RT folks!

I'm pleased to announce the v4.14.29-rt25 patch set. 

Changes since v4.14.29-rt24:

  - There are checks for disabled interrupts in the target code which do
    not work on RT. Reported by Arnaldo Carvalho de Melo.

Known issues
     - A warning triggered in "rcu_note_context_switch" originated from
       SyS_timer_gettime(). The issue was always there, it is now
       visible. Reported by Grygorii Strashko and Daniel Wagner.

The delta patch against v4.14.29-rt24 is appended below and can be found here:
 
     https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz

You can get this release via the git tree at:

    git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25

The RT patch against v4.14.29 can be found here:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz

The split quilt queue is available at:

    https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz

Sebastian

diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 9c7bc1ca341a..3d35dad1de2c 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -114,8 +114,6 @@ static bool __target_check_io_state(struct se_cmd *se_cmd,
 {
 	struct se_session *sess = se_cmd->se_sess;
 
-	assert_spin_locked(&sess->sess_cmd_lock);
-	WARN_ON_ONCE(!irqs_disabled());
 	/*
 	 * If command already reached CMD_T_COMPLETE state within
 	 * target_complete_cmd() or CMD_T_FABRIC_STOP due to shutdown,
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index e6d51135d105..b00829995131 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -2966,9 +2966,6 @@ __transport_wait_for_tasks(struct se_cmd *cmd, bool fabric_stop,
 	__acquires(&cmd->t_state_lock)
 {
 
-	assert_spin_locked(&cmd->t_state_lock);
-	WARN_ON_ONCE(!irqs_disabled());
-
 	if (fabric_stop)
 		cmd->transport_state |= CMD_T_FABRIC_STOP;
 
@@ -3238,9 +3235,6 @@ static int __transport_check_aborted_status(struct se_cmd *cmd, int send_status)
 {
 	int ret;
 
-	assert_spin_locked(&cmd->t_state_lock);
-	WARN_ON_ONCE(!irqs_disabled());
-
 	if (!(cmd->transport_state & CMD_T_ABORTED))
 		return 0;
 	/*
diff --git a/localversion-rt b/localversion-rt
index b2111a212663..c5b71f9a229d 100644
--- a/localversion-rt
+++ b/localversion-rt
@@ -1 +1 @@
--rt24
+-rt25

Re: [ANNOUNCE] v4.14.29-rt25

[Grygorii Strashko]

Hi Sebastian,

On 03/25/2018 05:12 AM, Sebastian Andrzej Siewior wrote:
> Dear RT folks!
> 
> I'm pleased to announce the v4.14.29-rt25 patch set.
> 
> Changes since v4.14.29-rt24:
> 
>    - There are checks for disabled interrupts in the target code which do
>      not work on RT. Reported by Arnaldo Carvalho de Melo.
> 
> Known issues
>       - A warning triggered in "rcu_note_context_switch" originated from
>         SyS_timer_gettime(). The issue was always there, it is now
>         visible. Reported by Grygorii Strashko and Daniel Wagner.
> 
> The delta patch against v4.14.29-rt24 is appended below and can be found here:
>   
>       https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz
> 
> You can get this release via the git tree at:
> 
>      git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25
> 
> The RT patch against v4.14.29 can be found here:
> 
>      https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz
> 
> The split quilt queue is available at:
> 
>      https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz
> 
> Sebastian
> 

I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
while holding the RCU lock" [1] on top. Then I run below tests:

##  stress-ng --class os --all 0 -t 5m
- and -
## stress-ng --class cpu --all 0 -t 5m &   stress-ng --class memory --all 0 --vm-bytes 90% -t 5m

no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings

[1] https://patchwork.kernel.org/patch/10305015/

==== WARN1 ====
[  104.967174] ------------[ cut here ]------------
[  104.967191] WARNING: CPU: 0 PID: 1676 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4
stress-ng: error: [913] stress-ng-fifo: read timeout!
stress-ng: error: [924] stress-ng-fifo: read timeout!
[  104.967194] Modules linked in:
[  104.967206] CPU: 0 PID: 1676 Comm: stress-ng-fstat Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269
stress-ng: error: [926] stress-ng-fifo: read timeout!
stress-ng: error: [920] stress-ng-fifo: read timeout!
[  104.967210] Hardware name: Generic DRA74X (Flattened Device Tree)
[  104.967227] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14)
[  104.967239] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4)
[  104.967251] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100)
[  104.967261] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28)
[  104.967271] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4)
stress-ng: fail:  [828] stress-ng-af-alg: socket failed, errno=9[  104.967285] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814)
7 (Address family not supported by protocol)
[  104.967294] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0)
[  104.967302] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4)
stress-ng: fail:  [1484] stress-ng-quota: quotactl command Q_GETQUOTA failed: errno=38 (Function not implemented)
stress-ng: fail:  [1484] stress-ng-quota: quotactl command Q_GETFMT failed: errno=38 (Function not implemented)
stress-ng: fail:  [1484] stress-ng-quota: quotactl command Q_GETINFO failed: errno=38 (Function not implemented)
stress-ng: fail:  [1484] stress-ng-quota: quotactl command Q_SYNC failed: errno=38 (Function not implemented)
stress-ng: error: [1484] stress-ng-quota: quotactl() failed, not available on this kernel
[  104.967311] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0)
[  104.967320] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c)
[  104.967329] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4)
[  104.967339] [<c015a5d8>] (migrate_disable) from [<c0842338>] (rt_spin_lock+0x18/0x60)
[  104.967349] [<c0842338>] (rt_spin_lock) from [<c0440994>] (lockref_get_not_dead+0xc/0x34)
[  104.967360] [<c0440994>] (lockref_get_not_dead) from [<c0273fb8>] (legitimize_path+0x2c/0x64)
[  104.967368] [<c0273fb8>] (legitimize_path) from [<c02740e4>] (unlazy_walk+0x3c/0x18c)
[  104.967376] [<c02740e4>] (unlazy_walk) from [<c0274258>] (complete_walk+0x24/0x78)
[  104.967384] [<c0274258>] (complete_walk) from [<c02768b8>] (path_lookupat+0xbc/0x1ec)
[  104.967393] [<c02768b8>] (path_lookupat) from [<c0278f98>] (filename_lookup+0x8c/0xe8)
[  104.967404] [<c0278f98>] (filename_lookup) from [<c026e1ac>] (vfs_statx+0x5c/0xb8)
[  104.967414] [<c026e1ac>] (vfs_statx) from [<c026e8ec>] (SyS_stat64+0x28/0x44)
[  104.967424] [<c026e8ec>] (SyS_stat64) from [<c0108480>] (ret_fast_syscall+0x0/0x28)
[  104.967429] ---[ end trace 0000000000000002 ]---


=== WARN2 ====
[  196.437276] ------------[ cut here ]------------
[  196.437293] WARNING: CPU: 0 PID: 1521 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4
[  196.437296] Modules linked in:
[  196.437307] CPU: 0 PID: 1521 Comm: stress-ng-opcod Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269
[  196.437311] Hardware name: Generic DRA74X (Flattened Device Tree)
[  196.437326] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14)
[  196.437338] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4)
[  196.437349] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100)
[  196.437359] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28)
[  196.437369] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4)
[  196.437381] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814)
[  196.437390] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0)
[  196.437398] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4)
[  196.437407] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0)
[  196.437416] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c)
[  196.437424] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4)
[  196.437433] [<c015a5d8>] (migrate_disable) from [<c011a3a4>] (kmap_atomic+0x34/0x174)
[  196.437443] [<c011a3a4>] (kmap_atomic) from [<c0236484>] (alloc_set_pte+0x288/0x320)
[  196.437453] [<c0236484>] (alloc_set_pte) from [<c01ffcb8>] (filemap_map_pages+0x2b0/0x4c8)
[  196.437461] [<c01ffcb8>] (filemap_map_pages) from [<c0236c4c>] (handle_mm_fault+0x690/0xc78)
[  196.437468] [<c0236c4c>] (handle_mm_fault) from [<c0115bd8>] (do_page_fault+0x2bc/0x398)
[  196.437476] [<c0115bd8>] (do_page_fault) from [<c0101474>] (do_PrefetchAbort+0x38/0xa0)
[  196.437484] [<c0101474>] (do_PrefetchAbort) from [<c010decc>] (ret_from_exception+0x0/0x14)
[  196.437488] Exception stack(0xe9943fb0 to 0xe9943ff8)
[  196.437494] 3fa0:                                     00000000 00000000 00000000 00000000
[  196.437502] 3fc0: b6f6e468 bee42888 b6f6e400 bee42898 b6f6e8c0 00000000 000003af 000003af
[  196.437508] 3fe0: 00000078 bee42888 b6d9ac91 b6d9ac90 200b0030 ffffffff
[  196.437513] ---[ end trace 0000000000000002 ]---


-- 
regards,
-grygorii

Re: [ANNOUNCE] v4.14.29-rt25

[Sebastian Andrzej Siewior]

On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote:
> Hi Sebastian,
Hi Grygorii,

> I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
> while holding the RCU lock" [1] on top. Then I run below tests:
…
> no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings

the warning is unrelated to the patch I posted. This should mute it:

Subject: [PATCH RT] rtmutex: annotate sleeping lock context

The RCU code complains on schedule() within a rcu_readlock() section.
The valid scenario on -RT is if a sleeping is held. In order to suppress
the warning the mirgrate_disable counter was used to identify the
invocation of schedule() due to lock contention.

Grygorii Strashko report that during CPU hotplug we might see the
warning via
 rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock()

because the counter is not yet set.
It is also possible to trigger the warning from cpu_chill()
(seen on a kblockd_mod_delayed_work_on() caller).

To address this RCU warning I annotate the sleeping lock context. The
counter is incremented before migrate_disable() so the warning Grygorii
should not trigger anymore. Additionally I use that counter in
cpu_chill() to avoid the RCU warning from there.

Reported-by: Grygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
 include/linux/sched.h      | 20 ++++++++++++++++++++
 kernel/locking/rtmutex.c   | 12 ++++++++++--
 kernel/locking/rwlock-rt.c | 18 ++++++++++++++----
 kernel/rcu/tree_plugin.h   |  8 ++++----
 kernel/time/hrtimer.c      |  2 ++
 5 files changed, 50 insertions(+), 10 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index cb78e2785f60..e90a15f38bf5 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -613,6 +613,9 @@ struct task_struct {
 	int				migrate_disable_atomic;
 # endif
 #endif
+#ifdef CONFIG_PREEMPT_RT_FULL
+	int				sleeping_lock;
+#endif
 
 #ifdef CONFIG_PREEMPT_RCU
 	int				rcu_read_lock_nesting;
@@ -1774,6 +1777,23 @@ static __always_inline bool need_resched(void)
 	return unlikely(tif_need_resched());
 }
 
+#ifdef CONFIG_PREEMPT_RT_FULL
+static inline void sleeping_lock_inc(void)
+{
+	current->sleeping_lock++;
+}
+
+static inline void sleeping_lock_dec(void)
+{
+	current->sleeping_lock--;
+}
+
+#else
+
+static inline void sleeping_lock_inc(void) { }
+static inline void sleeping_lock_dec(void) { }
+#endif
+
 /*
  * Wrappers for p->thread_info->cpu access. No-op on UP.
  */
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 65cf3f6851eb..761a910b8489 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
 
 void __lockfunc rt_spin_lock(spinlock_t *lock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
 	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
 	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
 	spin_release(&lock->dep_map, 1, _RET_IP_);
 	rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_spin_unlock);
 
@@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = __rt_mutex_trylock(&lock->lock);
-	if (ret)
+	if (ret) {
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_spin_trylock);
@@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
 	local_bh_disable();
 	ret = __rt_mutex_trylock(&lock->lock);
 	if (ret) {
+		sleeping_lock_inc();
 		migrate_disable();
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
 	} else
@@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
 	*flags = 0;
 	ret = __rt_mutex_trylock(&lock->lock);
 	if (ret) {
+		sleeping_lock_inc();
 		migrate_disable();
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
 	}
diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
index aebb7ce25bc6..f2e155b2c4a8 100644
--- a/kernel/locking/rwlock-rt.c
+++ b/kernel/locking/rwlock-rt.c
@@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = do_read_rt_trylock(rwlock);
-	if (ret)
+	if (ret) {
 		rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_read_trylock);
@@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = do_write_rt_trylock(rwlock);
-	if (ret)
+	if (ret) {
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_write_trylock);
 
 void __lockfunc rt_read_lock(rwlock_t *rwlock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
 	do_read_rt_lock(rwlock);
@@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
 
 void __lockfunc rt_write_lock(rwlock_t *rwlock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
 	do_write_rt_lock(rwlock);
@@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	do_read_rt_unlock(rwlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_read_unlock);
 
@@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	do_write_rt_unlock(rwlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_write_unlock);
 
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3933026838eb..181afc1ab6e2 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -300,13 +300,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
 	struct task_struct *t = current;
 	struct rcu_data *rdp;
 	struct rcu_node *rnp;
-	int mg_counter = 0;
+	int sleeping_l = 0;
 
 	lockdep_assert_irqs_disabled();
-#if defined(CONFIG_PREEMPT_RT_BASE)
-	mg_counter = t->migrate_disable;
+#if defined(CONFIG_PREEMPT_RT_FULL)
+	sleeping_l = t->sleeping_lock;
 #endif
-	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
+	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
 	if (t->rcu_read_lock_nesting > 0 &&
 	    !t->rcu_read_unlock_special.b.blocked) {
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 43c2f571a842..9e1b766dc35c 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1854,7 +1854,9 @@ void cpu_chill(void)
 	chill_time = ktime_set(0, NSEC_PER_MSEC);
 	set_current_state(TASK_UNINTERRUPTIBLE);
 	current->flags |= PF_NOFREEZE;
+	sleeping_lock_inc();
 	schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
+	sleeping_lock_dec();
 	if (!freeze_flag)
 		current->flags &= ~PF_NOFREEZE;
 }
-- 
2.17.0

Re: [ANNOUNCE] v4.14.29-rt25

[Grygorii Strashko]

On 04/20/2018 12:11 PM, Sebastian Andrzej Siewior wrote:
> On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote:
>> Hi Sebastian,
> Hi Grygorii,
> 
>> I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
>> while holding the RCU lock" [1] on top. Then I run below tests:
> …
>> no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings
> 
> the warning is unrelated to the patch I posted. This should mute it:
> 
> Subject: [PATCH RT] rtmutex: annotate sleeping lock context
> 
> The RCU code complains on schedule() within a rcu_readlock() section.
> The valid scenario on -RT is if a sleeping is held. In order to suppress
> the warning the mirgrate_disable counter was used to identify the
> invocation of schedule() due to lock contention.
> 
> Grygorii Strashko report that during CPU hotplug we might see the
> warning via
>   rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock()
> 
> because the counter is not yet set.
> It is also possible to trigger the warning from cpu_chill()
> (seen on a kblockd_mod_delayed_work_on() caller).
> 
> To address this RCU warning I annotate the sleeping lock context. The
> counter is incremented before migrate_disable() so the warning Grygorii
> should not trigger anymore. Additionally I use that counter in
> cpu_chill() to avoid the RCU warning from there.

Sry, but I can't apply it. What's you base?

> 
> Reported-by: Grygorii Strashko <grygorii.strashko@ti.com>
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
-- 
regards,
-grygorii

Re: [ANNOUNCE] v4.14.29-rt25

[Sebastian Andrzej Siewior]

On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote:
> Sry, but I can't apply it. What's you base?

please try this patch and latest v4.14-RT

diff --git a/include/linux/sched.h b/include/linux/sched.h
index e305a8f8cd7d..0322503084a5 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -600,6 +600,9 @@ struct task_struct {
 	int				migrate_disable_atomic;
 # endif
 #endif
+#ifdef CONFIG_PREEMPT_RT_FULL
+	int				sleeping_lock;
+#endif
 
 #ifdef CONFIG_PREEMPT_RCU
 	int				rcu_read_lock_nesting;
@@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void)
 	return unlikely(tif_need_resched());
 }
 
+#ifdef CONFIG_PREEMPT_RT_FULL
+static inline void sleeping_lock_inc(void)
+{
+	current->sleeping_lock++;
+}
+
+static inline void sleeping_lock_dec(void)
+{
+	current->sleeping_lock--;
+}
+
+#else
+
+static inline void sleeping_lock_inc(void) { }
+static inline void sleeping_lock_dec(void) { }
+#endif
+
 /*
  * Wrappers for p->thread_info->cpu access. No-op on UP.
  */
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 0cb716ba3be0..bac3fb580af6 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
 
 void __lockfunc rt_spin_lock(spinlock_t *lock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
 	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
 #ifdef CONFIG_DEBUG_LOCK_ALLOC
 void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
 	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
 	spin_release(&lock->dep_map, 1, _RET_IP_);
 	rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_spin_unlock);
 
@@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = __rt_mutex_trylock(&lock->lock);
-	if (ret)
+	if (ret) {
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_spin_trylock);
@@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
 	local_bh_disable();
 	ret = __rt_mutex_trylock(&lock->lock);
 	if (ret) {
+		sleeping_lock_inc();
 		migrate_disable();
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
 	} else
@@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
 	*flags = 0;
 	ret = __rt_mutex_trylock(&lock->lock);
 	if (ret) {
+		sleeping_lock_inc();
 		migrate_disable();
 		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
 	}
diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
index aebb7ce25bc6..f2e155b2c4a8 100644
--- a/kernel/locking/rwlock-rt.c
+++ b/kernel/locking/rwlock-rt.c
@@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = do_read_rt_trylock(rwlock);
-	if (ret)
+	if (ret) {
 		rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_read_trylock);
@@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
 {
 	int ret;
 
+	sleeping_lock_inc();
 	migrate_disable();
 	ret = do_write_rt_trylock(rwlock);
-	if (ret)
+	if (ret) {
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-	else
+	} else {
 		migrate_enable();
+		sleeping_lock_dec();
+	}
 	return ret;
 }
 EXPORT_SYMBOL(rt_write_trylock);
 
 void __lockfunc rt_read_lock(rwlock_t *rwlock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
 	do_read_rt_lock(rwlock);
@@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
 
 void __lockfunc rt_write_lock(rwlock_t *rwlock)
 {
+	sleeping_lock_inc();
 	migrate_disable();
 	rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
 	do_write_rt_lock(rwlock);
@@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	do_read_rt_unlock(rwlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_read_unlock);
 
@@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	do_write_rt_unlock(rwlock);
 	migrate_enable();
+	sleeping_lock_dec();
 }
 EXPORT_SYMBOL(rt_write_unlock);
 
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3315ebad932f..c6098f60e641 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
 	struct task_struct *t = current;
 	struct rcu_data *rdp;
 	struct rcu_node *rnp;
-	int mg_counter = 0;
+	int sleeping_l = 0;
 
 	RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n");
-#if defined(CONFIG_PREEMPT_RT_BASE)
-	mg_counter = t->migrate_disable;
+#if defined(CONFIG_PREEMPT_RT_FULL)
+	sleeping_l = t->sleeping_lock;
 #endif
-	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
+	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
 	if (t->rcu_read_lock_nesting > 0 &&
 	    !t->rcu_read_unlock_special.b.blocked) {
 
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ce2c2d04cbaa..b59e009087a9 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1870,7 +1870,9 @@ void cpu_chill(void)
 	chill_time = ktime_set(0, NSEC_PER_MSEC);
 	set_current_state(TASK_UNINTERRUPTIBLE);
 	current->flags |= PF_NOFREEZE;
+	sleeping_lock_inc();
 	schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
+	sleeping_lock_dec();
 	if (!freeze_flag)
 		current->flags &= ~PF_NOFREEZE;
 }

Sebastian

Re: [ANNOUNCE] v4.14.29-rt25

[Grygorii Strashko]

Hi Sebastian,

On 04/23/2018 02:10 PM, Sebastian Andrzej Siewior wrote:
> On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote:
>> Sry, but I can't apply it. What's you base?
> 
> please try this patch and latest v4.14-RT

I've tried this (with  v4.14.34-rt27) and I do not see rcu_note_context_switch() any more.

Sry, it took some time as i found some instability -
test "stress-ng --class os --all 0 -t 5m " not always finished :(
So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
pure rt-devel. Still not sure if this is some sort of regression or not.

> 
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index e305a8f8cd7d..0322503084a5 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -600,6 +600,9 @@ struct task_struct {
>   	int				migrate_disable_atomic;
>   # endif
>   #endif
> +#ifdef CONFIG_PREEMPT_RT_FULL
> +	int				sleeping_lock;
> +#endif
>   
>   #ifdef CONFIG_PREEMPT_RCU
>   	int				rcu_read_lock_nesting;
> @@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void)
>   	return unlikely(tif_need_resched());
>   }
>   
> +#ifdef CONFIG_PREEMPT_RT_FULL
> +static inline void sleeping_lock_inc(void)
> +{
> +	current->sleeping_lock++;
> +}
> +
> +static inline void sleeping_lock_dec(void)
> +{
> +	current->sleeping_lock--;
> +}
> +
> +#else
> +
> +static inline void sleeping_lock_inc(void) { }
> +static inline void sleeping_lock_dec(void) { }
> +#endif
> +
>   /*
>    * Wrappers for p->thread_info->cpu access. No-op on UP.
>    */
> diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
> index 0cb716ba3be0..bac3fb580af6 100644
> --- a/kernel/locking/rtmutex.c
> +++ b/kernel/locking/rtmutex.c
> @@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
>   
>   void __lockfunc rt_spin_lock(spinlock_t *lock)
>   {
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
>   	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
> @@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
>   #ifdef CONFIG_DEBUG_LOCK_ALLOC
>   void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
>   {
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
>   	rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
> @@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
>   	spin_release(&lock->dep_map, 1, _RET_IP_);
>   	rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
>   	migrate_enable();
> +	sleeping_lock_dec();
>   }
>   EXPORT_SYMBOL(rt_spin_unlock);
>   
> @@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
>   {
>   	int ret;
>   
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	ret = __rt_mutex_trylock(&lock->lock);
> -	if (ret)
> +	if (ret) {
>   		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
> -	else
> +	} else {
>   		migrate_enable();
> +		sleeping_lock_dec();
> +	}
>   	return ret;
>   }
>   EXPORT_SYMBOL(rt_spin_trylock);
> @@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
>   	local_bh_disable();
>   	ret = __rt_mutex_trylock(&lock->lock);
>   	if (ret) {
> +		sleeping_lock_inc();
>   		migrate_disable();
>   		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
>   	} else
> @@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
>   	*flags = 0;
>   	ret = __rt_mutex_trylock(&lock->lock);
>   	if (ret) {
> +		sleeping_lock_inc();
>   		migrate_disable();
>   		spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
>   	}
> diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
> index aebb7ce25bc6..f2e155b2c4a8 100644
> --- a/kernel/locking/rwlock-rt.c
> +++ b/kernel/locking/rwlock-rt.c
> @@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
>   {
>   	int ret;
>   
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	ret = do_read_rt_trylock(rwlock);
> -	if (ret)
> +	if (ret) {
>   		rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
> -	else
> +	} else {
>   		migrate_enable();
> +		sleeping_lock_dec();
> +	}
>   	return ret;
>   }
>   EXPORT_SYMBOL(rt_read_trylock);
> @@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
>   {
>   	int ret;
>   
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	ret = do_write_rt_trylock(rwlock);
> -	if (ret)
> +	if (ret) {
>   		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
> -	else
> +	} else {
>   		migrate_enable();
> +		sleeping_lock_dec();
> +	}
>   	return ret;
>   }
>   EXPORT_SYMBOL(rt_write_trylock);
>   
>   void __lockfunc rt_read_lock(rwlock_t *rwlock)
>   {
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
>   	do_read_rt_lock(rwlock);
> @@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
>   
>   void __lockfunc rt_write_lock(rwlock_t *rwlock)
>   {
> +	sleeping_lock_inc();
>   	migrate_disable();
>   	rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
>   	do_write_rt_lock(rwlock);
> @@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
>   	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
>   	do_read_rt_unlock(rwlock);
>   	migrate_enable();
> +	sleeping_lock_dec();
>   }
>   EXPORT_SYMBOL(rt_read_unlock);
>   
> @@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
>   	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
>   	do_write_rt_unlock(rwlock);
>   	migrate_enable();
> +	sleeping_lock_dec();
>   }
>   EXPORT_SYMBOL(rt_write_unlock);
>   
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 3315ebad932f..c6098f60e641 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
>   	struct task_struct *t = current;
>   	struct rcu_data *rdp;
>   	struct rcu_node *rnp;
> -	int mg_counter = 0;
> +	int sleeping_l = 0;
>   
>   	RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n");
> -#if defined(CONFIG_PREEMPT_RT_BASE)
> -	mg_counter = t->migrate_disable;
> +#if defined(CONFIG_PREEMPT_RT_FULL)
> +	sleeping_l = t->sleeping_lock;
>   #endif
> -	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
> +	WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
>   	if (t->rcu_read_lock_nesting > 0 &&
>   	    !t->rcu_read_unlock_special.b.blocked) {
>   
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index ce2c2d04cbaa..b59e009087a9 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -1870,7 +1870,9 @@ void cpu_chill(void)
>   	chill_time = ktime_set(0, NSEC_PER_MSEC);
>   	set_current_state(TASK_UNINTERRUPTIBLE);
>   	current->flags |= PF_NOFREEZE;
> +	sleeping_lock_inc();
>   	schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
> +	sleeping_lock_dec();
>   	if (!freeze_flag)
>   		current->flags &= ~PF_NOFREEZE;
>   }
> 
> Sebastian
> 

-- 
regards,
-grygorii

Re: [ANNOUNCE] v4.14.29-rt25

[Sebastian Andrzej Siewior]

On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote:
> Hi Sebastian,
Hi Grygorii,

> Sry, it took some time as i found some instability -
> test "stress-ng --class os --all 0 -t 5m " not always finished :(
> So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
> pure rt-devel. Still not sure if this is some sort of regression or not.

is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does
not? This patch just silences the warning so it should not break
anything…

Sebastian

Re: [ANNOUNCE] v4.14.29-rt25

[Grygorii Strashko]

On 05/03/2018 10:24 AM, Sebastian Andrzej Siewior wrote:
> On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote:
>> Hi Sebastian,
> Hi Grygorii,
> 
>> Sry, it took some time as i found some instability -
>> test "stress-ng --class os --all 0 -t 5m " not always finished :(
>> So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
>> pure rt-devel. Still not sure if this is some sort of regression or not.
> 
> is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does
> not? This patch just silences the warning so it should not break
> anything…

Yes. It seems not related.

-- 
regards,
-grygorii