* [ANNOUNCE] v4.14.29-rt25
@ 2018-03-25 10:12 Sebastian Andrzej Siewior
2018-03-27 18:01 ` Grygorii Strashko
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Andrzej Siewior @ 2018-03-25 10:12 UTC (permalink / raw)
To: Thomas Gleixner; +Cc: LKML, linux-rt-users, Steven Rostedt
Dear RT folks!
I'm pleased to announce the v4.14.29-rt25 patch set.
Changes since v4.14.29-rt24:
- There are checks for disabled interrupts in the target code which do
not work on RT. Reported by Arnaldo Carvalho de Melo.
Known issues
- A warning triggered in "rcu_note_context_switch" originated from
SyS_timer_gettime(). The issue was always there, it is now
visible. Reported by Grygorii Strashko and Daniel Wagner.
The delta patch against v4.14.29-rt24 is appended below and can be found here:
https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz
You can get this release via the git tree at:
git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25
The RT patch against v4.14.29 can be found here:
https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz
The split quilt queue is available at:
https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz
Sebastian
diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c
index 9c7bc1ca341a..3d35dad1de2c 100644
--- a/drivers/target/target_core_tmr.c
+++ b/drivers/target/target_core_tmr.c
@@ -114,8 +114,6 @@ static bool __target_check_io_state(struct se_cmd *se_cmd,
{
struct se_session *sess = se_cmd->se_sess;
- assert_spin_locked(&sess->sess_cmd_lock);
- WARN_ON_ONCE(!irqs_disabled());
/*
* If command already reached CMD_T_COMPLETE state within
* target_complete_cmd() or CMD_T_FABRIC_STOP due to shutdown,
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index e6d51135d105..b00829995131 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -2966,9 +2966,6 @@ __transport_wait_for_tasks(struct se_cmd *cmd, bool fabric_stop,
__acquires(&cmd->t_state_lock)
{
- assert_spin_locked(&cmd->t_state_lock);
- WARN_ON_ONCE(!irqs_disabled());
-
if (fabric_stop)
cmd->transport_state |= CMD_T_FABRIC_STOP;
@@ -3238,9 +3235,6 @@ static int __transport_check_aborted_status(struct se_cmd *cmd, int send_status)
{
int ret;
- assert_spin_locked(&cmd->t_state_lock);
- WARN_ON_ONCE(!irqs_disabled());
-
if (!(cmd->transport_state & CMD_T_ABORTED))
return 0;
/*
diff --git a/localversion-rt b/localversion-rt
index b2111a212663..c5b71f9a229d 100644
--- a/localversion-rt
+++ b/localversion-rt
@@ -1 +1 @@
--rt24
+-rt25
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-03-25 10:12 [ANNOUNCE] v4.14.29-rt25 Sebastian Andrzej Siewior
@ 2018-03-27 18:01 ` Grygorii Strashko
2018-04-20 17:11 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 8+ messages in thread
From: Grygorii Strashko @ 2018-03-27 18:01 UTC (permalink / raw)
To: Sebastian Andrzej Siewior, Thomas Gleixner
Cc: LKML, linux-rt-users, Steven Rostedt
Hi Sebastian,
On 03/25/2018 05:12 AM, Sebastian Andrzej Siewior wrote:
> Dear RT folks!
>
> I'm pleased to announce the v4.14.29-rt25 patch set.
>
> Changes since v4.14.29-rt24:
>
> - There are checks for disabled interrupts in the target code which do
> not work on RT. Reported by Arnaldo Carvalho de Melo.
>
> Known issues
> - A warning triggered in "rcu_note_context_switch" originated from
> SyS_timer_gettime(). The issue was always there, it is now
> visible. Reported by Grygorii Strashko and Daniel Wagner.
>
> The delta patch against v4.14.29-rt24 is appended below and can be found here:
>
> https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz
>
> You can get this release via the git tree at:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25
>
> The RT patch against v4.14.29 can be found here:
>
> https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz
>
> The split quilt queue is available at:
>
> https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz
>
> Sebastian
>
I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
while holding the RCU lock" [1] on top. Then I run below tests:
## stress-ng --class os --all 0 -t 5m
- and -
## stress-ng --class cpu --all 0 -t 5m & stress-ng --class memory --all 0 --vm-bytes 90% -t 5m
no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings
[1] https://patchwork.kernel.org/patch/10305015/
==== WARN1 ====
[ 104.967174] ------------[ cut here ]------------
[ 104.967191] WARNING: CPU: 0 PID: 1676 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4
stress-ng: error: [913] stress-ng-fifo: read timeout!
stress-ng: error: [924] stress-ng-fifo: read timeout!
[ 104.967194] Modules linked in:
[ 104.967206] CPU: 0 PID: 1676 Comm: stress-ng-fstat Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269
stress-ng: error: [926] stress-ng-fifo: read timeout!
stress-ng: error: [920] stress-ng-fifo: read timeout!
[ 104.967210] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 104.967227] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14)
[ 104.967239] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4)
[ 104.967251] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100)
[ 104.967261] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28)
[ 104.967271] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4)
stress-ng: fail: [828] stress-ng-af-alg: socket failed, errno=9[ 104.967285] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814)
7 (Address family not supported by protocol)
[ 104.967294] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0)
[ 104.967302] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4)
stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETQUOTA failed: errno=38 (Function not implemented)
stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETFMT failed: errno=38 (Function not implemented)
stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETINFO failed: errno=38 (Function not implemented)
stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_SYNC failed: errno=38 (Function not implemented)
stress-ng: error: [1484] stress-ng-quota: quotactl() failed, not available on this kernel
[ 104.967311] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0)
[ 104.967320] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c)
[ 104.967329] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4)
[ 104.967339] [<c015a5d8>] (migrate_disable) from [<c0842338>] (rt_spin_lock+0x18/0x60)
[ 104.967349] [<c0842338>] (rt_spin_lock) from [<c0440994>] (lockref_get_not_dead+0xc/0x34)
[ 104.967360] [<c0440994>] (lockref_get_not_dead) from [<c0273fb8>] (legitimize_path+0x2c/0x64)
[ 104.967368] [<c0273fb8>] (legitimize_path) from [<c02740e4>] (unlazy_walk+0x3c/0x18c)
[ 104.967376] [<c02740e4>] (unlazy_walk) from [<c0274258>] (complete_walk+0x24/0x78)
[ 104.967384] [<c0274258>] (complete_walk) from [<c02768b8>] (path_lookupat+0xbc/0x1ec)
[ 104.967393] [<c02768b8>] (path_lookupat) from [<c0278f98>] (filename_lookup+0x8c/0xe8)
[ 104.967404] [<c0278f98>] (filename_lookup) from [<c026e1ac>] (vfs_statx+0x5c/0xb8)
[ 104.967414] [<c026e1ac>] (vfs_statx) from [<c026e8ec>] (SyS_stat64+0x28/0x44)
[ 104.967424] [<c026e8ec>] (SyS_stat64) from [<c0108480>] (ret_fast_syscall+0x0/0x28)
[ 104.967429] ---[ end trace 0000000000000002 ]---
=== WARN2 ====
[ 196.437276] ------------[ cut here ]------------
[ 196.437293] WARNING: CPU: 0 PID: 1521 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4
[ 196.437296] Modules linked in:
[ 196.437307] CPU: 0 PID: 1521 Comm: stress-ng-opcod Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269
[ 196.437311] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 196.437326] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14)
[ 196.437338] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4)
[ 196.437349] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100)
[ 196.437359] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28)
[ 196.437369] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4)
[ 196.437381] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814)
[ 196.437390] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0)
[ 196.437398] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4)
[ 196.437407] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0)
[ 196.437416] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c)
[ 196.437424] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4)
[ 196.437433] [<c015a5d8>] (migrate_disable) from [<c011a3a4>] (kmap_atomic+0x34/0x174)
[ 196.437443] [<c011a3a4>] (kmap_atomic) from [<c0236484>] (alloc_set_pte+0x288/0x320)
[ 196.437453] [<c0236484>] (alloc_set_pte) from [<c01ffcb8>] (filemap_map_pages+0x2b0/0x4c8)
[ 196.437461] [<c01ffcb8>] (filemap_map_pages) from [<c0236c4c>] (handle_mm_fault+0x690/0xc78)
[ 196.437468] [<c0236c4c>] (handle_mm_fault) from [<c0115bd8>] (do_page_fault+0x2bc/0x398)
[ 196.437476] [<c0115bd8>] (do_page_fault) from [<c0101474>] (do_PrefetchAbort+0x38/0xa0)
[ 196.437484] [<c0101474>] (do_PrefetchAbort) from [<c010decc>] (ret_from_exception+0x0/0x14)
[ 196.437488] Exception stack(0xe9943fb0 to 0xe9943ff8)
[ 196.437494] 3fa0: 00000000 00000000 00000000 00000000
[ 196.437502] 3fc0: b6f6e468 bee42888 b6f6e400 bee42898 b6f6e8c0 00000000 000003af 000003af
[ 196.437508] 3fe0: 00000078 bee42888 b6d9ac91 b6d9ac90 200b0030 ffffffff
[ 196.437513] ---[ end trace 0000000000000002 ]---
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-03-27 18:01 ` Grygorii Strashko
@ 2018-04-20 17:11 ` Sebastian Andrzej Siewior
2018-04-23 16:57 ` Grygorii Strashko
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Andrzej Siewior @ 2018-04-20 17:11 UTC (permalink / raw)
To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote:
> Hi Sebastian,
Hi Grygorii,
> I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
> while holding the RCU lock" [1] on top. Then I run below tests:
…
> no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings
the warning is unrelated to the patch I posted. This should mute it:
Subject: [PATCH RT] rtmutex: annotate sleeping lock context
The RCU code complains on schedule() within a rcu_readlock() section.
The valid scenario on -RT is if a sleeping is held. In order to suppress
the warning the mirgrate_disable counter was used to identify the
invocation of schedule() due to lock contention.
Grygorii Strashko report that during CPU hotplug we might see the
warning via
rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock()
because the counter is not yet set.
It is also possible to trigger the warning from cpu_chill()
(seen on a kblockd_mod_delayed_work_on() caller).
To address this RCU warning I annotate the sleeping lock context. The
counter is incremented before migrate_disable() so the warning Grygorii
should not trigger anymore. Additionally I use that counter in
cpu_chill() to avoid the RCU warning from there.
Reported-by: Grygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
---
include/linux/sched.h | 20 ++++++++++++++++++++
kernel/locking/rtmutex.c | 12 ++++++++++--
kernel/locking/rwlock-rt.c | 18 ++++++++++++++----
kernel/rcu/tree_plugin.h | 8 ++++----
kernel/time/hrtimer.c | 2 ++
5 files changed, 50 insertions(+), 10 deletions(-)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index cb78e2785f60..e90a15f38bf5 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -613,6 +613,9 @@ struct task_struct {
int migrate_disable_atomic;
# endif
#endif
+#ifdef CONFIG_PREEMPT_RT_FULL
+ int sleeping_lock;
+#endif
#ifdef CONFIG_PREEMPT_RCU
int rcu_read_lock_nesting;
@@ -1774,6 +1777,23 @@ static __always_inline bool need_resched(void)
return unlikely(tif_need_resched());
}
+#ifdef CONFIG_PREEMPT_RT_FULL
+static inline void sleeping_lock_inc(void)
+{
+ current->sleeping_lock++;
+}
+
+static inline void sleeping_lock_dec(void)
+{
+ current->sleeping_lock--;
+}
+
+#else
+
+static inline void sleeping_lock_inc(void) { }
+static inline void sleeping_lock_dec(void) { }
+#endif
+
/*
* Wrappers for p->thread_info->cpu access. No-op on UP.
*/
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 65cf3f6851eb..761a910b8489 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
void __lockfunc rt_spin_lock(spinlock_t *lock)
{
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
#ifdef CONFIG_DEBUG_LOCK_ALLOC
void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
{
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
spin_release(&lock->dep_map, 1, _RET_IP_);
rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_spin_unlock);
@@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = __rt_mutex_trylock(&lock->lock);
- if (ret)
+ if (ret) {
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_spin_trylock);
@@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
local_bh_disable();
ret = __rt_mutex_trylock(&lock->lock);
if (ret) {
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
} else
@@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
*flags = 0;
ret = __rt_mutex_trylock(&lock->lock);
if (ret) {
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
}
diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
index aebb7ce25bc6..f2e155b2c4a8 100644
--- a/kernel/locking/rwlock-rt.c
+++ b/kernel/locking/rwlock-rt.c
@@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = do_read_rt_trylock(rwlock);
- if (ret)
+ if (ret) {
rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_read_trylock);
@@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = do_write_rt_trylock(rwlock);
- if (ret)
+ if (ret) {
rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_write_trylock);
void __lockfunc rt_read_lock(rwlock_t *rwlock)
{
+ sleeping_lock_inc();
migrate_disable();
rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
do_read_rt_lock(rwlock);
@@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
void __lockfunc rt_write_lock(rwlock_t *rwlock)
{
+ sleeping_lock_inc();
migrate_disable();
rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
do_write_rt_lock(rwlock);
@@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
do_read_rt_unlock(rwlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_read_unlock);
@@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
do_write_rt_unlock(rwlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_write_unlock);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3933026838eb..181afc1ab6e2 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -300,13 +300,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
struct task_struct *t = current;
struct rcu_data *rdp;
struct rcu_node *rnp;
- int mg_counter = 0;
+ int sleeping_l = 0;
lockdep_assert_irqs_disabled();
-#if defined(CONFIG_PREEMPT_RT_BASE)
- mg_counter = t->migrate_disable;
+#if defined(CONFIG_PREEMPT_RT_FULL)
+ sleeping_l = t->sleeping_lock;
#endif
- WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
+ WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
if (t->rcu_read_lock_nesting > 0 &&
!t->rcu_read_unlock_special.b.blocked) {
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 43c2f571a842..9e1b766dc35c 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1854,7 +1854,9 @@ void cpu_chill(void)
chill_time = ktime_set(0, NSEC_PER_MSEC);
set_current_state(TASK_UNINTERRUPTIBLE);
current->flags |= PF_NOFREEZE;
+ sleeping_lock_inc();
schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
+ sleeping_lock_dec();
if (!freeze_flag)
current->flags &= ~PF_NOFREEZE;
}
--
2.17.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-04-20 17:11 ` Sebastian Andrzej Siewior
@ 2018-04-23 16:57 ` Grygorii Strashko
2018-04-23 19:10 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 8+ messages in thread
From: Grygorii Strashko @ 2018-04-23 16:57 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
On 04/20/2018 12:11 PM, Sebastian Andrzej Siewior wrote:
> On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote:
>> Hi Sebastian,
> Hi Grygorii,
>
>> I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule()
>> while holding the RCU lock" [1] on top. Then I run below tests:
> …
>> no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings
>
> the warning is unrelated to the patch I posted. This should mute it:
>
> Subject: [PATCH RT] rtmutex: annotate sleeping lock context
>
> The RCU code complains on schedule() within a rcu_readlock() section.
> The valid scenario on -RT is if a sleeping is held. In order to suppress
> the warning the mirgrate_disable counter was used to identify the
> invocation of schedule() due to lock contention.
>
> Grygorii Strashko report that during CPU hotplug we might see the
> warning via
> rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock()
>
> because the counter is not yet set.
> It is also possible to trigger the warning from cpu_chill()
> (seen on a kblockd_mod_delayed_work_on() caller).
>
> To address this RCU warning I annotate the sleeping lock context. The
> counter is incremented before migrate_disable() so the warning Grygorii
> should not trigger anymore. Additionally I use that counter in
> cpu_chill() to avoid the RCU warning from there.
Sry, but I can't apply it. What's you base?
>
> Reported-by: Grygorii Strashko <grygorii.strashko@ti.com>
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> ---
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-04-23 16:57 ` Grygorii Strashko
@ 2018-04-23 19:10 ` Sebastian Andrzej Siewior
2018-05-01 15:04 ` Grygorii Strashko
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Andrzej Siewior @ 2018-04-23 19:10 UTC (permalink / raw)
To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote:
> Sry, but I can't apply it. What's you base?
please try this patch and latest v4.14-RT
diff --git a/include/linux/sched.h b/include/linux/sched.h
index e305a8f8cd7d..0322503084a5 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -600,6 +600,9 @@ struct task_struct {
int migrate_disable_atomic;
# endif
#endif
+#ifdef CONFIG_PREEMPT_RT_FULL
+ int sleeping_lock;
+#endif
#ifdef CONFIG_PREEMPT_RCU
int rcu_read_lock_nesting;
@@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void)
return unlikely(tif_need_resched());
}
+#ifdef CONFIG_PREEMPT_RT_FULL
+static inline void sleeping_lock_inc(void)
+{
+ current->sleeping_lock++;
+}
+
+static inline void sleeping_lock_dec(void)
+{
+ current->sleeping_lock--;
+}
+
+#else
+
+static inline void sleeping_lock_inc(void) { }
+static inline void sleeping_lock_dec(void) { }
+#endif
+
/*
* Wrappers for p->thread_info->cpu access. No-op on UP.
*/
diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
index 0cb716ba3be0..bac3fb580af6 100644
--- a/kernel/locking/rtmutex.c
+++ b/kernel/locking/rtmutex.c
@@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
void __lockfunc rt_spin_lock(spinlock_t *lock)
{
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
#ifdef CONFIG_DEBUG_LOCK_ALLOC
void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
{
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
@@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
spin_release(&lock->dep_map, 1, _RET_IP_);
rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_spin_unlock);
@@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = __rt_mutex_trylock(&lock->lock);
- if (ret)
+ if (ret) {
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_spin_trylock);
@@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
local_bh_disable();
ret = __rt_mutex_trylock(&lock->lock);
if (ret) {
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
} else
@@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
*flags = 0;
ret = __rt_mutex_trylock(&lock->lock);
if (ret) {
+ sleeping_lock_inc();
migrate_disable();
spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
}
diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
index aebb7ce25bc6..f2e155b2c4a8 100644
--- a/kernel/locking/rwlock-rt.c
+++ b/kernel/locking/rwlock-rt.c
@@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = do_read_rt_trylock(rwlock);
- if (ret)
+ if (ret) {
rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_read_trylock);
@@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
{
int ret;
+ sleeping_lock_inc();
migrate_disable();
ret = do_write_rt_trylock(rwlock);
- if (ret)
+ if (ret) {
rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
- else
+ } else {
migrate_enable();
+ sleeping_lock_dec();
+ }
return ret;
}
EXPORT_SYMBOL(rt_write_trylock);
void __lockfunc rt_read_lock(rwlock_t *rwlock)
{
+ sleeping_lock_inc();
migrate_disable();
rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
do_read_rt_lock(rwlock);
@@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
void __lockfunc rt_write_lock(rwlock_t *rwlock)
{
+ sleeping_lock_inc();
migrate_disable();
rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
do_write_rt_lock(rwlock);
@@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
do_read_rt_unlock(rwlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_read_unlock);
@@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
do_write_rt_unlock(rwlock);
migrate_enable();
+ sleeping_lock_dec();
}
EXPORT_SYMBOL(rt_write_unlock);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 3315ebad932f..c6098f60e641 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
struct task_struct *t = current;
struct rcu_data *rdp;
struct rcu_node *rnp;
- int mg_counter = 0;
+ int sleeping_l = 0;
RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n");
-#if defined(CONFIG_PREEMPT_RT_BASE)
- mg_counter = t->migrate_disable;
+#if defined(CONFIG_PREEMPT_RT_FULL)
+ sleeping_l = t->sleeping_lock;
#endif
- WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
+ WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
if (t->rcu_read_lock_nesting > 0 &&
!t->rcu_read_unlock_special.b.blocked) {
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index ce2c2d04cbaa..b59e009087a9 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -1870,7 +1870,9 @@ void cpu_chill(void)
chill_time = ktime_set(0, NSEC_PER_MSEC);
set_current_state(TASK_UNINTERRUPTIBLE);
current->flags |= PF_NOFREEZE;
+ sleeping_lock_inc();
schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
+ sleeping_lock_dec();
if (!freeze_flag)
current->flags &= ~PF_NOFREEZE;
}
Sebastian
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-04-23 19:10 ` Sebastian Andrzej Siewior
@ 2018-05-01 15:04 ` Grygorii Strashko
2018-05-03 15:24 ` Sebastian Andrzej Siewior
0 siblings, 1 reply; 8+ messages in thread
From: Grygorii Strashko @ 2018-05-01 15:04 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
Hi Sebastian,
On 04/23/2018 02:10 PM, Sebastian Andrzej Siewior wrote:
> On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote:
>> Sry, but I can't apply it. What's you base?
>
> please try this patch and latest v4.14-RT
I've tried this (with v4.14.34-rt27) and I do not see rcu_note_context_switch() any more.
Sry, it took some time as i found some instability -
test "stress-ng --class os --all 0 -t 5m " not always finished :(
So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
pure rt-devel. Still not sure if this is some sort of regression or not.
>
> diff --git a/include/linux/sched.h b/include/linux/sched.h
> index e305a8f8cd7d..0322503084a5 100644
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -600,6 +600,9 @@ struct task_struct {
> int migrate_disable_atomic;
> # endif
> #endif
> +#ifdef CONFIG_PREEMPT_RT_FULL
> + int sleeping_lock;
> +#endif
>
> #ifdef CONFIG_PREEMPT_RCU
> int rcu_read_lock_nesting;
> @@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void)
> return unlikely(tif_need_resched());
> }
>
> +#ifdef CONFIG_PREEMPT_RT_FULL
> +static inline void sleeping_lock_inc(void)
> +{
> + current->sleeping_lock++;
> +}
> +
> +static inline void sleeping_lock_dec(void)
> +{
> + current->sleeping_lock--;
> +}
> +
> +#else
> +
> +static inline void sleeping_lock_inc(void) { }
> +static inline void sleeping_lock_dec(void) { }
> +#endif
> +
> /*
> * Wrappers for p->thread_info->cpu access. No-op on UP.
> */
> diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c
> index 0cb716ba3be0..bac3fb580af6 100644
> --- a/kernel/locking/rtmutex.c
> +++ b/kernel/locking/rtmutex.c
> @@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock)
>
> void __lockfunc rt_spin_lock(spinlock_t *lock)
> {
> + sleeping_lock_inc();
> migrate_disable();
> spin_acquire(&lock->dep_map, 0, 0, _RET_IP_);
> rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
> @@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock)
> #ifdef CONFIG_DEBUG_LOCK_ALLOC
> void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass)
> {
> + sleeping_lock_inc();
> migrate_disable();
> spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_);
> rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock);
> @@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock)
> spin_release(&lock->dep_map, 1, _RET_IP_);
> rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock);
> migrate_enable();
> + sleeping_lock_dec();
> }
> EXPORT_SYMBOL(rt_spin_unlock);
>
> @@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock)
> {
> int ret;
>
> + sleeping_lock_inc();
> migrate_disable();
> ret = __rt_mutex_trylock(&lock->lock);
> - if (ret)
> + if (ret) {
> spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
> - else
> + } else {
> migrate_enable();
> + sleeping_lock_dec();
> + }
> return ret;
> }
> EXPORT_SYMBOL(rt_spin_trylock);
> @@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock)
> local_bh_disable();
> ret = __rt_mutex_trylock(&lock->lock);
> if (ret) {
> + sleeping_lock_inc();
> migrate_disable();
> spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
> } else
> @@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags)
> *flags = 0;
> ret = __rt_mutex_trylock(&lock->lock);
> if (ret) {
> + sleeping_lock_inc();
> migrate_disable();
> spin_acquire(&lock->dep_map, 0, 1, _RET_IP_);
> }
> diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c
> index aebb7ce25bc6..f2e155b2c4a8 100644
> --- a/kernel/locking/rwlock-rt.c
> +++ b/kernel/locking/rwlock-rt.c
> @@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
> {
> int ret;
>
> + sleeping_lock_inc();
> migrate_disable();
> ret = do_read_rt_trylock(rwlock);
> - if (ret)
> + if (ret) {
> rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_);
> - else
> + } else {
> migrate_enable();
> + sleeping_lock_dec();
> + }
> return ret;
> }
> EXPORT_SYMBOL(rt_read_trylock);
> @@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock)
> {
> int ret;
>
> + sleeping_lock_inc();
> migrate_disable();
> ret = do_write_rt_trylock(rwlock);
> - if (ret)
> + if (ret) {
> rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
> - else
> + } else {
> migrate_enable();
> + sleeping_lock_dec();
> + }
> return ret;
> }
> EXPORT_SYMBOL(rt_write_trylock);
>
> void __lockfunc rt_read_lock(rwlock_t *rwlock)
> {
> + sleeping_lock_inc();
> migrate_disable();
> rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_);
> do_read_rt_lock(rwlock);
> @@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock);
>
> void __lockfunc rt_write_lock(rwlock_t *rwlock)
> {
> + sleeping_lock_inc();
> migrate_disable();
> rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
> do_write_rt_lock(rwlock);
> @@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock)
> rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
> do_read_rt_unlock(rwlock);
> migrate_enable();
> + sleeping_lock_dec();
> }
> EXPORT_SYMBOL(rt_read_unlock);
>
> @@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock)
> rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
> do_write_rt_unlock(rwlock);
> migrate_enable();
> + sleeping_lock_dec();
> }
> EXPORT_SYMBOL(rt_write_unlock);
>
> diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
> index 3315ebad932f..c6098f60e641 100644
> --- a/kernel/rcu/tree_plugin.h
> +++ b/kernel/rcu/tree_plugin.h
> @@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt)
> struct task_struct *t = current;
> struct rcu_data *rdp;
> struct rcu_node *rnp;
> - int mg_counter = 0;
> + int sleeping_l = 0;
>
> RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n");
> -#if defined(CONFIG_PREEMPT_RT_BASE)
> - mg_counter = t->migrate_disable;
> +#if defined(CONFIG_PREEMPT_RT_FULL)
> + sleeping_l = t->sleeping_lock;
> #endif
> - WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter);
> + WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l);
> if (t->rcu_read_lock_nesting > 0 &&
> !t->rcu_read_unlock_special.b.blocked) {
>
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index ce2c2d04cbaa..b59e009087a9 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -1870,7 +1870,9 @@ void cpu_chill(void)
> chill_time = ktime_set(0, NSEC_PER_MSEC);
> set_current_state(TASK_UNINTERRUPTIBLE);
> current->flags |= PF_NOFREEZE;
> + sleeping_lock_inc();
> schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD);
> + sleeping_lock_dec();
> if (!freeze_flag)
> current->flags &= ~PF_NOFREEZE;
> }
>
> Sebastian
>
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-05-01 15:04 ` Grygorii Strashko
@ 2018-05-03 15:24 ` Sebastian Andrzej Siewior
2018-05-03 18:26 ` Grygorii Strashko
0 siblings, 1 reply; 8+ messages in thread
From: Sebastian Andrzej Siewior @ 2018-05-03 15:24 UTC (permalink / raw)
To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote:
> Hi Sebastian,
Hi Grygorii,
> Sry, it took some time as i found some instability -
> test "stress-ng --class os --all 0 -t 5m " not always finished :(
> So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
> pure rt-devel. Still not sure if this is some sort of regression or not.
is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does
not? This patch just silences the warning so it should not break
anything…
Sebastian
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25
2018-05-03 15:24 ` Sebastian Andrzej Siewior
@ 2018-05-03 18:26 ` Grygorii Strashko
0 siblings, 0 replies; 8+ messages in thread
From: Grygorii Strashko @ 2018-05-03 18:26 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt
On 05/03/2018 10:24 AM, Sebastian Andrzej Siewior wrote:
> On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote:
>> Hi Sebastian,
> Hi Grygorii,
>
>> Sry, it took some time as i found some instability -
>> test "stress-ng --class os --all 0 -t 5m " not always finished :(
>> So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as
>> pure rt-devel. Still not sure if this is some sort of regression or not.
>
> is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does
> not? This patch just silences the warning so it should not break
> anything…
Yes. It seems not related.
--
regards,
-grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-05-03 18:27 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-25 10:12 [ANNOUNCE] v4.14.29-rt25 Sebastian Andrzej Siewior
2018-03-27 18:01 ` Grygorii Strashko
2018-04-20 17:11 ` Sebastian Andrzej Siewior
2018-04-23 16:57 ` Grygorii Strashko
2018-04-23 19:10 ` Sebastian Andrzej Siewior
2018-05-01 15:04 ` Grygorii Strashko
2018-05-03 15:24 ` Sebastian Andrzej Siewior
2018-05-03 18:26 ` Grygorii Strashko
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).