* [ANNOUNCE] v4.14.29-rt25 @ 2018-03-25 10:12 Sebastian Andrzej Siewior 2018-03-27 18:01 ` Grygorii Strashko 0 siblings, 1 reply; 8+ messages in thread From: Sebastian Andrzej Siewior @ 2018-03-25 10:12 UTC (permalink / raw) To: Thomas Gleixner; +Cc: LKML, linux-rt-users, Steven Rostedt Dear RT folks! I'm pleased to announce the v4.14.29-rt25 patch set. Changes since v4.14.29-rt24: - There are checks for disabled interrupts in the target code which do not work on RT. Reported by Arnaldo Carvalho de Melo. Known issues - A warning triggered in "rcu_note_context_switch" originated from SyS_timer_gettime(). The issue was always there, it is now visible. Reported by Grygorii Strashko and Daniel Wagner. The delta patch against v4.14.29-rt24 is appended below and can be found here: https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz You can get this release via the git tree at: git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25 The RT patch against v4.14.29 can be found here: https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz The split quilt queue is available at: https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz Sebastian diff --git a/drivers/target/target_core_tmr.c b/drivers/target/target_core_tmr.c index 9c7bc1ca341a..3d35dad1de2c 100644 --- a/drivers/target/target_core_tmr.c +++ b/drivers/target/target_core_tmr.c @@ -114,8 +114,6 @@ static bool __target_check_io_state(struct se_cmd *se_cmd, { struct se_session *sess = se_cmd->se_sess; - assert_spin_locked(&sess->sess_cmd_lock); - WARN_ON_ONCE(!irqs_disabled()); /* * If command already reached CMD_T_COMPLETE state within * target_complete_cmd() or CMD_T_FABRIC_STOP due to shutdown, diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c index e6d51135d105..b00829995131 100644 --- a/drivers/target/target_core_transport.c +++ b/drivers/target/target_core_transport.c @@ -2966,9 +2966,6 @@ __transport_wait_for_tasks(struct se_cmd *cmd, bool fabric_stop, __acquires(&cmd->t_state_lock) { - assert_spin_locked(&cmd->t_state_lock); - WARN_ON_ONCE(!irqs_disabled()); - if (fabric_stop) cmd->transport_state |= CMD_T_FABRIC_STOP; @@ -3238,9 +3235,6 @@ static int __transport_check_aborted_status(struct se_cmd *cmd, int send_status) { int ret; - assert_spin_locked(&cmd->t_state_lock); - WARN_ON_ONCE(!irqs_disabled()); - if (!(cmd->transport_state & CMD_T_ABORTED)) return 0; /* diff --git a/localversion-rt b/localversion-rt index b2111a212663..c5b71f9a229d 100644 --- a/localversion-rt +++ b/localversion-rt @@ -1 +1 @@ --rt24 +-rt25 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-03-25 10:12 [ANNOUNCE] v4.14.29-rt25 Sebastian Andrzej Siewior @ 2018-03-27 18:01 ` Grygorii Strashko 2018-04-20 17:11 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 8+ messages in thread From: Grygorii Strashko @ 2018-03-27 18:01 UTC (permalink / raw) To: Sebastian Andrzej Siewior, Thomas Gleixner Cc: LKML, linux-rt-users, Steven Rostedt Hi Sebastian, On 03/25/2018 05:12 AM, Sebastian Andrzej Siewior wrote: > Dear RT folks! > > I'm pleased to announce the v4.14.29-rt25 patch set. > > Changes since v4.14.29-rt24: > > - There are checks for disabled interrupts in the target code which do > not work on RT. Reported by Arnaldo Carvalho de Melo. > > Known issues > - A warning triggered in "rcu_note_context_switch" originated from > SyS_timer_gettime(). The issue was always there, it is now > visible. Reported by Grygorii Strashko and Daniel Wagner. > > The delta patch against v4.14.29-rt24 is appended below and can be found here: > > https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/incr/patch-4.14.29-rt24-rt25.patch.xz > > You can get this release via the git tree at: > > git://git.kernel.org/pub/scm/linux/kernel/git/rt/linux-rt-devel.git v4.14.29-rt25 > > The RT patch against v4.14.29 can be found here: > > https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patch-4.14.29-rt25.patch.xz > > The split quilt queue is available at: > > https://cdn.kernel.org/pub/linux/kernel/projects/rt/4.14/older/patches-4.14.29-rt25.tar.xz > > Sebastian > I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule() while holding the RCU lock" [1] on top. Then I run below tests: ## stress-ng --class os --all 0 -t 5m - and - ## stress-ng --class cpu --all 0 -t 5m & stress-ng --class memory --all 0 --vm-bytes 90% -t 5m no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings [1] https://patchwork.kernel.org/patch/10305015/ ==== WARN1 ==== [ 104.967174] ------------[ cut here ]------------ [ 104.967191] WARNING: CPU: 0 PID: 1676 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4 stress-ng: error: [913] stress-ng-fifo: read timeout! stress-ng: error: [924] stress-ng-fifo: read timeout! [ 104.967194] Modules linked in: [ 104.967206] CPU: 0 PID: 1676 Comm: stress-ng-fstat Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269 stress-ng: error: [926] stress-ng-fifo: read timeout! stress-ng: error: [920] stress-ng-fifo: read timeout! [ 104.967210] Hardware name: Generic DRA74X (Flattened Device Tree) [ 104.967227] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14) [ 104.967239] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4) [ 104.967251] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100) [ 104.967261] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28) [ 104.967271] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4) stress-ng: fail: [828] stress-ng-af-alg: socket failed, errno=9[ 104.967285] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814) 7 (Address family not supported by protocol) [ 104.967294] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0) [ 104.967302] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4) stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETQUOTA failed: errno=38 (Function not implemented) stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETFMT failed: errno=38 (Function not implemented) stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_GETINFO failed: errno=38 (Function not implemented) stress-ng: fail: [1484] stress-ng-quota: quotactl command Q_SYNC failed: errno=38 (Function not implemented) stress-ng: error: [1484] stress-ng-quota: quotactl() failed, not available on this kernel [ 104.967311] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0) [ 104.967320] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c) [ 104.967329] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4) [ 104.967339] [<c015a5d8>] (migrate_disable) from [<c0842338>] (rt_spin_lock+0x18/0x60) [ 104.967349] [<c0842338>] (rt_spin_lock) from [<c0440994>] (lockref_get_not_dead+0xc/0x34) [ 104.967360] [<c0440994>] (lockref_get_not_dead) from [<c0273fb8>] (legitimize_path+0x2c/0x64) [ 104.967368] [<c0273fb8>] (legitimize_path) from [<c02740e4>] (unlazy_walk+0x3c/0x18c) [ 104.967376] [<c02740e4>] (unlazy_walk) from [<c0274258>] (complete_walk+0x24/0x78) [ 104.967384] [<c0274258>] (complete_walk) from [<c02768b8>] (path_lookupat+0xbc/0x1ec) [ 104.967393] [<c02768b8>] (path_lookupat) from [<c0278f98>] (filename_lookup+0x8c/0xe8) [ 104.967404] [<c0278f98>] (filename_lookup) from [<c026e1ac>] (vfs_statx+0x5c/0xb8) [ 104.967414] [<c026e1ac>] (vfs_statx) from [<c026e8ec>] (SyS_stat64+0x28/0x44) [ 104.967424] [<c026e8ec>] (SyS_stat64) from [<c0108480>] (ret_fast_syscall+0x0/0x28) [ 104.967429] ---[ end trace 0000000000000002 ]--- === WARN2 ==== [ 196.437276] ------------[ cut here ]------------ [ 196.437293] WARNING: CPU: 0 PID: 1521 at kernel/rcu/tree_plugin.h:310 rcu_note_context_switch+0x2e0/0x5f4 [ 196.437296] Modules linked in: [ 196.437307] CPU: 0 PID: 1521 Comm: stress-ng-opcod Not tainted 4.14.29-rt25-01568-g2ba67d4-dirty #269 [ 196.437311] Hardware name: Generic DRA74X (Flattened Device Tree) [ 196.437326] [<c011143c>] (unwind_backtrace) from [<c010cb40>] (show_stack+0x10/0x14) [ 196.437338] [<c010cb40>] (show_stack) from [<c0828c10>] (dump_stack+0x98/0xc4) [ 196.437349] [<c0828c10>] (dump_stack) from [<c012eee0>] (__warn+0xe8/0x100) [ 196.437359] [<c012eee0>] (__warn) from [<c012efa8>] (warn_slowpath_null+0x20/0x28) [ 196.437369] [<c012efa8>] (warn_slowpath_null) from [<c01aa470>] (rcu_note_context_switch+0x2e0/0x5f4) [ 196.437381] [<c01aa470>] (rcu_note_context_switch) from [<c083e2fc>] (__schedule+0xdc/0x814) [ 196.437390] [<c083e2fc>] (__schedule) from [<c083ea8c>] (schedule+0x58/0xf0) [ 196.437398] [<c083ea8c>] (schedule) from [<c0840604>] (rt_spin_lock_slowlock_locked+0xbc/0x2f4) [ 196.437407] [<c0840604>] (rt_spin_lock_slowlock_locked) from [<c08412b4>] (__read_rt_lock+0x84/0xc0) [ 196.437416] [<c08412b4>] (__read_rt_lock) from [<c0130918>] (pin_current_cpu+0x40/0x11c) [ 196.437424] [<c0130918>] (pin_current_cpu) from [<c015a5d8>] (migrate_disable+0x78/0x1e4) [ 196.437433] [<c015a5d8>] (migrate_disable) from [<c011a3a4>] (kmap_atomic+0x34/0x174) [ 196.437443] [<c011a3a4>] (kmap_atomic) from [<c0236484>] (alloc_set_pte+0x288/0x320) [ 196.437453] [<c0236484>] (alloc_set_pte) from [<c01ffcb8>] (filemap_map_pages+0x2b0/0x4c8) [ 196.437461] [<c01ffcb8>] (filemap_map_pages) from [<c0236c4c>] (handle_mm_fault+0x690/0xc78) [ 196.437468] [<c0236c4c>] (handle_mm_fault) from [<c0115bd8>] (do_page_fault+0x2bc/0x398) [ 196.437476] [<c0115bd8>] (do_page_fault) from [<c0101474>] (do_PrefetchAbort+0x38/0xa0) [ 196.437484] [<c0101474>] (do_PrefetchAbort) from [<c010decc>] (ret_from_exception+0x0/0x14) [ 196.437488] Exception stack(0xe9943fb0 to 0xe9943ff8) [ 196.437494] 3fa0: 00000000 00000000 00000000 00000000 [ 196.437502] 3fc0: b6f6e468 bee42888 b6f6e400 bee42898 b6f6e8c0 00000000 000003af 000003af [ 196.437508] 3fe0: 00000078 bee42888 b6d9ac91 b6d9ac90 200b0030 ffffffff [ 196.437513] ---[ end trace 0000000000000002 ]--- -- regards, -grygorii ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-03-27 18:01 ` Grygorii Strashko @ 2018-04-20 17:11 ` Sebastian Andrzej Siewior 2018-04-23 16:57 ` Grygorii Strashko 0 siblings, 1 reply; 8+ messages in thread From: Sebastian Andrzej Siewior @ 2018-04-20 17:11 UTC (permalink / raw) To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote: > Hi Sebastian, Hi Grygorii, > I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule() > while holding the RCU lock" [1] on top. Then I run below tests: … > no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings the warning is unrelated to the patch I posted. This should mute it: Subject: [PATCH RT] rtmutex: annotate sleeping lock context The RCU code complains on schedule() within a rcu_readlock() section. The valid scenario on -RT is if a sleeping is held. In order to suppress the warning the mirgrate_disable counter was used to identify the invocation of schedule() due to lock contention. Grygorii Strashko report that during CPU hotplug we might see the warning via rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock() because the counter is not yet set. It is also possible to trigger the warning from cpu_chill() (seen on a kblockd_mod_delayed_work_on() caller). To address this RCU warning I annotate the sleeping lock context. The counter is incremented before migrate_disable() so the warning Grygorii should not trigger anymore. Additionally I use that counter in cpu_chill() to avoid the RCU warning from there. Reported-by: Grygorii Strashko <grygorii.strashko@ti.com> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- include/linux/sched.h | 20 ++++++++++++++++++++ kernel/locking/rtmutex.c | 12 ++++++++++-- kernel/locking/rwlock-rt.c | 18 ++++++++++++++---- kernel/rcu/tree_plugin.h | 8 ++++---- kernel/time/hrtimer.c | 2 ++ 5 files changed, 50 insertions(+), 10 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index cb78e2785f60..e90a15f38bf5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -613,6 +613,9 @@ struct task_struct { int migrate_disable_atomic; # endif #endif +#ifdef CONFIG_PREEMPT_RT_FULL + int sleeping_lock; +#endif #ifdef CONFIG_PREEMPT_RCU int rcu_read_lock_nesting; @@ -1774,6 +1777,23 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } +#ifdef CONFIG_PREEMPT_RT_FULL +static inline void sleeping_lock_inc(void) +{ + current->sleeping_lock++; +} + +static inline void sleeping_lock_dec(void) +{ + current->sleeping_lock--; +} + +#else + +static inline void sleeping_lock_inc(void) { } +static inline void sleeping_lock_dec(void) { } +#endif + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 65cf3f6851eb..761a910b8489 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock) void __lockfunc rt_spin_lock(spinlock_t *lock) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 0, _RET_IP_); rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); @@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock) #ifdef CONFIG_DEBUG_LOCK_ALLOC void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_); rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); @@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock) spin_release(&lock->dep_map, 1, _RET_IP_); rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_spin_unlock); @@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = __rt_mutex_trylock(&lock->lock); - if (ret) + if (ret) { spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_spin_trylock); @@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock) local_bh_disable(); ret = __rt_mutex_trylock(&lock->lock); if (ret) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); } else @@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags) *flags = 0; ret = __rt_mutex_trylock(&lock->lock); if (ret) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); } diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c index aebb7ce25bc6..f2e155b2c4a8 100644 --- a/kernel/locking/rwlock-rt.c +++ b/kernel/locking/rwlock-rt.c @@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = do_read_rt_trylock(rwlock); - if (ret) + if (ret) { rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_read_trylock); @@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = do_write_rt_trylock(rwlock); - if (ret) + if (ret) { rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_write_trylock); void __lockfunc rt_read_lock(rwlock_t *rwlock) { + sleeping_lock_inc(); migrate_disable(); rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_); do_read_rt_lock(rwlock); @@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock); void __lockfunc rt_write_lock(rwlock_t *rwlock) { + sleeping_lock_inc(); migrate_disable(); rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_); do_write_rt_lock(rwlock); @@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock) rwlock_release(&rwlock->dep_map, 1, _RET_IP_); do_read_rt_unlock(rwlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_read_unlock); @@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock) rwlock_release(&rwlock->dep_map, 1, _RET_IP_); do_write_rt_unlock(rwlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_write_unlock); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 3933026838eb..181afc1ab6e2 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -300,13 +300,13 @@ static void rcu_preempt_note_context_switch(bool preempt) struct task_struct *t = current; struct rcu_data *rdp; struct rcu_node *rnp; - int mg_counter = 0; + int sleeping_l = 0; lockdep_assert_irqs_disabled(); -#if defined(CONFIG_PREEMPT_RT_BASE) - mg_counter = t->migrate_disable; +#if defined(CONFIG_PREEMPT_RT_FULL) + sleeping_l = t->sleeping_lock; #endif - WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter); + WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l); if (t->rcu_read_lock_nesting > 0 && !t->rcu_read_unlock_special.b.blocked) { diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c index 43c2f571a842..9e1b766dc35c 100644 --- a/kernel/time/hrtimer.c +++ b/kernel/time/hrtimer.c @@ -1854,7 +1854,9 @@ void cpu_chill(void) chill_time = ktime_set(0, NSEC_PER_MSEC); set_current_state(TASK_UNINTERRUPTIBLE); current->flags |= PF_NOFREEZE; + sleeping_lock_inc(); schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD); + sleeping_lock_dec(); if (!freeze_flag) current->flags &= ~PF_NOFREEZE; } -- 2.17.0 ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-04-20 17:11 ` Sebastian Andrzej Siewior @ 2018-04-23 16:57 ` Grygorii Strashko 2018-04-23 19:10 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 8+ messages in thread From: Grygorii Strashko @ 2018-04-23 16:57 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt On 04/20/2018 12:11 PM, Sebastian Andrzej Siewior wrote: > On 2018-03-27 13:01:07 [-0500], Grygorii Strashko wrote: >> Hi Sebastian, > Hi Grygorii, > >> I've took this RT version and applied "[RT] kernel/time/posix-timer: avoid schedule() >> while holding the RCU lock" [1] on top. Then I run below tests: > … >> no stall or crashes were observed, but I've caught two "rcu_note_context_switch()" warnings > > the warning is unrelated to the patch I posted. This should mute it: > > Subject: [PATCH RT] rtmutex: annotate sleeping lock context > > The RCU code complains on schedule() within a rcu_readlock() section. > The valid scenario on -RT is if a sleeping is held. In order to suppress > the warning the mirgrate_disable counter was used to identify the > invocation of schedule() due to lock contention. > > Grygorii Strashko report that during CPU hotplug we might see the > warning via > rt_spin_lock() -> migrate_disable() -> pin_current_cpu() -> __read_rt_lock() > > because the counter is not yet set. > It is also possible to trigger the warning from cpu_chill() > (seen on a kblockd_mod_delayed_work_on() caller). > > To address this RCU warning I annotate the sleeping lock context. The > counter is incremented before migrate_disable() so the warning Grygorii > should not trigger anymore. Additionally I use that counter in > cpu_chill() to avoid the RCU warning from there. Sry, but I can't apply it. What's you base? > > Reported-by: Grygorii Strashko <grygorii.strashko@ti.com> > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > --- -- regards, -grygorii ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-04-23 16:57 ` Grygorii Strashko @ 2018-04-23 19:10 ` Sebastian Andrzej Siewior 2018-05-01 15:04 ` Grygorii Strashko 0 siblings, 1 reply; 8+ messages in thread From: Sebastian Andrzej Siewior @ 2018-04-23 19:10 UTC (permalink / raw) To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote: > Sry, but I can't apply it. What's you base? please try this patch and latest v4.14-RT diff --git a/include/linux/sched.h b/include/linux/sched.h index e305a8f8cd7d..0322503084a5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -600,6 +600,9 @@ struct task_struct { int migrate_disable_atomic; # endif #endif +#ifdef CONFIG_PREEMPT_RT_FULL + int sleeping_lock; +#endif #ifdef CONFIG_PREEMPT_RCU int rcu_read_lock_nesting; @@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } +#ifdef CONFIG_PREEMPT_RT_FULL +static inline void sleeping_lock_inc(void) +{ + current->sleeping_lock++; +} + +static inline void sleeping_lock_dec(void) +{ + current->sleeping_lock--; +} + +#else + +static inline void sleeping_lock_inc(void) { } +static inline void sleeping_lock_dec(void) { } +#endif + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c index 0cb716ba3be0..bac3fb580af6 100644 --- a/kernel/locking/rtmutex.c +++ b/kernel/locking/rtmutex.c @@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock) void __lockfunc rt_spin_lock(spinlock_t *lock) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 0, _RET_IP_); rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); @@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock) #ifdef CONFIG_DEBUG_LOCK_ALLOC void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_); rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); @@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock) spin_release(&lock->dep_map, 1, _RET_IP_); rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_spin_unlock); @@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = __rt_mutex_trylock(&lock->lock); - if (ret) + if (ret) { spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_spin_trylock); @@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock) local_bh_disable(); ret = __rt_mutex_trylock(&lock->lock); if (ret) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); } else @@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags) *flags = 0; ret = __rt_mutex_trylock(&lock->lock); if (ret) { + sleeping_lock_inc(); migrate_disable(); spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); } diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c index aebb7ce25bc6..f2e155b2c4a8 100644 --- a/kernel/locking/rwlock-rt.c +++ b/kernel/locking/rwlock-rt.c @@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = do_read_rt_trylock(rwlock); - if (ret) + if (ret) { rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_read_trylock); @@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock) { int ret; + sleeping_lock_inc(); migrate_disable(); ret = do_write_rt_trylock(rwlock); - if (ret) + if (ret) { rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_); - else + } else { migrate_enable(); + sleeping_lock_dec(); + } return ret; } EXPORT_SYMBOL(rt_write_trylock); void __lockfunc rt_read_lock(rwlock_t *rwlock) { + sleeping_lock_inc(); migrate_disable(); rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_); do_read_rt_lock(rwlock); @@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock); void __lockfunc rt_write_lock(rwlock_t *rwlock) { + sleeping_lock_inc(); migrate_disable(); rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_); do_write_rt_lock(rwlock); @@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock) rwlock_release(&rwlock->dep_map, 1, _RET_IP_); do_read_rt_unlock(rwlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_read_unlock); @@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock) rwlock_release(&rwlock->dep_map, 1, _RET_IP_); do_write_rt_unlock(rwlock); migrate_enable(); + sleeping_lock_dec(); } EXPORT_SYMBOL(rt_write_unlock); diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h index 3315ebad932f..c6098f60e641 100644 --- a/kernel/rcu/tree_plugin.h +++ b/kernel/rcu/tree_plugin.h @@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt) struct task_struct *t = current; struct rcu_data *rdp; struct rcu_node *rnp; - int mg_counter = 0; + int sleeping_l = 0; RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n"); -#if defined(CONFIG_PREEMPT_RT_BASE) - mg_counter = t->migrate_disable; +#if defined(CONFIG_PREEMPT_RT_FULL) + sleeping_l = t->sleeping_lock; #endif - WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter); + WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l); if (t->rcu_read_lock_nesting > 0 && !t->rcu_read_unlock_special.b.blocked) { diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c index ce2c2d04cbaa..b59e009087a9 100644 --- a/kernel/time/hrtimer.c +++ b/kernel/time/hrtimer.c @@ -1870,7 +1870,9 @@ void cpu_chill(void) chill_time = ktime_set(0, NSEC_PER_MSEC); set_current_state(TASK_UNINTERRUPTIBLE); current->flags |= PF_NOFREEZE; + sleeping_lock_inc(); schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD); + sleeping_lock_dec(); if (!freeze_flag) current->flags &= ~PF_NOFREEZE; } Sebastian ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-04-23 19:10 ` Sebastian Andrzej Siewior @ 2018-05-01 15:04 ` Grygorii Strashko 2018-05-03 15:24 ` Sebastian Andrzej Siewior 0 siblings, 1 reply; 8+ messages in thread From: Grygorii Strashko @ 2018-05-01 15:04 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt Hi Sebastian, On 04/23/2018 02:10 PM, Sebastian Andrzej Siewior wrote: > On 2018-04-23 11:57:39 [-0500], Grygorii Strashko wrote: >> Sry, but I can't apply it. What's you base? > > please try this patch and latest v4.14-RT I've tried this (with v4.14.34-rt27) and I do not see rcu_note_context_switch() any more. Sry, it took some time as i found some instability - test "stress-ng --class os --all 0 -t 5m " not always finished :( So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as pure rt-devel. Still not sure if this is some sort of regression or not. > > diff --git a/include/linux/sched.h b/include/linux/sched.h > index e305a8f8cd7d..0322503084a5 100644 > --- a/include/linux/sched.h > +++ b/include/linux/sched.h > @@ -600,6 +600,9 @@ struct task_struct { > int migrate_disable_atomic; > # endif > #endif > +#ifdef CONFIG_PREEMPT_RT_FULL > + int sleeping_lock; > +#endif > > #ifdef CONFIG_PREEMPT_RCU > int rcu_read_lock_nesting; > @@ -1757,6 +1760,23 @@ static __always_inline bool need_resched(void) > return unlikely(tif_need_resched()); > } > > +#ifdef CONFIG_PREEMPT_RT_FULL > +static inline void sleeping_lock_inc(void) > +{ > + current->sleeping_lock++; > +} > + > +static inline void sleeping_lock_dec(void) > +{ > + current->sleeping_lock--; > +} > + > +#else > + > +static inline void sleeping_lock_inc(void) { } > +static inline void sleeping_lock_dec(void) { } > +#endif > + > /* > * Wrappers for p->thread_info->cpu access. No-op on UP. > */ > diff --git a/kernel/locking/rtmutex.c b/kernel/locking/rtmutex.c > index 0cb716ba3be0..bac3fb580af6 100644 > --- a/kernel/locking/rtmutex.c > +++ b/kernel/locking/rtmutex.c > @@ -1141,6 +1141,7 @@ void __sched rt_spin_lock_slowunlock(struct rt_mutex *lock) > > void __lockfunc rt_spin_lock(spinlock_t *lock) > { > + sleeping_lock_inc(); > migrate_disable(); > spin_acquire(&lock->dep_map, 0, 0, _RET_IP_); > rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); > @@ -1155,6 +1156,7 @@ void __lockfunc __rt_spin_lock(struct rt_mutex *lock) > #ifdef CONFIG_DEBUG_LOCK_ALLOC > void __lockfunc rt_spin_lock_nested(spinlock_t *lock, int subclass) > { > + sleeping_lock_inc(); > migrate_disable(); > spin_acquire(&lock->dep_map, subclass, 0, _RET_IP_); > rt_spin_lock_fastlock(&lock->lock, rt_spin_lock_slowlock); > @@ -1168,6 +1170,7 @@ void __lockfunc rt_spin_unlock(spinlock_t *lock) > spin_release(&lock->dep_map, 1, _RET_IP_); > rt_spin_lock_fastunlock(&lock->lock, rt_spin_lock_slowunlock); > migrate_enable(); > + sleeping_lock_dec(); > } > EXPORT_SYMBOL(rt_spin_unlock); > > @@ -1193,12 +1196,15 @@ int __lockfunc rt_spin_trylock(spinlock_t *lock) > { > int ret; > > + sleeping_lock_inc(); > migrate_disable(); > ret = __rt_mutex_trylock(&lock->lock); > - if (ret) > + if (ret) { > spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); > - else > + } else { > migrate_enable(); > + sleeping_lock_dec(); > + } > return ret; > } > EXPORT_SYMBOL(rt_spin_trylock); > @@ -1210,6 +1216,7 @@ int __lockfunc rt_spin_trylock_bh(spinlock_t *lock) > local_bh_disable(); > ret = __rt_mutex_trylock(&lock->lock); > if (ret) { > + sleeping_lock_inc(); > migrate_disable(); > spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); > } else > @@ -1225,6 +1232,7 @@ int __lockfunc rt_spin_trylock_irqsave(spinlock_t *lock, unsigned long *flags) > *flags = 0; > ret = __rt_mutex_trylock(&lock->lock); > if (ret) { > + sleeping_lock_inc(); > migrate_disable(); > spin_acquire(&lock->dep_map, 0, 1, _RET_IP_); > } > diff --git a/kernel/locking/rwlock-rt.c b/kernel/locking/rwlock-rt.c > index aebb7ce25bc6..f2e155b2c4a8 100644 > --- a/kernel/locking/rwlock-rt.c > +++ b/kernel/locking/rwlock-rt.c > @@ -305,12 +305,15 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock) > { > int ret; > > + sleeping_lock_inc(); > migrate_disable(); > ret = do_read_rt_trylock(rwlock); > - if (ret) > + if (ret) { > rwlock_acquire_read(&rwlock->dep_map, 0, 1, _RET_IP_); > - else > + } else { > migrate_enable(); > + sleeping_lock_dec(); > + } > return ret; > } > EXPORT_SYMBOL(rt_read_trylock); > @@ -319,18 +322,22 @@ int __lockfunc rt_write_trylock(rwlock_t *rwlock) > { > int ret; > > + sleeping_lock_inc(); > migrate_disable(); > ret = do_write_rt_trylock(rwlock); > - if (ret) > + if (ret) { > rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_); > - else > + } else { > migrate_enable(); > + sleeping_lock_dec(); > + } > return ret; > } > EXPORT_SYMBOL(rt_write_trylock); > > void __lockfunc rt_read_lock(rwlock_t *rwlock) > { > + sleeping_lock_inc(); > migrate_disable(); > rwlock_acquire_read(&rwlock->dep_map, 0, 0, _RET_IP_); > do_read_rt_lock(rwlock); > @@ -339,6 +346,7 @@ EXPORT_SYMBOL(rt_read_lock); > > void __lockfunc rt_write_lock(rwlock_t *rwlock) > { > + sleeping_lock_inc(); > migrate_disable(); > rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_); > do_write_rt_lock(rwlock); > @@ -350,6 +358,7 @@ void __lockfunc rt_read_unlock(rwlock_t *rwlock) > rwlock_release(&rwlock->dep_map, 1, _RET_IP_); > do_read_rt_unlock(rwlock); > migrate_enable(); > + sleeping_lock_dec(); > } > EXPORT_SYMBOL(rt_read_unlock); > > @@ -358,6 +367,7 @@ void __lockfunc rt_write_unlock(rwlock_t *rwlock) > rwlock_release(&rwlock->dep_map, 1, _RET_IP_); > do_write_rt_unlock(rwlock); > migrate_enable(); > + sleeping_lock_dec(); > } > EXPORT_SYMBOL(rt_write_unlock); > > diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h > index 3315ebad932f..c6098f60e641 100644 > --- a/kernel/rcu/tree_plugin.h > +++ b/kernel/rcu/tree_plugin.h > @@ -301,13 +301,13 @@ static void rcu_preempt_note_context_switch(bool preempt) > struct task_struct *t = current; > struct rcu_data *rdp; > struct rcu_node *rnp; > - int mg_counter = 0; > + int sleeping_l = 0; > > RCU_LOCKDEP_WARN(!irqs_disabled(), "rcu_preempt_note_context_switch() invoked with interrupts enabled!!!\n"); > -#if defined(CONFIG_PREEMPT_RT_BASE) > - mg_counter = t->migrate_disable; > +#if defined(CONFIG_PREEMPT_RT_FULL) > + sleeping_l = t->sleeping_lock; > #endif > - WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !mg_counter); > + WARN_ON_ONCE(!preempt && t->rcu_read_lock_nesting > 0 && !sleeping_l); > if (t->rcu_read_lock_nesting > 0 && > !t->rcu_read_unlock_special.b.blocked) { > > diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c > index ce2c2d04cbaa..b59e009087a9 100644 > --- a/kernel/time/hrtimer.c > +++ b/kernel/time/hrtimer.c > @@ -1870,7 +1870,9 @@ void cpu_chill(void) > chill_time = ktime_set(0, NSEC_PER_MSEC); > set_current_state(TASK_UNINTERRUPTIBLE); > current->flags |= PF_NOFREEZE; > + sleeping_lock_inc(); > schedule_hrtimeout(&chill_time, HRTIMER_MODE_REL_HARD); > + sleeping_lock_dec(); > if (!freeze_flag) > current->flags &= ~PF_NOFREEZE; > } > > Sebastian > -- regards, -grygorii ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-05-01 15:04 ` Grygorii Strashko @ 2018-05-03 15:24 ` Sebastian Andrzej Siewior 2018-05-03 18:26 ` Grygorii Strashko 0 siblings, 1 reply; 8+ messages in thread From: Sebastian Andrzej Siewior @ 2018-05-03 15:24 UTC (permalink / raw) To: Grygorii Strashko; +Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote: > Hi Sebastian, Hi Grygorii, > Sry, it took some time as i found some instability - > test "stress-ng --class os --all 0 -t 5m " not always finished :( > So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as > pure rt-devel. Still not sure if this is some sort of regression or not. is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does not? This patch just silences the warning so it should not break anything… Sebastian ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [ANNOUNCE] v4.14.29-rt25 2018-05-03 15:24 ` Sebastian Andrzej Siewior @ 2018-05-03 18:26 ` Grygorii Strashko 0 siblings, 0 replies; 8+ messages in thread From: Grygorii Strashko @ 2018-05-03 18:26 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Thomas Gleixner, LKML, linux-rt-users, Steven Rostedt On 05/03/2018 10:24 AM, Sebastian Andrzej Siewior wrote: > On 2018-05-01 10:04:13 [-0500], Grygorii Strashko wrote: >> Hi Sebastian, > Hi Grygorii, > >> Sry, it took some time as i found some instability - >> test "stress-ng --class os --all 0 -t 5m " not always finished :( >> So, I've tried to rollback to v4.14.29-rt25 and use as TI RT kernel as >> pure rt-devel. Still not sure if this is some sort of regression or not. > > is this new? So v4.14.29-rt25 works and v4.14.29-rt25 with this does > not? This patch just silences the warning so it should not break > anything… Yes. It seems not related. -- regards, -grygorii ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-05-03 18:27 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-25 10:12 [ANNOUNCE] v4.14.29-rt25 Sebastian Andrzej Siewior 2018-03-27 18:01 ` Grygorii Strashko 2018-04-20 17:11 ` Sebastian Andrzej Siewior 2018-04-23 16:57 ` Grygorii Strashko 2018-04-23 19:10 ` Sebastian Andrzej Siewior 2018-05-01 15:04 ` Grygorii Strashko 2018-05-03 15:24 ` Sebastian Andrzej Siewior 2018-05-03 18:26 ` Grygorii Strashko
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).