live-patching.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2] livepatch: Fix idle cpu's tasks transition
@ 2021-09-15 14:18 Vasily Gorbik
  2021-09-20  8:27 ` Petr Mladek
  2021-09-21 17:43 ` Peter Zijlstra
  0 siblings, 2 replies; 4+ messages in thread
From: Vasily Gorbik @ 2021-09-15 14:18 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Josh Poimboeuf, Jiri Kosina,
	Miroslav Benes, Petr Mladek
  Cc: Joe Lawrence, Frederic Weisbecker, Thomas Gleixner,
	Heiko Carstens, Sven Schnelle, Sumanth Korikkar, live-patching,
	linux-kernel

On an idle system with large amount of cpus it might happen that
klp_update_patch_state() is not reached in do_idle() for a long periods
of time. With debug messages enabled log is filled with:
[  499.442643] livepatch: klp_try_switch_task: swapper/63:0 is running

without any signs of progress. Ending up with "failed to complete
transition".

On s390 LPAR with 128 cpus not a single transition is able to complete
and livepatch kselftests fail. Tests on idling x86 kvm instance with 128
cpus demonstrate similar symptoms with and without CONFIG_NO_HZ.

To deal with that, since runqueue is already locked in
klp_try_switch_task() identify idling cpus and trigger rescheduling
potentially waking them up and making sure idle tasks break out of
do_idle() inner loop and reach klp_update_patch_state(). This helps to
speed up transition time while avoiding unnecessary extra system load.

Reviewed-by: Petr Mladek <pmladek@suse.com>
Acked-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
---
Ingo/Peter, as Josh mentioned, could you please ack if you are ok with
livepatch calling this private scheduler interface?

Changes since v1:
- added comments suggested by Petr
  lkml.kernel.org/r/patch.git-a4aad6b1540d.your-ad-here.call-01631177886-ext-3083@work.hours

Previous discussion and RFC PATCH:
  lkml.kernel.org/r/patch.git-b76842ceb035.your-ad-here.call-01625661932-ext-1304@work.hours

 kernel/livepatch/transition.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
index 291b857a6e20..2846a879f2dc 100644
--- a/kernel/livepatch/transition.c
+++ b/kernel/livepatch/transition.c
@@ -278,6 +278,8 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
  * Try to safely switch a task to the target patch state.  If it's currently
  * running, or it's sleeping on a to-be-patched or to-be-unpatched function, or
  * if the stack is unreliable, return false.
+ *
+ * Idle tasks are switched in the main loop when running.
  */
 static bool klp_try_switch_task(struct task_struct *task)
 {
@@ -308,6 +310,12 @@ static bool klp_try_switch_task(struct task_struct *task)
 	rq = task_rq_lock(task, &flags);
 
 	if (task_running(rq, task) && task != current) {
+		/*
+		 * Idle task might stay running for a long time. Switch them
+		 * in the main loop.
+		 */
+		if (is_idle_task(task))
+			resched_curr(rq);
 		snprintf(err_buf, STACK_ERR_BUF_SIZE,
 			 "%s: %s:%d is running\n", __func__, task->comm,
 			 task->pid);
-- 
2.25.4

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] livepatch: Fix idle cpu's tasks transition
  2021-09-15 14:18 [PATCH v2] livepatch: Fix idle cpu's tasks transition Vasily Gorbik
@ 2021-09-20  8:27 ` Petr Mladek
  2021-09-21 17:43 ` Peter Zijlstra
  1 sibling, 0 replies; 4+ messages in thread
From: Petr Mladek @ 2021-09-20  8:27 UTC (permalink / raw)
  To: Ingo Molnar, Peter Zijlstra, Josh Poimboeuf
  Cc: Vasily Gorbik, Jiri Kosina, Miroslav Benes, Joe Lawrence,
	Frederic Weisbecker, Thomas Gleixner, Heiko Carstens,
	Sven Schnelle, Sumanth Korikkar, live-patching, linux-kernel

On Wed 2021-09-15 16:18:01, Vasily Gorbik wrote:
> On an idle system with large amount of cpus it might happen that
> klp_update_patch_state() is not reached in do_idle() for a long periods
> of time. With debug messages enabled log is filled with:
> [  499.442643] livepatch: klp_try_switch_task: swapper/63:0 is running
> 
> without any signs of progress. Ending up with "failed to complete
> transition".
> 
> On s390 LPAR with 128 cpus not a single transition is able to complete
> and livepatch kselftests fail. Tests on idling x86 kvm instance with 128
> cpus demonstrate similar symptoms with and without CONFIG_NO_HZ.
> 
> To deal with that, since runqueue is already locked in
> klp_try_switch_task() identify idling cpus and trigger rescheduling
> potentially waking them up and making sure idle tasks break out of
> do_idle() inner loop and reach klp_update_patch_state(). This helps to
> speed up transition time while avoiding unnecessary extra system load.
> 
> Reviewed-by: Petr Mladek <pmladek@suse.com>
> Acked-by: Miroslav Benes <mbenes@suse.cz>
> Signed-off-by: Vasily Gorbik <gor@linux.ibm.com>
> ---
> Ingo/Peter, as Josh mentioned, could you please ack if you are ok with
> livepatch calling this private scheduler interface?

Ingo, Peter, Josh, could anyone please ack that it is acceptable
to call resched_curr(rq) from the livepatch code?

Or is there a better way to make an idle task go through
the main cycle?

Best Regards,
Petr

> 
> Changes since v1:
> - added comments suggested by Petr
>   lkml.kernel.org/r/patch.git-a4aad6b1540d.your-ad-here.call-01631177886-ext-3083@work.hours
> 
> Previous discussion and RFC PATCH:
>   lkml.kernel.org/r/patch.git-b76842ceb035.your-ad-here.call-01625661932-ext-1304@work.hours
> 
>  kernel/livepatch/transition.c | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> index 291b857a6e20..2846a879f2dc 100644
> --- a/kernel/livepatch/transition.c
> +++ b/kernel/livepatch/transition.c
> @@ -278,6 +278,8 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
>   * Try to safely switch a task to the target patch state.  If it's currently
>   * running, or it's sleeping on a to-be-patched or to-be-unpatched function, or
>   * if the stack is unreliable, return false.
> + *
> + * Idle tasks are switched in the main loop when running.
>   */
>  static bool klp_try_switch_task(struct task_struct *task)
>  {
> @@ -308,6 +310,12 @@ static bool klp_try_switch_task(struct task_struct *task)
>  	rq = task_rq_lock(task, &flags);
>  
>  	if (task_running(rq, task) && task != current) {
> +		/*
> +		 * Idle task might stay running for a long time. Switch them
> +		 * in the main loop.
> +		 */
> +		if (is_idle_task(task))
> +			resched_curr(rq);
>  		snprintf(err_buf, STACK_ERR_BUF_SIZE,
>  			 "%s: %s:%d is running\n", __func__, task->comm,
>  			 task->pid);
> -- 
> 2.25.4

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] livepatch: Fix idle cpu's tasks transition
  2021-09-15 14:18 [PATCH v2] livepatch: Fix idle cpu's tasks transition Vasily Gorbik
  2021-09-20  8:27 ` Petr Mladek
@ 2021-09-21 17:43 ` Peter Zijlstra
  2021-09-21 21:17   ` Peter Zijlstra
  1 sibling, 1 reply; 4+ messages in thread
From: Peter Zijlstra @ 2021-09-21 17:43 UTC (permalink / raw)
  To: Vasily Gorbik
  Cc: Ingo Molnar, Josh Poimboeuf, Jiri Kosina, Miroslav Benes,
	Petr Mladek, Joe Lawrence, Frederic Weisbecker, Thomas Gleixner,
	Heiko Carstens, Sven Schnelle, Sumanth Korikkar, live-patching,
	linux-kernel

On Wed, Sep 15, 2021 at 04:18:01PM +0200, Vasily Gorbik wrote:

> diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> index 291b857a6e20..2846a879f2dc 100644
> --- a/kernel/livepatch/transition.c
> +++ b/kernel/livepatch/transition.c
> @@ -278,6 +278,8 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
>   * Try to safely switch a task to the target patch state.  If it's currently
>   * running, or it's sleeping on a to-be-patched or to-be-unpatched function, or
>   * if the stack is unreliable, return false.
> + *
> + * Idle tasks are switched in the main loop when running.
>   */
>  static bool klp_try_switch_task(struct task_struct *task)
>  {
> @@ -308,6 +310,12 @@ static bool klp_try_switch_task(struct task_struct *task)
>  	rq = task_rq_lock(task, &flags);
>  
>  	if (task_running(rq, task) && task != current) {
> +		/*
> +		 * Idle task might stay running for a long time. Switch them
> +		 * in the main loop.
> +		 */
> +		if (is_idle_task(task))
> +			resched_curr(rq);
>  		snprintf(err_buf, STACK_ERR_BUF_SIZE,
>  			 "%s: %s:%d is running\n", __func__, task->comm,
>  			 task->pid);

So 'recently' we grew try_invoke_on_locked_down_task() (yes, that's a
crap name), and I'm thinking this code is very similar to that. So
perhaps we re-use that.

Perhaps have func() > 0 imply resched.

I'll have a play...

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [PATCH v2] livepatch: Fix idle cpu's tasks transition
  2021-09-21 17:43 ` Peter Zijlstra
@ 2021-09-21 21:17   ` Peter Zijlstra
  0 siblings, 0 replies; 4+ messages in thread
From: Peter Zijlstra @ 2021-09-21 21:17 UTC (permalink / raw)
  To: Vasily Gorbik
  Cc: Ingo Molnar, Josh Poimboeuf, Jiri Kosina, Miroslav Benes,
	Petr Mladek, Joe Lawrence, Frederic Weisbecker, Thomas Gleixner,
	Heiko Carstens, Sven Schnelle, Sumanth Korikkar, live-patching,
	linux-kernel

On Tue, Sep 21, 2021 at 07:43:16PM +0200, Peter Zijlstra wrote:
> On Wed, Sep 15, 2021 at 04:18:01PM +0200, Vasily Gorbik wrote:
> 
> > diff --git a/kernel/livepatch/transition.c b/kernel/livepatch/transition.c
> > index 291b857a6e20..2846a879f2dc 100644
> > --- a/kernel/livepatch/transition.c
> > +++ b/kernel/livepatch/transition.c
> > @@ -278,6 +278,8 @@ static int klp_check_stack(struct task_struct *task, char *err_buf)
> >   * Try to safely switch a task to the target patch state.  If it's currently
> >   * running, or it's sleeping on a to-be-patched or to-be-unpatched function, or
> >   * if the stack is unreliable, return false.
> > + *
> > + * Idle tasks are switched in the main loop when running.
> >   */
> >  static bool klp_try_switch_task(struct task_struct *task)
> >  {
> > @@ -308,6 +310,12 @@ static bool klp_try_switch_task(struct task_struct *task)
> >  	rq = task_rq_lock(task, &flags);
> >  
> >  	if (task_running(rq, task) && task != current) {
> > +		/*
> > +		 * Idle task might stay running for a long time. Switch them
> > +		 * in the main loop.
> > +		 */
> > +		if (is_idle_task(task))
> > +			resched_curr(rq);
> >  		snprintf(err_buf, STACK_ERR_BUF_SIZE,
> >  			 "%s: %s:%d is running\n", __func__, task->comm,
> >  			 task->pid);
> 
> So 'recently' we grew try_invoke_on_locked_down_task() (yes, that's a
> crap name), and I'm thinking this code is very similar to that. So
> perhaps we re-use that.
> 
> Perhaps have func() > 0 imply resched.
> 
> I'll have a play...

https://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git/log/?h=sched/livepatch

I'll write proper changelogs and post tomorrow,... (and fix robot fail,
if any).

(also, notes to later self:

 - task_try_func() might need to re-check on_rq after acquiring rq_lock
 - klp_send_signal() might want wake_up_process()/TASK_NORMAL due to TASK_IDLE
)

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-09-21 21:17 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-15 14:18 [PATCH v2] livepatch: Fix idle cpu's tasks transition Vasily Gorbik
2021-09-20  8:27 ` Petr Mladek
2021-09-21 17:43 ` Peter Zijlstra
2021-09-21 21:17   ` Peter Zijlstra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).