[PATCH] cpu/hotplug: Reset node state after operation

[PATCH] cpu/hotplug: Reset node state after operation

[Thomas Gleixner]

The recent rework of the cpu hotplug internals changed the usage of the per
cpu state->node field, but missed to clean it up after usage.

So subsequent hotplug operations use the stale pointer from a previous
operation and hand it into the callback functions. The callbacks then
dereference a pointer which either belongs to a different facility or
points to freed and potentially reused memory. In either case data
corruption and crashes are the obvious consequence.

Reset the node and the last pointers in the per cpu state to NULL after the
operation which set them has completed.

Fixes: 96abb968549c ("smp/hotplug: Allow external multi-instance rollback")
Reported-by: Tvrtko Ursulin <tursulin@ursulin.net>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
---
 kernel/cpu.c |    5 +++++
 1 file changed, 5 insertions(+)

--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -632,6 +632,11 @@ cpuhp_invoke_ap_callback(int cpu, enum c
 		__cpuhp_kick_ap(st);
 	}
 
+	/*
+	 * Clean up the leftovers so the next hotplug operation wont use stale
+	 * data.
+	 */
+	st->node = st->last = NULL;
 	return ret;
 }
 

[tip:smp/urgent] cpu/hotplug: Reset node state after operation

[tip-bot for Thomas Gleixner]

Commit-ID:  1f7c70d6b2bc5de301f30456621e1161fddf4242
Gitweb:     https://git.kernel.org/tip/1f7c70d6b2bc5de301f30456621e1161fddf4242
Author:     Thomas Gleixner <tglx@linutronix.de>
AuthorDate: Sat, 21 Oct 2017 16:06:52 +0200
Committer:  Thomas Gleixner <tglx@linutronix.de>
CommitDate: Sat, 21 Oct 2017 16:11:30 +0200

cpu/hotplug: Reset node state after operation

The recent rework of the cpu hotplug internals changed the usage of the per
cpu state->node field, but missed to clean it up after usage.

So subsequent hotplug operations use the stale pointer from a previous
operation and hand it into the callback functions. The callbacks then
dereference a pointer which either belongs to a different facility or
points to freed and potentially reused memory. In either case data
corruption and crashes are the obvious consequence.

Reset the node and the last pointers in the per cpu state to NULL after the
operation which set them has completed.

Fixes: 96abb968549c ("smp/hotplug: Allow external multi-instance rollback")
Reported-by: Tvrtko Ursulin <tursulin@ursulin.net>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Link: https://lkml.kernel.org/r/alpine.DEB.2.20.1710211606130.3213@nanos

---
 kernel/cpu.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/kernel/cpu.c b/kernel/cpu.c
index d851df2..04892a8 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -632,6 +632,11 @@ cpuhp_invoke_ap_callback(int cpu, enum cpuhp_state state, bool bringup,
 		__cpuhp_kick_ap(st);
 	}
 
+	/*
+	 * Clean up the leftovers so the next hotplug operation wont use stale
+	 * data.
+	 */
+	st->node = st->last = NULL;
 	return ret;
 }
 

Re: [PATCH] cpu/hotplug: Reset node state after operation

[Paul E. McKenney]

On Sat, Oct 21, 2017 at 04:06:52PM +0200, Thomas Gleixner wrote:
> The recent rework of the cpu hotplug internals changed the usage of the per
> cpu state->node field, but missed to clean it up after usage.
> 
> So subsequent hotplug operations use the stale pointer from a previous
> operation and hand it into the callback functions. The callbacks then
> dereference a pointer which either belongs to a different facility or
> points to freed and potentially reused memory. In either case data
> corruption and crashes are the obvious consequence.
> 
> Reset the node and the last pointers in the per cpu state to NULL after the
> operation which set them has completed.
> 
> Fixes: 96abb968549c ("smp/hotplug: Allow external multi-instance rollback")
> Reported-by: Tvrtko Ursulin <tursulin@ursulin.net>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

This does fine with rcutorture, though it does not fix the lost
(or egregiously delayed) timer problem that is still eluding me.
Nevertheless, for whatever it is worth:

Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

> ---
>  kernel/cpu.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -632,6 +632,11 @@ cpuhp_invoke_ap_callback(int cpu, enum c
>  		__cpuhp_kick_ap(st);
>  	}
> 
> +	/*
> +	 * Clean up the leftovers so the next hotplug operation wont use stale
> +	 * data.
> +	 */
> +	st->node = st->last = NULL;
>  	return ret;
>  }
> 
> 

Re: [PATCH] cpu/hotplug: Reset node state after operation

[Peter Zijlstra]

On Sat, Oct 21, 2017 at 04:06:52PM +0200, Thomas Gleixner wrote:
> The recent rework of the cpu hotplug internals changed the usage of the per
> cpu state->node field, but missed to clean it up after usage.
> 
> So subsequent hotplug operations use the stale pointer from a previous
> operation and hand it into the callback functions. The callbacks then
> dereference a pointer which either belongs to a different facility or
> points to freed and potentially reused memory. In either case data
> corruption and crashes are the obvious consequence.
> 
> Reset the node and the last pointers in the per cpu state to NULL after the
> operation which set them has completed.
> 
> Fixes: 96abb968549c ("smp/hotplug: Allow external multi-instance rollback")
> Reported-by: Tvrtko Ursulin <tursulin@ursulin.net>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

Thanks for digging into that during the conf :-) And sorry for wrecking
that.

Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org>

> ---
>  kernel/cpu.c |    5 +++++
>  1 file changed, 5 insertions(+)
> 
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -632,6 +632,11 @@ cpuhp_invoke_ap_callback(int cpu, enum c
>  		__cpuhp_kick_ap(st);
>  	}
>  
> +	/*
> +	 * Clean up the leftovers so the next hotplug operation wont use stale
> +	 * data.
> +	 */
> +	st->node = st->last = NULL;
>  	return ret;
>  }
>  

Re: [PATCH] cpu/hotplug: Reset node state after operation

[Tvrtko Ursulin]

On 21/10/17 15:06, Thomas Gleixner wrote:
> The recent rework of the cpu hotplug internals changed the usage of the per
> cpu state->node field, but missed to clean it up after usage.
> 
> So subsequent hotplug operations use the stale pointer from a previous
> operation and hand it into the callback functions. The callbacks then
> dereference a pointer which either belongs to a different facility or
> points to freed and potentially reused memory. In either case data
> corruption and crashes are the obvious consequence.
> 
> Reset the node and the last pointers in the per cpu state to NULL after the
> operation which set them has completed.
> 
> Fixes: 96abb968549c ("smp/hotplug: Allow external multi-instance rollback")
> Reported-by: Tvrtko Ursulin <tursulin@ursulin.net>
> Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
> ---
>   kernel/cpu.c |    5 +++++
>   1 file changed, 5 insertions(+)
> 
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -632,6 +632,11 @@ cpuhp_invoke_ap_callback(int cpu, enum c
>   		__cpuhp_kick_ap(st);
>   	}
>   
> +	/*
> +	 * Clean up the leftovers so the next hotplug operation wont use stale
> +	 * data.
> +	 */
> +	st->node = st->last = NULL;
>   	return ret;
>   }
>   
> 

Our CI farm is happy with this fix, so:

Tested-by: Tvrtko Ursulin <tvrtko.ursulin@intel.com>

Thanks for looking into is so quickly!

Regards,

Tvrtko