[PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[K. Y. Srinivasan]

From: Vitaly Kuznetsov <vkuznets@redhat.com>

Loaded Hyper-V module will use these functions to disable CPU
hotplug under certain circumstances. Convert cpu_hotplug_disabled
to a counter (protected by cpu_add_remove_lock) to support e.g.
disable -> disable -> enable call sequences.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
---
 Documentation/power/suspend-and-cpuhotplug.txt |    6 +++---
 kernel/cpu.c                                   |   13 ++++++++-----
 2 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/Documentation/power/suspend-and-cpuhotplug.txt b/Documentation/power/suspend-and-cpuhotplug.txt
index 2850df3..2fc9095 100644
--- a/Documentation/power/suspend-and-cpuhotplug.txt
+++ b/Documentation/power/suspend-and-cpuhotplug.txt
@@ -72,7 +72,7 @@ More details follow:
                                         |
                                         v
                            Disable regular cpu hotplug
-                        by setting cpu_hotplug_disabled=1
+                        by increasing cpu_hotplug_disabled
                                         |
                                         v
                             Release cpu_add_remove_lock
@@ -89,7 +89,7 @@ Resuming back is likewise, with the counterparts being (in the order of
 execution during resume):
 * enable_nonboot_cpus() which involves:
    |  Acquire cpu_add_remove_lock
-   |  Reset cpu_hotplug_disabled to 0, thereby enabling regular cpu hotplug
+   |  Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
    |  Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
    |  Release cpu_add_remove_lock
    v
@@ -120,7 +120,7 @@ after the entire cycle is complete (i.e., suspend + resume).
                            Acquire cpu_add_remove_lock
                                         |
                                         v
-                          If cpu_hotplug_disabled is 1
+                          If cpu_hotplug_disabled > 0
                                 return gracefully
                                         |
                                         |
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 94bbe46..8f35ee6 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
 void cpu_hotplug_disable(void)
 {
 	cpu_maps_update_begin();
-	cpu_hotplug_disabled = 1;
+	cpu_hotplug_disabled++;
 	cpu_maps_update_done();
 }
+EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
 
 void cpu_hotplug_enable(void)
 {
 	cpu_maps_update_begin();
-	cpu_hotplug_disabled = 0;
+	if (cpu_hotplug_disabled)
+		cpu_hotplug_disabled--;
 	cpu_maps_update_done();
 }
-
+EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
 #endif	/* CONFIG_HOTPLUG_CPU */
 
 /* Need to know about CPUs going up/down? */
@@ -600,7 +602,7 @@ int disable_nonboot_cpus(void)
 	if (!error) {
 		BUG_ON(num_online_cpus() > 1);
 		/* Make sure the CPUs won't be enabled by someone else */
-		cpu_hotplug_disabled = 1;
+		cpu_hotplug_disabled++;
 	} else {
 		pr_err("Non-boot CPUs are not disabled\n");
 	}
@@ -622,7 +624,8 @@ void __ref enable_nonboot_cpus(void)
 
 	/* Allow everyone to use the CPU hotplug again */
 	cpu_maps_update_begin();
-	cpu_hotplug_disabled = 0;
+	if (cpu_hotplug_disabled)
+		cpu_hotplug_disabled--;
 	if (cpumask_empty(frozen_cpus))
 		goto out;
 
-- 
1.7.4.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
Please read the FAQ at  http://www.tux.org/lkml/

RE: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[KY Srinivasan]

> -----Original Message-----
> From: K. Y. Srinivasan [mailto:kys@microsoft.com]
> Sent: Sunday, June 21, 2015 7:56 PM
> To: gregkh@linuxfoundation.org; linux-kernel@vger.kernel.org;
> devel@linuxdriverproject.org; mingo@kernel.org; peterz@infradead.org;
> rafael.j.wysocki@intel.com; tglx@linutronix.de
> Cc: Vitaly Kuznetsov; KY Srinivasan
> Subject: [PATCH 1/1] cpu-hotplug: export
> cpu_hotplug_enable/cpu_hotplug_disable
> 
> From: Vitaly Kuznetsov <vkuznets@redhat.com>
> 
> Loaded Hyper-V module will use these functions to disable CPU
> hotplug under certain circumstances. Convert cpu_hotplug_disabled
> to a counter (protected by cpu_add_remove_lock) to support e.g.
> disable -> disable -> enable call sequences.

Greg is willing to take this patch through his tree if gets an ack. Please provide your feedback.

Regards,

K. Y
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> Signed-off-by: K. Y. Srinivasan <kys@microsoft.com>
> ---
>  Documentation/power/suspend-and-cpuhotplug.txt |    6 +++---
>  kernel/cpu.c                                   |   13 ++++++++-----
>  2 files changed, 11 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/power/suspend-and-cpuhotplug.txt
> b/Documentation/power/suspend-and-cpuhotplug.txt
> index 2850df3..2fc9095 100644
> --- a/Documentation/power/suspend-and-cpuhotplug.txt
> +++ b/Documentation/power/suspend-and-cpuhotplug.txt
> @@ -72,7 +72,7 @@ More details follow:
>                                          |
>                                          v
>                             Disable regular cpu hotplug
> -                        by setting cpu_hotplug_disabled=1
> +                        by increasing cpu_hotplug_disabled
>                                          |
>                                          v
>                              Release cpu_add_remove_lock
> @@ -89,7 +89,7 @@ Resuming back is likewise, with the counterparts being
> (in the order of
>  execution during resume):
>  * enable_nonboot_cpus() which involves:
>     |  Acquire cpu_add_remove_lock
> -   |  Reset cpu_hotplug_disabled to 0, thereby enabling regular cpu hotplug
> +   |  Decrease cpu_hotplug_disabled, thereby enabling regular cpu hotplug
>     |  Call _cpu_up() [for all those cpus in the frozen_cpus mask, in a loop]
>     |  Release cpu_add_remove_lock
>     v
> @@ -120,7 +120,7 @@ after the entire cycle is complete (i.e., suspend +
> resume).
>                             Acquire cpu_add_remove_lock
>                                          |
>                                          v
> -                          If cpu_hotplug_disabled is 1
> +                          If cpu_hotplug_disabled > 0
>                                  return gracefully
>                                          |
>                                          |
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 94bbe46..8f35ee6 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
>  void cpu_hotplug_disable(void)
>  {
>  	cpu_maps_update_begin();
> -	cpu_hotplug_disabled = 1;
> +	cpu_hotplug_disabled++;
>  	cpu_maps_update_done();
>  }
> +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
> 
>  void cpu_hotplug_enable(void)
>  {
>  	cpu_maps_update_begin();
> -	cpu_hotplug_disabled = 0;
> +	if (cpu_hotplug_disabled)
> +		cpu_hotplug_disabled--;
>  	cpu_maps_update_done();
>  }
> -
> +EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
>  #endif	/* CONFIG_HOTPLUG_CPU */
> 
>  /* Need to know about CPUs going up/down? */
> @@ -600,7 +602,7 @@ int disable_nonboot_cpus(void)
>  	if (!error) {
>  		BUG_ON(num_online_cpus() > 1);
>  		/* Make sure the CPUs won't be enabled by someone else
> */
> -		cpu_hotplug_disabled = 1;
> +		cpu_hotplug_disabled++;
>  	} else {
>  		pr_err("Non-boot CPUs are not disabled\n");
>  	}
> @@ -622,7 +624,8 @@ void __ref enable_nonboot_cpus(void)
> 
>  	/* Allow everyone to use the CPU hotplug again */
>  	cpu_maps_update_begin();
> -	cpu_hotplug_disabled = 0;
> +	if (cpu_hotplug_disabled)
> +		cpu_hotplug_disabled--;
>  	if (cpumask_empty(frozen_cpus))
>  		goto out;
> 
> --
> 1.7.4.1

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Peter Zijlstra]

On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:

> > diff --git a/kernel/cpu.c b/kernel/cpu.c
> > index 94bbe46..8f35ee6 100644
> > --- a/kernel/cpu.c
> > +++ b/kernel/cpu.c
> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
> >  void cpu_hotplug_disable(void)
> >  {
> >  	cpu_maps_update_begin();
> > -	cpu_hotplug_disabled = 1;
> > +	cpu_hotplug_disabled++;
> >  	cpu_maps_update_done();
> >  }
> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
> > 
> >  void cpu_hotplug_enable(void)
> >  {
> >  	cpu_maps_update_begin();
> > -	cpu_hotplug_disabled = 0;
> > +	if (cpu_hotplug_disabled)
> > +		cpu_hotplug_disabled--;

No that just papers over bugs.

> >  	cpu_maps_update_done();
> >  }

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Vitaly Kuznetsov]

Peter Zijlstra <peterz@infradead.org> writes:

> On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:
>
>> > diff --git a/kernel/cpu.c b/kernel/cpu.c
>> > index 94bbe46..8f35ee6 100644
>> > --- a/kernel/cpu.c
>> > +++ b/kernel/cpu.c
>> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
>> >  void cpu_hotplug_disable(void)
>> >  {
>> >  	cpu_maps_update_begin();
>> > -	cpu_hotplug_disabled = 1;
>> > +	cpu_hotplug_disabled++;
>> >  	cpu_maps_update_done();
>> >  }
>> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
>> > 
>> >  void cpu_hotplug_enable(void)
>> >  {
>> >  	cpu_maps_update_begin();
>> > -	cpu_hotplug_disabled = 0;
>> > +	if (cpu_hotplug_disabled)
>> > +		cpu_hotplug_disabled--;
>
> No that just papers over bugs.
>

Yes, but these bugs are not in Linux. I don't see any other way for a
platform to enable/disable cpu hotplug in runtime.

-- 
  Vitaly

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Thomas Gleixner]

On Wed, 24 Jun 2015, Vitaly Kuznetsov wrote:

> Peter Zijlstra <peterz@infradead.org> writes:
> 
> > On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:
> >
> >> > diff --git a/kernel/cpu.c b/kernel/cpu.c
> >> > index 94bbe46..8f35ee6 100644
> >> > --- a/kernel/cpu.c
> >> > +++ b/kernel/cpu.c
> >> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
> >> >  void cpu_hotplug_disable(void)
> >> >  {
> >> >  	cpu_maps_update_begin();
> >> > -	cpu_hotplug_disabled = 1;
> >> > +	cpu_hotplug_disabled++;
> >> >  	cpu_maps_update_done();
> >> >  }
> >> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
> >> > 
> >> >  void cpu_hotplug_enable(void)
> >> >  {
> >> >  	cpu_maps_update_begin();
> >> > -	cpu_hotplug_disabled = 0;
> >> > +	if (cpu_hotplug_disabled)
> >> > +		cpu_hotplug_disabled--;
> >
> > No that just papers over bugs.
> >
> 
> Yes, but these bugs are not in Linux. I don't see any other way for a
> platform to enable/disable cpu hotplug in runtime.

What? cpu_hotplug_disable()/enable() do that today. They just do not
support recursion and we have not yet seen any argument WHY that's
required at all.

Thanks,

	tglx

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Peter Zijlstra]

On Wed, Jun 24, 2015 at 11:43:33AM +0200, Vitaly Kuznetsov wrote:
> Peter Zijlstra <peterz@infradead.org> writes:
> > On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:
> >
> >> > diff --git a/kernel/cpu.c b/kernel/cpu.c
> >> > index 94bbe46..8f35ee6 100644
> >> > --- a/kernel/cpu.c
> >> > +++ b/kernel/cpu.c
> >> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
> >> >  void cpu_hotplug_disable(void)
> >> >  {
> >> >  	cpu_maps_update_begin();
> >> > -	cpu_hotplug_disabled = 1;
> >> > +	cpu_hotplug_disabled++;
> >> >  	cpu_maps_update_done();
> >> >  }
> >> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
> >> > 
> >> >  void cpu_hotplug_enable(void)
> >> >  {
> >> >  	cpu_maps_update_begin();
> >> > -	cpu_hotplug_disabled = 0;
> >> > +	if (cpu_hotplug_disabled)
> >> > +		cpu_hotplug_disabled--;
> >
> > No that just papers over bugs.
> 
> Yes, but these bugs are not in Linux. I don't see any other way for a
> platform to enable/disable cpu hotplug in runtime.

Any code using this had better be open source, if not I really don't
care anyway.

That really ought to be:

	WARN_ON(--cpu_hotplug_disabled < 0);

You do not make underflows go away.

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Vitaly Kuznetsov]

Thomas Gleixner <tglx@linutronix.de> writes:

> On Wed, 24 Jun 2015, Vitaly Kuznetsov wrote:
>
>> Peter Zijlstra <peterz@infradead.org> writes:
>> 
>> > On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:
>> >
>> >> > diff --git a/kernel/cpu.c b/kernel/cpu.c
>> >> > index 94bbe46..8f35ee6 100644
>> >> > --- a/kernel/cpu.c
>> >> > +++ b/kernel/cpu.c
>> >> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
>> >> >  void cpu_hotplug_disable(void)
>> >> >  {
>> >> >  	cpu_maps_update_begin();
>> >> > -	cpu_hotplug_disabled = 1;
>> >> > +	cpu_hotplug_disabled++;
>> >> >  	cpu_maps_update_done();
>> >> >  }
>> >> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
>> >> > 
>> >> >  void cpu_hotplug_enable(void)
>> >> >  {
>> >> >  	cpu_maps_update_begin();
>> >> > -	cpu_hotplug_disabled = 0;
>> >> > +	if (cpu_hotplug_disabled)
>> >> > +		cpu_hotplug_disabled--;
>> >
>> > No that just papers over bugs.
>> >
>> 
>> Yes, but these bugs are not in Linux. I don't see any other way for a
>> platform to enable/disable cpu hotplug in runtime.
>
> What? cpu_hotplug_disable()/enable() do that today. They just do not
> support recursion and we have not yet seen any argument WHY that's
> required at all.

They do, my patch just exports them. Despite the fact that I have no
specific usecase in my mind at this time I still see a theoretical
possibility of this being useful - when two pieces of code are trying to
enable/disable cpu hotplug simultaneously. Without making
cpu_hotplug_disabled a counter disable->disable->enable sequence will
lead to cpu hotplug being enabled.

-- 
  Vitaly

Re: [PATCH 1/1] cpu-hotplug: export cpu_hotplug_enable/cpu_hotplug_disable

[Vitaly Kuznetsov]

Peter Zijlstra <peterz@infradead.org> writes:

> On Wed, Jun 24, 2015 at 11:43:33AM +0200, Vitaly Kuznetsov wrote:
>> Peter Zijlstra <peterz@infradead.org> writes:
>> > On Tue, Jun 23, 2015 at 04:57:05PM +0000, KY Srinivasan wrote:
>> >
>> >> > diff --git a/kernel/cpu.c b/kernel/cpu.c
>> >> > index 94bbe46..8f35ee6 100644
>> >> > --- a/kernel/cpu.c
>> >> > +++ b/kernel/cpu.c
>> >> > @@ -190,17 +190,19 @@ void cpu_hotplug_done(void)
>> >> >  void cpu_hotplug_disable(void)
>> >> >  {
>> >> >  	cpu_maps_update_begin();
>> >> > -	cpu_hotplug_disabled = 1;
>> >> > +	cpu_hotplug_disabled++;
>> >> >  	cpu_maps_update_done();
>> >> >  }
>> >> > +EXPORT_SYMBOL_GPL(cpu_hotplug_disable);
>> >> > 
>> >> >  void cpu_hotplug_enable(void)
>> >> >  {
>> >> >  	cpu_maps_update_begin();
>> >> > -	cpu_hotplug_disabled = 0;
>> >> > +	if (cpu_hotplug_disabled)
>> >> > +		cpu_hotplug_disabled--;
>> >
>> > No that just papers over bugs.
>> 
>> Yes, but these bugs are not in Linux. I don't see any other way for a
>> platform to enable/disable cpu hotplug in runtime.
>
> Any code using this had better be open source, if not I really don't
> care anyway.
>
> That really ought to be:
>
> 	WARN_ON(--cpu_hotplug_disabled < 0);
>
> You do not make underflows go away.

Sorry, I was under an impression you were referring to the patch as a
whole :-) Yes, I agree, let's add this WARN_ON().

-- 
  Vitaly