[PATCH 0/2 resend] show isolated & nohz_full cpus in sysfs

[PATCH 0/2 resend] show isolated & nohz_full cpus in sysfs

[riel]

Currently there is no good way to get the isolated and nohz_full
CPUs at runtime, because the kernel may have changed the CPUs
specified on the commandline (when specifying all CPUs as
isolated, or CPUs that do not exist, ...)

This series adds two files to /sys/devices/system/cpu, which can
be used by system management tools like libvirt, openstack, etc.
to ensure proper task placement.

These patches were kind of (but not formally) acked by
Mike and Frederic, see https://lkml.org/lkml/2015/3/27/852

[PATCH 1/2] show isolated cpus in sysfs

[riel]

From: Rik van Riel <riel@redhat.com>

After system bootup, there is no totally reliable way to see
which CPUs are isolated, because the kernel may modify the
CPUs specified on the isolcpus= kernel command line option.

Export the CPU list that actually got isolated in sysfs,
specifically in the file /sys/devices/system/cpu/isolated

This can be used by system management tools like libvirt,
openstack, and others to ensure proper placement of tasks.

Suggested-by: Li Zefan <lizefan@huawei.com>
Signed-off-by: Rik van Riel <riel@redhat.com>
---
 drivers/base/cpu.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index f160ea44a86d..ea23ee7b545b 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -265,6 +265,17 @@ static ssize_t print_cpus_offline(struct device *dev,
 }
 static DEVICE_ATTR(offline, 0444, print_cpus_offline, NULL);
 
+static ssize_t print_cpus_isolated(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	int n = 0, len = PAGE_SIZE-2;
+
+	n = scnprintf(buf, len, "%*pbl\n", cpumask_pr_args(cpu_isolated_map));
+
+	return n;
+}
+static DEVICE_ATTR(isolated, 0444, print_cpus_isolated, NULL);
+
 static void cpu_device_release(struct device *dev)
 {
 	/*
@@ -431,6 +442,7 @@ static struct attribute *cpu_root_attrs[] = {
 	&cpu_attrs[2].attr.attr,
 	&dev_attr_kernel_max.attr,
 	&dev_attr_offline.attr,
+	&dev_attr_isolated.attr,
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
 #endif
-- 
2.1.0

[PATCH 2/2] show nohz_full cpus in sysfs

[riel]

From: Rik van Riel <riel@redhat.com>

Currently there is no way to query which CPUs are in nohz_full
mode from userspace.

Export the CPU list running in nohz_full mode in sysfs,
specifically in the file /sys/devices/system/cpu/nohz_full

This can be used by system management tools like libvirt,
openstack, and others to ensure proper task placement.

Signed-off-by: Rik van Riel <riel@redhat.com>
---
 drivers/base/cpu.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index ea23ee7b545b..78720e706176 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -16,6 +16,7 @@
 #include <linux/acpi.h>
 #include <linux/of.h>
 #include <linux/cpufeature.h>
+#include <linux/tick.h>
 
 #include "base.h"
 
@@ -276,6 +277,19 @@ static ssize_t print_cpus_isolated(struct device *dev,
 }
 static DEVICE_ATTR(isolated, 0444, print_cpus_isolated, NULL);
 
+#ifdef CONFIG_NO_HZ_FULL
+static ssize_t print_cpus_nohz_full(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	int n = 0, len = PAGE_SIZE-2;
+
+	n = scnprintf(buf, len, "%*pbl\n", cpumask_pr_args(tick_nohz_full_mask));
+
+	return n;
+}
+static DEVICE_ATTR(nohz_full, 0444, print_cpus_nohz_full, NULL);
+#endif
+
 static void cpu_device_release(struct device *dev)
 {
 	/*
@@ -443,6 +457,9 @@ static struct attribute *cpu_root_attrs[] = {
 	&dev_attr_kernel_max.attr,
 	&dev_attr_offline.attr,
 	&dev_attr_isolated.attr,
+#ifdef CONFIG_NO_HZ_FULL
+	&dev_attr_nohz_full.attr,
+#endif
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
 #endif
-- 
2.1.0

Re: [PATCH 1/2] show isolated cpus in sysfs

[Frederic Weisbecker]

On Fri, Apr 24, 2015 at 03:24:27PM -0400, riel@redhat.com wrote:
> From: Rik van Riel <riel@redhat.com>
> 
> After system bootup, there is no totally reliable way to see
> which CPUs are isolated, because the kernel may modify the
> CPUs specified on the isolcpus= kernel command line option.
> 
> Export the CPU list that actually got isolated in sysfs,
> specifically in the file /sys/devices/system/cpu/isolated
> 
> This can be used by system management tools like libvirt,
> openstack, and others to ensure proper placement of tasks.
> 
> Suggested-by: Li Zefan <lizefan@huawei.com>
> Signed-off-by: Rik van Riel <riel@redhat.com>

This patch should go through Peterz.

> ---
>  drivers/base/cpu.c | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
> index f160ea44a86d..ea23ee7b545b 100644
> --- a/drivers/base/cpu.c
> +++ b/drivers/base/cpu.c
> @@ -265,6 +265,17 @@ static ssize_t print_cpus_offline(struct device *dev,
>  }
>  static DEVICE_ATTR(offline, 0444, print_cpus_offline, NULL);
>  
> +static ssize_t print_cpus_isolated(struct device *dev,
> +				  struct device_attribute *attr, char *buf)
> +{
> +	int n = 0, len = PAGE_SIZE-2;
> +
> +	n = scnprintf(buf, len, "%*pbl\n", cpumask_pr_args(cpu_isolated_map));
> +
> +	return n;
> +}
> +static DEVICE_ATTR(isolated, 0444, print_cpus_isolated, NULL);
> +
>  static void cpu_device_release(struct device *dev)
>  {
>  	/*
> @@ -431,6 +442,7 @@ static struct attribute *cpu_root_attrs[] = {
>  	&cpu_attrs[2].attr.attr,
>  	&dev_attr_kernel_max.attr,
>  	&dev_attr_offline.attr,
> +	&dev_attr_isolated.attr,
>  #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
>  	&dev_attr_modalias.attr,
>  #endif
> -- 
> 2.1.0
> 

Re: [PATCH 1/2] show isolated cpus in sysfs

[Tejun Heo]

On Fri, Apr 24, 2015 at 11:11:29PM +0200, Frederic Weisbecker wrote:
> > Suggested-by: Li Zefan <lizefan@huawei.com>
> > Signed-off-by: Rik van Riel <riel@redhat.com>
> 
> This patch should go through Peterz.

Right, that's why the two patches kinda got lost.  Rik, sorry about
that and yes these two should go through Peterz.

Thanks.

-- 
tejun

Re: [PATCH 1/2] show isolated cpus in sysfs

[Rik van Riel]

On 04/24/2015 05:11 PM, Frederic Weisbecker wrote:
> On Fri, Apr 24, 2015 at 03:24:27PM -0400, riel@redhat.com wrote:
>> From: Rik van Riel <riel@redhat.com>
>>
>> After system bootup, there is no totally reliable way to see
>> which CPUs are isolated, because the kernel may modify the
>> CPUs specified on the isolcpus= kernel command line option.
>>
>> Export the CPU list that actually got isolated in sysfs,
>> specifically in the file /sys/devices/system/cpu/isolated
>>
>> This can be used by system management tools like libvirt,
>> openstack, and others to ensure proper placement of tasks.
>>
>> Suggested-by: Li Zefan <lizefan@huawei.com>
>> Signed-off-by: Rik van Riel <riel@redhat.com>
> 
> This patch should go through Peterz.

Oh, fun. That's what I get for getting the get_maintainer.pl
script, which told me to go through Greg KH instead :)

$ ./scripts/get_maintainer.pl -f drivers/base/cpu.c
Greg Kroah-Hartman <gregkh@linuxfoundation.org> (supporter:DRIVER CORE,
KOBJ...)
linux-kernel@vger.kernel.org (open list)

-- 
All rights reversed

Re: [PATCH 1/2] show isolated cpus in sysfs

[Greg KH]

On Fri, Apr 24, 2015 at 05:22:12PM -0400, Rik van Riel wrote:
> On 04/24/2015 05:11 PM, Frederic Weisbecker wrote:
> > On Fri, Apr 24, 2015 at 03:24:27PM -0400, riel@redhat.com wrote:
> >> From: Rik van Riel <riel@redhat.com>
> >>
> >> After system bootup, there is no totally reliable way to see
> >> which CPUs are isolated, because the kernel may modify the
> >> CPUs specified on the isolcpus= kernel command line option.
> >>
> >> Export the CPU list that actually got isolated in sysfs,
> >> specifically in the file /sys/devices/system/cpu/isolated
> >>
> >> This can be used by system management tools like libvirt,
> >> openstack, and others to ensure proper placement of tasks.
> >>
> >> Suggested-by: Li Zefan <lizefan@huawei.com>
> >> Signed-off-by: Rik van Riel <riel@redhat.com>
> > 
> > This patch should go through Peterz.
> 
> Oh, fun. That's what I get for getting the get_maintainer.pl
> script, which told me to go through Greg KH instead :)
> 
> $ ./scripts/get_maintainer.pl -f drivers/base/cpu.c
> Greg Kroah-Hartman <gregkh@linuxfoundation.org> (supporter:DRIVER CORE,
> KOBJ...)
> linux-kernel@vger.kernel.org (open list)

That's right, I can take this, I missed this the last time Rik sent
these, that's my fault.

thanks,

greg k-h

Re: [PATCH 0/2 resend] show isolated & nohz_full cpus in sysfs

[Mike Galbraith]

On Fri, 2015-04-24 at 15:24 -0400, riel@redhat.com wrote:
> Currently there is no good way to get the isolated and nohz_full
> CPUs at runtime, because the kernel may have changed the CPUs
> specified on the commandline (when specifying all CPUs as
> isolated, or CPUs that do not exist, ...)
> 
> This series adds two files to /sys/devices/system/cpu, which can
> be used by system management tools like libvirt, openstack, etc.
> to ensure proper task placement.
> 
> These patches were kind of (but not formally) acked by
> Mike and Frederic, see https://lkml.org/lkml/2015/3/27/852

Acked-by: Mike Galbraith <umgwanakikbuti@gmail.com>

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Chris Metcalf]

On 04/24/2015 03:24 PM, riel@redhat.com wrote:
> From: Rik van Riel<riel@redhat.com>
>
> Currently there is no way to query which CPUs are in nohz_full
> mode from userspace.
>
> Export the CPU list running in nohz_full mode in sysfs,
> specifically in the file /sys/devices/system/cpu/nohz_full
>
> This can be used by system management tools like libvirt,
> openstack, and others to ensure proper task placement.
>
> Signed-off-by: Rik van Riel<riel@redhat.com>
> ---
>   drivers/base/cpu.c | 17 +++++++++++++++++
>   1 file changed, 17 insertions(+)

In the arch/tile distribution with our "dataplane" support, we have
traditionally had a /sys/devices/system/cpu/dataplane file with
the same semantics you are proposing.  It's definitely helpful
to userspace applications figuring out how to appropriately take
over the machine they are being run on.

Acked-by: Chris Metcalf <cmetcalf@ezchip.com>
-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com

Re: [PATCH 1/2] show isolated cpus in sysfs

[Rik van Riel]

On 04/24/2015 05:49 PM, Greg KH wrote:
> On Fri, Apr 24, 2015 at 05:22:12PM -0400, Rik van Riel wrote:
>> On 04/24/2015 05:11 PM, Frederic Weisbecker wrote:
>>> On Fri, Apr 24, 2015 at 03:24:27PM -0400, riel@redhat.com wrote:
>>>> From: Rik van Riel <riel@redhat.com>
>>>>
>>>> After system bootup, there is no totally reliable way to see
>>>> which CPUs are isolated, because the kernel may modify the
>>>> CPUs specified on the isolcpus= kernel command line option.
>>>>
>>>> Export the CPU list that actually got isolated in sysfs,
>>>> specifically in the file /sys/devices/system/cpu/isolated
>>>>
>>>> This can be used by system management tools like libvirt,
>>>> openstack, and others to ensure proper placement of tasks.
>>>>
>>>> Suggested-by: Li Zefan <lizefan@huawei.com>
>>>> Signed-off-by: Rik van Riel <riel@redhat.com>
>>>
>>> This patch should go through Peterz.
>>
>> Oh, fun. That's what I get for getting the get_maintainer.pl
>> script, which told me to go through Greg KH instead :)
>>
>> $ ./scripts/get_maintainer.pl -f drivers/base/cpu.c
>> Greg Kroah-Hartman <gregkh@linuxfoundation.org> (supporter:DRIVER CORE,
>> KOBJ...)
>> linux-kernel@vger.kernel.org (open list)
> 
> That's right, I can take this, I missed this the last time Rik sent
> these, that's my fault.

Ping?

Let me harass you before we both forget and this
thing falls through the cracks again :)

-- 
All rights reversed

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Frederic Weisbecker]

On Sat, Mar 28, 2015 at 12:15:56PM -0400, Rik van Riel wrote:
> On 03/28/2015 12:02 PM, Frederic Weisbecker wrote:
> > 2015-03-27 22:50 GMT+01:00  <riel@redhat.com>:
> >> From: Rik van Riel <riel@redhat.com>
> >>
> >> Currently there is no way to query which CPUs are in nohz_full
> >> mode from userspace.
> > 
> > Well you can watch dmesg | grep NO_HZ
> > But surely sysfs is more convenient from an app.
> > 
> > I guess it's ok, as long as it's strictly Read Only. Here it seems to
> > be the case. And it's not chmod'able, right?
> 
> I followed the other code for files in that directory.
> 
> Quick testing shows that the cpu info files in
> /sys/devices/system/cpu are chmoddable, but writing
> to them fails with -EIO because there is no function
> set up to handle writes. So yeah, read only :)

Perfect :-)

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Rik van Riel]

On 03/28/2015 12:02 PM, Frederic Weisbecker wrote:
> 2015-03-27 22:50 GMT+01:00  <riel@redhat.com>:
>> From: Rik van Riel <riel@redhat.com>
>>
>> Currently there is no way to query which CPUs are in nohz_full
>> mode from userspace.
> 
> Well you can watch dmesg | grep NO_HZ
> But surely sysfs is more convenient from an app.
> 
> I guess it's ok, as long as it's strictly Read Only. Here it seems to
> be the case. And it's not chmod'able, right?

I followed the other code for files in that directory.

Quick testing shows that the cpu info files in
/sys/devices/system/cpu are chmoddable, but writing
to them fails with -EIO because there is no function
set up to handle writes. So yeah, read only :)

-- 
All rights reversed

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Frederic Weisbecker]

2015-03-27 22:50 GMT+01:00  <riel@redhat.com>:
> From: Rik van Riel <riel@redhat.com>
>
> Currently there is no way to query which CPUs are in nohz_full
> mode from userspace.

Well you can watch dmesg | grep NO_HZ
But surely sysfs is more convenient from an app.

I guess it's ok, as long as it's strictly Read Only. Here it seems to
be the case. And it's not chmod'able, right?

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Mike Galbraith]

On Sat, 2015-03-28 at 09:35 -0400, Rik van Riel wrote:

> It can be used by programs like irqbalance to avoid binding IRQs
> to isolated or nohz_full CPUs, by libvirt to know which CPUs do
> not get load balancing of SCHED_OTHER tasks, etc...

Ok, other system packages learning to avoid these does make good sense.

	-Mike

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Rik van Riel]

On 03/28/2015 12:18 AM, Mike Galbraith wrote:
> On Fri, 2015-03-27 at 17:50 -0400, riel@redhat.com wrote:
>> From: Rik van Riel <riel@redhat.com>
>>
>> Currently there is no way to query which CPUs are in nohz_full
>> mode from userspace.
> 
> Hm, they're both (as of your last set) invariant. 

So are most of the others in /sys/device/system/cpu

That does not make it less useful to discover what
the system setup turned out to be after boot.

> Is this so an HPC app
> can automatically bind itself or something?  You can't have more than
> one such app, or rather if you did, they'd need more than which CPUs are
> HPC capable, they'd need occupancy too, so the query mechanism seems
> kinda useless.  Box driver has to allocate CPUs, and presumably knows
> the configuration of the box (those who don't become ex box drivers).

Knowing what system you bought does not reduce the usefulness
of /proc/cpuinfo.

The CPUs that actually end up isolated or nohz_full can differ
from what was specified on the kernel commandline, and being
able to see which CPUs ended up in the desired state seems useful.

It can be used by programs like irqbalance to avoid binding IRQs
to isolated or nohz_full CPUs, by libvirt to know which CPUs do
not get load balancing of SCHED_OTHER tasks, etc...

-- 
All rights reversed

Re: [PATCH 2/2] show nohz_full cpus in sysfs

[Mike Galbraith]

On Fri, 2015-03-27 at 17:50 -0400, riel@redhat.com wrote:
> From: Rik van Riel <riel@redhat.com>
> 
> Currently there is no way to query which CPUs are in nohz_full
> mode from userspace.

Hm, they're both (as of your last set) invariant.  Is this so an HPC app
can automatically bind itself or something?  You can't have more than
one such app, or rather if you did, they'd need more than which CPUs are
HPC capable, they'd need occupancy too, so the query mechanism seems
kinda useless.  Box driver has to allocate CPUs, and presumably knows
the configuration of the box (those who don't become ex box drivers).

	-Mike

[PATCH 2/2] show nohz_full cpus in sysfs

[riel]

From: Rik van Riel <riel@redhat.com>

Currently there is no way to query which CPUs are in nohz_full
mode from userspace.

Export the CPU list running in nohz_full mode in sysfs,
specifically in the file /sys/devices/system/cpu/nohz_full

Signed-off-by: Rik van Riel <riel@redhat.com>
---
 drivers/base/cpu.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index ea23ee7b545b..78720e706176 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -16,6 +16,7 @@
 #include <linux/acpi.h>
 #include <linux/of.h>
 #include <linux/cpufeature.h>
+#include <linux/tick.h>
 
 #include "base.h"
 
@@ -276,6 +277,19 @@ static ssize_t print_cpus_isolated(struct device *dev,
 }
 static DEVICE_ATTR(isolated, 0444, print_cpus_isolated, NULL);
 
+#ifdef CONFIG_NO_HZ_FULL
+static ssize_t print_cpus_nohz_full(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	int n = 0, len = PAGE_SIZE-2;
+
+	n = scnprintf(buf, len, "%*pbl\n", cpumask_pr_args(tick_nohz_full_mask));
+
+	return n;
+}
+static DEVICE_ATTR(nohz_full, 0444, print_cpus_nohz_full, NULL);
+#endif
+
 static void cpu_device_release(struct device *dev)
 {
 	/*
@@ -443,6 +457,9 @@ static struct attribute *cpu_root_attrs[] = {
 	&dev_attr_kernel_max.attr,
 	&dev_attr_offline.attr,
 	&dev_attr_isolated.attr,
+#ifdef CONFIG_NO_HZ_FULL
+	&dev_attr_nohz_full.attr,
+#endif
 #ifdef CONFIG_GENERIC_CPU_AUTOPROBE
 	&dev_attr_modalias.attr,
 #endif
-- 
2.1.0