[PATCH 0/2] mm: Fix N_CPU handlings of node_states

[PATCH 0/2] mm: Fix N_CPU handlings of node_states

[Toshi Kani]

node_stats[N_CPU] tracks which nodes have CPUs on the system.
vmstat.c updates this N_CPU information, but it has multiple
issues. 

Patch 1/2 changes setup_vmstat() to set up the N_CPU info at
boot.  Patch 2/2 changes vmstat_cpuup_callback() to udpate
the N_CPU info at CPU offline.

---
Toshi Kani (2)
  mm: Set N_CPU to node_states during boot
  mm: Clear N_CPU from node_states at CPU offline

---
 mm/vmstat.c | 21 ++++++++++++++++++++-
 1 file changed, 20 insertions(+), 1 deletion(-)

[PATCH 1/2] mm: Set N_CPU to node_states during boot

[Toshi Kani]

After a system booted, N_CPU is not set to any node as
has_cpu shows an empty line.

  # cat /sys/devices/system/node/has_cpu
  (show-empty-line)

setup_vmstat() registers its CPU notifier callback,
vmstat_cpuup_callback(), which marks N_CPU to a node when
a CPU is put into online.  However, setup_vmstat() is
called after all CPUs are launched in the boot sequence.

Changed setup_vmstat() to mark N_CPU to the nodes with
online CPUs at boot, which is consistent with other
operations in vmstat_cpuup_callback(), i.e. start_cpu_timer()
and refresh_zone_stat_thresholds().

Also added get_online_cpus() to protect the
for_each_online_cpu() loop.

Signed-off-by: Toshi Kani <toshi.kani@hp.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
---
 mm/vmstat.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 9bb3145..0a1f7de 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1276,8 +1276,12 @@ static int __init setup_vmstat(void)
 
 	register_cpu_notifier(&vmstat_notifier);
 
-	for_each_online_cpu(cpu)
+	get_online_cpus();
+	for_each_online_cpu(cpu) {
 		start_cpu_timer(cpu);
+		node_set_state(cpu_to_node(cpu), N_CPU);
+	}
+	put_online_cpus();
 #endif
 #ifdef CONFIG_PROC_FS
 	proc_create("buddyinfo", S_IRUGO, NULL, &fragmentation_file_operations);

[PATCH 2/2] mm: Clear N_CPU from node_states at CPU offline

[Toshi Kani]

vmstat_cpuup_callback() is a CPU notifier callback, which
marks N_CPU to a node at CPU online event.  However, it
does not update this N_CPU info at CPU offline event.

Changed vmstat_cpuup_callback() to clear N_CPU when the last
CPU in the node is put into offline, i.e. the node no longer
has any online CPU.

Signed-off-by: Toshi Kani <toshi.kani@hp.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
---
 mm/vmstat.c |   15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/mm/vmstat.c b/mm/vmstat.c
index 0a1f7de..b6d17ed 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -1229,6 +1229,20 @@ static void start_cpu_timer(int cpu)
 	schedule_delayed_work_on(cpu, work, __round_jiffies_relative(HZ, cpu));
 }
 
+static void vmstat_cpu_dead(int node)
+{
+	int cpu;
+
+	get_online_cpus();
+	for_each_online_cpu(cpu)
+		if (cpu_to_node(cpu) == node)
+			goto end;
+
+	node_clear_state(node, N_CPU);
+end:
+	put_online_cpus();
+}
+
 /*
  * Use the cpu notifier to insure that the thresholds are recalculated
  * when necessary.
@@ -1258,6 +1272,7 @@ static int vmstat_cpuup_callback(struct notifier_block *nfb,
 	case CPU_DEAD:
 	case CPU_DEAD_FROZEN:
 		refresh_zone_stat_thresholds();
+		vmstat_cpu_dead(cpu_to_node(cpu));
 		break;
 	default:
 		break;

Re: [PATCH 1/2] mm: Set N_CPU to node_states during boot

[Christoph Lameter]

On Tue, 15 Oct 2013, Toshi Kani wrote:

> Changed setup_vmstat() to mark N_CPU to the nodes with
> online CPUs at boot, which is consistent with other
> operations in vmstat_cpuup_callback(), i.e. start_cpu_timer()
> and refresh_zone_stat_thresholds().

Acked-by: Christoph Lameter <cl@linux.com>

Re: [PATCH 2/2] mm: Clear N_CPU from node_states at CPU offline

[Christoph Lameter]

On Tue, 15 Oct 2013, Toshi Kani wrote:

> vmstat_cpuup_callback() is a CPU notifier callback, which
> marks N_CPU to a node at CPU online event.  However, it
> does not update this N_CPU info at CPU offline event.

Acked-by: Christoph Lameter <cl@linux.com>

Re: [PATCH 1/2] mm: Set N_CPU to node_states during boot

[Toshi Kani]

On Tue, 2013-10-15 at 17:22 +0000, Christoph Lameter wrote:
> On Tue, 15 Oct 2013, Toshi Kani wrote:
> 
> > Changed setup_vmstat() to mark N_CPU to the nodes with
> > online CPUs at boot, which is consistent with other
> > operations in vmstat_cpuup_callback(), i.e. start_cpu_timer()
> > and refresh_zone_stat_thresholds().
> 
> Acked-by: Christoph Lameter <cl@linux.com>

Thanks Christoph for the quick review to the patchset!
-Toshi