[PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[Wei Yang]

When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu
affinity.

Current routine does like this:

  a) Iterate all the numa node
  b) Adjust cpu affinity when node has an online cpu

For a) this is not necessary, since the particular online cpu belongs to
a particular numa node. So it is not necessary to iterate on every nodes
on the system. This new onlined cpu just affect kswapd cpu affinity of
this particular node.

For b) several cpumask operation is used to check whether the node has
an online CPU. Since at this point we are sure one of our CPU onlined,
we can set the cpu affinity directly to current cpumask_of_node().

This patch simplifies the logic by set cpu affinity of the affected
kswapd.

---
v2:
  * rephrase the changelog

Signed-off-by: Wei Yang <richardw.yang@linux.intel.com>
---
 mm/vmscan.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 665f33258cd7..acc5af82b6ed 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4029,18 +4029,19 @@ unsigned long shrink_all_memory(unsigned long nr_to_reclaim)
    restore their cpu bindings. */
 static int kswapd_cpu_online(unsigned int cpu)
 {
-	int nid;
+	int nid = cpu_to_node(cpu);
+	pg_data_t *pgdat;
+	const struct cpumask *mask;
 
-	for_each_node_state(nid, N_MEMORY) {
-		pg_data_t *pgdat = NODE_DATA(nid);
-		const struct cpumask *mask;
+	if (!node_state(nid, N_MEMORY))
+		return 0;
 
-		mask = cpumask_of_node(pgdat->node_id);
+	pgdat = NODE_DATA(nid);
+	mask = cpumask_of_node(nid);
+
+	/* One of our CPUs online: restore mask */
+	set_cpus_allowed_ptr(pgdat->kswapd, mask);
 
-		if (cpumask_any_and(cpu_online_mask, mask) < nr_cpu_ids)
-			/* One of our CPUs online: restore mask */
-			set_cpus_allowed_ptr(pgdat->kswapd, mask);
-	}
 	return 0;
 }
 
-- 
2.17.1

Re: [PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[Michal Hocko]

On Fri 14-02-20 15:33:20, Wei Yang wrote:
> When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu
> affinity.
> 
> Current routine does like this:
> 
>   a) Iterate all the numa node
>   b) Adjust cpu affinity when node has an online cpu
> 
> For a) this is not necessary, since the particular online cpu belongs to
> a particular numa node. So it is not necessary to iterate on every nodes
> on the system. This new onlined cpu just affect kswapd cpu affinity of
> this particular node.
> 
> For b) several cpumask operation is used to check whether the node has
> an online CPU. Since at this point we are sure one of our CPU onlined,
> we can set the cpu affinity directly to current cpumask_of_node().
> 
> This patch simplifies the logic by set cpu affinity of the affected
> kswapd.

How have you tested this patch?

Also this is an old code and quite convoluted but does it still work as
inteded? I mean, I do not see any cpu offline callback to reduce the
cpu mask as all the CPUs for the given node go offline? Wouldn't the
scheduler simply go and fallback to no affinity if that happens?
In other words what is the value of kswapd_cpu_online in the first
place?
-- 
Michal Hocko
SUSE Labs

Re: [PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[Wei Yang]

On Fri, Feb 14, 2020 at 09:51:13AM +0100, Michal Hocko wrote:
>On Fri 14-02-20 15:33:20, Wei Yang wrote:
>> When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu
>> affinity.
>> 
>> Current routine does like this:
>> 
>>   a) Iterate all the numa node
>>   b) Adjust cpu affinity when node has an online cpu
>> 
>> For a) this is not necessary, since the particular online cpu belongs to
>> a particular numa node. So it is not necessary to iterate on every nodes
>> on the system. This new onlined cpu just affect kswapd cpu affinity of
>> this particular node.
>> 
>> For b) several cpumask operation is used to check whether the node has
>> an online CPU. Since at this point we are sure one of our CPU onlined,
>> we can set the cpu affinity directly to current cpumask_of_node().
>> 
>> This patch simplifies the logic by set cpu affinity of the affected
>> kswapd.
>
>How have you tested this patch?
>

I online one cpu and confirm the "cpu" is the one we just onlined.

If my understanding is correct, this is the expected behavior.

>Also this is an old code and quite convoluted but does it still work as
>inteded? I mean, I do not see any cpu offline callback to reduce the
>cpu mask as all the CPUs for the given node go offline? Wouldn't the

You are right, I didn't see the counterpart for cpu offline. This is the
question I want to ask. Seems we didn't handle it at the very beginning.

>scheduler simply go and fallback to no affinity if that happens?
>In other words what is the value of kswapd_cpu_online in the first
>place?

Some cases may this function be useful.

If we have a memory node which doesn't have any online cpu, the default
cpumask is not set. After one of the cpu online, we want to change cpu
affinity.

Or we want to add more cpu to the system, we could allow kswapd use more cpu
resources. Otherwise, kswapd would be limited to those original cpus.

>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me

Re: [PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[Michal Hocko]

On Sat 15-02-20 08:37:53, Wei Yang wrote:
> On Fri, Feb 14, 2020 at 09:51:13AM +0100, Michal Hocko wrote:
> >On Fri 14-02-20 15:33:20, Wei Yang wrote:
> >> When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu
> >> affinity.
> >> 
> >> Current routine does like this:
> >> 
> >>   a) Iterate all the numa node
> >>   b) Adjust cpu affinity when node has an online cpu
> >> 
> >> For a) this is not necessary, since the particular online cpu belongs to
> >> a particular numa node. So it is not necessary to iterate on every nodes
> >> on the system. This new onlined cpu just affect kswapd cpu affinity of
> >> this particular node.
> >> 
> >> For b) several cpumask operation is used to check whether the node has
> >> an online CPU. Since at this point we are sure one of our CPU onlined,
> >> we can set the cpu affinity directly to current cpumask_of_node().
> >> 
> >> This patch simplifies the logic by set cpu affinity of the affected
> >> kswapd.
> >
> >How have you tested this patch?
> >
> 
> I online one cpu and confirm the "cpu" is the one we just onlined.
> 
> If my understanding is correct, this is the expected behavior.
> 
> >Also this is an old code and quite convoluted but does it still work as
> >inteded? I mean, I do not see any cpu offline callback to reduce the
> >cpu mask as all the CPUs for the given node go offline? Wouldn't the
> 
> You are right, I didn't see the counterpart for cpu offline. This is the
> question I want to ask. Seems we didn't handle it at the very beginning.
> 
> >scheduler simply go and fallback to no affinity if that happens?
> >In other words what is the value of kswapd_cpu_online in the first
> >place?
> 
> Some cases may this function be useful.
> 
> If we have a memory node which doesn't have any online cpu, the default
> cpumask is not set. After one of the cpu online, we want to change cpu
> affinity.
> 
> Or we want to add more cpu to the system, we could allow kswapd use more cpu
> resources. Otherwise, kswapd would be limited to those original cpus.

OK, so the usecase is when a NUMA node gains a new CPU which wasn't
there at the time when the node got onlined. Is this a scenario we
really do care about? While not completely impossible I haven't seen
a system which would allow such a runtime configurability. Maybe it
would be simply easier to drop the callback for now until we have a real
world usecase to support it and have it documented.
-- 
Michal Hocko
SUSE Labs

Re: [PATCH v2] mm/vmscan.c: only adjust related kswapd cpu affinity when online cpu

[Wei Yang]

On Mon, Feb 17, 2020 at 10:31:24AM +0100, Michal Hocko wrote:
>On Sat 15-02-20 08:37:53, Wei Yang wrote:
>> On Fri, Feb 14, 2020 at 09:51:13AM +0100, Michal Hocko wrote:
>> >On Fri 14-02-20 15:33:20, Wei Yang wrote:
>> >> When onlining a cpu, kswapd_cpu_online() is called to adjust kswapd cpu
>> >> affinity.
>> >> 
>> >> Current routine does like this:
>> >> 
>> >>   a) Iterate all the numa node
>> >>   b) Adjust cpu affinity when node has an online cpu
>> >> 
>> >> For a) this is not necessary, since the particular online cpu belongs to
>> >> a particular numa node. So it is not necessary to iterate on every nodes
>> >> on the system. This new onlined cpu just affect kswapd cpu affinity of
>> >> this particular node.
>> >> 
>> >> For b) several cpumask operation is used to check whether the node has
>> >> an online CPU. Since at this point we are sure one of our CPU onlined,
>> >> we can set the cpu affinity directly to current cpumask_of_node().
>> >> 
>> >> This patch simplifies the logic by set cpu affinity of the affected
>> >> kswapd.
>> >
>> >How have you tested this patch?
>> >
>> 
>> I online one cpu and confirm the "cpu" is the one we just onlined.
>> 
>> If my understanding is correct, this is the expected behavior.
>> 
>> >Also this is an old code and quite convoluted but does it still work as
>> >inteded? I mean, I do not see any cpu offline callback to reduce the
>> >cpu mask as all the CPUs for the given node go offline? Wouldn't the
>> 
>> You are right, I didn't see the counterpart for cpu offline. This is the
>> question I want to ask. Seems we didn't handle it at the very beginning.
>> 
>> >scheduler simply go and fallback to no affinity if that happens?
>> >In other words what is the value of kswapd_cpu_online in the first
>> >place?
>> 
>> Some cases may this function be useful.
>> 
>> If we have a memory node which doesn't have any online cpu, the default
>> cpumask is not set. After one of the cpu online, we want to change cpu
>> affinity.
>> 
>> Or we want to add more cpu to the system, we could allow kswapd use more cpu
>> resources. Otherwise, kswapd would be limited to those original cpus.
>
>OK, so the usecase is when a NUMA node gains a new CPU which wasn't
>there at the time when the node got onlined. Is this a scenario we
>really do care about? While not completely impossible I haven't seen
>a system which would allow such a runtime configurability. Maybe it
>would be simply easier to drop the callback for now until we have a real
>world usecase to support it and have it documented.

I am fine with this suggestion. Let me prepare v3.

>-- 
>Michal Hocko
>SUSE Labs

-- 
Wei Yang
Help you, Help me