[RFC] mem-hotplug: shall we skip unmovable node when doing numa balance?

[RFC] mem-hotplug: shall we skip unmovable node when doing numa balance?

[Xishi Qiu]

On mem-hotplug system, there is a problem, please see the following case.

memtester xxG, the memory will be alloced on a movable node. And after numa
balancing, the memory may be migrated to the other node, it may be a unmovable
node. This will reduce the free memory of the unmovable node, and may be oom
later.

My question is that shall we skip unmovable node when doing numa balance?
or just let the manager set some numa policies?

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 057964d..f0954ac 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2334,6 +2334,13 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 out:
 	mpol_cond_put(pol);
 
+	/* Skip unmovable nodes when do numa balancing */
+	if (movable_node_enabled && ret != -1) {
+		zone = NODE_DATA(ret)->node_zones + MAX_NR_ZONES - 1;
+		if (!populated_zone(zone))
+			ret = -1;
+	}
+
 	return ret;
 }

Thanks,
Xishi Qiu

Re: [RFC] mem-hotplug: shall we skip unmovable node when doing numa balance?

[Mel Gorman]

On Tue, Nov 08, 2016 at 12:43:17PM +0800, Xishi Qiu wrote:
> On mem-hotplug system, there is a problem, please see the following case.
> 
> memtester xxG, the memory will be alloced on a movable node. And after numa
> balancing, the memory may be migrated to the other node, it may be a unmovable
> node. This will reduce the free memory of the unmovable node, and may be oom
> later.
> 

How would it OOM later? It's movable memmory that is moving via
automatic NUMA balancing so at the very least it can be reclaimed. If
the memory is mlocked or unable to migrate then it's irrelevant if
automatic balancing put it there.

> My question is that shall we skip unmovable node when doing numa balance?
> or just let the manager set some numa policies?
> 

If the unmovable node must be protected from automatic NUMA balancing
then policies are the appropriate step to prevent the processes running
on that node or from allocating memory on that node.

Either way, protecting unmovable nodes in the name of hotplug is pretty
much guaranteed to be a performance black hole because at the very
least, page table pages will always be remote accesses for processes
running on the unmovable node.

> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 057964d..f0954ac 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -2334,6 +2334,13 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
>  out:
>  	mpol_cond_put(pol);
>  
> +	/* Skip unmovable nodes when do numa balancing */
> +	if (movable_node_enabled && ret != -1) {
> +		zone = NODE_DATA(ret)->node_zones + MAX_NR_ZONES - 1;
> +		if (!populated_zone(zone))
> +			ret = -1;
> +	}
> +
>  	return ret;
>  }

Nak.

-- 
Mel Gorman
SUSE Labs

Re: [RFC] mem-hotplug: shall we skip unmovable node when doing numa balance?

[Xishi Qiu]

On 2016/11/9 19:58, Mel Gorman wrote:

> On Tue, Nov 08, 2016 at 12:43:17PM +0800, Xishi Qiu wrote:
>> On mem-hotplug system, there is a problem, please see the following case.
>>
>> memtester xxG, the memory will be alloced on a movable node. And after numa
>> balancing, the memory may be migrated to the other node, it may be a unmovable
>> node. This will reduce the free memory of the unmovable node, and may be oom
>> later.
>>
> 
> How would it OOM later? It's movable memmory that is moving via
> automatic NUMA balancing so at the very least it can be reclaimed. If
> the memory is mlocked or unable to migrate then it's irrelevant if
> automatic balancing put it there.
> 

Hi Mel,

memtester will mlock the memory, so we can not reclaim, then maybe oom, right?
So let the manager set some numa policies to prevent the above case, right?

Thanks,
Xishi Qiu

>> My question is that shall we skip unmovable node when doing numa balance?
>> or just let the manager set some numa policies?
>>
> 
> If the unmovable node must be protected from automatic NUMA balancing
> then policies are the appropriate step to prevent the processes running
> on that node or from allocating memory on that node.
> 
> Either way, protecting unmovable nodes in the name of hotplug is pretty
> much guaranteed to be a performance black hole because at the very
> least, page table pages will always be remote accesses for processes
> running on the unmovable node.
> 
>> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
>> index 057964d..f0954ac 100644
>> --- a/mm/mempolicy.c
>> +++ b/mm/mempolicy.c
>> @@ -2334,6 +2334,13 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
>>  out:
>>  	mpol_cond_put(pol);
>>  
>> +	/* Skip unmovable nodes when do numa balancing */
>> +	if (movable_node_enabled && ret != -1) {
>> +		zone = NODE_DATA(ret)->node_zones + MAX_NR_ZONES - 1;
>> +		if (!populated_zone(zone))
>> +			ret = -1;
>> +	}
>> +
>>  	return ret;
>>  }
> 
> Nak.
> 

Re: [RFC] mem-hotplug: shall we skip unmovable node when doing numa balance?

[Mel Gorman]

On Thu, Nov 10, 2016 at 11:17:03AM +0800, Xishi Qiu wrote:
> On 2016/11/9 19:58, Mel Gorman wrote:
> 
> > On Tue, Nov 08, 2016 at 12:43:17PM +0800, Xishi Qiu wrote:
> >> On mem-hotplug system, there is a problem, please see the following case.
> >>
> >> memtester xxG, the memory will be alloced on a movable node. And after numa
> >> balancing, the memory may be migrated to the other node, it may be a unmovable
> >> node. This will reduce the free memory of the unmovable node, and may be oom
> >> later.
> >>
> > 
> > How would it OOM later? It's movable memmory that is moving via
> > automatic NUMA balancing so at the very least it can be reclaimed. If
> > the memory is mlocked or unable to migrate then it's irrelevant if
> > automatic balancing put it there.
> > 
> 
> memtester will mlock the memory, so we can not reclaim, then maybe oom, right?
> So let the manager set some numa policies to prevent the above case, right?
> 

Deal with it using policies.

-- 
Mel Gorman
SUSE Labs