mm, page_alloc: Swap likely to unlikely as code logic is different for next_zones_zonelist()

mm, page_alloc: Swap likely to unlikely as code logic is different for next_zones_zonelist()

[Steven Rostedt]

Commit 682a3385e773 "mm, page_alloc: inline the fast path of the
zonelist iterator" changed how next_zones_zonelist() is called, by
adding a static inline function to do the fast path. This function adds:

       if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))
               return z;
       return __next_zones_zonelist(z, highest_zoneidx, nodes);

Where __next_zones_zonelist() is only called when nodes is not NULL or
zonelist_zone_idx(z) is less than highest_zoneidx.

The original next_zone_zonelist() was converted to
__next_zones_zonelist() but it still maintained:

	if (likely(nodes == NULL))

Which is now actually a very unlikely, as it is only called with nodes
equal to NULL when zonelist_zone_idx(z) is greater than highest_zoneidx.

Before this commit, this if had this statistic:

 correct incorrect  %        Function                  File              Line
 ------- ---------  -        --------                  ----              ----
  837895   446078  34 next_zones_zonelist            mmzone.c             63

After this commit, it has:

 correct incorrect  %        Function                  File              Line
 ------- ---------  -        --------                  ----              ----
      10   173840  99 __next_zones_zonelist          mmzone.c             63

Thus, the if statement is now much more unlikely than it ever was as a
likely.

Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
---
diff --git a/mm/mmzone.c b/mm/mmzone.c
index 5652be858e5e..a51c0a67ea3d 100644
--- a/mm/mmzone.c
+++ b/mm/mmzone.c
@@ -60,7 +60,7 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
 	 * Find the next suitable zone to use for the allocation.
 	 * Only filter based on nodemask if it's set
 	 */
-	if (likely(nodes == NULL))
+	if (unlikely(nodes == NULL))
 		while (zonelist_zone_idx(z) > highest_zoneidx)
 			z++;
 	else

Re: mm, page_alloc: Swap likely to unlikely as code logic is different for next_zones_zonelist()

[Vlastimil Babka]

On 01/06/2017 02:01 AM, Steven Rostedt wrote:
> Commit 682a3385e773 "mm, page_alloc: inline the fast path of the
> zonelist iterator" changed how next_zones_zonelist() is called, by
> adding a static inline function to do the fast path. This function adds:
> 
>        if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))

Your profiling stats posted online suggested even this one was wrong:

          function.file.line         correct 	incorrect 	percent
next_zones_zonelist.mmzone.h.965    216840653	25415261543	%0.85

Was it bogus then, or is there other explanation? I have speculated that
this might be due to multiple cpusets configured in the system.

>                return z;
>        return __next_zones_zonelist(z, highest_zoneidx, nodes);
> 
> Where __next_zones_zonelist() is only called when nodes is not NULL or
> zonelist_zone_idx(z) is less than highest_zoneidx.
> 
> The original next_zone_zonelist() was converted to
> __next_zones_zonelist() but it still maintained:
> 
> 	if (likely(nodes == NULL))
> 
> Which is now actually a very unlikely, as it is only called with nodes
> equal to NULL when zonelist_zone_idx(z) is greater than highest_zoneidx.
> 
> Before this commit, this if had this statistic:
> 
>  correct incorrect  %        Function                  File              Line
>  ------- ---------  -        --------                  ----              ----
>   837895   446078  34 next_zones_zonelist            mmzone.c             63
> 
> After this commit, it has:
> 
>  correct incorrect  %        Function                  File              Line
>  ------- ---------  -        --------                  ----              ----
>       10   173840  99 __next_zones_zonelist          mmzone.c             63
> 
> Thus, the if statement is now much more unlikely than it ever was as a
> likely.
> 
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

Anyway this makes sense.

Acked-by: Vlastimil Babka <vbabka@suse.cz>

> ---
> diff --git a/mm/mmzone.c b/mm/mmzone.c
> index 5652be858e5e..a51c0a67ea3d 100644
> --- a/mm/mmzone.c
> +++ b/mm/mmzone.c
> @@ -60,7 +60,7 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
>  	 * Find the next suitable zone to use for the allocation.
>  	 * Only filter based on nodemask if it's set
>  	 */
> -	if (likely(nodes == NULL))
> +	if (unlikely(nodes == NULL))
>  		while (zonelist_zone_idx(z) > highest_zoneidx)
>  			z++;
>  	else
> 

Re: mm, page_alloc: Swap likely to unlikely as code logic is different for next_zones_zonelist()

[Steven Rostedt]

On Fri, 6 Jan 2017 09:00:23 +0100
Vlastimil Babka <vbabka@suse.cz> wrote:

> On 01/06/2017 02:01 AM, Steven Rostedt wrote:
> > Commit 682a3385e773 "mm, page_alloc: inline the fast path of the
> > zonelist iterator" changed how next_zones_zonelist() is called, by
> > adding a static inline function to do the fast path. This function adds:
> > 
> >        if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))  
> 
> Your profiling stats posted online suggested even this one was wrong:
> 
>           function.file.line         correct 	incorrect 	percent
> next_zones_zonelist.mmzone.h.965    216840653	25415261543	%0.85
> 
> Was it bogus then, or is there other explanation? I have speculated that
> this might be due to multiple cpusets configured in the system.

I didn't have any cpusets configured on either system.

But the number is exaggerated, as it too was affected by the "constant"
optimization by gcc. But when I fixed that, it still gives me about
50%. Probably best just to remove the likely() in that case.

> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>  
> 
> Anyway this makes sense.
> 
> Acked-by: Vlastimil Babka <vbabka@suse.cz>

Thanks!

-- Steve

Re: mm, page_alloc: Swap likely to unlikely as code logic is different for next_zones_zonelist()

[Mel Gorman]

On Thu, Jan 05, 2017 at 08:01:02PM -0500, Steven Rostedt wrote:
> Commit 682a3385e773 "mm, page_alloc: inline the fast path of the
> zonelist iterator" changed how next_zones_zonelist() is called, by
> adding a static inline function to do the fast path. This function adds:
> 
>        if (likely(!nodes && zonelist_zone_idx(z) <= highest_zoneidx))
>                return z;
>        return __next_zones_zonelist(z, highest_zoneidx, nodes);
> 
> Where __next_zones_zonelist() is only called when nodes is not NULL or
> zonelist_zone_idx(z) is less than highest_zoneidx.
> 
> The original next_zone_zonelist() was converted to
> __next_zones_zonelist() but it still maintained:
> 
> 	if (likely(nodes == NULL))
> 
> Which is now actually a very unlikely, as it is only called with nodes
> equal to NULL when zonelist_zone_idx(z) is greater than highest_zoneidx.
> 
> Before this commit, this if had this statistic:
> 
>  correct incorrect  %        Function                  File              Line
>  ------- ---------  -        --------                  ----              ----
>   837895   446078  34 next_zones_zonelist            mmzone.c             63
> 
> After this commit, it has:
> 
>  correct incorrect  %        Function                  File              Line
>  ------- ---------  -        --------                  ----              ----
>       10   173840  99 __next_zones_zonelist          mmzone.c             63
> 
> Thus, the if statement is now much more unlikely than it ever was as a
> likely.
> 
> Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>

Well spotted;

Acked-by: Mel Gorman <mgorman@techsingularity.net>

-- 
Mel Gorman
SUSE Labs