Re: [PATCH 00/46] Automatic NUMA Balancing V4

[Mel Gorman <mgorman@suse.de>]

On Thu, Nov 22, 2012 at 10:05:14AM +0100, Ingo Molnar wrote:
> 
> > Thanks!
> > 
> > I ran a quick test with your 'balancenuma v4' tree and while 
> > numa02 and numa01-THREAD-ALLOC performance is looking good, 
> > numa01 performance does not look very good:
> > 
> >                     mainline    numa/core      balancenuma-v4
> >      numa01:           340.3       139.4          276 secs
> > 
> > 97% slower than numa/core.
> 
> I mean numa/core was 97% faster. That transforms into 
> balancenuma-v4 being 50.5% slower.
> 

I've asked the other questions on this already so I won't ask again.

> Your numbers from yesterday showed an even bigger proportion:
> 
> AUTONUMA BENCH
>                                           3.7.0                 3.7.0                 3.7.0                 3.7.0                 
> 3.7.0                 3.7.0
>                                 rc6-stats-v4r12   rc6-schednuma-v16r2 rc6-autonuma-v28fastr3	  rc6-moron-v4r38    rc6-twostage-v4r38  rc6-thpmigrate-v4r38
> Elapsed NUMA01                1668.03 (  0.00%)      486.04 ( 70.86%) 	   794.10 ( 52.39%)	 601.19 ( 63.96%)     1575.52 (  5.55%)     1066.67 ( 36.05%)
> 
> In your test numa/core was 240% times faster than mainline, 63% 
> faster than autonuma and 119% faster than 
> balancenuma-"rc6-thpmigrate-v4r38".
> 

Yes, I know and I know why. It's the two-filter thing and how it deals
with an adverse workload. Here is my description in mmtests comments on
what numa01 is.

# NUMA01
#   Two processes
#   NUM_CPUS/2 number of threads so all CPUs are in use
#   
#   On startup, the process forks
#   Each process mallocs a 3G buffer but there is no communication
#       between the processes.
#   Threads are created that zeros out the full buffer 1000 times
#
#   The objective of the test is that initially the two processes
#   allocate their memory on the same node. As the threads are
#   are created the memory will migrate from the initial node to
#   nodes that are closer to the referencing thread.
#
#   It is worth noting that this benchmark is specifically tuned
#   for two nodes and the expectation is that the two processes
#   and their threads split so that all process A runs on node 0
#   and all threads on process B run in node 1
#
#   With 4 and more nodes, this is actually an adverse workload.
#   As all the buffer is zeroed in both processes, there is an
#   expectation that it will continually bounce between two nodes.
#
#   So, on 2 nodes, this benchmark tests convergence. On 4 or more
#   nodes, this partially measures how much busy work automatic
#   NUMA migrate does and it'll be very noisy due to cache conflicts.

Wow, that's badly written! STill, on two nodes, numa01 is meant to
converge. On 4 nodes it is an adverse workload where the only reasonable
response is to interleave across all nodes. What likely happens is that
one process bounces between 2 nodes and the second process bounces between
the other two nodes -- all uselessly unless it's interleaving and then
backing off. I'm running on 4 nodes.

My tree has no interleaving policy. All it'll notice is that it may be
bouncing between nodes and stop migrating on the assumption that a remote
access is cheaper than a lot of migration. A smart policy is expected to
be able to overcome this.

Look at the other figures which are for a reasonable NUMA workload.
balancenuma is beating numacore there and it shouldn't be able to. One strong
likelihood is that we differ in the base mechanics. Another possibility
is that there is a bug in the policies somewhere. If you rebased on top,
we'd find out which.

-- 
Mel Gorman
SUSE Labs