From: Mel Gorman <mgorman@suse.de>
To: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>,
Andrea Arcangeli <aarcange@redhat.com>,
Rik van Riel <riel@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Hugh Dickins <hughd@google.com>,
Thomas Gleixner <tglx@linutronix.de>,
Paul Turner <pjt@google.com>, Hillf Danton <dhillf@gmail.com>,
David Rientjes <rientjes@google.com>,
Lee Schermerhorn <Lee.Schermerhorn@hp.com>,
Alex Shi <lkml.alex@gmail.com>,
Srikar Dronamraju <srikar@linux.vnet.ibm.com>,
Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Andrew Morton <akpm@linux-foundation.org>,
Linux-MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 00/49] Automatic NUMA Balancing v10
Date: Sun, 9 Dec 2012 20:36:31 +0000 [thread overview]
Message-ID: <20121209203630.GC1009@suse.de> (raw)
In-Reply-To: <20121207110113.GB21482@gmail.com>
On Fri, Dec 07, 2012 at 12:01:13PM +0100, Ingo Molnar wrote:
>
> * Mel Gorman <mgorman@suse.de> wrote:
>
> > This is a full release of all the patches so apologies for the
> > flood. [...]
>
> I have yet to process all your mails, but assuming I address all
> your review feedback and the latest unified tree in tip:master
> shows no regression in your testing, would you be willing to
> start using it for ongoing work?
>
Ingo,
If you had read the second paragraph of the mail you just responded to or
the results at the end then you would have seen that I had problems with
the performance. You would also know that tip/master testing for the last
week was failing due to a boot problem (issue was in mainline not tip and
has been already fixed) and would have known that since the -v18 release
that numacore was effectively disabled on my test machine.
Clearly you are not reading the bug reports you are receiving and you're not
seeing the small bit of review feedback or answering the review questions
you have received either. Why would I be more forthcoming when I feel that
it'll simply be ignored? You simply assume that each batch of patches
you place on top must be fixing all known regressions and ignoring any
evidence to the contrary.
If you had read my mail from last Tuesday you would even know which patch
was causing the problem that effectively disabled numacore although not
why. The comment about p->numa_faults was completely off the mark (long
journey, was tired, assumed numa_faults was a counter and not a pointer
which was careless). If you had called me on it then I would have spotted
the actual problem sooner. The problem was indeed with the nr_cpus_allowed
== num_online_cpus()s check which I had pointed out was a suspicious check
although for different reasons. As it turns out, a printk() bodge showed
that nr_cpus_allowed == 80 set in sched_init_smp() while num_online_cpus()
== 48. This effectively disabling numacore. If you had responded to the
bug report, this would likely have been found last Wednesday.
As for my ongoing work, I have not actually changed much in the last
two weeks or so -- build fixes and your scalability patches. As I've
said multiple times, my primary objective was to build something minimal
that did something better than mainline although not necessarily as good
as the kernel potentially if either numacore or autonuma were rebased
on top. I left the tree so that other testing might validate it was
correct and avoid changing the tree too much prior to the merge window.
I deliberately avoided working on anything that would directly collide
with what numacore was trying to achieve.
> It would make it much easier for me to pick up your
> enhancements, fixes, etc.
>
> > Changelog since V9
> > o Migration scalability (mingo)
>
> To *really* see migration scalability bottlenecks you need to
> remove the migration-bandwidth throttling kludge from your tree
> (or configure it up very high if you want to do it simple).
>
Why is it a kludge? I already explained what the rational behind the rate
limiting was. It's not about scalability, it's about mitigating worse-case
behaviour and the amount of time the kernel spends moving data around which
a deliberately adverse workload can trigger. It is unacceptable if during a
phase change that a process would stall potentially for milliseconds (seconds
if the node is large enough I guess) while the data is being migrated. Here
is it again -- http://www.spinics.net/lists/linux-mm/msg47440.html . You
either ignored the mail or simply could not be bothered explaining why
you thought this was the incorrect decision or why the concerns about an
adverse workload were unimportant.
I have a vague suspicion actually that when you are modelling the task->data
relationship that you make an implicit assumption that moving data has
zero or near-zero cost. In such a model it would always make sense to move
quickly and immediately but in practice the cost of moving can exceed the
performance benefit of accessing local data and lead to regressions. It
becomes more pronounced if the nodes are not fully connected.
> Some (certainly not all) of the performance regressions you
> reported were certainly due to numa/core code hitting the
> migration codepaths as aggressively as the workload demanded -
> and hitting scalability bottlenecks.
>
How are you so certain? How do you not know it's because your code is
migrating excessively for no good reason because the algorithm has a flaw
in it? Or that the cost of excessive migration is not being offset by
local data accesses? The critical point to note is that if it really was
only scalability problems then autonuma would suffer the same problems
and would be impossible to autonumas performance to exceed numacores.
This isn't the case making it unlikely the scalability is your only problem.
Either way, last night I applied a patch on top of latest tip/master to
remove the nr_cpus_allowed check so that numacore would be enabled again
and tested that. In some places it has indeed much improved. In others
it is still regressing badly and in two case, it's corrupting memory --
specjbb when THP is enabled crashes when running for single or multiple
JVMs. It is likely that a zero page is being inserted due to a race with
migration and causes the JVM to throw a null pointer exception. Here is
the comparison on the rough off-chance you actually read it this time.
stats-v8r6 Same collection of TLB flush fixes and stats
numacore-20121130 Roughly numacore v17
numafix-20121209 numacore as of December 9th with the nr_cpus_allowed check removed.
Note that this is a 3.7-rc8 based test because that's what tip/master
is.
autonuma-v28fastr4 Autonuma v28fast with THP patch on top
balancenuma-v9r2 Balance numa v9
balancenuma-v10r3 V9 + the migration scalability patches
AutoNUMA Benchmark
==================
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
User NUMA01 65230.85 ( 0.00%) 24835.22 ( 61.93%) 21882.80 ( 66.45%) 30410.22 ( 53.38%) 52436.65 ( 19.61%) 59949.95 ( 8.10%)
User NUMA01_THEADLOCAL 60794.67 ( 0.00%) 17856.17 ( 70.63%) 18367.20 ( 69.79%) 17185.34 ( 71.73%) 17829.96 ( 70.67%) 17501.83 ( 71.21%)
User NUMA02 7031.50 ( 0.00%) 2084.38 ( 70.36%) 2391.47 ( 65.99%) 2238.73 ( 68.16%) 2079.48 ( 70.43%) 2094.68 ( 70.21%)
User NUMA02_SMT 2916.19 ( 0.00%) 1009.28 ( 65.39%) 1046.49 ( 64.11%) 1037.07 ( 64.44%) 997.57 ( 65.79%) 1010.15 ( 65.36%)
System NUMA01 39.66 ( 0.00%) 926.55 (-2236.23%) 134.00 (-237.87%) 236.83 (-497.15%) 275.09 (-593.62%) 265.02 (-568.23%)
System NUMA01_THEADLOCAL 42.33 ( 0.00%) 513.99 (-1114.25%) 201.65 (-376.38%) 70.90 (-67.49%) 110.82 (-161.80%) 130.30 (-207.82%)
System NUMA02 1.25 ( 0.00%) 18.57 (-1385.60%) 13.00 (-940.00%) 6.39 (-411.20%) 6.42 (-413.60%) 9.17 (-633.60%)
System NUMA02_SMT 16.66 ( 0.00%) 12.32 ( 26.05%) 7.26 ( 56.42%) 3.17 ( 80.97%) 3.58 ( 78.51%) 6.21 ( 62.73%)
Elapsed NUMA01 1511.76 ( 0.00%) 575.93 ( 61.90%) 475.26 ( 68.56%) 701.62 ( 53.59%) 1185.53 ( 21.58%) 1352.74 ( 10.52%)
Elapsed NUMA01_THEADLOCAL 1387.17 ( 0.00%) 398.55 ( 71.27%) 405.25 ( 70.79%) 378.47 ( 72.72%) 397.37 ( 71.35%) 387.93 ( 72.03%)
Elapsed NUMA02 176.81 ( 0.00%) 51.14 ( 71.08%) 62.08 ( 64.89%) 53.45 ( 69.77%) 49.51 ( 72.00%) 49.77 ( 71.85%)
Elapsed NUMA02_SMT 163.96 ( 0.00%) 48.92 ( 70.16%) 54.45 ( 66.79%) 48.17 ( 70.62%) 47.71 ( 70.90%) 48.63 ( 70.34%)
CPU NUMA01 4317.00 ( 0.00%) 4473.00 ( -3.61%) 4632.00 ( -7.30%) 4368.00 ( -1.18%) 4446.00 ( -2.99%) 4451.00 ( -3.10%)
CPU NUMA01_THEADLOCAL 4385.00 ( 0.00%) 4609.00 ( -5.11%) 4582.00 ( -4.49%) 4559.00 ( -3.97%) 4514.00 ( -2.94%) 4545.00 ( -3.65%)
CPU NUMA02 3977.00 ( 0.00%) 4111.00 ( -3.37%) 3873.00 ( 2.62%) 4200.00 ( -5.61%) 4212.00 ( -5.91%) 4226.00 ( -6.26%)
CPU NUMA02_SMT 1788.00 ( 0.00%) 2087.00 (-16.72%) 1935.00 ( -8.22%) 2159.00 (-20.75%) 2098.00 (-17.34%) 2089.00 (-16.83%)
Latest numacore has improved on the numa01 case quite a bit. However, this is
the adverse workload. For the workloads that actually do something sensible,
autonuma and balancenuma are both beating numacore by a good margin.
numacores system CPU usage continues to be excessive -- over triple
balancenumas in the numa01 case. Over quadruple in the numa01_threadlocal
case. Double in numa02 and over double in the numa02_smt case.
Duration and vmstats showed nothing interesting so I excluded them this time.
SpecJBB, Multiple JVMs, THP is enabled
======================================
There is no latest numacore figures available because the JVM in two
separate tests crashed with this report
Input Properties:
per_jvm_warehouse_rampup = 3.0
per_jvm_warehouse_rampdown = 20.0
jvm_instances = 4
deterministic_random_seed = false
ramp_up_seconds = 30
measurement_seconds = 240
starting_number_warehouses = 1
increment_number_warehouses = 1
ending_number_warehouses = 24
expected_peak_warehouse = 12
Waiting on instance 1 pid 4028 to finish.
Accepted client /127.0.0.1:59130
Accepted client /127.0.0.1:58393
Accepted client /127.0.0.1:53374
Accepted client /127.0.0.1:40128
java.lang.NullPointerException: error
/root/git-private/autonuma-test/shellpacks/shellpack-bench-specjbb: line 203: 4028 Aborted java $USE_HUGEPAGE $SPECJBB_MAXHEAP spec.jbb.JBBmain -propfile SPECjbb.props -id $INSTANCE > $SHELLPACK_TEMP/jvm-instance-$INSTANCE.log
Waiting on instance 1 pid 4029 to finish.
Exception in thread "main" java.lang.NullPointerException
at spec.jbb.Company.displayResultTotals(Unknown Source)
at spec.jbb.JBBmain.DoARun(Unknown Source)
at spec.jbb.JBBmain.runWarehouse(Unknown Source)
at spec.jbb.JBBmain.doIt(Unknown Source)
at spec.jbb.JBBmain.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
at spec.jbb.Company.displayResultTotals(Unknown Source)
at spec.jbb.JBBmain.DoARun(Unknown Source)
at spec.jbb.JBBmain.runWarehouse(Unknown Source)
at spec.jbb.JBBmain.doIt(Unknown Source)
at spec.jbb.JBBmain.main(Unknown Source)
Exception in thread "main" java.lang.NullPointerException
at spec.jbb.Company.displayResultTotals(Unknown Source)
at spec.jbb.JBBmain.DoARun(Unknown Source)
at spec.jbb.JBBmain.runWarehouse(Unknown Source)
at spec.jbb.JBBmain.doIt(Unknown Source)
at spec.jbb.JBBmain.main(Unknown Source)
Read from remote host compass: Connection reset by peer
Here are the results for the kernels that succeeded
3.7.0-rc7 3.7.0-rc6 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Mean 1 31311.75 ( 0.00%) 27938.00 (-10.77%) 31474.25 ( 0.52%) 31112.00 ( -0.64%) 31281.50 ( -0.10%)
Mean 2 62972.75 ( 0.00%) 51899.00 (-17.58%) 66654.00 ( 5.85%) 62937.50 ( -0.06%) 62483.50 ( -0.78%)
Mean 3 91292.00 ( 0.00%) 80908.00 (-11.37%) 97177.50 ( 6.45%) 90665.50 ( -0.69%) 90667.00 ( -0.68%)
Mean 4 115768.75 ( 0.00%) 99497.25 (-14.06%) 125596.00 ( 8.49%) 116812.50 ( 0.90%) 116193.50 ( 0.37%)
Mean 5 137248.50 ( 0.00%) 92837.75 (-32.36%) 152795.25 ( 11.33%) 139037.75 ( 1.30%) 139055.50 ( 1.32%)
Mean 6 155528.50 ( 0.00%) 105554.50 (-32.13%) 177455.25 ( 14.10%) 155769.25 ( 0.15%) 159129.50 ( 2.32%)
Mean 7 156747.50 ( 0.00%) 122582.25 (-21.80%) 184578.75 ( 17.76%) 157103.25 ( 0.23%) 163234.00 ( 4.14%)
Mean 8 152069.50 ( 0.00%) 122439.00 (-19.48%) 186619.25 ( 22.72%) 157631.00 ( 3.66%) 163077.75 ( 7.24%)
Mean 9 146609.75 ( 0.00%) 112410.00 (-23.33%) 186165.00 ( 26.98%) 152561.00 ( 4.06%) 159656.00 ( 8.90%)
Mean 10 142819.00 ( 0.00%) 111456.00 (-21.96%) 182569.75 ( 27.83%) 145320.00 ( 1.75%) 153414.25 ( 7.42%)
Mean 11 128292.25 ( 0.00%) 98027.00 (-23.59%) 176104.75 ( 37.27%) 138599.50 ( 8.03%) 147194.25 ( 14.73%)
Mean 12 128769.75 ( 0.00%) 129469.50 ( 0.54%) 169003.00 ( 31.24%) 131994.75 ( 2.50%) 140049.75 ( 8.76%)
Mean 13 126488.50 ( 0.00%) 110133.75 (-12.93%) 162725.75 ( 28.65%) 130005.25 ( 2.78%) 139109.75 ( 9.98%)
Mean 14 123400.00 ( 0.00%) 117929.75 ( -4.43%) 163781.25 ( 32.72%) 126340.75 ( 2.38%) 137883.00 ( 11.74%)
Mean 15 122139.50 ( 0.00%) 122404.25 ( 0.22%) 160800.25 ( 31.65%) 128612.75 ( 5.30%) 136624.00 ( 11.86%)
Mean 16 116413.50 ( 0.00%) 124573.50 ( 7.01%) 160882.75 ( 38.20%) 117793.75 ( 1.19%) 134005.75 ( 15.11%)
Mean 17 117263.25 ( 0.00%) 121937.25 ( 3.99%) 159069.75 ( 35.65%) 121991.75 ( 4.03%) 133444.50 ( 13.80%)
Mean 18 117277.00 ( 0.00%) 116633.75 ( -0.55%) 158694.75 ( 35.32%) 119089.75 ( 1.55%) 129650.75 ( 10.55%)
Mean 19 113231.00 ( 0.00%) 111035.75 ( -1.94%) 155563.25 ( 37.39%) 119699.75 ( 5.71%) 123403.25 ( 8.98%)
Mean 20 113628.75 ( 0.00%) 113451.25 ( -0.16%) 154779.75 ( 36.22%) 118400.75 ( 4.20%) 126041.25 ( 10.92%)
Mean 21 110982.50 ( 0.00%) 107660.50 ( -2.99%) 151147.25 ( 36.19%) 115663.25 ( 4.22%) 121906.50 ( 9.84%)
Mean 22 107660.25 ( 0.00%) 104771.50 ( -2.68%) 151180.50 ( 40.42%) 111038.00 ( 3.14%) 125519.00 ( 16.59%)
Mean 23 105320.50 ( 0.00%) 88275.25 (-16.18%) 147032.00 ( 39.60%) 112817.50 ( 7.12%) 124148.25 ( 17.88%)
Mean 24 110900.50 ( 0.00%) 85169.00 (-23.20%) 147407.00 ( 32.92%) 109556.50 ( -1.21%) 122544.00 ( 10.50%)
Stddev 1 720.83 ( 0.00%) 982.31 (-36.28%) 942.80 (-30.79%) 1170.23 (-62.35%) 539.84 ( 25.11%)
Stddev 2 466.00 ( 0.00%) 1770.75 (-279.99%) 1327.32 (-184.83%) 1368.51 (-193.67%) 2103.32 (-351.35%)
Stddev 3 509.61 ( 0.00%) 4849.62 (-851.63%) 1803.72 (-253.94%) 1088.04 (-113.50%) 410.73 ( 19.40%)
Stddev 4 1750.10 ( 0.00%) 10708.16 (-511.86%) 2010.11 (-14.86%) 1456.90 ( 16.75%) 1370.22 ( 21.71%)
Stddev 5 700.05 ( 0.00%) 16497.79 (-2256.66%) 2354.70 (-236.36%) 759.38 ( -8.48%) 1869.54 (-167.06%)
Stddev 6 2259.33 ( 0.00%) 24221.98 (-972.09%) 1516.32 ( 32.89%) 1032.39 ( 54.31%) 1720.87 ( 23.83%)
Stddev 7 3390.99 ( 0.00%) 4721.80 (-39.25%) 2398.34 ( 29.27%) 2487.08 ( 26.66%) 4327.85 (-27.63%)
Stddev 8 7533.18 ( 0.00%) 8609.90 (-14.29%) 2895.55 ( 61.56%) 3902.53 ( 48.20%) 2536.68 ( 66.33%)
Stddev 9 9223.98 ( 0.00%) 10731.70 (-16.35%) 4726.23 ( 48.76%) 5673.20 ( 38.50%) 3377.59 ( 63.38%)
Stddev 10 4578.09 ( 0.00%) 11136.27 (-143.25%) 6705.48 (-46.47%) 5516.47 (-20.50%) 7227.58 (-57.87%)
Stddev 11 8201.30 ( 0.00%) 3580.27 ( 56.35%) 10915.90 (-33.10%) 4757.42 ( 41.99%) 4056.02 ( 50.54%)
Stddev 12 5713.70 ( 0.00%) 13923.12 (-143.68%) 16555.64 (-189.75%) 4573.05 ( 19.96%) 3678.89 ( 35.61%)
Stddev 13 5878.95 ( 0.00%) 10471.09 (-78.11%) 18628.01 (-216.86%) 1680.65 ( 71.41%) 3947.39 ( 32.86%)
Stddev 14 4783.95 ( 0.00%) 4051.35 ( 15.31%) 18324.63 (-283.04%) 2637.82 ( 44.86%) 4806.09 ( -0.46%)
Stddev 15 6281.48 ( 0.00%) 3357.07 ( 46.56%) 17654.58 (-181.06%) 2003.38 ( 68.11%) 3005.22 ( 52.16%)
Stddev 16 6948.12 ( 0.00%) 3763.32 ( 45.84%) 18280.52 (-163.10%) 3526.10 ( 49.25%) 3309.24 ( 52.37%)
Stddev 17 5603.77 ( 0.00%) 1452.04 ( 74.09%) 18230.53 (-225.33%) 1712.95 ( 69.43%) 3516.09 ( 37.25%)
Stddev 18 6200.90 ( 0.00%) 1870.12 ( 69.84%) 18486.73 (-198.13%) 751.36 ( 87.88%) 2412.60 ( 61.09%)
Stddev 19 6726.31 ( 0.00%) 1045.21 ( 84.46%) 18465.25 (-174.52%) 1750.49 ( 73.98%) 4482.82 ( 33.35%)
Stddev 20 5713.58 ( 0.00%) 2066.90 ( 63.82%) 19947.77 (-249.13%) 1892.91 ( 66.87%) 2612.62 ( 54.27%)
Stddev 21 4566.92 ( 0.00%) 2460.40 ( 46.13%) 21189.08 (-363.97%) 3639.75 ( 20.30%) 1963.17 ( 57.01%)
Stddev 22 6168.05 ( 0.00%) 2770.81 ( 55.08%) 20033.82 (-224.80%) 3682.20 ( 40.30%) 1159.17 ( 81.21%)
Stddev 23 6295.45 ( 0.00%) 1337.32 ( 78.76%) 22610.91 (-259.16%) 2013.53 ( 68.02%) 3842.61 ( 38.96%)
Stddev 24 3108.17 ( 0.00%) 1381.20 ( 55.56%) 21243.56 (-583.47%) 4044.16 (-30.11%) 2673.39 ( 13.99%)
TPut 1 125247.00 ( 0.00%) 111752.00 (-10.77%) 125897.00 ( 0.52%) 124448.00 ( -0.64%) 125126.00 ( -0.10%)
TPut 2 251891.00 ( 0.00%) 207596.00 (-17.58%) 266616.00 ( 5.85%) 251750.00 ( -0.06%) 249934.00 ( -0.78%)
TPut 3 365168.00 ( 0.00%) 323632.00 (-11.37%) 388710.00 ( 6.45%) 362662.00 ( -0.69%) 362668.00 ( -0.68%)
TPut 4 463075.00 ( 0.00%) 397989.00 (-14.06%) 502384.00 ( 8.49%) 467250.00 ( 0.90%) 464774.00 ( 0.37%)
TPut 5 548994.00 ( 0.00%) 371351.00 (-32.36%) 611181.00 ( 11.33%) 556151.00 ( 1.30%) 556222.00 ( 1.32%)
TPut 6 622114.00 ( 0.00%) 422218.00 (-32.13%) 709821.00 ( 14.10%) 623077.00 ( 0.15%) 636518.00 ( 2.32%)
TPut 7 626990.00 ( 0.00%) 490329.00 (-21.80%) 738315.00 ( 17.76%) 628413.00 ( 0.23%) 652936.00 ( 4.14%)
TPut 8 608278.00 ( 0.00%) 489756.00 (-19.48%) 746477.00 ( 22.72%) 630524.00 ( 3.66%) 652311.00 ( 7.24%)
TPut 9 586439.00 ( 0.00%) 449640.00 (-23.33%) 744660.00 ( 26.98%) 610244.00 ( 4.06%) 638624.00 ( 8.90%)
TPut 10 571276.00 ( 0.00%) 445824.00 (-21.96%) 730279.00 ( 27.83%) 581280.00 ( 1.75%) 613657.00 ( 7.42%)
TPut 11 513169.00 ( 0.00%) 392108.00 (-23.59%) 704419.00 ( 37.27%) 554398.00 ( 8.03%) 588777.00 ( 14.73%)
TPut 12 515079.00 ( 0.00%) 517878.00 ( 0.54%) 676012.00 ( 31.24%) 527979.00 ( 2.50%) 560199.00 ( 8.76%)
TPut 13 505954.00 ( 0.00%) 440535.00 (-12.93%) 650903.00 ( 28.65%) 520021.00 ( 2.78%) 556439.00 ( 9.98%)
TPut 14 493600.00 ( 0.00%) 471719.00 ( -4.43%) 655125.00 ( 32.72%) 505363.00 ( 2.38%) 551532.00 ( 11.74%)
TPut 15 488558.00 ( 0.00%) 489617.00 ( 0.22%) 643201.00 ( 31.65%) 514451.00 ( 5.30%) 546496.00 ( 11.86%)
TPut 16 465654.00 ( 0.00%) 498294.00 ( 7.01%) 643531.00 ( 38.20%) 471175.00 ( 1.19%) 536023.00 ( 15.11%)
TPut 17 469053.00 ( 0.00%) 487749.00 ( 3.99%) 636279.00 ( 35.65%) 487967.00 ( 4.03%) 533778.00 ( 13.80%)
TPut 18 469108.00 ( 0.00%) 466535.00 ( -0.55%) 634779.00 ( 35.32%) 476359.00 ( 1.55%) 518603.00 ( 10.55%)
TPut 19 452924.00 ( 0.00%) 444143.00 ( -1.94%) 622253.00 ( 37.39%) 478799.00 ( 5.71%) 493613.00 ( 8.98%)
TPut 20 454515.00 ( 0.00%) 453805.00 ( -0.16%) 619119.00 ( 36.22%) 473603.00 ( 4.20%) 504165.00 ( 10.92%)
TPut 21 443930.00 ( 0.00%) 430642.00 ( -2.99%) 604589.00 ( 36.19%) 462653.00 ( 4.22%) 487626.00 ( 9.84%)
TPut 22 430641.00 ( 0.00%) 419086.00 ( -2.68%) 604722.00 ( 40.42%) 444152.00 ( 3.14%) 502076.00 ( 16.59%)
TPut 23 421282.00 ( 0.00%) 353101.00 (-16.18%) 588128.00 ( 39.60%) 451270.00 ( 7.12%) 496593.00 ( 17.88%)
TPut 24 443602.00 ( 0.00%) 340676.00 (-23.20%) 589628.00 ( 32.92%) 438226.00 ( -1.21%) 490176.00 ( 10.50%)
numacore v17 regressed but we knew that already.
autonuma does the best overall
balancenuma does all right and the scalability patches help quite a bit.
SPECJBB PEAKS
3.7.0-rc7 3.7.0-rc6 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%)
Expctd Peak Bops 515079.00 ( 0.00%) 517878.00 ( 0.54%) 676012.00 ( 31.24%) 527979.00 ( 2.50%) 560199.00 ( 8.76%)
Actual Warehouse 7.00 ( 0.00%) 12.00 ( 71.43%) 8.00 ( 14.29%) 8.00 ( 14.29%) 7.00 ( 0.00%)
Actual Peak Bops 626990.00 ( 0.00%) 517878.00 (-17.40%) 746477.00 ( 19.06%) 630524.00 ( 0.56%) 652936.00 ( 4.14%)
SpecJBB Bops 465685.00 ( 0.00%) 447214.00 ( -3.97%) 628328.00 ( 34.93%) 480925.00 ( 3.27%) 521332.00 ( 11.95%)
SpecJBB Bops/JVM 116421.00 ( 0.00%) 111804.00 ( -3.97%) 157082.00 ( 34.93%) 120231.00 ( 3.27%) 130333.00 ( 11.95%)
numacore is pretty old here so ignore the regression.
autonuma is the best but balancenuma sees some of the performance gain.
MMTests Statistics: vmstat
3.7.0-rc7 3.7.0-rc6 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins 37116 36404 36740 35664 34832
Page Outs 30340 33624 29428 29656 30320
Swap Ins 0 0 0 0 0
Swap Outs 0 0 0 0 0
Direct pages scanned 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0
Kswapd efficiency 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0
Page writes file 0 0 0 0 0
Page writes anon 0 0 0 0 0
Page reclaim immediate 0 0 0 0 0
Page rescued immediate 0 0 0 0 0
Slabs scanned 0 0 0 0 0
Direct inode steals 0 0 0 0 0
Kswapd inode steals 0 0 0 0 0
Kswapd skipped wait 0 0 0 0 0
THP fault alloc 63322 49889 52514 65794 66963
THP collapse alloc 130 53 463 128 121
THP splits 355 192 376 371 362
THP fault fallback 0 0 0 0 0
THP collapse fail 0 0 0 0 0
Compaction stalls 0 0 0 0 0
Compaction success 0 0 0 0 0
Compaction failures 0 0 0 0 0
Page migrate success 0 0 0 51424061 50195011
Page migrate failure 0 0 0 0 0
Compaction pages isolated 0 0 0 0 0
Compaction migrate scanned 0 0 0 0 0
Compaction free scanned 0 0 0 0 0
Compaction cost 0 0 0 53378 52102
NUMA PTE updates 0 0 0 411047238 404964644
NUMA hint faults 0 0 0 3077302 3075026
NUMA hint local faults 0 0 0 958617 870171
NUMA pages migrated 0 0 0 51424061 50195011
AutoNUMA cost 0 0 0 19240 19163
All it shows really is that THP is enabled and that balancenuma is migrating
more than I'd like -- 48MB/sec on average throughout the test.
SpecJBB, Multiple JVMs, THP is disabled
=======================================
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Mean 1 26036.50 ( 0.00%) 19595.00 (-24.74%) 23791.25 ( -8.62%) 24738.25 ( -4.99%) 25595.00 ( -1.70%) 25610.50 ( -1.64%)
Mean 2 53629.75 ( 0.00%) 38481.50 (-28.25%) 46966.75 (-12.42%) 55646.75 ( 3.76%) 53045.25 ( -1.09%) 53383.00 ( -0.46%)
Mean 3 77385.00 ( 0.00%) 53685.50 (-30.63%) 66913.25 (-13.53%) 82714.75 ( 6.89%) 76596.00 ( -1.02%) 76502.75 ( -1.14%)
Mean 4 100097.75 ( 0.00%) 68253.50 (-31.81%) 72186.50 (-27.88%) 107883.25 ( 7.78%) 98618.00 ( -1.48%) 99786.50 ( -0.31%)
Mean 5 119012.75 ( 0.00%) 74164.50 (-37.68%) 72126.50 (-39.40%) 130260.25 ( 9.45%) 119354.50 ( 0.29%) 121741.75 ( 2.29%)
Mean 6 137419.25 ( 0.00%) 86158.50 (-37.30%) 52123.00 (-62.07%) 154244.50 ( 12.24%) 136901.75 ( -0.38%) 136990.50 ( -0.31%)
Mean 7 138018.25 ( 0.00%) 96059.25 (-30.40%) 55582.50 (-59.73%) 159501.00 ( 15.57%) 138265.50 ( 0.18%) 139398.75 ( 1.00%)
Mean 8 136774.00 ( 0.00%) 97003.50 (-29.08%) 30208.25 (-77.91%) 162868.00 ( 19.08%) 138554.50 ( 1.30%) 137340.75 ( 0.41%)
Mean 9 127966.50 ( 0.00%) 95261.00 (-25.56%) 125900.50 ( -1.61%) 163008.00 ( 27.38%) 137954.00 ( 7.80%) 134200.50 ( 4.87%)
Mean 10 124628.75 ( 0.00%) 96202.25 (-22.81%) 73809.00 (-40.78%) 159696.50 ( 28.14%) 131322.25 ( 5.37%) 126927.50 ( 1.84%)
Mean 11 117269.00 ( 0.00%) 95924.25 (-18.20%) 127804.25 ( 8.98%) 154701.50 ( 31.92%) 125032.75 ( 6.62%) 122925.00 ( 4.82%)
Mean 12 111962.25 ( 0.00%) 94247.25 (-15.82%) 146580.25 ( 30.92%) 150936.50 ( 34.81%) 118119.50 ( 5.50%) 119931.75 ( 7.12%)
Mean 13 111595.50 ( 0.00%) 106538.50 ( -4.53%) 134462.75 ( 20.49%) 147193.25 ( 31.90%) 116398.75 ( 4.30%) 117349.75 ( 5.16%)
Mean 14 110881.00 ( 0.00%) 103549.00 ( -6.61%) 137573.25 ( 24.07%) 144584.00 ( 30.40%) 114934.50 ( 3.66%) 115838.25 ( 4.47%)
Mean 15 109337.50 ( 0.00%) 101729.00 ( -6.96%) 139722.50 ( 27.79%) 143333.00 ( 31.09%) 115523.75 ( 5.66%) 115151.25 ( 5.32%)
Mean 16 107031.75 ( 0.00%) 101983.75 ( -4.72%) 121221.75 ( 13.26%) 141907.75 ( 32.58%) 113666.00 ( 6.20%) 113673.50 ( 6.21%)
Mean 17 105491.25 ( 0.00%) 100205.75 ( -5.01%) 129429.75 ( 22.69%) 140691.00 ( 33.37%) 112751.50 ( 6.88%) 113221.25 ( 7.33%)
Mean 18 101102.75 ( 0.00%) 96635.50 ( -4.42%) 115086.50 ( 13.83%) 137784.25 ( 36.28%) 112582.50 ( 11.35%) 111533.50 ( 10.32%)
Mean 19 103907.25 ( 0.00%) 94578.25 ( -8.98%) 126392.75 ( 21.64%) 135719.25 ( 30.62%) 110152.25 ( 6.01%) 113959.25 ( 9.67%)
Mean 20 100496.00 ( 0.00%) 92683.75 ( -7.77%) 123318.75 ( 22.71%) 135264.25 ( 34.60%) 108861.50 ( 8.32%) 113746.00 ( 13.18%)
Mean 21 99570.00 ( 0.00%) 92955.75 ( -6.64%) 111293.00 ( 11.77%) 133891.00 ( 34.47%) 110094.00 ( 10.57%) 109462.50 ( 9.94%)
Mean 22 98611.75 ( 0.00%) 89781.75 ( -8.95%) 118218.50 ( 19.88%) 132399.75 ( 34.26%) 109322.75 ( 10.86%) 110502.75 ( 12.06%)
Mean 23 98173.00 ( 0.00%) 88846.00 ( -9.50%) 118210.00 ( 20.41%) 130726.00 ( 33.16%) 106046.25 ( 8.02%) 107304.25 ( 9.30%)
Mean 24 92074.75 ( 0.00%) 88581.00 ( -3.79%) 111965.00 ( 21.60%) 127552.25 ( 38.53%) 102362.00 ( 11.17%) 107119.25 ( 16.34%)
Stddev 1 735.13 ( 0.00%) 538.24 ( 26.78%) 854.37 (-16.22%) 121.08 ( 83.53%) 906.62 (-23.33%) 788.06 ( -7.20%)
Stddev 2 406.26 ( 0.00%) 3458.87 (-751.39%) 4220.03 (-938.75%) 477.32 (-17.49%) 1322.57 (-225.55%) 468.57 (-15.34%)
Stddev 3 644.20 ( 0.00%) 1360.89 (-111.25%) 2573.27 (-299.45%) 922.47 (-43.20%) 609.27 ( 5.42%) 599.26 ( 6.98%)
Stddev 4 743.93 ( 0.00%) 2149.34 (-188.92%) 14533.01 (-1853.53%) 1385.42 (-86.23%) 1119.02 (-50.42%) 801.13 ( -7.69%)
Stddev 5 898.53 ( 0.00%) 2521.01 (-180.57%) 15303.97 (-1603.23%) 763.24 ( 15.06%) 942.52 ( -4.90%) 1718.19 (-91.22%)
Stddev 6 1126.61 ( 0.00%) 3818.22 (-238.91%) 23616.59 (-1996.26%) 1527.03 (-35.54%) 2445.69 (-117.08%) 1754.32 (-55.72%)
Stddev 7 2907.61 ( 0.00%) 4419.29 (-51.99%) 29664.97 (-920.25%) 1536.66 ( 47.15%) 4881.65 (-67.89%) 4863.83 (-67.28%)
Stddev 8 3200.64 ( 0.00%) 382.01 ( 88.06%) 10743.99 (-235.68%) 1228.09 ( 61.63%) 5459.06 (-70.56%) 5583.95 (-74.46%)
Stddev 9 2907.92 ( 0.00%) 1813.39 ( 37.64%) 11763.90 (-304.55%) 1502.61 ( 48.33%) 2501.16 ( 13.99%) 2525.02 ( 13.17%)
Stddev 10 5093.23 ( 0.00%) 1313.58 ( 74.21%) 34926.95 (-585.75%) 2763.19 ( 45.75%) 2973.78 ( 41.61%) 2005.95 ( 60.62%)
Stddev 11 4982.41 ( 0.00%) 1163.02 ( 76.66%) 13792.07 (-176.81%) 4776.28 ( 4.14%) 6068.34 (-21.80%) 4256.77 ( 14.56%)
Stddev 12 3051.38 ( 0.00%) 2117.59 ( 30.60%) 5819.48 (-90.72%) 9252.59 (-203.23%) 3885.96 (-27.35%) 2580.44 ( 15.43%)
Stddev 13 2918.03 ( 0.00%) 2252.11 ( 22.82%) 8340.05 (-185.81%) 9384.83 (-221.62%) 1833.07 ( 37.18%) 2523.28 ( 13.53%)
Stddev 14 3178.97 ( 0.00%) 2337.49 ( 26.47%) 6166.98 (-93.99%) 9353.03 (-194.22%) 1072.60 ( 66.26%) 1140.55 ( 64.12%)
Stddev 15 2438.31 ( 0.00%) 1707.72 ( 29.96%) 10687.74 (-338.33%) 10494.03 (-330.38%) 2295.76 ( 5.85%) 1213.75 ( 50.22%)
Stddev 16 2682.25 ( 0.00%) 840.47 ( 68.67%) 10963.32 (-308.74%) 10343.25 (-285.62%) 2416.09 ( 9.92%) 1697.27 ( 36.72%)
Stddev 17 2807.66 ( 0.00%) 1546.16 ( 44.93%) 10755.81 (-283.09%) 11446.15 (-307.68%) 2484.08 ( 11.52%) 563.50 ( 79.93%)
Stddev 18 3049.27 ( 0.00%) 934.11 ( 69.37%) 8523.80 (-179.54%) 11779.80 (-286.31%) 1472.27 ( 51.72%) 1533.68 ( 49.70%)
Stddev 19 2782.65 ( 0.00%) 735.28 ( 73.58%) 9045.84 (-225.08%) 11416.35 (-310.27%) 514.78 ( 81.50%) 1283.38 ( 53.88%)
Stddev 20 2379.12 ( 0.00%) 956.25 ( 59.81%) 3789.62 (-59.29%) 10511.63 (-341.83%) 1641.25 ( 31.01%) 1758.22 ( 26.10%)
Stddev 21 2975.22 ( 0.00%) 438.31 ( 85.27%) 8160.39 (-174.28%) 11292.91 (-279.57%) 1087.60 ( 63.44%) 434.51 ( 85.40%)
Stddev 22 2260.61 ( 0.00%) 718.23 ( 68.23%) 10418.90 (-360.89%) 11993.84 (-430.56%) 909.16 ( 59.78%) 322.32 ( 85.74%)
Stddev 23 2900.85 ( 0.00%) 275.47 ( 90.50%) 9829.57 (-238.85%) 12234.80 (-321.77%) 701.39 ( 75.82%) 1444.19 ( 50.21%)
Stddev 24 2578.98 ( 0.00%) 481.68 ( 81.32%) 7696.37 (-198.43%) 12769.61 (-395.14%) 732.56 ( 71.60%) 1777.60 ( 31.07%)
TPut 1 104146.00 ( 0.00%) 78380.00 (-24.74%) 95165.00 ( -8.62%) 98953.00 ( -4.99%) 102380.00 ( -1.70%) 102442.00 ( -1.64%)
TPut 2 214519.00 ( 0.00%) 153926.00 (-28.25%) 187867.00 (-12.42%) 222587.00 ( 3.76%) 212181.00 ( -1.09%) 213532.00 ( -0.46%)
TPut 3 309540.00 ( 0.00%) 214742.00 (-30.63%) 267653.00 (-13.53%) 330859.00 ( 6.89%) 306384.00 ( -1.02%) 306011.00 ( -1.14%)
TPut 4 400391.00 ( 0.00%) 273014.00 (-31.81%) 288746.00 (-27.88%) 431533.00 ( 7.78%) 394472.00 ( -1.48%) 399146.00 ( -0.31%)
TPut 5 476051.00 ( 0.00%) 296658.00 (-37.68%) 288506.00 (-39.40%) 521041.00 ( 9.45%) 477418.00 ( 0.29%) 486967.00 ( 2.29%)
TPut 6 549677.00 ( 0.00%) 344634.00 (-37.30%) 208492.00 (-62.07%) 616978.00 ( 12.24%) 547607.00 ( -0.38%) 547962.00 ( -0.31%)
TPut 7 552073.00 ( 0.00%) 384237.00 (-30.40%) 222330.00 (-59.73%) 638004.00 ( 15.57%) 553062.00 ( 0.18%) 557595.00 ( 1.00%)
TPut 8 547096.00 ( 0.00%) 388014.00 (-29.08%) 120833.00 (-77.91%) 651472.00 ( 19.08%) 554218.00 ( 1.30%) 549363.00 ( 0.41%)
TPut 9 511866.00 ( 0.00%) 381044.00 (-25.56%) 503602.00 ( -1.61%) 652032.00 ( 27.38%) 551816.00 ( 7.80%) 536802.00 ( 4.87%)
TPut 10 498515.00 ( 0.00%) 384809.00 (-22.81%) 295236.00 (-40.78%) 638786.00 ( 28.14%) 525289.00 ( 5.37%) 507710.00 ( 1.84%)
TPut 11 469076.00 ( 0.00%) 383697.00 (-18.20%) 511217.00 ( 8.98%) 618806.00 ( 31.92%) 500131.00 ( 6.62%) 491700.00 ( 4.82%)
TPut 12 447849.00 ( 0.00%) 376989.00 (-15.82%) 586321.00 ( 30.92%) 603746.00 ( 34.81%) 472478.00 ( 5.50%) 479727.00 ( 7.12%)
TPut 13 446382.00 ( 0.00%) 426154.00 ( -4.53%) 537851.00 ( 20.49%) 588773.00 ( 31.90%) 465595.00 ( 4.30%) 469399.00 ( 5.16%)
TPut 14 443524.00 ( 0.00%) 414196.00 ( -6.61%) 550293.00 ( 24.07%) 578336.00 ( 30.40%) 459738.00 ( 3.66%) 463353.00 ( 4.47%)
TPut 15 437350.00 ( 0.00%) 406916.00 ( -6.96%) 558890.00 ( 27.79%) 573332.00 ( 31.09%) 462095.00 ( 5.66%) 460605.00 ( 5.32%)
TPut 16 428127.00 ( 0.00%) 407935.00 ( -4.72%) 484887.00 ( 13.26%) 567631.00 ( 32.58%) 454664.00 ( 6.20%) 454694.00 ( 6.21%)
TPut 17 421965.00 ( 0.00%) 400823.00 ( -5.01%) 517719.00 ( 22.69%) 562764.00 ( 33.37%) 451006.00 ( 6.88%) 452885.00 ( 7.33%)
TPut 18 404411.00 ( 0.00%) 386542.00 ( -4.42%) 460346.00 ( 13.83%) 551137.00 ( 36.28%) 450330.00 ( 11.35%) 446134.00 ( 10.32%)
TPut 19 415629.00 ( 0.00%) 378313.00 ( -8.98%) 505571.00 ( 21.64%) 542877.00 ( 30.62%) 440609.00 ( 6.01%) 455837.00 ( 9.67%)
TPut 20 401984.00 ( 0.00%) 370735.00 ( -7.77%) 493275.00 ( 22.71%) 541057.00 ( 34.60%) 435446.00 ( 8.32%) 454984.00 ( 13.18%)
TPut 21 398280.00 ( 0.00%) 371823.00 ( -6.64%) 445172.00 ( 11.77%) 535564.00 ( 34.47%) 440376.00 ( 10.57%) 437850.00 ( 9.94%)
TPut 22 394447.00 ( 0.00%) 359127.00 ( -8.95%) 472874.00 ( 19.88%) 529599.00 ( 34.26%) 437291.00 ( 10.86%) 442011.00 ( 12.06%)
TPut 23 392692.00 ( 0.00%) 355384.00 ( -9.50%) 472840.00 ( 20.41%) 522904.00 ( 33.16%) 424185.00 ( 8.02%) 429217.00 ( 9.30%)
TPut 24 368299.00 ( 0.00%) 354324.00 ( -3.79%) 447860.00 ( 21.60%) 510209.00 ( 38.53%) 409448.00 ( 11.17%) 428477.00 ( 16.34%)
Latest numacore has improved dramatically here. In v17, it was regressing
heavily across the board. The latest figures show that it regresses heavily
for small numbers of warehouses and shows very large performance gains
for larger numbers of warehouses. This problem with regressions for smaller
numbers of warehouses has been reported repeatedly and it has been pointed out
multiple times that specjbb by default ignores these results which can be
very misleading.
autonuma shows large gains even for small numbers of warehouses and larger
performnace gains than numacore does. This is without the TLB optimisations.
balancenuma is not great, but it's better than mainline.
SPECJBB PEAKS
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Expctd Warehouse 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%) 12.00 ( 0.00%)
Expctd Peak Bops 447849.00 ( 0.00%) 376989.00 (-15.82%) 586321.00 ( 30.92%) 603746.00 ( 34.81%) 472478.00 ( 5.50%) 479727.00 ( 7.12%)
Actual Warehouse 7.00 ( 0.00%) 13.00 ( 85.71%) 12.00 ( 71.43%) 9.00 ( 28.57%) 8.00 ( 14.29%) 7.00 ( 0.00%)
Actual Peak Bops 552073.00 ( 0.00%) 426154.00 (-22.81%) 586321.00 ( 6.20%) 652032.00 ( 18.11%) 554218.00 ( 0.39%) 557595.00 ( 1.00%)
SpecJBB Bops 415458.00 ( 0.00%) 385328.00 ( -7.25%) 502608.00 ( 20.98%) 554456.00 ( 33.46%) 446405.00 ( 7.45%) 451937.00 ( 8.78%)
SpecJBB Bops/JVM 103865.00 ( 0.00%) 96332.00 ( -7.25%) 125652.00 ( 20.98%) 138614.00 ( 33.46%) 111601.00 ( 7.45%) 112984.00 ( 8.78%)
numacore is showing good performance gains both at the peak and in the
specjbb score. Note that the specjbb score ignored the regressions for
smaller numbers of warehouses.
autonuma was still better.
balancenuma was all right, better than mainline.
MMTests Statistics: duration
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User 177832.71 148340.09 165197.46 177337.90 176411.93 176466.36
System 89.07 28052.02 12438.18 287.31 1464.93 1467.74
Elapsed 4035.81 4041.26 4038.34 4028.05 4041.53 4031.74
numacores system CPU usage is incredibly high -- over 8 times higher
than balancenumas.
balancenumas system CPU usage also sucks to be honest.
MMTests Statistics: vmstat
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins 37380 66040 34576 36416 35452 34948
Page Outs 29224 46900 31972 29584 29612 30892
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0
Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0
Page writes file 0 0 0 0 0 0
Page writes anon 0 0 0 0 0 0
Page reclaim immediate 0 0 0 0 0 0
Page rescued immediate 0 0 0 0 0 0
Slabs scanned 0 0 0 0 0 0
Direct inode steals 0 0 0 0 0 0
Kswapd inode steals 0 0 0 0 0 0
Kswapd skipped wait 0 0 0 0 0 0
THP fault alloc 2 3 1 2 2 2
THP collapse alloc 0 0 0 0 0 0
THP splits 0 0 0 0 0 0
THP fault fallback 0 0 0 0 0 0
THP collapse fail 0 0 0 0 0 0
Compaction stalls 0 0 0 0 0 0
Compaction success 0 0 0 0 0 0
Compaction failures 0 0 0 0 0 0
Page migrate success 0 0 193988041 0 37611432 39796961
Page migrate failure 0 0 0 0 0 0
Compaction pages isolated 0 0 0 0 0 0
Compaction migrate scanned 0 0 0 0 0 0
Compaction free scanned 0 0 0 0 0 0
Compaction cost 0 0 201359 0 39040 41309
NUMA PTE updates 0 0 904384590 0 288455303 286931926
NUMA hint faults 0 0 0 0 270103189 269176121
NUMA hint local faults 0 0 0 0 70822016 70400386
NUMA pages migrated 0 0 193988041 0 37611432 39796961
AutoNUMA cost 0 0 10016 0 1353249 1348645
According to this, numacore never had a NUMA fault. This is completely broken
obviously and it's because PTE NUMA hinting faults are not accounted for
by numacore because that path does not call numa_migration_target(). The
consequences are not that great, it just means that the notional "AutoNUMA
cost" is meaningless for numacore.
What is interesting is numacores migration rate -- 187MB/sec on average. This
is over quadruple balancenumas migration rate of 38MB/sec on average.
SpecJBB, Single JVM, THP is enabled
===================================
As with the Multiple JVM test with THP enabled, numacore crashes. This
time the message is
Timing Measurement began Sun Dec 09 17:12:53 GMT 2012 for 0.5 minutes
Exception in thread "Thread-1040" java.lang.NullPointerException
at java.util.TreeMap.access$100(Unknown Source)
at java.util.TreeMap$PrivateEntryIterator.nextEntry(Unknown Source)
at java.util.TreeMap$ValueIterator.next(Unknown Source)
at spec.jbb.DeliveryTransaction.preprocess(Unknown Source)
at spec.jbb.DeliveryHandler.handleDelivery(Unknown Source)
at spec.jbb.DeliveryTransaction.process(Unknown Source)
at spec.jbb.TransactionManager.runTxn(Unknown Source)
at spec.jbb.TransactionManager.goManual(Unknown Source)
at spec.jbb.TransactionManager.go(Unknown Source)
at spec.jbb.JBBmain.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Timing Measurement ended Sun Dec 09 17:13:23 GMT 2012
Here are the rest of the resutls
3.7.0-rc7 3.7.0-rc6 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
TPut 1 25550.00 ( 0.00%) 25491.00 ( -0.23%) 24233.00 ( -5.15%) 24913.00 ( -2.49%) 26480.00 ( 3.64%)
TPut 2 55943.00 ( 0.00%) 51630.00 ( -7.71%) 55312.00 ( -1.13%) 55042.00 ( -1.61%) 56920.00 ( 1.75%)
TPut 3 87707.00 ( 0.00%) 74497.00 (-15.06%) 88569.00 ( 0.98%) 86135.00 ( -1.79%) 88608.00 ( 1.03%)
TPut 4 117911.00 ( 0.00%) 98435.00 (-16.52%) 118561.00 ( 0.55%) 117486.00 ( -0.36%) 117953.00 ( 0.04%)
TPut 5 143285.00 ( 0.00%) 133964.00 ( -6.51%) 145703.00 ( 1.69%) 142821.00 ( -0.32%) 144926.00 ( 1.15%)
TPut 6 171208.00 ( 0.00%) 152795.00 (-10.75%) 171006.00 ( -0.12%) 170635.00 ( -0.33%) 169394.00 ( -1.06%)
TPut 7 195635.00 ( 0.00%) 162517.00 (-16.93%) 198699.00 ( 1.57%) 196108.00 ( 0.24%) 196491.00 ( 0.44%)
TPut 8 222655.00 ( 0.00%) 168679.00 (-24.24%) 224903.00 ( 1.01%) 223494.00 ( 0.38%) 225978.00 ( 1.49%)
TPut 9 244787.00 ( 0.00%) 193394.00 (-20.99%) 248313.00 ( 1.44%) 251858.00 ( 2.89%) 251569.00 ( 2.77%)
TPut 10 271565.00 ( 0.00%) 237987.00 (-12.36%) 272148.00 ( 0.21%) 275869.00 ( 1.58%) 279049.00 ( 2.76%)
TPut 11 298270.00 ( 0.00%) 207908.00 (-30.30%) 303749.00 ( 1.84%) 301763.00 ( 1.17%) 301399.00 ( 1.05%)
TPut 12 320867.00 ( 0.00%) 257937.00 (-19.61%) 327808.00 ( 2.16%) 329681.00 ( 2.75%) 330506.00 ( 3.00%)
TPut 13 343514.00 ( 0.00%) 248474.00 (-27.67%) 349080.00 ( 1.62%) 340606.00 ( -0.85%) 350817.00 ( 2.13%)
TPut 14 365321.00 ( 0.00%) 298876.00 (-18.19%) 370026.00 ( 1.29%) 379939.00 ( 4.00%) 361752.00 ( -0.98%)
TPut 15 377071.00 ( 0.00%) 296562.00 (-21.35%) 329847.00 (-12.52%) 395421.00 ( 4.87%) 396091.00 ( 5.04%)
TPut 16 404979.00 ( 0.00%) 287964.00 (-28.89%) 411066.00 ( 1.50%) 420551.00 ( 3.85%) 411673.00 ( 1.65%)
TPut 17 420593.00 ( 0.00%) 342590.00 (-18.55%) 428242.00 ( 1.82%) 437461.00 ( 4.01%) 428270.00 ( 1.83%)
TPut 18 440178.00 ( 0.00%) 377508.00 (-14.24%) 440392.00 ( 0.05%) 455014.00 ( 3.37%) 447671.00 ( 1.70%)
TPut 19 448876.00 ( 0.00%) 397727.00 (-11.39%) 462036.00 ( 2.93%) 479223.00 ( 6.76%) 461881.00 ( 2.90%)
TPut 20 460513.00 ( 0.00%) 411831.00 (-10.57%) 476437.00 ( 3.46%) 493176.00 ( 7.09%) 474824.00 ( 3.11%)
TPut 21 474161.00 ( 0.00%) 442153.00 ( -6.75%) 487513.00 ( 2.82%) 505246.00 ( 6.56%) 468938.00 ( -1.10%)
TPut 22 474493.00 ( 0.00%) 429921.00 ( -9.39%) 487920.00 ( 2.83%) 527360.00 ( 11.14%) 475208.00 ( 0.15%)
TPut 23 489559.00 ( 0.00%) 460354.00 ( -5.97%) 508298.00 ( 3.83%) 534820.00 ( 9.25%) 490743.00 ( 0.24%)
TPut 24 495378.00 ( 0.00%) 486826.00 ( -1.73%) 514403.00 ( 3.84%) 545294.00 ( 10.08%) 493974.00 ( -0.28%)
TPut 25 491795.00 ( 0.00%) 520474.00 ( 5.83%) 507373.00 ( 3.17%) 543526.00 ( 10.52%) 489850.00 ( -0.40%)
TPut 26 490038.00 ( 0.00%) 465587.00 ( -4.99%) 376322.00 (-23.21%) 545175.00 ( 11.25%) 491352.00 ( 0.27%)
TPut 27 491233.00 ( 0.00%) 469764.00 ( -4.37%) 366225.00 (-25.45%) 536927.00 ( 9.30%) 489611.00 ( -0.33%)
TPut 28 489058.00 ( 0.00%) 489561.00 ( 0.10%) 414027.00 (-15.34%) 543127.00 ( 11.06%) 473835.00 ( -3.11%)
TPut 29 471539.00 ( 0.00%) 492496.00 ( 4.44%) 400529.00 (-15.06%) 541615.00 ( 14.86%) 486009.00 ( 3.07%)
TPut 30 480343.00 ( 0.00%) 488349.00 ( 1.67%) 405612.00 (-15.56%) 542904.00 ( 13.02%) 478384.00 ( -0.41%)
TPut 31 478109.00 ( 0.00%) 460043.00 ( -3.78%) 401471.00 (-16.03%) 529079.00 ( 10.66%) 466457.00 ( -2.44%)
TPut 32 475736.00 ( 0.00%) 472007.00 ( -0.78%) 401075.00 (-15.69%) 532423.00 ( 11.92%) 467866.00 ( -1.65%)
TPut 33 470758.00 ( 0.00%) 474348.00 ( 0.76%) 399592.00 (-15.12%) 518811.00 ( 10.21%) 464764.00 ( -1.27%)
TPut 34 467304.00 ( 0.00%) 475878.00 ( 1.83%) 394589.00 (-15.56%) 518334.00 ( 10.92%) 446719.00 ( -4.41%)
TPut 35 466391.00 ( 0.00%) 487411.00 ( 4.51%) 382799.00 (-17.92%) 513591.00 ( 10.12%) 447071.00 ( -4.14%)
TPut 36 452722.00 ( 0.00%) 478050.00 ( 5.59%) 381120.00 (-15.82%) 503801.00 ( 11.28%) 452243.00 ( -0.11%)
TPut 37 447878.00 ( 0.00%) 478467.00 ( 6.83%) 382803.00 (-14.53%) 494555.00 ( 10.42%) 442751.00 ( -1.14%)
TPut 38 447907.00 ( 0.00%) 455542.00 ( 1.70%) 341693.00 (-23.71%) 482758.00 ( 7.78%) 444023.00 ( -0.87%)
TPut 39 428322.00 ( 0.00%) 367921.00 (-14.10%) 404210.00 ( -5.63%) 464550.00 ( 8.46%) 440482.00 ( 2.84%)
TPut 40 429157.00 ( 0.00%) 394277.00 ( -8.13%) 378554.00 (-11.79%) 467767.00 ( 9.00%) 411807.00 ( -4.04%)
TPut 41 424339.00 ( 0.00%) 415413.00 ( -2.10%) 399220.00 ( -5.92%) 457669.00 ( 7.85%) 428273.00 ( 0.93%)
TPut 42 397440.00 ( 0.00%) 421027.00 ( 5.93%) 372161.00 ( -6.36%) 458156.00 ( 15.28%) 422535.00 ( 6.31%)
TPut 43 405391.00 ( 0.00%) 433900.00 ( 7.03%) 383936.00 ( -5.29%) 438929.00 ( 8.27%) 410196.00 ( 1.19%)
TPut 44 400692.00 ( 0.00%) 427504.00 ( 6.69%) 374757.00 ( -6.47%) 423538.00 ( 5.70%) 399471.00 ( -0.30%)
TPut 45 399623.00 ( 0.00%) 372622.00 ( -6.76%) 379797.00 ( -4.96%) 407255.00 ( 1.91%) 374068.00 ( -6.39%)
TPut 46 391920.00 ( 0.00%) 351205.00 (-10.39%) 368042.00 ( -6.09%) 411353.00 ( 4.96%) 384363.00 ( -1.93%)
TPut 47 378199.00 ( 0.00%) 358150.00 ( -5.30%) 368744.00 ( -2.50%) 408739.00 ( 8.08%) 385670.00 ( 1.98%)
TPut 48 379346.00 ( 0.00%) 387287.00 ( 2.09%) 373581.00 ( -1.52%) 423791.00 ( 11.72%) 380665.00 ( 0.35%)
TPut 49 373614.00 ( 0.00%) 395793.00 ( 5.94%) 372621.00 ( -0.27%) 423024.00 ( 13.22%) 377985.00 ( 1.17%)
TPut 50 372494.00 ( 0.00%) 366488.00 ( -1.61%) 388778.00 ( 4.37%) 410647.00 ( 10.24%) 378831.00 ( 1.70%)
TPut 51 382195.00 ( 0.00%) 381771.00 ( -0.11%) 387687.00 ( 1.44%) 423249.00 ( 10.74%) 402233.00 ( 5.24%)
TPut 52 369118.00 ( 0.00%) 429441.00 ( 16.34%) 390226.00 ( 5.72%) 410023.00 ( 11.08%) 396558.00 ( 7.43%)
TPut 53 366453.00 ( 0.00%) 445744.00 ( 21.64%) 399257.00 ( 8.95%) 405937.00 ( 10.77%) 383916.00 ( 4.77%)
TPut 54 366571.00 ( 0.00%) 375762.00 ( 2.51%) 395098.00 ( 7.78%) 402220.00 ( 9.72%) 395417.00 ( 7.87%)
TPut 55 367580.00 ( 0.00%) 336113.00 ( -8.56%) 400550.00 ( 8.97%) 420978.00 ( 14.53%) 398098.00 ( 8.30%)
TPut 56 367056.00 ( 0.00%) 375635.00 ( 2.34%) 385743.00 ( 5.09%) 412685.00 ( 12.43%) 384029.00 ( 4.62%)
TPut 57 359163.00 ( 0.00%) 354001.00 ( -1.44%) 389827.00 ( 8.54%) 394688.00 ( 9.89%) 381032.00 ( 6.09%)
TPut 58 360552.00 ( 0.00%) 353312.00 ( -2.01%) 394099.00 ( 9.30%) 388655.00 ( 7.79%) 378132.00 ( 4.88%)
TPut 59 354967.00 ( 0.00%) 368534.00 ( 3.82%) 390746.00 ( 10.08%) 399086.00 ( 12.43%) 387101.00 ( 9.05%)
TPut 60 362976.00 ( 0.00%) 388472.00 ( 7.02%) 383073.00 ( 5.54%) 399713.00 ( 10.12%) 390635.00 ( 7.62%)
TPut 61 368072.00 ( 0.00%) 399476.00 ( 8.53%) 380807.00 ( 3.46%) 372060.00 ( 1.08%) 383187.00 ( 4.11%)
TPut 62 356938.00 ( 0.00%) 385648.00 ( 8.04%) 387736.00 ( 8.63%) 377183.00 ( 5.67%) 378484.00 ( 6.04%)
TPut 63 357491.00 ( 0.00%) 404325.00 ( 13.10%) 396672.00 ( 10.96%) 384221.00 ( 7.48%) 378907.00 ( 5.99%)
TPut 64 357322.00 ( 0.00%) 389552.00 ( 9.02%) 386826.00 ( 8.26%) 378601.00 ( 5.96%) 369852.00 ( 3.51%)
TPut 65 341262.00 ( 0.00%) 394964.00 ( 15.74%) 380271.00 ( 11.43%) 382896.00 ( 12.20%) 382897.00 ( 12.20%)
TPut 66 357807.00 ( 0.00%) 384846.00 ( 7.56%) 362723.00 ( 1.37%) 361530.00 ( 1.04%) 380023.00 ( 6.21%)
TPut 67 345092.00 ( 0.00%) 376842.00 ( 9.20%) 364193.00 ( 5.54%) 374449.00 ( 8.51%) 373877.00 ( 8.34%)
TPut 68 350334.00 ( 0.00%) 358330.00 ( 2.28%) 359368.00 ( 2.58%) 384920.00 ( 9.87%) 381888.00 ( 9.01%)
TPut 69 348372.00 ( 0.00%) 356188.00 ( 2.24%) 364449.00 ( 4.61%) 395611.00 ( 13.56%) 375892.00 ( 7.90%)
TPut 70 335077.00 ( 0.00%) 359313.00 ( 7.23%) 356418.00 ( 6.37%) 375448.00 ( 12.05%) 372358.00 ( 11.13%)
TPut 71 341197.00 ( 0.00%) 364168.00 ( 6.73%) 343847.00 ( 0.78%) 376113.00 ( 10.23%) 384292.00 ( 12.63%)
TPut 72 345032.00 ( 0.00%) 356934.00 ( 3.45%) 345007.00 ( -0.01%) 375313.00 ( 8.78%) 381504.00 ( 10.57%)
numacore v17 was doing reasonably well but we knew that already.
autonuma does not do great on this test.
balancenuma does all right. The scalability patches actually hurt in this case
but it's likely down to varability in the decisions made by the scheduler as much
as anything else.
SPECJBB PEAKS
3.7.0-rc7 3.7.0-rc6 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Expctd Warehouse 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%)
Expctd Peak Bops 379346.00 ( 0.00%) 387287.00 ( 2.09%) 373581.00 ( -1.52%) 423791.00 ( 11.72%) 380665.00 ( 0.35%)
Actual Warehouse 24.00 ( 0.00%) 25.00 ( 4.17%) 24.00 ( 0.00%) 24.00 ( 0.00%) 24.00 ( 0.00%)
Actual Peak Bops 495378.00 ( 0.00%) 520474.00 ( 5.07%) 514403.00 ( 3.84%) 545294.00 ( 10.08%) 493974.00 ( -0.28%)
SpecJBB Bops 183389.00 ( 0.00%) 193652.00 ( 5.60%) 193461.00 ( 5.49%) 201083.00 ( 9.65%) 195465.00 ( 6.58%)
SpecJBB Bops/JVM 183389.00 ( 0.00%) 193652.00 ( 5.60%) 193461.00 ( 5.49%) 201083.00 ( 9.65%) 195465.00 ( 6.58%)
Balancenuma does all right on its specjbb score but the peak score with
the migration scalability patches applied is hurt. At least it's still
comparable to mainline.
MMTests Statistics: duration
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User 316340.52 311420.23 31308.52 314589.64 316061.23 315584.37
System 102.08 3067.27 803.23 352.70 428.76 450.71
Elapsed 7433.22 7436.63 1398.05 7434.74 7432.60 7435.03
Usual comments about system CPU usage. You actually see latest numacore
figures here because they are based on what happened up until the crash.
MMTests Statistics: vmstat
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins 66212 36180 31560 36152 36188 63852
Page Outs 31248 35544 12016 28388 28024 42360
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0
Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0
Page writes file 0 0 0 0 0 0
Page writes anon 0 0 0 0 0 0
Page reclaim immediate 0 0 0 0 0 0
Page rescued immediate 0 0 0 0 0 0
Slabs scanned 0 0 0 0 0 0
Direct inode steals 0 0 0 0 0 0
Kswapd inode steals 0 0 0 0 0 0
Kswapd skipped wait 0 0 0 0 0 0
THP fault alloc 48874 45657 34986 48296 48697 47056
THP collapse alloc 51 2 9 157 53 69
THP splits 70 37 28 83 78 56
THP fault fallback 0 0 0 0 0 0
THP collapse fail 0 0 0 0 0 0
Compaction stalls 0 0 0 0 0 0
Compaction success 0 0 0 0 0 0
Compaction failures 0 0 0 0 0 0
Page migrate success 0 0 110442307 0 45908125 46995604
Page migrate failure 0 0 0 0 0 0
Compaction pages isolated 0 0 0 0 0 0
Compaction migrate scanned 0 0 0 0 0 0
Compaction free scanned 0 0 0 0 0 0
Compaction cost 0 0 114639 0 47652 48781
NUMA PTE updates 0 0 391813174 0 351907231 361308027
NUMA hint faults 0 0 796717 0 2010327 1867697
NUMA hint local faults 0 0 261885 0 677602 572742
NUMA pages migrated 0 0 110442307 0 45908125 46995604
AutoNUMA cost 0 0 8824 0 13387 12760
THP was certainly enabled.
numacores migration rate is extremely high until it crashed -- 308MB/sec
as opposed to balancenumas 24MB/sec on average.
SpecJBB, Single JVM, THP is disabled
====================================
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
TPut 1 19861.00 ( 0.00%) 18255.00 ( -8.09%) 20169.00 ( 1.55%) 19636.00 ( -1.13%) 19838.00 ( -0.12%) 20650.00 ( 3.97%)
TPut 2 47613.00 ( 0.00%) 37136.00 (-22.00%) 45050.00 ( -5.38%) 47153.00 ( -0.97%) 47481.00 ( -0.28%) 48199.00 ( 1.23%)
TPut 3 72438.00 ( 0.00%) 55692.00 (-23.12%) 64075.00 (-11.55%) 69394.00 ( -4.20%) 72029.00 ( -0.56%) 72932.00 ( 0.68%)
TPut 4 98455.00 ( 0.00%) 81301.00 (-17.42%) 93595.00 ( -4.94%) 98577.00 ( 0.12%) 98437.00 ( -0.02%) 99748.00 ( 1.31%)
TPut 5 120831.00 ( 0.00%) 89067.00 (-26.29%) 115796.00 ( -4.17%) 120805.00 ( -0.02%) 117218.00 ( -2.99%) 121254.00 ( 0.35%)
TPut 6 140013.00 ( 0.00%) 108349.00 (-22.62%) 116704.00 (-16.65%) 125079.00 (-10.67%) 139878.00 ( -0.10%) 145360.00 ( 3.82%)
TPut 7 163553.00 ( 0.00%) 116192.00 (-28.96%) 118711.00 (-27.42%) 164368.00 ( 0.50%) 167133.00 ( 2.19%) 169539.00 ( 3.66%)
TPut 8 190148.00 ( 0.00%) 125955.00 (-33.76%) 118079.00 (-37.90%) 188906.00 ( -0.65%) 183058.00 ( -3.73%) 188936.00 ( -0.64%)
TPut 9 211343.00 ( 0.00%) 144068.00 (-31.83%) 170067.00 (-19.53%) 206645.00 ( -2.22%) 205699.00 ( -2.67%) 217322.00 ( 2.83%)
TPut 10 233190.00 ( 0.00%) 148098.00 (-36.49%) 133365.00 (-42.81%) 234533.00 ( 0.58%) 233632.00 ( 0.19%) 227292.00 ( -2.53%)
TPut 11 253333.00 ( 0.00%) 146043.00 (-42.35%) 108866.00 (-57.03%) 254167.00 ( 0.33%) 251938.00 ( -0.55%) 259924.00 ( 2.60%)
TPut 12 270661.00 ( 0.00%) 131739.00 (-51.33%) 146170.00 (-46.00%) 271490.00 ( 0.31%) 271393.00 ( 0.27%) 272536.00 ( 0.69%)
TPut 13 299807.00 ( 0.00%) 169396.00 (-43.50%) 134946.00 (-54.99%) 299758.00 ( -0.02%) 270594.00 ( -9.74%) 299110.00 ( -0.23%)
TPut 14 319243.00 ( 0.00%) 150705.00 (-52.79%) 145135.00 (-54.54%) 318481.00 ( -0.24%) 318566.00 ( -0.21%) 325133.00 ( 1.84%)
TPut 15 339054.00 ( 0.00%) 116872.00 (-65.53%) 127277.00 (-62.46%) 331534.00 ( -2.22%) 344672.00 ( 1.66%) 318119.00 ( -6.17%)
TPut 16 354315.00 ( 0.00%) 124346.00 (-64.91%) 86657.00 (-75.54%) 352600.00 ( -0.48%) 316761.00 (-10.60%) 364648.00 ( 2.92%)
TPut 17 371306.00 ( 0.00%) 118493.00 (-68.09%) 93297.00 (-74.87%) 368260.00 ( -0.82%) 328888.00 (-11.42%) 371088.00 ( -0.06%)
TPut 18 386361.00 ( 0.00%) 138571.00 (-64.13%) 208447.00 (-46.05%) 374358.00 ( -3.11%) 356148.00 ( -7.82%) 399913.00 ( 3.51%)
TPut 19 401827.00 ( 0.00%) 118855.00 (-70.42%) 155803.00 (-61.23%) 399476.00 ( -0.59%) 393918.00 ( -1.97%) 405771.00 ( 0.98%)
TPut 20 411130.00 ( 0.00%) 144024.00 (-64.97%) 116524.00 (-71.66%) 407799.00 ( -0.81%) 377706.00 ( -8.13%) 406038.00 ( -1.24%)
TPut 21 425352.00 ( 0.00%) 154264.00 (-63.73%) 144766.00 (-65.97%) 429226.00 ( 0.91%) 431677.00 ( 1.49%) 431583.00 ( 1.46%)
TPut 22 438150.00 ( 0.00%) 153892.00 (-64.88%) 222211.00 (-49.28%) 385827.00 (-11.94%) 440379.00 ( 0.51%) 438861.00 ( 0.16%)
TPut 23 438425.00 ( 0.00%) 146506.00 (-66.58%) 213367.00 (-51.33%) 433963.00 ( -1.02%) 361427.00 (-17.56%) 445293.00 ( 1.57%)
TPut 24 461598.00 ( 0.00%) 138869.00 (-69.92%) 189745.00 (-58.89%) 439691.00 ( -4.75%) 471567.00 ( 2.16%) 488259.00 ( 5.78%)
TPut 25 459475.00 ( 0.00%) 141698.00 (-69.16%) 105196.00 (-77.11%) 431373.00 ( -6.12%) 487921.00 ( 6.19%) 447353.00 ( -2.64%)
TPut 26 452651.00 ( 0.00%) 142844.00 (-68.44%) 125573.00 (-72.26%) 447517.00 ( -1.13%) 425336.00 ( -6.03%) 469793.00 ( 3.79%)
TPut 27 450436.00 ( 0.00%) 140870.00 (-68.73%) 68802.00 (-84.73%) 430805.00 ( -4.36%) 456114.00 ( 1.26%) 461172.00 ( 2.38%)
TPut 28 459770.00 ( 0.00%) 143078.00 (-68.88%) 144373.00 (-68.60%) 432260.00 ( -5.98%) 478317.00 ( 4.03%) 452144.00 ( -1.66%)
TPut 29 450347.00 ( 0.00%) 142076.00 (-68.45%) 221760.00 (-50.76%) 440423.00 ( -2.20%) 388175.00 (-13.81%) 473273.00 ( 5.09%)
TPut 30 449252.00 ( 0.00%) 146900.00 (-67.30%) 139971.00 (-68.84%) 435082.00 ( -3.15%) 440795.00 ( -1.88%) 435189.00 ( -3.13%)
TPut 31 446802.00 ( 0.00%) 148008.00 (-66.87%) 195143.00 (-56.32%) 418684.00 ( -6.29%) 417343.00 ( -6.59%) 437562.00 ( -2.07%)
TPut 32 439701.00 ( 0.00%) 149591.00 (-65.98%) 159107.00 (-63.81%) 421866.00 ( -4.06%) 438719.00 ( -0.22%) 469763.00 ( 6.84%)
TPut 33 434477.00 ( 0.00%) 142801.00 (-67.13%) 110758.00 (-74.51%) 420631.00 ( -3.19%) 454673.00 ( 4.65%) 451224.00 ( 3.85%)
TPut 34 423014.00 ( 0.00%) 152308.00 (-63.99%) 111701.00 (-73.59%) 415202.00 ( -1.85%) 415194.00 ( -1.85%) 446735.00 ( 5.61%)
TPut 35 429012.00 ( 0.00%) 154116.00 (-64.08%) 118968.00 (-72.27%) 402395.00 ( -6.20%) 425151.00 ( -0.90%) 434230.00 ( 1.22%)
TPut 36 421097.00 ( 0.00%) 157571.00 (-62.58%) 174626.00 (-58.53%) 404770.00 ( -3.88%) 430480.00 ( 2.23%) 425324.00 ( 1.00%)
TPut 37 414815.00 ( 0.00%) 150771.00 (-63.65%) 238764.00 (-42.44%) 388842.00 ( -6.26%) 393351.00 ( -5.17%) 405824.00 ( -2.17%)
TPut 38 412361.00 ( 0.00%) 157070.00 (-61.91%) 173206.00 (-58.00%) 398947.00 ( -3.25%) 401555.00 ( -2.62%) 432074.00 ( 4.78%)
TPut 39 402234.00 ( 0.00%) 161487.00 (-59.85%) 119790.00 (-70.22%) 382645.00 ( -4.87%) 423106.00 ( 5.19%) 401091.00 ( -0.28%)
TPut 40 380278.00 ( 0.00%) 165947.00 (-56.36%) 309375.00 (-18.65%) 394039.00 ( 3.62%) 405371.00 ( 6.60%) 410739.00 ( 8.01%)
TPut 41 393204.00 ( 0.00%) 160540.00 (-59.17%) 146153.00 (-62.83%) 385605.00 ( -1.93%) 403383.00 ( 2.59%) 372466.00 ( -5.27%)
TPut 42 380622.00 ( 0.00%) 151946.00 (-60.08%) 269523.00 (-29.19%) 374843.00 ( -1.52%) 380797.00 ( 0.05%) 396227.00 ( 4.10%)
TPut 43 371566.00 ( 0.00%) 162369.00 (-56.30%) 344584.00 ( -7.26%) 347951.00 ( -6.36%) 386765.00 ( 4.09%) 345633.00 ( -6.98%)
TPut 44 365538.00 ( 0.00%) 161127.00 (-55.92%) 147195.00 (-59.73%) 355070.00 ( -2.86%) 344701.00 ( -5.70%) 391276.00 ( 7.04%)
TPut 45 359305.00 ( 0.00%) 159062.00 (-55.73%) 102716.00 (-71.41%) 350973.00 ( -2.32%) 370666.00 ( 3.16%) 331191.00 ( -7.82%)
TPut 46 343160.00 ( 0.00%) 163889.00 (-52.24%) 309203.00 ( -9.90%) 347960.00 ( 1.40%) 380147.00 ( 10.78%) 323176.00 ( -5.82%)
TPut 47 346983.00 ( 0.00%) 168666.00 (-51.39%) 330345.00 ( -4.80%) 313612.00 ( -9.62%) 362189.00 ( 4.38%) 343154.00 ( -1.10%)
TPut 48 338143.00 ( 0.00%) 153448.00 (-54.62%) 291944.00 (-13.66%) 341809.00 ( 1.08%) 365342.00 ( 8.04%) 354348.00 ( 4.79%)
TPut 49 333941.00 ( 0.00%) 142784.00 (-57.24%) 252850.00 (-24.28%) 336174.00 ( 0.67%) 371700.00 ( 11.31%) 353148.00 ( 5.75%)
TPut 50 334001.00 ( 0.00%) 135713.00 (-59.37%) 252350.00 (-24.45%) 322489.00 ( -3.45%) 367963.00 ( 10.17%) 355823.00 ( 6.53%)
TPut 51 338310.00 ( 0.00%) 133402.00 (-60.57%) 232361.00 (-31.32%) 354805.00 ( 4.88%) 372592.00 ( 10.13%) 351194.00 ( 3.81%)
TPut 52 322897.00 ( 0.00%) 150293.00 (-53.45%) 193895.00 (-39.95%) 353169.00 ( 9.38%) 363024.00 ( 12.43%) 344846.00 ( 6.80%)
TPut 53 329801.00 ( 0.00%) 160792.00 (-51.25%) 180672.00 (-45.22%) 353588.00 ( 7.21%) 365359.00 ( 10.78%) 355499.00 ( 7.79%)
TPut 54 336610.00 ( 0.00%) 164696.00 (-51.07%) 248332.00 (-26.23%) 361189.00 ( 7.30%) 377851.00 ( 12.25%) 363987.00 ( 8.13%)
TPut 55 325920.00 ( 0.00%) 172380.00 (-47.11%) 271331.00 (-16.75%) 365678.00 ( 12.20%) 375735.00 ( 15.28%) 363697.00 ( 11.59%)
TPut 56 318997.00 ( 0.00%) 176071.00 (-44.80%) 155354.00 (-51.30%) 367048.00 ( 15.06%) 380588.00 ( 19.31%) 362614.00 ( 13.67%)
TPut 57 321776.00 ( 0.00%) 174531.00 (-45.76%) 279294.00 (-13.20%) 341874.00 ( 6.25%) 378996.00 ( 17.78%) 360366.00 ( 11.99%)
TPut 58 308532.00 ( 0.00%) 174202.00 (-43.54%) 170351.00 (-44.79%) 348156.00 ( 12.84%) 361623.00 ( 17.21%) 369693.00 ( 19.82%)
TPut 59 318974.00 ( 0.00%) 175343.00 (-45.03%) 243463.00 (-23.67%) 358252.00 ( 12.31%) 360457.00 ( 13.01%) 364556.00 ( 14.29%)
TPut 60 325465.00 ( 0.00%) 173694.00 (-46.63%) 222867.00 (-31.52%) 360808.00 ( 10.86%) 362745.00 ( 11.45%) 354232.00 ( 8.84%)
TPut 61 319151.00 ( 0.00%) 172320.00 (-46.01%) 218542.00 (-31.52%) 350597.00 ( 9.85%) 371277.00 ( 16.33%) 352478.00 ( 10.44%)
TPut 62 320837.00 ( 0.00%) 172312.00 (-46.29%) 251630.00 (-21.57%) 359062.00 ( 11.91%) 361009.00 ( 12.52%) 352930.00 ( 10.00%)
TPut 63 318198.00 ( 0.00%) 172297.00 (-45.85%) 172040.00 (-45.93%) 356137.00 ( 11.92%) 347637.00 ( 9.25%) 335322.00 ( 5.38%)
TPut 64 321438.00 ( 0.00%) 171894.00 (-46.52%) 151337.00 (-52.92%) 347376.00 ( 8.07%) 346756.00 ( 7.88%) 351410.00 ( 9.32%)
TPut 65 314482.00 ( 0.00%) 169147.00 (-46.21%) 143487.00 (-54.37%) 351726.00 ( 11.84%) 357429.00 ( 13.66%) 351236.00 ( 11.69%)
TPut 66 316802.00 ( 0.00%) 170234.00 (-46.26%) 230207.00 (-27.33%) 344548.00 ( 8.76%) 362143.00 ( 14.31%) 347058.00 ( 9.55%)
TPut 67 312139.00 ( 0.00%) 168180.00 (-46.12%) 148468.00 (-52.44%) 329030.00 ( 5.41%) 353305.00 ( 13.19%) 345903.00 ( 10.82%)
TPut 68 323918.00 ( 0.00%) 168392.00 (-48.01%) 184696.00 (-42.98%) 319985.00 ( -1.21%) 344250.00 ( 6.28%) 345703.00 ( 6.73%)
TPut 69 307506.00 ( 0.00%) 167082.00 (-45.67%) 221855.00 (-27.85%) 340673.00 ( 10.79%) 339346.00 ( 10.35%) 336071.00 ( 9.29%)
TPut 70 306799.00 ( 0.00%) 165764.00 (-45.97%) 246518.00 (-19.65%) 331678.00 ( 8.11%) 349583.00 ( 13.95%) 341944.00 ( 11.46%)
TPut 71 304232.00 ( 0.00%) 165289.00 (-45.67%) 225582.00 (-25.85%) 319824.00 ( 5.13%) 335238.00 ( 10.19%) 343396.00 ( 12.87%)
TPut 72 301619.00 ( 0.00%) 163909.00 (-45.66%) 154552.00 (-48.76%) 326875.00 ( 8.37%) 345999.00 ( 14.71%) 343949.00 ( 14.03%)
Latest numacore is regressing really badly here.
autonuma is all right.
balancenuma is all right. Migration scalability patches actually seem to
hurt a little.
SPECJBB PEAKS
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Expctd Warehouse 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%) 48.00 ( 0.00%)
Expctd Peak Bops 338143.00 ( 0.00%) 153448.00 (-54.62%) 291944.00 (-13.66%) 341809.00 ( 1.08%) 365342.00 ( 8.04%) 354348.00 ( 4.79%)
Actual Warehouse 24.00 ( 0.00%) 56.00 (133.33%) 43.00 ( 79.17%) 26.00 ( 8.33%) 25.00 ( 4.17%) 24.00 ( 0.00%)
Actual Peak Bops 461598.00 ( 0.00%) 176071.00 (-61.86%) 344584.00 (-25.35%) 447517.00 ( -3.05%) 487921.00 ( 5.70%) 488259.00 ( 5.78%)
SpecJBB Bops 163683.00 ( 0.00%) 83963.00 (-48.70%) 109061.00 (-33.37%) 176379.00 ( 7.76%) 184040.00 ( 12.44%) 179621.00 ( 9.74%)
SpecJBB Bops/JVM 163683.00 ( 0.00%) 83963.00 (-48.70%) 109061.00 (-33.37%) 176379.00 ( 7.76%) 184040.00 ( 12.44%) 179621.00 ( 9.74%)
numacore regresses 25.35% at the peak and 33.37% on its specjbb score.
balancenuma does all right -- 5.78% gain at the peak, 9.74% on its overall
specjbb score.
MMTests Statistics: duration
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
User 316751.91 167098.56 227496.59 307598.67 309109.47 313644.48
System 60.28 122511.08 72477.33 4411.81 1820.70 2654.77
Elapsed 7434.08 7451.36 7476.09 7437.52 7438.28 7438.19
numacores system CPu usage has improved but it's still insane -- 27 times
higher than balancenumas which itself is high. Put another way, numacore
is using over 1000 times more system CPU than the mainline kernel is.
MMTests Statistics: vmstat
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins 37112 36416 34572 37436 35400 34708
Page Outs 29252 35664 29788 28120 28504 28292
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0
Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0
Page writes file 0 0 0 0 0 0
Page writes anon 0 0 0 0 0 0
Page reclaim immediate 0 0 0 0 0 0
Page rescued immediate 0 0 0 0 0 0
Slabs scanned 0 0 0 0 0 0
Direct inode steals 0 0 0 0 0 0
Kswapd inode steals 0 0 0 0 0 0
Kswapd skipped wait 0 0 0 0 0 0
THP fault alloc 3 2 3 2 2 2
THP collapse alloc 0 0 0 4 0 0
THP splits 0 0 0 1 0 0
THP fault fallback 0 0 0 0 0 0
THP collapse fail 0 0 0 0 0 0
Compaction stalls 0 0 0 0 0 0
Compaction success 0 0 0 0 0 0
Compaction failures 0 0 0 0 0 0
Page migrate success 0 0 472734998 0 24675369 36216149
Page migrate failure 0 0 0 0 0 0
Compaction pages isolated 0 0 0 0 0 0
Compaction migrate scanned 0 0 0 0 0 0
Compaction free scanned 0 0 0 0 0 0
Compaction cost 0 0 490698 0 25613 37592
NUMA PTE updates 0 0 2978374076 0 200854895 256255594
NUMA hint faults 0 0 0 0 195451244 250219588
NUMA hint local faults 0 0 0 0 50377035 63739483
NUMA pages migrated 0 0 472734998 0 24675369 36216149
AutoNUMA cost 0 0 29830 0 979131 1253579
numacore is migrating on average 247MB/sec. balancenuma is migrating
19MB/sec on average.
I ran the other normal benchmarks too. kernbench and aim9 are more or less ok. The impact is on hackbench
HACKBENCH PIPES
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Procs 1 0.0250 ( 0.00%) 0.0260 ( -4.00%) 0.0246 ( 1.48%) 0.0261 ( -4.27%) 0.0325 (-30.07%) 0.0353 (-41.14%)
Procs 4 0.0696 ( 0.00%) 0.0702 ( -0.84%) 0.0602 ( 13.57%) 0.0707 ( -1.65%) 0.0760 ( -9.20%) 0.0738 ( -5.98%)
Procs 8 0.0836 ( 0.00%) 0.0973 (-16.43%) 0.0949 (-13.53%) 0.1030 (-23.21%) 0.0887 ( -6.15%) 0.1031 (-23.36%)
Procs 12 0.0971 ( 0.00%) 0.0969 ( 0.21%) 0.1447 (-49.00%) 0.1235 (-27.19%) 0.0953 ( 1.88%) 0.1394 (-43.56%)
Procs 16 0.1218 ( 0.00%) 0.1286 ( -5.52%) 0.2214 (-81.70%) 0.1775 (-45.69%) 0.1105 ( 9.33%) 0.2188 (-79.57%)
Procs 20 0.1472 ( 0.00%) 0.1508 ( -2.48%) 0.2744 (-86.43%) 0.1584 ( -7.64%) 0.1378 ( 6.38%) 0.2567 (-74.37%)
Procs 24 0.1684 ( 0.00%) 0.1823 ( -8.20%) 0.3602 (-113.82%) 0.4648 (-175.96%) 0.1623 ( 3.68%) 0.3118 (-85.12%)
Procs 28 0.1919 ( 0.00%) 0.1969 ( -2.61%) 0.4632 (-141.39%) 0.5287 (-175.57%) 0.1900 ( 0.96%) 0.4326 (-125.48%)
Procs 32 0.2256 ( 0.00%) 0.2163 ( 4.12%) 0.5040 (-123.40%) 0.4607 (-104.23%) 0.2163 ( 4.13%) 0.4583 (-103.16%)
Procs 36 0.2228 ( 0.00%) 0.2658 (-19.29%) 0.5481 (-145.98%) 0.6190 (-177.83%) 0.2570 (-15.33%) 0.5267 (-136.38%)
Procs 40 0.2811 ( 0.00%) 0.2906 ( -3.37%) 0.6223 (-121.36%) 0.2595 ( 7.69%) 0.2638 ( 6.15%) 0.5941 (-111.35%)
HACKBENCH SOCKETS
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6 numacore-20121130 numafix-20121209 autonuma-v28fastr4 balancenuma-v9r2 balancenuma-v10r3
Procs 1 0.0220 ( 0.00%) 0.0220 ( 0.00%) 0.0229 ( -4.20%) 0.0283 (-28.66%) 0.0216 ( 1.89%) 0.0256 (-16.36%)
Procs 4 0.0456 ( 0.00%) 0.0513 (-12.51%) 0.0559 (-22.50%) 0.0820 (-79.73%) 0.0407 ( 10.76%) 0.0627 (-37.46%)
Procs 8 0.0679 ( 0.00%) 0.0714 ( -5.20%) 0.1472 (-116.82%) 0.2772 (-308.32%) 0.0697 ( -2.60%) 0.1715 (-152.63%)
Procs 12 0.0940 ( 0.00%) 0.0973 ( -3.56%) 0.2259 (-140.32%) 0.1155 (-22.87%) 0.0973 ( -3.55%) 0.2459 (-161.55%)
Procs 16 0.1181 ( 0.00%) 0.1263 ( -6.96%) 0.3248 (-174.92%) 0.4467 (-278.19%) 0.1234 ( -4.46%) 0.3231 (-173.55%)
Procs 20 0.1504 ( 0.00%) 0.1531 ( -1.83%) 0.4039 (-168.54%) 0.4917 (-226.94%) 0.1534 ( -1.97%) 0.4172 (-177.36%)
Procs 24 0.1757 ( 0.00%) 0.1826 ( -3.92%) 0.3965 (-125.60%) 0.5142 (-192.57%) 0.1826 ( -3.89%) 0.4759 (-170.78%)
Procs 28 0.2044 ( 0.00%) 0.2166 ( -5.93%) 0.5438 (-165.99%) 0.6600 (-222.85%) 0.2164 ( -5.88%) 0.5455 (-166.83%)
Procs 32 0.2456 ( 0.00%) 0.2501 ( -1.86%) 0.6261 (-154.93%) 0.6391 (-160.22%) 0.2449 ( 0.27%) 0.6093 (-148.11%)
Procs 36 0.2649 ( 0.00%) 0.2747 ( -3.70%) 0.7066 (-166.71%) 0.5775 (-117.97%) 0.2815 ( -6.27%) 0.6840 (-158.19%)
Procs 40 0.3067 ( 0.00%) 0.3114 ( -1.56%) 0.7588 (-147.42%) 0.7517 (-145.12%) 0.3081 ( -0.48%) 0.8871 (-189.27%)
Latest numacore, autonuma and balancenuma are all butchering hackbench
performance. Considering that balancenuma started hurting performance with
the migration scalability patches leads me to conclude that they might be
directly or indirectly responsible.
MMTests Statistics: vmstat
3.7.0-rc7 3.7.0-rc6 3.7.0-rc8 3.7.0-rc7 3.7.0-rc7 3.7.0-rc7
stats-v8r6numacore-20121130numafix-20121209autonuma-v28fastr4balancenuma-v9r2balancenuma-v10r3
Page Ins 4 4 4 4 4 4
Page Outs 1540 1636 2568 2264 1548 2484
Swap Ins 0 0 0 0 0 0
Swap Outs 0 0 0 0 0 0
Direct pages scanned 0 0 0 0 0 0
Kswapd pages scanned 0 0 0 0 0 0
Kswapd pages reclaimed 0 0 0 0 0 0
Direct pages reclaimed 0 0 0 0 0 0
Kswapd efficiency 100% 100% 100% 100% 100% 100%
Kswapd velocity 0.000 0.000 0.000 0.000 0.000 0.000
Direct efficiency 100% 100% 100% 100% 100% 100%
Direct velocity 0.000 0.000 0.000 0.000 0.000 0.000
Percentage direct scans 0% 0% 0% 0% 0% 0%
Page writes by reclaim 0 0 0 0 0 0
Page writes file 0 0 0 0 0 0
Page writes anon 0 0 0 0 0 0
Page reclaim immediate 0 0 0 0 0 0
Page rescued immediate 0 0 0 0 0 0
Slabs scanned 0 0 0 0 0 0
Direct inode steals 0 0 0 0 0 0
Kswapd inode steals 0 0 0 0 0 0
Kswapd skipped wait 0 0 0 0 0 0
THP fault alloc 5 0 0 0 6 5
THP collapse alloc 0 0 0 0 0 0
THP splits 0 0 0 0 0 0
THP fault fallback 0 0 0 0 0 0
THP collapse fail 0 0 0 0 0 0
Compaction stalls 0 0 0 0 0 0
Compaction success 0 0 0 0 0 0
Compaction failures 0 0 0 0 0 0
Page migrate success 0 0 0 0 1649 49
Page migrate failure 0 0 0 0 0 0
Compaction pages isolated 0 0 0 0 0 0
Compaction migrate scanned 0 0 0 0 0 0
Compaction free scanned 0 0 0 0 0 0
Compaction cost 0 0 0 0 1 0
NUMA PTE updates 0 0 0 0 21646 22884
NUMA hint faults 0 0 0 0 1045 2131
NUMA hint local faults 0 0 0 0 40 1218
NUMA pages migrated 0 0 0 0 1649 49
AutoNUMA cost 0 0 0 0 5 10
Based on this, I believe the migration patches are only indirectly
responsible. No way should hackbench be migrating or receiving a PTE update
at all. Rather than withdrawing the scalability patches it might make more
sense to either increase the length of time before a PTE takes place or
to delay NUMA PTE updates until the RSS reaches a particular size instead
of just relying on where the task gets scheduled.
So overall, I still believe that balancenuma should be merged at this point
based on these results. Nothing stops you doing a rebase of numacore on
top afterwards and introduce it in parts validating at each point it's
actually improving performance and not just assuming it does.
--
Mel Gorman
SUSE Labs
next prev parent reply other threads:[~2012-12-09 20:36 UTC|newest]
Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-12-07 10:23 [PATCH 00/49] Automatic NUMA Balancing v10 Mel Gorman
2012-12-07 10:23 ` [PATCH 01/49] x86: mm: only do a local tlb flush in ptep_set_access_flags() Mel Gorman
2012-12-07 10:23 ` [PATCH 02/49] x86: mm: drop TLB flush from ptep_set_access_flags Mel Gorman
2012-12-07 10:23 ` [PATCH 03/49] mm,generic: only flush the local TLB in ptep_set_access_flags Mel Gorman
2012-12-07 10:23 ` [PATCH 04/49] x86/mm: Introduce pte_accessible() Mel Gorman
2012-12-07 10:23 ` [PATCH 05/49] mm: Only flush the TLB when clearing an accessible pte Mel Gorman
2012-12-07 10:23 ` [PATCH 06/49] mm: Count the number of pages affected in change_protection() Mel Gorman
2012-12-07 10:23 ` [PATCH 07/49] mm: Optimize the TLB flush of sys_mprotect() and change_protection() users Mel Gorman
2012-12-07 10:23 ` [PATCH 08/49] mm: compaction: Move migration fail/success stats to migrate.c Mel Gorman
2012-12-07 10:23 ` [PATCH 09/49] mm: migrate: Add a tracepoint for migrate_pages Mel Gorman
2012-12-07 10:23 ` [PATCH 10/49] mm: compaction: Add scanned and isolated counters for compaction Mel Gorman
2012-12-07 10:23 ` [PATCH 11/49] mm: numa: define _PAGE_NUMA Mel Gorman
2012-12-07 10:23 ` [PATCH 12/49] mm: numa: pte_numa() and pmd_numa() Mel Gorman
2012-12-07 10:23 ` [PATCH 13/49] mm: numa: Support NUMA hinting page faults from gup/gup_fast Mel Gorman
2012-12-07 10:23 ` [PATCH 14/49] mm: numa: split_huge_page: transfer the NUMA type from the pmd to the pte Mel Gorman
2012-12-07 10:23 ` [PATCH 15/49] mm: numa: Create basic numa page hinting infrastructure Mel Gorman
2012-12-07 10:23 ` [PATCH 16/49] mm: mempolicy: Make MPOL_LOCAL a real policy Mel Gorman
2012-12-07 10:23 ` [PATCH 17/49] mm: mempolicy: Add MPOL_NOOP Mel Gorman
2012-12-07 10:23 ` [PATCH 18/49] mm: mempolicy: Check for misplaced page Mel Gorman
2012-12-07 10:23 ` [PATCH 19/49] mm: migrate: Introduce migrate_misplaced_page() Mel Gorman
2012-12-07 10:23 ` [PATCH 20/49] mm: migrate: Drop the misplaced pages reference count if the target node is full Mel Gorman
2012-12-07 10:23 ` [PATCH 21/49] mm: mempolicy: Use _PAGE_NUMA to migrate pages Mel Gorman
2012-12-07 10:23 ` [PATCH 22/49] mm: mempolicy: Add MPOL_MF_LAZY Mel Gorman
2013-01-05 5:18 ` Simon Jeons
2013-01-07 15:14 ` Mel Gorman
2012-12-07 10:23 ` [PATCH 23/49] mm: mempolicy: Implement change_prot_numa() in terms of change_protection() Mel Gorman
2012-12-07 10:23 ` [PATCH 24/49] mm: mempolicy: Hide MPOL_NOOP and MPOL_MF_LAZY from userspace for now Mel Gorman
2012-12-07 10:23 ` [PATCH 25/49] mm: numa: Add fault driven placement and migration Mel Gorman
2013-01-04 11:56 ` Simon Jeons
2012-12-07 10:23 ` [PATCH 26/49] mm: sched: numa: Implement constant, per task Working Set Sampling (WSS) rate Mel Gorman
2012-12-07 10:23 ` [PATCH 27/49] sched, numa, mm: Count WS scanning against present PTEs, not virtual memory ranges Mel Gorman
2012-12-07 10:23 ` [PATCH 28/49] mm: sched: numa: Implement slow start for working set sampling Mel Gorman
2012-12-07 10:23 ` [PATCH 29/49] mm: numa: Add pte updates, hinting and migration stats Mel Gorman
2013-01-04 11:42 ` Simon Jeons
2013-01-07 15:29 ` Mel Gorman
2012-12-07 10:23 ` [PATCH 30/49] mm: numa: Migrate on reference policy Mel Gorman
2012-12-07 10:23 ` [PATCH 31/49] mm: numa: Migrate pages handled during a pmd_numa hinting fault Mel Gorman
2012-12-07 10:23 ` [PATCH 32/49] mm: numa: Structures for Migrate On Fault per NUMA migration rate limiting Mel Gorman
2012-12-07 10:23 ` [PATCH 33/49] mm: numa: Rate limit the amount of memory that is migrated between nodes Mel Gorman
2012-12-07 10:23 ` [PATCH 34/49] mm: numa: Rate limit setting of pte_numa if node is saturated Mel Gorman
2012-12-07 10:23 ` [PATCH 35/49] sched: numa: Slowly increase the scanning period as NUMA faults are handled Mel Gorman
2012-12-07 10:23 ` [PATCH 36/49] mm: numa: Introduce last_nid to the page frame Mel Gorman
2012-12-07 10:23 ` [PATCH 37/49] mm: numa: split_huge_page: Transfer last_nid on tail page Mel Gorman
2012-12-07 10:23 ` [PATCH 38/49] mm: numa: migrate: Set last_nid on newly allocated page Mel Gorman
2012-12-07 10:23 ` [PATCH 39/49] mm: numa: Use a two-stage filter to restrict pages being migrated for unlikely task<->node relationships Mel Gorman
2012-12-07 10:23 ` [PATCH 40/49] mm: sched: Adapt the scanning rate if a NUMA hinting fault does not migrate Mel Gorman
2012-12-07 10:23 ` [PATCH 41/49] mm: sched: numa: Control enabling and disabling of NUMA balancing Mel Gorman
2012-12-07 10:23 ` [PATCH 42/49] mm: sched: numa: Control enabling and disabling of NUMA balancing if !SCHED_DEBUG Mel Gorman
2012-12-07 10:23 ` [PATCH 43/49] mm: sched: numa: Delay PTE scanning until a task is scheduled on a new node Mel Gorman
2012-12-07 10:23 ` [PATCH 44/49] mm: numa: Add THP migration for the NUMA working set scanning fault case Mel Gorman
[not found] ` <20130105084229.GA3208@hacker.(null)>
2013-01-07 15:37 ` Mel Gorman
2012-12-07 10:23 ` [PATCH 45/49] mm: numa: Add THP migration for the NUMA working set scanning fault case build fix Mel Gorman
2012-12-07 10:23 ` [PATCH 46/49] mm: numa: Account for failed allocations and isolations as migration failures Mel Gorman
2012-12-07 10:23 ` [PATCH 47/49] mm: migrate: Account a transhuge page properly when rate limiting Mel Gorman
2012-12-07 10:23 ` [PATCH 48/49] mm/rmap: Convert the struct anon_vma::mutex to an rwsem Mel Gorman
2012-12-07 10:23 ` [PATCH 49/49] mm/rmap, migration: Make rmap_walk_anon() and try_to_unmap_anon() more scalable Mel Gorman
2012-12-07 11:01 ` [PATCH 00/49] Automatic NUMA Balancing v10 Ingo Molnar
2012-12-09 20:36 ` Mel Gorman [this message]
2012-12-09 21:17 ` Kirill A. Shutemov
2012-12-10 8:44 ` Mel Gorman
2012-12-10 5:07 ` Srikar Dronamraju
2012-12-10 6:28 ` Srikar Dronamraju
2012-12-10 12:44 ` [PATCH] sched: Fix task_numa_fault() + KSM crash Ingo Molnar
2012-12-13 13:57 ` Srikar Dronamraju
2012-12-10 8:46 ` [PATCH 00/49] Automatic NUMA Balancing v10 Mel Gorman
2012-12-10 12:35 ` Ingo Molnar
2012-12-10 11:39 ` Ingo Molnar
2012-12-10 11:53 ` Ingo Molnar
2012-12-10 15:24 ` Mel Gorman
2012-12-11 1:02 ` Mel Gorman
2012-12-11 8:52 ` Ingo Molnar
2012-12-11 9:18 ` Ingo Molnar
2012-12-11 15:22 ` Mel Gorman
2012-12-11 16:30 ` Mel Gorman
2012-12-17 10:33 ` Ingo Molnar
2012-12-10 16:42 ` Srikar Dronamraju
2012-12-10 19:23 ` Ingo Molnar
2012-12-10 23:35 ` Srikar Dronamraju
2012-12-10 23:40 ` Srikar Dronamraju
2012-12-13 13:21 ` Srikar Dronamraju
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121209203630.GC1009@suse.de \
--to=mgorman@suse.de \
--cc=Lee.Schermerhorn@hp.com \
--cc=a.p.zijlstra@chello.nl \
--cc=aarcange@redhat.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.vnet.ibm.com \
--cc=dhillf@gmail.com \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=lkml.alex@gmail.com \
--cc=mingo@kernel.org \
--cc=pjt@google.com \
--cc=riel@redhat.com \
--cc=rientjes@google.com \
--cc=srikar@linux.vnet.ibm.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).