From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758702AbcEFSym (ORCPT ); Fri, 6 May 2016 14:54:42 -0400 Received: from mx2.suse.de ([195.135.220.15]:59059 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758166AbcEFSyl (ORCPT ); Fri, 6 May 2016 14:54:41 -0400 Message-ID: <1462560878.5119.9.camel@suse.de> Subject: Re: sched: tweak select_idle_sibling to look for idle threads From: Mike Galbraith To: Matt Fleming , Peter Zijlstra Cc: Chris Mason , Ingo Molnar , linux-kernel@vger.kernel.org Date: Fri, 06 May 2016 20:54:38 +0200 In-Reply-To: <20160505220306.GO2839@codeblueprint.co.uk> References: <20160409190554.honue3gtian2p6vr@floor.thefacebook.com> <20160430124731.GE2975@worktop.cust.blueprintrf.com> <1462086753.9717.29.camel@suse.de> <20160502084615.GB3430@twins.programming.kicks-ass.net> <1462200604.3736.42.camel@suse.de> <20160502145817.GW3408@twins.programming.kicks-ass.net> <20160502154725.ckiewczbdubudyc7@floor.masoncoding.com> <20160503143225.GG3448@twins.programming.kicks-ass.net> <20160503151153.wp6jcnjadmw2ypmx@floor.masoncoding.com> <20160504103701.GO3430@twins.programming.kicks-ass.net> <20160505220306.GO2839@codeblueprint.co.uk> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.16.5 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 2016-05-05 at 23:03 +0100, Matt Fleming wrote: > One thing I haven't yet done is twiddled the bits individually to see > what the best combination is. Have you settled on the right settings > yet? Lighter configs, revert sched/fair: Fix fairness issue on migration, twiddle knobs. Added an IDLE_SIBLING knob to ~virgin master.. only sorta virgin because I always throttle nohz. 1 x i4790 master for i in 1 2 4 8; do tbench.sh $i 30 2>&1|grep Throughput; done Throughput 871.785 MB/sec 1 clients 1 procs max_latency=0.324 ms Throughput 1514.5 MB/sec 2 clients 2 procs max_latency=0.411 ms Throughput 2722.43 MB/sec 4 clients 4 procs max_latency=2.400 ms Throughput 4334.46 MB/sec 8 clients 8 procs max_latency=3.561 ms echo NO_IDLE_SIBLING > /sys/kernel/debug/sched_features Throughput 1078.69 MB/sec 1 clients 1 procs max_latency=2.274 ms Throughput 2130.33 MB/sec 2 clients 2 procs max_latency=1.451 ms Throughput 3484.18 MB/sec 4 clients 4 procs max_latency=3.430 ms Throughput 4423.69 MB/sec 8 clients 8 procs max_latency=5.363 ms masterx for i in 1 2 4 8; do tbench.sh $i 30 2>&1|grep Throughput; done Throughput 707.673 MB/sec 1 clients 1 procs max_latency=2.279 ms Throughput 1503.55 MB/sec 2 clients 2 procs max_latency=0.695 ms Throughput 2527.73 MB/sec 4 clients 4 procs max_latency=2.321 ms Throughput 4291.26 MB/sec 8 clients 8 procs max_latency=3.815 ms echo NO_IDLE_CPU > /sys/kernel/debug/sched_features homer:~ # for i in 1 2 4 8; do tbench.sh $i 30 2>&1|grep Throughput; done Throughput 865.936 MB/sec 1 clients 1 procs max_latency=0.411 ms Throughput 1586.41 MB/sec 2 clients 2 procs max_latency=2.293 ms Throughput 2638.39 MB/sec 4 clients 4 procs max_latency=2.037 ms Throughput 4405.43 MB/sec 8 clients 8 procs max_latency=3.581 ms + echo NO_AVG_CPU > /sys/kernel/debug/sched_features + echo IDLE_SMT > /sys/kernel/debug/sched_features Throughput 697.126 MB/sec 1 clients 1 procs max_latency=2.220 ms Throughput 1562.82 MB/sec 2 clients 2 procs max_latency=0.526 ms Throughput 2620.62 MB/sec 4 clients 4 procs max_latency=6.460 ms Throughput 4345.13 MB/sec 8 clients 8 procs max_latency=27.921 ms 4 x E7-8890 master for i in 1 2 4 8 16 32 64 128 256; do tbench.sh $i 30 2>&1| grep Throughput; done Throughput 615.663 MB/sec 1 clients 1 procs max_latency=0.087 ms Throughput 1171.53 MB/sec 2 clients 2 procs max_latency=0.087 ms Throughput 2251.22 MB/sec 4 clients 4 procs max_latency=0.078 ms Throughput 4090.76 MB/sec 8 clients 8 procs max_latency=0.801 ms Throughput 7695.92 MB/sec 16 clients 16 procs max_latency=0.235 ms Throughput 15152 MB/sec 32 clients 32 procs max_latency=0.693 ms Throughput 21628.2 MB/sec 64 clients 64 procs max_latency=4.666 ms Throughput 43185.7 MB/sec 128 clients 128 procs max_latency=7.280 ms Throughput 72144.5 MB/sec 256 clients 256 procs max_latency=8.194 ms echo NO_IDLE_SIBLING > /sys/kernel/debug/sched_features Throughput 954.593 MB/sec 1 clients 1 procs max_latency=0.185 ms Throughput 1882.65 MB/sec 2 clients 2 procs max_latency=0.278 ms Throughput 3457.03 MB/sec 4 clients 4 procs max_latency=0.431 ms Throughput 6279.38 MB/sec 8 clients 8 procs max_latency=0.730 ms Throughput 11170.4 MB/sec 16 clients 16 procs max_latency=0.500 ms Throughput 21940.9 MB/sec 32 clients 32 procs max_latency=0.475 ms Throughput 41738.8 MB/sec 64 clients 64 procs max_latency=3.669 ms Throughput 67634.6 MB/sec 128 clients 128 procs max_latency=6.676 ms Throughput 76299.7 MB/sec 256 clients 256 procs max_latency=7.878 ms masterx for i in 1 2 4 8 16 32 64 128 256; do tbench.sh $i 30 2>&1| grep Throughput; done Throughput 587.956 MB/sec 1 clients 1 procs max_latency=0.124 ms Throughput 1140.16 MB/sec 2 clients 2 procs max_latency=0.476 ms Throughput 2296.03 MB/sec 4 clients 4 procs max_latency=0.142 ms Throughput 4116.65 MB/sec 8 clients 8 procs max_latency=0.464 ms Throughput 7820.27 MB/sec 16 clients 16 procs max_latency=0.238 ms Throughput 14899.2 MB/sec 32 clients 32 procs max_latency=0.321 ms Throughput 21909.8 MB/sec 64 clients 64 procs max_latency=0.905 ms Throughput 35495.2 MB/sec 128 clients 128 procs max_latency=6.158 ms Throughput 75863.2 MB/sec 256 clients 256 procs max_latency=7.650 ms echo NO_IDLE_CPU > /sys/kernel/debug/sched_features Throughput 555.15 MB/sec 1 clients 1 procs max_latency=0.096 ms Throughput 1195.12 MB/sec 2 clients 2 procs max_latency=0.131 ms Throughput 2276.97 MB/sec 4 clients 4 procs max_latency=0.105 ms Throughput 4248.14 MB/sec 8 clients 8 procs max_latency=0.131 ms Throughput 7860.86 MB/sec 16 clients 16 procs max_latency=0.210 ms Throughput 15178.6 MB/sec 32 clients 32 procs max_latency=0.229 ms Throughput 21523.9 MB/sec 64 clients 64 procs max_latency=0.842 ms Throughput 31082.1 MB/sec 128 clients 128 procs max_latency=7.311 ms Throughput 75887.9 MB/sec 256 clients 256 procs max_latency=7.764 ms + echo NO_AVG_CPU > /sys/kernel/debug/sched_features Throughput 598.063 MB/sec 1 clients 1 procs max_latency=0.131 ms Throughput 1140.2 MB/sec 2 clients 2 procs max_latency=0.092 ms Throughput 2268.68 MB/sec 4 clients 4 procs max_latency=0.170 ms Throughput 4259.7 MB/sec 8 clients 8 procs max_latency=0.212 ms Throughput 7904.15 MB/sec 16 clients 16 procs max_latency=0.191 ms Throughput 14840 MB/sec 32 clients 32 procs max_latency=0.279 ms Throughput 21701.5 MB/sec 64 clients 64 procs max_latency=0.856 ms Throughput 38945 MB/sec 128 clients 128 procs max_latency=7.501 ms Throughput 75669.4 MB/sec 256 clients 256 procs max_latency=14.984 ms + echo IDLE_SMT > /sys/kernel/debug/sched_features Throughput 592.799 MB/sec 1 clients 1 procs max_latency=0.120 ms Throughput 1208.28 MB/sec 2 clients 2 procs max_latency=0.078 ms Throughput 2319.22 MB/sec 4 clients 4 procs max_latency=0.141 ms Throughput 4196.64 MB/sec 8 clients 8 procs max_latency=0.253 ms Throughput 7816.47 MB/sec 16 clients 16 procs max_latency=0.117 ms Throughput 14990.8 MB/sec 32 clients 32 procs max_latency=0.189 ms Throughput 21809.4 MB/sec 64 clients 64 procs max_latency=0.832 ms Throughput 44813 MB/sec 128 clients 128 procs max_latency=7.930 ms Throughput 75978.1 MB/sec 256 clients 256 procs max_latency=7.337 ms