From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751112AbeAVMJc convert rfc822-to-8bit (ORCPT ); Mon, 22 Jan 2018 07:09:32 -0500 Received: from mout.gmx.net ([212.227.17.20]:56884 "EHLO mout.gmx.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750886AbeAVMJa (ORCPT ); Mon, 22 Jan 2018 07:09:30 -0500 Message-ID: <1516622939.24679.5.camel@gmx.de> Subject: Re: unixbench context switch perfomance & cpu topology From: Mike Galbraith To: Wanpeng Li , linux-kernel@vger.kernel.org, kvm Cc: Paolo Bonzini , Peter Zijlstra , Radim Krcmar , Frederic Weisbecker , Thomas Gleixner , Ingo Molnar Date: Mon, 22 Jan 2018 13:08:59 +0100 In-Reply-To: References: Content-Type: text/plain; charset="ISO-8859-15" X-Mailer: Evolution 3.20.5 Mime-Version: 1.0 Content-Transfer-Encoding: 8BIT X-Provags-ID: V03:K0:H7IeXNz8mYP+OLl/n/BhszEkiF/affEzwj4yFhuq0IXvHPmIm5b ua5TINXbKvrz+qTrKTdkb7oUiLzWPNZphigp6IeJYncS2U45ivMlLSRBDfMWoEij/QMYCQw kPqoHilCFhv9j/GBv1mA1ynb1+cEyROy/K4HL3SWWOxyAMuT3JLhGrCNty4Rg7tgbodo638 HWhWMrBYxG979suG8MOmA== X-UI-Out-Filterresults: notjunk:1;V01:K0:TC/8DPdS54Y=:j4E7rqe8i5OmXIM5w4Ht0x XorolM4/kZaMcWcIza8XAXrUrAh1+Zy+EgYnlWxw363BWVBv/3mpEJGTz3AdFY0eW1ONIEVK4 BxiLj3FGanZGuIsPjn9KFOISIfzMA6qf4eu7xrOKj53PxutTGqklTsO+z6om9iisaupYL/l74 e/96S+Mk4I3ZIctEyq63cnXIScstust6kvO1HmLIBXgTF4LzDOREy1Qw/XYFZpctsrQe3Md7j Cnlxz1NZg5iLRoOzOK6G4lb2rjoRk4THVOgRT3FSfAVP3tm9HPBY16hb+teVkduHRibdR58+K mkWkAtc5kRyNYpl+QXHjeEaeCmD4sy3/B+3ajIn06NlvSXA/QmbZzDdxomQmX2fAjTuYX6bQm XRzA64R2FYkY+GtncGm3aS1P2vBfi+9U31PK1uVOTgHYCB4b3prwjfBJgEqgahfRCIzN1bmuz 2jANu1prvL3thr7soUrveSJqiUm4cb18E5zFduWzRmwYSJb/3bagCgOBp9KcV2NF9ijTxp492 0Ylt+MzdIk7TQ9Jn74lbSe57e6hOce/fmfWz/0HOXXJXhIlCIYiEVvwvz5cF+eBntcCeKtG+t duGl3+Pi6eNDqJzhgPIOIKKI3+WZQ9avAhvK9UBrMBxXRUaHzI2WuLmhFQXdKpWTkvUMO0ddc MD8AJBZ2N/Rphx/aMfgpgeqdjj/XLf1sDIuzkNbj1UjOMNNnYhMXtXQbHuukIyB9hXa3Lcb8J l0SwqqJSo3Io/hJxPvS8sMIdThvNF+i/b+o2Nzfn8uZwi6kS/GEBAPT9rvrt4EBAzPqPztp9h DeAxxkx2y9/mkCzIEyGfEH74tSfEN+3GMp8fHOyhj6Ar8TmQGI= Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 2018-01-22 at 19:47 +0800, Wanpeng Li wrote: > Hi all, > > We can observe unixbench context switch performance is heavily > influenced by cpu topology which is exposed to the guest. the score is > posted below, bigger is better, both the guest and the host kernel are > 3.15-rc3(we can also reproduce against centos 7.4 693 guest/host), LLC > is exposed to the guest, kvm adaptive halt-polling is default enabled, > then start a guest w/ 8 logical cpus. > > > > unixbench context switch > -smp 8, sockets=8, cores=1, threads=1 382036 > -smp 8, sockets=4, cores=2, threads=1 132480 > -smp 8, sockets=2, cores=4, threads=1 128032 > -smp 8, sockets=2, cores=2, threads=2 131767 > -smp 8, sockets=1, cores=4, threads=2 132742 > -smp 8, sockets=1, cores=4, threads=2 (guest w/ nohz=off idle=poll) 331471 > > I can observe there are a lot of reschedule IPIs sent from one vCPU to > another vCPU, the context switch workload switches between running and > idle frequently which results in HLT instruction in the idle path, I > use idle=poll to avoid vmexit due to HLT and to avoid reschedule IPIs > since idle task checks TIF_NEED_RESCHED flags in a loop, nohz=off can > stop to program lapic timer/other nohz stuffs. Any idea why sockets=8 > can get best performance? Probably because with that topology, there is no shared llc, thus no cross-core scheduling, micro-benchmark waker/wakee are stacked.  If your benchmark does nothing but schedule, stacking makes beautiful (but utterly meaningless) numbers. -Mike