From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <linux-kernel-owner@vger.kernel.org>
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
        id S1751233AbeAVM2A (ORCPT <rfc822;w@1wt.eu>);
        Mon, 22 Jan 2018 07:28:00 -0500
Received: from mail-ot0-f178.google.com ([74.125.82.178]:40811 "EHLO
        mail-ot0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
        with ESMTP id S1751041AbeAVM17 (ORCPT
        <rfc822;linux-kernel@vger.kernel.org>);
        Mon, 22 Jan 2018 07:27:59 -0500
X-Google-Smtp-Source: AH8x227zjqgpr2beTX5+aDqFkSRNO8Rp1h+jhoBP4qAkcnmEhcfGYauGINQm8IM5kCi+/441J8gXqCFZEISKrUOpYTs=
MIME-Version: 1.0
In-Reply-To: <1516622939.24679.5.camel@gmx.de>
References: <CANRm+CwpGoSTLNG3D8KN+fgfc+20-gFND7xGAYo2b5EuwzPeOg@mail.gmail.com>
 <1516622939.24679.5.camel@gmx.de>
From: Wanpeng Li <kernellwp@gmail.com>
Date: Mon, 22 Jan 2018 20:27:58 +0800
Message-ID: <CANRm+CyXVkQKbHFMQG4v3fjUuPOHpY15_c6gU+gaUzf_67xnFg@mail.gmail.com>
Subject: Re: unixbench context switch perfomance & cpu topology
To: Mike Galbraith <efault@gmx.de>
Cc: linux-kernel@vger.kernel.org, kvm <kvm@vger.kernel.org>,
        Paolo Bonzini <pbonzini@redhat.com>,
        Peter Zijlstra <peterz@infradead.org>,
        Radim Krcmar <rkrcmar@redhat.com>,
        Frederic Weisbecker <fweisbec@gmail.com>,
        Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@kernel.org>
Content-Type: text/plain; charset="UTF-8"
Sender: linux-kernel-owner@vger.kernel.org
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org

2018-01-22 20:08 GMT+08:00 Mike Galbraith <efault@gmx.de>:
> On Mon, 2018-01-22 at 19:47 +0800, Wanpeng Li wrote:
>> Hi all,
>>
>> We can observe unixbench context switch performance is heavily
>> influenced by cpu topology which is exposed to the guest. the score is
>> posted below, bigger is better, both the guest and the host kernel are
>> 3.15-rc3(we can also reproduce against centos 7.4 693 guest/host), LLC
>> is exposed to the guest, kvm adaptive halt-polling is default enabled,
>> then start a guest w/ 8 logical cpus.
>>
>>
>>
>> unixbench context switch
>> -smp 8, sockets=8, cores=1, threads=1    382036
>> -smp 8, sockets=4, cores=2, threads=1    132480
>> -smp 8, sockets=2, cores=4, threads=1    128032
>> -smp 8, sockets=2, cores=2, threads=2    131767
>> -smp 8, sockets=1, cores=4, threads=2    132742
>> -smp 8, sockets=1, cores=4, threads=2 (guest w/ nohz=off idle=poll)    331471
>>
>> I can observe there are a lot of reschedule IPIs sent from one vCPU to
>> another vCPU, the context switch workload switches between running and
>> idle frequently which results in HLT instruction in the idle path, I
>> use idle=poll to avoid vmexit due to HLT and to avoid reschedule IPIs
>> since idle task checks TIF_NEED_RESCHED flags in a loop, nohz=off can
>> stop to program lapic timer/other nohz stuffs. Any idea why sockets=8
>> can get best performance?
>
> Probably because with that topology, there is no shared llc, thus no
> cross-core scheduling, micro-benchmark waker/wakee are stacked.  If
> your benchmark does nothing but schedule, stacking makes beautiful (but
> utterly meaningless) numbers.

The waker and wakee are just sporadic on the same logical cpu in the
guest(-smp 8, sockets=8, cores=1, threads=1) during the testing, in
addition, binding the waker/wakee to one logical cpu in the guest(-smp
8, sockets=1, cores=4, threads=2) also can get the performance as
better as 8 sockets setup.

Regards,
Wanpeng Li