From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71316C41514 for ; Sun, 25 Aug 2019 10:20:10 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 0861F217D7 for ; Sun, 25 Aug 2019 10:20:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0861F217D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 816AA4A5A0; Sun, 25 Aug 2019 06:20:09 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jvHqUg+xK03s; Sun, 25 Aug 2019 06:20:08 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 5F2304A5A8; Sun, 25 Aug 2019 06:20:08 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 190394A5A0 for ; Sun, 25 Aug 2019 06:20:08 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QkEuie4zo7JU for ; Sun, 25 Aug 2019 06:20:06 -0400 (EDT) Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by mm01.cs.columbia.edu (Postfix) with ESMTP id BD9B54A4DF for ; Sun, 25 Aug 2019 06:20:06 -0400 (EDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 62E03344; Sun, 25 Aug 2019 03:20:06 -0700 (PDT) Received: from big-swifty.misterjones.org (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBC403F246; Sun, 25 Aug 2019 03:20:02 -0700 (PDT) Date: Sun, 25 Aug 2019 11:20:00 +0100 Message-ID: <86wof1mtrj.wl-maz@kernel.org> From: Marc Zyngier To: Auger Eric Subject: Re: [PATCH] KVM: arm/arm64: vgic: Use a single IO device per redistributor In-Reply-To: References: <20190823173330.23342-1-eric.auger@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/26 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Approximate MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Cc: zhang.zhanghailiang@huawei.com, kvm@vger.kernel.org, andre.przywara@arm.com, linux-kernel@vger.kernel.org, qemu-arm@nongnu.org, kvmarm@lists.cs.columbia.edu, eric.auger.pro@gmail.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Fri, 23 Aug 2019 18:52:32 +0100, Auger Eric wrote: > > Hi Zenghui, Marc, > > On 8/23/19 7:33 PM, Eric Auger wrote: > > At the moment we use 2 IO devices per GICv3 redistributor: one > > one for the RD_base frame and one for the SGI_base frame. > > > > Instead we can use a single IO device per redistributor (the 2 > > frames are contiguous). This saves slots on the KVM_MMIO_BUS > > which is currently limited to NR_IOBUS_DEVS (1000). > > > > This change allows to instantiate up to 512 redistributors and may > > speed the guest boot with a large number of VCPUs. > > > > Signed-off-by: Eric Auger > > I tested this patch with below kernel and QEMU branches: > kernel: https://github.com/eauger/linux/tree/256fix-v1 > (Marc's patch + this patch) > https://github.com/eauger/qemu/tree/v4.1.0-256fix-rfc1-rc0 > (header update + kvm_arm_gic_set_irq modification) A small comment on this: you don't seem to check that KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 is available before allowing more than 256 vcpus. It'd be worth doing that before allowing a guest to start. > > On a machine with 224 pcpus, I was able to boot a 512 vcpu guest. > > As expected, qemu outputs warnings: > > qemu-system-aarch64: warning: Number of SMP cpus requested (512) exceeds > the recommended cpus supported by KVM (224) > qemu-system-aarch64: warning: Number of hotpluggable cpus requested > (512) exceeds the recommended cpus supported by KVM (224) > > on the guest: getconf _NPROCESSORS_ONLN returns 512 > > Then I have no clue about what can be expected of such overcommit config > and I have not further exercised the guest at the moment. It will just work, albeit slowly. I often boot 64 vcpu guests on 4 or 8 core systems, just to check that we don't regress too much. > But at least > it seems to boot properly. I also tested without overcommit and it seems > to behave as before (boot, migration). > > I still need to look at the migration of > 256vcpu guest at qemu level. OK, please let us know how it goes. I'd like some more reviewing on the userspace ABI change before merging it though. Peter, your input would be very good to have. Thanks, M. -- Jazz is not dead, it just smells funny. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm