From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EDFEC3A5A4 for ; Sun, 25 Aug 2019 10:20:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 579662190F for ; Sun, 25 Aug 2019 10:20:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1566728410; bh=J+6+FalZW9fTPxHV4gxPkSUwM/FzSjxZg37dWQ8Mqso=; h=Date:From:To:Cc:Subject:In-Reply-To:References:List-ID:From; b=UTA8hWTcKfeexKJ6eIfT3Z3CP3ZlxN59d61EE2P+1Da+Z99UX6QDliY0aXy1LpG31 h8ZbI/0yePnlOc8TM1iGfZ9EVSX3VGw6T92jDFGAZwk1qDeS/EVIVyAPnnsipy3k3o y4s32b2nyS7nLp69MuK/AK7VA08RtVjrfmz7V+7c= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726660AbfHYKUH (ORCPT ); Sun, 25 Aug 2019 06:20:07 -0400 Received: from foss.arm.com ([217.140.110.172]:48800 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725995AbfHYKUH (ORCPT ); Sun, 25 Aug 2019 06:20:07 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 62E03344; Sun, 25 Aug 2019 03:20:06 -0700 (PDT) Received: from big-swifty.misterjones.org (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DBC403F246; Sun, 25 Aug 2019 03:20:02 -0700 (PDT) Date: Sun, 25 Aug 2019 11:20:00 +0100 Message-ID: <86wof1mtrj.wl-maz@kernel.org> From: Marc Zyngier To: Auger Eric Cc: eric.auger.pro@gmail.com, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, yuzenghui@huawei.com, zhang.zhanghailiang@huawei.com, wanghaibin.wang@huawei.com, james.morse@arm.com, qemu-arm@nongnu.org, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com, peter.maydell@linaro.org, andre.przywara@arm.com Subject: Re: [PATCH] KVM: arm/arm64: vgic: Use a single IO device per redistributor In-Reply-To: References: <20190823173330.23342-1-eric.auger@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL/10.8 EasyPG/1.0.0 Emacs/26 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Organization: Approximate MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Fri, 23 Aug 2019 18:52:32 +0100, Auger Eric wrote: > > Hi Zenghui, Marc, > > On 8/23/19 7:33 PM, Eric Auger wrote: > > At the moment we use 2 IO devices per GICv3 redistributor: one > > one for the RD_base frame and one for the SGI_base frame. > > > > Instead we can use a single IO device per redistributor (the 2 > > frames are contiguous). This saves slots on the KVM_MMIO_BUS > > which is currently limited to NR_IOBUS_DEVS (1000). > > > > This change allows to instantiate up to 512 redistributors and may > > speed the guest boot with a large number of VCPUs. > > > > Signed-off-by: Eric Auger > > I tested this patch with below kernel and QEMU branches: > kernel: https://github.com/eauger/linux/tree/256fix-v1 > (Marc's patch + this patch) > https://github.com/eauger/qemu/tree/v4.1.0-256fix-rfc1-rc0 > (header update + kvm_arm_gic_set_irq modification) A small comment on this: you don't seem to check that KVM_CAP_ARM_IRQ_LINE_LAYOUT_2 is available before allowing more than 256 vcpus. It'd be worth doing that before allowing a guest to start. > > On a machine with 224 pcpus, I was able to boot a 512 vcpu guest. > > As expected, qemu outputs warnings: > > qemu-system-aarch64: warning: Number of SMP cpus requested (512) exceeds > the recommended cpus supported by KVM (224) > qemu-system-aarch64: warning: Number of hotpluggable cpus requested > (512) exceeds the recommended cpus supported by KVM (224) > > on the guest: getconf _NPROCESSORS_ONLN returns 512 > > Then I have no clue about what can be expected of such overcommit config > and I have not further exercised the guest at the moment. It will just work, albeit slowly. I often boot 64 vcpu guests on 4 or 8 core systems, just to check that we don't regress too much. > But at least > it seems to boot properly. I also tested without overcommit and it seems > to behave as before (boot, migration). > > I still need to look at the migration of > 256vcpu guest at qemu level. OK, please let us know how it goes. I'd like some more reviewing on the userspace ABI change before merging it though. Peter, your input would be very good to have. Thanks, M. -- Jazz is not dead, it just smells funny.