From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE25646AA for ; Fri, 3 Mar 2023 13:06:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677848773; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=v07IDWMxZry0XXohrnatEt4gHWwOiAbWwK6lrNuE9qk=; b=YEVCgr67jqOP7fUqN94e9uSC7SbvQBKo9872P6vrfxhLO/Wxa+hm0YCPhR8dYFrPjZZD3v ZhdwSG3p+MHNGbWZdi7bMSYwBxfMEgwQ6y5o2nH63e7kj9WB3BQjPOcnA+IdG/N8kiHZmP 9Ig/bqHV5yrY9+xbIbaToem4cozwuKI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-356-caECKmB_PNWeUU7H5QlysA-1; Fri, 03 Mar 2023 08:06:10 -0500 X-MC-Unique: caECKmB_PNWeUU7H5QlysA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 991A0802D2F; Fri, 3 Mar 2023 13:06:09 +0000 (UTC) Received: from localhost (unknown [10.39.194.66]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 05B5140C6EC4; Fri, 3 Mar 2023 13:06:08 +0000 (UTC) From: Cornelia Huck To: Suzuki K Poulose , Andrew Jones , Jean-Philippe Brucker Cc: Jean-Philippe Brucker , Itaru Kitayama , linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Catalin Marinas , Chao Peng , Christoffer Dall , Fuad Tabba , James Morse , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Sean Christopherson , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.cs.columbia.edu Subject: Re: [RFC] Support for Arm CCA VMs on Linux In-Reply-To: <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com> Organization: Red Hat GmbH References: <20230127112248.136810-1-suzuki.poulose@arm.com> <20230303094618.GC361458@myrica> <1c91b777-982e-e71a-4829-51744e9555c5@arm.com> <20230303113905.GD361458@myrica> <20230303120800.ahtyc6et77ig4s27@orel> <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com> User-Agent: Notmuch/0.37 (https://notmuchmail.org) Date: Fri, 03 Mar 2023 14:06:07 +0100 Message-ID: <875ybi0ytc.fsf@redhat.com> Precedence: bulk X-Mailing-List: linux-coco@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 On Fri, Mar 03 2023, Suzuki K Poulose wrote: > On 03/03/2023 12:08, Andrew Jones wrote: >> On Fri, Mar 03, 2023 at 11:39:05AM +0000, Jean-Philippe Brucker wrote: >>> On Fri, Mar 03, 2023 at 09:54:47AM +0000, Suzuki K Poulose wrote: >>>> On 03/03/2023 09:46, Jean-Philippe Brucker wrote: >>>>> On Thu, Mar 02, 2023 at 07:12:24AM +0900, Itaru Kitayama wrote: >>>>>>>> I've tried your series in Real on CCA Host, but the KVM arch init >>>>>>>> emits an Invalid argument error and terminates. >>>>> >>>>> This was the KVM_SET_ONE_REG for the SVE vector size. During my tests I >>>>> didn't enable SVE in the host but shrinkwrap enables more options. >>>> >>>> Does the Qemu check for SVE capability on /dev/kvm ? For kvmtool, we >>>> changed to using the VM instance and that would prevent using SVE, >>>> until the RMM supports it. >>> >>> Yes, QEMU does check the SVE cap on /dev/kvm. I can propose changing it or >>> complementing it with a VM check in my next version, it seems to work >>> (though I need to double-check the VM fd lifetime). Same goes for >>> KVM_CAP_STEAL_TIME, which I need to disable explicitly at the moment. >> >> I'm probably missing something since I haven't looked at this, but I'm >> wondering what the "VM instance" check is and why it should be necessary. > > Userspace can check for a KVM_CAP_ on KVM fd (/dev/kvm) or a VM fd > (returned via KVM_CREATE_VM). > >> Shouldn't KVM only expose capabilities which it can provide? I.e. the > > Correct, given now that we have different "types" of VMs possible on > Arm64, (Normal vs Realm vs pVM), the capabilities of each of these > could be different and thus we should use the KVM_CAP_ on the VM fd ( > referred to VM instance above) and not the generic KVM fd. Using the vm ioctl is even encouraged in the doc for KVM_CHECK_EXTENSION: "Based on their initialization different VMs may have different capabilities. It is thus encouraged to use the vm ioctl to query for capabilities" It would probably make sense to convert QEMU to use the vm ioctl everywhere (the wrapper falls back to the global version on failure anyway.) > >> "VM instance" check should be done by KVM and, when it fails, the SVE and >> steal-time capabilities should return 0. >> > > Correct. > > Suzuki > >> Thanks, >> drew From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BFED6C64EC4 for ; Fri, 3 Mar 2023 13:07:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=oz9H2PjC8tvshaF1jBe7m9iLeiHPe4M+T7TNmgnm9fk=; b=gAqf86YSDuywnr BVe0LQyqTRvMbRhufDQahAOgczSuvrftJwl1SvDc/tnc79fBxp3lf1lBFiB7so5yoKD2qHBPUht2h txmYgsqTnMFGBbtG62w1Hs/Hvbgstee6b8oh1asrqczTnO8LUyPIc2upxD3trOC4echtOmqvUCfly PwhUHQn5VyJFSgzfzCJ4ycfz1w95+k06p17YetLQCokjxAr7QuXywbl8YoX/wdFDFn668mijqv8Np rLVFm3wl0SobtqPREi4gVdC0TWn5P0nd3QoUC0JTJlPdbGOOy5qsv4+CNfZv3vHVlEHw49Yo8odRy rTy22emlKZN4gH7UcgOA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pY57B-006OsM-4y; Fri, 03 Mar 2023 13:06:25 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pY577-006Onp-DV for linux-arm-kernel@lists.infradead.org; Fri, 03 Mar 2023 13:06:23 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1677848775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=v07IDWMxZry0XXohrnatEt4gHWwOiAbWwK6lrNuE9qk=; b=MBtxn6/mrvR815duzE8UUgfpB0d63sXIvWJCH4gA45maxvWAxeuHNBq5Ul+PRfBUuiyvwK qF0Dbd5avIYGzPzCX8y+RJfWb5P2rlE1TRjTx09hoeF2IFCFQOqgLizR+Ba4Jb/chjLmMr 2d99yE6X+AnKwksaYK9FnOai9CbBW/c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-356-caECKmB_PNWeUU7H5QlysA-1; Fri, 03 Mar 2023 08:06:10 -0500 X-MC-Unique: caECKmB_PNWeUU7H5QlysA-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 991A0802D2F; Fri, 3 Mar 2023 13:06:09 +0000 (UTC) Received: from localhost (unknown [10.39.194.66]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 05B5140C6EC4; Fri, 3 Mar 2023 13:06:08 +0000 (UTC) From: Cornelia Huck To: Suzuki K Poulose , Andrew Jones , Jean-Philippe Brucker Cc: Jean-Philippe Brucker , Itaru Kitayama , linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Catalin Marinas , Chao Peng , Christoffer Dall , Fuad Tabba , James Morse , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Sean Christopherson , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.cs.columbia.edu Subject: Re: [RFC] Support for Arm CCA VMs on Linux In-Reply-To: <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com> Organization: Red Hat GmbH References: <20230127112248.136810-1-suzuki.poulose@arm.com> <20230303094618.GC361458@myrica> <1c91b777-982e-e71a-4829-51744e9555c5@arm.com> <20230303113905.GD361458@myrica> <20230303120800.ahtyc6et77ig4s27@orel> <2418536c-2658-18d6-f70c-c1af5adaa816@arm.com> User-Agent: Notmuch/0.37 (https://notmuchmail.org) Date: Fri, 03 Mar 2023 14:06:07 +0100 Message-ID: <875ybi0ytc.fsf@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230303_050621_567121_7A0F4C1A X-CRM114-Status: GOOD ( 21.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Mar 03 2023, Suzuki K Poulose wrote: > On 03/03/2023 12:08, Andrew Jones wrote: >> On Fri, Mar 03, 2023 at 11:39:05AM +0000, Jean-Philippe Brucker wrote: >>> On Fri, Mar 03, 2023 at 09:54:47AM +0000, Suzuki K Poulose wrote: >>>> On 03/03/2023 09:46, Jean-Philippe Brucker wrote: >>>>> On Thu, Mar 02, 2023 at 07:12:24AM +0900, Itaru Kitayama wrote: >>>>>>>> I've tried your series in Real on CCA Host, but the KVM arch init >>>>>>>> emits an Invalid argument error and terminates. >>>>> >>>>> This was the KVM_SET_ONE_REG for the SVE vector size. During my tests I >>>>> didn't enable SVE in the host but shrinkwrap enables more options. >>>> >>>> Does the Qemu check for SVE capability on /dev/kvm ? For kvmtool, we >>>> changed to using the VM instance and that would prevent using SVE, >>>> until the RMM supports it. >>> >>> Yes, QEMU does check the SVE cap on /dev/kvm. I can propose changing it or >>> complementing it with a VM check in my next version, it seems to work >>> (though I need to double-check the VM fd lifetime). Same goes for >>> KVM_CAP_STEAL_TIME, which I need to disable explicitly at the moment. >> >> I'm probably missing something since I haven't looked at this, but I'm >> wondering what the "VM instance" check is and why it should be necessary. > > Userspace can check for a KVM_CAP_ on KVM fd (/dev/kvm) or a VM fd > (returned via KVM_CREATE_VM). > >> Shouldn't KVM only expose capabilities which it can provide? I.e. the > > Correct, given now that we have different "types" of VMs possible on > Arm64, (Normal vs Realm vs pVM), the capabilities of each of these > could be different and thus we should use the KVM_CAP_ on the VM fd ( > referred to VM instance above) and not the generic KVM fd. Using the vm ioctl is even encouraged in the doc for KVM_CHECK_EXTENSION: "Based on their initialization different VMs may have different capabilities. It is thus encouraged to use the vm ioctl to query for capabilities" It would probably make sense to convert QEMU to use the vm ioctl everywhere (the wrapper falls back to the global version on failure anyway.) > >> "VM instance" check should be done by KVM and, when it fails, the SVE and >> steal-time capabilities should return 0. >> > > Correct. > > Suzuki > >> Thanks, >> drew _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel