From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9A52AC54EAA for ; Fri, 27 Jan 2023 11:24:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=/MxNpFbG8lZNSaixYDbXcChuNth1TFznPJItxukFxLc=; b=E7U+fvb277RJAa fjuirEHQdGyxeo3Pnz23MwpjsQZC5+H/u1U4xQ48KjtXHaqb5HzGBsQ2NfjEVpAcFtpIjQJX6mety uAjdSNw+fvHzBfr3zQdSP4lCnGWtCTQPWdfTKNN8OMhq9qZR/S/m4MD8m9ApXDQrzVNRWeSIsVH7J WklfqtDGoqN3gIzPEMhNnfUnqKzUQQWZlpro85clSGGnUfWBdQz8XEmsGZMhfLjUs3g6WbQlLln7H TcEB7zSD7apW9AcstNxONM/HRAHtBgMLDMQAjVU2yATUZ2G9xfsTLVJOLvMbYqkfi8pJT86kTpd0Q BkJ/cV3xvGsQK7R2KLPQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLMpV-00E6kR-QQ; Fri, 27 Jan 2023 11:23:37 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pLMpR-00E6jh-Th for linux-arm-kernel@lists.infradead.org; Fri, 27 Jan 2023 11:23:36 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6DE432B; Fri, 27 Jan 2023 03:24:13 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9335C3F64C; Fri, 27 Jan 2023 03:23:28 -0800 (PST) From: Suzuki K Poulose To: linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andrew Jones , Catalin Marinas , Chao Peng , Christoffer Dall , Fuad Tabba , James Morse , Jean-Philippe Brucker , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Sean Christopherson , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.cs.columbia.edu Subject: [RFC] Support for Arm CCA VMs on Linux Date: Fri, 27 Jan 2023 11:22:48 +0000 Message-Id: <20230127112248.136810-1-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230127_032334_078024_BE0E5E8E X-CRM114-Status: GOOD ( 22.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org We are happy to announce the early RFC version of the Arm Confidential Compute Architecture (CCA) support for the Linux stack. The intention is to seek early feedback in the following areas: * KVM integration of the Arm CCA * KVM UABI for managing the Realms, seeking to generalise the operations wherever possible with other Confidential Compute solutions. Note: This version doesn't support Guest Private memory, which will be added later (see below). * Linux Guest support for Realms Arm CCA Introduction ===================== The Arm CCA is a reference software architecture and implementation that builds on the Realm Management Extension (RME), enabling the execution of Virtual machines, while preventing access by more privileged software, such as hypervisor. The Arm CCA allows the hypervisor to control the VM, but removes the right for access to the code, register state or data that is used by VM. More information on the architecture is available here[0]. Arm CCA Reference Software Architecture Realm World || Normal World || Secure World || || | || || EL0 x-------x || x----x | x------x || || | Realm | || | | | | | || || | | || | VM | | | | || || ----| VM* |---------||-| |---| |-||----------------|| | | || | | | | H | || || EL1 x-------x || x----x | | | || || ^ || | | o | || || | || | | | || || ------- R*------------------------| s -|--------------------- S || | | || || I || | t | || || | || | | || || v || x------x || || EL2 RMM* || ^ || || ^ || | || || ========|=============================|======================== | | SMC x--------- *RMI* -------------x EL3 Root World EL3 Firmware =============================================================== Where : RMM - Realm Management Monitor RMI - Realm Management Interface RSI - Realm Service Interface SMC - Secure Monitor Call RME introduces a new security state "Realm world", in addition to the traditional Secure and Non-Secure states. The Arm CCA defines a new component, Realm Management Monitor (RMM) that runs at R-EL2. This is a standard piece of firmware, verified, installed and loaded by the EL3 firmware (e.g, TF-A), at system boot. The RMM provides standard interfaces - Realm Management Interface (RMI) - to the Normal world hypervisor to manage the VMs running in the Realm world (also called Realms in short). These are exposed via SMC and are routed through the EL3 firmwre. The RMI interface includes: - Move a physical page from the Normal world to the Realm world - Creating a Realm with requested parameters, tracked via Realm Descriptor (RD) - Creating VCPUs aka Realm Execution Context (REC), with initial register state. - Create stage2 translation table at any level. - Load initial images into Realm Memory from normal world memory - Schedule RECs (vCPUs) and handle exits - Inject virtual interrupts into the Realm - Service stage2 runtime faults with pages (provided by host, scrubbed by RMM). - Create "shared" mappings that can be accessed by VMM/Hyp. - Reclaim the memory allocated for the RAM and RTTs (Realm Translation Tables) However v1.0 of RMM specifications doesn't support: - Paging protected memory of a Realm VM. Thus the pages backing the protected memory region must be pinned. - Live migration of Realms. - Trusted Device assignment. - Physical interrupt backed Virtual interrupts for Realms RMM also provides certain services to the Realms via SMC, called Realm Service Interface (RSI). These include: - Realm Guest Configuration. - Attestation & Measurement services - Managing the state of an Intermediate Physical Address (IPA aka GPA) page. - Host Call service (Communication with the Normal world Hypervisor) The specifications for the RMM software is currently at *v1.0-Beta2* and the latest version is available here [1]. The Trusted Firmware foundation has an implementation of the RMM - TF-RMM - available here [3]. Implementation ================= This version of the stack is based on the RMM specification v1.0-Beta0[2], with following exceptions : - TF-RMM/KVM currently doesn't support the optional features of PMU, SVE and Self-hosted debug (coming soon). - The RSI_HOST_CALL structure alignment requirement is reduced to match RMM v1.0 Beta1 - RMI/RSI version numbers do not match the RMM spec. This will be resolved once the spec/implementation is complete, across TF-RMM+Linux stack. We plan to update the stack to support the latest version of the RMMv1.0 spec in the coming revisions. This release includes the following components : a) Linux Kernel i) Host / KVM support - Support for driving the Realms via RMI. This is dependent on running in the Kernel at EL2 (aka VHE mode). Also provides UABI for VMMs to manage the Realm VMs. The support is restricted to 4K page size, matching the Stage2 granule supported by RMM. The VMM is responsible for making sure the guest memory is locked. TODO: Guest Private memory[10] integration - We have been following the series and support will be added once it is merged upstream. ii) Guest support - Support for a Linux Kernel to run in the Realm VM at Realm-EL1, using RSI services. This includes virtio support (virtio-v1.0 only). All I/O are treated as non-secure/shared. c) kvmtool - VMM changes required to manage Realm VMs. No guest private memory as mentioned above. d) kvm-unit-tests - Support for running in Realms along with additional tests for RSI ABI. Running the stack ==================== To run/test the stack, you would need the following components : 1) FVP Base AEM RevC model with FEAT_RME support [4] 2) TF-A firmware for EL3 [5] 3) TF-A RMM for R-EL2 [3] 4) Linux Kernel [6] 5) kvmtool [7] 6) kvm-unit-tests [8] Instructions for building the firmware components and running the model are available here [9]. Once, the host kernel is booted, a Realm can be launched by invoking the `lkvm` commad as follows: $ lkvm run --realm \ --measurement-algo=["sha256", "sha512"] \ --disable-sve \ Where: * --measurement-algo (Optional) specifies the algorithm selected for creating the initial measurements by the RMM for this Realm (defaults to sha256). * GICv3 is mandatory for the Realms. * SVE is not yet supported in the TF-RMM, and thus must be disabled using --disable-sve You may also run the kvm-unit-tests inside the Realm world, using the similar options as above. Links ============ [0] Arm CCA Landing page (See Key Resources section for various documentations) https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture [1] RMM Specification Latest https://developer.arm.com/documentation/den0137/latest [2] RMM v1.0-Beta0 specification https://developer.arm.com/documentation/den0137/1-0bet0/ [3] Trusted Firmware RMM - TF-RMM https://www.trustedfirmware.org/projects/tf-rmm/ GIT: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git [4] FVP Base RevC AEM Model (available on x86_64 / Arm64 Linux) https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms [5] Trusted Firmware for A class https://www.trustedfirmware.org/projects/tf-a/ [6] Linux kernel support for Arm-CCA https://gitlab.arm.com/linux-arm/linux-cca Host Support branch: cca-host/rfc-v1 Guest Support branch: cca-guest/rfc-v1 [7] kvmtool support for Arm CCA https://gitlab.arm.com/linux-arm/kvmtool-cca cca/rfc-v1 [8] kvm-unit-tests support for Arm CCA https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca cca/rfc-v1 [9] Instructions for Building Firmware components and running the model, see section 4.19.2 "Building and running TF-A with RME" https://trustedfirmware-a.readthedocs.io/en/latest/components/realm-management-extension.html#building-and-running-tf-a-with-rme [10] fd based Guest Private memory for KVM https://lkml.kernel.org/r/20221202061347.1070246-1-chao.p.peng@linux.intel.com Cc: Alexandru Elisei Cc: Andrew Jones Cc: Catalin Marinas Cc: Chao Peng Cc: Christoffer Dall Cc: Fuad Tabba Cc: James Morse Cc: Jean-Philippe Brucker Cc: Joey Gouly Cc: Marc Zyngier Cc: Mark Rutland Cc: Oliver Upton Cc: Paolo Bonzini Cc: Quentin Perret Cc: Sean Christopherson Cc: Steven Price Cc: Thomas Huth Cc: Will Deacon Cc: Zenghui Yu To: linux-coco@lists.linux.dev To: kvmarm@lists.linux.dev Cc: kvmarm@lists.cs.columbia.edu Cc: linux-arm-kernel@lists.infradead.org To: linux-kernel@vger.kernel.org To: kvm@vger.kernel.org _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 8C73D28F2; Fri, 27 Jan 2023 11:30:46 +0000 (UTC) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6DE432B; Fri, 27 Jan 2023 03:24:13 -0800 (PST) Received: from ewhatever.cambridge.arm.com (ewhatever.cambridge.arm.com [10.1.197.1]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9335C3F64C; Fri, 27 Jan 2023 03:23:28 -0800 (PST) From: Suzuki K Poulose To: linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org Cc: Alexandru Elisei , Andrew Jones , Catalin Marinas , Chao Peng , Christoffer Dall , Fuad Tabba , James Morse , Jean-Philippe Brucker , Joey Gouly , Marc Zyngier , Mark Rutland , Oliver Upton , Paolo Bonzini , Quentin Perret , Sean Christopherson , Steven Price , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.cs.columbia.edu Subject: [RFC] Support for Arm CCA VMs on Linux Date: Fri, 27 Jan 2023 11:22:48 +0000 Message-Id: <20230127112248.136810-1-suzuki.poulose@arm.com> X-Mailer: git-send-email 2.34.1 Precedence: bulk X-Mailing-List: linux-coco@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit We are happy to announce the early RFC version of the Arm Confidential Compute Architecture (CCA) support for the Linux stack. The intention is to seek early feedback in the following areas: * KVM integration of the Arm CCA * KVM UABI for managing the Realms, seeking to generalise the operations wherever possible with other Confidential Compute solutions. Note: This version doesn't support Guest Private memory, which will be added later (see below). * Linux Guest support for Realms Arm CCA Introduction ===================== The Arm CCA is a reference software architecture and implementation that builds on the Realm Management Extension (RME), enabling the execution of Virtual machines, while preventing access by more privileged software, such as hypervisor. The Arm CCA allows the hypervisor to control the VM, but removes the right for access to the code, register state or data that is used by VM. More information on the architecture is available here[0]. Arm CCA Reference Software Architecture Realm World || Normal World || Secure World || || | || || EL0 x-------x || x----x | x------x || || | Realm | || | | | | | || || | | || | VM | | | | || || ----| VM* |---------||-| |---| |-||----------------|| | | || | | | | H | || || EL1 x-------x || x----x | | | || || ^ || | | o | || || | || | | | || || ------- R*------------------------| s -|--------------------- S || | | || || I || | t | || || | || | | || || v || x------x || || EL2 RMM* || ^ || || ^ || | || || ========|=============================|======================== | | SMC x--------- *RMI* -------------x EL3 Root World EL3 Firmware =============================================================== Where : RMM - Realm Management Monitor RMI - Realm Management Interface RSI - Realm Service Interface SMC - Secure Monitor Call RME introduces a new security state "Realm world", in addition to the traditional Secure and Non-Secure states. The Arm CCA defines a new component, Realm Management Monitor (RMM) that runs at R-EL2. This is a standard piece of firmware, verified, installed and loaded by the EL3 firmware (e.g, TF-A), at system boot. The RMM provides standard interfaces - Realm Management Interface (RMI) - to the Normal world hypervisor to manage the VMs running in the Realm world (also called Realms in short). These are exposed via SMC and are routed through the EL3 firmwre. The RMI interface includes: - Move a physical page from the Normal world to the Realm world - Creating a Realm with requested parameters, tracked via Realm Descriptor (RD) - Creating VCPUs aka Realm Execution Context (REC), with initial register state. - Create stage2 translation table at any level. - Load initial images into Realm Memory from normal world memory - Schedule RECs (vCPUs) and handle exits - Inject virtual interrupts into the Realm - Service stage2 runtime faults with pages (provided by host, scrubbed by RMM). - Create "shared" mappings that can be accessed by VMM/Hyp. - Reclaim the memory allocated for the RAM and RTTs (Realm Translation Tables) However v1.0 of RMM specifications doesn't support: - Paging protected memory of a Realm VM. Thus the pages backing the protected memory region must be pinned. - Live migration of Realms. - Trusted Device assignment. - Physical interrupt backed Virtual interrupts for Realms RMM also provides certain services to the Realms via SMC, called Realm Service Interface (RSI). These include: - Realm Guest Configuration. - Attestation & Measurement services - Managing the state of an Intermediate Physical Address (IPA aka GPA) page. - Host Call service (Communication with the Normal world Hypervisor) The specifications for the RMM software is currently at *v1.0-Beta2* and the latest version is available here [1]. The Trusted Firmware foundation has an implementation of the RMM - TF-RMM - available here [3]. Implementation ================= This version of the stack is based on the RMM specification v1.0-Beta0[2], with following exceptions : - TF-RMM/KVM currently doesn't support the optional features of PMU, SVE and Self-hosted debug (coming soon). - The RSI_HOST_CALL structure alignment requirement is reduced to match RMM v1.0 Beta1 - RMI/RSI version numbers do not match the RMM spec. This will be resolved once the spec/implementation is complete, across TF-RMM+Linux stack. We plan to update the stack to support the latest version of the RMMv1.0 spec in the coming revisions. This release includes the following components : a) Linux Kernel i) Host / KVM support - Support for driving the Realms via RMI. This is dependent on running in the Kernel at EL2 (aka VHE mode). Also provides UABI for VMMs to manage the Realm VMs. The support is restricted to 4K page size, matching the Stage2 granule supported by RMM. The VMM is responsible for making sure the guest memory is locked. TODO: Guest Private memory[10] integration - We have been following the series and support will be added once it is merged upstream. ii) Guest support - Support for a Linux Kernel to run in the Realm VM at Realm-EL1, using RSI services. This includes virtio support (virtio-v1.0 only). All I/O are treated as non-secure/shared. c) kvmtool - VMM changes required to manage Realm VMs. No guest private memory as mentioned above. d) kvm-unit-tests - Support for running in Realms along with additional tests for RSI ABI. Running the stack ==================== To run/test the stack, you would need the following components : 1) FVP Base AEM RevC model with FEAT_RME support [4] 2) TF-A firmware for EL3 [5] 3) TF-A RMM for R-EL2 [3] 4) Linux Kernel [6] 5) kvmtool [7] 6) kvm-unit-tests [8] Instructions for building the firmware components and running the model are available here [9]. Once, the host kernel is booted, a Realm can be launched by invoking the `lkvm` commad as follows: $ lkvm run --realm \ --measurement-algo=["sha256", "sha512"] \ --disable-sve \ Where: * --measurement-algo (Optional) specifies the algorithm selected for creating the initial measurements by the RMM for this Realm (defaults to sha256). * GICv3 is mandatory for the Realms. * SVE is not yet supported in the TF-RMM, and thus must be disabled using --disable-sve You may also run the kvm-unit-tests inside the Realm world, using the similar options as above. Links ============ [0] Arm CCA Landing page (See Key Resources section for various documentations) https://www.arm.com/architecture/security-features/arm-confidential-compute-architecture [1] RMM Specification Latest https://developer.arm.com/documentation/den0137/latest [2] RMM v1.0-Beta0 specification https://developer.arm.com/documentation/den0137/1-0bet0/ [3] Trusted Firmware RMM - TF-RMM https://www.trustedfirmware.org/projects/tf-rmm/ GIT: https://git.trustedfirmware.org/TF-RMM/tf-rmm.git [4] FVP Base RevC AEM Model (available on x86_64 / Arm64 Linux) https://developer.arm.com/Tools%20and%20Software/Fixed%20Virtual%20Platforms [5] Trusted Firmware for A class https://www.trustedfirmware.org/projects/tf-a/ [6] Linux kernel support for Arm-CCA https://gitlab.arm.com/linux-arm/linux-cca Host Support branch: cca-host/rfc-v1 Guest Support branch: cca-guest/rfc-v1 [7] kvmtool support for Arm CCA https://gitlab.arm.com/linux-arm/kvmtool-cca cca/rfc-v1 [8] kvm-unit-tests support for Arm CCA https://gitlab.arm.com/linux-arm/kvm-unit-tests-cca cca/rfc-v1 [9] Instructions for Building Firmware components and running the model, see section 4.19.2 "Building and running TF-A with RME" https://trustedfirmware-a.readthedocs.io/en/latest/components/realm-management-extension.html#building-and-running-tf-a-with-rme [10] fd based Guest Private memory for KVM https://lkml.kernel.org/r/20221202061347.1070246-1-chao.p.peng@linux.intel.com Cc: Alexandru Elisei Cc: Andrew Jones Cc: Catalin Marinas Cc: Chao Peng Cc: Christoffer Dall Cc: Fuad Tabba Cc: James Morse Cc: Jean-Philippe Brucker Cc: Joey Gouly Cc: Marc Zyngier Cc: Mark Rutland Cc: Oliver Upton Cc: Paolo Bonzini Cc: Quentin Perret Cc: Sean Christopherson Cc: Steven Price Cc: Thomas Huth Cc: Will Deacon Cc: Zenghui Yu To: linux-coco@lists.linux.dev To: kvmarm@lists.linux.dev Cc: kvmarm@lists.cs.columbia.edu Cc: linux-arm-kernel@lists.infradead.org To: linux-kernel@vger.kernel.org To: kvm@vger.kernel.org