From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C79EC433DF for ; Mon, 22 Jun 2020 20:03:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA4122076A for ; Mon, 22 Jun 2020 20:03:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="b4uBcbjS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728698AbgFVUDv (ORCPT ); Mon, 22 Jun 2020 16:03:51 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:63405 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728435AbgFVUDu (ORCPT ); Mon, 22 Jun 2020 16:03:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1592856228; x=1624392228; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=EiUt6Oedptz4f3XfYR/pew/AoUF85rOaT6U9ZvC05Jc=; b=b4uBcbjSlpaCjz4Efya5+fsfln0Qs+FIg2uVhm08aeW8Kw1QOQsSEanr mcr3dYJyQO7btZIREiMiV9hROq7bi6rnmLYi++iuqrTzQpfpg+iSAfF7J NfKc3wq2+6etUVzxZhLiC6w0NVWpAHf3hpgauu9I8pk0xAsYqIO3bI/X7 s=; IronPort-SDR: Mb5swMn+KcsFsZBMoka4+NcvherPj4dLUTzXSqCBvliS1qm01gNYlkmsZoG6juXvOsbsCPjMLQ Cje3C9vWRVSA== X-IronPort-AV: E=Sophos;i="5.75,268,1589241600"; d="scan'208";a="46040836" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-cc689b93.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 22 Jun 2020 20:03:46 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-cc689b93.us-west-2.amazon.com (Postfix) with ESMTPS id DAFA1120F63; Mon, 22 Jun 2020 20:03:45 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 22 Jun 2020 20:03:45 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.145) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 22 Jun 2020 20:03:35 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , Greg KH , Martin Pohlack , "Matt Wilson" , Paolo Bonzini , Balbir Singh , Stefano Garzarella , "Stefan Hajnoczi" , Stewart Smith , "Uwe Dannowski" , , , Andra Paraschiv Subject: [PATCH v4 00/18] Add support for Nitro Enclaves Date: Mon, 22 Jun 2020 23:03:11 +0300 Message-ID: <20200622200329.52996-1-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) MIME-Version: 1.0 X-Originating-IP: [10.43.161.145] X-ClientProxiedBy: EX13D12UWC002.ant.amazon.com (10.43.162.253) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Nitro Enclaves (NE) is a new Amazon Elastic Compute Cloud (EC2) capability that allows customers to carve out isolated compute environments within EC2 instances [1]. For example, an application that processes sensitive data and runs in a VM, can be separated from other applications running in the same VM. This application then runs in a separate VM than the primary VM, namely an enclave. An enclave runs alongside the VM that spawned it. This setup matches low latency applications needs. The resources that are allocated for the enclave, such as memory and CPU, are carved out of the primary VM. Each enclave is mapped to a process running in the primary VM, that communicates with the NE driver via an ioctl interface. In this sense, there are two components: 1. An enclave abstraction process - a user space process running in the primary VM guest that uses the provided ioctl interface of the NE driver to spawn an enclave VM (that's 2 below). There is a NE emulated PCI device exposed to the primary VM. The driver for this new PCI device is included in the NE driver. The ioctl logic is mapped to PCI device commands e.g. the NE_START_ENCLAVE ioctl maps to an enclave start PCI command. The PCI device commands are then translated into actions taken on the hypervisor side; that's the Nitro hypervisor running on the host where the primary VM is running. The Nitro hypervisor is based on core KVM technology. 2. The enclave itself - a VM running on the same host as the primary VM that spawned it. Memory and CPUs are carved out of the primary VM and are dedicated for the enclave VM. An enclave does not have persistent storage attached. The memory regions carved out of the primary VM and given to an enclave need to be aligned 2 MiB / 1 GiB physically contiguous memory regions (or multiple of this size e.g. 8 MiB). The memory can be allocated e.g. by using hugetlbfs from user space [2][3]. The memory size for an enclave needs to be at least 64 MiB. The enclave memory and CPUs need to be from the same NUMA node. An enclave runs on dedicated cores. CPU 0 and its CPU siblings need to remain available for the primary VM. A CPU pool has to be set for NE purposes by an user with admin capability. See the cpu list section from the kernel documentation [4] for how a CPU pool format looks. An enclave communicates with the primary VM via a local communication channel, using virtio-vsock [5]. The primary VM has virtio-pci vsock emulated device, while the enclave VM has a virtio-mmio vsock emulated device. The vsock device uses eventfd for signaling. The enclave VM sees the usual interfaces - local APIC and IOAPIC - to get interrupts from virtio-vsock device. The virtio-mmio device is placed in memory below the typical 4 GiB. The application that runs in the enclave needs to be packaged in an enclave image together with the OS ( e.g. kernel, ramdisk, init ) that will run in the enclave VM. The enclave VM has its own kernel and follows the standard Linux boot protocol. The kernel bzImage, the kernel command line, the ramdisk(s) are part of the Enclave Image Format (EIF); plus an EIF header including metadata such as magic number, eif version, image size and CRC. Hash values are computed for the entire enclave image (EIF), the kernel and ramdisk(s). That's used, for example, to check that the enclave image that is loaded in the enclave VM is the one that was intended to be run. These crypto measurements are included in a signed attestation document generated by the Nitro Hypervisor and further used to prove the identity of the enclave; KMS is an example of service that NE is integrated with and that checks the attestation doc. The enclave image (EIF) is loaded in the enclave memory at offset 8 MiB. The init process in the enclave connects to the vsock CID of the primary VM and a predefined port - 9000 - to send a heartbeat value - 0xb7. This mechanism is used to check in the primary VM that the enclave has booted. If the enclave VM crashes or gracefully exits, an interrupt event is received by the NE driver. This event is sent further to the user space enclave process running in the primary VM via a poll notification mechanism. Then the user space enclave process can exit. Thank you. Andra [1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/ [2] https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt [3] https://lwn.net/Articles/807108/ [4] https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html [5] https://man7.org/linux/man-pages/man7/vsock.7.html --- Patch Series Changelog The patch series is built on top of v5.8-rc2. v3 -> v4 * Rebase on top of v5.8-rc2. * Add NE API version and the corresponding ioctl call. * Add enclave / image load flags options. * Decouple NE ioctl interface from KVM API. * Remove the "packed" attribute and include padding in the NE data structures. * Update documentation based on the changes from v4. * Update sample to match the updates in v4. * Remove the NE CPU pool init during NE kernel module loading. * Setup the NE CPU pool at runtime via a sysfs file for the kernel parameter. * Check if the enclave memory and CPUs are from the same NUMA node. * Add minimum enclave memory size definition. * v3: https://lore.kernel.org/lkml/20200525221334.62966-1-andraprs@amazon.com/ v2 -> v3 * Rebase on top of v5.7-rc7. * Add changelog to each patch in the series. * Remove "ratelimited" from the logs that are not in the ioctl call paths. * Update static calls sanity checks. * Remove file ops that do nothing for now. * Remove GPL additional wording as SPDX-License-Identifier is already in place. * v2: https://lore.kernel.org/lkml/20200522062946.28973-1-andraprs@amazon.com/ v1 -> v2 * Rebase on top of v5.7-rc6. * Adapt codebase based on feedback from v1. * Update ioctl number definition - major and minor. * Add sample / documentation for the ioctl interface basic flow usage. * Update cover letter to include more context on the NE overall. * Add fix for the enclave / vcpu fd creation error cleanup path. * Add fix reported by kbuild test robot . * v1: https://lore.kernel.org/lkml/20200421184150.68011-1-andraprs@amazon.com/ --- Andra Paraschiv (18): nitro_enclaves: Add ioctl interface definition nitro_enclaves: Define the PCI device interface nitro_enclaves: Define enclave info for internal bookkeeping nitro_enclaves: Init PCI device driver nitro_enclaves: Handle PCI device command requests nitro_enclaves: Handle out-of-band PCI device events nitro_enclaves: Init misc device providing the ioctl interface nitro_enclaves: Add logic for enclave vm creation nitro_enclaves: Add logic for enclave vcpu creation nitro_enclaves: Add logic for enclave image load info nitro_enclaves: Add logic for enclave memory region set nitro_enclaves: Add logic for enclave start nitro_enclaves: Add logic for enclave termination nitro_enclaves: Add Kconfig for the Nitro Enclaves driver nitro_enclaves: Add Makefile for the Nitro Enclaves driver nitro_enclaves: Add sample for ioctl interface usage nitro_enclaves: Add overview documentation MAINTAINERS: Add entry for the Nitro Enclaves driver Documentation/nitro_enclaves/ne_overview.rst | 87 ++ .../userspace-api/ioctl/ioctl-number.rst | 5 +- MAINTAINERS | 13 + drivers/virt/Kconfig | 2 + drivers/virt/Makefile | 2 + drivers/virt/nitro_enclaves/Kconfig | 16 + drivers/virt/nitro_enclaves/Makefile | 11 + drivers/virt/nitro_enclaves/ne_misc_dev.c | 1364 +++++++++++++++++ drivers/virt/nitro_enclaves/ne_misc_dev.h | 115 ++ drivers/virt/nitro_enclaves/ne_pci_dev.c | 626 ++++++++ drivers/virt/nitro_enclaves/ne_pci_dev.h | 264 ++++ include/linux/nitro_enclaves.h | 11 + include/uapi/linux/nitro_enclaves.h | 137 ++ samples/nitro_enclaves/.gitignore | 2 + samples/nitro_enclaves/Makefile | 16 + samples/nitro_enclaves/ne_ioctl_sample.c | 520 +++++++ 16 files changed, 3190 insertions(+), 1 deletion(-) create mode 100644 Documentation/nitro_enclaves/ne_overview.rst create mode 100644 drivers/virt/nitro_enclaves/Kconfig create mode 100644 drivers/virt/nitro_enclaves/Makefile create mode 100644 drivers/virt/nitro_enclaves/ne_misc_dev.c create mode 100644 drivers/virt/nitro_enclaves/ne_misc_dev.h create mode 100644 drivers/virt/nitro_enclaves/ne_pci_dev.c create mode 100644 drivers/virt/nitro_enclaves/ne_pci_dev.h create mode 100644 include/linux/nitro_enclaves.h create mode 100644 include/uapi/linux/nitro_enclaves.h create mode 100644 samples/nitro_enclaves/.gitignore create mode 100644 samples/nitro_enclaves/Makefile create mode 100644 samples/nitro_enclaves/ne_ioctl_sample.c -- 2.20.1 (Apple Git-117) Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.