[PATCH v2 0/7] nitro_enclaves: Add support for Arm64

[PATCH v2 0/7] nitro_enclaves: Add support for Arm64

[Andra Paraschiv]

Update the kernel config of the Nitro Enclaves kernel driver to enable Arm64
support. Add Arm64 specific references to its documentation.

While at it, fix a set of reports from checkpatch and kernel-doc scripts.

Thank you,
Andra

---

Patch Series Changelog

The patch series is built on top of v5.14-rc7.

GitHub repo branch for the latest version of the patch series:

* https://github.com/andraprs/linux/tree/ne-driver-arm-support-v2

v1 -> v2

* Add information about supported architectures for the NE kernel driver.
* Update comments for send / receive buffer sizes for the NE PCI device.
* Split patch 3 that includes fixes for the checkpatch and kernel-doc reports
  into multiple ones.
* v1: https://lore.kernel.org/lkml/20210826173451.93165-1-andraprs@amazon.com/

---

Andra Paraschiv (7):
  nitro_enclaves: Enable Arm64 support
  nitro_enclaves: Update documentation for Arm64 support
  nitro_enclaves: Add fix for the kernel-doc report
  nitro_enclaves: Update copyright statement to include 2021
  nitro_enclaves: Add fixes for checkpatch match open parenthesis
    reports
  nitro_enclaves: Add fixes for checkpatch spell check reports
  nitro_enclaves: Add fixes for checkpatch blank line reports

 Documentation/virt/ne_overview.rst        | 21 +++++++++++++--------
 drivers/virt/nitro_enclaves/Kconfig       |  8 ++------
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 17 +++++++++--------
 drivers/virt/nitro_enclaves/ne_pci_dev.c  |  2 +-
 drivers/virt/nitro_enclaves/ne_pci_dev.h  |  8 ++++++--
 include/uapi/linux/nitro_enclaves.h       | 10 +++++-----
 samples/nitro_enclaves/ne_ioctl_sample.c  |  7 +++----
 7 files changed, 39 insertions(+), 34 deletions(-)

-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 1/7] nitro_enclaves: Enable Arm64 support

[Andra Paraschiv]

Update the kernel config to enable the Nitro Enclaves kernel driver for
Arm64 support.

Changelog

v1 -> v2

* No changes.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
Acked-by: Stefano Garzarella <sgarzare@redhat.com>
---
 drivers/virt/nitro_enclaves/Kconfig | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/Kconfig b/drivers/virt/nitro_enclaves/Kconfig
index 8c9387a232df8..f53740b941c0f 100644
--- a/drivers/virt/nitro_enclaves/Kconfig
+++ b/drivers/virt/nitro_enclaves/Kconfig
@@ -1,17 +1,13 @@
 # SPDX-License-Identifier: GPL-2.0
 #
-# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 
 # Amazon Nitro Enclaves (NE) support.
 # Nitro is a hypervisor that has been developed by Amazon.
 
-# TODO: Add dependency for ARM64 once NE is supported on Arm platforms. For now,
-# the NE kernel driver can be built for aarch64 arch.
-# depends on (ARM64 || X86) && HOTPLUG_CPU && PCI && SMP
-
 config NITRO_ENCLAVES
 	tristate "Nitro Enclaves Support"
-	depends on X86 && HOTPLUG_CPU && PCI && SMP
+	depends on (ARM64 || X86) && HOTPLUG_CPU && PCI && SMP
 	help
 	  This driver consists of support for enclave lifetime management
 	  for Nitro Enclaves (NE).
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 2/7] nitro_enclaves: Update documentation for Arm64 support

[Andra Paraschiv]

Add references for hugepages and booting steps for Arm64.

Include info about the current supported architectures for the
NE kernel driver.

Changelog

v1 -> v2

* Add information about supported architectures for the NE kernel
driver.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 Documentation/virt/ne_overview.rst | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/Documentation/virt/ne_overview.rst b/Documentation/virt/ne_overview.rst
index 39b0c8fe2654a..74c2f5919c886 100644
--- a/Documentation/virt/ne_overview.rst
+++ b/Documentation/virt/ne_overview.rst
@@ -14,12 +14,15 @@ instances [1].
 For example, an application that processes sensitive data and runs in a VM,
 can be separated from other applications running in the same VM. This
 application then runs in a separate VM than the primary VM, namely an enclave.
+It runs alongside the VM that spawned it. This setup matches low latency
+applications needs.
 
-An enclave runs alongside the VM that spawned it. This setup matches low latency
-applications needs. The resources that are allocated for the enclave, such as
-memory and CPUs, are carved out of the primary VM. Each enclave is mapped to a
-process running in the primary VM, that communicates with the NE driver via an
-ioctl interface.
+The current supported architectures for the NE kernel driver, available in the
+upstream Linux kernel, are x86 and ARM64.
+
+The resources that are allocated for the enclave, such as memory and CPUs, are
+carved out of the primary VM. Each enclave is mapped to a process running in the
+primary VM, that communicates with the NE kernel driver via an ioctl interface.
 
 In this sense, there are two components:
 
@@ -43,8 +46,8 @@ for the enclave VM. An enclave does not have persistent storage attached.
 The memory regions carved out of the primary VM and given to an enclave need to
 be aligned 2 MiB / 1 GiB physically contiguous memory regions (or multiple of
 this size e.g. 8 MiB). The memory can be allocated e.g. by using hugetlbfs from
-user space [2][3]. The memory size for an enclave needs to be at least 64 MiB.
-The enclave memory and CPUs need to be from the same NUMA node.
+user space [2][3][7]. The memory size for an enclave needs to be at least
+64 MiB. The enclave memory and CPUs need to be from the same NUMA node.
 
 An enclave runs on dedicated cores. CPU 0 and its CPU siblings need to remain
 available for the primary VM. A CPU pool has to be set for NE purposes by an
@@ -61,7 +64,7 @@ device is placed in memory below the typical 4 GiB.
 The application that runs in the enclave needs to be packaged in an enclave
 image together with the OS ( e.g. kernel, ramdisk, init ) that will run in the
 enclave VM. The enclave VM has its own kernel and follows the standard Linux
-boot protocol [6].
+boot protocol [6][8].
 
 The kernel bzImage, the kernel command line, the ramdisk(s) are part of the
 Enclave Image Format (EIF); plus an EIF header including metadata such as magic
@@ -93,3 +96,5 @@ enclave process can exit.
 [4] https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html
 [5] https://man7.org/linux/man-pages/man7/vsock.7.html
 [6] https://www.kernel.org/doc/html/latest/x86/boot.html
+[7] https://www.kernel.org/doc/html/latest/arm64/hugetlbpage.html
+[8] https://www.kernel.org/doc/html/latest/arm64/booting.html
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 3/7] nitro_enclaves: Add fix for the kernel-doc report

[Andra Paraschiv]

Fix the reported issue from the kernel-doc script, to have a comment per
identifier.

Changelog

v1 -> v2

* Update comments for send / receive buffer sizes for the NE PCI device.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.h | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h
index 8bfbc66078185..6e9f28971a4e0 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.h
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 */
 /*
- * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  */
 
 #ifndef _NE_PCI_DEV_H_
@@ -84,9 +84,13 @@
  */
 
 /**
- * NE_SEND_DATA_SIZE / NE_RECV_DATA_SIZE - 240 bytes for send / recv buffer.
+ * NE_SEND_DATA_SIZE - Size of the send buffer, in bytes.
  */
 #define NE_SEND_DATA_SIZE	(240)
+
+/**
+ * NE_RECV_DATA_SIZE - Size of the receive buffer, in bytes.
+ */
 #define NE_RECV_DATA_SIZE	(240)
 
 /**
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 4/7] nitro_enclaves: Update copyright statement to include 2021

[Andra Paraschiv]

Update the copyright statement to include 2021, as a change has been
made over this year.

Check commit d874742f6a73 ("nitro_enclaves: Set Bus Master for the NE
PCI device") for the codebase update from this file (ne_pci_dev.c).

Changelog

v1 -> v2

* No codebase changes, it was split from the patch 3 in the v1 of the
patch series.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_pci_dev.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c
index 143207e9b9698..40b49ec8e30b1 100644
--- a/drivers/virt/nitro_enclaves/ne_pci_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  */
 
 /**
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 5/7] nitro_enclaves: Add fixes for checkpatch match open parenthesis reports

[Andra Paraschiv]

Update the codebase formatting to fix the reports from the checkpatch
script, to match the open parenthesis.

Changelog

v1 -> v2

* No codebase changes, it was split from the patch 3 in the v1 of the
patch series.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 drivers/virt/nitro_enclaves/ne_misc_dev.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index e21e1e86ad15f..8939612ee0e08 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  */
 
 /**
@@ -284,8 +284,8 @@ static int ne_setup_cpu_pool(const char *ne_cpu_list)
 	ne_cpu_pool.nr_parent_vm_cores = nr_cpu_ids / ne_cpu_pool.nr_threads_per_core;
 
 	ne_cpu_pool.avail_threads_per_core = kcalloc(ne_cpu_pool.nr_parent_vm_cores,
-					     sizeof(*ne_cpu_pool.avail_threads_per_core),
-					     GFP_KERNEL);
+						     sizeof(*ne_cpu_pool.avail_threads_per_core),
+						     GFP_KERNEL);
 	if (!ne_cpu_pool.avail_threads_per_core) {
 		rc = -ENOMEM;
 
@@ -735,7 +735,7 @@ static int ne_add_vcpu_ioctl(struct ne_enclave *ne_enclave, u32 vcpu_id)
  * * Negative return value on failure.
  */
 static int ne_sanity_check_user_mem_region(struct ne_enclave *ne_enclave,
-	struct ne_user_memory_region mem_region)
+					   struct ne_user_memory_region mem_region)
 {
 	struct ne_mem_region *ne_mem_region = NULL;
 
@@ -771,7 +771,7 @@ static int ne_sanity_check_user_mem_region(struct ne_enclave *ne_enclave,
 		u64 userspace_addr = ne_mem_region->userspace_addr;
 
 		if ((userspace_addr <= mem_region.userspace_addr &&
-		    mem_region.userspace_addr < (userspace_addr + memory_size)) ||
+		     mem_region.userspace_addr < (userspace_addr + memory_size)) ||
 		    (mem_region.userspace_addr <= userspace_addr &&
 		    (mem_region.userspace_addr + mem_region.memory_size) > userspace_addr)) {
 			dev_err_ratelimited(ne_misc_dev.this_device,
@@ -836,7 +836,7 @@ static int ne_sanity_check_user_mem_region_page(struct ne_enclave *ne_enclave,
  * * Negative return value on failure.
  */
 static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave,
-	struct ne_user_memory_region mem_region)
+					   struct ne_user_memory_region mem_region)
 {
 	long gup_rc = 0;
 	unsigned long i = 0;
@@ -1014,7 +1014,7 @@ static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave,
  * * Negative return value on failure.
  */
 static int ne_start_enclave_ioctl(struct ne_enclave *ne_enclave,
-	struct ne_enclave_start_info *enclave_start_info)
+				  struct ne_enclave_start_info *enclave_start_info)
 {
 	struct ne_pci_dev_cmd_reply cmd_reply = {};
 	unsigned int cpu = 0;
@@ -1574,7 +1574,8 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui
 	mutex_unlock(&ne_cpu_pool.mutex);
 
 	ne_enclave->threads_per_core = kcalloc(ne_enclave->nr_parent_vm_cores,
-		sizeof(*ne_enclave->threads_per_core), GFP_KERNEL);
+					       sizeof(*ne_enclave->threads_per_core),
+					       GFP_KERNEL);
 	if (!ne_enclave->threads_per_core) {
 		rc = -ENOMEM;
 
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 6/7] nitro_enclaves: Add fixes for checkpatch spell check reports

[Andra Paraschiv]

Fix the typos in the words spelling as per the checkpatch script
reports.

Changelog

v1 -> v2

* No codebase changes, it was split from the patch 3 in the v1 of the
patch series.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 include/uapi/linux/nitro_enclaves.h      | 10 +++++-----
 samples/nitro_enclaves/ne_ioctl_sample.c |  4 ++--
 2 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
index b945073fe544d..e808f5ba124d4 100644
--- a/include/uapi/linux/nitro_enclaves.h
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -1,6 +1,6 @@
 /* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
 /*
- * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  */
 
 #ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
@@ -60,7 +60,7 @@
  *
  * Context: Process context.
  * Return:
- * * 0					- Logic succesfully completed.
+ * * 0					- Logic successfully completed.
  * *  -1				- There was a failure in the ioctl logic.
  * On failure, errno is set to:
  * * EFAULT				- copy_from_user() / copy_to_user() failure.
@@ -95,7 +95,7 @@
  *
  * Context: Process context.
  * Return:
- * * 0				- Logic succesfully completed.
+ * * 0				- Logic successfully completed.
  * *  -1			- There was a failure in the ioctl logic.
  * On failure, errno is set to:
  * * EFAULT			- copy_from_user() / copy_to_user() failure.
@@ -118,7 +118,7 @@
  *
  * Context: Process context.
  * Return:
- * * 0					- Logic succesfully completed.
+ * * 0					- Logic successfully completed.
  * *  -1				- There was a failure in the ioctl logic.
  * On failure, errno is set to:
  * * EFAULT				- copy_from_user() failure.
@@ -161,7 +161,7 @@
  *
  * Context: Process context.
  * Return:
- * * 0					- Logic succesfully completed.
+ * * 0					- Logic successfully completed.
  * *  -1				- There was a failure in the ioctl logic.
  * On failure, errno is set to:
  * * EFAULT				- copy_from_user() / copy_to_user() failure.
diff --git a/samples/nitro_enclaves/ne_ioctl_sample.c b/samples/nitro_enclaves/ne_ioctl_sample.c
index 480b763142b34..6a60990b2e202 100644
--- a/samples/nitro_enclaves/ne_ioctl_sample.c
+++ b/samples/nitro_enclaves/ne_ioctl_sample.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0
 /*
- * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ * Copyright 2020-2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  */
 
 /**
@@ -638,7 +638,7 @@ static int ne_start_enclave(int enclave_fd,  struct ne_enclave_start_info *encla
 }
 
 /**
- * ne_start_enclave_check_booted() - Start the enclave and wait for a hearbeat
+ * ne_start_enclave_check_booted() - Start the enclave and wait for a heartbeat
  *				     from it, on a newly created vsock channel,
  *				     to check it has booted.
  * @enclave_fd :	The file descriptor associated with the enclave.
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[PATCH v2 7/7] nitro_enclaves: Add fixes for checkpatch blank line reports

[Andra Paraschiv]

Remove blank lines that are not necessary, fixing the checkpatch script
reports. While at it, add a blank line after the switch default block,
similar to the other parts of the codebase.

Changelog

v1 -> v2

* No codebase changes, it was split from the patch 3 in the v1 of the
patch series.

Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
 samples/nitro_enclaves/ne_ioctl_sample.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/samples/nitro_enclaves/ne_ioctl_sample.c b/samples/nitro_enclaves/ne_ioctl_sample.c
index 6a60990b2e202..765b131c73190 100644
--- a/samples/nitro_enclaves/ne_ioctl_sample.c
+++ b/samples/nitro_enclaves/ne_ioctl_sample.c
@@ -185,7 +185,6 @@ static int ne_create_vm(int ne_dev_fd, unsigned long *slot_uid, int *enclave_fd)
 	return 0;
 }
 
-
 /**
  * ne_poll_enclave_fd() - Thread function for polling the enclave fd.
  * @data:	Argument provided for the polling function.
@@ -560,8 +559,8 @@ static int ne_add_vcpu(int enclave_fd, unsigned int *vcpu_id)
 
 		default:
 			printf("Error in add vcpu [%m]\n");
-
 		}
+
 		return rc;
 	}
 
-- 
2.20.1 (Apple Git-117)




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

Re: [PATCH v2 1/7] nitro_enclaves: Enable Arm64 support

[Greg KH]

On Fri, Aug 27, 2021 at 04:32:24PM +0300, Andra Paraschiv wrote:
> Update the kernel config to enable the Nitro Enclaves kernel driver for
> Arm64 support.
> 
> Changelog
> 
> v1 -> v2
> 
> * No changes.
> 

changelogs for different all go below the --- line, as is documented.
No need for them here in the changelog text itself, right?

Please fix up and send a v3 series.

thanks,

greg k-h

Re: [PATCH v2 1/7] nitro_enclaves: Enable Arm64 support

[Paraschiv, Andra-Irina]

On 27/08/2021 17:25, Greg KH wrote:
> On Fri, Aug 27, 2021 at 04:32:24PM +0300, Andra Paraschiv wrote:
>> Update the kernel config to enable the Nitro Enclaves kernel driver for
>> Arm64 support.
>>
>> Changelog
>>
>> v1 -> v2
>>
>> * No changes.
>>
> changelogs for different all go below the --- line, as is documented.
> No need for them here in the changelog text itself, right?
>
> Please fix up and send a v3 series.

Alright, I can modify the patches so that the changelog is after the line.

I followed the same pattern as the initial time, when I received 
feedback to have the changelogs in the commit message, before SoB(s).

But that's fine with me, I can switch to this way of doing it, as 
mentioned also in the docs.

Thanks,
Andra



Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

Re: [PATCH v2 1/7] nitro_enclaves: Enable Arm64 support

[Greg KH]

On Fri, Aug 27, 2021 at 06:02:57PM +0300, Paraschiv, Andra-Irina wrote:
> 
> 
> On 27/08/2021 17:25, Greg KH wrote:
> > On Fri, Aug 27, 2021 at 04:32:24PM +0300, Andra Paraschiv wrote:
> > > Update the kernel config to enable the Nitro Enclaves kernel driver for
> > > Arm64 support.
> > > 
> > > Changelog
> > > 
> > > v1 -> v2
> > > 
> > > * No changes.
> > > 
> > changelogs for different all go below the --- line, as is documented.
> > No need for them here in the changelog text itself, right?
> > 
> > Please fix up and send a v3 series.
> 
> Alright, I can modify the patches so that the changelog is after the line.
> 
> I followed the same pattern as the initial time, when I received feedback to
> have the changelogs in the commit message, before SoB(s).

Only the crazy drm developers seem to use that format :)

> But that's fine with me, I can switch to this way of doing it, as mentioned
> also in the docs.

Thank you.

greg k-h