[kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

Hello,

This patch series adds initial support for building arm64 tests as EFI
tests and running them under QEMU. Much like x86_64 we import external
dependencies from gnu-efi and adapt them to work with types and other
assumptions from kvm-unit-tests. In addition, this series adds support
for enumerating parts of the system using ACPI.

The first set of patches moves the existing ACPI code to the common
lib path. Then, it extends definitions and functions to allow for more
robust discovery of ACPI tables. In arm64, we add support for setting
up the PSCI conduit, discovering the UART, timers and cpus via
ACPI. The code retains existing behavior and gives priority to
discovery through DT when one has been provided.

In the second set of patches, we add support for getting the command
line from the EFI shell. This is a requirement for many of the
existing arm64 tests.

In the third set of patches, we import code from gnu-efi, make minor
changes and add an alternative setup sequence from arm64 systems that
boot through EFI. Finally, we add support in the build system and a
run script which is used to run an EFI app.

After this set of patches one can build arm64 EFI tests:

$> ./configure --enable-efi
$> make

And use the run script to run an EFI tests:

$> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"

Or all tests:

$> ./run_tests.sh

There are a few items that this series does not address but they would
be useful to have:
 - Support for booting the system from EL2. Currently, we assume that a
   tests starts running at EL1. This the case when we run with EFI, it's
   not always the case in hardware.
 - Support for reading environment variables and populating __envp.
 - Support for discovering the PCI subsystem using ACPI.
 - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.

git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased

v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/

Changes in v3:
 - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
 - Added support for discovering the GIC through ACPI
 - Added a missing header file (<elf.h>)
 - Added support for correctly parsing the outcome of tests (./run_tests)

Thanks,

Nikos

Alexandru Elisei (1):
  lib: arm: Print test exit status

Andrew Jones (3):
  arm/arm64: mmu_disable: Clean and invalidate before disabling
  arm/arm64: Rename etext to _etext
  arm64: Add a new type of memory type flag MR_F_RESERVED

Nikos Nikoleris (23):
  lib: Fix style for acpi.{c,h}
  x86: Avoid references to fields of ACPI tables
  lib: Ensure all struct definition for ACPI tables are packed
  lib: Add support for the XSDT ACPI table
  lib: Extend the definition of the ACPI table FADT
  devicetree: Check if fdt is NULL before returning that a DT is
    available
  arm/arm64: Add support for setting up the PSCI conduit through ACPI
  arm/arm64: Add support for discovering the UART through ACPI
  arm/arm64: Add support for timer initialization through ACPI
  arm/arm64: Add support for cpu initialization through ACPI
  arm/arm64: Add support for gic initialization through ACPI
  lib/printf: Support for precision modifier in printf
  lib/printf: Add support for printing wide strings
  lib/efi: Add support for getting the cmdline
  lib: Avoid ms_abi for calls related to EFI on arm64
  arm/arm64: Add a setup sequence for systems that boot through EFI
  arm64: Copy code from GNU-EFI
  arm64: Change GNU-EFI imported file to use defined types
  arm64: Use code from the gnu-efi when booting with EFI
  lib: Avoid external dependency in libelf
  x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
  arm64: Add support for efi in Makefile
  arm64: Add an efi/run script

 scripts/runtime.bash        |  14 +-
 arm/efi/run                 |  61 +++++++
 arm/run                     |  14 +-
 configure                   |  15 +-
 Makefile                    |   4 -
 arm/Makefile.arm            |   6 +
 arm/Makefile.arm64          |  18 +-
 arm/Makefile.common         |  48 +++--
 x86/Makefile.x86_64         |   4 +
 lib/linux/efi.h             |  25 +++
 lib/arm/asm/setup.h         |   3 +
 lib/arm/asm/timer.h         |   2 +
 lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
 lib/argv.h                  |   1 +
 lib/elf.h                   |  57 ++++++
 lib/libcflat.h              |   1 +
 lib/acpi.c                  | 129 ++++++++-----
 lib/argv.c                  |   2 +-
 lib/arm/gic.c               | 127 ++++++++++++-
 lib/arm/io.c                |  29 ++-
 lib/arm/mmu.c               |   4 +
 lib/arm/psci.c              |  25 ++-
 lib/arm/setup.c             | 247 ++++++++++++++++++++-----
 lib/arm/timer.c             |  79 ++++++++
 lib/devicetree.c            |   2 +-
 lib/efi.c                   | 102 +++++++++++
 lib/printf.c                | 194 ++++++++++++++++++--
 arm/efi/elf_aarch64_efi.lds |  63 +++++++
 arm/flat.lds                |   2 +-
 arm/cstart.S                |  29 ++-
 arm/cstart64.S              |  28 ++-
 arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
 arm/dummy.c                 |   4 +
 arm/efi/reloc_aarch64.c     |  93 ++++++++++
 arm/micro-bench.c           |   4 +-
 arm/timer.c                 |  10 +-
 x86/s3.c                    |  19 +-
 x86/vmexit.c                |   2 +-
 38 files changed, 1700 insertions(+), 258 deletions(-)
 create mode 100755 arm/efi/run
 create mode 100644 lib/elf.h
 create mode 100644 lib/arm/timer.c
 create mode 100644 arm/efi/elf_aarch64_efi.lds
 create mode 100644 arm/efi/crt0-efi-aarch64.S
 create mode 100644 arm/dummy.c
 create mode 100644 arm/efi/reloc_aarch64.c

-- 
2.25.1

[kvm-unit-tests PATCH v3 01/27] lib: Fix style for acpi.{c,h}

[Nikos Nikoleris]

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/acpi.h | 148 ++++++++++++++++++++++++++---------------------------
 lib/acpi.c |  70 ++++++++++++-------------
 2 files changed, 108 insertions(+), 110 deletions(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index 1e89840..456e62d 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -3,7 +3,7 @@
 
 #include "libcflat.h"
 
-#define ACPI_SIGNATURE(c1, c2, c3, c4) \
+#define ACPI_SIGNATURE(c1, c2, c3, c4)				\
 	((c1) | ((c2) << 8) | ((c3) << 16) | ((c4) << 24))
 
 #define RSDP_SIGNATURE ACPI_SIGNATURE('R','S','D','P')
@@ -12,98 +12,98 @@
 #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
 
 
-#define ACPI_SIGNATURE_8BYTE(c1, c2, c3, c4, c5, c6, c7, c8) \
-	((uint64_t)(ACPI_SIGNATURE(c1, c2, c3, c4))) |       \
+#define ACPI_SIGNATURE_8BYTE(c1, c2, c3, c4, c5, c6, c7, c8)	\
+	((uint64_t)(ACPI_SIGNATURE(c1, c2, c3, c4))) |		\
 	((uint64_t)(ACPI_SIGNATURE(c5, c6, c7, c8)) << 32)
 
 #define RSDP_SIGNATURE_8BYTE (ACPI_SIGNATURE_8BYTE('R', 'S', 'D', ' ', 'P', 'T', 'R', ' '))
 
-struct rsdp_descriptor {        /* Root System Descriptor Pointer */
-    u64 signature;              /* ACPI signature, contains "RSD PTR " */
-    u8  checksum;               /* To make sum of struct == 0 */
-    u8  oem_id [6];             /* OEM identification */
-    u8  revision;               /* Must be 0 for 1.0, 2 for 2.0 */
-    u32 rsdt_physical_address;  /* 32-bit physical address of RSDT */
-    u32 length;                 /* XSDT Length in bytes including hdr */
-    u64 xsdt_physical_address;  /* 64-bit physical address of XSDT */
-    u8  extended_checksum;      /* Checksum of entire table */
-    u8  reserved [3];           /* Reserved field must be 0 */
+struct rsdp_descriptor {		/* Root System Descriptor Pointer */
+	u64 signature;			/* ACPI signature, contains "RSD PTR " */
+	u8  checksum;			/* To make sum of struct == 0 */
+	u8  oem_id [6];			/* OEM identification */
+	u8  revision;			/* Must be 0 for 1.0, 2 for 2.0 */
+	u32 rsdt_physical_address;	/* 32-bit physical address of RSDT */
+	u32 length;			/* XSDT Length in bytes including hdr */
+	u64 xsdt_physical_address;	/* 64-bit physical address of XSDT */
+	u8  extended_checksum;		/* Checksum of entire table */
+	u8  reserved [3];		/* Reserved field must be 0 */
 };
 
-#define ACPI_TABLE_HEADER_DEF   /* ACPI common table header */ \
-    u32 signature;          /* ACPI signature (4 ASCII characters) */ \
-    u32 length;                 /* Length of table, in bytes, including header */ \
-    u8  revision;               /* ACPI Specification minor version # */ \
-    u8  checksum;               /* To make sum of entire table == 0 */ \
-    u8  oem_id [6];             /* OEM identification */ \
-    u8  oem_table_id [8];       /* OEM table identification */ \
-    u32 oem_revision;           /* OEM revision number */ \
-    u8  asl_compiler_id [4];    /* ASL compiler vendor ID */ \
-    u32 asl_compiler_revision;  /* ASL compiler revision number */
+#define ACPI_TABLE_HEADER_DEF		/* ACPI common table header */			\
+	u32 signature;			/* ACPI signature (4 ASCII characters) */	\
+	u32 length;			/* Length of table, in bytes, including header */ \
+	u8  revision;			/* ACPI Specification minor version # */	\
+	u8  checksum;			/* To make sum of entire table == 0 */		\
+	u8  oem_id [6];			/* OEM identification */			\
+	u8  oem_table_id [8];		/* OEM table identification */			\
+	u32 oem_revision;		/* OEM revision number */			\
+	u8  asl_compiler_id [4];	/* ASL compiler vendor ID */			\
+	u32 asl_compiler_revision;	/* ASL compiler revision number */
 
 struct acpi_table {
-    ACPI_TABLE_HEADER_DEF
-    char data[0];
+	ACPI_TABLE_HEADER_DEF
+	char data[0];
 };
 
 struct rsdt_descriptor_rev1 {
-    ACPI_TABLE_HEADER_DEF
-    u32 table_offset_entry[0];
+	ACPI_TABLE_HEADER_DEF
+	u32 table_offset_entry[1];
 };
 
 struct fadt_descriptor_rev1
 {
-    ACPI_TABLE_HEADER_DEF     /* ACPI common table header */
-    u32 firmware_ctrl;          /* Physical address of FACS */
-    u32 dsdt;                   /* Physical address of DSDT */
-    u8  model;                  /* System Interrupt Model */
-    u8  reserved1;              /* Reserved */
-    u16 sci_int;                /* System vector of SCI interrupt */
-    u32 smi_cmd;                /* Port address of SMI command port */
-    u8  acpi_enable;            /* Value to write to smi_cmd to enable ACPI */
-    u8  acpi_disable;           /* Value to write to smi_cmd to disable ACPI */
-    u8  S4bios_req;             /* Value to write to SMI CMD to enter S4BIOS state */
-    u8  reserved2;              /* Reserved - must be zero */
-    u32 pm1a_evt_blk;           /* Port address of Power Mgt 1a acpi_event Reg Blk */
-    u32 pm1b_evt_blk;           /* Port address of Power Mgt 1b acpi_event Reg Blk */
-    u32 pm1a_cnt_blk;           /* Port address of Power Mgt 1a Control Reg Blk */
-    u32 pm1b_cnt_blk;           /* Port address of Power Mgt 1b Control Reg Blk */
-    u32 pm2_cnt_blk;            /* Port address of Power Mgt 2 Control Reg Blk */
-    u32 pm_tmr_blk;             /* Port address of Power Mgt Timer Ctrl Reg Blk */
-    u32 gpe0_blk;               /* Port addr of General Purpose acpi_event 0 Reg Blk */
-    u32 gpe1_blk;               /* Port addr of General Purpose acpi_event 1 Reg Blk */
-    u8  pm1_evt_len;            /* Byte length of ports at pm1_x_evt_blk */
-    u8  pm1_cnt_len;            /* Byte length of ports at pm1_x_cnt_blk */
-    u8  pm2_cnt_len;            /* Byte Length of ports at pm2_cnt_blk */
-    u8  pm_tmr_len;             /* Byte Length of ports at pm_tm_blk */
-    u8  gpe0_blk_len;           /* Byte Length of ports at gpe0_blk */
-    u8  gpe1_blk_len;           /* Byte Length of ports at gpe1_blk */
-    u8  gpe1_base;              /* Offset in gpe model where gpe1 events start */
-    u8  reserved3;              /* Reserved */
-    u16 plvl2_lat;              /* Worst case HW latency to enter/exit C2 state */
-    u16 plvl3_lat;              /* Worst case HW latency to enter/exit C3 state */
-    u16 flush_size;             /* Size of area read to flush caches */
-    u16 flush_stride;           /* Stride used in flushing caches */
-    u8  duty_offset;            /* Bit location of duty cycle field in p_cnt reg */
-    u8  duty_width;             /* Bit width of duty cycle field in p_cnt reg */
-    u8  day_alrm;               /* Index to day-of-month alarm in RTC CMOS RAM */
-    u8  mon_alrm;               /* Index to month-of-year alarm in RTC CMOS RAM */
-    u8  century;                /* Index to century in RTC CMOS RAM */
-    u8  reserved4;              /* Reserved */
-    u8  reserved4a;             /* Reserved */
-    u8  reserved4b;             /* Reserved */
+	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
+	u32 firmware_ctrl;	/* Physical address of FACS */
+	u32 dsdt;		/* Physical address of DSDT */
+	u8  model;		/* System Interrupt Model */
+	u8  reserved1;		/* Reserved */
+	u16 sci_int;		/* System vector of SCI interrupt */
+	u32 smi_cmd;		/* Port address of SMI command port */
+	u8  acpi_enable;	/* Value to write to smi_cmd to enable ACPI */
+	u8  acpi_disable;	/* Value to write to smi_cmd to disable ACPI */
+	u8  S4bios_req;		/* Value to write to SMI CMD to enter S4BIOS state */
+	u8  reserved2;		/* Reserved - must be zero */
+	u32 pm1a_evt_blk;	/* Port address of Power Mgt 1a acpi_event Reg Blk */
+	u32 pm1b_evt_blk;	/* Port address of Power Mgt 1b acpi_event Reg Blk */
+	u32 pm1a_cnt_blk;	/* Port address of Power Mgt 1a Control Reg Blk */
+	u32 pm1b_cnt_blk;	/* Port address of Power Mgt 1b Control Reg Blk */
+	u32 pm2_cnt_blk;	/* Port address of Power Mgt 2 Control Reg Blk */
+	u32 pm_tmr_blk;		/* Port address of Power Mgt Timer Ctrl Reg Blk */
+	u32 gpe0_blk;		/* Port addr of General Purpose acpi_event 0 Reg Blk */
+	u32 gpe1_blk;		/* Port addr of General Purpose acpi_event 1 Reg Blk */
+	u8  pm1_evt_len;	/* Byte length of ports at pm1_x_evt_blk */
+	u8  pm1_cnt_len;	/* Byte length of ports at pm1_x_cnt_blk */
+	u8  pm2_cnt_len;	/* Byte Length of ports at pm2_cnt_blk */
+	u8  pm_tmr_len;		/* Byte Length of ports at pm_tm_blk */
+	u8  gpe0_blk_len;	/* Byte Length of ports at gpe0_blk */
+	u8  gpe1_blk_len;	/* Byte Length of ports at gpe1_blk */
+	u8  gpe1_base;		/* Offset in gpe model where gpe1 events start */
+	u8  reserved3;		/* Reserved */
+	u16 plvl2_lat;		/* Worst case HW latency to enter/exit C2 state */
+	u16 plvl3_lat;		/* Worst case HW latency to enter/exit C3 state */
+	u16 flush_size;		/* Size of area read to flush caches */
+	u16 flush_stride;	/* Stride used in flushing caches */
+	u8  duty_offset;	/* Bit location of duty cycle field in p_cnt reg */
+	u8  duty_width;		/* Bit width of duty cycle field in p_cnt reg */
+	u8  day_alrm;		/* Index to day-of-month alarm in RTC CMOS RAM */
+	u8  mon_alrm;		/* Index to month-of-year alarm in RTC CMOS RAM */
+	u8  century;		/* Index to century in RTC CMOS RAM */
+	u8  reserved4;		/* Reserved */
+	u8  reserved4a;		/* Reserved */
+	u8  reserved4b;		/* Reserved */
 };
 
 struct facs_descriptor_rev1
 {
-    u32 signature;           /* ACPI Signature */
-    u32 length;                 /* Length of structure, in bytes */
-    u32 hardware_signature;     /* Hardware configuration signature */
-    u32 firmware_waking_vector; /* ACPI OS waking vector */
-    u32 global_lock;            /* Global Lock */
-    u32 S4bios_f        : 1;    /* Indicates if S4BIOS support is present */
-    u32 reserved1       : 31;   /* Must be 0 */
-    u8  reserved3 [40];         /* Reserved - must be zero */
+	u32 signature;			/* ACPI Signature */
+	u32 length;			/* Length of structure, in bytes */
+	u32 hardware_signature;		/* Hardware configuration signature */
+	u32 firmware_waking_vector;	/* ACPI OS waking vector */
+	u32 global_lock;		/* Global Lock */
+	u32 S4bios_f	: 1;		/* Indicates if S4BIOS support is present */
+	u32 reserved1	: 31;		/* Must be 0 */
+	u8  reserved3 [40];		/* Reserved - must be zero */
 };
 
 void set_efi_rsdp(struct rsdp_descriptor *rsdp);
diff --git a/lib/acpi.c b/lib/acpi.c
index de275ca..b7fd923 100644
--- a/lib/acpi.c
+++ b/lib/acpi.c
@@ -36,47 +36,45 @@ static struct rsdp_descriptor *get_rsdp(void)
 }
 #endif /* CONFIG_EFI */
 
-void* find_acpi_table_addr(u32 sig)
+void *find_acpi_table_addr(u32 sig)
 {
-    struct rsdp_descriptor *rsdp;
-    struct rsdt_descriptor_rev1 *rsdt;
-    void *end;
-    int i;
+	struct rsdp_descriptor *rsdp;
+	struct rsdt_descriptor_rev1 *rsdt;
+	void *end;
+	int i;
 
-    /* FACS is special... */
-    if (sig == FACS_SIGNATURE) {
-        struct fadt_descriptor_rev1 *fadt;
-        fadt = find_acpi_table_addr(FACP_SIGNATURE);
-        if (!fadt) {
-            return NULL;
-        }
-        return (void*)(ulong)fadt->firmware_ctrl;
-    }
+	/* FACS is special... */
+	if (sig == FACS_SIGNATURE) {
+		struct fadt_descriptor_rev1 *fadt;
+		fadt = find_acpi_table_addr(FACP_SIGNATURE);
+		if (!fadt)
+			return NULL;
 
-    rsdp = get_rsdp();
-    if (rsdp == NULL) {
-        printf("Can't find RSDP\n");
-        return 0;
-    }
+		return (void *)(ulong)fadt->firmware_ctrl;
+	}
 
-    if (sig == RSDP_SIGNATURE) {
-        return rsdp;
-    }
+	rsdp = get_rsdp();
+	if (rsdp == NULL) {
+		printf("Can't find RSDP\n");
+		return NULL;
+	}
 
-    rsdt = (void*)(ulong)rsdp->rsdt_physical_address;
-    if (!rsdt || rsdt->signature != RSDT_SIGNATURE)
-        return 0;
+	if (sig == RSDP_SIGNATURE)
+		return rsdp;
 
-    if (sig == RSDT_SIGNATURE) {
-        return rsdt;
-    }
+	rsdt = (void *)(ulong)rsdp->rsdt_physical_address;
+	if (!rsdt || rsdt->signature != RSDT_SIGNATURE)
+		return NULL;
+
+	if (sig == RSDT_SIGNATURE)
+		return rsdt;
 
-    end = (void*)rsdt + rsdt->length;
-    for (i=0; (void*)&rsdt->table_offset_entry[i] < end; i++) {
-        struct acpi_table *t = (void*)(ulong)rsdt->table_offset_entry[i];
-        if (t && t->signature == sig) {
-            return t;
-        }
-    }
-   return NULL;
+	end = (void *)rsdt + rsdt->length;
+	for (i = 0; (void *)&rsdt->table_offset_entry[i] < end; i++) {
+		struct acpi_table *t = (void *)(ulong)rsdt->table_offset_entry[i];
+		if (t && t->signature == sig) {
+			return t;
+		}
+	}
+	return NULL;
 }
-- 
2.25.1

[kvm-unit-tests PATCH v3 02/27] x86: Avoid references to fields of ACPI tables

[Nikos Nikoleris]

ACPI table definitions in <acpi.h> have to be packed.  However, once
we do that, direct references to members of the packed struct might
result in unaligned pointers and gcc complains about them. This change
modifies the code to avoid such references in preparation for making
the APCI table definitions packed.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 x86/s3.c | 17 +++++------------
 1 file changed, 5 insertions(+), 12 deletions(-)

diff --git a/x86/s3.c b/x86/s3.c
index 378d37a..0bcbe18 100644
--- a/x86/s3.c
+++ b/x86/s3.c
@@ -2,15 +2,6 @@
 #include "acpi.h"
 #include "asm/io.h"
 
-static u32* find_resume_vector_addr(void)
-{
-    struct facs_descriptor_rev1 *facs = find_acpi_table_addr(FACS_SIGNATURE);
-    if (!facs)
-        return 0;
-    printf("FACS is at %p\n", facs);
-    return &facs->firmware_waking_vector;
-}
-
 #define RTC_SECONDS_ALARM       1
 #define RTC_MINUTES_ALARM       3
 #define RTC_HOURS_ALARM         5
@@ -40,12 +31,14 @@ extern char resume_start, resume_end;
 int main(int argc, char **argv)
 {
 	struct fadt_descriptor_rev1 *fadt = find_acpi_table_addr(FACP_SIGNATURE);
-	volatile u32 *resume_vector_ptr = find_resume_vector_addr();
+	struct facs_descriptor_rev1 *facs = find_acpi_table_addr(FACS_SIGNATURE);
 	char *addr, *resume_vec = (void*)0x1000;
 
-	*resume_vector_ptr = (u32)(ulong)resume_vec;
+	assert(facs);
+	facs->firmware_waking_vector = (u32)(ulong)resume_vec;
 
-	printf("resume vector addr is %p\n", resume_vector_ptr);
+	printf("FACS is at %p\n", facs);
+	printf("resume vector addr is %p\n", &facs->firmware_waking_vector);
 	for (addr = &resume_start; addr < &resume_end; addr++)
 		*resume_vec++ = *addr;
 	printf("copy resume code from %p\n", &resume_start);
-- 
2.25.1

[kvm-unit-tests PATCH v3 03/27] lib: Ensure all struct definition for ACPI tables are packed

[Nikos Nikoleris]

All ACPI table definitions are provided with precise definitions of
field sizes and offsets, make sure that no compiler optimization can
interfere with the memory layout of the corresponding structs.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/acpi.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/lib/acpi.h b/lib/acpi.h
index 456e62d..b853a55 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -3,6 +3,12 @@
 
 #include "libcflat.h"
 
+/*
+ * All tables and structures must be byte-packed to match the ACPI
+ * specification, since the tables are provided by the system BIOS
+ */
+#pragma pack(1)
+
 #define ACPI_SIGNATURE(c1, c2, c3, c4)				\
 	((c1) | ((c2) << 8) | ((c3) << 16) | ((c4) << 24))
 
@@ -106,6 +112,8 @@ struct facs_descriptor_rev1
 	u8  reserved3 [40];		/* Reserved - must be zero */
 };
 
+#pragma pack(0)
+
 void set_efi_rsdp(struct rsdp_descriptor *rsdp);
 void* find_acpi_table_addr(u32 sig);
 
-- 
2.25.1

[kvm-unit-tests PATCH v3 04/27] lib: Add support for the XSDT ACPI table

[Nikos Nikoleris]

XSDT provides pointers to other ACPI tables much like RSDT. However,
contrary to RSDT that provides 32-bit addresses, XSDT provides 64-bit
pointers. ACPI requires that if XSDT is valid then it takes precedence
over RSDT.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/acpi.h |  6 ++++++
 lib/acpi.c | 37 +++++++++++++++++++++++++++++++------
 2 files changed, 37 insertions(+), 6 deletions(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index b853a55..c5f0aa5 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -14,6 +14,7 @@
 
 #define RSDP_SIGNATURE ACPI_SIGNATURE('R','S','D','P')
 #define RSDT_SIGNATURE ACPI_SIGNATURE('R','S','D','T')
+#define XSDT_SIGNATURE ACPI_SIGNATURE('X','S','D','T')
 #define FACP_SIGNATURE ACPI_SIGNATURE('F','A','C','P')
 #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
 
@@ -57,6 +58,11 @@ struct rsdt_descriptor_rev1 {
 	u32 table_offset_entry[1];
 };
 
+struct acpi_table_xsdt {
+	ACPI_TABLE_HEADER_DEF
+	u64 table_offset_entry[1];
+};
+
 struct fadt_descriptor_rev1
 {
 	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
diff --git a/lib/acpi.c b/lib/acpi.c
index b7fd923..240a922 100644
--- a/lib/acpi.c
+++ b/lib/acpi.c
@@ -40,6 +40,7 @@ void *find_acpi_table_addr(u32 sig)
 {
 	struct rsdp_descriptor *rsdp;
 	struct rsdt_descriptor_rev1 *rsdt;
+	struct acpi_table_xsdt *xsdt = NULL;
 	void *end;
 	int i;
 
@@ -64,17 +65,41 @@ void *find_acpi_table_addr(u32 sig)
 
 	rsdt = (void *)(ulong)rsdp->rsdt_physical_address;
 	if (!rsdt || rsdt->signature != RSDT_SIGNATURE)
-		return NULL;
+		rsdt = NULL;
 
 	if (sig == RSDT_SIGNATURE)
 		return rsdt;
 
-	end = (void *)rsdt + rsdt->length;
-	for (i = 0; (void *)&rsdt->table_offset_entry[i] < end; i++) {
-		struct acpi_table *t = (void *)(ulong)rsdt->table_offset_entry[i];
-		if (t && t->signature == sig) {
-			return t;
+	/*
+	 * When the system implements APCI 2.0 and above and XSDT is
+	 * valid we have use XSDT to find other ACPI tables,
+	 * otherwise, we use RSDT.
+	 */
+	if (rsdp->revision == 2)
+		xsdt = (void *)(ulong)rsdp->xsdt_physical_address;
+	if (!xsdt || xsdt->signature != XSDT_SIGNATURE)
+		xsdt = NULL;
+
+	if (sig == XSDT_SIGNATURE)
+		return xsdt;
+
+	if (xsdt) {
+		end = (void *)(ulong)xsdt + xsdt->length;
+		for (i = 0; (void *)&xsdt->table_offset_entry[i] < end; i++) {
+			struct acpi_table *t = (void *)xsdt->table_offset_entry[i];
+
+			if (t && t->signature == sig)
+				return t;
+		}
+	} else if (rsdt) {
+		end = (void *)rsdt + rsdt->length;
+		for (i = 0; (void *)&rsdt->table_offset_entry[i] < end; i++) {
+			struct acpi_table *t = (void *)(ulong)rsdt->table_offset_entry[i];
+
+			if (t && t->signature == sig)
+				return t;
 		}
 	}
+
 	return NULL;
 }
-- 
2.25.1

[kvm-unit-tests PATCH v3 05/27] lib: Extend the definition of the ACPI table FADT

[Nikos Nikoleris]

This change add more fields in the APCI table FADT to allow for the
discovery of the PSCI conduit in arm64 systems. The definition for
FADT is similar to the one in include/acpi/actbl.h in Linux.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 lib/acpi.h   | 33 +++++++++++++++++++++++++++++----
 lib/acpi.c   |  3 ++-
 x86/s3.c     |  2 +-
 x86/vmexit.c |  2 +-
 4 files changed, 33 insertions(+), 7 deletions(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index c5f0aa5..e49647b 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -63,7 +63,15 @@ struct acpi_table_xsdt {
 	u64 table_offset_entry[1];
 };
 
-struct fadt_descriptor_rev1
+struct acpi_generic_address {
+	u8 space_id;            /* Address space where struct or register exists */
+	u8 bit_width;           /* Size in bits of given register */
+	u8 bit_offset;          /* Bit offset within the register */
+	u8 access_width;        /* Minimum Access size (ACPI 3.0) */
+	u64 address;            /* 64-bit address of struct or register */
+};
+
+struct acpi_table_fadt
 {
 	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
 	u32 firmware_ctrl;	/* Physical address of FACS */
@@ -101,9 +109,26 @@ struct fadt_descriptor_rev1
 	u8  day_alrm;		/* Index to day-of-month alarm in RTC CMOS RAM */
 	u8  mon_alrm;		/* Index to month-of-year alarm in RTC CMOS RAM */
 	u8  century;		/* Index to century in RTC CMOS RAM */
-	u8  reserved4;		/* Reserved */
-	u8  reserved4a;		/* Reserved */
-	u8  reserved4b;		/* Reserved */
+	u16 boot_flags;		/* IA-PC Boot Architecture Flags (see below for individual flags) */
+	u8 reserved;		/* Reserved, must be zero */
+	u32 flags;		/* Miscellaneous flag bits (see below for individual flags) */
+	struct acpi_generic_address reset_register;	/* 64-bit address of the Reset register */
+	u8 reset_value;		/* Value to write to the reset_register port to reset the system */
+	u16 arm_boot_flags;	/* ARM-Specific Boot Flags (see below for individual flags) (ACPI 5.1) */
+	u8 minor_revision;	/* FADT Minor Revision (ACPI 5.1) */
+	u64 Xfacs;		/* 64-bit physical address of FACS */
+	u64 Xdsdt;		/* 64-bit physical address of DSDT */
+	struct acpi_generic_address xpm1a_event_block;		/* 64-bit Extended Power Mgt 1a Event Reg Blk address */
+	struct acpi_generic_address xpm1b_event_block;		/* 64-bit Extended Power Mgt 1b Event Reg Blk address */
+	struct acpi_generic_address xpm1a_control_block;	/* 64-bit Extended Power Mgt 1a Control Reg Blk address */
+	struct acpi_generic_address xpm1b_control_block;	/* 64-bit Extended Power Mgt 1b Control Reg Blk address */
+	struct acpi_generic_address xpm2_control_block;		/* 64-bit Extended Power Mgt 2 Control Reg Blk address */
+	struct acpi_generic_address xpm_timer_block;		/* 64-bit Extended Power Mgt Timer Ctrl Reg Blk address */
+	struct acpi_generic_address xgpe0_block;		/* 64-bit Extended General Purpose Event 0 Reg Blk address */
+	struct acpi_generic_address xgpe1_block;		/* 64-bit Extended General Purpose Event 1 Reg Blk address */
+	struct acpi_generic_address sleep_control;		/* 64-bit Sleep Control register (ACPI 5.0) */
+	struct acpi_generic_address sleep_status;		/* 64-bit Sleep Status register (ACPI 5.0) */
+	u64 hypervisor_id;	/* Hypervisor Vendor ID (ACPI 6.0) */
 };
 
 struct facs_descriptor_rev1
diff --git a/lib/acpi.c b/lib/acpi.c
index 240a922..aeaccb8 100644
--- a/lib/acpi.c
+++ b/lib/acpi.c
@@ -46,7 +46,8 @@ void *find_acpi_table_addr(u32 sig)
 
 	/* FACS is special... */
 	if (sig == FACS_SIGNATURE) {
-		struct fadt_descriptor_rev1 *fadt;
+		struct acpi_table_fadt *fadt;
+
 		fadt = find_acpi_table_addr(FACP_SIGNATURE);
 		if (!fadt)
 			return NULL;
diff --git a/x86/s3.c b/x86/s3.c
index 0bcbe18..b891a95 100644
--- a/x86/s3.c
+++ b/x86/s3.c
@@ -30,7 +30,7 @@ extern char resume_start, resume_end;
 
 int main(int argc, char **argv)
 {
-	struct fadt_descriptor_rev1 *fadt = find_acpi_table_addr(FACP_SIGNATURE);
+	struct acpi_table_fadt *fadt = find_acpi_table_addr(FACP_SIGNATURE);
 	struct facs_descriptor_rev1 *facs = find_acpi_table_addr(FACS_SIGNATURE);
 	char *addr, *resume_vec = (void*)0x1000;
 
diff --git a/x86/vmexit.c b/x86/vmexit.c
index 8015cc7..7e00bce 100644
--- a/x86/vmexit.c
+++ b/x86/vmexit.c
@@ -206,7 +206,7 @@ int pm_tmr_blk;
 static void inl_pmtimer(void)
 {
     if (!pm_tmr_blk) {
-	struct fadt_descriptor_rev1 *fadt;
+	struct acpi_table_fadt *fadt;
 
 	fadt = find_acpi_table_addr(FACP_SIGNATURE);
 	pm_tmr_blk = fadt->pm_tmr_blk;
-- 
2.25.1

[kvm-unit-tests PATCH v3 06/27] devicetree: Check if fdt is NULL before returning that a DT is available

[Nikos Nikoleris]

Up until now, for platfroms that support DT, KUT would unconditionally
use DT to configure the system and the code made the assumption that
the fdt will always be a valid pointer.

In Arm systems that support both ACPI and DT, we plat to follow the
same convension as the Linux kernel. First, we attempt to configure
the system using the DT. If an FDT was not provided then we attempt to
configure the system using ACPI.

As a result, for Arm systems with support for ACPI the fdt pointer can
be NULL. This change modifies dt_available() to check if the fdt is a
valid pointer and return if we can use information from the DT.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/devicetree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/devicetree.c b/lib/devicetree.c
index 78c1f6f..3ff9d16 100644
--- a/lib/devicetree.c
+++ b/lib/devicetree.c
@@ -16,7 +16,7 @@ const void *dt_fdt(void)
 
 bool dt_available(void)
 {
-	return fdt_check_header(fdt) == 0;
+	return fdt && fdt_check_header(fdt) == 0;
 }
 
 int dt_get_nr_cells(int fdtnode, u32 *nr_address_cells, u32 *nr_size_cells)
-- 
2.25.1

[kvm-unit-tests PATCH v3 07/27] arm/arm64: Add support for setting up the PSCI conduit through ACPI

[Nikos Nikoleris]

In systems with ACPI support and when a DT is not provided, we can use
the FADT to discover whether PSCI calls need to use smc or hvc
calls. This change implements this but retains the default behavior;
we check if a valid DT is provided, if not, we try to setup the PSCI
conduit using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 arm/Makefile.common |  1 +
 lib/acpi.h          |  5 +++++
 lib/arm/psci.c      | 25 ++++++++++++++++++++++++-
 3 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/arm/Makefile.common b/arm/Makefile.common
index 38385e0..8e9b3bb 100644
--- a/arm/Makefile.common
+++ b/arm/Makefile.common
@@ -38,6 +38,7 @@ cflatobjs += lib/alloc_page.o
 cflatobjs += lib/vmalloc.o
 cflatobjs += lib/alloc.o
 cflatobjs += lib/devicetree.o
+cflatobjs += lib/acpi.o
 cflatobjs += lib/pci.o
 cflatobjs += lib/pci-host-generic.o
 cflatobjs += lib/pci-testdev.o
diff --git a/lib/acpi.h b/lib/acpi.h
index e49647b..6c18f73 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -131,6 +131,11 @@ struct acpi_table_fadt
 	u64 hypervisor_id;	/* Hypervisor Vendor ID (ACPI 6.0) */
 };
 
+/* Masks for FADT ARM Boot Architecture Flags (arm_boot_flags) ACPI 5.1 */
+
+#define ACPI_FADT_PSCI_COMPLIANT    (1)         /* 00: [V5+] PSCI 0.2+ is implemented */
+#define ACPI_FADT_PSCI_USE_HVC      (1<<1)      /* 01: [V5+] HVC must be used instead of SMC as the PSCI conduit */
+
 struct facs_descriptor_rev1
 {
 	u32 signature;			/* ACPI Signature */
diff --git a/lib/arm/psci.c b/lib/arm/psci.c
index 9c031a1..afbc33d 100644
--- a/lib/arm/psci.c
+++ b/lib/arm/psci.c
@@ -6,6 +6,7 @@
  *
  * This work is licensed under the terms of the GNU LGPL, version 2.
  */
+#include <acpi.h>
 #include <devicetree.h>
 #include <asm/psci.h>
 #include <asm/setup.h>
@@ -56,7 +57,7 @@ void psci_system_off(void)
 	printf("CPU%d unable to do system off (error = %d)\n", smp_processor_id(), err);
 }
 
-void psci_set_conduit(void)
+static void psci_set_conduit_fdt(void)
 {
 	const void *fdt = dt_fdt();
 	const struct fdt_property *method;
@@ -75,3 +76,25 @@ void psci_set_conduit(void)
 	else
 		assert_msg(false, "Unknown PSCI conduit: %s", method->data);
 }
+
+static void psci_set_conduit_acpi(void)
+{
+	struct acpi_table_fadt *fadt = find_acpi_table_addr(FACP_SIGNATURE);
+
+	assert_msg(fadt, "Unable to find ACPI FADT");
+	assert_msg(fadt->arm_boot_flags & ACPI_FADT_PSCI_COMPLIANT,
+		   "PSCI is not supported in this platform");
+
+	if (fadt->arm_boot_flags & ACPI_FADT_PSCI_USE_HVC)
+		psci_invoke = psci_invoke_hvc;
+	else
+		psci_invoke = psci_invoke_smc;
+}
+
+void psci_set_conduit(void)
+{
+	if (dt_available())
+		psci_set_conduit_fdt();
+	else
+		psci_set_conduit_acpi();
+}
-- 
2.25.1

[kvm-unit-tests PATCH v3 08/27] arm/arm64: Add support for discovering the UART through ACPI

[Nikos Nikoleris]

In systems with ACPI support and when a DT is not provided, we can use
the SPCR to discover the serial port address range. This change
implements this but retains the default behavior; we check if a valid
DT is provided, if not, we try to discover the UART using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 lib/acpi.h   | 25 +++++++++++++++++++++++++
 lib/arm/io.c | 23 ++++++++++++++++++-----
 2 files changed, 43 insertions(+), 5 deletions(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index 6c18f73..1cd99c5 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -17,6 +17,7 @@
 #define XSDT_SIGNATURE ACPI_SIGNATURE('X','S','D','T')
 #define FACP_SIGNATURE ACPI_SIGNATURE('F','A','C','P')
 #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
+#define SPCR_SIGNATURE ACPI_SIGNATURE('S','P','C','R')
 
 
 #define ACPI_SIGNATURE_8BYTE(c1, c2, c3, c4, c5, c6, c7, c8)	\
@@ -148,6 +149,30 @@ struct facs_descriptor_rev1
 	u8  reserved3 [40];		/* Reserved - must be zero */
 };
 
+struct spcr_descriptor {
+	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
+	u8 interface_type;	/* 0=full 16550, 1=subset of 16550 */
+	u8 reserved[3];
+	struct acpi_generic_address serial_port;
+	u8 interrupt_type;
+	u8 pc_interrupt;
+	u32 interrupt;
+	u8 baud_rate;
+	u8 parity;
+	u8 stop_bits;
+	u8 flow_control;
+	u8 terminal_type;
+	u8 reserved1;
+	u16 pci_device_id;
+	u16 pci_vendor_id;
+	u8 pci_bus;
+	u8 pci_device;
+	u8 pci_function;
+	u32 pci_flags;
+	u8 pci_segment;
+	u32 reserved2;
+};
+
 #pragma pack(0)
 
 void set_efi_rsdp(struct rsdp_descriptor *rsdp);
diff --git a/lib/arm/io.c b/lib/arm/io.c
index 343e108..a91f116 100644
--- a/lib/arm/io.c
+++ b/lib/arm/io.c
@@ -9,6 +9,7 @@
  * This work is licensed under the terms of the GNU LGPL, version 2.
  */
 #include <libcflat.h>
+#include <acpi.h>
 #include <devicetree.h>
 #include <chr-testdev.h>
 #include <config.h>
@@ -29,7 +30,7 @@ static struct spinlock uart_lock;
 #define UART_EARLY_BASE (u8 *)(unsigned long)CONFIG_UART_EARLY_BASE
 static volatile u8 *uart0_base = UART_EARLY_BASE;
 
-static void uart0_init(void)
+static void uart0_init_fdt(void)
 {
 	/*
 	 * kvm-unit-tests uses the uart only for output. Both uart models have
@@ -65,17 +66,29 @@ static void uart0_init(void)
 	}
 
 	uart0_base = ioremap(base.addr, base.size);
+}
+
+static void uart0_init_acpi(void)
+{
+	struct spcr_descriptor *spcr = find_acpi_table_addr(SPCR_SIGNATURE);
+
+	assert_msg(spcr, "Unable to find ACPI SPCR");
+	uart0_base = ioremap(spcr->serial_port.address, spcr->serial_port.bit_width);
+}
+
+void io_init(void)
+{
+	if (dt_available())
+		uart0_init_fdt();
+	else
+		uart0_init_acpi();
 
 	if (uart0_base != UART_EARLY_BASE) {
 		printf("WARNING: early print support may not work. "
 		       "Found uart at %p, but early base is %p.\n",
 			uart0_base, UART_EARLY_BASE);
 	}
-}
 
-void io_init(void)
-{
-	uart0_init();
 	chr_testdev_init();
 }
 
-- 
2.25.1

[kvm-unit-tests PATCH v3 09/27] arm/arm64: Add support for timer initialization through ACPI

[Nikos Nikoleris]

For systems with ACPI support, we can discover timers through the ACPI
GTDT table. This change implements the code to discover timers through
the GTDT and adds ACPI support in timer_save_state. This change
retains the default behavior; we check if a valid DT is provided, if
not, we try to discover timers using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 arm/Makefile.common |  1 +
 lib/arm/asm/timer.h |  2 ++
 lib/acpi.h          | 18 +++++++++++
 lib/arm/setup.c     | 39 ----------------------
 lib/arm/timer.c     | 79 +++++++++++++++++++++++++++++++++++++++++++++
 arm/micro-bench.c   |  4 +--
 arm/timer.c         | 10 +++---
 7 files changed, 107 insertions(+), 46 deletions(-)
 create mode 100644 lib/arm/timer.c

diff --git a/arm/Makefile.common b/arm/Makefile.common
index 8e9b3bb..5be42c0 100644
--- a/arm/Makefile.common
+++ b/arm/Makefile.common
@@ -53,6 +53,7 @@ cflatobjs += lib/arm/psci.o
 cflatobjs += lib/arm/smp.o
 cflatobjs += lib/arm/delay.o
 cflatobjs += lib/arm/gic.o lib/arm/gic-v2.o lib/arm/gic-v3.o
+cflatobjs += lib/arm/timer.o
 
 OBJDIRS += lib/arm
 
diff --git a/lib/arm/asm/timer.h b/lib/arm/asm/timer.h
index f75cc67..aaf839f 100644
--- a/lib/arm/asm/timer.h
+++ b/lib/arm/asm/timer.h
@@ -27,5 +27,7 @@ extern struct timer_state __timer_state;
 #define TIMER_PTIMER_IRQ (__timer_state.ptimer.irq)
 #define TIMER_VTIMER_IRQ (__timer_state.vtimer.irq)
 
+void timer_save_state(void);
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _ASMARM_TIMER_H_ */
diff --git a/lib/acpi.h b/lib/acpi.h
index 1cd99c5..9910c39 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -18,6 +18,7 @@
 #define FACP_SIGNATURE ACPI_SIGNATURE('F','A','C','P')
 #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
 #define SPCR_SIGNATURE ACPI_SIGNATURE('S','P','C','R')
+#define GTDT_SIGNATURE ACPI_SIGNATURE('G','T','D','T')
 
 
 #define ACPI_SIGNATURE_8BYTE(c1, c2, c3, c4, c5, c6, c7, c8)	\
@@ -173,6 +174,23 @@ struct spcr_descriptor {
 	u32 reserved2;
 };
 
+struct acpi_table_gtdt {
+	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
+	u64 counter_block_addresss;
+	u32 reserved;
+	u32 secure_el1_interrupt;
+	u32 secure_el1_flags;
+	u32 non_secure_el1_interrupt;
+	u32 non_secure_el1_flags;
+	u32 virtual_timer_interrupt;
+	u32 virtual_timer_flags;
+	u32 non_secure_el2_interrupt;
+	u32 non_secure_el2_flags;
+	u64 counter_read_block_address;
+	u32 platform_timer_count;
+	u32 platform_timer_offset;
+};
+
 #pragma pack(0)
 
 void set_efi_rsdp(struct rsdp_descriptor *rsdp);
diff --git a/lib/arm/setup.c b/lib/arm/setup.c
index bcdf0d7..1572c64 100644
--- a/lib/arm/setup.c
+++ b/lib/arm/setup.c
@@ -35,8 +35,6 @@
 
 extern unsigned long etext;
 
-struct timer_state __timer_state;
-
 char *initrd;
 u32 initrd_size;
 
@@ -199,43 +197,6 @@ static void mem_init(phys_addr_t freemem_start)
 	page_alloc_ops_enable();
 }
 
-static void timer_save_state(void)
-{
-	const struct fdt_property *prop;
-	const void *fdt = dt_fdt();
-	int node, len;
-	u32 *data;
-
-	node = fdt_node_offset_by_compatible(fdt, -1, "arm,armv8-timer");
-	assert(node >= 0 || node == -FDT_ERR_NOTFOUND);
-
-	if (node == -FDT_ERR_NOTFOUND) {
-		__timer_state.ptimer.irq = -1;
-		__timer_state.vtimer.irq = -1;
-		return;
-	}
-
-	/*
-	 * From Linux devicetree timer binding documentation
-	 *
-	 * interrupts <type irq flags>:
-	 *	secure timer irq
-	 *	non-secure timer irq		(ptimer)
-	 *	virtual timer irq		(vtimer)
-	 *	hypervisor timer irq
-	 */
-	prop = fdt_get_property(fdt, node, "interrupts", &len);
-	assert(prop && len == (4 * 3 * sizeof(u32)));
-
-	data = (u32 *)prop->data;
-	assert(fdt32_to_cpu(data[3]) == 1 /* PPI */);
-	__timer_state.ptimer.irq = fdt32_to_cpu(data[4]);
-	__timer_state.ptimer.irq_flags = fdt32_to_cpu(data[5]);
-	assert(fdt32_to_cpu(data[6]) == 1 /* PPI */);
-	__timer_state.vtimer.irq = fdt32_to_cpu(data[7]);
-	__timer_state.vtimer.irq_flags = fdt32_to_cpu(data[8]);
-}
-
 void setup(const void *fdt, phys_addr_t freemem_start)
 {
 	void *freemem;
diff --git a/lib/arm/timer.c b/lib/arm/timer.c
new file mode 100644
index 0000000..507d9c8
--- /dev/null
+++ b/lib/arm/timer.c
@@ -0,0 +1,79 @@
+/*
+ * Initialize timers.
+ *
+ * Copyright (C) 2022, Arm Ltd., Nikos Nikoleris <nikos.nikoleris@arm.com>
+ * Copyright (C) 2014, Red Hat Inc, Andrew Jones <drjones@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU LGPL, version 2.
+ */
+#include <libcflat.h>
+#include <acpi.h>
+#include <devicetree.h>
+#include <libfdt/libfdt.h>
+#include <asm/gic.h>
+#include <asm/timer.h>
+
+struct timer_state __timer_state;
+
+static void timer_save_state_fdt(void)
+{
+	const struct fdt_property *prop;
+	const void *fdt = dt_fdt();
+	int node, len;
+	u32 *data;
+
+	node = fdt_node_offset_by_compatible(fdt, -1, "arm,armv8-timer");
+	assert(node >= 0 || node == -FDT_ERR_NOTFOUND);
+
+	if (node == -FDT_ERR_NOTFOUND) {
+		__timer_state.ptimer.irq = -1;
+		__timer_state.vtimer.irq = -1;
+		return;
+	}
+
+	/*
+	 * From Linux devicetree timer binding documentation
+	 *
+	 * interrupts <type irq flags>:
+	 *	secure timer irq
+	 *	non-secure timer irq		(ptimer)
+	 *	virtual timer irq		(vtimer)
+	 *	hypervisor timer irq
+	 */
+	prop = fdt_get_property(fdt, node, "interrupts", &len);
+	assert(prop && len == (4 * 3 * sizeof(u32)));
+
+	data = (u32 *)prop->data;
+	assert(fdt32_to_cpu(data[3]) == 1 /* PPI */);
+	__timer_state.ptimer.irq = PPI(fdt32_to_cpu(data[4]));
+	__timer_state.ptimer.irq_flags = fdt32_to_cpu(data[5]);
+	assert(fdt32_to_cpu(data[6]) == 1 /* PPI */);
+	__timer_state.vtimer.irq = PPI(fdt32_to_cpu(data[7]));
+	__timer_state.vtimer.irq_flags = fdt32_to_cpu(data[8]);
+}
+
+static void timer_save_state_acpi(void)
+{
+	struct acpi_table_gtdt *gtdt = find_acpi_table_addr(GTDT_SIGNATURE);
+
+	if (!gtdt) {
+		printf("Cannot find ACPI GTDT");
+		__timer_state.ptimer.irq = -1;
+		__timer_state.vtimer.irq = -1;
+		return;
+	}
+
+	__timer_state.ptimer.irq = gtdt->non_secure_el1_interrupt;
+	__timer_state.ptimer.irq_flags = gtdt->non_secure_el1_flags;
+
+	__timer_state.vtimer.irq = gtdt->virtual_timer_interrupt;
+	__timer_state.vtimer.irq_flags = gtdt->virtual_timer_flags;
+}
+
+void timer_save_state(void)
+{
+	if (dt_available())
+		timer_save_state_fdt();
+	else
+		timer_save_state_acpi();
+}
diff --git a/arm/micro-bench.c b/arm/micro-bench.c
index 8436612..5be9b18 100644
--- a/arm/micro-bench.c
+++ b/arm/micro-bench.c
@@ -43,7 +43,7 @@ static void gic_irq_handler(struct pt_regs *regs)
 	irq_received = true;
 	gic_write_eoir(irqstat);
 
-	if (irqstat == PPI(TIMER_VTIMER_IRQ)) {
+	if (irqstat == TIMER_VTIMER_IRQ) {
 		write_sysreg((ARCH_TIMER_CTL_IMASK | ARCH_TIMER_CTL_ENABLE),
 			     cntv_ctl_el0);
 		isb();
@@ -229,7 +229,7 @@ static bool timer_prep(void)
 		assert_msg(0, "Unreachable");
 	}
 
-	writel(1 << PPI(TIMER_VTIMER_IRQ), gic_isenabler);
+	writel(1 << TIMER_VTIMER_IRQ, gic_isenabler);
 	write_sysreg(ARCH_TIMER_CTL_IMASK | ARCH_TIMER_CTL_ENABLE, cntv_ctl_el0);
 	isb();
 
diff --git a/arm/timer.c b/arm/timer.c
index c4e7b10..a4a8281 100644
--- a/arm/timer.c
+++ b/arm/timer.c
@@ -139,7 +139,7 @@ static struct timer_info ptimer_info = {
 
 static void set_timer_irq_enabled(struct timer_info *info, bool enabled)
 {
-	u32 val = 1 << PPI(info->irq);
+	u32 val = 1 << info->irq;
 
 	if (enabled)
 		writel(val, gic_isenabler);
@@ -153,9 +153,9 @@ static void irq_handler(struct pt_regs *regs)
 	u32 irqstat = gic_read_iar();
 	u32 irqnr = gic_iar_irqnr(irqstat);
 
-	if (irqnr == PPI(vtimer_info.irq)) {
+	if (irqnr == vtimer_info.irq) {
 		info = &vtimer_info;
-	} else if (irqnr == PPI(ptimer_info.irq)) {
+	} else if (irqnr == ptimer_info.irq) {
 		info = &ptimer_info;
 	} else {
 		if (irqnr != GICC_INT_SPURIOUS)
@@ -185,9 +185,9 @@ static bool gic_timer_check_state(struct timer_info *info,
 	/* Wait for up to 1s for the GIC to sample the interrupt. */
 	for (i = 0; i < 10; i++) {
 		mdelay(100);
-		if (gic_irq_state(PPI(info->irq)) == expected_state) {
+		if (gic_irq_state(info->irq) == expected_state) {
 			mdelay(100);
-			if (gic_irq_state(PPI(info->irq)) == expected_state)
+			if (gic_irq_state(info->irq) == expected_state)
 				return true;
 		}
 	}
-- 
2.25.1

[kvm-unit-tests PATCH v3 10/27] arm/arm64: Add support for cpu initialization through ACPI

[Nikos Nikoleris]

In systems with ACPI support and when a DT is not provided, we can use
the MADTs to discover the number of CPUs and their corresponding MIDR.
This change implements this but retains the default behavior; we check
if a valid DT is provided, if not, we try to discover the cores in the
system using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 lib/acpi.h      | 63 +++++++++++++++++++++++++++++++++++++++++++++++++
 lib/acpi.c      | 21 +++++++++++++++++
 lib/arm/setup.c | 25 +++++++++++++++++---
 3 files changed, 106 insertions(+), 3 deletions(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index 9910c39..160bfbe 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -17,6 +17,7 @@
 #define XSDT_SIGNATURE ACPI_SIGNATURE('X','S','D','T')
 #define FACP_SIGNATURE ACPI_SIGNATURE('F','A','C','P')
 #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
+#define MADT_SIGNATURE ACPI_SIGNATURE('A','P','I','C')
 #define SPCR_SIGNATURE ACPI_SIGNATURE('S','P','C','R')
 #define GTDT_SIGNATURE ACPI_SIGNATURE('G','T','D','T')
 
@@ -150,6 +151,67 @@ struct facs_descriptor_rev1
 	u8  reserved3 [40];		/* Reserved - must be zero */
 };
 
+struct acpi_table_madt {
+	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
+	u32 address;		/* Physical address of local APIC */
+	u32 flags;
+};
+
+struct acpi_subtable_header {
+	u8 type;
+	u8 length;
+};
+
+typedef int (*acpi_table_handler)(struct acpi_subtable_header *header);
+
+/* 11: Generic interrupt - GICC (ACPI 5.0 + ACPI 6.0 + ACPI 6.3 changes) */
+
+struct acpi_madt_generic_interrupt {
+	u8 type;
+	u8 length;
+	u16 reserved;		/* reserved - must be zero */
+	u32 cpu_interface_number;
+	u32 uid;
+	u32 flags;
+	u32 parking_version;
+	u32 performance_interrupt;
+	u64 parked_address;
+	u64 base_address;
+	u64 gicv_base_address;
+	u64 gich_base_address;
+	u32 vgic_interrupt;
+	u64 gicr_base_address;
+	u64 arm_mpidr;
+	u8 efficiency_class;
+	u8 reserved2[1];
+	u16 spe_interrupt;	/* ACPI 6.3 */
+};
+
+/* Values for MADT subtable type in struct acpi_subtable_header */
+
+enum acpi_madt_type {
+	ACPI_MADT_TYPE_LOCAL_APIC = 0,
+	ACPI_MADT_TYPE_IO_APIC = 1,
+	ACPI_MADT_TYPE_INTERRUPT_OVERRIDE = 2,
+	ACPI_MADT_TYPE_NMI_SOURCE = 3,
+	ACPI_MADT_TYPE_LOCAL_APIC_NMI = 4,
+	ACPI_MADT_TYPE_LOCAL_APIC_OVERRIDE = 5,
+	ACPI_MADT_TYPE_IO_SAPIC = 6,
+	ACPI_MADT_TYPE_LOCAL_SAPIC = 7,
+	ACPI_MADT_TYPE_INTERRUPT_SOURCE = 8,
+	ACPI_MADT_TYPE_LOCAL_X2APIC = 9,
+	ACPI_MADT_TYPE_LOCAL_X2APIC_NMI = 10,
+	ACPI_MADT_TYPE_GENERIC_INTERRUPT = 11,
+	ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR = 12,
+	ACPI_MADT_TYPE_GENERIC_MSI_FRAME = 13,
+	ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR = 14,
+	ACPI_MADT_TYPE_GENERIC_TRANSLATOR = 15,
+	ACPI_MADT_TYPE_RESERVED = 16	/* 16 and greater are reserved */
+};
+
+/* MADT Local APIC flags */
+#define ACPI_MADT_ENABLED		(1) /* 00: Processor is usable if set */
+
 struct spcr_descriptor {
 	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
 	u8 interface_type;	/* 0=full 16550, 1=subset of 16550 */
@@ -195,5 +257,6 @@ struct acpi_table_gtdt {
 
 void set_efi_rsdp(struct rsdp_descriptor *rsdp);
 void* find_acpi_table_addr(u32 sig);
+void acpi_table_parse_madt(enum acpi_madt_type mtype, acpi_table_handler handler);
 
 #endif
diff --git a/lib/acpi.c b/lib/acpi.c
index aeaccb8..b652d18 100644
--- a/lib/acpi.c
+++ b/lib/acpi.c
@@ -104,3 +104,24 @@ void *find_acpi_table_addr(u32 sig)
 
 	return NULL;
 }
+
+void acpi_table_parse_madt(enum acpi_madt_type mtype,
+			   acpi_table_handler handler)
+{
+	struct acpi_table_madt *madt;
+	struct acpi_subtable_header *header;
+	void *end;
+
+	madt = find_acpi_table_addr(MADT_SIGNATURE);
+	assert(madt);
+
+	header = (void *)(ulong)madt + sizeof(struct acpi_table_madt);
+	end = (void *)((ulong)madt + madt->length);
+
+	while ((void *)header < end) {
+		if (header->type == mtype)
+			handler(header);
+
+		header = (void *)(ulong)header + header->length;
+	}
+}
diff --git a/lib/arm/setup.c b/lib/arm/setup.c
index 1572c64..3612991 100644
--- a/lib/arm/setup.c
+++ b/lib/arm/setup.c
@@ -13,6 +13,7 @@
 #include <libcflat.h>
 #include <libfdt/libfdt.h>
 #include <devicetree.h>
+#include <acpi.h>
 #include <alloc.h>
 #include <alloc_phys.h>
 #include <alloc_page.h>
@@ -55,7 +56,7 @@ int mpidr_to_cpu(uint64_t mpidr)
 	return -1;
 }
 
-static void cpu_set(int fdtnode __unused, u64 regval, void *info __unused)
+static void cpu_set_fdt(int fdtnode __unused, u64 regval, void *info __unused)
 {
 	int cpu = nr_cpus++;
 
@@ -65,13 +66,31 @@ static void cpu_set(int fdtnode __unused, u64 regval, void *info __unused)
 	set_cpu_present(cpu, true);
 }
 
+static int cpu_set_acpi(struct acpi_subtable_header *header)
+{
+	int cpu = nr_cpus++;
+	struct acpi_madt_generic_interrupt *gicc = (void *)header;
+
+	assert_msg(cpu < NR_CPUS, "Number cpus exceeds maximum supported (%d).", NR_CPUS);
+
+	cpus[cpu] = gicc->arm_mpidr;
+	set_cpu_present(cpu, true);
+
+	return 0;
+}
+
 static void cpu_init(void)
 {
 	int ret;
 
 	nr_cpus = 0;
-	ret = dt_for_each_cpu_node(cpu_set, NULL);
-	assert(ret == 0);
+	if (dt_available()) {
+		ret = dt_for_each_cpu_node(cpu_set_fdt, NULL);
+		assert(ret == 0);
+	} else {
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT, cpu_set_acpi);
+	}
+
 	set_cpu_online(0, true);
 }
 
-- 
2.25.1

[kvm-unit-tests PATCH v3 11/27] arm/arm64: Add support for gic initialization through ACPI

[Nikos Nikoleris]

In systems with ACPI support and when a DT is not provided, we can use
the MADTs to figure out if it implements a GICv2 or a GICv3 and
discover the GIC parameters. This change implements this but retains
the default behavior; we check if a valid DT is provided, if not, we
try to discover the cores in the system using ACPI.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/acpi.h     |  44 +++++++++++++++++
 lib/libcflat.h |   1 +
 lib/arm/gic.c  | 127 ++++++++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 171 insertions(+), 1 deletion(-)

diff --git a/lib/acpi.h b/lib/acpi.h
index 160bfbe..9f6fbe4 100644
--- a/lib/acpi.h
+++ b/lib/acpi.h
@@ -187,6 +187,50 @@ struct acpi_madt_generic_interrupt {
 	u16 spe_interrupt;	/* ACPI 6.3 */
 };
 
+#define ACPI_MADT_ENABLED           (1) /* 00: Processor is usable if set */
+
+/* 12: Generic Distributor (ACPI 5.0 + ACPI 6.0 changes) */
+
+struct acpi_madt_generic_distributor {
+	struct acpi_subtable_header header;
+	u16 reserved;           /* reserved - must be zero */
+	u32 gic_id;
+	u64 base_address;
+	u32 global_irq_base;
+	u8 version;
+	u8 reserved2[3];        /* reserved - must be zero */
+};
+
+/* Values for Version field above */
+
+enum acpi_madt_gic_version {
+	ACPI_MADT_GIC_VERSION_NONE = 0,
+	ACPI_MADT_GIC_VERSION_V1 = 1,
+	ACPI_MADT_GIC_VERSION_V2 = 2,
+	ACPI_MADT_GIC_VERSION_V3 = 3,
+	ACPI_MADT_GIC_VERSION_V4 = 4,
+	ACPI_MADT_GIC_VERSION_RESERVED = 5      /* 5 and greater are reserved */
+};
+
+/* 14: Generic Redistributor (ACPI 5.1) */
+
+struct acpi_madt_generic_redistributor {
+	struct acpi_subtable_header header;
+	u16 reserved;           /* reserved - must be zero */
+	u64 base_address;
+	u32 length;
+};
+
+/* 15: Generic Translator (ACPI 6.0) */
+
+struct acpi_madt_generic_translator {
+	struct acpi_subtable_header header;
+	u16 reserved;           /* reserved - must be zero */
+	u32 translation_id;
+	u64 base_address;
+	u32 reserved2;
+};
+
 /* Values for MADT subtable type in struct acpi_subtable_header */
 
 enum acpi_madt_type {
diff --git a/lib/libcflat.h b/lib/libcflat.h
index c1fd31f..700f435 100644
--- a/lib/libcflat.h
+++ b/lib/libcflat.h
@@ -161,6 +161,7 @@ extern void setup_vm(void);
 #define SZ_8K			(1 << 13)
 #define SZ_16K			(1 << 14)
 #define SZ_64K			(1 << 16)
+#define SZ_128K			(1 << 17)
 #define SZ_1M			(1 << 20)
 #define SZ_2M			(1 << 21)
 #define SZ_1G			(1 << 30)
diff --git a/lib/arm/gic.c b/lib/arm/gic.c
index 1bfcfcf..c346d1f 100644
--- a/lib/arm/gic.c
+++ b/lib/arm/gic.c
@@ -3,6 +3,7 @@
  *
  * This work is licensed under the terms of the GNU LGPL, version 2.
  */
+#include <acpi.h>
 #include <devicetree.h>
 #include <asm/gic.h>
 #include <asm/io.h>
@@ -120,7 +121,7 @@ int gic_version(void)
 	return 0;
 }
 
-int gic_init(void)
+static int gic_init_fdt(void)
 {
 	if (gicv2_init()) {
 		gic_common_ops = &gicv2_common_ops;
@@ -133,6 +134,130 @@ int gic_init(void)
 	return gic_version();
 }
 
+#define ACPI_GICV2_DIST_MEM_SIZE	(SZ_4K)
+#define ACPI_GIC_CPU_IF_MEM_SIZE	(SZ_8K)
+#define ACPI_GICV3_DIST_MEM_SIZE	(SZ_64K)
+#define ACPI_GICV3_ITS_MEM_SIZE		(SZ_128K)
+
+static int gic_acpi_version(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_distributor *dist = (void *)header;
+	int version = dist->version;
+
+	if (version == 2)
+		gic_common_ops = &gicv2_common_ops;
+	else if (version == 3)
+		gic_common_ops = &gicv3_common_ops;
+
+	return version;
+}
+
+static int gicv2_acpi_parse_madt_cpu(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_interrupt *gicc = (void *)header;
+	static phys_addr_t gicc_base_address;
+
+	if (!(gicc->flags & ACPI_MADT_ENABLED))
+		return 0;
+
+	if (!gicc_base_address) {
+		gicc_base_address = gicc->base_address;
+		gicv2_data.cpu_base = ioremap(gicc_base_address, ACPI_GIC_CPU_IF_MEM_SIZE);
+	}
+	assert(gicc_base_address == gicc->base_address);
+
+	return 0;
+}
+
+static int gicv2_acpi_parse_madt_dist(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_distributor *dist = (void *)header;
+
+	gicv2_data.dist_base = ioremap(dist->base_address, ACPI_GICV2_DIST_MEM_SIZE);
+
+	return 0;
+}
+
+static int gicv3_acpi_parse_madt_gicc(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_interrupt *gicc = (void *)header;
+	static phys_addr_t gicr_base_address;
+
+	if (!(gicc->flags & ACPI_MADT_ENABLED))
+		return 0;
+
+	if (!gicr_base_address) {
+		gicr_base_address = gicc->gicr_base_address;
+		gicv3_data.redist_bases[0] = ioremap(gicr_base_address, SZ_64K * 2);
+	}
+	assert(gicr_base_address == gicc->gicr_base_address);
+
+	return 0;
+}
+
+static int gicv3_acpi_parse_madt_dist(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_distributor *dist = (void *)header;
+
+	gicv3_data.dist_base = ioremap(dist->base_address, ACPI_GICV3_DIST_MEM_SIZE);
+
+	return 0;
+}
+
+static int gicv3_acpi_parse_madt_redist(struct acpi_subtable_header *header)
+{
+	static int i;
+	struct acpi_madt_generic_redistributor *redist = (void *)header;
+
+	gicv3_data.redist_bases[i++] = ioremap(redist->base_address, redist->length);
+
+	return 0;
+}
+
+static int gicv3_acpi_parse_madt_its(struct acpi_subtable_header *header)
+{
+	struct acpi_madt_generic_translator *its_entry = (void *)header;
+
+	its_data.base = ioremap(its_entry->base_address, ACPI_GICV3_ITS_MEM_SIZE - 1);
+
+	return 0;
+}
+
+static int gic_init_acpi(void)
+{
+	acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR,
+			      gic_acpi_version);
+	if (gic_version() == 2) {
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
+				      gicv2_acpi_parse_madt_cpu);
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR,
+				      gicv2_acpi_parse_madt_dist);
+	} else if (gic_version() == 3) {
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_DISTRIBUTOR,
+				      gicv3_acpi_parse_madt_dist);
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
+				      gicv3_acpi_parse_madt_redist);
+		if (!gicv3_data.redist_base)
+			acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
+					      gicv3_acpi_parse_madt_gicc);
+		acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_TRANSLATOR,
+				      gicv3_acpi_parse_madt_its);
+#ifdef __aarch64__
+		its_init();
+#endif
+	}
+
+	return gic_version();
+}
+
+int gic_init(void)
+{
+	if (dt_available())
+		return gic_init_fdt();
+	else
+		return gic_init_acpi();
+}
+
 void gic_enable_defaults(void)
 {
 	if (!gic_common_ops) {
-- 
2.25.1

[kvm-unit-tests PATCH v3 12/27] lib/printf: Support for precision modifier in printf

[Nikos Nikoleris]

This follows the typical format of:

printf("%.Ns", *str);

Where N might be a decimal digit string or '*'. This feature is used
by a future change.

See also: man 3 printf

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 lib/printf.c | 95 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 78 insertions(+), 17 deletions(-)

diff --git a/lib/printf.c b/lib/printf.c
index 383799e..d600199 100644
--- a/lib/printf.c
+++ b/lib/printf.c
@@ -6,6 +6,7 @@
  */
 
 #include "libcflat.h"
+#include "ctype.h"
 
 #define BUFSZ 2000
 
@@ -19,6 +20,7 @@ typedef struct strprops {
 	char pad;
 	int npad;
 	bool alternate;
+	int precision;
 } strprops_t;
 
 static void addchar(pstream_t *p, char c)
@@ -43,7 +45,7 @@ static void print_str(pstream_t *p, const char *s, strprops_t props)
 		}
 	}
 
-	while (*s)
+	while (*s && props.precision--)
 		addchar(p, *s++);
 
 	if (npad < 0) {
@@ -73,12 +75,13 @@ static void print_int(pstream_t *ps, long long n, int base, strprops_t props)
 		n /= base;
 	}
 
+	while (p == buf || (p - buf < props.precision))
+		*p++ = '0';
+	props.precision = -1;
+
 	if (s)
 		*p++ = '-';
 
-	if (p == buf)
-		*p++ = '0';
-
 	for (i = 0; i < (p - buf) / 2; ++i) {
 		char tmp;
 
@@ -104,8 +107,13 @@ static void print_unsigned(pstream_t *ps, unsigned long long n, int base,
 	}
 
 	if (p == buf)
+		props.alternate = false;
+
+	while (p == buf || (p - buf < props.precision))
 		*p++ = '0';
-	else if (props.alternate && base == 16) {
+	props.precision = -1;
+
+	if (props.alternate && base == 16) {
 		if (props.pad == '0') {
 			addchar(ps, '0');
 			addchar(ps, 'x');
@@ -147,9 +155,56 @@ static int fmtnum(const char **fmt)
 	return num;
 }
 
+/*
+ * Adapted from drivers/firmware/efi/libstub/vsprintf.c
+ */
+static int skip_atoi(const char **s)
+{
+	int i = 0;
+
+	do {
+		i = i*10 + *((*s)++) - '0';
+	} while (isdigit(**s));
+
+	return i;
+}
+
+/*
+ * Adapted from drivers/firmware/efi/libstub/vsprintf.c
+ */
+static int get_int(const char **fmt, va_list *ap)
+{
+	if (isdigit(**fmt))
+		return skip_atoi(fmt);
+
+	if (**fmt == '*') {
+		++(*fmt);
+		/* it's the next argument */
+		return va_arg(*ap, int);
+	}
+	return 0;
+}
+
 int vsnprintf(char *buf, int size, const char *fmt, va_list va)
 {
 	pstream_t s;
+	va_list args;
+
+	/*
+	 * We want to pass our input va_list to helper functions by reference,
+	 * but there's an annoying edge case. If va_list was originally passed
+	 * to us by value, we could just pass &ap down to the helpers. This is
+	 * the case on, for example, X86_32.
+	 * However, on X86_64 (and possibly others), va_list is actually a
+	 * size-1 array containing a structure. Our function parameter ap has
+	 * decayed from T[1] to T*, and &ap has type T** rather than T(*)[1],
+	 * which is what will be expected by a function taking a va_list *
+	 * parameter.
+	 * One standard way to solve this mess is by creating a copy in a local
+	 * variable of type va_list and then passing a pointer to that local
+	 * copy instead, which is what we do here.
+	 */
+	va_copy(args, va);
 
 	s.buffer = buf;
 	s.remain = size - 1;
@@ -160,6 +215,7 @@ int vsnprintf(char *buf, int size, const char *fmt, va_list va)
 		strprops_t props;
 		memset(&props, 0, sizeof(props));
 		props.pad = ' ';
+		props.precision = -1;
 
 		if (f != '%') {
 			addchar(&s, f);
@@ -172,11 +228,15 @@ morefmt:
 			addchar(&s, '%');
 			break;
 		case 'c':
-			addchar(&s, va_arg(va, int));
+			addchar(&s, va_arg(args, int));
 			break;
 		case '\0':
 			--fmt;
 			break;
+		case '.':
+			props.pad = ' ';
+			props.precision = get_int(&fmt, &args);
+			goto morefmt;
 		case '#':
 			props.alternate = true;
 			goto morefmt;
@@ -204,54 +264,55 @@ morefmt:
 		case 'd':
 			switch (nlong) {
 			case 0:
-				print_int(&s, va_arg(va, int), 10, props);
+				print_int(&s, va_arg(args, int), 10, props);
 				break;
 			case 1:
-				print_int(&s, va_arg(va, long), 10, props);
+				print_int(&s, va_arg(args, long), 10, props);
 				break;
 			default:
-				print_int(&s, va_arg(va, long long), 10, props);
+				print_int(&s, va_arg(args, long long), 10, props);
 				break;
 			}
 			break;
 		case 'u':
 			switch (nlong) {
 			case 0:
-				print_unsigned(&s, va_arg(va, unsigned), 10, props);
+				print_unsigned(&s, va_arg(args, unsigned int), 10, props);
 				break;
 			case 1:
-				print_unsigned(&s, va_arg(va, unsigned long), 10, props);
+				print_unsigned(&s, va_arg(args, unsigned long), 10, props);
 				break;
 			default:
-				print_unsigned(&s, va_arg(va, unsigned long long), 10, props);
+				print_unsigned(&s, va_arg(args, unsigned long long), 10, props);
 				break;
 			}
 			break;
 		case 'x':
 			switch (nlong) {
 			case 0:
-				print_unsigned(&s, va_arg(va, unsigned), 16, props);
+				print_unsigned(&s, va_arg(args, unsigned int), 16, props);
 				break;
 			case 1:
-				print_unsigned(&s, va_arg(va, unsigned long), 16, props);
+				print_unsigned(&s, va_arg(args, unsigned long), 16, props);
 				break;
 			default:
-				print_unsigned(&s, va_arg(va, unsigned long long), 16, props);
+				print_unsigned(&s, va_arg(args, unsigned long long), 16, props);
 				break;
 			}
 			break;
 		case 'p':
 			props.alternate = true;
-			print_unsigned(&s, (unsigned long)va_arg(va, void *), 16, props);
+			print_unsigned(&s, (unsigned long)va_arg(args, void *), 16, props);
 			break;
 		case 's':
-			print_str(&s, va_arg(va, const char *), props);
+			print_str(&s, va_arg(args, const char *), props);
 			break;
 		default:
 			addchar(&s, f);
 			break;
 		}
 	}
+	va_end(args);
 	*s.buffer = 0;
 	return s.added;
 }
-- 
2.25.1

[kvm-unit-tests PATCH v3 13/27] lib/printf: Add support for printing wide strings

[Nikos Nikoleris]

This change adds support for wide strings (u16*) to printf()
variants. This feature is used by a future change.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 lib/printf.c | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 100 insertions(+), 1 deletion(-)

diff --git a/lib/printf.c b/lib/printf.c
index d600199..27041bb 100644
--- a/lib/printf.c
+++ b/lib/printf.c
@@ -58,6 +58,102 @@ static void print_str(pstream_t *p, const char *s, strprops_t props)
 	}
 }
 
+/*
+ * Adapted from drivers/firmware/efi/libstub/vsprintf.c
+ */
+static u32 utf16_to_utf32(const u16 **s16)
+{
+	u16 c0, c1;
+
+	c0 = *(*s16)++;
+	/* not a surrogate */
+	if ((c0 & 0xf800) != 0xd800)
+		return c0;
+	/* invalid: low surrogate instead of high */
+	if (c0 & 0x0400)
+		return 0xfffd;
+	c1 = **s16;
+	/* invalid: missing low surrogate */
+	if ((c1 & 0xfc00) != 0xdc00)
+		return 0xfffd;
+	/* valid surrogate pair */
+	++(*s16);
+	return (0x10000 - (0xd800 << 10) - 0xdc00) + (c0 << 10) + c1;
+}
+
+/*
+ * Adapted from drivers/firmware/efi/libstub/vsprintf.c
+ */
+static size_t utf16s_utf8nlen(const u16 *s16, size_t maxlen)
+{
+	size_t len, clen;
+
+	for (len = 0; len < maxlen && *s16; len += clen) {
+		u16 c0 = *s16++;
+
+		/* First, get the length for a BMP character */
+		clen = 1 + (c0 >= 0x80) + (c0 >= 0x800);
+		if (len + clen > maxlen)
+			break;
+		/*
+		 * If this is a high surrogate, and we're already at maxlen, we
+		 * can't include the character if it's a valid surrogate pair.
+		 * Avoid accessing one extra word just to check if it's valid
+		 * or not.
+		 */
+		if ((c0 & 0xfc00) == 0xd800) {
+			if (len + clen == maxlen)
+				break;
+			if ((*s16 & 0xfc00) == 0xdc00) {
+				++s16;
+				++clen;
+			}
+		}
+	}
+
+	return len;
+}
+
+/*
+ * Adapted from drivers/firmware/efi/libstub/vsprintf.c
+ */
+static void print_wstring(pstream_t *p, const u16 *s, strprops_t props)
+{
+	const u16 *ws = (const u16 *)s;
+	size_t pos = 0, size = p->remain + 1, len = utf16s_utf8nlen(ws, props.precision);
+
+	while (len-- > 0) {
+		u32 c32 = utf16_to_utf32(&ws);
+		u8 *s8;
+		size_t clen;
+
+		if (c32 < 0x80) {
+			addchar(p, c32);
+			continue;
+		}
+
+		/* Number of trailing octets */
+		clen = 1 + (c32 >= 0x800) + (c32 >= 0x10000);
+
+		len -= clen;
+		s8 = (u8 *)(p->buffer - p->added + pos);
+
+		/* Avoid writing partial character */
+		addchar(p, '\0');
+		pos += clen;
+		if (pos >= size)
+			continue;
+
+		/* Set high bits of leading octet */
+		*s8 = (0xf00 >> 1) >> clen;
+		/* Write trailing octets in reverse order */
+		for (s8 += clen; clen; --clen, c32 >>= 6)
+			*s8-- = 0x80 | (c32 & 0x3f);
+		/* Set low bits of leading octet */
+		*s8 |= c32;
+	}
+}
+
 static char digits[16] = "0123456789abcdef";
 
 static void print_int(pstream_t *ps, long long n, int base, strprops_t props)
@@ -305,7 +401,10 @@ morefmt:
 			print_unsigned(&s, (unsigned long)va_arg(args, void *), 16, props);
 			break;
 		case 's':
-			print_str(&s, va_arg(args, const char *), props);
+			if (nlong)
+				print_wstring(&s, va_arg(args, const u16 *), props);
+			else
+				print_str(&s, va_arg(args, const char *), props);
 			break;
 		default:
 			addchar(&s, f);
-- 
2.25.1

[kvm-unit-tests PATCH v3 14/27] lib/efi: Add support for getting the cmdline

[Nikos Nikoleris]

This change adds support for discovering the command line arguments,
as a string. Then, we parse this string to populate __argc and __argv
for EFI tests.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/linux/efi.h |  20 ++++++++++
 lib/argv.h      |   1 +
 lib/argv.c      |   2 +-
 lib/efi.c       | 102 ++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 124 insertions(+), 1 deletion(-)

diff --git a/lib/linux/efi.h b/lib/linux/efi.h
index 455625a..9a1cf87 100644
--- a/lib/linux/efi.h
+++ b/lib/linux/efi.h
@@ -60,6 +60,8 @@ typedef guid_t efi_guid_t;
 
 #define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)
 
+#define LOADED_IMAGE_PROTOCOL_GUID EFI_GUID(0x5b1b31a1, 0x9562, 0x11d2,  0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b)
+
 typedef struct {
 	efi_guid_t guid;
 	void *table;
@@ -416,6 +418,24 @@ struct efi_boot_memmap {
 	unsigned long           *buff_size;
 };
 
+#define __aligned_u64 u64 __attribute__((aligned(8)))
+
+struct efi_loaded_image_64 {
+	u32			revision;
+	efi_handle_t		parent_handle;
+	efi_system_table_t	*system_table;
+	efi_handle_t		device_handle;
+	void			*file_path;
+	void			*reserved;
+	u32			load_options_size;
+	void			*load_options;
+	void			*image_base;
+	__aligned_u64		image_size;
+	unsigned int		image_code_type;
+	unsigned int		image_data_type;
+	efi_status_t		(__efiapi * unload)(efi_handle_t image_handle);
+};
+
 #define efi_bs_call(func, ...) efi_system_table->boottime->func(__VA_ARGS__)
 #define efi_rs_call(func, ...) efi_system_table->runtime->func(__VA_ARGS__)
 
diff --git a/lib/argv.h b/lib/argv.h
index 1fd746d..0fa7772 100644
--- a/lib/argv.h
+++ b/lib/argv.h
@@ -9,6 +9,7 @@
 #define _ARGV_H_
 
 extern void __setup_args(void);
+extern void setup_args(const char *args);
 extern void setup_args_progname(const char *args);
 extern void setup_env(char *env, int size);
 extern void add_setup_arg(const char *arg);
diff --git a/lib/argv.c b/lib/argv.c
index 9ffa673..fa5ff9a 100644
--- a/lib/argv.c
+++ b/lib/argv.c
@@ -41,7 +41,7 @@ void __setup_args(void)
 	__argc = argv - __argv;
 }
 
-static void setup_args(const char *args)
+void setup_args(const char *args)
 {
 	if (!args)
 		return;
diff --git a/lib/efi.c b/lib/efi.c
index 64cc978..f524ec9 100644
--- a/lib/efi.c
+++ b/lib/efi.c
@@ -8,6 +8,9 @@
  */
 
 #include "efi.h"
+#include <argv.h>
+#include <stdlib.h>
+#include <ctype.h>
 #include <libcflat.h>
 #include <asm/setup.h>
 
@@ -96,6 +99,80 @@ static void efi_exit(efi_status_t code)
 	efi_rs_call(reset_system, EFI_RESET_SHUTDOWN, code, 0, NULL);
 }
 
+/* Adapted from drivers/firmware/efi/libstub/efi-stub.c */
+static char *efi_convert_cmdline(struct efi_loaded_image_64 *image, int *cmd_line_len)
+{
+	const u16 *s2;
+	unsigned long cmdline_addr = 0;
+	int options_chars = image->load_options_size;
+	const u16 *options = image->load_options;
+	int options_bytes = 0, safe_options_bytes = 0;  /* UTF-8 bytes */
+	bool in_quote = false;
+	efi_status_t status;
+	const int COMMAND_LINE_SIZE = 2048;
+
+	if (options) {
+		s2 = options;
+		while (options_bytes < COMMAND_LINE_SIZE && options_chars--) {
+			u16 c = *s2++;
+
+			if (c < 0x80) {
+				if (c == L'\0' || c == L'\n')
+					break;
+				if (c == L'"')
+					in_quote = !in_quote;
+				else if (!in_quote && isspace((char)c))
+					safe_options_bytes = options_bytes;
+
+				options_bytes++;
+				continue;
+			}
+
+			/*
+			 * Get the number of UTF-8 bytes corresponding to a
+			 * UTF-16 character.
+			 * The first part handles everything in the BMP.
+			 */
+			options_bytes += 2 + (c >= 0x800);
+			/*
+			 * Add one more byte for valid surrogate pairs. Invalid
+			 * surrogates will be replaced with 0xfffd and take up
+			 * only 3 bytes.
+			 */
+			if ((c & 0xfc00) == 0xd800) {
+				/*
+				 * If the very last word is a high surrogate,
+				 * we must ignore it since we can't access the
+				 * low surrogate.
+				 */
+				if (!options_chars) {
+					options_bytes -= 3;
+				} else if ((*s2 & 0xfc00) == 0xdc00) {
+					options_bytes++;
+					options_chars--;
+					s2++;
+				}
+			}
+		}
+		if (options_bytes >= COMMAND_LINE_SIZE) {
+			options_bytes = safe_options_bytes;
+			printf("Command line is too long: truncated to %d bytes\n",
+			       options_bytes);
+		}
+	}
+
+	options_bytes++;        /* NUL termination */
+
+	status = efi_bs_call(allocate_pool, EFI_LOADER_DATA, options_bytes, (void **)&cmdline_addr);
+	if (status != EFI_SUCCESS)
+		return NULL;
+
+	snprintf((char *)cmdline_addr, options_bytes, "%.*ls", options_bytes - 1, options);
+
+	*cmd_line_len = options_bytes;
+	return (char *)cmdline_addr;
+}
+
 efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
 {
 	int ret;
@@ -109,6 +186,31 @@ efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
 	unsigned long map_size = 0, desc_size = 0, key = 0, buff_size = 0;
 	u32 desc_ver;
 
+	/* Helper variables needed to get the cmdline */
+	struct efi_loaded_image_64 *image;
+	efi_guid_t loaded_image_proto = LOADED_IMAGE_PROTOCOL_GUID;
+	char *cmdline_ptr = NULL;
+	int cmdline_size = 0;
+
+	/*
+	 * Get a handle to the loaded image protocol.  This is used to get
+	 * information about the running image, such as size and the command
+	 * line.
+	 */
+	status = efi_bs_call(handle_protocol, handle, &loaded_image_proto, (void *)&image);
+	if (status != EFI_SUCCESS) {
+		printf("Failed to get loaded image protocol\n");
+		goto efi_main_error;
+	}
+
+	cmdline_ptr = efi_convert_cmdline(image, &cmdline_size);
+	if (!cmdline_ptr) {
+		printf("getting command line via LOADED_IMAGE_PROTOCOL\n");
+		status = EFI_OUT_OF_RESOURCES;
+		goto efi_main_error;
+	}
+	setup_args(cmdline_ptr);
+
 	/* Set up efi_bootinfo */
 	efi_bootinfo.mem_map.map = &map;
 	efi_bootinfo.mem_map.map_size = &map_size;
-- 
2.25.1

[kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Nikos Nikoleris]

From: Andrew Jones <drjones@redhat.com>

The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
clean + invalidate after turning MMU off") justifies cleaning and
invalidating the dcache after disabling the MMU by saying it's nice
not to rely on the current page tables and that it should still work
(per the spec), as long as there's an identity map in the current
tables. Doing the invalidation after also somewhat helped with
reenabling the MMU without seeing stale data, but the real problem
with reenabling was because the cache needs to be disabled with
the MMU, but it wasn't.

Since we have to trust/validate that the current page tables have an
identity map anyway, then there's no harm in doing the clean
and invalidate first (it feels a little better to do so, anyway,
considering the cache maintenance instructions take virtual
addresses). Then, also disable the cache with the MMU to avoid
problems when reenabling. We invalidate the Icache and disable
that too for good measure. And, a final TLB invalidation ensures
we're crystal clean when we return from asm_mmu_disable().

Cc: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 arm/cstart.S   | 28 +++++++++++++++++++++-------
 arm/cstart64.S | 21 ++++++++++++++++-----
 2 files changed, 37 insertions(+), 12 deletions(-)

diff --git a/arm/cstart.S b/arm/cstart.S
index 7036e67..dc324c5 100644
--- a/arm/cstart.S
+++ b/arm/cstart.S
@@ -179,6 +179,7 @@ halt:
 .globl asm_mmu_enable
 asm_mmu_enable:
 	/* TLBIALL */
+	mov	r2, #0
 	mcr	p15, 0, r2, c8, c7, 0
 	dsb	nsh
 
@@ -211,12 +212,7 @@ asm_mmu_enable:
 
 .globl asm_mmu_disable
 asm_mmu_disable:
-	/* SCTLR */
-	mrc	p15, 0, r0, c1, c0, 0
-	bic	r0, #CR_M
-	mcr	p15, 0, r0, c1, c0, 0
-	isb
-
+	/* Clean + invalidate the entire memory */
 	ldr	r0, =__phys_offset
 	ldr	r0, [r0]
 	ldr	r1, =__phys_end
@@ -224,7 +220,25 @@ asm_mmu_disable:
 	sub	r1, r1, r0
 	dcache_by_line_op dccimvac, sy, r0, r1, r2, r3
 
-	mov     pc, lr
+	/* Invalidate Icache */
+	mov	r0, #0
+	mcr	p15, 0, r0, c7, c5, 0
+	isb
+
+	/*  Disable cache, Icache and MMU */
+	mrc	p15, 0, r0, c1, c0, 0
+	bic	r0, #CR_C
+	bic	r0, #CR_I
+	bic	r0, #CR_M
+	mcr	p15, 0, r0, c1, c0, 0
+	isb
+
+	/* Invalidate TLB */
+	mov	r0, #0
+	mcr	p15, 0, r0, c8, c7, 0
+	dsb	nsh
+
+	mov	pc, lr
 
 /*
  * Vectors
diff --git a/arm/cstart64.S b/arm/cstart64.S
index e4ab7d0..390feb9 100644
--- a/arm/cstart64.S
+++ b/arm/cstart64.S
@@ -246,11 +246,6 @@ asm_mmu_enable:
 
 .globl asm_mmu_disable
 asm_mmu_disable:
-	mrs	x0, sctlr_el1
-	bic	x0, x0, SCTLR_EL1_M
-	msr	sctlr_el1, x0
-	isb
-
 	/* Clean + invalidate the entire memory */
 	adrp	x0, __phys_offset
 	ldr	x0, [x0, :lo12:__phys_offset]
@@ -259,6 +254,22 @@ asm_mmu_disable:
 	sub	x1, x1, x0
 	dcache_by_line_op civac, sy, x0, x1, x2, x3
 
+	/* Invalidate Icache */
+	ic	iallu
+	isb
+
+	/* Disable cache, Icache and MMU */
+	mrs	x0, sctlr_el1
+	bic	x0, x0, SCTLR_EL1_C
+	bic	x0, x0, SCTLR_EL1_I
+	bic	x0, x0, SCTLR_EL1_M
+	msr	sctlr_el1, x0
+	isb
+
+	/* Invalidate TLB */
+	tlbi	vmalle1
+	dsb	nsh
+
 	ret
 
 /*
-- 
2.25.1

[kvm-unit-tests PATCH v3 16/27] arm/arm64: Rename etext to _etext

[Nikos Nikoleris]

From: Andrew Jones <drjones@redhat.com>

Rename etext to the more popular _etext allowing different linker
scripts to more easily be used.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 lib/arm/setup.c | 4 ++--
 arm/flat.lds    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/lib/arm/setup.c b/lib/arm/setup.c
index 3612991..13513d0 100644
--- a/lib/arm/setup.c
+++ b/lib/arm/setup.c
@@ -34,7 +34,7 @@
 #define NR_EXTRA_MEM_REGIONS	16
 #define NR_INITIAL_MEM_REGIONS	(MAX_DT_MEM_REGIONS + NR_EXTRA_MEM_REGIONS)
 
-extern unsigned long etext;
+extern unsigned long _etext;
 
 char *initrd;
 u32 initrd_size;
@@ -140,7 +140,7 @@ unsigned int mem_region_get_flags(phys_addr_t paddr)
 
 static void mem_regions_add_assumed(void)
 {
-	phys_addr_t code_end = (phys_addr_t)(unsigned long)&etext;
+	phys_addr_t code_end = (phys_addr_t)(unsigned long)&_etext;
 	struct mem_region *r;
 
 	r = mem_region_find(code_end - 1);
diff --git a/arm/flat.lds b/arm/flat.lds
index 47fcb64..9016ac9 100644
--- a/arm/flat.lds
+++ b/arm/flat.lds
@@ -27,7 +27,7 @@ SECTIONS
     PROVIDE(_text = .);
     .text : { *(.init) *(.text) *(.text.*) }
     . = ALIGN(64K);
-    PROVIDE(etext = .);
+    PROVIDE(_etext = .);
 
     PROVIDE(reloc_start = .);
     .rela.dyn : { *(.rela.dyn) }
-- 
2.25.1

[kvm-unit-tests PATCH v3 17/27] lib: Avoid ms_abi for calls related to EFI on arm64

[Nikos Nikoleris]

x86_64 requires that EFI calls use the ms_abi calling convention. For
arm64 this is unnecessary.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Andrew Jones <drjones@redhat.com>
---
 lib/linux/efi.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/linux/efi.h b/lib/linux/efi.h
index 9a1cf87..53748dd 100644
--- a/lib/linux/efi.h
+++ b/lib/linux/efi.h
@@ -33,7 +33,11 @@ typedef u16 efi_char16_t;		/* UNICODE character */
 typedef u64 efi_physical_addr_t;
 typedef void *efi_handle_t;
 
+#ifdef __x86_64__
 #define __efiapi __attribute__((ms_abi))
+#else
+#define __efiapi
+#endif
 
 /*
  * The UEFI spec and EDK2 reference implementation both define EFI_GUID as
-- 
2.25.1

[kvm-unit-tests PATCH v3 18/27] arm64: Add a new type of memory type flag MR_F_RESERVED

[Nikos Nikoleris]

From: Andrew Jones <drjones@redhat.com>

This will be used by future change to add PTE entries for special EFI
memory regions.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 lib/arm/asm/setup.h | 1 +
 lib/arm/mmu.c       | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/lib/arm/asm/setup.h b/lib/arm/asm/setup.h
index f0e70b1..64cd379 100644
--- a/lib/arm/asm/setup.h
+++ b/lib/arm/asm/setup.h
@@ -15,6 +15,7 @@ extern int nr_cpus;
 
 #define MR_F_IO			(1U << 0)
 #define MR_F_CODE		(1U << 1)
+#define MR_F_RESERVED		(1U << 2)
 #define MR_F_UNKNOWN		(1U << 31)
 
 struct mem_region {
diff --git a/lib/arm/mmu.c b/lib/arm/mmu.c
index e1a72fe..931be98 100644
--- a/lib/arm/mmu.c
+++ b/lib/arm/mmu.c
@@ -174,6 +174,10 @@ void *setup_mmu(phys_addr_t phys_end, void *unused)
 	for (r = mem_regions; r->end; ++r) {
 		if (r->flags & MR_F_IO) {
 			continue;
+		} else if (r->flags & MR_F_RESERVED) {
+			/* Reserved pages need to be writable for whatever reserved them */
+			mmu_set_range_ptes(mmu_idmap, r->start, r->start, r->end,
+					   __pgprot(PTE_WBWA));
 		} else if (r->flags & MR_F_CODE) {
 			/* armv8 requires code shared between EL1 and EL0 to be read-only */
 			mmu_set_range_ptes(mmu_idmap, r->start, r->start, r->end,
-- 
2.25.1

[kvm-unit-tests PATCH v3 19/27] arm/arm64: Add a setup sequence for systems that boot through EFI

[Nikos Nikoleris]

This change implements an alternative setup sequence for the system
when we are booting through EFI. The memory map is discovered through
EFI boot services and devices through ACPI.

This change is based on a change initially proposed by
Andrew Jones <drjones@redhat.com>

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/linux/efi.h     |   1 +
 lib/arm/asm/setup.h |   2 +
 lib/arm/setup.c     | 181 +++++++++++++++++++++++++++++++++++++++++++-
 arm/cstart.S        |   1 +
 arm/cstart64.S      |   1 +
 5 files changed, 184 insertions(+), 2 deletions(-)

diff --git a/lib/linux/efi.h b/lib/linux/efi.h
index 53748dd..89f9a9e 100644
--- a/lib/linux/efi.h
+++ b/lib/linux/efi.h
@@ -63,6 +63,7 @@ typedef guid_t efi_guid_t;
 	(c) & 0xff, ((c) >> 8) & 0xff, d } }
 
 #define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)
+#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3,  0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)
 
 #define LOADED_IMAGE_PROTOCOL_GUID EFI_GUID(0x5b1b31a1, 0x9562, 0x11d2,  0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b)
 
diff --git a/lib/arm/asm/setup.h b/lib/arm/asm/setup.h
index 64cd379..c4cd485 100644
--- a/lib/arm/asm/setup.h
+++ b/lib/arm/asm/setup.h
@@ -6,6 +6,7 @@
  * This work is licensed under the terms of the GNU LGPL, version 2.
  */
 #include <libcflat.h>
+#include <efi.h>
 #include <asm/page.h>
 #include <asm/pgtable-hwdef.h>
 
@@ -37,5 +38,6 @@ extern unsigned int mem_region_get_flags(phys_addr_t paddr);
 #define SMP_CACHE_BYTES		L1_CACHE_BYTES
 
 void setup(const void *fdt, phys_addr_t freemem_start);
+efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo);
 
 #endif /* _ASMARM_SETUP_H_ */
diff --git a/lib/arm/setup.c b/lib/arm/setup.c
index 13513d0..30d04d0 100644
--- a/lib/arm/setup.c
+++ b/lib/arm/setup.c
@@ -34,7 +34,7 @@
 #define NR_EXTRA_MEM_REGIONS	16
 #define NR_INITIAL_MEM_REGIONS	(MAX_DT_MEM_REGIONS + NR_EXTRA_MEM_REGIONS)
 
-extern unsigned long _etext;
+extern unsigned long _text, _etext, _data, _edata;
 
 char *initrd;
 u32 initrd_size;
@@ -44,7 +44,10 @@ int nr_cpus;
 
 static struct mem_region __initial_mem_regions[NR_INITIAL_MEM_REGIONS + 1];
 struct mem_region *mem_regions = __initial_mem_regions;
-phys_addr_t __phys_offset, __phys_end;
+phys_addr_t __phys_offset = (phys_addr_t)-1, __phys_end = 0;
+
+extern void exceptions_init(void);
+extern void asm_mmu_disable(void);
 
 int mpidr_to_cpu(uint64_t mpidr)
 {
@@ -272,3 +275,177 @@ void setup(const void *fdt, phys_addr_t freemem_start)
 	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
 		setup_vm();
 }
+
+#ifdef CONFIG_EFI
+
+#include <efi.h>
+
+static efi_status_t setup_rsdp(efi_bootinfo_t *efi_bootinfo)
+{
+	efi_status_t status;
+	struct rsdp_descriptor *rsdp;
+
+	/*
+	 * RSDP resides in an EFI_ACPI_RECLAIM_MEMORY region, which is not used
+	 * by kvm-unit-tests arm64 memory allocator. So it is not necessary to
+	 * copy the data structure to another memory region to prevent
+	 * unintentional overwrite.
+	 */
+	status = efi_get_system_config_table(ACPI_20_TABLE_GUID, (void **)&rsdp);
+	if (status != EFI_SUCCESS)
+		return status;
+
+	set_efi_rsdp(rsdp);
+
+	return EFI_SUCCESS;
+}
+
+static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
+{
+	int i;
+	unsigned long free_mem_pages = 0;
+	unsigned long free_mem_start = 0;
+	struct efi_boot_memmap *map = &(efi_bootinfo->mem_map);
+	efi_memory_desc_t *buffer = *map->map;
+	efi_memory_desc_t *d = NULL;
+	phys_addr_t base, top;
+	struct mem_region *r;
+	uintptr_t text = (uintptr_t)&_text, etext = __ALIGN((uintptr_t)&_etext, 4096);
+	uintptr_t data = (uintptr_t)&_data, edata = __ALIGN((uintptr_t)&_edata, 4096);
+
+	/*
+	 * Record the largest free EFI_CONVENTIONAL_MEMORY region
+	 * which will be used to set up the memory allocator, so that
+	 * the memory allocator can work in the largest free
+	 * continuous memory region.
+	 */
+	for (i = 0, r = &mem_regions[0]; i < *(map->map_size); i += *(map->desc_size), ++r) {
+		d = (efi_memory_desc_t *)(&((u8 *)buffer)[i]);
+
+		r->start = d->phys_addr;
+		r->end = d->phys_addr + d->num_pages * EFI_PAGE_SIZE;
+
+		switch (d->type) {
+		case EFI_RESERVED_TYPE:
+		case EFI_LOADER_DATA:
+		case EFI_BOOT_SERVICES_CODE:
+		case EFI_BOOT_SERVICES_DATA:
+		case EFI_RUNTIME_SERVICES_CODE:
+		case EFI_RUNTIME_SERVICES_DATA:
+		case EFI_UNUSABLE_MEMORY:
+		case EFI_ACPI_RECLAIM_MEMORY:
+		case EFI_ACPI_MEMORY_NVS:
+		case EFI_PAL_CODE:
+			r->flags = MR_F_RESERVED;
+			break;
+		case EFI_MEMORY_MAPPED_IO:
+		case EFI_MEMORY_MAPPED_IO_PORT_SPACE:
+			r->flags = MR_F_IO;
+			break;
+		case EFI_LOADER_CODE:
+			if (r->start <= text && r->end > text) {
+				/* This is the unit test region. Flag the code separately. */
+				phys_addr_t tmp = r->end;
+
+				assert(etext <= data);
+				assert(edata <= r->end);
+				r->flags = MR_F_CODE;
+				r->end = data;
+				++r;
+				r->start = data;
+				r->end = tmp;
+			} else {
+				r->flags = MR_F_RESERVED;
+			}
+			break;
+		case EFI_CONVENTIONAL_MEMORY:
+			if (free_mem_pages < d->num_pages) {
+				free_mem_pages = d->num_pages;
+				free_mem_start = d->phys_addr;
+			}
+			break;
+		}
+
+		if (!(r->flags & MR_F_IO)) {
+			if (r->start < __phys_offset)
+				__phys_offset = r->start;
+			if (r->end > __phys_end)
+				__phys_end = r->end;
+		}
+	}
+	__phys_end &= PHYS_MASK;
+	asm_mmu_disable();
+
+	if (free_mem_pages == 0)
+		return EFI_OUT_OF_RESOURCES;
+
+	assert(sizeof(long) == 8 || free_mem_start < (3ul << 30));
+
+	phys_alloc_init(free_mem_start, free_mem_pages << EFI_PAGE_SHIFT);
+	phys_alloc_set_minimum_alignment(SMP_CACHE_BYTES);
+
+	phys_alloc_get_unused(&base, &top);
+	base = PAGE_ALIGN(base);
+	top = top & PAGE_MASK;
+	assert(sizeof(long) == 8 || !(base >> 32));
+	if (sizeof(long) != 8 && (top >> 32) != 0)
+		top = ((uint64_t)1 << 32);
+	page_alloc_init_area(0, base >> PAGE_SHIFT, top >> PAGE_SHIFT);
+	page_alloc_ops_enable();
+
+	return EFI_SUCCESS;
+}
+
+efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo)
+{
+	efi_status_t status;
+
+	struct thread_info *ti = current_thread_info();
+
+	memset(ti, 0, sizeof(*ti));
+
+	exceptions_init();
+
+	status = efi_mem_init(efi_bootinfo);
+	if (status != EFI_SUCCESS) {
+		printf("Failed to initialize memory: ");
+		switch (status) {
+		case EFI_OUT_OF_RESOURCES:
+			printf("No free memory region\n");
+			break;
+		default:
+			printf("Unknown error\n");
+			break;
+		}
+		return status;
+	}
+
+	status = setup_rsdp(efi_bootinfo);
+	if (status != EFI_SUCCESS) {
+		printf("Cannot find RSDP in EFI system table\n");
+		return status;
+	}
+
+	psci_set_conduit();
+	cpu_init();
+	/* cpu_init must be called before thread_info_init */
+	thread_info_init(current_thread_info(), 0);
+	/* mem_init must be called before io_init */
+	io_init();
+
+	timer_save_state();
+	if (initrd) {
+		/* environ is currently the only file in the initrd */
+		char *env = malloc(initrd_size);
+
+		memcpy(env, initrd, initrd_size);
+		setup_env(env, initrd_size);
+	}
+
+	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
+		setup_vm();
+
+	return EFI_SUCCESS;
+}
+
+#endif
diff --git a/arm/cstart.S b/arm/cstart.S
index dc324c5..66a55b9 100644
--- a/arm/cstart.S
+++ b/arm/cstart.S
@@ -256,6 +256,7 @@ asm_mmu_disable:
  *
  * Input r0 is the stack top, which is the exception stacks base
  */
+.globl exceptions_init
 exceptions_init:
 	mrc	p15, 0, r2, c1, c0, 0	@ read SCTLR
 	bic	r2, #CR_V		@ SCTLR.V := 0
diff --git a/arm/cstart64.S b/arm/cstart64.S
index 390feb9..55b41ea 100644
--- a/arm/cstart64.S
+++ b/arm/cstart64.S
@@ -276,6 +276,7 @@ asm_mmu_disable:
  * Vectors
  */
 
+.globl exceptions_init
 exceptions_init:
 	adrp	x4, vector_table
 	add	x4, x4, :lo12:vector_table
-- 
2.25.1

[kvm-unit-tests PATCH v3 20/27] arm64: Copy code from GNU-EFI

[Nikos Nikoleris]

This change adds unmodified dependencies that we need from GNU-EFI in
order to build arm64 EFI apps.

GNU-EFI sources from  https://git.code.sf.net/p/gnu-efi/code v3.0.14

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 arm/efi/elf_aarch64_efi.lds |  63 +++++++++++++++++
 arm/efi/crt0-efi-aarch64.S  | 130 ++++++++++++++++++++++++++++++++++++
 arm/efi/reloc_aarch64.c     |  97 +++++++++++++++++++++++++++
 3 files changed, 290 insertions(+)
 create mode 100644 arm/efi/elf_aarch64_efi.lds
 create mode 100644 arm/efi/crt0-efi-aarch64.S
 create mode 100644 arm/efi/reloc_aarch64.c

diff --git a/arm/efi/elf_aarch64_efi.lds b/arm/efi/elf_aarch64_efi.lds
new file mode 100644
index 0000000..836d982
--- /dev/null
+++ b/arm/efi/elf_aarch64_efi.lds
@@ -0,0 +1,63 @@
+OUTPUT_FORMAT("elf64-littleaarch64", "elf64-littleaarch64", "elf64-littleaarch64")
+OUTPUT_ARCH(aarch64)
+ENTRY(_start)
+SECTIONS
+{
+  .text 0x0 : {
+    _text = .;
+    *(.text.head)
+    *(.text)
+    *(.text.*)
+    *(.gnu.linkonce.t.*)
+    *(.srodata)
+    *(.rodata*)
+    . = ALIGN(16);
+  }
+  _etext = .;
+  _text_size = . - _text;
+  .dynamic  : { *(.dynamic) }
+  .data : ALIGN(4096)
+  {
+   _data = .;
+   *(.sdata)
+   *(.data)
+   *(.data1)
+   *(.data.*)
+   *(.got.plt)
+   *(.got)
+
+   /* the EFI loader doesn't seem to like a .bss section, so we stick
+      it all into .data: */
+   . = ALIGN(16);
+   _bss = .;
+   *(.sbss)
+   *(.scommon)
+   *(.dynbss)
+   *(.bss)
+   *(COMMON)
+   . = ALIGN(16);
+   _bss_end = .;
+  }
+
+  .rela.dyn : { *(.rela.dyn) }
+  .rela.plt : { *(.rela.plt) }
+  .rela.got : { *(.rela.got) }
+  .rela.data : { *(.rela.data) *(.rela.data*) }
+  . = ALIGN(512);
+  _edata = .;
+  _data_size = . - _data;
+
+  . = ALIGN(4096);
+  .dynsym   : { *(.dynsym) }
+  . = ALIGN(4096);
+  .dynstr   : { *(.dynstr) }
+  . = ALIGN(4096);
+  .note.gnu.build-id : { *(.note.gnu.build-id) }
+  /DISCARD/ :
+  {
+    *(.rel.reloc)
+    *(.eh_frame)
+    *(.note.GNU-stack)
+  }
+  .comment 0 : { *(.comment) }
+}
diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
new file mode 100644
index 0000000..d50e78d
--- /dev/null
+++ b/arm/efi/crt0-efi-aarch64.S
@@ -0,0 +1,130 @@
+/*
+ * crt0-efi-aarch64.S - PE/COFF header for AArch64 EFI applications
+ *
+ * Copright (C) 2014 Linaro Ltd. <ard.biesheuvel@linaro.org>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice and this list of conditions, without modification.
+ * 2. The name of the author may not be used to endorse or promote products
+ *    derived from this software without specific prior written permission.
+ *
+ * Alternatively, this software may be distributed under the terms of the
+ * GNU General Public License as published by the Free Software Foundation;
+ * either version 2 of the License, or (at your option) any later version.
+ */
+
+	.section	.text.head
+
+	/*
+	 * Magic "MZ" signature for PE/COFF
+	 */
+	.globl	ImageBase
+ImageBase:
+	.ascii	"MZ"
+	.skip	58				// 'MZ' + pad + offset == 64
+	.long	pe_header - ImageBase		// Offset to the PE header.
+pe_header:
+	.ascii	"PE"
+	.short 	0
+coff_header:
+	.short	0xaa64				// AArch64
+	.short	2				// nr_sections
+	.long	0 				// TimeDateStamp
+	.long	0				// PointerToSymbolTable
+	.long	0				// NumberOfSymbols
+	.short	section_table - optional_header	// SizeOfOptionalHeader
+	.short	0x206				// Characteristics.
+						// IMAGE_FILE_DEBUG_STRIPPED |
+						// IMAGE_FILE_EXECUTABLE_IMAGE |
+						// IMAGE_FILE_LINE_NUMS_STRIPPED
+optional_header:
+	.short	0x20b				// PE32+ format
+	.byte	0x02				// MajorLinkerVersion
+	.byte	0x14				// MinorLinkerVersion
+	.long	_data - _start			// SizeOfCode
+	.long	_data_size			// SizeOfInitializedData
+	.long	0				// SizeOfUninitializedData
+	.long	_start - ImageBase		// AddressOfEntryPoint
+	.long	_start - ImageBase		// BaseOfCode
+
+extra_header_fields:
+	.quad	0				// ImageBase
+	.long	0x1000				// SectionAlignment
+	.long	0x200				// FileAlignment
+	.short	0				// MajorOperatingSystemVersion
+	.short	0				// MinorOperatingSystemVersion
+	.short	0				// MajorImageVersion
+	.short	0				// MinorImageVersion
+	.short	0				// MajorSubsystemVersion
+	.short	0				// MinorSubsystemVersion
+	.long	0				// Win32VersionValue
+
+	.long	_edata - ImageBase		// SizeOfImage
+
+	// Everything before the kernel image is considered part of the header
+	.long	_start - ImageBase		// SizeOfHeaders
+	.long	0				// CheckSum
+	.short	EFI_SUBSYSTEM			// Subsystem
+	.short	0				// DllCharacteristics
+	.quad	0				// SizeOfStackReserve
+	.quad	0				// SizeOfStackCommit
+	.quad	0				// SizeOfHeapReserve
+	.quad	0				// SizeOfHeapCommit
+	.long	0				// LoaderFlags
+	.long	0x6				// NumberOfRvaAndSizes
+
+	.quad	0				// ExportTable
+	.quad	0				// ImportTable
+	.quad	0				// ResourceTable
+	.quad	0				// ExceptionTable
+	.quad	0				// CertificationTable
+	.quad	0				// BaseRelocationTable
+
+	// Section table
+section_table:
+	.ascii	".text\0\0\0"
+	.long	_data - _start		// VirtualSize
+	.long	_start - ImageBase	// VirtualAddress
+	.long	_data - _start		// SizeOfRawData
+	.long	_start - ImageBase	// PointerToRawData
+
+	.long	0		// PointerToRelocations (0 for executables)
+	.long	0		// PointerToLineNumbers (0 for executables)
+	.short	0		// NumberOfRelocations  (0 for executables)
+	.short	0		// NumberOfLineNumbers  (0 for executables)
+	.long	0x60000020	// Characteristics (section flags)
+
+	.ascii	".data\0\0\0"
+	.long	_data_size		// VirtualSize
+	.long	_data - ImageBase	// VirtualAddress
+	.long	_data_size		// SizeOfRawData
+	.long	_data - ImageBase	// PointerToRawData
+
+	.long	0		// PointerToRelocations (0 for executables)
+	.long	0		// PointerToLineNumbers (0 for executables)
+	.short	0		// NumberOfRelocations  (0 for executables)
+	.short	0		// NumberOfLineNumbers  (0 for executables)
+	.long	0xc0000040	// Characteristics (section flags)
+
+	.align		12
+_start:
+	stp		x29, x30, [sp, #-32]!
+	mov		x29, sp
+
+	stp		x0, x1, [sp, #16]
+	mov		x2, x0
+	mov		x3, x1
+	adr		x0, ImageBase
+	adrp		x1, _DYNAMIC
+	add		x1, x1, #:lo12:_DYNAMIC
+	bl		_relocate
+	cbnz		x0, 0f
+
+	ldp		x0, x1, [sp, #16]
+	bl		efi_main
+
+0:	ldp		x29, x30, [sp], #32
+	ret
diff --git a/arm/efi/reloc_aarch64.c b/arm/efi/reloc_aarch64.c
new file mode 100644
index 0000000..0867279
--- /dev/null
+++ b/arm/efi/reloc_aarch64.c
@@ -0,0 +1,97 @@
+/* reloc_aarch64.c - position independent x86 ELF shared object relocator
+   Copyright (C) 2014 Linaro Ltd. <ard.biesheuvel@linaro.org>
+   Copyright (C) 1999 Hewlett-Packard Co.
+	Contributed by David Mosberger <davidm@hpl.hp.com>.
+
+    All rights reserved.
+
+    Redistribution and use in source and binary forms, with or without
+    modification, are permitted provided that the following conditions
+    are met:
+
+    * Redistributions of source code must retain the above copyright
+      notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above
+      copyright notice, this list of conditions and the following
+      disclaimer in the documentation and/or other materials
+      provided with the distribution.
+    * Neither the name of Hewlett-Packard Co. nor the names of its
+      contributors may be used to endorse or promote products derived
+      from this software without specific prior written permission.
+
+    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
+    CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
+    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+    MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+    DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS
+    BE LIABLE FOR ANYDIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
+    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+    PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+    PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+    THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR
+    TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
+    THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+    SUCH DAMAGE.
+*/
+
+#include <efi.h>
+#include <efilib.h>
+
+#include <elf.h>
+
+EFI_STATUS _relocate (long ldbase, Elf64_Dyn *dyn,
+		      EFI_HANDLE image EFI_UNUSED,
+		      EFI_SYSTEM_TABLE *systab EFI_UNUSED)
+{
+	long relsz = 0, relent = 0;
+	Elf64_Rela *rel = 0;
+	unsigned long *addr;
+	int i;
+
+	for (i = 0; dyn[i].d_tag != DT_NULL; ++i) {
+		switch (dyn[i].d_tag) {
+			case DT_RELA:
+				rel = (Elf64_Rela*)
+					((unsigned long)dyn[i].d_un.d_ptr
+					 + ldbase);
+				break;
+
+			case DT_RELASZ:
+				relsz = dyn[i].d_un.d_val;
+				break;
+
+			case DT_RELAENT:
+				relent = dyn[i].d_un.d_val;
+				break;
+
+			default:
+				break;
+		}
+	}
+
+	if (!rel && relent == 0)
+		return EFI_SUCCESS;
+
+	if (!rel || relent == 0)
+		return EFI_LOAD_ERROR;
+
+	while (relsz > 0) {
+		/* apply the relocs */
+		switch (ELF64_R_TYPE (rel->r_info)) {
+			case R_AARCH64_NONE:
+				break;
+
+			case R_AARCH64_RELATIVE:
+				addr = (unsigned long *)
+					(ldbase + rel->r_offset);
+				*addr = ldbase + rel->r_addend;
+				break;
+
+			default:
+				break;
+		}
+		rel = (Elf64_Rela*) ((char *) rel + relent);
+		relsz -= relent;
+	}
+	return EFI_SUCCESS;
+}
-- 
2.25.1

[kvm-unit-tests PATCH v3 21/27] arm64: Change GNU-EFI imported file to use defined types

[Nikos Nikoleris]

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 arm/efi/reloc_aarch64.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/arm/efi/reloc_aarch64.c b/arm/efi/reloc_aarch64.c
index 0867279..fa0cd6b 100644
--- a/arm/efi/reloc_aarch64.c
+++ b/arm/efi/reloc_aarch64.c
@@ -34,14 +34,11 @@
     SUCH DAMAGE.
 */
 
-#include <efi.h>
-#include <efilib.h>
-
+#include "efi.h"
 #include <elf.h>
 
-EFI_STATUS _relocate (long ldbase, Elf64_Dyn *dyn,
-		      EFI_HANDLE image EFI_UNUSED,
-		      EFI_SYSTEM_TABLE *systab EFI_UNUSED)
+efi_status_t _relocate(long ldbase, Elf64_Dyn *dyn, efi_handle_t image,
+		       efi_system_table_t *sys_tab)
 {
 	long relsz = 0, relent = 0;
 	Elf64_Rela *rel = 0;
-- 
2.25.1

[kvm-unit-tests PATCH v3 22/27] arm64: Use code from the gnu-efi when booting with EFI

[Nikos Nikoleris]

arm/efi/crt0-efi-aarch64.S defines the header and the handover
sequence from EFI to a efi_main. This change includes the whole file
in arm/cstart64.S when we compile with EFI support.

In addition, we change the handover code in arm/efi/crt0-efi-aarch64.S
to align the stack pointer. This alignment is necessary because we
make assumptions about cpu0's stack alignment and most importantly we
place its thread_info at the bottom of this stack.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 arm/cstart64.S             |  6 ++++++
 arm/efi/crt0-efi-aarch64.S | 21 +++++++++++++++++----
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/arm/cstart64.S b/arm/cstart64.S
index 55b41ea..08cf02f 100644
--- a/arm/cstart64.S
+++ b/arm/cstart64.S
@@ -15,6 +15,10 @@
 #include <asm/thread_info.h>
 #include <asm/sysreg.h>
 
+#ifdef CONFIG_EFI
+#include "efi/crt0-efi-aarch64.S"
+#else
+
 .macro zero_range, tmp1, tmp2
 9998:	cmp	\tmp1, \tmp2
 	b.eq	9997f
@@ -107,6 +111,8 @@ start:
 	bl	exit
 	b	halt
 
+#endif
+
 .text
 
 /*
diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
index d50e78d..03d29b0 100644
--- a/arm/efi/crt0-efi-aarch64.S
+++ b/arm/efi/crt0-efi-aarch64.S
@@ -111,10 +111,19 @@ section_table:
 
 	.align		12
 _start:
-	stp		x29, x30, [sp, #-32]!
+	stp		x29, x30, [sp, #-16]!
+
+	/* Align sp; this is necessary due to way we store cpu0's thread_info */
 	mov		x29, sp
+	and		x29, x29, #THREAD_MASK
+	mov		x30, sp
+	mov		sp, x29
+	str		x30, [sp, #-16]!
+
+	mov             x29, sp
+
+	stp		x0, x1, [sp, #-16]!
 
-	stp		x0, x1, [sp, #16]
 	mov		x2, x0
 	mov		x3, x1
 	adr		x0, ImageBase
@@ -123,8 +132,12 @@ _start:
 	bl		_relocate
 	cbnz		x0, 0f
 
-	ldp		x0, x1, [sp, #16]
+	ldp		x0, x1, [sp], #16
 	bl		efi_main
 
-0:	ldp		x29, x30, [sp], #32
+	/* Restore sp */
+	ldr		x30, [sp], #16
+	mov             sp, x30
+
+0:	ldp		x29, x30, [sp], #16
 	ret
-- 
2.25.1

[kvm-unit-tests PATCH v3 23/27] lib: Avoid external dependency in libelf

[Nikos Nikoleris]

There is just a small number of definitions we need from
uapi/linux/elf.h and asm/elf.h and the relocation code is
self-contained.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 lib/elf.h               | 57 +++++++++++++++++++++++++++++++++++++++++
 arm/efi/reloc_aarch64.c |  3 +--
 2 files changed, 58 insertions(+), 2 deletions(-)
 create mode 100644 lib/elf.h

diff --git a/lib/elf.h b/lib/elf.h
new file mode 100644
index 0000000..abd5cf4
--- /dev/null
+++ b/lib/elf.h
@@ -0,0 +1,57 @@
+/* SPDX-License-Identifier: LGPL-2.0-or-later */
+/*
+ * Relevant definitions from uapi/linux/elf.h and asm/elf.h
+ */
+
+#ifndef _ELF_H_
+#define _ELF_H_
+
+#include <libcflat.h>
+
+/* 64-bit ELF base types. */
+typedef u64	Elf64_Addr;
+typedef u64	Elf64_Xword;
+typedef s64	Elf64_Sxword;
+
+typedef struct {
+	Elf64_Sxword d_tag;             /* entry tag value */
+	union {
+		Elf64_Xword d_val;
+		Elf64_Addr d_ptr;
+	} d_un;
+} Elf64_Dyn;
+
+typedef struct elf64_rel {
+	Elf64_Addr r_offset;    /* Location at which to apply the action */
+	Elf64_Xword r_info;     /* index and type of relocation */
+} Elf64_Rel;
+
+typedef struct elf64_rela {
+	Elf64_Addr r_offset;    /* Location at which to apply the action */
+	Elf64_Xword r_info;     /* index and type of relocation */
+	Elf64_Sxword r_addend;  /* Constant addend used to compute value */
+} Elf64_Rela;
+
+/* This is the info that is needed to parse the dynamic section of the file */
+#define DT_NULL		0
+#define DT_RELA		7
+#define DT_RELASZ	8
+#define DT_RELAENT	9
+
+/* x86 relocation types. */
+#define R_X86_64_NONE		0       /* No reloc */
+#define R_X86_64_RELATIVE	8       /* Adjust by program base */
+
+
+/*
+ * AArch64 static relocation types.
+ */
+
+/* Miscellaneous. */
+#define R_AARCH64_NONE		256
+#define R_AARCH64_RELATIVE	1027
+
+/* The following are used with relocations */
+#define ELF64_R_TYPE(i)		((i) & 0xffffffff)
+
+#endif /* _ELF_H_ */
diff --git a/arm/efi/reloc_aarch64.c b/arm/efi/reloc_aarch64.c
index fa0cd6b..3f6d9a6 100644
--- a/arm/efi/reloc_aarch64.c
+++ b/arm/efi/reloc_aarch64.c
@@ -34,8 +34,7 @@
     SUCH DAMAGE.
 */
 
-#include "efi.h"
-#include <elf.h>
+#include <efi.h>
 
 efi_status_t _relocate(long ldbase, Elf64_Dyn *dyn, efi_handle_t image,
 		       efi_system_table_t *sys_tab)
-- 
2.25.1

[kvm-unit-tests PATCH v3 24/27] x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile

[Nikos Nikoleris]

Compiler flag -macculate-outgoing-args is only needed by the x86_64
ABI. Move it to the relevant Makefile.

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 Makefile            | 4 ----
 x86/Makefile.x86_64 | 4 ++++
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/Makefile b/Makefile
index 6ed5dea..307bc29 100644
--- a/Makefile
+++ b/Makefile
@@ -40,14 +40,10 @@ OBJDIRS += $(LIBFDT_objdir)
 
 # EFI App
 ifeq ($(CONFIG_EFI),y)
-EFI_ARCH = x86_64
 EFI_CFLAGS := -DCONFIG_EFI
 # The following CFLAGS and LDFLAGS come from:
 #   - GNU-EFI/Makefile.defaults
 #   - GNU-EFI/apps/Makefile
-# Function calls must include the number of arguments passed to the functions
-# More details: https://wiki.osdev.org/GNU-EFI
-EFI_CFLAGS += -maccumulate-outgoing-args
 # GCC defines wchar to be 32 bits, but EFI expects 16 bits
 EFI_CFLAGS += -fshort-wchar
 # EFI applications use PIC as they are loaded to dynamic addresses, not a fixed
diff --git a/x86/Makefile.x86_64 b/x86/Makefile.x86_64
index e19284a..c0ffac4 100644
--- a/x86/Makefile.x86_64
+++ b/x86/Makefile.x86_64
@@ -2,6 +2,10 @@ cstart.o = $(TEST_DIR)/cstart64.o
 bits = 64
 ldarch = elf64-x86-64
 ifeq ($(CONFIG_EFI),y)
+# Function calls must include the number of arguments passed to the functions
+# More details: https://wiki.osdev.org/GNU-EFI
+CFLAGS += -maccumulate-outgoing-args
+
 exe = efi
 bin = so
 FORMAT = efi-app-x86_64
-- 
2.25.1

[kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Nikos Nikoleris]

Users can now build kvm-unit-tests as efi apps by supplying an extra
argument when invoking configure:

$> ./configure --enable-efi

This patch is based on an earlier version by
Andrew Jones <drjones@redhat.com>

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 configure           | 15 ++++++++++++---
 arm/Makefile.arm    |  6 ++++++
 arm/Makefile.arm64  | 18 ++++++++++++++----
 arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
 4 files changed, 66 insertions(+), 18 deletions(-)

diff --git a/configure b/configure
index 5b7daac..2ff9881 100755
--- a/configure
+++ b/configure
@@ -196,14 +196,19 @@ else
     fi
 fi
 
-if [ "$efi" ] && [ "$arch" != "x86_64" ]; then
+if [ "$efi" ] && [ "$arch" != "x86_64" ] && [ "$arch" != "arm64" ]; then
     echo "--[enable|disable]-efi is not supported for $arch"
     usage
 fi
 
 if [ -z "$page_size" ]; then
-    [ "$arch" = "arm64" ] && page_size="65536"
-    [ "$arch" = "arm" ] && page_size="4096"
+    if [ "$efi" = 'y' ] && [ "$arch" = "arm64" ]; then
+        page_size="4096"
+    elif [ "$arch" = "arm64" ]; then
+        page_size="65536"
+    elif [ "$arch" = "arm" ]; then
+        page_size="4096"
+    fi
 else
     if [ "$arch" != "arm64" ]; then
         echo "--page-size is not supported for $arch"
@@ -218,6 +223,10 @@ else
         echo "arm64 doesn't support page size of $page_size"
         usage
     fi
+    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
+        echo "efi must use 4K pages"
+        exit 1
+    fi
 fi
 
 [ -z "$processor" ] && processor="$arch"
diff --git a/arm/Makefile.arm b/arm/Makefile.arm
index 01fd4c7..2ce00f5 100644
--- a/arm/Makefile.arm
+++ b/arm/Makefile.arm
@@ -7,6 +7,10 @@ bits = 32
 ldarch = elf32-littlearm
 machine = -marm -mfpu=vfp
 
+ifeq ($(CONFIG_EFI),y)
+$(error Cannot build arm32 tests as EFI apps)
+endif
+
 # stack.o relies on frame pointers.
 KEEP_FRAME_POINTER := y
 
@@ -32,6 +36,8 @@ cflatobjs += lib/arm/stack.o
 cflatobjs += lib/ldiv32.o
 cflatobjs += lib/arm/ldivmod.o
 
+exe = flat
+
 # arm specific tests
 tests =
 
diff --git a/arm/Makefile.arm64 b/arm/Makefile.arm64
index 42e18e7..fd3d681 100644
--- a/arm/Makefile.arm64
+++ b/arm/Makefile.arm64
@@ -27,11 +27,21 @@ cflatobjs += lib/arm64/gic-v3-its.o lib/arm64/gic-v3-its-cmd.o
 
 OBJDIRS += lib/arm64
 
+ifeq ($(CONFIG_EFI),y)
+# avoid jump tables before all relocations have been processed
+arm/efi/reloc_aarch64.o: CFLAGS += -fno-jump-tables
+cflatobjs += arm/efi/reloc_aarch64.o
+
+exe = efi
+else
+exe = flat
+endif
+
 # arm64 specific tests
-tests = $(TEST_DIR)/timer.flat
-tests += $(TEST_DIR)/micro-bench.flat
-tests += $(TEST_DIR)/cache.flat
-tests += $(TEST_DIR)/debug.flat
+tests = $(TEST_DIR)/timer.$(exe)
+tests += $(TEST_DIR)/micro-bench.$(exe)
+tests += $(TEST_DIR)/cache.$(exe)
+tests += $(TEST_DIR)/debug.$(exe)
 
 include $(SRCDIR)/$(TEST_DIR)/Makefile.common
 
diff --git a/arm/Makefile.common b/arm/Makefile.common
index 5be42c0..a8007f4 100644
--- a/arm/Makefile.common
+++ b/arm/Makefile.common
@@ -4,14 +4,14 @@
 # Authors: Andrew Jones <drjones@redhat.com>
 #
 
-tests-common  = $(TEST_DIR)/selftest.flat
-tests-common += $(TEST_DIR)/spinlock-test.flat
-tests-common += $(TEST_DIR)/pci-test.flat
-tests-common += $(TEST_DIR)/pmu.flat
-tests-common += $(TEST_DIR)/gic.flat
-tests-common += $(TEST_DIR)/psci.flat
-tests-common += $(TEST_DIR)/sieve.flat
-tests-common += $(TEST_DIR)/pl031.flat
+tests-common  = $(TEST_DIR)/selftest.$(exe)
+tests-common += $(TEST_DIR)/spinlock-test.$(exe)
+tests-common += $(TEST_DIR)/pci-test.$(exe)
+tests-common += $(TEST_DIR)/pmu.$(exe)
+tests-common += $(TEST_DIR)/gic.$(exe)
+tests-common += $(TEST_DIR)/psci.$(exe)
+tests-common += $(TEST_DIR)/sieve.$(exe)
+tests-common += $(TEST_DIR)/pl031.$(exe)
 
 tests-all = $(tests-common) $(tests)
 all: directories $(tests-all)
@@ -54,6 +54,9 @@ cflatobjs += lib/arm/smp.o
 cflatobjs += lib/arm/delay.o
 cflatobjs += lib/arm/gic.o lib/arm/gic-v2.o lib/arm/gic-v3.o
 cflatobjs += lib/arm/timer.o
+ifeq ($(CONFIG_EFI),y)
+cflatobjs += lib/efi.o
+endif
 
 OBJDIRS += lib/arm
 
@@ -61,6 +64,25 @@ libeabi = lib/arm/libeabi.a
 eabiobjs = lib/arm/eabi_compat.o
 
 FLATLIBS = $(libcflat) $(LIBFDT_archive) $(libeabi)
+
+ifeq ($(CONFIG_EFI),y)
+%.so: EFI_LDFLAGS += -defsym=EFI_SUBSYSTEM=0xa --no-undefined
+%.so: %.o $(FLATLIBS) $(SRCDIR)/arm/efi/elf_aarch64_efi.lds $(cstart.o)
+	$(CC) $(CFLAGS) -c -o $(@:.so=.aux.o) $(SRCDIR)/lib/auxinfo.c \
+		-DPROGNAME=\"$(@:.so=.efi)\" -DAUXFLAGS=$(AUXFLAGS)
+	$(LD) $(EFI_LDFLAGS) -o $@ -T $(SRCDIR)/arm/efi/elf_aarch64_efi.lds \
+		$(filter %.o, $^) $(FLATLIBS) $(@:.so=.aux.o) \
+		$(EFI_LIBS)
+	$(RM) $(@:.so=.aux.o)
+
+%.efi: %.so
+	$(call arch_elf_check, $^)
+	$(OBJCOPY) \
+		-j .text -j .sdata -j .data -j .dynamic -j .dynsym \
+		-j .rel -j .rela -j .rel.* -j .rela.* -j .rel* -j .rela* \
+		-j .reloc \
+		-O binary $^ $@
+else
 %.elf: LDFLAGS = -nostdlib $(arch_LDFLAGS)
 %.elf: %.o $(FLATLIBS) $(SRCDIR)/arm/flat.lds $(cstart.o)
 	$(CC) $(CFLAGS) -c -o $(@:.elf=.aux.o) $(SRCDIR)/lib/auxinfo.c \
@@ -74,13 +96,14 @@ FLATLIBS = $(libcflat) $(LIBFDT_archive) $(libeabi)
 	$(call arch_elf_check, $^)
 	$(OBJCOPY) -O binary $^ $@
 	@chmod a-x $@
+endif
 
 $(libeabi): $(eabiobjs)
 	$(AR) rcs $@ $^
 
 arm_clean: asm_offsets_clean
-	$(RM) $(TEST_DIR)/*.{o,flat,elf} $(libeabi) $(eabiobjs) \
-	      $(TEST_DIR)/.*.d lib/arm/.*.d
+	$(RM) $(TEST_DIR)/*.{o,flat,elf,so,efi} $(libeabi) $(eabiobjs) \
+	      $(TEST_DIR)/.*.d $(TEST_DIR)/efi/.*.d lib/arm/.*.d
 
 generated-files = $(asm-offsets)
-$(tests-all:.flat=.o) $(cstart.o) $(cflatobjs): $(generated-files)
+$(tests-all:.$(exe)=.o) $(cstart.o) $(cflatobjs): $(generated-files)
-- 
2.25.1

[kvm-unit-tests PATCH v3 26/27] lib: arm: Print test exit status

[Nikos Nikoleris]

From: Alexandru Elisei <alexandru.elisei@arm.com>

The arm tests can be run under kvmtool, which doesn't emulate a chr-testdev
device. Print the test exit status to make it possible for the runner
scripts to pick it up when they have support for it.

Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
---
 lib/arm/io.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/lib/arm/io.c b/lib/arm/io.c
index a91f116..337cf1b 100644
--- a/lib/arm/io.c
+++ b/lib/arm/io.c
@@ -138,6 +138,12 @@ extern void halt(int code);
 
 void exit(int code)
 {
+	/*
+	 * Print the test return code in the format used by chr-testdev so the
+	 * runner can pick it up if there is chr-testdev is not present.
+	 */
+	printf("\nEXIT: STATUS=%d\n", ((code) << 1) | 1);
+
 	chr_testdev_exit(code);
 	psci_system_off();
 	halt(code);
-- 
2.25.1

[kvm-unit-tests PATCH v3 27/27] arm64: Add an efi/run script

[Nikos Nikoleris]

This change adds a efi/run script inspired by the one in x86. This
script will setup a folder with the test compiled as an EFI app and a
startup.nsh script. The script launches QEMU providing an image with
EDKII and the path to the folder with the test which is executed
automatically.

For example:

$> ./arm/efi/run ./arm/selftest.efi setup smp=2 mem=256

Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
Reviewed-by: Ricardo Koller <ricarkol@google.com>
---
 scripts/runtime.bash | 14 +++++-----
 arm/efi/run          | 61 ++++++++++++++++++++++++++++++++++++++++++++
 arm/run              | 14 +++++++---
 arm/Makefile.common  |  1 +
 arm/dummy.c          |  4 +++
 5 files changed, 82 insertions(+), 12 deletions(-)
 create mode 100755 arm/efi/run
 create mode 100644 arm/dummy.c

diff --git a/scripts/runtime.bash b/scripts/runtime.bash
index 7d0180b..dc28f24 100644
--- a/scripts/runtime.bash
+++ b/scripts/runtime.bash
@@ -131,14 +131,12 @@ function run()
     fi
 
     last_line=$(premature_failure > >(tail -1)) && {
-        skip=true
-        if [ "${CONFIG_EFI}" == "y" ] && [[ "${last_line}" =~ "enabling apic" ]]; then
-            skip=false
-        fi
-        if [ ${skip} == true ]; then
-            print_result "SKIP" $testname "" "$last_line"
-            return 77
-        fi
+        if [ "${CONFIG_EFI}" == "y" ] && [ "${ARCH}" = x86_64 ]; then
+		if ! [[ "${last_line}" =~ "enabling apic" ]]; then
+			print_result "SKIP" $testname "" "$last_line"
+			return 77
+		fi
+	fi
     }
 
     cmdline=$(get_cmdline $kernel)
diff --git a/arm/efi/run b/arm/efi/run
new file mode 100755
index 0000000..dfff717
--- /dev/null
+++ b/arm/efi/run
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+set -e
+
+if [ $# -eq 0 ]; then
+	echo "Usage $0 TEST_CASE [QEMU_ARGS]"
+	exit 2
+fi
+
+if [ ! -f config.mak ]; then
+	echo "run './configure --enable-efi && make' first. See ./configure -h"
+	exit 2
+fi
+source config.mak
+source scripts/arch-run.bash
+source scripts/common.bash
+
+: "${EFI_SRC:=$(realpath "$(dirname "$0")/../")}"
+: "${EFI_UEFI:=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd}"
+: "${EFI_TEST:=efi-tests}"
+: "${EFI_CASE:=$(basename $1 .efi)}"
+
+if [ ! -f "$EFI_UEFI" ]; then
+	echo "UEFI firmware not found: $EFI_UEFI"
+	echo "Please install the UEFI firmware to this path"
+	echo "Or specify the correct path with the env variable EFI_UEFI"
+	exit 2
+fi
+
+# Remove the TEST_CASE from $@
+shift 1
+
+# Fish out the arguments for the test, they should be the next string
+# after the "-append" option
+qemu_args=()
+cmd_args=()
+while (( "$#" )); do
+	if [ "$1" = "-append" ]; then
+		cmd_args=$2
+		shift 2
+	else
+		qemu_args+=("$1")
+		shift 1
+	fi
+done
+
+if [ "$EFI_CASE" = "_NO_FILE_4Uhere_" ]; then
+	EFI_CASE=dummy
+fi
+
+: "${EFI_CASE_DIR:="$EFI_TEST/$EFI_CASE"}"
+mkdir -p "$EFI_CASE_DIR"
+
+cp "$EFI_SRC/$EFI_CASE.efi" "$EFI_TEST/$EFI_CASE/"
+echo "@echo -off" > "$EFI_TEST/$EFI_CASE/startup.nsh"
+echo "$EFI_CASE.efi" "${cmd_args[@]}" >> "$EFI_TEST/$EFI_CASE/startup.nsh"
+
+EFI_RUN=y $TEST_DIR/run \
+       -bios "$EFI_UEFI" \
+       -drive file.dir="$EFI_TEST/$EFI_CASE/",file.driver=vvfat,file.rw=on,format=raw,if=virtio \
+       "${qemu_args[@]}"
diff --git a/arm/run b/arm/run
index 1284891..62f845b 100755
--- a/arm/run
+++ b/arm/run
@@ -65,8 +65,10 @@ if $qemu $M -chardev testdev,id=id -initrd . 2>&1 \
 	exit 2
 fi
 
-chr_testdev='-device virtio-serial-device'
-chr_testdev+=' -device virtconsole,chardev=ctd -chardev testdev,id=ctd'
+if [ "$EFI_RUN" != "y" ]; then
+	chr_testdev='-device virtio-serial-device'
+	chr_testdev+=' -device virtconsole,chardev=ctd -chardev testdev,id=ctd'
+fi
 
 pci_testdev=
 if $qemu $M -device '?' 2>&1 | grep pci-testdev > /dev/null; then
@@ -75,7 +77,11 @@ fi
 
 A="-accel $ACCEL"
 command="$qemu -nodefaults $M $A -cpu $processor $chr_testdev $pci_testdev"
-command+=" -display none -serial stdio -kernel"
+command+=" -display none -serial stdio"
 command="$(migration_cmd) $(timeout_cmd) $command"
 
-run_qemu $command "$@"
+if [ "$EFI_RUN" = "y" ]; then
+	ENVIRON_DEFAULT=n run_qemu_status $command "$@"
+else
+	run_qemu $command -kernel "$@"
+fi
diff --git a/arm/Makefile.common b/arm/Makefile.common
index a8007f4..aabd335 100644
--- a/arm/Makefile.common
+++ b/arm/Makefile.common
@@ -12,6 +12,7 @@ tests-common += $(TEST_DIR)/gic.$(exe)
 tests-common += $(TEST_DIR)/psci.$(exe)
 tests-common += $(TEST_DIR)/sieve.$(exe)
 tests-common += $(TEST_DIR)/pl031.$(exe)
+tests-common += $(TEST_DIR)/dummy.$(exe)
 
 tests-all = $(tests-common) $(tests)
 all: directories $(tests-all)
diff --git a/arm/dummy.c b/arm/dummy.c
new file mode 100644
index 0000000..5019e79
--- /dev/null
+++ b/arm/dummy.c
@@ -0,0 +1,4 @@
+int main(int argc, char **argv)
+{
+	return 0;
+}
-- 
2.25.1

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi,

On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> From: Andrew Jones <drjones@redhat.com>
> 
> The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> clean + invalidate after turning MMU off") justifies cleaning and
> invalidating the dcache after disabling the MMU by saying it's nice
> not to rely on the current page tables and that it should still work
> (per the spec), as long as there's an identity map in the current
> tables. Doing the invalidation after also somewhat helped with
> reenabling the MMU without seeing stale data, but the real problem
> with reenabling was because the cache needs to be disabled with
> the MMU, but it wasn't.
> 
> Since we have to trust/validate that the current page tables have an
> identity map anyway, then there's no harm in doing the clean
> and invalidate first (it feels a little better to do so, anyway,
> considering the cache maintenance instructions take virtual
> addresses). Then, also disable the cache with the MMU to avoid
> problems when reenabling. We invalidate the Icache and disable
> that too for good measure. And, a final TLB invalidation ensures
> we're crystal clean when we return from asm_mmu_disable().

I'll point you to my previous reply [1] to this exact patch which explains
why it's incorrect and is only papering over another problem.

[1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/

Thanks,
Alex

> 
> Cc: Alexandru Elisei <alexandru.elisei@arm.com>
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  arm/cstart.S   | 28 +++++++++++++++++++++-------
>  arm/cstart64.S | 21 ++++++++++++++++-----
>  2 files changed, 37 insertions(+), 12 deletions(-)
> 
> diff --git a/arm/cstart.S b/arm/cstart.S
> index 7036e67..dc324c5 100644
> --- a/arm/cstart.S
> +++ b/arm/cstart.S
> @@ -179,6 +179,7 @@ halt:
>  .globl asm_mmu_enable
>  asm_mmu_enable:
>  	/* TLBIALL */
> +	mov	r2, #0
>  	mcr	p15, 0, r2, c8, c7, 0
>  	dsb	nsh
>  
> @@ -211,12 +212,7 @@ asm_mmu_enable:
>  
>  .globl asm_mmu_disable
>  asm_mmu_disable:
> -	/* SCTLR */
> -	mrc	p15, 0, r0, c1, c0, 0
> -	bic	r0, #CR_M
> -	mcr	p15, 0, r0, c1, c0, 0
> -	isb
> -
> +	/* Clean + invalidate the entire memory */
>  	ldr	r0, =__phys_offset
>  	ldr	r0, [r0]
>  	ldr	r1, =__phys_end
> @@ -224,7 +220,25 @@ asm_mmu_disable:
>  	sub	r1, r1, r0
>  	dcache_by_line_op dccimvac, sy, r0, r1, r2, r3
>  
> -	mov     pc, lr
> +	/* Invalidate Icache */
> +	mov	r0, #0
> +	mcr	p15, 0, r0, c7, c5, 0
> +	isb
> +
> +	/*  Disable cache, Icache and MMU */
> +	mrc	p15, 0, r0, c1, c0, 0
> +	bic	r0, #CR_C
> +	bic	r0, #CR_I
> +	bic	r0, #CR_M
> +	mcr	p15, 0, r0, c1, c0, 0
> +	isb
> +
> +	/* Invalidate TLB */
> +	mov	r0, #0
> +	mcr	p15, 0, r0, c8, c7, 0
> +	dsb	nsh
> +
> +	mov	pc, lr
>  
>  /*
>   * Vectors
> diff --git a/arm/cstart64.S b/arm/cstart64.S
> index e4ab7d0..390feb9 100644
> --- a/arm/cstart64.S
> +++ b/arm/cstart64.S
> @@ -246,11 +246,6 @@ asm_mmu_enable:
>  
>  .globl asm_mmu_disable
>  asm_mmu_disable:
> -	mrs	x0, sctlr_el1
> -	bic	x0, x0, SCTLR_EL1_M
> -	msr	sctlr_el1, x0
> -	isb
> -
>  	/* Clean + invalidate the entire memory */
>  	adrp	x0, __phys_offset
>  	ldr	x0, [x0, :lo12:__phys_offset]
> @@ -259,6 +254,22 @@ asm_mmu_disable:
>  	sub	x1, x1, x0
>  	dcache_by_line_op civac, sy, x0, x1, x2, x3
>  
> +	/* Invalidate Icache */
> +	ic	iallu
> +	isb
> +
> +	/* Disable cache, Icache and MMU */
> +	mrs	x0, sctlr_el1
> +	bic	x0, x0, SCTLR_EL1_C
> +	bic	x0, x0, SCTLR_EL1_I
> +	bic	x0, x0, SCTLR_EL1_M
> +	msr	sctlr_el1, x0
> +	isb
> +
> +	/* Invalidate TLB */
> +	tlbi	vmalle1
> +	dsb	nsh
> +
>  	ret
>  
>  /*
> -- 
> 2.25.1
> 

Re: [kvm-unit-tests PATCH v3 19/27] arm/arm64: Add a setup sequence for systems that boot through EFI

[Alexandru Elisei]

Hi,

I missed your reply [1] to my comment [2] regarding v2 of this patch, I'm
going to do my best to reply here. I'll try to copy your reply, I hope it
doesn't turn into something unreadable.

You mentioned the stack in your reply [1], that got me thinking. I guess I
have a question now, are you using the stack set by EFI or another stack
that you have control over? If you're using the EFI stack, then there
should be a set of properties for that stack that EFI guarantees. What are
those guarantees?

[1] https://lore.kernel.org/all/6c5a3ef7-3742-c4e9-5a94-c702a5b3ebca@arm.com/
[2] https://lore.kernel.org/all/Yn5dhgVGoZUgYGUi@monolith.localdoman/

On Thu, Jun 30, 2022 at 11:03:16AM +0100, Nikos Nikoleris wrote:
> This change implements an alternative setup sequence for the system
> when we are booting through EFI. The memory map is discovered through
> EFI boot services and devices through ACPI.
> 
> This change is based on a change initially proposed by
> Andrew Jones <drjones@redhat.com>
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/linux/efi.h     |   1 +
>  lib/arm/asm/setup.h |   2 +
>  lib/arm/setup.c     | 181 +++++++++++++++++++++++++++++++++++++++++++-
>  arm/cstart.S        |   1 +
>  arm/cstart64.S      |   1 +
>  5 files changed, 184 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/linux/efi.h b/lib/linux/efi.h
> index 53748dd..89f9a9e 100644
> --- a/lib/linux/efi.h
> +++ b/lib/linux/efi.h
> @@ -63,6 +63,7 @@ typedef guid_t efi_guid_t;
>  	(c) & 0xff, ((c) >> 8) & 0xff, d } }
>  
>  #define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)
> +#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3,  0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)
>  
>  #define LOADED_IMAGE_PROTOCOL_GUID EFI_GUID(0x5b1b31a1, 0x9562, 0x11d2,  0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b)
>  
> diff --git a/lib/arm/asm/setup.h b/lib/arm/asm/setup.h
> index 64cd379..c4cd485 100644
> --- a/lib/arm/asm/setup.h
> +++ b/lib/arm/asm/setup.h
> @@ -6,6 +6,7 @@
>   * This work is licensed under the terms of the GNU LGPL, version 2.
>   */
>  #include <libcflat.h>
> +#include <efi.h>
>  #include <asm/page.h>
>  #include <asm/pgtable-hwdef.h>
>  
> @@ -37,5 +38,6 @@ extern unsigned int mem_region_get_flags(phys_addr_t paddr);
>  #define SMP_CACHE_BYTES		L1_CACHE_BYTES
>  
>  void setup(const void *fdt, phys_addr_t freemem_start);
> +efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo);
>  
>  #endif /* _ASMARM_SETUP_H_ */
> diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> index 13513d0..30d04d0 100644
> --- a/lib/arm/setup.c
> +++ b/lib/arm/setup.c
> @@ -34,7 +34,7 @@
>  #define NR_EXTRA_MEM_REGIONS	16
>  #define NR_INITIAL_MEM_REGIONS	(MAX_DT_MEM_REGIONS + NR_EXTRA_MEM_REGIONS)
>  
> -extern unsigned long _etext;
> +extern unsigned long _text, _etext, _data, _edata;
>  
>  char *initrd;
>  u32 initrd_size;
> @@ -44,7 +44,10 @@ int nr_cpus;
>  
>  static struct mem_region __initial_mem_regions[NR_INITIAL_MEM_REGIONS + 1];
>  struct mem_region *mem_regions = __initial_mem_regions;
> -phys_addr_t __phys_offset, __phys_end;
> +phys_addr_t __phys_offset = (phys_addr_t)-1, __phys_end = 0;
> +
> +extern void exceptions_init(void);
> +extern void asm_mmu_disable(void);
>  
>  int mpidr_to_cpu(uint64_t mpidr)
>  {
> @@ -272,3 +275,177 @@ void setup(const void *fdt, phys_addr_t freemem_start)
>  	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
>  		setup_vm();
>  }
> +
> +#ifdef CONFIG_EFI
> +
> +#include <efi.h>
> +
> +static efi_status_t setup_rsdp(efi_bootinfo_t *efi_bootinfo)
> +{
> +	efi_status_t status;
> +	struct rsdp_descriptor *rsdp;
> +
> +	/*
> +	 * RSDP resides in an EFI_ACPI_RECLAIM_MEMORY region, which is not used
> +	 * by kvm-unit-tests arm64 memory allocator. So it is not necessary to
> +	 * copy the data structure to another memory region to prevent
> +	 * unintentional overwrite.
> +	 */
> +	status = efi_get_system_config_table(ACPI_20_TABLE_GUID, (void **)&rsdp);
> +	if (status != EFI_SUCCESS)
> +		return status;
> +
> +	set_efi_rsdp(rsdp);
> +
> +	return EFI_SUCCESS;
> +}
> +
> +static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
> +{
> +	int i;
> +	unsigned long free_mem_pages = 0;
> +	unsigned long free_mem_start = 0;
> +	struct efi_boot_memmap *map = &(efi_bootinfo->mem_map);
> +	efi_memory_desc_t *buffer = *map->map;
> +	efi_memory_desc_t *d = NULL;
> +	phys_addr_t base, top;
> +	struct mem_region *r;
> +	uintptr_t text = (uintptr_t)&_text, etext = __ALIGN((uintptr_t)&_etext, 4096);
> +	uintptr_t data = (uintptr_t)&_data, edata = __ALIGN((uintptr_t)&_edata, 4096);
> +
> +	/*
> +	 * Record the largest free EFI_CONVENTIONAL_MEMORY region
> +	 * which will be used to set up the memory allocator, so that
> +	 * the memory allocator can work in the largest free
> +	 * continuous memory region.
> +	 */
> +	for (i = 0, r = &mem_regions[0]; i < *(map->map_size); i += *(map->desc_size), ++r) {
> +		d = (efi_memory_desc_t *)(&((u8 *)buffer)[i]);
> +
> +		r->start = d->phys_addr;
> +		r->end = d->phys_addr + d->num_pages * EFI_PAGE_SIZE;
> +
> +		switch (d->type) {
> +		case EFI_RESERVED_TYPE:
> +		case EFI_LOADER_DATA:
> +		case EFI_BOOT_SERVICES_CODE:
> +		case EFI_BOOT_SERVICES_DATA:
> +		case EFI_RUNTIME_SERVICES_CODE:
> +		case EFI_RUNTIME_SERVICES_DATA:
> +		case EFI_UNUSABLE_MEMORY:
> +		case EFI_ACPI_RECLAIM_MEMORY:
> +		case EFI_ACPI_MEMORY_NVS:
> +		case EFI_PAL_CODE:
> +			r->flags = MR_F_RESERVED;
> +			break;
> +		case EFI_MEMORY_MAPPED_IO:
> +		case EFI_MEMORY_MAPPED_IO_PORT_SPACE:
> +			r->flags = MR_F_IO;
> +			break;
> +		case EFI_LOADER_CODE:
> +			if (r->start <= text && r->end > text) {
> +				/* This is the unit test region. Flag the code separately. */
> +				phys_addr_t tmp = r->end;
> +
> +				assert(etext <= data);
> +				assert(edata <= r->end);
> +				r->flags = MR_F_CODE;
> +				r->end = data;
> +				++r;
> +				r->start = data;
> +				r->end = tmp;
> +			} else {
> +				r->flags = MR_F_RESERVED;
> +			}
> +			break;
> +		case EFI_CONVENTIONAL_MEMORY:
> +			if (free_mem_pages < d->num_pages) {
> +				free_mem_pages = d->num_pages;
> +				free_mem_start = d->phys_addr;
> +			}
> +			break;
> +		}
> +
> +		if (!(r->flags & MR_F_IO)) {
> +			if (r->start < __phys_offset)
> +				__phys_offset = r->start;
> +			if (r->end > __phys_end)
> +				__phys_end = r->end;
> +		}
> +	}
> +	__phys_end &= PHYS_MASK;
>
> I think, there is much more that we need to clean, we're still using the
> same stack, we've loaded the code with the MMU on and we're using some

The assembly routing to disable the MMU doesn't use the stack, so I'm not
really sure what you trying to say here. If you're worried that the region
described by __phys_offset and __phys_end doesn't include the stack, then
that means that mmu_disable() will never work for a test (it will not clean
+ invalidate the stack), and that should be definitely fixed.

> memory up until the point we got here. I don't think, it would be safe to
> clean only __phys_offset and __phys_end and move on. Unless what you're
> suggesting is to clean __phys_offset and __phys_end after we switch the MMU

The purpose of cleaning __phys_offset and __phys_end is to for the latest
values to be in memory. It doesn't matter if it's done with the MMU off or
on, as long as it's done after the last write to the variables, and before
they are loaded, obviously.

> off. But then I have two questions:
> * If the CPU can still speculatively load __phys_offset and __phys_end at
> least the invalidate operation is still questionable.

Like I've said above, the purpose of the clean (you don't need an
invalidate) is to make sure that the CPU loads the latest values from
memory when the MMU is off. It doesn't matter if the value is in the cache,
as the subsequent clean + invalidate sequence will invalidate it. Which
also does not matter since this is the last time kvm-unit-tests writes to
the variables (the invalidate is needed so the CPU sees the values written
with the MMU off instead of stale values in the cache).

> * If we switch off the MMU and then clean the cache, are we causing
> coherence issues by issuing the CMOs with different memory attributes
> (Device-nGnRnE vs Normal Write-back).

This is interesting, but I find it very vague. Would you care to elaborate
how that might happen? A specific scenario would be helpful here, I
believe.

> +	asm_mmu_disable();

Thanks,
Alex

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Nikos Nikoleris]

Hi Alex,

On 30/06/2022 11:20, Alexandru Elisei wrote:
> Hi,
> 
> On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
>> From: Andrew Jones <drjones@redhat.com>
>>
>> The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
>> clean + invalidate after turning MMU off") justifies cleaning and
>> invalidating the dcache after disabling the MMU by saying it's nice
>> not to rely on the current page tables and that it should still work
>> (per the spec), as long as there's an identity map in the current
>> tables. Doing the invalidation after also somewhat helped with
>> reenabling the MMU without seeing stale data, but the real problem
>> with reenabling was because the cache needs to be disabled with
>> the MMU, but it wasn't.
>>
>> Since we have to trust/validate that the current page tables have an
>> identity map anyway, then there's no harm in doing the clean
>> and invalidate first (it feels a little better to do so, anyway,
>> considering the cache maintenance instructions take virtual
>> addresses). Then, also disable the cache with the MMU to avoid
>> problems when reenabling. We invalidate the Icache and disable
>> that too for good measure. And, a final TLB invalidation ensures
>> we're crystal clean when we return from asm_mmu_disable().
> 
> I'll point you to my previous reply [1] to this exact patch which explains
> why it's incorrect and is only papering over another problem.
> 
> [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> 

Apologies, I didn't mean to ignore your feedback on this. There was a 
parallel discussion in [2] which I thought makes the problem more concrete.

This is Drew's patch as soon as he confirms he's also happy with the 
change you suggested in the patch description I am happy to make it.

Generally, a test will start off with the MMU enabled. At this point, we 
access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of 
the two regions could be mapped as any type of memory (I need to have 
another look to confirm if it's Normal Memory). Then we want to take 
over control of the page tables and for that reason we have to switch 
off the MMU. And any access to code or data will be with Device-nGnRnE 
as you pointed out. If we don't clean and invalidate, instructions and 
data might be in the cache and we will be mixing memory attributes, 
won't we?

[2]: 
https://lore.kernel.org/all/6c5a3ef7-3742-c4e9-5a94-c702a5b3ebca@arm.com/

Thanks,

Nikos

> Thanks,
> Alex
> 
>>
>> Cc: Alexandru Elisei <alexandru.elisei@arm.com>
>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>> ---
>>   arm/cstart.S   | 28 +++++++++++++++++++++-------
>>   arm/cstart64.S | 21 ++++++++++++++++-----
>>   2 files changed, 37 insertions(+), 12 deletions(-)
>>
>> diff --git a/arm/cstart.S b/arm/cstart.S
>> index 7036e67..dc324c5 100644
>> --- a/arm/cstart.S
>> +++ b/arm/cstart.S
>> @@ -179,6 +179,7 @@ halt:
>>   .globl asm_mmu_enable
>>   asm_mmu_enable:
>>   	/* TLBIALL */
>> +	mov	r2, #0
>>   	mcr	p15, 0, r2, c8, c7, 0
>>   	dsb	nsh
>>   
>> @@ -211,12 +212,7 @@ asm_mmu_enable:
>>   
>>   .globl asm_mmu_disable
>>   asm_mmu_disable:
>> -	/* SCTLR */
>> -	mrc	p15, 0, r0, c1, c0, 0
>> -	bic	r0, #CR_M
>> -	mcr	p15, 0, r0, c1, c0, 0
>> -	isb
>> -
>> +	/* Clean + invalidate the entire memory */
>>   	ldr	r0, =__phys_offset
>>   	ldr	r0, [r0]
>>   	ldr	r1, =__phys_end
>> @@ -224,7 +220,25 @@ asm_mmu_disable:
>>   	sub	r1, r1, r0
>>   	dcache_by_line_op dccimvac, sy, r0, r1, r2, r3
>>   
>> -	mov     pc, lr
>> +	/* Invalidate Icache */
>> +	mov	r0, #0
>> +	mcr	p15, 0, r0, c7, c5, 0
>> +	isb
>> +
>> +	/*  Disable cache, Icache and MMU */
>> +	mrc	p15, 0, r0, c1, c0, 0
>> +	bic	r0, #CR_C
>> +	bic	r0, #CR_I
>> +	bic	r0, #CR_M
>> +	mcr	p15, 0, r0, c1, c0, 0
>> +	isb
>> +
>> +	/* Invalidate TLB */
>> +	mov	r0, #0
>> +	mcr	p15, 0, r0, c8, c7, 0
>> +	dsb	nsh
>> +
>> +	mov	pc, lr
>>   
>>   /*
>>    * Vectors
>> diff --git a/arm/cstart64.S b/arm/cstart64.S
>> index e4ab7d0..390feb9 100644
>> --- a/arm/cstart64.S
>> +++ b/arm/cstart64.S
>> @@ -246,11 +246,6 @@ asm_mmu_enable:
>>   
>>   .globl asm_mmu_disable
>>   asm_mmu_disable:
>> -	mrs	x0, sctlr_el1
>> -	bic	x0, x0, SCTLR_EL1_M
>> -	msr	sctlr_el1, x0
>> -	isb
>> -
>>   	/* Clean + invalidate the entire memory */
>>   	adrp	x0, __phys_offset
>>   	ldr	x0, [x0, :lo12:__phys_offset]
>> @@ -259,6 +254,22 @@ asm_mmu_disable:
>>   	sub	x1, x1, x0
>>   	dcache_by_line_op civac, sy, x0, x1, x2, x3
>>   
>> +	/* Invalidate Icache */
>> +	ic	iallu
>> +	isb
>> +
>> +	/* Disable cache, Icache and MMU */
>> +	mrs	x0, sctlr_el1
>> +	bic	x0, x0, SCTLR_EL1_C
>> +	bic	x0, x0, SCTLR_EL1_I
>> +	bic	x0, x0, SCTLR_EL1_M
>> +	msr	sctlr_el1, x0
>> +	isb
>> +
>> +	/* Invalidate TLB */
>> +	tlbi	vmalle1
>> +	dsb	nsh
>> +
>>   	ret
>>   
>>   /*
>> -- 
>> 2.25.1
>>

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi,

On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
> Hi Alex,
> 
> On 30/06/2022 11:20, Alexandru Elisei wrote:
> > Hi,
> > 
> > On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> > > From: Andrew Jones <drjones@redhat.com>
> > > 
> > > The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> > > clean + invalidate after turning MMU off") justifies cleaning and
> > > invalidating the dcache after disabling the MMU by saying it's nice
> > > not to rely on the current page tables and that it should still work
> > > (per the spec), as long as there's an identity map in the current
> > > tables. Doing the invalidation after also somewhat helped with
> > > reenabling the MMU without seeing stale data, but the real problem
> > > with reenabling was because the cache needs to be disabled with
> > > the MMU, but it wasn't.
> > > 
> > > Since we have to trust/validate that the current page tables have an
> > > identity map anyway, then there's no harm in doing the clean
> > > and invalidate first (it feels a little better to do so, anyway,
> > > considering the cache maintenance instructions take virtual
> > > addresses). Then, also disable the cache with the MMU to avoid
> > > problems when reenabling. We invalidate the Icache and disable
> > > that too for good measure. And, a final TLB invalidation ensures
> > > we're crystal clean when we return from asm_mmu_disable().
> > 
> > I'll point you to my previous reply [1] to this exact patch which explains
> > why it's incorrect and is only papering over another problem.
> > 
> > [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> > 
> 
> Apologies, I didn't mean to ignore your feedback on this. There was a
> parallel discussion in [2] which I thought makes the problem more concrete.

No problem, I figured as much :).

> 
> This is Drew's patch as soon as he confirms he's also happy with the change
> you suggested in the patch description I am happy to make it.
> 
> Generally, a test will start off with the MMU enabled. At this point, we
> access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
> two regions could be mapped as any type of memory (I need to have another
> look to confirm if it's Normal Memory). Then we want to take over control of
> the page tables and for that reason we have to switch off the MMU. And any
> access to code or data will be with Device-nGnRnE as you pointed out. If we
> don't clean and invalidate, instructions and data might be in the cache and
> we will be mixing memory attributes, won't we?

I missed that comment, sorry. I've replied to that comment made in v2,
here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
systems that boot through EFI").

This is the second time you've mentioned mixed memory attributes, so I'm
going to reiterate the question I asked in patch #19: what do you mean by
"mixing memory attributes" and what is wrong with it? Because it looks to
me like you're saying that you cannot access data written with the MMU on
when the MMU is off (and I assume the other way around, you cannot data
written with the MMU off when the MMU is on).

Thanks,
Alex

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Nikos Nikoleris]

Hi Alex,

On 30/06/2022 12:24, Alexandru Elisei wrote:
> Hi,
> 
> On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
>> Hi Alex,
>>
>> On 30/06/2022 11:20, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
>>>> From: Andrew Jones <drjones@redhat.com>
>>>>
>>>> The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
>>>> clean + invalidate after turning MMU off") justifies cleaning and
>>>> invalidating the dcache after disabling the MMU by saying it's nice
>>>> not to rely on the current page tables and that it should still work
>>>> (per the spec), as long as there's an identity map in the current
>>>> tables. Doing the invalidation after also somewhat helped with
>>>> reenabling the MMU without seeing stale data, but the real problem
>>>> with reenabling was because the cache needs to be disabled with
>>>> the MMU, but it wasn't.
>>>>
>>>> Since we have to trust/validate that the current page tables have an
>>>> identity map anyway, then there's no harm in doing the clean
>>>> and invalidate first (it feels a little better to do so, anyway,
>>>> considering the cache maintenance instructions take virtual
>>>> addresses). Then, also disable the cache with the MMU to avoid
>>>> problems when reenabling. We invalidate the Icache and disable
>>>> that too for good measure. And, a final TLB invalidation ensures
>>>> we're crystal clean when we return from asm_mmu_disable().
>>>
>>> I'll point you to my previous reply [1] to this exact patch which explains
>>> why it's incorrect and is only papering over another problem.
>>>
>>> [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
>>>
>>
>> Apologies, I didn't mean to ignore your feedback on this. There was a
>> parallel discussion in [2] which I thought makes the problem more concrete.
> 
> No problem, I figured as much :).
> 
>>
>> This is Drew's patch as soon as he confirms he's also happy with the change
>> you suggested in the patch description I am happy to make it.
>>
>> Generally, a test will start off with the MMU enabled. At this point, we
>> access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
>> two regions could be mapped as any type of memory (I need to have another
>> look to confirm if it's Normal Memory). Then we want to take over control of
>> the page tables and for that reason we have to switch off the MMU. And any
>> access to code or data will be with Device-nGnRnE as you pointed out. If we
>> don't clean and invalidate, instructions and data might be in the cache and
>> we will be mixing memory attributes, won't we?
> 
> I missed that comment, sorry. I've replied to that comment made in v2,
> here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
> systems that boot through EFI").
> 
> This is the second time you've mentioned mixed memory attributes, so I'm
> going to reiterate the question I asked in patch #19: what do you mean by
> "mixing memory attributes" and what is wrong with it? Because it looks to
> me like you're saying that you cannot access data written with the MMU on
> when the MMU is off (and I assume the other way around, you cannot data
> written with the MMU off when the MMU is on).
> 

What I mean by mixing memory attributes is illustrated by the following 
example.

Take a memory location x, for which the page table entry maps to a 
physical location as Normal, Inner-Shareable, Inner-writeback and 
Outer-writeback. If we access it when the MMU is on and subquently when 
the MMU is off (treated as Device-nGnRnE), then we have two accesses 
with mismatched memory attributes to the same location. There is a whole 
section in the Arm ARM on why this needs to be avoided (B2.8 Mismatched 
memory attributes) but the result is "a loss of the uniprocessor 
semantics, ordering, or coherency". As I understand, the solution to 
this is:

"If the mismatched attributes for a Location mean that multiple 
cacheable accesses to the Location might be made with different 
shareability attributes, then uniprocessor semantics, ordering, and 
coherency are guaranteed only if:
• Software running on a PE cleans and invalidates a Location from cache 
before and after each read or write to that Location by that PE.
• A DMB barrier with scope that covers the full shareability of the 
accesses is placed between any accesses to the same memory Location that 
use different attributes."

So unless UEFI maps all memory as Device-nGnRnE we have to do something. 
I will try to find out more about UEFI's page tables.

Thanks,

Nikos


> Thanks,
> Alex

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi,

On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
> Hi Alex,
> 
> On 30/06/2022 12:24, Alexandru Elisei wrote:
> > Hi,
> > 
> > On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
> > > Hi Alex,
> > > 
> > > On 30/06/2022 11:20, Alexandru Elisei wrote:
> > > > Hi,
> > > > 
> > > > On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> > > > > From: Andrew Jones <drjones@redhat.com>
> > > > > 
> > > > > The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> > > > > clean + invalidate after turning MMU off") justifies cleaning and
> > > > > invalidating the dcache after disabling the MMU by saying it's nice
> > > > > not to rely on the current page tables and that it should still work
> > > > > (per the spec), as long as there's an identity map in the current
> > > > > tables. Doing the invalidation after also somewhat helped with
> > > > > reenabling the MMU without seeing stale data, but the real problem
> > > > > with reenabling was because the cache needs to be disabled with
> > > > > the MMU, but it wasn't.
> > > > > 
> > > > > Since we have to trust/validate that the current page tables have an
> > > > > identity map anyway, then there's no harm in doing the clean
> > > > > and invalidate first (it feels a little better to do so, anyway,
> > > > > considering the cache maintenance instructions take virtual
> > > > > addresses). Then, also disable the cache with the MMU to avoid
> > > > > problems when reenabling. We invalidate the Icache and disable
> > > > > that too for good measure. And, a final TLB invalidation ensures
> > > > > we're crystal clean when we return from asm_mmu_disable().
> > > > 
> > > > I'll point you to my previous reply [1] to this exact patch which explains
> > > > why it's incorrect and is only papering over another problem.
> > > > 
> > > > [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> > > > 
> > > 
> > > Apologies, I didn't mean to ignore your feedback on this. There was a
> > > parallel discussion in [2] which I thought makes the problem more concrete.
> > 
> > No problem, I figured as much :).
> > 
> > > 
> > > This is Drew's patch as soon as he confirms he's also happy with the change
> > > you suggested in the patch description I am happy to make it.
> > > 
> > > Generally, a test will start off with the MMU enabled. At this point, we
> > > access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
> > > two regions could be mapped as any type of memory (I need to have another
> > > look to confirm if it's Normal Memory). Then we want to take over control of
> > > the page tables and for that reason we have to switch off the MMU. And any
> > > access to code or data will be with Device-nGnRnE as you pointed out. If we
> > > don't clean and invalidate, instructions and data might be in the cache and
> > > we will be mixing memory attributes, won't we?
> > 
> > I missed that comment, sorry. I've replied to that comment made in v2,
> > here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
> > systems that boot through EFI").
> > 
> > This is the second time you've mentioned mixed memory attributes, so I'm
> > going to reiterate the question I asked in patch #19: what do you mean by
> > "mixing memory attributes" and what is wrong with it? Because it looks to
> > me like you're saying that you cannot access data written with the MMU on
> > when the MMU is off (and I assume the other way around, you cannot data
> > written with the MMU off when the MMU is on).
> > 
> 
> What I mean by mixing memory attributes is illustrated by the following
> example.
> 
> Take a memory location x, for which the page table entry maps to a physical
> location as Normal, Inner-Shareable, Inner-writeback and Outer-writeback. If
> we access it when the MMU is on and subquently when the MMU is off (treated
> as Device-nGnRnE), then we have two accesses with mismatched memory
> attributes to the same location. There is a whole section in the Arm ARM on
> why this needs to be avoided (B2.8 Mismatched memory attributes) but the
> result is "a loss of the uniprocessor semantics, ordering, or coherency". As
> I understand, the solution to this is:
> 
> "If the mismatched attributes for a Location mean that multiple cacheable
> accesses to the Location might be made with different shareability
> attributes, then uniprocessor semantics, ordering, and coherency are
> guaranteed only if:
> • Software running on a PE cleans and invalidates a Location from cache
> before and after each read or write to that Location by that PE.
> • A DMB barrier with scope that covers the full shareability of the accesses
> is placed between any accesses to the same memory Location that use
> different attributes."

Ok, so this is about *mismatched* memory attributes. I searched the Arm ARM
for the string "mixed" and nothing relevant came up.

Device-whatever memory is outer shareable and kvm-unit-tests maps memory as
inner shareable, so that matches the "different shareability attributes"
part of the paragraph.

But I would like to point out that there is only one type of cacheable
access that is being performed, when the MMU is on. When the MMU is off,
the access is not cacheable. So there are two types of accesses being
performed:

- cacheable + inner-shareable (MMU on)
- non-cacheable + outer-shareable (MMU off)

It looks to me like the paragraph doesn't apply to our case, because there
are no "multiple cacheable accesses [..] made with different shareability
attributes". Do you agree?

> 
> So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
> will try to find out more about UEFI's page tables.

That's important to know, especially regarding the text section of the
image. If UEFI doesnt' clean it to PoC, kvm-unit-tests must do it in order
to execute correctly with the MMU off.

Thanks,
Alex

> 
> Thanks,
> 
> Nikos
> 
> 
> > Thanks,
> > Alex

Re: [kvm-unit-tests PATCH v3 22/27] arm64: Use code from the gnu-efi when booting with EFI

[Ricardo Koller]

On Thu, Jun 30, 2022 at 11:03:19AM +0100, Nikos Nikoleris wrote:
> arm/efi/crt0-efi-aarch64.S defines the header and the handover
> sequence from EFI to a efi_main. This change includes the whole file
> in arm/cstart64.S when we compile with EFI support.
> 
> In addition, we change the handover code in arm/efi/crt0-efi-aarch64.S
> to align the stack pointer. This alignment is necessary because we
> make assumptions about cpu0's stack alignment and most importantly we
> place its thread_info at the bottom of this stack.
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  arm/cstart64.S             |  6 ++++++
>  arm/efi/crt0-efi-aarch64.S | 21 +++++++++++++++++----
>  2 files changed, 23 insertions(+), 4 deletions(-)
> 
> diff --git a/arm/cstart64.S b/arm/cstart64.S
> index 55b41ea..08cf02f 100644
> --- a/arm/cstart64.S
> +++ b/arm/cstart64.S
> @@ -15,6 +15,10 @@
>  #include <asm/thread_info.h>
>  #include <asm/sysreg.h>
>  
> +#ifdef CONFIG_EFI
> +#include "efi/crt0-efi-aarch64.S"
> +#else
> +
>  .macro zero_range, tmp1, tmp2
>  9998:	cmp	\tmp1, \tmp2
>  	b.eq	9997f
> @@ -107,6 +111,8 @@ start:
>  	bl	exit
>  	b	halt
>  
> +#endif
> +
>  .text
>  
>  /*
> diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
> index d50e78d..03d29b0 100644
> --- a/arm/efi/crt0-efi-aarch64.S
> +++ b/arm/efi/crt0-efi-aarch64.S
> @@ -111,10 +111,19 @@ section_table:
>  
>  	.align		12
>  _start:
> -	stp		x29, x30, [sp, #-32]!
> +	stp		x29, x30, [sp, #-16]!
> +
> +	/* Align sp; this is necessary due to way we store cpu0's thread_info */
>  	mov		x29, sp
> +	and		x29, x29, #THREAD_MASK
> +	mov		x30, sp
> +	mov		sp, x29
> +	str		x30, [sp, #-16]!
> +
> +	mov             x29, sp

I wasn't sure what was this x29 for. But after some googling, this is
what I found [0]:

	The frame pointer (X29) should point to the previous frame pointer saved
	on stack, with the saved LR (X30) stored after it. 

The old code ended up with x29 pointing to the right place: the previous
(x29,x30).

	|   ...  |
	|   x1   |
	|   x0   |
	|   x30  |
x29 ->	|   x29  |

In the new code x29 is pointing to:

	|   ...  |
	|   x30  |
old_sp->|   x29  |
	|   ...  |
	|   x1   |
	|   x0   |
	|   pad  |
x29 ->	| old_sp |

I think the new version can be fixed by setting x29 to the old_sp,
conveniently stored in x30:

+	mov             x30, sp

> +
> +	stp		x0, x1, [sp, #-16]!
>  
> -	stp		x0, x1, [sp, #16]
>  	mov		x2, x0
>  	mov		x3, x1
>  	adr		x0, ImageBase
> @@ -123,8 +132,12 @@ _start:
>  	bl		_relocate
>  	cbnz		x0, 0f
>  
> -	ldp		x0, x1, [sp, #16]
> +	ldp		x0, x1, [sp], #16
>  	bl		efi_main
>  
> -0:	ldp		x29, x30, [sp], #32
> +	/* Restore sp */
> +	ldr		x30, [sp], #16
> +	mov             sp, x30
> +
> +0:	ldp		x29, x30, [sp], #16
>  	ret
> -- 
> 2.25.1
> 

[0] https://developer.arm.com/documentation/den0024/a/The-ABI-for-ARM-64-bit-Architecture/Register-use-in-the-AArch64-Procedure-Call-Standard/Indirect-result-location

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Andrew Jones]

On Thu, Jun 30, 2022 at 04:57:39PM +0100, Alexandru Elisei wrote:
> Hi,
> 
> On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
> > Hi Alex,
> > 
> > On 30/06/2022 12:24, Alexandru Elisei wrote:
> > > Hi,
> > > 
> > > On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
> > > > Hi Alex,
> > > > 
> > > > On 30/06/2022 11:20, Alexandru Elisei wrote:
> > > > > Hi,
> > > > > 
> > > > > On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> > > > > > From: Andrew Jones <drjones@redhat.com>
> > > > > > 
> > > > > > The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> > > > > > clean + invalidate after turning MMU off") justifies cleaning and
> > > > > > invalidating the dcache after disabling the MMU by saying it's nice
> > > > > > not to rely on the current page tables and that it should still work
> > > > > > (per the spec), as long as there's an identity map in the current
> > > > > > tables. Doing the invalidation after also somewhat helped with
> > > > > > reenabling the MMU without seeing stale data, but the real problem
> > > > > > with reenabling was because the cache needs to be disabled with
> > > > > > the MMU, but it wasn't.
> > > > > > 
> > > > > > Since we have to trust/validate that the current page tables have an
> > > > > > identity map anyway, then there's no harm in doing the clean
> > > > > > and invalidate first (it feels a little better to do so, anyway,
> > > > > > considering the cache maintenance instructions take virtual
> > > > > > addresses). Then, also disable the cache with the MMU to avoid
> > > > > > problems when reenabling. We invalidate the Icache and disable
> > > > > > that too for good measure. And, a final TLB invalidation ensures
> > > > > > we're crystal clean when we return from asm_mmu_disable().
> > > > > 
> > > > > I'll point you to my previous reply [1] to this exact patch which explains
> > > > > why it's incorrect and is only papering over another problem.
> > > > > 
> > > > > [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> > > > > 
> > > > 
> > > > Apologies, I didn't mean to ignore your feedback on this. There was a
> > > > parallel discussion in [2] which I thought makes the problem more concrete.
> > > 
> > > No problem, I figured as much :).
> > > 
> > > > 
> > > > This is Drew's patch as soon as he confirms he's also happy with the change
> > > > you suggested in the patch description I am happy to make it.
> > > > 
> > > > Generally, a test will start off with the MMU enabled. At this point, we
> > > > access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
> > > > two regions could be mapped as any type of memory (I need to have another
> > > > look to confirm if it's Normal Memory). Then we want to take over control of
> > > > the page tables and for that reason we have to switch off the MMU. And any
> > > > access to code or data will be with Device-nGnRnE as you pointed out. If we
> > > > don't clean and invalidate, instructions and data might be in the cache and
> > > > we will be mixing memory attributes, won't we?
> > > 
> > > I missed that comment, sorry. I've replied to that comment made in v2,
> > > here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
> > > systems that boot through EFI").
> > > 
> > > This is the second time you've mentioned mixed memory attributes, so I'm
> > > going to reiterate the question I asked in patch #19: what do you mean by
> > > "mixing memory attributes" and what is wrong with it? Because it looks to
> > > me like you're saying that you cannot access data written with the MMU on
> > > when the MMU is off (and I assume the other way around, you cannot data
> > > written with the MMU off when the MMU is on).
> > > 
> > 
> > What I mean by mixing memory attributes is illustrated by the following
> > example.
> > 
> > Take a memory location x, for which the page table entry maps to a physical
> > location as Normal, Inner-Shareable, Inner-writeback and Outer-writeback. If
> > we access it when the MMU is on and subquently when the MMU is off (treated
> > as Device-nGnRnE), then we have two accesses with mismatched memory
> > attributes to the same location. There is a whole section in the Arm ARM on
> > why this needs to be avoided (B2.8 Mismatched memory attributes) but the
> > result is "a loss of the uniprocessor semantics, ordering, or coherency". As
> > I understand, the solution to this is:
> > 
> > "If the mismatched attributes for a Location mean that multiple cacheable
> > accesses to the Location might be made with different shareability
> > attributes, then uniprocessor semantics, ordering, and coherency are
> > guaranteed only if:
> > • Software running on a PE cleans and invalidates a Location from cache
> > before and after each read or write to that Location by that PE.
> > • A DMB barrier with scope that covers the full shareability of the accesses
> > is placed between any accesses to the same memory Location that use
> > different attributes."
> 
> Ok, so this is about *mismatched* memory attributes. I searched the Arm ARM
> for the string "mixed" and nothing relevant came up.
> 
> Device-whatever memory is outer shareable and kvm-unit-tests maps memory as
> inner shareable, so that matches the "different shareability attributes"
> part of the paragraph.
> 
> But I would like to point out that there is only one type of cacheable
> access that is being performed, when the MMU is on. When the MMU is off,
> the access is not cacheable. So there are two types of accesses being
> performed:
> 
> - cacheable + inner-shareable (MMU on)
> - non-cacheable + outer-shareable (MMU off)
> 
> It looks to me like the paragraph doesn't apply to our case, because there
> are no "multiple cacheable accesses [..] made with different shareability
> attributes". Do you agree?
> 
> > 
> > So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
> > will try to find out more about UEFI's page tables.
> 
> That's important to know, especially regarding the text section of the
> image. If UEFI doesnt' clean it to PoC, kvm-unit-tests must do it in order
> to execute correctly with the MMU off.
>

Hi Alex and Nikos,

Indeed my experiments on bare-metal made this change necessary. I'm happy
to see this discussion, though, as this patch could be tweaked or at least
the commit message improved in order to better explain what's going on and
why the changes are necessary. IOW, I have no problem with this patch
being dropped and replaced by one of you with something that "makes
more sense" as long as the outcome (coherent execution on bare-metal)
still works.

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 01/27] lib: Fix style for acpi.{c,h}

[Andrew Jones]

Hi Nikos,

I guess you used Linux's scripts/Lindent or something for this
conversion. Can you please specify what you used/did in the
commit message?

On Thu, Jun 30, 2022 at 11:02:58AM +0100, Nikos Nikoleris wrote:
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/acpi.h | 148 ++++++++++++++++++++++++++---------------------------
>  lib/acpi.c |  70 ++++++++++++-------------
>  2 files changed, 108 insertions(+), 110 deletions(-)

It looks like the series is missing the file move patch. Latest master
still doesn't have lib/acpi.*

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 03/27] lib: Ensure all struct definition for ACPI tables are packed

[Andrew Jones]

On Thu, Jun 30, 2022 at 11:03:00AM +0100, Nikos Nikoleris wrote:
> All ACPI table definitions are provided with precise definitions of
> field sizes and offsets, make sure that no compiler optimization can
> interfere with the memory layout of the corresponding structs.
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/acpi.h | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/lib/acpi.h b/lib/acpi.h
> index 456e62d..b853a55 100644
> --- a/lib/acpi.h
> +++ b/lib/acpi.h
> @@ -3,6 +3,12 @@
>  
>  #include "libcflat.h"
>  
> +/*
> + * All tables and structures must be byte-packed to match the ACPI
> + * specification, since the tables are provided by the system BIOS
> + */
> +#pragma pack(1)
> +
>  #define ACPI_SIGNATURE(c1, c2, c3, c4)				\
>  	((c1) | ((c2) << 8) | ((c3) << 16) | ((c4) << 24))
>  
> @@ -106,6 +112,8 @@ struct facs_descriptor_rev1
>  	u8  reserved3 [40];		/* Reserved - must be zero */
>  };
>  
> +#pragma pack(0)
> +
>  void set_efi_rsdp(struct rsdp_descriptor *rsdp);
>  void* find_acpi_table_addr(u32 sig);
>  
> -- 
> 2.25.1
>

Reviewed-by: Andrew Jones <andrew.jones@linux.dev>

Re: [kvm-unit-tests PATCH v3 04/27] lib: Add support for the XSDT ACPI table

[Andrew Jones]

On Thu, Jun 30, 2022 at 11:03:01AM +0100, Nikos Nikoleris wrote:
> XSDT provides pointers to other ACPI tables much like RSDT. However,
> contrary to RSDT that provides 32-bit addresses, XSDT provides 64-bit
> pointers. ACPI requires that if XSDT is valid then it takes precedence
> over RSDT.
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/acpi.h |  6 ++++++
>  lib/acpi.c | 37 +++++++++++++++++++++++++++++++------
>  2 files changed, 37 insertions(+), 6 deletions(-)
> 
> diff --git a/lib/acpi.h b/lib/acpi.h
> index b853a55..c5f0aa5 100644
> --- a/lib/acpi.h
> +++ b/lib/acpi.h
> @@ -14,6 +14,7 @@
>  
>  #define RSDP_SIGNATURE ACPI_SIGNATURE('R','S','D','P')
>  #define RSDT_SIGNATURE ACPI_SIGNATURE('R','S','D','T')
> +#define XSDT_SIGNATURE ACPI_SIGNATURE('X','S','D','T')
>  #define FACP_SIGNATURE ACPI_SIGNATURE('F','A','C','P')
>  #define FACS_SIGNATURE ACPI_SIGNATURE('F','A','C','S')
>  
> @@ -57,6 +58,11 @@ struct rsdt_descriptor_rev1 {
>  	u32 table_offset_entry[1];
>  };
>  
> +struct acpi_table_xsdt {
> +	ACPI_TABLE_HEADER_DEF
> +	u64 table_offset_entry[1];
> +};
> +
>  struct fadt_descriptor_rev1
>  {
>  	ACPI_TABLE_HEADER_DEF	/* ACPI common table header */
> diff --git a/lib/acpi.c b/lib/acpi.c
> index b7fd923..240a922 100644
> --- a/lib/acpi.c
> +++ b/lib/acpi.c
> @@ -40,6 +40,7 @@ void *find_acpi_table_addr(u32 sig)
>  {
>  	struct rsdp_descriptor *rsdp;
>  	struct rsdt_descriptor_rev1 *rsdt;
> +	struct acpi_table_xsdt *xsdt = NULL;
>  	void *end;
>  	int i;
>  
> @@ -64,17 +65,41 @@ void *find_acpi_table_addr(u32 sig)
>  
>  	rsdt = (void *)(ulong)rsdp->rsdt_physical_address;
>  	if (!rsdt || rsdt->signature != RSDT_SIGNATURE)
> -		return NULL;
> +		rsdt = NULL;

Now that we set rsdt to NULL in the body of this if, the
condition may look a bit better as

  (rsdt && rsdt->signature != RSDT_SIGNATURE)

>  
>  	if (sig == RSDT_SIGNATURE)
>  		return rsdt;
>  
> -	end = (void *)rsdt + rsdt->length;
> -	for (i = 0; (void *)&rsdt->table_offset_entry[i] < end; i++) {
> -		struct acpi_table *t = (void *)(ulong)rsdt->table_offset_entry[i];
> -		if (t && t->signature == sig) {
> -			return t;
> +	/*
> +	 * When the system implements APCI 2.0 and above and XSDT is
> +	 * valid we have use XSDT to find other ACPI tables,
                        ^ to

> +	 * otherwise, we use RSDT.
> +	 */
> +	if (rsdp->revision == 2)
> +		xsdt = (void *)(ulong)rsdp->xsdt_physical_address;

The (ulong) cast shouldn't be necessary

maybe add a blank line here

> +	if (!xsdt || xsdt->signature != XSDT_SIGNATURE)

Same comment as above

> +		xsdt = NULL;
> +
> +	if (sig == XSDT_SIGNATURE)
> +		return xsdt;
> +
> +	if (xsdt) {
> +		end = (void *)(ulong)xsdt + xsdt->length;

The (ulong) cast shouldn't be necessary.

> +		for (i = 0; (void *)&xsdt->table_offset_entry[i] < end; i++) {
> +			struct acpi_table *t = (void *)xsdt->table_offset_entry[i];
> +
> +			if (t && t->signature == sig)
> +				return t;
> +		}
> +	} else if (rsdt) {
> +		end = (void *)rsdt + rsdt->length;
> +		for (i = 0; (void *)&rsdt->table_offset_entry[i] < end; i++) {
> +			struct acpi_table *t = (void *)(ulong)rsdt->table_offset_entry[i];
> +
> +			if (t && t->signature == sig)
> +				return t;
>  		}
>  	}
> +
>  	return NULL;
>  }
> -- 
> 2.25.1
>

Besides the nits

Reviewed-by: Andrew Jones <andrew.jones@linux.dev>

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 01/27] lib: Fix style for acpi.{c,h}

[Nikos Nikoleris]

Hi Drew

Thanks for the review!

On 01/07/2022 10:27, Andrew Jones wrote:
> Hi Nikos,
> 
> I guess you used Linux's scripts/Lindent or something for this
> conversion. Can you please specify what you used/did in the
> commit message?
> 

I fixed the style by hand but happy to use Lindent in the next iteration.

> On Thu, Jun 30, 2022 at 11:02:58AM +0100, Nikos Nikoleris wrote:
>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>> ---
>>   lib/acpi.h | 148 ++++++++++++++++++++++++++---------------------------
>>   lib/acpi.c |  70 ++++++++++++-------------
>>   2 files changed, 108 insertions(+), 110 deletions(-)
> 
> It looks like the series is missing the file move patch. Latest master
> still doesn't have lib/acpi.*
> 

I am sorry, I missed the first patch. The missing patch is doing a move 
of acpi.{h,c} [1].

FWIW, I tried combining the patches in one but I ended up with a big 
diff. I found it much easier to check that everything looks ok when the 
overall change was split in two patches.

[1]: 
https://github.com/relokin/kvm-unit-tests/commit/959ca08c23dbaa490b936303b94b006352a29d43

Thanks,

Nikos

> Thanks,
> drew

Re: [kvm-unit-tests PATCH v3 06/27] devicetree: Check if fdt is NULL before returning that a DT is available

[Andrew Jones]

Hi Nikos,

nit: The summary is a bit long. We could probably drop the "before..."
part.

On Thu, Jun 30, 2022 at 11:03:03AM +0100, Nikos Nikoleris wrote:
> Up until now, for platfroms that support DT, KUT would unconditionally
> use DT to configure the system and the code made the assumption that
> the fdt will always be a valid pointer.
> 
> In Arm systems that support both ACPI and DT, we plat to follow the

s/plat/plan/

> same convension as the Linux kernel. First, we attempt to configure
> the system using the DT. If an FDT was not provided then we attempt to
> configure the system using ACPI.
> 
> As a result, for Arm systems with support for ACPI the fdt pointer can
> be NULL. This change modifies dt_available() to check if the fdt is a
> valid pointer and return if we can use information from the DT.
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/devicetree.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/lib/devicetree.c b/lib/devicetree.c
> index 78c1f6f..3ff9d16 100644
> --- a/lib/devicetree.c
> +++ b/lib/devicetree.c
> @@ -16,7 +16,7 @@ const void *dt_fdt(void)
>  
>  bool dt_available(void)
>  {
> -	return fdt_check_header(fdt) == 0;
> +	return fdt && fdt_check_header(fdt) == 0;
>  }
>  
>  int dt_get_nr_cells(int fdtnode, u32 *nr_address_cells, u32 *nr_size_cells)
> -- 
> 2.25.1
>

Reviewed-by: Andrew Jones <andrew.jones@linux.dev>

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 01/27] lib: Fix style for acpi.{c,h}

[Andrew Jones]

(Dropping the drjones@redhat.com address from CC since, as of today,
it's generating auto address-not-valid messages.)

On Fri, Jul 01, 2022 at 10:52:28AM +0100, Nikos Nikoleris wrote:
> Hi Drew
> 
> Thanks for the review!
> 
> On 01/07/2022 10:27, Andrew Jones wrote:
> > Hi Nikos,
> > 
> > I guess you used Linux's scripts/Lindent or something for this
> > conversion. Can you please specify what you used/did in the
> > commit message?
> > 
> 
> I fixed the style by hand but happy to use Lindent in the next iteration.

Yeah, I recommend it. I used it for commit 0e9812980ee5 ("lib: Fix
whitespace"). I did need to modify it to allow 100 columns instead of 80,
though. Also, I then looked at the changes with 'git diff -b' and saw a
few other things to modify by hand.

> 
> > On Thu, Jun 30, 2022 at 11:02:58AM +0100, Nikos Nikoleris wrote:
> > > Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> > > ---
> > >   lib/acpi.h | 148 ++++++++++++++++++++++++++---------------------------
> > >   lib/acpi.c |  70 ++++++++++++-------------
> > >   2 files changed, 108 insertions(+), 110 deletions(-)
> > 
> > It looks like the series is missing the file move patch. Latest master
> > still doesn't have lib/acpi.*
> > 
> 
> I am sorry, I missed the first patch. The missing patch is doing a move of
> acpi.{h,c} [1].
> 
> FWIW, I tried combining the patches in one but I ended up with a big diff. I
> found it much easier to check that everything looks ok when the overall
> change was split in two patches.

Yes, please do it in two separate patches with the move patch generated
with -M, as you've done in [1].

Thanks,
drew

> 
> [1]: https://github.com/relokin/kvm-unit-tests/commit/959ca08c23dbaa490b936303b94b006352a29d43
> 
> Thanks,
> 
> Nikos
> 
> > Thanks,
> > drew

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi,

On Fri, Jul 01, 2022 at 11:12:14AM +0200, Andrew Jones wrote:
> On Thu, Jun 30, 2022 at 04:57:39PM +0100, Alexandru Elisei wrote:
> > Hi,
> > 
> > On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
> > > Hi Alex,
> > > 
> > > On 30/06/2022 12:24, Alexandru Elisei wrote:
> > > > Hi,
> > > > 
> > > > On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
> > > > > Hi Alex,
> > > > > 
> > > > > On 30/06/2022 11:20, Alexandru Elisei wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> > > > > > > From: Andrew Jones <drjones@redhat.com>
> > > > > > > 
> > > > > > > The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> > > > > > > clean + invalidate after turning MMU off") justifies cleaning and
> > > > > > > invalidating the dcache after disabling the MMU by saying it's nice
> > > > > > > not to rely on the current page tables and that it should still work
> > > > > > > (per the spec), as long as there's an identity map in the current
> > > > > > > tables. Doing the invalidation after also somewhat helped with
> > > > > > > reenabling the MMU without seeing stale data, but the real problem
> > > > > > > with reenabling was because the cache needs to be disabled with
> > > > > > > the MMU, but it wasn't.
> > > > > > > 
> > > > > > > Since we have to trust/validate that the current page tables have an
> > > > > > > identity map anyway, then there's no harm in doing the clean
> > > > > > > and invalidate first (it feels a little better to do so, anyway,
> > > > > > > considering the cache maintenance instructions take virtual
> > > > > > > addresses). Then, also disable the cache with the MMU to avoid
> > > > > > > problems when reenabling. We invalidate the Icache and disable
> > > > > > > that too for good measure. And, a final TLB invalidation ensures
> > > > > > > we're crystal clean when we return from asm_mmu_disable().
> > > > > > 
> > > > > > I'll point you to my previous reply [1] to this exact patch which explains
> > > > > > why it's incorrect and is only papering over another problem.
> > > > > > 
> > > > > > [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> > > > > > 
> > > > > 
> > > > > Apologies, I didn't mean to ignore your feedback on this. There was a
> > > > > parallel discussion in [2] which I thought makes the problem more concrete.
> > > > 
> > > > No problem, I figured as much :).
> > > > 
> > > > > 
> > > > > This is Drew's patch as soon as he confirms he's also happy with the change
> > > > > you suggested in the patch description I am happy to make it.
> > > > > 
> > > > > Generally, a test will start off with the MMU enabled. At this point, we
> > > > > access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
> > > > > two regions could be mapped as any type of memory (I need to have another
> > > > > look to confirm if it's Normal Memory). Then we want to take over control of
> > > > > the page tables and for that reason we have to switch off the MMU. And any
> > > > > access to code or data will be with Device-nGnRnE as you pointed out. If we
> > > > > don't clean and invalidate, instructions and data might be in the cache and
> > > > > we will be mixing memory attributes, won't we?
> > > > 
> > > > I missed that comment, sorry. I've replied to that comment made in v2,
> > > > here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
> > > > systems that boot through EFI").
> > > > 
> > > > This is the second time you've mentioned mixed memory attributes, so I'm
> > > > going to reiterate the question I asked in patch #19: what do you mean by
> > > > "mixing memory attributes" and what is wrong with it? Because it looks to
> > > > me like you're saying that you cannot access data written with the MMU on
> > > > when the MMU is off (and I assume the other way around, you cannot data
> > > > written with the MMU off when the MMU is on).
> > > > 
> > > 
> > > What I mean by mixing memory attributes is illustrated by the following
> > > example.
> > > 
> > > Take a memory location x, for which the page table entry maps to a physical
> > > location as Normal, Inner-Shareable, Inner-writeback and Outer-writeback. If
> > > we access it when the MMU is on and subquently when the MMU is off (treated
> > > as Device-nGnRnE), then we have two accesses with mismatched memory
> > > attributes to the same location. There is a whole section in the Arm ARM on
> > > why this needs to be avoided (B2.8 Mismatched memory attributes) but the
> > > result is "a loss of the uniprocessor semantics, ordering, or coherency". As
> > > I understand, the solution to this is:
> > > 
> > > "If the mismatched attributes for a Location mean that multiple cacheable
> > > accesses to the Location might be made with different shareability
> > > attributes, then uniprocessor semantics, ordering, and coherency are
> > > guaranteed only if:
> > > • Software running on a PE cleans and invalidates a Location from cache
> > > before and after each read or write to that Location by that PE.
> > > • A DMB barrier with scope that covers the full shareability of the accesses
> > > is placed between any accesses to the same memory Location that use
> > > different attributes."
> > 
> > Ok, so this is about *mismatched* memory attributes. I searched the Arm ARM
> > for the string "mixed" and nothing relevant came up.
> > 
> > Device-whatever memory is outer shareable and kvm-unit-tests maps memory as
> > inner shareable, so that matches the "different shareability attributes"
> > part of the paragraph.
> > 
> > But I would like to point out that there is only one type of cacheable
> > access that is being performed, when the MMU is on. When the MMU is off,
> > the access is not cacheable. So there are two types of accesses being
> > performed:
> > 
> > - cacheable + inner-shareable (MMU on)
> > - non-cacheable + outer-shareable (MMU off)
> > 
> > It looks to me like the paragraph doesn't apply to our case, because there
> > are no "multiple cacheable accesses [..] made with different shareability
> > attributes". Do you agree?
> > 
> > > 
> > > So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
> > > will try to find out more about UEFI's page tables.
> > 
> > That's important to know, especially regarding the text section of the
> > image. If UEFI doesnt' clean it to PoC, kvm-unit-tests must do it in order
> > to execute correctly with the MMU off.
> >
> 
> Hi Alex and Nikos,
> 
> Indeed my experiments on bare-metal made this change necessary. I'm happy
> to see this discussion, though, as this patch could be tweaked or at least
> the commit message improved in order to better explain what's going on and
> why the changes are necessary. IOW, I have no problem with this patch

If you fix the commit message to be architecturally correct then you will
come to the conclusion that the patch is architecturally incorrect because
while it fixes the problem you were seeing, it breaks asm_mmu_disable in
all other cases.

The problem you were seeing according to my investigation was this:

__phys_offset and __phys_end are written with the MMU on and the most up to
date value is in the cache. When the MMU is turned off, the value that
asm_mmu_disable reads is the stale value from main memory and it will not
clean + invalidate all the memory, which is what we want. This assumes that
UEFI cleaned the image to PoC, otherwise, that will need to be cleaned too
by kvm-unit-tests before turning off the MMU.

This was explained before, both on your original UEFI support series on
github [1], and on this list.

As for why it breaks asm_mmu_disable for all other cases:

The purpose of the clean in asm_mmu_disable is for the CPU to sync the
caches with main memory when the MMU is turned off (to propagate the most
up-to-date value from the cache to main memory); the purpose of the
invalidate is to make sure that the CPU reads from main memory instead of
the cache once the MMU is turned back on - if the cache line is still
valid, the CPU wll read the values written *before* the MMU was turned
off, not the values written *after* the MMU was turned off.

If you do the dcache clean + invalidate *before* turning the MMU on, the
CPU can speculate a read and allocate a new cache line before the MMU is
turned off, which would make the invalidate useless. Speculation is
prohibited with the MMU off, that's why the invalidate must be done with
the MMU off.

Because of this reason I believe the patch is incorrect.

> being dropped and replaced by one of you with something that "makes
> more sense" as long as the outcome (coherent execution on bare-metal)
> still works.

Hmm... maybe an experiment will work. I propose the following:

1. Revert this patch.
2. Apply this diff on top of the series:

diff --git a/lib/arm/setup.c b/lib/arm/setup.c
index 30d04d0eb100..913f4088d96c 100644
--- a/lib/arm/setup.c
+++ b/lib/arm/setup.c
@@ -374,6 +374,11 @@ static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
                }
        }
        __phys_end &= PHYS_MASK;
+
+       asm volatile("dc cvau, %0\n" :: "r" (&__phys_offset) : "memory");
+       asm volatile("dc cvau, %0\n" :: "r" (&__phys_end) : "memory");
+       dsb(sy);
+
        asm_mmu_disable();

        if (free_mem_pages == 0)

This is the solution, based on an architectural explanation of what we were
observing, that I proposed on your github branch, a solution that you've
tested with the result:

"I tested at least 10 times (lost count) with a build where "arm/arm64:
mmu_disable: Clean and invalidate before disabling" was reverted from the
target-efi branch and your hack was applied. It worked every time."

[1] https://github.com/rhdrjones/kvm-unit-tests/commit/fc58684bc47b7d07d75098fdfddb6083e9b12104#commitcomment-44222926

Thanks,
Alex

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Andrew Jones]

On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
...
> So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
> will try to find out more about UEFI's page tables.
>

UEFI will map memory as normal-cacheable. You may find section 2.3.6 of
the UEFI spec informative.

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 26/27] lib: arm: Print test exit status

[Andrew Jones]

On Thu, Jun 30, 2022 at 11:03:23AM +0100, Nikos Nikoleris wrote:
> From: Alexandru Elisei <alexandru.elisei@arm.com>
> 
> The arm tests can be run under kvmtool, which doesn't emulate a chr-testdev
> device. Print the test exit status to make it possible for the runner
> scripts to pick it up when they have support for it.
> 
> Signed-off-by: Alexandru Elisei <alexandru.elisei@arm.com>
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/arm/io.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/lib/arm/io.c b/lib/arm/io.c
> index a91f116..337cf1b 100644
> --- a/lib/arm/io.c
> +++ b/lib/arm/io.c
> @@ -138,6 +138,12 @@ extern void halt(int code);
>  
>  void exit(int code)
>  {
> +	/*
> +	 * Print the test return code in the format used by chr-testdev so the
> +	 * runner can pick it up if there is chr-testdev is not present.

nit: The comment isn't worded quite right...

The printed format ("EXIT: STATUS=") isn't chr-testdev's format, but it
is the format we want, because it's consistent with powerpc and s390x.
The exit code format '(code << 1) | 1' is chr-testdev's and
isa-debug-exit's exit format.

> +	 */
> +	printf("\nEXIT: STATUS=%d\n", ((code) << 1) | 1);
> +
>  	chr_testdev_exit(code);
>  	psci_system_off();
>  	halt(code);
> -- 
> 2.25.1
>

Anyway,

Reviewed-by: Andrew Jones <andrew.jones@linux.dev>

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Andrew Jones]

On Fri, Jul 01, 2022 at 11:24:44AM +0100, Alexandru Elisei wrote:
...
> > being dropped and replaced by one of you with something that "makes
> > more sense" as long as the outcome (coherent execution on bare-metal)
> > still works.
> 
> Hmm... maybe an experiment will work. I propose the following:
> 
> 1. Revert this patch.
> 2. Apply this diff on top of the series:
> 
> diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> index 30d04d0eb100..913f4088d96c 100644
> --- a/lib/arm/setup.c
> +++ b/lib/arm/setup.c
> @@ -374,6 +374,11 @@ static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
>                 }
>         }
>         __phys_end &= PHYS_MASK;
> +
> +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_offset) : "memory");
> +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_end) : "memory");
> +       dsb(sy);
> +
>         asm_mmu_disable();
> 
>         if (free_mem_pages == 0)
> 
> This is the solution, based on an architectural explanation of what we were
> observing, that I proposed on your github branch, a solution that you've
> tested with the result:
> 
> "I tested at least 10 times (lost count) with a build where "arm/arm64:
> mmu_disable: Clean and invalidate before disabling" was reverted from the
> target-efi branch and your hack was applied. It worked every time."
> 
> [1] https://github.com/rhdrjones/kvm-unit-tests/commit/fc58684bc47b7d07d75098fdfddb6083e9b12104#commitcomment-44222926
>

Hi Alex,

Thanks for digging that back up. I had lost track of it. The last comment
is you saying that you'll send a proper patch. Did you send one that got
lost? If not, would you like to send one now that Nikos can incorporate?

Thanks,
drew

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Nikos Nikoleris]

On 01/07/2022 11:24, Alexandru Elisei wrote:
> Hi,
> 
> On Fri, Jul 01, 2022 at 11:12:14AM +0200, Andrew Jones wrote:
>> On Thu, Jun 30, 2022 at 04:57:39PM +0100, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
>>>> Hi Alex,
>>>>
>>>> On 30/06/2022 12:24, Alexandru Elisei wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
>>>>>> Hi Alex,
>>>>>>
>>>>>> On 30/06/2022 11:20, Alexandru Elisei wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
>>>>>>>> From: Andrew Jones <drjones@redhat.com>
>>>>>>>>
>>>>>>>> The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
>>>>>>>> clean + invalidate after turning MMU off") justifies cleaning and
>>>>>>>> invalidating the dcache after disabling the MMU by saying it's nice
>>>>>>>> not to rely on the current page tables and that it should still work
>>>>>>>> (per the spec), as long as there's an identity map in the current
>>>>>>>> tables. Doing the invalidation after also somewhat helped with
>>>>>>>> reenabling the MMU without seeing stale data, but the real problem
>>>>>>>> with reenabling was because the cache needs to be disabled with
>>>>>>>> the MMU, but it wasn't.
>>>>>>>>
>>>>>>>> Since we have to trust/validate that the current page tables have an
>>>>>>>> identity map anyway, then there's no harm in doing the clean
>>>>>>>> and invalidate first (it feels a little better to do so, anyway,
>>>>>>>> considering the cache maintenance instructions take virtual
>>>>>>>> addresses). Then, also disable the cache with the MMU to avoid
>>>>>>>> problems when reenabling. We invalidate the Icache and disable
>>>>>>>> that too for good measure. And, a final TLB invalidation ensures
>>>>>>>> we're crystal clean when we return from asm_mmu_disable().
>>>>>>>
>>>>>>> I'll point you to my previous reply [1] to this exact patch which explains
>>>>>>> why it's incorrect and is only papering over another problem.
>>>>>>>
>>>>>>> [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
>>>>>>>
>>>>>>
>>>>>> Apologies, I didn't mean to ignore your feedback on this. There was a
>>>>>> parallel discussion in [2] which I thought makes the problem more concrete.
>>>>>
>>>>> No problem, I figured as much :).
>>>>>
>>>>>>
>>>>>> This is Drew's patch as soon as he confirms he's also happy with the change
>>>>>> you suggested in the patch description I am happy to make it.
>>>>>>
>>>>>> Generally, a test will start off with the MMU enabled. At this point, we
>>>>>> access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
>>>>>> two regions could be mapped as any type of memory (I need to have another
>>>>>> look to confirm if it's Normal Memory). Then we want to take over control of
>>>>>> the page tables and for that reason we have to switch off the MMU. And any
>>>>>> access to code or data will be with Device-nGnRnE as you pointed out. If we
>>>>>> don't clean and invalidate, instructions and data might be in the cache and
>>>>>> we will be mixing memory attributes, won't we?
>>>>>
>>>>> I missed that comment, sorry. I've replied to that comment made in v2,
>>>>> here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
>>>>> systems that boot through EFI").
>>>>>
>>>>> This is the second time you've mentioned mixed memory attributes, so I'm
>>>>> going to reiterate the question I asked in patch #19: what do you mean by
>>>>> "mixing memory attributes" and what is wrong with it? Because it looks to
>>>>> me like you're saying that you cannot access data written with the MMU on
>>>>> when the MMU is off (and I assume the other way around, you cannot data
>>>>> written with the MMU off when the MMU is on).
>>>>>
>>>>
>>>> What I mean by mixing memory attributes is illustrated by the following
>>>> example.
>>>>
>>>> Take a memory location x, for which the page table entry maps to a physical
>>>> location as Normal, Inner-Shareable, Inner-writeback and Outer-writeback. If
>>>> we access it when the MMU is on and subquently when the MMU is off (treated
>>>> as Device-nGnRnE), then we have two accesses with mismatched memory
>>>> attributes to the same location. There is a whole section in the Arm ARM on
>>>> why this needs to be avoided (B2.8 Mismatched memory attributes) but the
>>>> result is "a loss of the uniprocessor semantics, ordering, or coherency". As
>>>> I understand, the solution to this is:
>>>>
>>>> "If the mismatched attributes for a Location mean that multiple cacheable
>>>> accesses to the Location might be made with different shareability
>>>> attributes, then uniprocessor semantics, ordering, and coherency are
>>>> guaranteed only if:
>>>> • Software running on a PE cleans and invalidates a Location from cache
>>>> before and after each read or write to that Location by that PE.
>>>> • A DMB barrier with scope that covers the full shareability of the accesses
>>>> is placed between any accesses to the same memory Location that use
>>>> different attributes."
>>>
>>> Ok, so this is about *mismatched* memory attributes. I searched the Arm ARM
>>> for the string "mixed" and nothing relevant came up.
>>>
>>> Device-whatever memory is outer shareable and kvm-unit-tests maps memory as
>>> inner shareable, so that matches the "different shareability attributes"
>>> part of the paragraph.
>>>
>>> But I would like to point out that there is only one type of cacheable
>>> access that is being performed, when the MMU is on. When the MMU is off,
>>> the access is not cacheable. So there are two types of accesses being
>>> performed:
>>>
>>> - cacheable + inner-shareable (MMU on)
>>> - non-cacheable + outer-shareable (MMU off)
>>>
>>> It looks to me like the paragraph doesn't apply to our case, because there
>>> are no "multiple cacheable accesses [..] made with different shareability
>>> attributes". Do you agree?
>>>

No, I think the rules on mismatched memory attributes still apply.

"Physical memory locations are accessed with mismatched attributes if 
all accesses to the location do not use a common definition of all of 
the following attributes of that location:
• Memory type: Device-nGnRnE, Device-nGnRE, Device-nGRE, Device-GRE or 
Normal."

When we're switching off the MMU, we're changing the type of memory for 
many of the efi regions and therefore the rules on mismatched memory 
attributes still apply.

>>>>
>>>> So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
>>>> will try to find out more about UEFI's page tables.
>>>
>>> That's important to know, especially regarding the text section of the
>>> image. If UEFI doesnt' clean it to PoC, kvm-unit-tests must do it in order
>>> to execute correctly with the MMU off.
>>>

Drew confirmed that UEFI page tables map most memory as Normal, Cacheable.

>>
>> Hi Alex and Nikos,
>>
>> Indeed my experiments on bare-metal made this change necessary. I'm happy
>> to see this discussion, though, as this patch could be tweaked or at least
>> the commit message improved in order to better explain what's going on and
>> why the changes are necessary. IOW, I have no problem with this patch
> 
> If you fix the commit message to be architecturally correct then you will
> come to the conclusion that the patch is architecturally incorrect because
> while it fixes the problem you were seeing, it breaks asm_mmu_disable in
> all other cases.
> > The problem you were seeing according to my investigation was this:
> 
> __phys_offset and __phys_end are written with the MMU on and the most up to
> date value is in the cache. When the MMU is turned off, the value that
> asm_mmu_disable reads is the stale value from main memory and it will not
> clean + invalidate all the memory, which is what we want. This assumes that
> UEFI cleaned the image to PoC, otherwise, that will need to be cleaned too
> by kvm-unit-tests before turning off the MMU.

The clean and/or invalidate instructions are also affected by the Memory 
type attributes so any operation after we switch off the MMU will be 
affected. Having said that, we might be fine:

"For Device memory and Normal memory that is Inner Non-cacheable, Outer 
Non-cacheable, these instructions must affect the caches of all PEs in 
the Outer Shareable shareability domain of the PE on which the 
instruction is operating."

> 
> This was explained before, both on your original UEFI support series on
> github [1], and on this list.
> 
> As for why it breaks asm_mmu_disable for all other cases:
> 
> The purpose of the clean in asm_mmu_disable is for the CPU to sync the
> caches with main memory when the MMU is turned off (to propagate the most
> up-to-date value from the cache to main memory); the purpose of the
> invalidate is to make sure that the CPU reads from main memory instead of
> the cache once the MMU is turned back on - if the cache line is still
> valid, the CPU wll read the values written *before* the MMU was turned
> off, not the values written *after* the MMU was turned off.
> 
> If you do the dcache clean + invalidate *before* turning the MMU on, the
> CPU can speculate a read and allocate a new cache line before the MMU is
> turned off, which would make the invalidate useless. Speculation is
> prohibited with the MMU off, that's why the invalidate must be done with
> the MMU off.
> 
> Because of this reason I believe the patch is incorrect.
> 

Given the effect of the DC instructions for Device Memory maybe keeping 
the invalidation after switching the MMU off seems to be the right 
approach. We only need to clean anything we use during the clean and 
invalidate as you suggest.

>> being dropped and replaced by one of you with something that "makes
>> more sense" as long as the outcome (coherent execution on bare-metal)
>> still works.
> 
> Hmm... maybe an experiment will work. I propose the following:
> > 1. Revert this patch.
> 2. Apply this diff on top of the series:
> 
> diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> index 30d04d0eb100..913f4088d96c 100644
> --- a/lib/arm/setup.c
> +++ b/lib/arm/setup.c
> @@ -374,6 +374,11 @@ static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
>                  }
>          }
>          __phys_end &= PHYS_MASK;
> +
> +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_offset) : "memory");
> +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_end) : "memory");

These need to be dc cvac.

> +       dsb(sy);
> +
>          asm_mmu_disable();
> 
>          if (free_mem_pages == 0)
> 
> This is the solution, based on an architectural explanation of what we were
> observing, that I proposed on your github branch, a solution that you've
> tested with the result:
> 
> "I tested at least 10 times (lost count) with a build where "arm/arm64:
> mmu_disable: Clean and invalidate before disabling" was reverted from the
> target-efi branch and your hack was applied. It worked every time."
>

FWIW, I don't think running 10 times on one machine shows much about the 
architectural correctness of either solution.

Thanks,

Nikos

> [1] https://github.com/rhdrjones/kvm-unit-tests/commit/fc58684bc47b7d07d75098fdfddb6083e9b12104#commitcomment-44222926
> 
> Thanks,
> Alex

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi,

On Fri, Jul 01, 2022 at 12:34:43PM +0100, Nikos Nikoleris wrote:
> On 01/07/2022 11:24, Alexandru Elisei wrote:
> > Hi,
> > 
> > On Fri, Jul 01, 2022 at 11:12:14AM +0200, Andrew Jones wrote:
> > > On Thu, Jun 30, 2022 at 04:57:39PM +0100, Alexandru Elisei wrote:
> > > > Hi,
> > > > 
> > > > On Thu, Jun 30, 2022 at 04:16:09PM +0100, Nikos Nikoleris wrote:
> > > > > Hi Alex,
> > > > > 
> > > > > On 30/06/2022 12:24, Alexandru Elisei wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > On Thu, Jun 30, 2022 at 12:08:41PM +0100, Nikos Nikoleris wrote:
> > > > > > > Hi Alex,
> > > > > > > 
> > > > > > > On 30/06/2022 11:20, Alexandru Elisei wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > On Thu, Jun 30, 2022 at 11:03:12AM +0100, Nikos Nikoleris wrote:
> > > > > > > > > From: Andrew Jones <drjones@redhat.com>
> > > > > > > > > 
> > > > > > > > > The commit message of commit 410b3bf09e76 ("arm/arm64: Perform dcache
> > > > > > > > > clean + invalidate after turning MMU off") justifies cleaning and
> > > > > > > > > invalidating the dcache after disabling the MMU by saying it's nice
> > > > > > > > > not to rely on the current page tables and that it should still work
> > > > > > > > > (per the spec), as long as there's an identity map in the current
> > > > > > > > > tables. Doing the invalidation after also somewhat helped with
> > > > > > > > > reenabling the MMU without seeing stale data, but the real problem
> > > > > > > > > with reenabling was because the cache needs to be disabled with
> > > > > > > > > the MMU, but it wasn't.
> > > > > > > > > 
> > > > > > > > > Since we have to trust/validate that the current page tables have an
> > > > > > > > > identity map anyway, then there's no harm in doing the clean
> > > > > > > > > and invalidate first (it feels a little better to do so, anyway,
> > > > > > > > > considering the cache maintenance instructions take virtual
> > > > > > > > > addresses). Then, also disable the cache with the MMU to avoid
> > > > > > > > > problems when reenabling. We invalidate the Icache and disable
> > > > > > > > > that too for good measure. And, a final TLB invalidation ensures
> > > > > > > > > we're crystal clean when we return from asm_mmu_disable().
> > > > > > > > 
> > > > > > > > I'll point you to my previous reply [1] to this exact patch which explains
> > > > > > > > why it's incorrect and is only papering over another problem.
> > > > > > > > 
> > > > > > > > [1] https://lore.kernel.org/all/Yn5Z6Kyj62cUNgRN@monolith.localdoman/
> > > > > > > > 
> > > > > > > 
> > > > > > > Apologies, I didn't mean to ignore your feedback on this. There was a
> > > > > > > parallel discussion in [2] which I thought makes the problem more concrete.
> > > > > > 
> > > > > > No problem, I figured as much :).
> > > > > > 
> > > > > > > 
> > > > > > > This is Drew's patch as soon as he confirms he's also happy with the change
> > > > > > > you suggested in the patch description I am happy to make it.
> > > > > > > 
> > > > > > > Generally, a test will start off with the MMU enabled. At this point, we
> > > > > > > access code, use and modify data (EfiLoaderData, EfiLoaderCode). Any of the
> > > > > > > two regions could be mapped as any type of memory (I need to have another
> > > > > > > look to confirm if it's Normal Memory). Then we want to take over control of
> > > > > > > the page tables and for that reason we have to switch off the MMU. And any
> > > > > > > access to code or data will be with Device-nGnRnE as you pointed out. If we
> > > > > > > don't clean and invalidate, instructions and data might be in the cache and
> > > > > > > we will be mixing memory attributes, won't we?
> > > > > > 
> > > > > > I missed that comment, sorry. I've replied to that comment made in v2,
> > > > > > here, in this ieration, in patch #19 ("arm/arm64: Add a setup sequence for
> > > > > > systems that boot through EFI").
> > > > > > 
> > > > > > This is the second time you've mentioned mixed memory attributes, so I'm
> > > > > > going to reiterate the question I asked in patch #19: what do you mean by
> > > > > > "mixing memory attributes" and what is wrong with it? Because it looks to
> > > > > > me like you're saying that you cannot access data written with the MMU on
> > > > > > when the MMU is off (and I assume the other way around, you cannot data
> > > > > > written with the MMU off when the MMU is on).
> > > > > > 
> > > > > 
> > > > > What I mean by mixing memory attributes is illustrated by the following
> > > > > example.
> > > > > 
> > > > > Take a memory location x, for which the page table entry maps to a physical
> > > > > location as Normal, Inner-Shareable, Inner-writeback and Outer-writeback. If
> > > > > we access it when the MMU is on and subquently when the MMU is off (treated
> > > > > as Device-nGnRnE), then we have two accesses with mismatched memory
> > > > > attributes to the same location. There is a whole section in the Arm ARM on
> > > > > why this needs to be avoided (B2.8 Mismatched memory attributes) but the
> > > > > result is "a loss of the uniprocessor semantics, ordering, or coherency". As
> > > > > I understand, the solution to this is:
> > > > > 
> > > > > "If the mismatched attributes for a Location mean that multiple cacheable
> > > > > accesses to the Location might be made with different shareability
> > > > > attributes, then uniprocessor semantics, ordering, and coherency are
> > > > > guaranteed only if:
> > > > > • Software running on a PE cleans and invalidates a Location from cache
> > > > > before and after each read or write to that Location by that PE.
> > > > > • A DMB barrier with scope that covers the full shareability of the accesses
> > > > > is placed between any accesses to the same memory Location that use
> > > > > different attributes."
> > > > 
> > > > Ok, so this is about *mismatched* memory attributes. I searched the Arm ARM
> > > > for the string "mixed" and nothing relevant came up.
> > > > 
> > > > Device-whatever memory is outer shareable and kvm-unit-tests maps memory as
> > > > inner shareable, so that matches the "different shareability attributes"
> > > > part of the paragraph.
> > > > 
> > > > But I would like to point out that there is only one type of cacheable
> > > > access that is being performed, when the MMU is on. When the MMU is off,
> > > > the access is not cacheable. So there are two types of accesses being
> > > > performed:
> > > > 
> > > > - cacheable + inner-shareable (MMU on)
> > > > - non-cacheable + outer-shareable (MMU off)
> > > > 
> > > > It looks to me like the paragraph doesn't apply to our case, because there
> > > > are no "multiple cacheable accesses [..] made with different shareability
> > > > attributes". Do you agree?
> > > > 
> 
> No, I think the rules on mismatched memory attributes still apply.
> 
> "Physical memory locations are accessed with mismatched attributes if all
> accesses to the location do not use a common definition of all of the
> following attributes of that location:
> • Memory type: Device-nGnRnE, Device-nGnRE, Device-nGRE, Device-GRE or
> Normal."
> 
> When we're switching off the MMU, we're changing the type of memory for many
> of the efi regions and therefore the rules on mismatched memory attributes
> still apply.

Ok, you've successfully argued that every memory access done with the MMU
off (or on) is done with mismatched memory attributes with regard to
accesses done with the MMU on (or, respectively, off).

Taking a step back, what is the problem that this patch is supposed to
solve and how is this patch solving it, taking into account mismatched
memory attributes?

> 
> > > > > 
> > > > > So unless UEFI maps all memory as Device-nGnRnE we have to do something. I
> > > > > will try to find out more about UEFI's page tables.
> > > > 
> > > > That's important to know, especially regarding the text section of the
> > > > image. If UEFI doesnt' clean it to PoC, kvm-unit-tests must do it in order
> > > > to execute correctly with the MMU off.
> > > > 
> 
> Drew confirmed that UEFI page tables map most memory as Normal, Cacheable.

Saw that, thanks!

> 
> > > 
> > > Hi Alex and Nikos,
> > > 
> > > Indeed my experiments on bare-metal made this change necessary. I'm happy
> > > to see this discussion, though, as this patch could be tweaked or at least
> > > the commit message improved in order to better explain what's going on and
> > > why the changes are necessary. IOW, I have no problem with this patch
> > 
> > If you fix the commit message to be architecturally correct then you will
> > come to the conclusion that the patch is architecturally incorrect because
> > while it fixes the problem you were seeing, it breaks asm_mmu_disable in
> > all other cases.
> > > The problem you were seeing according to my investigation was this:
> > 
> > __phys_offset and __phys_end are written with the MMU on and the most up to
> > date value is in the cache. When the MMU is turned off, the value that
> > asm_mmu_disable reads is the stale value from main memory and it will not
> > clean + invalidate all the memory, which is what we want. This assumes that
> > UEFI cleaned the image to PoC, otherwise, that will need to be cleaned too
> > by kvm-unit-tests before turning off the MMU.
> 
> The clean and/or invalidate instructions are also affected by the Memory
> type attributes so any operation after we switch off the MMU will be
> affected. Having said that, we might be fine:
> 
> "For Device memory and Normal memory that is Inner Non-cacheable, Outer
> Non-cacheable, these instructions must affect the caches of all PEs in the
> Outer Shareable shareability domain of the PE on which the instruction is
> operating."
> 
> > 
> > This was explained before, both on your original UEFI support series on
> > github [1], and on this list.
> > 
> > As for why it breaks asm_mmu_disable for all other cases:
> > 
> > The purpose of the clean in asm_mmu_disable is for the CPU to sync the
> > caches with main memory when the MMU is turned off (to propagate the most
> > up-to-date value from the cache to main memory); the purpose of the
> > invalidate is to make sure that the CPU reads from main memory instead of
> > the cache once the MMU is turned back on - if the cache line is still
> > valid, the CPU wll read the values written *before* the MMU was turned
> > off, not the values written *after* the MMU was turned off.
> > 
> > If you do the dcache clean + invalidate *before* turning the MMU on, the
> > CPU can speculate a read and allocate a new cache line before the MMU is
> > turned off, which would make the invalidate useless. Speculation is
> > prohibited with the MMU off, that's why the invalidate must be done with
> > the MMU off.
> > 
> > Because of this reason I believe the patch is incorrect.
> > 
> 
> Given the effect of the DC instructions for Device Memory maybe keeping the
> invalidation after switching the MMU off seems to be the right approach. We
> only need to clean anything we use during the clean and invalidate as you
> suggest.
> 
> > > being dropped and replaced by one of you with something that "makes
> > > more sense" as long as the outcome (coherent execution on bare-metal)
> > > still works.
> > 
> > Hmm... maybe an experiment will work. I propose the following:
> > > 1. Revert this patch.
> > 2. Apply this diff on top of the series:
> > 
> > diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> > index 30d04d0eb100..913f4088d96c 100644
> > --- a/lib/arm/setup.c
> > +++ b/lib/arm/setup.c
> > @@ -374,6 +374,11 @@ static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
> >                  }
> >          }
> >          __phys_end &= PHYS_MASK;
> > +
> > +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_offset) : "memory");
> > +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_end) : "memory");
> 
> These need to be dc cvac.

Yes, it needs to be cleaned to PoC, not PoU, my bad.

Thanks,
Alex

> 
> > +       dsb(sy);
> > +
> >          asm_mmu_disable();
> > 
> >          if (free_mem_pages == 0)
> > 
> > This is the solution, based on an architectural explanation of what we were
> > observing, that I proposed on your github branch, a solution that you've
> > tested with the result:
> > 
> > "I tested at least 10 times (lost count) with a build where "arm/arm64:
> > mmu_disable: Clean and invalidate before disabling" was reverted from the
> > target-efi branch and your hack was applied. It worked every time."
> > 
> 
> FWIW, I don't think running 10 times on one machine shows much about the
> architectural correctness of either solution.
> 
> Thanks,
> 
> Nikos
> 
> > [1] https://github.com/rhdrjones/kvm-unit-tests/commit/fc58684bc47b7d07d75098fdfddb6083e9b12104#commitcomment-44222926
> > 
> > Thanks,
> > Alex

Re: [kvm-unit-tests PATCH v3 22/27] arm64: Use code from the gnu-efi when booting with EFI

[Nikos Nikoleris]

On 01/07/2022 01:43, Ricardo Koller wrote:
> On Thu, Jun 30, 2022 at 11:03:19AM +0100, Nikos Nikoleris wrote:
>> arm/efi/crt0-efi-aarch64.S defines the header and the handover
>> sequence from EFI to a efi_main. This change includes the whole file
>> in arm/cstart64.S when we compile with EFI support.
>>
>> In addition, we change the handover code in arm/efi/crt0-efi-aarch64.S
>> to align the stack pointer. This alignment is necessary because we
>> make assumptions about cpu0's stack alignment and most importantly we
>> place its thread_info at the bottom of this stack.
>>
>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>> ---
>>   arm/cstart64.S             |  6 ++++++
>>   arm/efi/crt0-efi-aarch64.S | 21 +++++++++++++++++----
>>   2 files changed, 23 insertions(+), 4 deletions(-)
>>
>> diff --git a/arm/cstart64.S b/arm/cstart64.S
>> index 55b41ea..08cf02f 100644
>> --- a/arm/cstart64.S
>> +++ b/arm/cstart64.S
>> @@ -15,6 +15,10 @@
>>   #include <asm/thread_info.h>
>>   #include <asm/sysreg.h>
>>   
>> +#ifdef CONFIG_EFI
>> +#include "efi/crt0-efi-aarch64.S"
>> +#else
>> +
>>   .macro zero_range, tmp1, tmp2
>>   9998:	cmp	\tmp1, \tmp2
>>   	b.eq	9997f
>> @@ -107,6 +111,8 @@ start:
>>   	bl	exit
>>   	b	halt
>>   
>> +#endif
>> +
>>   .text
>>   
>>   /*
>> diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
>> index d50e78d..03d29b0 100644
>> --- a/arm/efi/crt0-efi-aarch64.S
>> +++ b/arm/efi/crt0-efi-aarch64.S
>> @@ -111,10 +111,19 @@ section_table:
>>   
>>   	.align		12
>>   _start:
>> -	stp		x29, x30, [sp, #-32]!
>> +	stp		x29, x30, [sp, #-16]!
>> +
>> +	/* Align sp; this is necessary due to way we store cpu0's thread_info */
>>   	mov		x29, sp
>> +	and		x29, x29, #THREAD_MASK
>> +	mov		x30, sp
>> +	mov		sp, x29
>> +	str		x30, [sp, #-16]!
>> +
>> +	mov             x29, sp
> 
> I wasn't sure what was this x29 for. But after some googling, this is
> what I found [0]:
> 
> 	The frame pointer (X29) should point to the previous frame pointer saved
> 	on stack, with the saved LR (X30) stored after it.
> 
> The old code ended up with x29 pointing to the right place: the previous
> (x29,x30).
> 
> 	|   ...  |
> 	|   x1   |
> 	|   x0   |
> 	|   x30  |
> x29 ->	|   x29  |
> 
> In the new code x29 is pointing to:
> 
> 	|   ...  |
> 	|   x30  |
> old_sp->|   x29  |
> 	|   ...  |
> 	|   x1   |
> 	|   x0   |
> 	|   pad  |
> x29 ->	| old_sp |
> 
> I think the new version can be fixed by setting x29 to the old_sp,
> conveniently stored in x30:

That's a good point, I'll swap x29 with x30 (x29 saves the old sp and 
x30 is used to calculate the new value for sp) to make sure that x29 
points to the right location in the stack.

Thanks,

Nikos

> 
> +	mov             x30, sp
> 
>> +
>> +	stp		x0, x1, [sp, #-16]!
>>   
>> -	stp		x0, x1, [sp, #16]
>>   	mov		x2, x0
>>   	mov		x3, x1
>>   	adr		x0, ImageBase
>> @@ -123,8 +132,12 @@ _start:
>>   	bl		_relocate
>>   	cbnz		x0, 0f
>>   
>> -	ldp		x0, x1, [sp, #16]
>> +	ldp		x0, x1, [sp], #16
>>   	bl		efi_main
>>   
>> -0:	ldp		x29, x30, [sp], #32
>> +	/* Restore sp */
>> +	ldr		x30, [sp], #16
>> +	mov             sp, x30
>> +
>> +0:	ldp		x29, x30, [sp], #16
>>   	ret
>> -- 
>> 2.25.1
>>
> 
> [0] https://developer.arm.com/documentation/den0024/a/The-ABI-for-ARM-64-bit-Architecture/Register-use-in-the-AArch64-Procedure-Call-Standard/Indirect-result-location

Re: [kvm-unit-tests PATCH v3 15/27] arm/arm64: mmu_disable: Clean and invalidate before disabling

[Alexandru Elisei]

Hi Drew,

Sorry, I lost track of this thread.

On Fri, Jul 01, 2022 at 01:16:27PM +0200, Andrew Jones wrote:
> On Fri, Jul 01, 2022 at 11:24:44AM +0100, Alexandru Elisei wrote:
> ...
> > > being dropped and replaced by one of you with something that "makes
> > > more sense" as long as the outcome (coherent execution on bare-metal)
> > > still works.
> > 
> > Hmm... maybe an experiment will work. I propose the following:
> > 
> > 1. Revert this patch.
> > 2. Apply this diff on top of the series:
> > 
> > diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> > index 30d04d0eb100..913f4088d96c 100644
> > --- a/lib/arm/setup.c
> > +++ b/lib/arm/setup.c
> > @@ -374,6 +374,11 @@ static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
> >                 }
> >         }
> >         __phys_end &= PHYS_MASK;
> > +
> > +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_offset) : "memory");
> > +       asm volatile("dc cvau, %0\n" :: "r" (&__phys_end) : "memory");
> > +       dsb(sy);
> > +
> >         asm_mmu_disable();
> > 
> >         if (free_mem_pages == 0)
> > 
> > This is the solution, based on an architectural explanation of what we were
> > observing, that I proposed on your github branch, a solution that you've
> > tested with the result:
> > 
> > "I tested at least 10 times (lost count) with a build where "arm/arm64:
> > mmu_disable: Clean and invalidate before disabling" was reverted from the
> > target-efi branch and your hack was applied. It worked every time."
> > 
> > [1] https://github.com/rhdrjones/kvm-unit-tests/commit/fc58684bc47b7d07d75098fdfddb6083e9b12104#commitcomment-44222926
> >
> 
> Hi Alex,
> 
> Thanks for digging that back up. I had lost track of it. The last comment
> is you saying that you'll send a proper patch. Did you send one that got
> lost? If not, would you like to send one now that Nikos can incorporate?

The "proper patch" that I was referring to was to skip cache maintenance
when the MMU is already off. Based on the ongoing thread with Nikos, we
might have to rethink asm_mmu_disable, so I'm waiting for a conclusion
before I make any changes to asm_mmu_disable.

Thanks,
Alex

> 
> Thanks,
> drew

Re: [kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Alexandru Elisei]

Hi,

On Thu, Jun 30, 2022 at 11:03:22AM +0100, Nikos Nikoleris wrote:
> Users can now build kvm-unit-tests as efi apps by supplying an extra
> argument when invoking configure:
> 
> $> ./configure --enable-efi
> 
> This patch is based on an earlier version by
> Andrew Jones <drjones@redhat.com>
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> Reviewed-by: Ricardo Koller <ricarkol@google.com>
> ---
>  configure           | 15 ++++++++++++---
>  arm/Makefile.arm    |  6 ++++++
>  arm/Makefile.arm64  | 18 ++++++++++++++----
>  arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
>  4 files changed, 66 insertions(+), 18 deletions(-)
> 
> diff --git a/configure b/configure
> index 5b7daac..2ff9881 100755
> --- a/configure
> +++ b/configure
[..]
> @@ -218,6 +223,10 @@ else
>          echo "arm64 doesn't support page size of $page_size"
>          usage
>      fi
> +    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
> +        echo "efi must use 4K pages"
> +        exit 1

Why this restriction?

The Makefile compiles kvm-unit-tests to run as an UEFI app, it doesn't
compile UEFI itself. As far as I can tell, UEFI is designed to run payloads
with larger page size (it would be pretty silly to not be able to boot a
kernel built for 16k or 64k pages with UEFI).

Is there some limitation that I'm missing?

Thanks,
Alex

Re: [kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Nikos Nikoleris]

Hi Alex,

On 12/07/2022 14:39, Alexandru Elisei wrote:
> Hi,
> 
> On Thu, Jun 30, 2022 at 11:03:22AM +0100, Nikos Nikoleris wrote:
>> Users can now build kvm-unit-tests as efi apps by supplying an extra
>> argument when invoking configure:
>>
>> $> ./configure --enable-efi
>>
>> This patch is based on an earlier version by
>> Andrew Jones <drjones@redhat.com>
>>
>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>> Reviewed-by: Ricardo Koller <ricarkol@google.com>
>> ---
>>   configure           | 15 ++++++++++++---
>>   arm/Makefile.arm    |  6 ++++++
>>   arm/Makefile.arm64  | 18 ++++++++++++++----
>>   arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
>>   4 files changed, 66 insertions(+), 18 deletions(-)
>>
>> diff --git a/configure b/configure
>> index 5b7daac..2ff9881 100755
>> --- a/configure
>> +++ b/configure
> [..]
>> @@ -218,6 +223,10 @@ else
>>           echo "arm64 doesn't support page size of $page_size"
>>           usage
>>       fi
>> +    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
>> +        echo "efi must use 4K pages"
>> +        exit 1
> 
> Why this restriction?
> 
> The Makefile compiles kvm-unit-tests to run as an UEFI app, it doesn't
> compile UEFI itself. As far as I can tell, UEFI is designed to run payloads
> with larger page size (it would be pretty silly to not be able to boot a
> kernel built for 16k or 64k pages with UEFI).
> 
> Is there some limitation that I'm missing?
> 

Technically, we could allow 16k or 64k granules. But to do that we would 
have to handle cases where the memory map we get from EFI cannot be 
remapped with the new granules. For example, a region might be 12kB and 
mapping it with 16k or 64k granules without moving it is impossible.

Thanks,

Nikos

> Thanks,
> Alex

Re: [kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Alexandru Elisei]

Hi,

On Tue, Jul 12, 2022 at 09:50:51PM +0100, Nikos Nikoleris wrote:
> Hi Alex,
> 
> On 12/07/2022 14:39, Alexandru Elisei wrote:
> > Hi,
> > 
> > On Thu, Jun 30, 2022 at 11:03:22AM +0100, Nikos Nikoleris wrote:
> > > Users can now build kvm-unit-tests as efi apps by supplying an extra
> > > argument when invoking configure:
> > > 
> > > $> ./configure --enable-efi
> > > 
> > > This patch is based on an earlier version by
> > > Andrew Jones <drjones@redhat.com>
> > > 
> > > Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> > > Reviewed-by: Ricardo Koller <ricarkol@google.com>
> > > ---
> > >   configure           | 15 ++++++++++++---
> > >   arm/Makefile.arm    |  6 ++++++
> > >   arm/Makefile.arm64  | 18 ++++++++++++++----
> > >   arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
> > >   4 files changed, 66 insertions(+), 18 deletions(-)
> > > 
> > > diff --git a/configure b/configure
> > > index 5b7daac..2ff9881 100755
> > > --- a/configure
> > > +++ b/configure
> > [..]
> > > @@ -218,6 +223,10 @@ else
> > >           echo "arm64 doesn't support page size of $page_size"
> > >           usage
> > >       fi
> > > +    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
> > > +        echo "efi must use 4K pages"
> > > +        exit 1
> > 
> > Why this restriction?
> > 
> > The Makefile compiles kvm-unit-tests to run as an UEFI app, it doesn't
> > compile UEFI itself. As far as I can tell, UEFI is designed to run payloads
> > with larger page size (it would be pretty silly to not be able to boot a
> > kernel built for 16k or 64k pages with UEFI).
> > 
> > Is there some limitation that I'm missing?
> > 
> 
> Technically, we could allow 16k or 64k granules. But to do that we would
> have to handle cases where the memory map we get from EFI cannot be remapped
> with the new granules. For example, a region might be 12kB and mapping it
> with 16k or 64k granules without moving it is impossible.

Hm... From UEFI Specification, Version 2.8, page 35:

"The ARM architecture allows mapping pages at a variety of granularities,
including 4KiB and 64KiB. If a 64KiB physical page contains any 4KiB page
with any of the following types listed below, then all 4KiB pages in the
64KiB page must use identical ARM Memory Page Attributes (as described in
Table 7):

— EfiRuntimeServicesCode
— EfiRuntimeServicesData
— EfiReserved
— EfiACPIMemoryNVS

Mixed attribute mappings within a larger page are not allowed.

Note: This constraint allows a 64K paged based Operating System to safely
map runtime services memory."

Looking at Table 30. Memory Type Usage after ExitBootServices(), on page
160 (I am going to assume that EfiReservedMemoryType is the same as
EfiReserved), the only region that is required to be mapped for runtime
services, but isn't mentioned above, is EfiPalCode. The bit about mixed
attribute mappings within a larger page not being allowed makes me think
that EfiPalCode can be mapped even if isn't mapped at the start of a 64KiB
page, as no other memory type can be withing a 64KiB granule. What do you
think?

There's no pressing need to have support for all page sizes, from my point
of view, it's fine if it's missing from the initial UEFI support. But I
would appreciate a comment in the code or an explanation in the commit
message (or both), because it looks very arbitrary as it is right now. At
the very least this will serve as a nice reminder of what still needs to be
done for full UEFI support.

Thanks,
Alex

> 
> Thanks,
> 
> Nikos
> 
> > Thanks,
> > Alex

Re: [kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Nikos Nikoleris]

On 13/07/2022 09:46, Alexandru Elisei wrote:
> Hi,
>
> On Tue, Jul 12, 2022 at 09:50:51PM +0100, Nikos Nikoleris wrote:
>> Hi Alex,
>>
>> On 12/07/2022 14:39, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> On Thu, Jun 30, 2022 at 11:03:22AM +0100, Nikos Nikoleris wrote:
>>>> Users can now build kvm-unit-tests as efi apps by supplying an extra
>>>> argument when invoking configure:
>>>>
>>>> $> ./configure --enable-efi
>>>>
>>>> This patch is based on an earlier version by
>>>> Andrew Jones <drjones@redhat.com>
>>>>
>>>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>>>> Reviewed-by: Ricardo Koller <ricarkol@google.com>
>>>> ---
>>>>    configure           | 15 ++++++++++++---
>>>>    arm/Makefile.arm    |  6 ++++++
>>>>    arm/Makefile.arm64  | 18 ++++++++++++++----
>>>>    arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
>>>>    4 files changed, 66 insertions(+), 18 deletions(-)
>>>>
>>>> diff --git a/configure b/configure
>>>> index 5b7daac..2ff9881 100755
>>>> --- a/configure
>>>> +++ b/configure
>>> [..]
>>>> @@ -218,6 +223,10 @@ else
>>>>            echo "arm64 doesn't support page size of $page_size"
>>>>            usage
>>>>        fi
>>>> +    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
>>>> +        echo "efi must use 4K pages"
>>>> +        exit 1
>>>
>>> Why this restriction?
>>>
>>> The Makefile compiles kvm-unit-tests to run as an UEFI app, it doesn't
>>> compile UEFI itself. As far as I can tell, UEFI is designed to run payloads
>>> with larger page size (it would be pretty silly to not be able to boot a
>>> kernel built for 16k or 64k pages with UEFI).
>>>
>>> Is there some limitation that I'm missing?
>>>
>>
>> Technically, we could allow 16k or 64k granules. But to do that we would
>> have to handle cases where the memory map we get from EFI cannot be remapped
>> with the new granules. For example, a region might be 12kB and mapping it
>> with 16k or 64k granules without moving it is impossible.
>
> Hm... From UEFI Specification, Version 2.8, page 35:
>
> "The ARM architecture allows mapping pages at a variety of granularities,
> including 4KiB and 64KiB. If a 64KiB physical page contains any 4KiB page
> with any of the following types listed below, then all 4KiB pages in the
> 64KiB page must use identical ARM Memory Page Attributes (as described in
> Table 7):
>
> — EfiRuntimeServicesCode
> — EfiRuntimeServicesData
> — EfiReserved
> — EfiACPIMemoryNVS
>
> Mixed attribute mappings within a larger page are not allowed.
>
> Note: This constraint allows a 64K paged based Operating System to safely
> map runtime services memory."
>
> Looking at Table 30. Memory Type Usage after ExitBootServices(), on page
> 160 (I am going to assume that EfiReservedMemoryType is the same as
> EfiReserved), the only region that is required to be mapped for runtime
> services, but isn't mentioned above, is EfiPalCode. The bit about mixed
> attribute mappings within a larger page not being allowed makes me think
> that EfiPalCode can be mapped even if isn't mapped at the start of a 64KiB
> page, as no other memory type can be withing a 64KiB granule. What do you
> think?
>
I wasn't aware of this. So from your explanation, it sounds like if we
have multiple regions in any 64k aligned block then it should be
possible to map them all using the same mapping?

I'll check if we can add rely on this and add some assertions.

> There's no pressing need to have support for all page sizes, from my point
> of view, it's fine if it's missing from the initial UEFI support. But I
> would appreciate a comment in the code or an explanation in the commit
> message (or both), because it looks very arbitrary as it is right now. At
> the very least this will serve as a nice reminder of what still needs to be
> done for full UEFI support.

If it's just removing the check in configure and adding assertions in
lib/arm/setup.c it shouldn't be a big problem.

Thanks,

Nikos

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

Re: [kvm-unit-tests PATCH v3 25/27] arm64: Add support for efi in Makefile

[Nikos Nikoleris]

Hi,

On 13/07/2022 10:17, Nikos Nikoleris wrote:
> On 13/07/2022 09:46, Alexandru Elisei wrote:
>> Hi,
>>
>> On Tue, Jul 12, 2022 at 09:50:51PM +0100, Nikos Nikoleris wrote:
>>> Hi Alex,
>>>
>>> On 12/07/2022 14:39, Alexandru Elisei wrote:
>>>> Hi,
>>>>
>>>> On Thu, Jun 30, 2022 at 11:03:22AM +0100, Nikos Nikoleris wrote:
>>>>> Users can now build kvm-unit-tests as efi apps by supplying an extra
>>>>> argument when invoking configure:
>>>>>
>>>>> $> ./configure --enable-efi
>>>>>
>>>>> This patch is based on an earlier version by
>>>>> Andrew Jones <drjones@redhat.com>
>>>>>
>>>>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>>>>> Reviewed-by: Ricardo Koller <ricarkol@google.com>
>>>>> ---
>>>>>     configure           | 15 ++++++++++++---
>>>>>     arm/Makefile.arm    |  6 ++++++
>>>>>     arm/Makefile.arm64  | 18 ++++++++++++++----
>>>>>     arm/Makefile.common | 45 ++++++++++++++++++++++++++++++++++-----------
>>>>>     4 files changed, 66 insertions(+), 18 deletions(-)
>>>>>
>>>>> diff --git a/configure b/configure
>>>>> index 5b7daac..2ff9881 100755
>>>>> --- a/configure
>>>>> +++ b/configure
>>>> [..]
>>>>> @@ -218,6 +223,10 @@ else
>>>>>             echo "arm64 doesn't support page size of $page_size"
>>>>>             usage
>>>>>         fi
>>>>> +    if [ "$efi" = 'y' ] && [ "$page_size" != "4096" ]; then
>>>>> +        echo "efi must use 4K pages"
>>>>> +        exit 1
>>>>
>>>> Why this restriction?
>>>>
>>>> The Makefile compiles kvm-unit-tests to run as an UEFI app, it doesn't
>>>> compile UEFI itself. As far as I can tell, UEFI is designed to run payloads
>>>> with larger page size (it would be pretty silly to not be able to boot a
>>>> kernel built for 16k or 64k pages with UEFI).
>>>>
>>>> Is there some limitation that I'm missing?
>>>>
>>>
>>> Technically, we could allow 16k or 64k granules. But to do that we would
>>> have to handle cases where the memory map we get from EFI cannot be remapped
>>> with the new granules. For example, a region might be 12kB and mapping it
>>> with 16k or 64k granules without moving it is impossible.
>>
>> Hm... From UEFI Specification, Version 2.8, page 35:
>>
>> "The ARM architecture allows mapping pages at a variety of granularities,
>> including 4KiB and 64KiB. If a 64KiB physical page contains any 4KiB page
>> with any of the following types listed below, then all 4KiB pages in the
>> 64KiB page must use identical ARM Memory Page Attributes (as described in
>> Table 7):
>>
>> — EfiRuntimeServicesCode
>> — EfiRuntimeServicesData
>> — EfiReserved
>> — EfiACPIMemoryNVS
>>
>> Mixed attribute mappings within a larger page are not allowed.
>>
>> Note: This constraint allows a 64K paged based Operating System to safely
>> map runtime services memory."
>>
>> Looking at Table 30. Memory Type Usage after ExitBootServices(), on page
>> 160 (I am going to assume that EfiReservedMemoryType is the same as
>> EfiReserved), the only region that is required to be mapped for runtime
>> services, but isn't mentioned above, is EfiPalCode. The bit about mixed
>> attribute mappings within a larger page not being allowed makes me think
>> that EfiPalCode can be mapped even if isn't mapped at the start of a 64KiB
>> page, as no other memory type can be withing a 64KiB granule. What do you
>> think?
>>
> I wasn't aware of this. So from your explanation, it sounds like if we
> have multiple regions in any 64k aligned block then it should be
> possible to map them all using the same mapping?
> 
> I'll check if we can add rely on this and add some assertions.
> 

I've been looking into this and it doesn't seem very straightforward. 
For example, in my system I run into the case where the first two 
regions as we get them from EFI are:

Region 0x40000000 - 0x48386000 type: EfiConventionalMemory
Region 0x48386000 - 0x48450000 type: EfiLoaderCode

If we map these two regions in a system with 16k granule that uses 
identity mapping (virtual address = physical address), the end of the 
1st region overlaps with the start of the 2nd region for the range 
0x48380000 - 0x48390000. While the 2 regions will have the same memory 
type, they need to have different permissions. We map 
EfiConventionalMemory as read/write but EfiLoaderCode as read only (this 
is to allow shared code between EL0 and EL1).

>> There's no pressing need to have support for all page sizes, from my point
>> of view, it's fine if it's missing from the initial UEFI support. But I
>> would appreciate a comment in the code or an explanation in the commit
>> message (or both), because it looks very arbitrary as it is right now. At
>> the very least this will serve as a nice reminder of what still needs to be
>> done for full UEFI support.
> 
> If it's just removing the check in configure and adding assertions in
> lib/arm/setup.c it shouldn't be a big problem.
> 

It seems that dealing with this limitation is not straightforward and 
unless I am missing something I would leave is as future work. What do 
you think?

Thanks,

Nikos

Re: [kvm-unit-tests PATCH v3 19/27] arm/arm64: Add a setup sequence for systems that boot through EFI

[Alexandru Elisei]

Hi,

On Thu, Jun 30, 2022 at 11:03:16AM +0100, Nikos Nikoleris wrote:
> This change implements an alternative setup sequence for the system
> when we are booting through EFI. The memory map is discovered through
> EFI boot services and devices through ACPI.
> 
> This change is based on a change initially proposed by
> Andrew Jones <drjones@redhat.com>
> 
> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
> ---
>  lib/linux/efi.h     |   1 +
>  lib/arm/asm/setup.h |   2 +
>  lib/arm/setup.c     | 181 +++++++++++++++++++++++++++++++++++++++++++-
>  arm/cstart.S        |   1 +
>  arm/cstart64.S      |   1 +
>  5 files changed, 184 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/linux/efi.h b/lib/linux/efi.h
> index 53748dd..89f9a9e 100644
> --- a/lib/linux/efi.h
> +++ b/lib/linux/efi.h
> @@ -63,6 +63,7 @@ typedef guid_t efi_guid_t;
>  	(c) & 0xff, ((c) >> 8) & 0xff, d } }
>  
>  #define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)
> +#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3,  0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)
>  
>  #define LOADED_IMAGE_PROTOCOL_GUID EFI_GUID(0x5b1b31a1, 0x9562, 0x11d2,  0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b)
>  
> diff --git a/lib/arm/asm/setup.h b/lib/arm/asm/setup.h
> index 64cd379..c4cd485 100644
> --- a/lib/arm/asm/setup.h
> +++ b/lib/arm/asm/setup.h
> @@ -6,6 +6,7 @@
>   * This work is licensed under the terms of the GNU LGPL, version 2.
>   */
>  #include <libcflat.h>
> +#include <efi.h>
>  #include <asm/page.h>
>  #include <asm/pgtable-hwdef.h>
>  
> @@ -37,5 +38,6 @@ extern unsigned int mem_region_get_flags(phys_addr_t paddr);
>  #define SMP_CACHE_BYTES		L1_CACHE_BYTES
>  
>  void setup(const void *fdt, phys_addr_t freemem_start);
> +efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo);
>  
>  #endif /* _ASMARM_SETUP_H_ */
> diff --git a/lib/arm/setup.c b/lib/arm/setup.c
> index 13513d0..30d04d0 100644
> --- a/lib/arm/setup.c
> +++ b/lib/arm/setup.c
> @@ -34,7 +34,7 @@
>  #define NR_EXTRA_MEM_REGIONS	16
>  #define NR_INITIAL_MEM_REGIONS	(MAX_DT_MEM_REGIONS + NR_EXTRA_MEM_REGIONS)
>  
> -extern unsigned long _etext;
> +extern unsigned long _text, _etext, _data, _edata;
>  
>  char *initrd;
>  u32 initrd_size;
> @@ -44,7 +44,10 @@ int nr_cpus;
>  
>  static struct mem_region __initial_mem_regions[NR_INITIAL_MEM_REGIONS + 1];
>  struct mem_region *mem_regions = __initial_mem_regions;
> -phys_addr_t __phys_offset, __phys_end;
> +phys_addr_t __phys_offset = (phys_addr_t)-1, __phys_end = 0;
> +
> +extern void exceptions_init(void);
> +extern void asm_mmu_disable(void);
>  
>  int mpidr_to_cpu(uint64_t mpidr)
>  {
> @@ -272,3 +275,177 @@ void setup(const void *fdt, phys_addr_t freemem_start)
>  	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
>  		setup_vm();
>  }
> +
> +#ifdef CONFIG_EFI
> +
> +#include <efi.h>
> +
> +static efi_status_t setup_rsdp(efi_bootinfo_t *efi_bootinfo)
> +{
> +	efi_status_t status;
> +	struct rsdp_descriptor *rsdp;
> +
> +	/*
> +	 * RSDP resides in an EFI_ACPI_RECLAIM_MEMORY region, which is not used
> +	 * by kvm-unit-tests arm64 memory allocator. So it is not necessary to
> +	 * copy the data structure to another memory region to prevent
> +	 * unintentional overwrite.
> +	 */
> +	status = efi_get_system_config_table(ACPI_20_TABLE_GUID, (void **)&rsdp);
> +	if (status != EFI_SUCCESS)
> +		return status;
> +
> +	set_efi_rsdp(rsdp);
> +
> +	return EFI_SUCCESS;
> +}
> +
> +static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
> +{
> +	int i;
> +	unsigned long free_mem_pages = 0;
> +	unsigned long free_mem_start = 0;
> +	struct efi_boot_memmap *map = &(efi_bootinfo->mem_map);
> +	efi_memory_desc_t *buffer = *map->map;
> +	efi_memory_desc_t *d = NULL;
> +	phys_addr_t base, top;
> +	struct mem_region *r;
> +	uintptr_t text = (uintptr_t)&_text, etext = __ALIGN((uintptr_t)&_etext, 4096);
> +	uintptr_t data = (uintptr_t)&_data, edata = __ALIGN((uintptr_t)&_edata, 4096);
> +
> +	/*
> +	 * Record the largest free EFI_CONVENTIONAL_MEMORY region
> +	 * which will be used to set up the memory allocator, so that
> +	 * the memory allocator can work in the largest free
> +	 * continuous memory region.
> +	 */
> +	for (i = 0, r = &mem_regions[0]; i < *(map->map_size); i += *(map->desc_size), ++r) {
> +		d = (efi_memory_desc_t *)(&((u8 *)buffer)[i]);
> +
> +		r->start = d->phys_addr;
> +		r->end = d->phys_addr + d->num_pages * EFI_PAGE_SIZE;
> +
> +		switch (d->type) {
> +		case EFI_RESERVED_TYPE:
> +		case EFI_LOADER_DATA:
> +		case EFI_BOOT_SERVICES_CODE:
> +		case EFI_BOOT_SERVICES_DATA:
> +		case EFI_RUNTIME_SERVICES_CODE:
> +		case EFI_RUNTIME_SERVICES_DATA:
> +		case EFI_UNUSABLE_MEMORY:
> +		case EFI_ACPI_RECLAIM_MEMORY:
> +		case EFI_ACPI_MEMORY_NVS:
> +		case EFI_PAL_CODE:
> +			r->flags = MR_F_RESERVED;
> +			break;
> +		case EFI_MEMORY_MAPPED_IO:
> +		case EFI_MEMORY_MAPPED_IO_PORT_SPACE:
> +			r->flags = MR_F_IO;
> +			break;
> +		case EFI_LOADER_CODE:
> +			if (r->start <= text && r->end > text) {
> +				/* This is the unit test region. Flag the code separately. */
> +				phys_addr_t tmp = r->end;
> +
> +				assert(etext <= data);
> +				assert(edata <= r->end);
> +				r->flags = MR_F_CODE;
> +				r->end = data;
> +				++r;
> +				r->start = data;
> +				r->end = tmp;
> +			} else {
> +				r->flags = MR_F_RESERVED;
> +			}
> +			break;
> +		case EFI_CONVENTIONAL_MEMORY:
> +			if (free_mem_pages < d->num_pages) {
> +				free_mem_pages = d->num_pages;
> +				free_mem_start = d->phys_addr;
> +			}
> +			break;
> +		}
> +
> +		if (!(r->flags & MR_F_IO)) {
> +			if (r->start < __phys_offset)
> +				__phys_offset = r->start;
> +			if (r->end > __phys_end)
> +				__phys_end = r->end;

What happens if there are holes between __phys_offset and __phys_end? The
boot path for KVM makes sure there are no holes. Wouldn't asm_mmu_disable()
trigger a translation fault if the address is not mapped because it
corresponds to a hole in the EFI provided memory map?

What happens if the region [__phys_offset, __phys_end) contains one of the
EFI reserved memory types? That's not really something that kvm-unit-tests
should be poking.

The efi boot code path changes the semantics for __phys_offset and
__phys_end, and that's a recipe for introducing bugs.

I would suggest changing __phys_offset and __phys_end to represent
something that applies to both the KVM boot path and the EFI boot path.

One idea that occured to me would be to have separate text, data and
available memory regions.  Have __phys_offset and __phys_end express the
start and end of the largest contiguous memory region, and initialize the
memory allocator from this region. That will also pave the way for handling
multiple memory regions from the DT.

Or, if you can prove and EFI_LOADER_CODE is always adjacent to
EFI_CONVENTIONAL_MEMORY, you can have __phys_offset and __phys_end describe
the region from the start of text to the end of EFI_CONVENTIONAL_MEMORY.

Thoughts? Suggestions?

Thanks,
Alex

> +		}
> +	}
> +	__phys_end &= PHYS_MASK;
> +	asm_mmu_disable();
> +
> +	if (free_mem_pages == 0)
> +		return EFI_OUT_OF_RESOURCES;
> +
> +	assert(sizeof(long) == 8 || free_mem_start < (3ul << 30));
> +
> +	phys_alloc_init(free_mem_start, free_mem_pages << EFI_PAGE_SHIFT);
> +	phys_alloc_set_minimum_alignment(SMP_CACHE_BYTES);
> +
> +	phys_alloc_get_unused(&base, &top);
> +	base = PAGE_ALIGN(base);
> +	top = top & PAGE_MASK;
> +	assert(sizeof(long) == 8 || !(base >> 32));
> +	if (sizeof(long) != 8 && (top >> 32) != 0)
> +		top = ((uint64_t)1 << 32);
> +	page_alloc_init_area(0, base >> PAGE_SHIFT, top >> PAGE_SHIFT);
> +	page_alloc_ops_enable();
> +
> +	return EFI_SUCCESS;
> +}
> +
> +efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo)
> +{
> +	efi_status_t status;
> +
> +	struct thread_info *ti = current_thread_info();
> +
> +	memset(ti, 0, sizeof(*ti));
> +
> +	exceptions_init();
> +
> +	status = efi_mem_init(efi_bootinfo);
> +	if (status != EFI_SUCCESS) {
> +		printf("Failed to initialize memory: ");
> +		switch (status) {
> +		case EFI_OUT_OF_RESOURCES:
> +			printf("No free memory region\n");
> +			break;
> +		default:
> +			printf("Unknown error\n");
> +			break;
> +		}
> +		return status;
> +	}
> +
> +	status = setup_rsdp(efi_bootinfo);
> +	if (status != EFI_SUCCESS) {
> +		printf("Cannot find RSDP in EFI system table\n");
> +		return status;
> +	}
> +
> +	psci_set_conduit();
> +	cpu_init();
> +	/* cpu_init must be called before thread_info_init */
> +	thread_info_init(current_thread_info(), 0);
> +	/* mem_init must be called before io_init */
> +	io_init();
> +
> +	timer_save_state();
> +	if (initrd) {
> +		/* environ is currently the only file in the initrd */
> +		char *env = malloc(initrd_size);
> +
> +		memcpy(env, initrd, initrd_size);
> +		setup_env(env, initrd_size);
> +	}
> +
> +	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
> +		setup_vm();
> +
> +	return EFI_SUCCESS;
> +}
> +
> +#endif
> diff --git a/arm/cstart.S b/arm/cstart.S
> index dc324c5..66a55b9 100644
> --- a/arm/cstart.S
> +++ b/arm/cstart.S
> @@ -256,6 +256,7 @@ asm_mmu_disable:
>   *
>   * Input r0 is the stack top, which is the exception stacks base
>   */
> +.globl exceptions_init
>  exceptions_init:
>  	mrc	p15, 0, r2, c1, c0, 0	@ read SCTLR
>  	bic	r2, #CR_V		@ SCTLR.V := 0
> diff --git a/arm/cstart64.S b/arm/cstart64.S
> index 390feb9..55b41ea 100644
> --- a/arm/cstart64.S
> +++ b/arm/cstart64.S
> @@ -276,6 +276,7 @@ asm_mmu_disable:
>   * Vectors
>   */
>  
> +.globl exceptions_init
>  exceptions_init:
>  	adrp	x4, vector_table
>  	add	x4, x4, :lo12:vector_table
> -- 
> 2.25.1
> 

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi,

I've been trying to test the seris and I've come across some issues.

I've been using the target-efi-upstream-v3-rebased branch.

When compiling, I encounter this error:

gcc -mstrict-align  -mno-outline-atomics -std=gnu99 -ffreestanding -O2 -I /path/to/kvm-unit-tests/lib -I /path/to/kvm-unit-tests/lib/libfdt -I lib -g -MMD -MF lib/arm/.timer.d -fno-strict-aliasing -fno-common -Wall -Wwrite-strings -Wempty-body -Wuninitialized -Wignored-qualifiers -Wno-missing-braces -Werror  -fomit-frame-pointer  -fno-stack-protector    -Wno-frame-address   -fno-pic  -no-pie  -Wclobbered  -Wunused-but-set-parameter  -Wmissing-parameter-type  -Wold-style-declaration -Woverride-init -Wmissing-prototypes -Wstrict-prototypes   -c -o lib/arm/timer.o lib/arm/timer.c
lib/arm/gic.c: In function ‘gic_init_acpi’:
lib/arm/gic.c:241:21: error: the comparison will always evaluate as ‘true’ for the address of ‘redist_base’ will never be NULL [-Werror=address]
  241 |                 if (!gicv3_data.redist_base)
      |                     ^
In file included from /path/to//kvm-unit-tests/lib/asm/gic-v3.h:1,
                 from /path/to//kvm-unit-tests/lib/asm/../../arm/asm/gic.h:43,
                 from /path/to//kvm-unit-tests/lib/asm/gic.h:1,
                 from lib/arm/gic.c:8:
/path/to//kvm-unit-tests/lib/asm/../../arm/asm/gic-v3.h:82:15: note: ‘redist_base’ declared here
   82 |         void *redist_base[NR_CPUS];
      |               ^~~~~~~~~~~

This happens with --enable-efi both set and unset (the above snippet is
from when I didn't specify --enable-efi).

For reference:

$ gcc --version
gcc (GCC) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

I managed to fix the compilation error by commenting out the
gicv3_acpi_parse_madt_gicc call:

diff --git a/lib/arm/gic.c b/lib/arm/gic.c
index 69521c3fde4f..66066ca84a96 100644
--- a/lib/arm/gic.c
+++ b/lib/arm/gic.c
@@ -179,6 +179,7 @@ static int gicv2_acpi_parse_madt_dist(struct acpi_subtable_header *header)
        return 0;
 }

+/*
 static int gicv3_acpi_parse_madt_gicc(struct acpi_subtable_header *header)
 {
        struct acpi_madt_generic_interrupt *gicc = (void *)header;
@@ -195,6 +196,7 @@ static int gicv3_acpi_parse_madt_gicc(struct acpi_subtable_header *header)

        return 0;
 }
+*/

 static int gicv3_acpi_parse_madt_dist(struct acpi_subtable_header *header)
 {
@@ -238,9 +240,11 @@ static int gic_init_acpi(void)
                                      gicv3_acpi_parse_madt_dist);
                acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
                                      gicv3_acpi_parse_madt_redist);
+               /*
                if (!gicv3_data.redist_base)
                        acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
                                              gicv3_acpi_parse_madt_gicc);
+                                             */
                acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_TRANSLATOR,
                                      gicv3_acpi_parse_madt_its);

I don't think this is the right fix, but I made the changes to get
kvm-unit-test to build.

The second error I'm encountering is when I try the selftest-setup test:

[..]
ProtectUefiImageCommon - 0x4D046040
  - 0x000000004BEC4000 - 0x000000000001F600
SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
Load address: 4bec4000
PC: 4beca400 PC offset: 6400
Unhandled exception ec=0x25 (DABT_EL1)
Vector: 4 (el1h_sync)
ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
FAR_EL1: 0000fffffffff0f8 (valid)
Exception frame registers:
pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
sp : 000000004f7ffe40
x29: 000000004f7ffff0 x28: 0000000000000000
x27: 000000004d046040 x26: 0000000000000000
x25: 0000000000000703 x24: 0000000000000050
x23: 0000000009011000 x22: 0000000000000000
x21: 000000000000001f x20: 0000fffffffff000
x19: 0000000043f92000 x18: 0000000000000000
x17: 00000000ffffa6ab x16: 000000004f513ebc
x15: 0000000000000002 x14: 000000004bed5000
x13: 000000004bee4000 x12: 000000004bed4000
x11: 000000004bec4000 x10: 000000004c03febc
x9 : 000000004bee2938 x8 : 0000000000000000
x7 : 0000000000000000 x6 : 000000004bee2900
x5 : 000000004bee2908 x4 : 0000000048000000
x3 : 0000000048000000 x2 : 000000004bee2928
x1 : 0000000000000003 x0 : ffffffffffffffff


EXIT: STATUS=127

The preceding lines were omitted for brevity, the entire log can be found
at [1] (expires in 6 months).

Command used to launch the test:

$ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"

qemu has been built from source, tag v7.0.0, configured with:

$ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf

EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
Wpa3 support in WifiConnectManager"):

$ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG

I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
were no debug symbols in the output and it was impossible to figure what is
going on.

[1] https://pastebin.com/0mcap1BU

Thanks,
Alex

On Thu, Jun 30, 2022 at 11:02:57AM +0100, Nikos Nikoleris wrote:
> Hello,
> 
> This patch series adds initial support for building arm64 tests as EFI
> tests and running them under QEMU. Much like x86_64 we import external
> dependencies from gnu-efi and adapt them to work with types and other
> assumptions from kvm-unit-tests. In addition, this series adds support
> for enumerating parts of the system using ACPI.
> 
> The first set of patches moves the existing ACPI code to the common
> lib path. Then, it extends definitions and functions to allow for more
> robust discovery of ACPI tables. In arm64, we add support for setting
> up the PSCI conduit, discovering the UART, timers and cpus via
> ACPI. The code retains existing behavior and gives priority to
> discovery through DT when one has been provided.
> 
> In the second set of patches, we add support for getting the command
> line from the EFI shell. This is a requirement for many of the
> existing arm64 tests.
> 
> In the third set of patches, we import code from gnu-efi, make minor
> changes and add an alternative setup sequence from arm64 systems that
> boot through EFI. Finally, we add support in the build system and a
> run script which is used to run an EFI app.
> 
> After this set of patches one can build arm64 EFI tests:
> 
> $> ./configure --enable-efi
> $> make
> 
> And use the run script to run an EFI tests:
> 
> $> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> 
> Or all tests:
> 
> $> ./run_tests.sh
> 
> There are a few items that this series does not address but they would
> be useful to have:
>  - Support for booting the system from EL2. Currently, we assume that a
>    tests starts running at EL1. This the case when we run with EFI, it's
>    not always the case in hardware.
>  - Support for reading environment variables and populating __envp.
>  - Support for discovering the PCI subsystem using ACPI.
>  - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.
> 
> git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased
> 
> v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/
> 
> Changes in v3:
>  - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
>  - Added support for discovering the GIC through ACPI
>  - Added a missing header file (<elf.h>)
>  - Added support for correctly parsing the outcome of tests (./run_tests)
> 
> Thanks,
> 
> Nikos
> 
> Alexandru Elisei (1):
>   lib: arm: Print test exit status
> 
> Andrew Jones (3):
>   arm/arm64: mmu_disable: Clean and invalidate before disabling
>   arm/arm64: Rename etext to _etext
>   arm64: Add a new type of memory type flag MR_F_RESERVED
> 
> Nikos Nikoleris (23):
>   lib: Fix style for acpi.{c,h}
>   x86: Avoid references to fields of ACPI tables
>   lib: Ensure all struct definition for ACPI tables are packed
>   lib: Add support for the XSDT ACPI table
>   lib: Extend the definition of the ACPI table FADT
>   devicetree: Check if fdt is NULL before returning that a DT is
>     available
>   arm/arm64: Add support for setting up the PSCI conduit through ACPI
>   arm/arm64: Add support for discovering the UART through ACPI
>   arm/arm64: Add support for timer initialization through ACPI
>   arm/arm64: Add support for cpu initialization through ACPI
>   arm/arm64: Add support for gic initialization through ACPI
>   lib/printf: Support for precision modifier in printf
>   lib/printf: Add support for printing wide strings
>   lib/efi: Add support for getting the cmdline
>   lib: Avoid ms_abi for calls related to EFI on arm64
>   arm/arm64: Add a setup sequence for systems that boot through EFI
>   arm64: Copy code from GNU-EFI
>   arm64: Change GNU-EFI imported file to use defined types
>   arm64: Use code from the gnu-efi when booting with EFI
>   lib: Avoid external dependency in libelf
>   x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
>   arm64: Add support for efi in Makefile
>   arm64: Add an efi/run script
> 
>  scripts/runtime.bash        |  14 +-
>  arm/efi/run                 |  61 +++++++
>  arm/run                     |  14 +-
>  configure                   |  15 +-
>  Makefile                    |   4 -
>  arm/Makefile.arm            |   6 +
>  arm/Makefile.arm64          |  18 +-
>  arm/Makefile.common         |  48 +++--
>  x86/Makefile.x86_64         |   4 +
>  lib/linux/efi.h             |  25 +++
>  lib/arm/asm/setup.h         |   3 +
>  lib/arm/asm/timer.h         |   2 +
>  lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
>  lib/argv.h                  |   1 +
>  lib/elf.h                   |  57 ++++++
>  lib/libcflat.h              |   1 +
>  lib/acpi.c                  | 129 ++++++++-----
>  lib/argv.c                  |   2 +-
>  lib/arm/gic.c               | 127 ++++++++++++-
>  lib/arm/io.c                |  29 ++-
>  lib/arm/mmu.c               |   4 +
>  lib/arm/psci.c              |  25 ++-
>  lib/arm/setup.c             | 247 ++++++++++++++++++++-----
>  lib/arm/timer.c             |  79 ++++++++
>  lib/devicetree.c            |   2 +-
>  lib/efi.c                   | 102 +++++++++++
>  lib/printf.c                | 194 ++++++++++++++++++--
>  arm/efi/elf_aarch64_efi.lds |  63 +++++++
>  arm/flat.lds                |   2 +-
>  arm/cstart.S                |  29 ++-
>  arm/cstart64.S              |  28 ++-
>  arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
>  arm/dummy.c                 |   4 +
>  arm/efi/reloc_aarch64.c     |  93 ++++++++++
>  arm/micro-bench.c           |   4 +-
>  arm/timer.c                 |  10 +-
>  x86/s3.c                    |  19 +-
>  x86/vmexit.c                |   2 +-
>  38 files changed, 1700 insertions(+), 258 deletions(-)
>  create mode 100755 arm/efi/run
>  create mode 100644 lib/elf.h
>  create mode 100644 lib/arm/timer.c
>  create mode 100644 arm/efi/elf_aarch64_efi.lds
>  create mode 100644 arm/efi/crt0-efi-aarch64.S
>  create mode 100644 arm/dummy.c
>  create mode 100644 arm/efi/reloc_aarch64.c
> 
> -- 
> 2.25.1
> 

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

Hi Alex,

On 19/07/2022 16:28, Alexandru Elisei wrote:
> Hi,
> 
> I've been trying to test the seris and I've come across some issues.
> 
> I've been using the target-efi-upstream-v3-rebased branch.
> 
> When compiling, I encounter this error:
> 
> gcc -mstrict-align  -mno-outline-atomics -std=gnu99 -ffreestanding -O2 -I /path/to/kvm-unit-tests/lib -I /path/to/kvm-unit-tests/lib/libfdt -I lib -g -MMD -MF lib/arm/.timer.d -fno-strict-aliasing -fno-common -Wall -Wwrite-strings -Wempty-body -Wuninitialized -Wignored-qualifiers -Wno-missing-braces -Werror  -fomit-frame-pointer  -fno-stack-protector    -Wno-frame-address   -fno-pic  -no-pie  -Wclobbered  -Wunused-but-set-parameter  -Wmissing-parameter-type  -Wold-style-declaration -Woverride-init -Wmissing-prototypes -Wstrict-prototypes   -c -o lib/arm/timer.o lib/arm/timer.c
> lib/arm/gic.c: In function ‘gic_init_acpi’:
> lib/arm/gic.c:241:21: error: the comparison will always evaluate as ‘true’ for the address of ‘redist_base’ will never be NULL [-Werror=address]
>    241 |                 if (!gicv3_data.redist_base)
>        |                     ^
> In file included from /path/to//kvm-unit-tests/lib/asm/gic-v3.h:1,
>                   from /path/to//kvm-unit-tests/lib/asm/../../arm/asm/gic.h:43,
>                   from /path/to//kvm-unit-tests/lib/asm/gic.h:1,
>                   from lib/arm/gic.c:8:
> /path/to//kvm-unit-tests/lib/asm/../../arm/asm/gic-v3.h:82:15: note: ‘redist_base’ declared here
>     82 |         void *redist_base[NR_CPUS];
>        |               ^~~~~~~~~~~
> 
> This happens with --enable-efi both set and unset (the above snippet is
> from when I didn't specify --enable-efi).
> 
> For reference:
> 
> $ gcc --version
> gcc (GCC) 12.1.0
> Copyright (C) 2022 Free Software Foundation, Inc.
> This is free software; see the source for copying conditions.  There is NO
> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> 
> I managed to fix the compilation error by commenting out the
> gicv3_acpi_parse_madt_gicc call:
> 
> diff --git a/lib/arm/gic.c b/lib/arm/gic.c
> index 69521c3fde4f..66066ca84a96 100644
> --- a/lib/arm/gic.c
> +++ b/lib/arm/gic.c
> @@ -179,6 +179,7 @@ static int gicv2_acpi_parse_madt_dist(struct acpi_subtable_header *header)
>          return 0;
>   }
> 
> +/*
>   static int gicv3_acpi_parse_madt_gicc(struct acpi_subtable_header *header)
>   {
>          struct acpi_madt_generic_interrupt *gicc = (void *)header;
> @@ -195,6 +196,7 @@ static int gicv3_acpi_parse_madt_gicc(struct acpi_subtable_header *header)
> 
>          return 0;
>   }
> +*/
> 
>   static int gicv3_acpi_parse_madt_dist(struct acpi_subtable_header *header)
>   {
> @@ -238,9 +240,11 @@ static int gic_init_acpi(void)
>                                        gicv3_acpi_parse_madt_dist);
>                  acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
>                                        gicv3_acpi_parse_madt_redist);
> +               /*
>                  if (!gicv3_data.redist_base)
>                          acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR,
>                                                gicv3_acpi_parse_madt_gicc);
> +                                             */
>                  acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_TRANSLATOR,
>                                        gicv3_acpi_parse_madt_its);
> 
> I don't think this is the right fix, but I made the changes to get
> kvm-unit-test to build.
> 

Thanks, that's obviously a bug. I have a fix and I will include it in 
the next revision. This branch: 
https://github.com/relokin/kvm-unit-tests/tree/target-efi-upstream-v4 
contains fixes for the feedback I received and I've been able to address 
so far.

> The second error I'm encountering is when I try the selftest-setup test:
> 
> [..]
> ProtectUefiImageCommon - 0x4D046040
>    - 0x000000004BEC4000 - 0x000000000001F600
> SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
> SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
> SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
> InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
> SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
> SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
> Load address: 4bec4000
> PC: 4beca400 PC offset: 6400
> Unhandled exception ec=0x25 (DABT_EL1)
> Vector: 4 (el1h_sync)
> ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
> FAR_EL1: 0000fffffffff0f8 (valid)
> Exception frame registers:
> pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
> sp : 000000004f7ffe40
> x29: 000000004f7ffff0 x28: 0000000000000000
> x27: 000000004d046040 x26: 0000000000000000
> x25: 0000000000000703 x24: 0000000000000050
> x23: 0000000009011000 x22: 0000000000000000
> x21: 000000000000001f x20: 0000fffffffff000
> x19: 0000000043f92000 x18: 0000000000000000
> x17: 00000000ffffa6ab x16: 000000004f513ebc
> x15: 0000000000000002 x14: 000000004bed5000
> x13: 000000004bee4000 x12: 000000004bed4000
> x11: 000000004bec4000 x10: 000000004c03febc
> x9 : 000000004bee2938 x8 : 0000000000000000
> x7 : 0000000000000000 x6 : 000000004bee2900
> x5 : 000000004bee2908 x4 : 0000000048000000
> x3 : 0000000048000000 x2 : 000000004bee2928
> x1 : 0000000000000003 x0 : ffffffffffffffff
> 
> 
> EXIT: STATUS=127
> 
> The preceding lines were omitted for brevity, the entire log can be found
> at [1] (expires in 6 months).
> 
> Command used to launch the test:
> 
> $ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> 
> qemu has been built from source, tag v7.0.0, configured with:
> 
> $ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf
> 
> EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
> Wpa3 support in WifiConnectManager"):
> 
> $ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG
> 
> I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
> were no debug symbols in the output and it was impossible to figure what is
> going on.
> 
> [1] https://pastebin.com/0mcap1BU

I haven't been to able to reproduce this. I've build from source qemu 
and EDK2 from source (the revisions you provided) and I've used gcc-10 
to compile KUT but selftest-smp passes.

Thanks,

Nikos

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi Nikos,

On Fri, Jul 22, 2022 at 11:57:09AM +0100, Nikos Nikoleris wrote:
> Hi Alex,
> 
> On 19/07/2022 16:28, Alexandru Elisei wrote:
> > Hi,
> > 
> > I've been trying to test the seris and I've come across some issues.
[..]
> > 
> > The second error I'm encountering is when I try the selftest-setup test:
> > 
> > [..]
> > ProtectUefiImageCommon - 0x4D046040
> >    - 0x000000004BEC4000 - 0x000000000001F600
> > SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
> > SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
> > SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
> > InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
> > SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
> > SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
> > Load address: 4bec4000
> > PC: 4beca400 PC offset: 6400
> > Unhandled exception ec=0x25 (DABT_EL1)
> > Vector: 4 (el1h_sync)
> > ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
> > FAR_EL1: 0000fffffffff0f8 (valid)
> > Exception frame registers:
> > pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
> > sp : 000000004f7ffe40
> > x29: 000000004f7ffff0 x28: 0000000000000000
> > x27: 000000004d046040 x26: 0000000000000000
> > x25: 0000000000000703 x24: 0000000000000050
> > x23: 0000000009011000 x22: 0000000000000000
> > x21: 000000000000001f x20: 0000fffffffff000
> > x19: 0000000043f92000 x18: 0000000000000000
> > x17: 00000000ffffa6ab x16: 000000004f513ebc
> > x15: 0000000000000002 x14: 000000004bed5000
> > x13: 000000004bee4000 x12: 000000004bed4000
> > x11: 000000004bec4000 x10: 000000004c03febc
> > x9 : 000000004bee2938 x8 : 0000000000000000
> > x7 : 0000000000000000 x6 : 000000004bee2900
> > x5 : 000000004bee2908 x4 : 0000000048000000
> > x3 : 0000000048000000 x2 : 000000004bee2928
> > x1 : 0000000000000003 x0 : ffffffffffffffff
> > 
> > 
> > EXIT: STATUS=127
> > 
> > The preceding lines were omitted for brevity, the entire log can be found
> > at [1] (expires in 6 months).
> > 
> > Command used to launch the test:
> > 
> > $ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> > 
> > qemu has been built from source, tag v7.0.0, configured with:
> > 
> > $ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf
> > 
> > EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
> > Wpa3 support in WifiConnectManager"):
> > 
> > $ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG
> > 
> > I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
> > were no debug symbols in the output and it was impossible to figure what is
> > going on.
> > 
> > [1] https://pastebin.com/0mcap1BU
> 
> I haven't been to able to reproduce this. I've build from source qemu and
> EDK2 from source (the revisions you provided) and I've used gcc-10 to
> compile KUT but selftest-smp passes.

That's weird, I've compiled kvm-unit-tests with gcc 10.3.0 [1] and I'm still
seeing the error (tried it on my x86 machine), for both selftest-setup
selftest-smp.

Did you compile qemu and edk2 with gcc 10.3.0? Or did you use some other
compiler?

[1] https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

Thanks,
Alex

> 
> Thanks,
> 
> Nikos
> 

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

Hi Alex,

On 22/07/2022 15:41, Alexandru Elisei wrote:
> Hi Nikos,
> 
> On Fri, Jul 22, 2022 at 11:57:09AM +0100, Nikos Nikoleris wrote:
>> Hi Alex,
>>
>> On 19/07/2022 16:28, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> I've been trying to test the seris and I've come across some issues.
> [..]
>>>
>>> The second error I'm encountering is when I try the selftest-setup test:
>>>
>>> [..]
>>> ProtectUefiImageCommon - 0x4D046040
>>>     - 0x000000004BEC4000 - 0x000000000001F600
>>> SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
>>> SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
>>> SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
>>> InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
>>> SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
>>> SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
>>> Load address: 4bec4000
>>> PC: 4beca400 PC offset: 6400
>>> Unhandled exception ec=0x25 (DABT_EL1)
>>> Vector: 4 (el1h_sync)
>>> ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
>>> FAR_EL1: 0000fffffffff0f8 (valid)
>>> Exception frame registers:
>>> pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
>>> sp : 000000004f7ffe40
>>> x29: 000000004f7ffff0 x28: 0000000000000000
>>> x27: 000000004d046040 x26: 0000000000000000
>>> x25: 0000000000000703 x24: 0000000000000050
>>> x23: 0000000009011000 x22: 0000000000000000
>>> x21: 000000000000001f x20: 0000fffffffff000
>>> x19: 0000000043f92000 x18: 0000000000000000
>>> x17: 00000000ffffa6ab x16: 000000004f513ebc
>>> x15: 0000000000000002 x14: 000000004bed5000
>>> x13: 000000004bee4000 x12: 000000004bed4000
>>> x11: 000000004bec4000 x10: 000000004c03febc
>>> x9 : 000000004bee2938 x8 : 0000000000000000
>>> x7 : 0000000000000000 x6 : 000000004bee2900
>>> x5 : 000000004bee2908 x4 : 0000000048000000
>>> x3 : 0000000048000000 x2 : 000000004bee2928
>>> x1 : 0000000000000003 x0 : ffffffffffffffff
>>>
>>>
>>> EXIT: STATUS=127
>>>
>>> The preceding lines were omitted for brevity, the entire log can be found
>>> at [1] (expires in 6 months).
>>>
>>> Command used to launch the test:
>>>
>>> $ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
>>>
>>> qemu has been built from source, tag v7.0.0, configured with:
>>>
>>> $ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf
>>>
>>> EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
>>> Wpa3 support in WifiConnectManager"):
>>>
>>> $ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG
>>>
>>> I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
>>> were no debug symbols in the output and it was impossible to figure what is
>>> going on.
>>>
>>> [1] https://pastebin.com/0mcap1BU
>>
>> I haven't been to able to reproduce this. I've build from source qemu and
>> EDK2 from source (the revisions you provided) and I've used gcc-10 to
>> compile KUT but selftest-smp passes.
> 
> That's weird, I've compiled kvm-unit-tests with gcc 10.3.0 [1] and I'm still
> seeing the error (tried it on my x86 machine), for both selftest-setup
> selftest-smp.
> 
> Did you compile qemu and edk2 with gcc 10.3.0? Or did you use some other
> compiler?
> 
> [1] https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz
> 

Thanks, I managed to reproduce this and found that with this 
configuration there are more mem_regions than the assumed max. To work 
around it you can change NR_EXTRA_MEM_REGIONS in lib/arm/setup.c to

#define NR_EXTRA_MEM_REGIONS    32

Doubling it would hopefully be enough to run tests for now but I will 
also try to find out a better way to do this.

Thanks,

Nikos

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi,

Doubling the number of memory regions worked, thanks. Like you've said,
that's just a band-aid, as there could be platforms out there for which
UEFI reports more regions than the static value that kvm-unit-tests
assumes.

If you don't mind a suggestion, you could run two passes on the UEFI memory
map: the first pass finds the largest available memory region and uses that
for initializing the memory allocators (could also count the number of
memory regions that it finds, for example), the second pass creates the
mem_regions array by allocating it dynamically (the allocators have been
initialized in the previous pass). The same approach could be used when
booting without UEFI.

Or you can just set NR_EXTRA_MEM_REGIONS to something very large and call
it a day :)

Thanks,
Alex

On Mon, Aug 01, 2022 at 07:23:05PM +0100, Nikos Nikoleris wrote:
> Hi Alex,
> 
> On 22/07/2022 15:41, Alexandru Elisei wrote:
> > Hi Nikos,
> > 
> > On Fri, Jul 22, 2022 at 11:57:09AM +0100, Nikos Nikoleris wrote:
> > > Hi Alex,
> > > 
> > > On 19/07/2022 16:28, Alexandru Elisei wrote:
> > > > Hi,
> > > > 
> > > > I've been trying to test the seris and I've come across some issues.
> > [..]
> > > > 
> > > > The second error I'm encountering is when I try the selftest-setup test:
> > > > 
> > > > [..]
> > > > ProtectUefiImageCommon - 0x4D046040
> > > >     - 0x000000004BEC4000 - 0x000000000001F600
> > > > SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
> > > > SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
> > > > SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
> > > > InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
> > > > SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
> > > > SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
> > > > Load address: 4bec4000
> > > > PC: 4beca400 PC offset: 6400
> > > > Unhandled exception ec=0x25 (DABT_EL1)
> > > > Vector: 4 (el1h_sync)
> > > > ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
> > > > FAR_EL1: 0000fffffffff0f8 (valid)
> > > > Exception frame registers:
> > > > pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
> > > > sp : 000000004f7ffe40
> > > > x29: 000000004f7ffff0 x28: 0000000000000000
> > > > x27: 000000004d046040 x26: 0000000000000000
> > > > x25: 0000000000000703 x24: 0000000000000050
> > > > x23: 0000000009011000 x22: 0000000000000000
> > > > x21: 000000000000001f x20: 0000fffffffff000
> > > > x19: 0000000043f92000 x18: 0000000000000000
> > > > x17: 00000000ffffa6ab x16: 000000004f513ebc
> > > > x15: 0000000000000002 x14: 000000004bed5000
> > > > x13: 000000004bee4000 x12: 000000004bed4000
> > > > x11: 000000004bec4000 x10: 000000004c03febc
> > > > x9 : 000000004bee2938 x8 : 0000000000000000
> > > > x7 : 0000000000000000 x6 : 000000004bee2900
> > > > x5 : 000000004bee2908 x4 : 0000000048000000
> > > > x3 : 0000000048000000 x2 : 000000004bee2928
> > > > x1 : 0000000000000003 x0 : ffffffffffffffff
> > > > 
> > > > 
> > > > EXIT: STATUS=127
> > > > 
> > > > The preceding lines were omitted for brevity, the entire log can be found
> > > > at [1] (expires in 6 months).
> > > > 
> > > > Command used to launch the test:
> > > > 
> > > > $ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> > > > 
> > > > qemu has been built from source, tag v7.0.0, configured with:
> > > > 
> > > > $ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf
> > > > 
> > > > EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
> > > > Wpa3 support in WifiConnectManager"):
> > > > 
> > > > $ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG
> > > > 
> > > > I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
> > > > were no debug symbols in the output and it was impossible to figure what is
> > > > going on.
> > > > 
> > > > [1] https://pastebin.com/0mcap1BU
> > > 
> > > I haven't been to able to reproduce this. I've build from source qemu and
> > > EDK2 from source (the revisions you provided) and I've used gcc-10 to
> > > compile KUT but selftest-smp passes.
> > 
> > That's weird, I've compiled kvm-unit-tests with gcc 10.3.0 [1] and I'm still
> > seeing the error (tried it on my x86 machine), for both selftest-setup
> > selftest-smp.
> > 
> > Did you compile qemu and edk2 with gcc 10.3.0? Or did you use some other
> > compiler?
> > 
> > [1] https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz
> > 
> 
> Thanks, I managed to reproduce this and found that with this configuration
> there are more mem_regions than the assumed max. To work around it you can
> change NR_EXTRA_MEM_REGIONS in lib/arm/setup.c to
> 
> #define NR_EXTRA_MEM_REGIONS    32
> 
> Doubling it would hopefully be enough to run tests for now but I will also
> try to find out a better way to do this.
> 
> Thanks,
> 
> Nikos

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Andrew Jones]

On Tue, Aug 02, 2022 at 11:19:00AM +0100, Alexandru Elisei wrote:
> Hi,
> 
> Doubling the number of memory regions worked, thanks. Like you've said,
> that's just a band-aid, as there could be platforms out there for which
> UEFI reports more regions than the static value that kvm-unit-tests
> assumes.
> 
> If you don't mind a suggestion, you could run two passes on the UEFI memory
> map: the first pass finds the largest available memory region and uses that
> for initializing the memory allocators (could also count the number of
> memory regions that it finds, for example), the second pass creates the
> mem_regions array by allocating it dynamically (the allocators have been
> initialized in the previous pass). The same approach could be used when
> booting without UEFI.
> 
> Or you can just set NR_EXTRA_MEM_REGIONS to something very large and call
> it a day :)

Yeah, let's just bump it to something large. We should also ensure it's
easy to debug when we hit the limit though. We have an assert() in
mem_region_add() already, but maybe we don't have an assert() in the
EFI code paths?

Thanks,
drew

> 
> Thanks,
> Alex
> 
> On Mon, Aug 01, 2022 at 07:23:05PM +0100, Nikos Nikoleris wrote:
> > Hi Alex,
> > 
> > On 22/07/2022 15:41, Alexandru Elisei wrote:
> > > Hi Nikos,
> > > 
> > > On Fri, Jul 22, 2022 at 11:57:09AM +0100, Nikos Nikoleris wrote:
> > > > Hi Alex,
> > > > 
> > > > On 19/07/2022 16:28, Alexandru Elisei wrote:
> > > > > Hi,
> > > > > 
> > > > > I've been trying to test the seris and I've come across some issues.
> > > [..]
> > > > > 
> > > > > The second error I'm encountering is when I try the selftest-setup test:
> > > > > 
> > > > > [..]
> > > > > ProtectUefiImageCommon - 0x4D046040
> > > > >     - 0x000000004BEC4000 - 0x000000000001F600
> > > > > SetUefiImageMemoryAttributes - 0x000000004BEC4000 - 0x0000000000001000 (0x0000000000004008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004BEC5000 - 0x0000000000010000 (0x0000000000020008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004BED5000 - 0x000000000000F000 (0x0000000000004008)
> > > > > InstallProtocolInterface: 752F3136-4E16-4FDC-A22A-E5F46812F4CA 4F8014E8
> > > > > SetUefiImageMemoryAttributes - 0x000000004F640000 - 0x0000000000040000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004C2D0000 - 0x0000000000040000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004C280000 - 0x0000000000040000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004C230000 - 0x0000000000040000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004C140000 - 0x0000000000040000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004F600000 - 0x0000000000030000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004C040000 - 0x0000000000030000 (0x0000000000000008)
> > > > > SetUefiImageMemoryAttributes - 0x000000004BFC0000 - 0x0000000000030000 (0x0000000000000008)
> > > > > Load address: 4bec4000
> > > > > PC: 4beca400 PC offset: 6400
> > > > > Unhandled exception ec=0x25 (DABT_EL1)
> > > > > Vector: 4 (el1h_sync)
> > > > > ESR_EL1:         96000000, ec=0x25 (DABT_EL1)
> > > > > FAR_EL1: 0000fffffffff0f8 (valid)
> > > > > Exception frame registers:
> > > > > pc : [<000000004beca400>] lr : [<000000004beca42c>] pstate: 400002c5
> > > > > sp : 000000004f7ffe40
> > > > > x29: 000000004f7ffff0 x28: 0000000000000000
> > > > > x27: 000000004d046040 x26: 0000000000000000
> > > > > x25: 0000000000000703 x24: 0000000000000050
> > > > > x23: 0000000009011000 x22: 0000000000000000
> > > > > x21: 000000000000001f x20: 0000fffffffff000
> > > > > x19: 0000000043f92000 x18: 0000000000000000
> > > > > x17: 00000000ffffa6ab x16: 000000004f513ebc
> > > > > x15: 0000000000000002 x14: 000000004bed5000
> > > > > x13: 000000004bee4000 x12: 000000004bed4000
> > > > > x11: 000000004bec4000 x10: 000000004c03febc
> > > > > x9 : 000000004bee2938 x8 : 0000000000000000
> > > > > x7 : 0000000000000000 x6 : 000000004bee2900
> > > > > x5 : 000000004bee2908 x4 : 0000000048000000
> > > > > x3 : 0000000048000000 x2 : 000000004bee2928
> > > > > x1 : 0000000000000003 x0 : ffffffffffffffff
> > > > > 
> > > > > 
> > > > > EXIT: STATUS=127
> > > > > 
> > > > > The preceding lines were omitted for brevity, the entire log can be found
> > > > > at [1] (expires in 6 months).
> > > > > 
> > > > > Command used to launch the test:
> > > > > 
> > > > > $ QEMU=/path/to/qemu/build/qemu-system-aarch64 EFI_UEFI=/path/to/QEMU_EFI.fd taskset -c 4-5 arm/efi/run arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> > > > > 
> > > > > qemu has been built from source, tag v7.0.0, configured with:
> > > > > 
> > > > > $ ./configure --target-list=aarch64-softmmu --disable-vnc --disable-gtk --disable-bpf
> > > > > 
> > > > > EDK2 image has been built from commit e1eef3a8b01a ("NetworkPkg: Add Wi-Fi
> > > > > Wpa3 support in WifiConnectManager"):
> > > > > 
> > > > > $ build -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc -b DEBUG
> > > > > 
> > > > > I tried to disassemble selftest.efi: $ objdump -d selftest.efi, but there
> > > > > were no debug symbols in the output and it was impossible to figure what is
> > > > > going on.
> > > > > 
> > > > > [1] https://pastebin.com/0mcap1BU
> > > > 
> > > > I haven't been to able to reproduce this. I've build from source qemu and
> > > > EDK2 from source (the revisions you provided) and I've used gcc-10 to
> > > > compile KUT but selftest-smp passes.
> > > 
> > > That's weird, I've compiled kvm-unit-tests with gcc 10.3.0 [1] and I'm still
> > > seeing the error (tried it on my x86 machine), for both selftest-setup
> > > selftest-smp.
> > > 
> > > Did you compile qemu and edk2 with gcc 10.3.0? Or did you use some other
> > > compiler?
> > > 
> > > [1] https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz
> > > 
> > 
> > Thanks, I managed to reproduce this and found that with this configuration
> > there are more mem_regions than the assumed max. To work around it you can
> > change NR_EXTRA_MEM_REGIONS in lib/arm/setup.c to
> > 
> > #define NR_EXTRA_MEM_REGIONS    32
> > 
> > Doubling it would hopefully be enough to run tests for now but I will also
> > try to find out a better way to do this.
> > 
> > Thanks,
> > 
> > Nikos

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

On 02/08/2022 11:46, Andrew Jones wrote:
> On Tue, Aug 02, 2022 at 11:19:00AM +0100, Alexandru Elisei wrote:
>> Hi,
>>
>> Doubling the number of memory regions worked, thanks. Like you've said,
>> that's just a band-aid, as there could be platforms out there for which
>> UEFI reports more regions than the static value that kvm-unit-tests
>> assumes.
>>
>> If you don't mind a suggestion, you could run two passes on the UEFI memory
>> map: the first pass finds the largest available memory region and uses that
>> for initializing the memory allocators (could also count the number of
>> memory regions that it finds, for example), the second pass creates the
>> mem_regions array by allocating it dynamically (the allocators have been
>> initialized in the previous pass). The same approach could be used when
>> booting without UEFI.
>>
>> Or you can just set NR_EXTRA_MEM_REGIONS to something very large and call
>> it a day :)
> 
> Yeah, let's just bump it to something large. We should also ensure it's
> easy to debug when we hit the limit though. We have an assert() in
> mem_region_add() already, but maybe we don't have an assert() in the
> EFI code paths?
> 
> Thanks,
> drew
> 

Thank you both. I'll make these two changes in the next revision of the 
series. By the way, I'm tracking changes on top of v3 here: 
https://github.com/relokin/kvm-unit-tests/tree/target-efi-upstream-v3-fixups

Thanks,

Nikos

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi,

Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
support.

This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
after performing the relocation. I'll post an abbreviated/simplified
version of efi_main() for reference:

efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
{
	/* Get image, cmdline and memory map parameters from UEFI */

        efi_exit_boot_services(handle, &efi_bootinfo.mem_map);

        /* Set up arch-specific resources */
        setup_efi(&efi_bootinfo);

        /* Run the test case */
        ret = main(__argc, __argv, __environ);

        /* Shutdown the guest VM */
        efi_exit(ret);

        /* Unreachable */
        return EFI_UNSUPPORTED;
}

Note that the assumption that efi_main() makes is that setup_efi() doesn't
change the stack from the stack that the UEFI implementation allocated, in
order for setup_efi() to be able to return to efi_main().

arm64 requires explicit data cache maintenance to keep the contents of the
caches in sync with memory when writing with MMU off/reading with MMU on
and viceversa. More details of what is needed is why here [1] and here [2].
These operations must also be performed for the stack because the stack is
always used when running C code.

What this means is that if arm64 wants to be able to run C code when the
MMU is disabled and when it is enabled, then it must perform data cache
operations for the stack memory. Which is impossible if the stack has been
allocated by UEFI, as kvm-unit-tests has no way of knowing its size, as it
isn't specified in the UEFI spec*. As a result, either efi_main needs to be
changed such that setup_efi() never returns, or arm64 must implement its
own version of efi_main() (or however it will end up being called).

One way to get around this is never to run C code with the MMU off. That's
relatively easy to do in the boot code, as the translation tables can be
constructed with the MMU on, and then a fairly small assembly sequence is
required to install them.

But arm64 also has two mechanisms for disabling the MMU:

1. At compile time, the user can request a test to start with the MMU off,
by setting the flag AUXINFO_MMU_OFF. So when booting as an UEFI app,
kvm-unit-tests must disable the MMU.

2. A function called mmu_disable() which allows a test to explicitly
disable the MMU.

If we want to keep the UEFI allocated stack, then both mechanism must be
forbidden when running under UEFI. I dislike this idea, because those two
mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
been possible with a normal operating system, which, except for the early
boot code, runs with the MMU enabled.

Any thoughts or comments about this?

*UEFI v2.8 states about the stack: "128 KiB or more of available stack
space" (page 35), but EDK2 allocates 64KiB [3]. So without any firmware
call to query the size of the stack, kvm-unit-tests cannot rely on it being
a specific size.

[1] https://lore.kernel.org/kvm/20220809091558.14379-19-alexandru.elisei@arm.com/
[2] https://lore.kernel.org/kvm/20220809091558.14379-20-alexandru.elisei@arm.com/
[3] https://github.com/tianocore/edk2/blob/master/ArmPlatformPkg/ArmPlatformPkg.dec#L71

Thanks,
Alex

On Thu, Jun 30, 2022 at 11:02:57AM +0100, Nikos Nikoleris wrote:
> Hello,
> 
> This patch series adds initial support for building arm64 tests as EFI
> tests and running them under QEMU. Much like x86_64 we import external
> dependencies from gnu-efi and adapt them to work with types and other
> assumptions from kvm-unit-tests. In addition, this series adds support
> for enumerating parts of the system using ACPI.
> 
> The first set of patches moves the existing ACPI code to the common
> lib path. Then, it extends definitions and functions to allow for more
> robust discovery of ACPI tables. In arm64, we add support for setting
> up the PSCI conduit, discovering the UART, timers and cpus via
> ACPI. The code retains existing behavior and gives priority to
> discovery through DT when one has been provided.
> 
> In the second set of patches, we add support for getting the command
> line from the EFI shell. This is a requirement for many of the
> existing arm64 tests.
> 
> In the third set of patches, we import code from gnu-efi, make minor
> changes and add an alternative setup sequence from arm64 systems that
> boot through EFI. Finally, we add support in the build system and a
> run script which is used to run an EFI app.
> 
> After this set of patches one can build arm64 EFI tests:
> 
> $> ./configure --enable-efi
> $> make
> 
> And use the run script to run an EFI tests:
> 
> $> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> 
> Or all tests:
> 
> $> ./run_tests.sh
> 
> There are a few items that this series does not address but they would
> be useful to have:
>  - Support for booting the system from EL2. Currently, we assume that a
>    tests starts running at EL1. This the case when we run with EFI, it's
>    not always the case in hardware.
>  - Support for reading environment variables and populating __envp.
>  - Support for discovering the PCI subsystem using ACPI.
>  - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.
> 
> git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased
> 
> v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/
> 
> Changes in v3:
>  - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
>  - Added support for discovering the GIC through ACPI
>  - Added a missing header file (<elf.h>)
>  - Added support for correctly parsing the outcome of tests (./run_tests)
> 
> Thanks,
> 
> Nikos
> 
> Alexandru Elisei (1):
>   lib: arm: Print test exit status
> 
> Andrew Jones (3):
>   arm/arm64: mmu_disable: Clean and invalidate before disabling
>   arm/arm64: Rename etext to _etext
>   arm64: Add a new type of memory type flag MR_F_RESERVED
> 
> Nikos Nikoleris (23):
>   lib: Fix style for acpi.{c,h}
>   x86: Avoid references to fields of ACPI tables
>   lib: Ensure all struct definition for ACPI tables are packed
>   lib: Add support for the XSDT ACPI table
>   lib: Extend the definition of the ACPI table FADT
>   devicetree: Check if fdt is NULL before returning that a DT is
>     available
>   arm/arm64: Add support for setting up the PSCI conduit through ACPI
>   arm/arm64: Add support for discovering the UART through ACPI
>   arm/arm64: Add support for timer initialization through ACPI
>   arm/arm64: Add support for cpu initialization through ACPI
>   arm/arm64: Add support for gic initialization through ACPI
>   lib/printf: Support for precision modifier in printf
>   lib/printf: Add support for printing wide strings
>   lib/efi: Add support for getting the cmdline
>   lib: Avoid ms_abi for calls related to EFI on arm64
>   arm/arm64: Add a setup sequence for systems that boot through EFI
>   arm64: Copy code from GNU-EFI
>   arm64: Change GNU-EFI imported file to use defined types
>   arm64: Use code from the gnu-efi when booting with EFI
>   lib: Avoid external dependency in libelf
>   x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
>   arm64: Add support for efi in Makefile
>   arm64: Add an efi/run script
> 
>  scripts/runtime.bash        |  14 +-
>  arm/efi/run                 |  61 +++++++
>  arm/run                     |  14 +-
>  configure                   |  15 +-
>  Makefile                    |   4 -
>  arm/Makefile.arm            |   6 +
>  arm/Makefile.arm64          |  18 +-
>  arm/Makefile.common         |  48 +++--
>  x86/Makefile.x86_64         |   4 +
>  lib/linux/efi.h             |  25 +++
>  lib/arm/asm/setup.h         |   3 +
>  lib/arm/asm/timer.h         |   2 +
>  lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
>  lib/argv.h                  |   1 +
>  lib/elf.h                   |  57 ++++++
>  lib/libcflat.h              |   1 +
>  lib/acpi.c                  | 129 ++++++++-----
>  lib/argv.c                  |   2 +-
>  lib/arm/gic.c               | 127 ++++++++++++-
>  lib/arm/io.c                |  29 ++-
>  lib/arm/mmu.c               |   4 +
>  lib/arm/psci.c              |  25 ++-
>  lib/arm/setup.c             | 247 ++++++++++++++++++++-----
>  lib/arm/timer.c             |  79 ++++++++
>  lib/devicetree.c            |   2 +-
>  lib/efi.c                   | 102 +++++++++++
>  lib/printf.c                | 194 ++++++++++++++++++--
>  arm/efi/elf_aarch64_efi.lds |  63 +++++++
>  arm/flat.lds                |   2 +-
>  arm/cstart.S                |  29 ++-
>  arm/cstart64.S              |  28 ++-
>  arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
>  arm/dummy.c                 |   4 +
>  arm/efi/reloc_aarch64.c     |  93 ++++++++++
>  arm/micro-bench.c           |   4 +-
>  arm/timer.c                 |  10 +-
>  x86/s3.c                    |  19 +-
>  x86/vmexit.c                |   2 +-
>  38 files changed, 1700 insertions(+), 258 deletions(-)
>  create mode 100755 arm/efi/run
>  create mode 100644 lib/elf.h
>  create mode 100644 lib/arm/timer.c
>  create mode 100644 arm/efi/elf_aarch64_efi.lds
>  create mode 100644 arm/efi/crt0-efi-aarch64.S
>  create mode 100644 arm/dummy.c
>  create mode 100644 arm/efi/reloc_aarch64.c
> 
> -- 
> 2.25.1
> 

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Sean Christopherson]

On Tue, Aug 09, 2022, Alexandru Elisei wrote:
> Hi,
> 
> Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
> support.
> 
> This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
> after performing the relocation. I'll post an abbreviated/simplified
> version of efi_main() for reference:
> 
> efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> {
> 	/* Get image, cmdline and memory map parameters from UEFI */
> 
>         efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> 
>         /* Set up arch-specific resources */
>         setup_efi(&efi_bootinfo);
> 
>         /* Run the test case */
>         ret = main(__argc, __argv, __environ);
> 
>         /* Shutdown the guest VM */
>         efi_exit(ret);
> 
>         /* Unreachable */
>         return EFI_UNSUPPORTED;
> }
> 
> Note that the assumption that efi_main() makes is that setup_efi() doesn't
> change the stack from the stack that the UEFI implementation allocated, in
> order for setup_efi() to be able to return to efi_main().

On the x86 side, efi_main() now runs with a KUT-controlled stack since commit

  d316d12a ("x86: efi: Provide a stack within testcase memory")

> If we want to keep the UEFI allocated stack, then both mechanism must be
> forbidden when running under UEFI. I dislike this idea, because those two
> mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> been possible with a normal operating system, which, except for the early
> boot code, runs with the MMU enabled.

Agreed.  IMO, KUT should stop using UEFI-controlled data as early as possible.
The original x86 behavior was effectively a temporary solution to get UEFI working
without needing to simultaneously rework the common early boot flows.

Side topic, I think the x86 code now has a benign bug.  The old code contained an
adjustment to RSP to undo some stack shenanigans (can't figure out why those
shenanigans exist), but now the adjustment happens on the KUT stack, which doesn't
need to be fixed up.

It's a moot point since efi_main() should never return, but it looks odd.  And it
seems like KUT should intentionally explode if efi_main() returns, e.g. do this
over two patches:

diff --git a/x86/efi/crt0-efi-x86_64.S b/x86/efi/crt0-efi-x86_64.S
index 1708ed55..e62891bc 100644
--- a/x86/efi/crt0-efi-x86_64.S
+++ b/x86/efi/crt0-efi-x86_64.S
@@ -62,10 +62,7 @@ _start:
        lea stacktop(%rip), %rsp
 
        call efi_main
-       addq $8, %rsp
-
-.exit: 
-       ret
+       ud2
 
        // hand-craft a dummy .reloc section so EFI knows it's a relocatable executable:

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

Hi,

On 09/08/2022 12:16, Alexandru Elisei wrote:
> Hi,
> 
> Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
> support.
> 
> This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
> after performing the relocation. I'll post an abbreviated/simplified
> version of efi_main() for reference:
> 
> efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> {
> 	/* Get image, cmdline and memory map parameters from UEFI */
> 
>          efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> 
>          /* Set up arch-specific resources */
>          setup_efi(&efi_bootinfo);
> 
>          /* Run the test case */
>          ret = main(__argc, __argv, __environ);
> 
>          /* Shutdown the guest VM */
>          efi_exit(ret);
> 
>          /* Unreachable */
>          return EFI_UNSUPPORTED;
> }
> 
> Note that the assumption that efi_main() makes is that setup_efi() doesn't
> change the stack from the stack that the UEFI implementation allocated, in
> order for setup_efi() to be able to return to efi_main().
> 
> arm64 requires explicit data cache maintenance to keep the contents of the
> caches in sync with memory when writing with MMU off/reading with MMU on
> and viceversa. More details of what is needed is why here [1] and here [2].
> These operations must also be performed for the stack because the stack is
> always used when running C code.
> 
> What this means is that if arm64 wants to be able to run C code when the
> MMU is disabled and when it is enabled, then it must perform data cache
> operations for the stack memory. Which is impossible if the stack has been
> allocated by UEFI, as kvm-unit-tests has no way of knowing its size, as it
> isn't specified in the UEFI spec*. As a result, either efi_main needs to be
> changed such that setup_efi() never returns, or arm64 must implement its
> own version of efi_main() (or however it will end up being called).
> 

I think it's possible to know the size of the stack. In 22/27 "arm64: 
Use code from the gnu-efi when booting with EFI", we change the top of 
the stack we start executing C code. Then at any point we can get the 
bottom of the stack.

But I doubt cleaning the stack is sufficient. What about the .data 
segment. Take for example, mem_regions. We populate them with the MMU 
on. We then use them to create page tables. We will need to clean them 
too. I won't be surprised if there is more data we would need to clean.

Thanks,

Nikos

> One way to get around this is never to run C code with the MMU off. That's
> relatively easy to do in the boot code, as the translation tables can be
> constructed with the MMU on, and then a fairly small assembly sequence is
> required to install them.
> 
> But arm64 also has two mechanisms for disabling the MMU:
> 
> 1. At compile time, the user can request a test to start with the MMU off,
> by setting the flag AUXINFO_MMU_OFF. So when booting as an UEFI app,
> kvm-unit-tests must disable the MMU.
> 
> 2. A function called mmu_disable() which allows a test to explicitly
> disable the MMU.
> 
> If we want to keep the UEFI allocated stack, then both mechanism must be
> forbidden when running under UEFI. I dislike this idea, because those two
> mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> been possible with a normal operating system, which, except for the early
> boot code, runs with the MMU enabled.
> 
> Any thoughts or comments about this?
> 
> *UEFI v2.8 states about the stack: "128 KiB or more of available stack
> space" (page 35), but EDK2 allocates 64KiB [3]. So without any firmware
> call to query the size of the stack, kvm-unit-tests cannot rely on it being
> a specific size.
> 
> [1] https://lore.kernel.org/kvm/20220809091558.14379-19-alexandru.elisei@arm.com/
> [2] https://lore.kernel.org/kvm/20220809091558.14379-20-alexandru.elisei@arm.com/
> [3] https://github.com/tianocore/edk2/blob/master/ArmPlatformPkg/ArmPlatformPkg.dec#L71
> 
> Thanks,
> Alex
> 
> On Thu, Jun 30, 2022 at 11:02:57AM +0100, Nikos Nikoleris wrote:
>> Hello,
>>
>> This patch series adds initial support for building arm64 tests as EFI
>> tests and running them under QEMU. Much like x86_64 we import external
>> dependencies from gnu-efi and adapt them to work with types and other
>> assumptions from kvm-unit-tests. In addition, this series adds support
>> for enumerating parts of the system using ACPI.
>>
>> The first set of patches moves the existing ACPI code to the common
>> lib path. Then, it extends definitions and functions to allow for more
>> robust discovery of ACPI tables. In arm64, we add support for setting
>> up the PSCI conduit, discovering the UART, timers and cpus via
>> ACPI. The code retains existing behavior and gives priority to
>> discovery through DT when one has been provided.
>>
>> In the second set of patches, we add support for getting the command
>> line from the EFI shell. This is a requirement for many of the
>> existing arm64 tests.
>>
>> In the third set of patches, we import code from gnu-efi, make minor
>> changes and add an alternative setup sequence from arm64 systems that
>> boot through EFI. Finally, we add support in the build system and a
>> run script which is used to run an EFI app.
>>
>> After this set of patches one can build arm64 EFI tests:
>>
>> $> ./configure --enable-efi
>> $> make
>>
>> And use the run script to run an EFI tests:
>>
>> $> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
>>
>> Or all tests:
>>
>> $> ./run_tests.sh
>>
>> There are a few items that this series does not address but they would
>> be useful to have:
>>   - Support for booting the system from EL2. Currently, we assume that a
>>     tests starts running at EL1. This the case when we run with EFI, it's
>>     not always the case in hardware.
>>   - Support for reading environment variables and populating __envp.
>>   - Support for discovering the PCI subsystem using ACPI.
>>   - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.
>>
>> git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased
>>
>> v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/
>>
>> Changes in v3:
>>   - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
>>   - Added support for discovering the GIC through ACPI
>>   - Added a missing header file (<elf.h>)
>>   - Added support for correctly parsing the outcome of tests (./run_tests)
>>
>> Thanks,
>>
>> Nikos
>>
>> Alexandru Elisei (1):
>>    lib: arm: Print test exit status
>>
>> Andrew Jones (3):
>>    arm/arm64: mmu_disable: Clean and invalidate before disabling
>>    arm/arm64: Rename etext to _etext
>>    arm64: Add a new type of memory type flag MR_F_RESERVED
>>
>> Nikos Nikoleris (23):
>>    lib: Fix style for acpi.{c,h}
>>    x86: Avoid references to fields of ACPI tables
>>    lib: Ensure all struct definition for ACPI tables are packed
>>    lib: Add support for the XSDT ACPI table
>>    lib: Extend the definition of the ACPI table FADT
>>    devicetree: Check if fdt is NULL before returning that a DT is
>>      available
>>    arm/arm64: Add support for setting up the PSCI conduit through ACPI
>>    arm/arm64: Add support for discovering the UART through ACPI
>>    arm/arm64: Add support for timer initialization through ACPI
>>    arm/arm64: Add support for cpu initialization through ACPI
>>    arm/arm64: Add support for gic initialization through ACPI
>>    lib/printf: Support for precision modifier in printf
>>    lib/printf: Add support for printing wide strings
>>    lib/efi: Add support for getting the cmdline
>>    lib: Avoid ms_abi for calls related to EFI on arm64
>>    arm/arm64: Add a setup sequence for systems that boot through EFI
>>    arm64: Copy code from GNU-EFI
>>    arm64: Change GNU-EFI imported file to use defined types
>>    arm64: Use code from the gnu-efi when booting with EFI
>>    lib: Avoid external dependency in libelf
>>    x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
>>    arm64: Add support for efi in Makefile
>>    arm64: Add an efi/run script
>>
>>   scripts/runtime.bash        |  14 +-
>>   arm/efi/run                 |  61 +++++++
>>   arm/run                     |  14 +-
>>   configure                   |  15 +-
>>   Makefile                    |   4 -
>>   arm/Makefile.arm            |   6 +
>>   arm/Makefile.arm64          |  18 +-
>>   arm/Makefile.common         |  48 +++--
>>   x86/Makefile.x86_64         |   4 +
>>   lib/linux/efi.h             |  25 +++
>>   lib/arm/asm/setup.h         |   3 +
>>   lib/arm/asm/timer.h         |   2 +
>>   lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
>>   lib/argv.h                  |   1 +
>>   lib/elf.h                   |  57 ++++++
>>   lib/libcflat.h              |   1 +
>>   lib/acpi.c                  | 129 ++++++++-----
>>   lib/argv.c                  |   2 +-
>>   lib/arm/gic.c               | 127 ++++++++++++-
>>   lib/arm/io.c                |  29 ++-
>>   lib/arm/mmu.c               |   4 +
>>   lib/arm/psci.c              |  25 ++-
>>   lib/arm/setup.c             | 247 ++++++++++++++++++++-----
>>   lib/arm/timer.c             |  79 ++++++++
>>   lib/devicetree.c            |   2 +-
>>   lib/efi.c                   | 102 +++++++++++
>>   lib/printf.c                | 194 ++++++++++++++++++--
>>   arm/efi/elf_aarch64_efi.lds |  63 +++++++
>>   arm/flat.lds                |   2 +-
>>   arm/cstart.S                |  29 ++-
>>   arm/cstart64.S              |  28 ++-
>>   arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
>>   arm/dummy.c                 |   4 +
>>   arm/efi/reloc_aarch64.c     |  93 ++++++++++
>>   arm/micro-bench.c           |   4 +-
>>   arm/timer.c                 |  10 +-
>>   x86/s3.c                    |  19 +-
>>   x86/vmexit.c                |   2 +-
>>   38 files changed, 1700 insertions(+), 258 deletions(-)
>>   create mode 100755 arm/efi/run
>>   create mode 100644 lib/elf.h
>>   create mode 100644 lib/arm/timer.c
>>   create mode 100644 arm/efi/elf_aarch64_efi.lds
>>   create mode 100644 arm/efi/crt0-efi-aarch64.S
>>   create mode 100644 arm/dummy.c
>>   create mode 100644 arm/efi/reloc_aarch64.c
>>
>> -- 
>> 2.25.1
>>

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi Sean,

On Tue, Aug 09, 2022 at 03:29:58PM +0000, Sean Christopherson wrote:
> On Tue, Aug 09, 2022, Alexandru Elisei wrote:
> > Hi,
> > 
> > Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
> > support.
> > 
> > This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
> > after performing the relocation. I'll post an abbreviated/simplified
> > version of efi_main() for reference:
> > 
> > efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> > {
> > 	/* Get image, cmdline and memory map parameters from UEFI */
> > 
> >         efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> > 
> >         /* Set up arch-specific resources */
> >         setup_efi(&efi_bootinfo);
> > 
> >         /* Run the test case */
> >         ret = main(__argc, __argv, __environ);
> > 
> >         /* Shutdown the guest VM */
> >         efi_exit(ret);
> > 
> >         /* Unreachable */
> >         return EFI_UNSUPPORTED;
> > }
> > 
> > Note that the assumption that efi_main() makes is that setup_efi() doesn't
> > change the stack from the stack that the UEFI implementation allocated, in
> > order for setup_efi() to be able to return to efi_main().
> 
> On the x86 side, efi_main() now runs with a KUT-controlled stack since commit
> 
>   d316d12a ("x86: efi: Provide a stack within testcase memory")
> 
> > If we want to keep the UEFI allocated stack, then both mechanism must be
> > forbidden when running under UEFI. I dislike this idea, because those two
> > mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> > been possible with a normal operating system, which, except for the early
> > boot code, runs with the MMU enabled.
> 
> Agreed.  IMO, KUT should stop using UEFI-controlled data as early as possible.
> The original x86 behavior was effectively a temporary solution to get UEFI working
> without needing to simultaneously rework the common early boot flows.

Yes, this is also what I am thinking, the stack is poorly specified in the
specification because the specification doesn't expect an application to
keep using it after calling EFI_BOOT_SERVICES.Exit(). Plus, using the UEFI
allocated stack makes the test less reproducible, as even EDK2 today
diverges from the spec wrt the stack, and other UEFI implementations might
do things differently. And with just like all software, there might be bugs
in the firmware. IMO, the more control kvm-unit-tests has over its
resources, the more robust the tests are.

What I was thinking is rewriting efi_main to return setup_efi(),
something like this:

void efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
{
	/* Get image, cmdline and memory map parameters from UEFI */

        efi_exit_boot_services(handle, &efi_bootinfo.mem_map);

        /* Set up arch-specific resources, not expected to return. */
        setup_efi(&efi_bootinfo);
}

Which would allow all architectures to change their environment as they see
fit, as setup_efi() is not expected to return. Architectures would have to
be made aware of the efi_exit() function though.

If you like that approach, I can give it a go, though I'm very rusty when
it comes to x86.

Thanks,
Alex

> 
> Side topic, I think the x86 code now has a benign bug.  The old code contained an
> adjustment to RSP to undo some stack shenanigans (can't figure out why those
> shenanigans exist), but now the adjustment happens on the KUT stack, which doesn't
> need to be fixed up.
> 
> It's a moot point since efi_main() should never return, but it looks odd.  And it
> seems like KUT should intentionally explode if efi_main() returns, e.g. do this
> over two patches:
> 
> diff --git a/x86/efi/crt0-efi-x86_64.S b/x86/efi/crt0-efi-x86_64.S
> index 1708ed55..e62891bc 100644
> --- a/x86/efi/crt0-efi-x86_64.S
> +++ b/x86/efi/crt0-efi-x86_64.S
> @@ -62,10 +62,7 @@ _start:
>         lea stacktop(%rip), %rsp
>  
>         call efi_main
> -       addq $8, %rsp
> -
> -.exit: 
> -       ret
> +       ud2
>  
>         // hand-craft a dummy .reloc section so EFI knows it's a relocatable executable:

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Sean Christopherson]

On Wed, Aug 10, 2022, Alexandru Elisei wrote:
> Hi Sean,
> 
> On Tue, Aug 09, 2022 at 03:29:58PM +0000, Sean Christopherson wrote:
> > On Tue, Aug 09, 2022, Alexandru Elisei wrote:
> > > Note that the assumption that efi_main() makes is that setup_efi() doesn't
> > > change the stack from the stack that the UEFI implementation allocated, in
> > > order for setup_efi() to be able to return to efi_main().
> > 
> > On the x86 side, efi_main() now runs with a KUT-controlled stack since commit
> > 
> >   d316d12a ("x86: efi: Provide a stack within testcase memory")
> > 
> > > If we want to keep the UEFI allocated stack, then both mechanism must be
> > > forbidden when running under UEFI. I dislike this idea, because those two
> > > mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> > > been possible with a normal operating system, which, except for the early
> > > boot code, runs with the MMU enabled.
> > 
> > Agreed.  IMO, KUT should stop using UEFI-controlled data as early as possible.
> > The original x86 behavior was effectively a temporary solution to get UEFI working
> > without needing to simultaneously rework the common early boot flows.
> 
> Yes, this is also what I am thinking, the stack is poorly specified in the
> specification because the specification doesn't expect an application to
> keep using it after calling EFI_BOOT_SERVICES.Exit(). Plus, using the UEFI
> allocated stack makes the test less reproducible, as even EDK2 today
> diverges from the spec wrt the stack, and other UEFI implementations might
> do things differently. And with just like all software, there might be bugs
> in the firmware. IMO, the more control kvm-unit-tests has over its
> resources, the more robust the tests are.
> 
> What I was thinking is rewriting efi_main to return setup_efi(),
> something like this:
> 
> void efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> {
> 	/* Get image, cmdline and memory map parameters from UEFI */
> 
>         efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> 
>         /* Set up arch-specific resources, not expected to return. */
>         setup_efi(&efi_bootinfo);
> }
> 
> Which would allow all architectures to change their environment as they see
> fit, as setup_efi() is not expected to return. Architectures would have to
> be made aware of the efi_exit() function though.

And they'd also have to invoke the test's main().  Why can't ARM switch to a KUT
stack before calling efi_main()?  The stack is a rather special case, I don't think
it's unreasonable to require architectures to handle that before efi_main().

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi Sean,

On Wed, Aug 10, 2022 at 02:58:15PM +0000, Sean Christopherson wrote:
> On Wed, Aug 10, 2022, Alexandru Elisei wrote:
> > Hi Sean,
> > 
> > On Tue, Aug 09, 2022 at 03:29:58PM +0000, Sean Christopherson wrote:
> > > On Tue, Aug 09, 2022, Alexandru Elisei wrote:
> > > > Note that the assumption that efi_main() makes is that setup_efi() doesn't
> > > > change the stack from the stack that the UEFI implementation allocated, in
> > > > order for setup_efi() to be able to return to efi_main().
> > > 
> > > On the x86 side, efi_main() now runs with a KUT-controlled stack since commit
> > > 
> > >   d316d12a ("x86: efi: Provide a stack within testcase memory")
> > > 
> > > > If we want to keep the UEFI allocated stack, then both mechanism must be
> > > > forbidden when running under UEFI. I dislike this idea, because those two
> > > > mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> > > > been possible with a normal operating system, which, except for the early
> > > > boot code, runs with the MMU enabled.
> > > 
> > > Agreed.  IMO, KUT should stop using UEFI-controlled data as early as possible.
> > > The original x86 behavior was effectively a temporary solution to get UEFI working
> > > without needing to simultaneously rework the common early boot flows.
> > 
> > Yes, this is also what I am thinking, the stack is poorly specified in the
> > specification because the specification doesn't expect an application to
> > keep using it after calling EFI_BOOT_SERVICES.Exit(). Plus, using the UEFI
> > allocated stack makes the test less reproducible, as even EDK2 today
> > diverges from the spec wrt the stack, and other UEFI implementations might
> > do things differently. And with just like all software, there might be bugs
> > in the firmware. IMO, the more control kvm-unit-tests has over its
> > resources, the more robust the tests are.
> > 
> > What I was thinking is rewriting efi_main to return setup_efi(),
> > something like this:
> > 
> > void efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> > {
> > 	/* Get image, cmdline and memory map parameters from UEFI */
> > 
> >         efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> > 
> >         /* Set up arch-specific resources, not expected to return. */
> >         setup_efi(&efi_bootinfo);
> > }
> > 
> > Which would allow all architectures to change their environment as they see
> > fit, as setup_efi() is not expected to return. Architectures would have to
> > be made aware of the efi_exit() function though.
> 
> And they'd also have to invoke the test's main().  Why can't ARM switch to a KUT
> stack before calling efi_main()?  The stack is a rather special case, I don't think
> it's unreasonable to require architectures to handle that before efi_main().

Sorry, missed the part from your earlier reply about x86 switching to a
kvm-unit-tests controlled stack. I was just going to say that, when I saw
your reply.

I agree, it's entirely feasible for arm64 to switch to its own stack before
calling efi_main(), in which case it can call efi_main() without any
changes to efi_main().

Thanks for your input.

Alex

Re: [kvm-unit-tests PATCH v3 19/27] arm/arm64: Add a setup sequence for systems that boot through EFI

[Nikos Nikoleris]

On 19/07/2022 15:08, Alexandru Elisei wrote:
> Hi,
> 
> On Thu, Jun 30, 2022 at 11:03:16AM +0100, Nikos Nikoleris wrote:
>> This change implements an alternative setup sequence for the system
>> when we are booting through EFI. The memory map is discovered through
>> EFI boot services and devices through ACPI.
>>
>> This change is based on a change initially proposed by
>> Andrew Jones <drjones@redhat.com>
>>
>> Signed-off-by: Nikos Nikoleris <nikos.nikoleris@arm.com>
>> ---
>>   lib/linux/efi.h     |   1 +
>>   lib/arm/asm/setup.h |   2 +
>>   lib/arm/setup.c     | 181 +++++++++++++++++++++++++++++++++++++++++++-
>>   arm/cstart.S        |   1 +
>>   arm/cstart64.S      |   1 +
>>   5 files changed, 184 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/linux/efi.h b/lib/linux/efi.h
>> index 53748dd..89f9a9e 100644
>> --- a/lib/linux/efi.h
>> +++ b/lib/linux/efi.h
>> @@ -63,6 +63,7 @@ typedef guid_t efi_guid_t;
>>   	(c) & 0xff, ((c) >> 8) & 0xff, d } }
>>   
>>   #define ACPI_TABLE_GUID EFI_GUID(0xeb9d2d30, 0x2d88, 0x11d3, 0x9a, 0x16, 0x00, 0x90, 0x27, 0x3f, 0xc1, 0x4d)
>> +#define ACPI_20_TABLE_GUID EFI_GUID(0x8868e871, 0xe4f1, 0x11d3,  0xbc, 0x22, 0x00, 0x80, 0xc7, 0x3c, 0x88, 0x81)
>>   
>>   #define LOADED_IMAGE_PROTOCOL_GUID EFI_GUID(0x5b1b31a1, 0x9562, 0x11d2,  0x8e, 0x3f, 0x00, 0xa0, 0xc9, 0x69, 0x72, 0x3b)
>>   
>> diff --git a/lib/arm/asm/setup.h b/lib/arm/asm/setup.h
>> index 64cd379..c4cd485 100644
>> --- a/lib/arm/asm/setup.h
>> +++ b/lib/arm/asm/setup.h
>> @@ -6,6 +6,7 @@
>>    * This work is licensed under the terms of the GNU LGPL, version 2.
>>    */
>>   #include <libcflat.h>
>> +#include <efi.h>
>>   #include <asm/page.h>
>>   #include <asm/pgtable-hwdef.h>
>>   
>> @@ -37,5 +38,6 @@ extern unsigned int mem_region_get_flags(phys_addr_t paddr);
>>   #define SMP_CACHE_BYTES		L1_CACHE_BYTES
>>   
>>   void setup(const void *fdt, phys_addr_t freemem_start);
>> +efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo);
>>   
>>   #endif /* _ASMARM_SETUP_H_ */
>> diff --git a/lib/arm/setup.c b/lib/arm/setup.c
>> index 13513d0..30d04d0 100644
>> --- a/lib/arm/setup.c
>> +++ b/lib/arm/setup.c
>> @@ -34,7 +34,7 @@
>>   #define NR_EXTRA_MEM_REGIONS	16
>>   #define NR_INITIAL_MEM_REGIONS	(MAX_DT_MEM_REGIONS + NR_EXTRA_MEM_REGIONS)
>>   
>> -extern unsigned long _etext;
>> +extern unsigned long _text, _etext, _data, _edata;
>>   
>>   char *initrd;
>>   u32 initrd_size;
>> @@ -44,7 +44,10 @@ int nr_cpus;
>>   
>>   static struct mem_region __initial_mem_regions[NR_INITIAL_MEM_REGIONS + 1];
>>   struct mem_region *mem_regions = __initial_mem_regions;
>> -phys_addr_t __phys_offset, __phys_end;
>> +phys_addr_t __phys_offset = (phys_addr_t)-1, __phys_end = 0;
>> +
>> +extern void exceptions_init(void);
>> +extern void asm_mmu_disable(void);
>>   
>>   int mpidr_to_cpu(uint64_t mpidr)
>>   {
>> @@ -272,3 +275,177 @@ void setup(const void *fdt, phys_addr_t freemem_start)
>>   	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
>>   		setup_vm();
>>   }
>> +
>> +#ifdef CONFIG_EFI
>> +
>> +#include <efi.h>
>> +
>> +static efi_status_t setup_rsdp(efi_bootinfo_t *efi_bootinfo)
>> +{
>> +	efi_status_t status;
>> +	struct rsdp_descriptor *rsdp;
>> +
>> +	/*
>> +	 * RSDP resides in an EFI_ACPI_RECLAIM_MEMORY region, which is not used
>> +	 * by kvm-unit-tests arm64 memory allocator. So it is not necessary to
>> +	 * copy the data structure to another memory region to prevent
>> +	 * unintentional overwrite.
>> +	 */
>> +	status = efi_get_system_config_table(ACPI_20_TABLE_GUID, (void **)&rsdp);
>> +	if (status != EFI_SUCCESS)
>> +		return status;
>> +
>> +	set_efi_rsdp(rsdp);
>> +
>> +	return EFI_SUCCESS;
>> +}
>> +
>> +static efi_status_t efi_mem_init(efi_bootinfo_t *efi_bootinfo)
>> +{
>> +	int i;
>> +	unsigned long free_mem_pages = 0;
>> +	unsigned long free_mem_start = 0;
>> +	struct efi_boot_memmap *map = &(efi_bootinfo->mem_map);
>> +	efi_memory_desc_t *buffer = *map->map;
>> +	efi_memory_desc_t *d = NULL;
>> +	phys_addr_t base, top;
>> +	struct mem_region *r;
>> +	uintptr_t text = (uintptr_t)&_text, etext = __ALIGN((uintptr_t)&_etext, 4096);
>> +	uintptr_t data = (uintptr_t)&_data, edata = __ALIGN((uintptr_t)&_edata, 4096);
>> +
>> +	/*
>> +	 * Record the largest free EFI_CONVENTIONAL_MEMORY region
>> +	 * which will be used to set up the memory allocator, so that
>> +	 * the memory allocator can work in the largest free
>> +	 * continuous memory region.
>> +	 */
>> +	for (i = 0, r = &mem_regions[0]; i < *(map->map_size); i += *(map->desc_size), ++r) {
>> +		d = (efi_memory_desc_t *)(&((u8 *)buffer)[i]);
>> +
>> +		r->start = d->phys_addr;
>> +		r->end = d->phys_addr + d->num_pages * EFI_PAGE_SIZE;
>> +
>> +		switch (d->type) {
>> +		case EFI_RESERVED_TYPE:
>> +		case EFI_LOADER_DATA:
>> +		case EFI_BOOT_SERVICES_CODE:
>> +		case EFI_BOOT_SERVICES_DATA:
>> +		case EFI_RUNTIME_SERVICES_CODE:
>> +		case EFI_RUNTIME_SERVICES_DATA:
>> +		case EFI_UNUSABLE_MEMORY:
>> +		case EFI_ACPI_RECLAIM_MEMORY:
>> +		case EFI_ACPI_MEMORY_NVS:
>> +		case EFI_PAL_CODE:
>> +			r->flags = MR_F_RESERVED;
>> +			break;
>> +		case EFI_MEMORY_MAPPED_IO:
>> +		case EFI_MEMORY_MAPPED_IO_PORT_SPACE:
>> +			r->flags = MR_F_IO;
>> +			break;
>> +		case EFI_LOADER_CODE:
>> +			if (r->start <= text && r->end > text) {
>> +				/* This is the unit test region. Flag the code separately. */
>> +				phys_addr_t tmp = r->end;
>> +
>> +				assert(etext <= data);
>> +				assert(edata <= r->end);
>> +				r->flags = MR_F_CODE;
>> +				r->end = data;
>> +				++r;
>> +				r->start = data;
>> +				r->end = tmp;
>> +			} else {
>> +				r->flags = MR_F_RESERVED;
>> +			}
>> +			break;
>> +		case EFI_CONVENTIONAL_MEMORY:
>> +			if (free_mem_pages < d->num_pages) {
>> +				free_mem_pages = d->num_pages;
>> +				free_mem_start = d->phys_addr;
>> +			}
>> +			break;
>> +		}
>> +
>> +		if (!(r->flags & MR_F_IO)) {
>> +			if (r->start < __phys_offset)
>> +				__phys_offset = r->start;
>> +			if (r->end > __phys_end)
>> +				__phys_end = r->end;
> 
> What happens if there are holes between __phys_offset and __phys_end? The
> boot path for KVM makes sure there are no holes. Wouldn't asm_mmu_disable()
> trigger a translation fault if the address is not mapped because it
> corresponds to a hole in the EFI provided memory map?
> 
> What happens if the region [__phys_offset, __phys_end) contains one of the
> EFI reserved memory types? That's not really something that kvm-unit-tests
> should be poking.
> 

That's a good point.

> The efi boot code path changes the semantics for __phys_offset and
> __phys_end, and that's a recipe for introducing bugs.
> 
> I would suggest changing __phys_offset and __phys_end to represent
> something that applies to both the KVM boot path and the EFI boot path.
> 

Currently we're using __phys_offset and __phys_end in two places:
* When we disable the MMU, to clean and invalidate the whole physical 
address space.
* In selftest-mem, To compute the amount of memory available to the test.

> One idea that occured to me would be to have separate text, data and
> available memory regions.  Have __phys_offset and __phys_end express the
> start and end of the largest contiguous memory region, and initialize the
> memory allocator from this region. That will also pave the way for handling
> multiple memory regions from the DT.

For EFI, this didn't work. __phys_end - __phys_offset doesn't produce 
the amount of memory expected by selftest-mem.

> 
> Or, if you can prove and EFI_LOADER_CODE is always adjacent to
> EFI_CONVENTIONAL_MEMORY, you can have __phys_offset and __phys_end describe
> the region from the start of text to the end of EFI_CONVENTIONAL_MEMORY.
> 

I don't think we can rely on this assumption. It doesn't hold on when I 
run experiments locally.

> Thoughts? Suggestions?

Could we get rid of __phys_offset and __phys_end? You wanted to move 
away from cleaning the whole memory for the purposes of disabling the 
MMU. We could just clean the regions as we discover them and keep a 
__phys_size for the purposes of selftest-mem. What do you think?

Thanks,

Nikos

> 
> Thanks,
> Alex
> 
>> +		}
>> +	}
>> +	__phys_end &= PHYS_MASK;
>> +	asm_mmu_disable();
>> +
>> +	if (free_mem_pages == 0)
>> +		return EFI_OUT_OF_RESOURCES;
>> +
>> +	assert(sizeof(long) == 8 || free_mem_start < (3ul << 30));
>> +
>> +	phys_alloc_init(free_mem_start, free_mem_pages << EFI_PAGE_SHIFT);
>> +	phys_alloc_set_minimum_alignment(SMP_CACHE_BYTES);
>> +
>> +	phys_alloc_get_unused(&base, &top);
>> +	base = PAGE_ALIGN(base);
>> +	top = top & PAGE_MASK;
>> +	assert(sizeof(long) == 8 || !(base >> 32));
>> +	if (sizeof(long) != 8 && (top >> 32) != 0)
>> +		top = ((uint64_t)1 << 32);
>> +	page_alloc_init_area(0, base >> PAGE_SHIFT, top >> PAGE_SHIFT);
>> +	page_alloc_ops_enable();
>> +
>> +	return EFI_SUCCESS;
>> +}
>> +
>> +efi_status_t setup_efi(efi_bootinfo_t *efi_bootinfo)
>> +{
>> +	efi_status_t status;
>> +
>> +	struct thread_info *ti = current_thread_info();
>> +
>> +	memset(ti, 0, sizeof(*ti));
>> +
>> +	exceptions_init();
>> +
>> +	status = efi_mem_init(efi_bootinfo);
>> +	if (status != EFI_SUCCESS) {
>> +		printf("Failed to initialize memory: ");
>> +		switch (status) {
>> +		case EFI_OUT_OF_RESOURCES:
>> +			printf("No free memory region\n");
>> +			break;
>> +		default:
>> +			printf("Unknown error\n");
>> +			break;
>> +		}
>> +		return status;
>> +	}
>> +
>> +	status = setup_rsdp(efi_bootinfo);
>> +	if (status != EFI_SUCCESS) {
>> +		printf("Cannot find RSDP in EFI system table\n");
>> +		return status;
>> +	}
>> +
>> +	psci_set_conduit();
>> +	cpu_init();
>> +	/* cpu_init must be called before thread_info_init */
>> +	thread_info_init(current_thread_info(), 0);
>> +	/* mem_init must be called before io_init */
>> +	io_init();
>> +
>> +	timer_save_state();
>> +	if (initrd) {
>> +		/* environ is currently the only file in the initrd */
>> +		char *env = malloc(initrd_size);
>> +
>> +		memcpy(env, initrd, initrd_size);
>> +		setup_env(env, initrd_size);
>> +	}
>> +
>> +	if (!(auxinfo.flags & AUXINFO_MMU_OFF))
>> +		setup_vm();
>> +
>> +	return EFI_SUCCESS;
>> +}
>> +
>> +#endif
>> diff --git a/arm/cstart.S b/arm/cstart.S
>> index dc324c5..66a55b9 100644
>> --- a/arm/cstart.S
>> +++ b/arm/cstart.S
>> @@ -256,6 +256,7 @@ asm_mmu_disable:
>>    *
>>    * Input r0 is the stack top, which is the exception stacks base
>>    */
>> +.globl exceptions_init
>>   exceptions_init:
>>   	mrc	p15, 0, r2, c1, c0, 0	@ read SCTLR
>>   	bic	r2, #CR_V		@ SCTLR.V := 0
>> diff --git a/arm/cstart64.S b/arm/cstart64.S
>> index 390feb9..55b41ea 100644
>> --- a/arm/cstart64.S
>> +++ b/arm/cstart64.S
>> @@ -276,6 +276,7 @@ asm_mmu_disable:
>>    * Vectors
>>    */
>>   
>> +.globl exceptions_init
>>   exceptions_init:
>>   	adrp	x4, vector_table
>>   	add	x4, x4, :lo12:vector_table
>> -- 
>> 2.25.1
>>

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Alexandru Elisei]

Hi,

On Tue, Aug 09, 2022 at 05:09:42PM +0100, Nikos Nikoleris wrote:
> Hi,
> 
> On 09/08/2022 12:16, Alexandru Elisei wrote:
> > Hi,
> > 
> > Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
> > support.
> > 
> > This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
> > after performing the relocation. I'll post an abbreviated/simplified
> > version of efi_main() for reference:
> > 
> > efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
> > {
> > 	/* Get image, cmdline and memory map parameters from UEFI */
> > 
> >          efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
> > 
> >          /* Set up arch-specific resources */
> >          setup_efi(&efi_bootinfo);
> > 
> >          /* Run the test case */
> >          ret = main(__argc, __argv, __environ);
> > 
> >          /* Shutdown the guest VM */
> >          efi_exit(ret);
> > 
> >          /* Unreachable */
> >          return EFI_UNSUPPORTED;
> > }
> > 
> > Note that the assumption that efi_main() makes is that setup_efi() doesn't
> > change the stack from the stack that the UEFI implementation allocated, in
> > order for setup_efi() to be able to return to efi_main().
> > 
> > arm64 requires explicit data cache maintenance to keep the contents of the
> > caches in sync with memory when writing with MMU off/reading with MMU on
> > and viceversa. More details of what is needed is why here [1] and here [2].
> > These operations must also be performed for the stack because the stack is
> > always used when running C code.
> > 
> > What this means is that if arm64 wants to be able to run C code when the
> > MMU is disabled and when it is enabled, then it must perform data cache
> > operations for the stack memory. Which is impossible if the stack has been
> > allocated by UEFI, as kvm-unit-tests has no way of knowing its size, as it
> > isn't specified in the UEFI spec*. As a result, either efi_main needs to be
> > changed such that setup_efi() never returns, or arm64 must implement its
> > own version of efi_main() (or however it will end up being called).
> > 
> 
> I think it's possible to know the size of the stack. In 22/27 "arm64: Use
> code from the gnu-efi when booting with EFI", we change the top of the stack
> we start executing C code. Then at any point we can get the bottom of the
> stack.

For reference, I believe this is the code you are referring to (removed
some parts because they were irrelevant):

diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
index d50e78dd1faa..03d29b01cb38 100644
--- a/arm/efi/crt0-efi-aarch64.S
+++ b/arm/efi/crt0-efi-aarch64.S
@@ -111,10 +111,19 @@ section_table:

	[..]
+       /* Align sp; this is necessary due to way we store cpu0's thread_info */
        mov             x29, sp
+       and             x29, x29, #THREAD_MASK
+       mov             x30, sp
+       mov             sp, x29


It looks to me like the code doesn't do anything to detect the size and
top/bottom of the stack. What it looks me, is that the code conveniently
assumes that 1. the stack is bigger than THREAD_SIZE = THREAD_MASK + 1
(which is 16K) and 2. the stack is aligned to THREAD_SIZE. As
under-specified the stack size is, I couldn't find any mention of the stack
alignment in the spec.

But it's more than about knowing the size and address of the stack. It's
about reproducibility and having as much control as possible over the
environment. Today, if the stack overflows/underflows you can detect that
by comparing the stack pointer with edata/stacktop. What happens if that
happens to the UEFI stack?  Who knows.

Also, I think we can agree that it is not inconceivable that there can be a
bug with how firmware manages the stack for an app. How can that be
reproduced if the bug manifests itself only on a very particular
combination of hardware and firmware? How can that be debugged? And how can
that be fixed? Fixing it in firmware might be impossible, same as expecting
people to upgrade their firmware, which might not be possible for their
particular setups. Wouldn't it be much simpler to modify kvm-unit-tests to
use its own stack?

That's why I think that arm64 doing what x86 did and replace the UEFI stack
is a good idea.

> 
> But I doubt cleaning the stack is sufficient. What about the .data segment.
> Take for example, mem_regions. We populate them with the MMU on. We then use
> them to create page tables. We will need to clean them too. I won't be
> surprised if there is more data we would need to clean.

I agree completely! The setup code should definitely to do dcache
maintenance.

If the UEFI setup code does dcache maintenance, I believe it would look
very strange if the normal setup code (as a kernel inside a VM) doesn't do
any by relying instead on the cache maintenance that KVM does. That's why I
think the best way forward would be to share as much of the setup code as
possible between the two.

Something like this (this is just an example off the top of my head):

- setup_efi() does whatever it's needed before disabling the MMU.
- it branches to the assembly routine that the disables the MMU and
  whatever else it needs to do (maybe exceptions_init like start does).
- it calls the C function lib/arm/setup.c::setup(), which can skip some
  steps under UEFI (like the mem_regions setup).
- setup_efi() then returns to efi_main().

That way the UEFI code can reuse most of the dcache maintenance that is
proposed in this series [1], and reuse most of the existing boot code. What
do you think?

[1] https://lore.kernel.org/kvmarm/20220809091558.14379-1-alexandru.elisei@arm.com/

Thanks,
Alex

> 
> Thanks,
> 
> Nikos
> 
> > One way to get around this is never to run C code with the MMU off. That's
> > relatively easy to do in the boot code, as the translation tables can be
> > constructed with the MMU on, and then a fairly small assembly sequence is
> > required to install them.
> > 
> > But arm64 also has two mechanisms for disabling the MMU:
> > 
> > 1. At compile time, the user can request a test to start with the MMU off,
> > by setting the flag AUXINFO_MMU_OFF. So when booting as an UEFI app,
> > kvm-unit-tests must disable the MMU.
> > 
> > 2. A function called mmu_disable() which allows a test to explicitly
> > disable the MMU.
> > 
> > If we want to keep the UEFI allocated stack, then both mechanism must be
> > forbidden when running under UEFI. I dislike this idea, because those two
> > mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
> > been possible with a normal operating system, which, except for the early
> > boot code, runs with the MMU enabled.
> > 
> > Any thoughts or comments about this?
> > 
> > *UEFI v2.8 states about the stack: "128 KiB or more of available stack
> > space" (page 35), but EDK2 allocates 64KiB [3]. So without any firmware
> > call to query the size of the stack, kvm-unit-tests cannot rely on it being
> > a specific size.
> > 
> > [1] https://lore.kernel.org/kvm/20220809091558.14379-19-alexandru.elisei@arm.com/
> > [2] https://lore.kernel.org/kvm/20220809091558.14379-20-alexandru.elisei@arm.com/
> > [3] https://github.com/tianocore/edk2/blob/master/ArmPlatformPkg/ArmPlatformPkg.dec#L71
> > 
> > Thanks,
> > Alex
> > 
> > On Thu, Jun 30, 2022 at 11:02:57AM +0100, Nikos Nikoleris wrote:
> > > Hello,
> > > 
> > > This patch series adds initial support for building arm64 tests as EFI
> > > tests and running them under QEMU. Much like x86_64 we import external
> > > dependencies from gnu-efi and adapt them to work with types and other
> > > assumptions from kvm-unit-tests. In addition, this series adds support
> > > for enumerating parts of the system using ACPI.
> > > 
> > > The first set of patches moves the existing ACPI code to the common
> > > lib path. Then, it extends definitions and functions to allow for more
> > > robust discovery of ACPI tables. In arm64, we add support for setting
> > > up the PSCI conduit, discovering the UART, timers and cpus via
> > > ACPI. The code retains existing behavior and gives priority to
> > > discovery through DT when one has been provided.
> > > 
> > > In the second set of patches, we add support for getting the command
> > > line from the EFI shell. This is a requirement for many of the
> > > existing arm64 tests.
> > > 
> > > In the third set of patches, we import code from gnu-efi, make minor
> > > changes and add an alternative setup sequence from arm64 systems that
> > > boot through EFI. Finally, we add support in the build system and a
> > > run script which is used to run an EFI app.
> > > 
> > > After this set of patches one can build arm64 EFI tests:
> > > 
> > > $> ./configure --enable-efi
> > > $> make
> > > 
> > > And use the run script to run an EFI tests:
> > > 
> > > $> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
> > > 
> > > Or all tests:
> > > 
> > > $> ./run_tests.sh
> > > 
> > > There are a few items that this series does not address but they would
> > > be useful to have:
> > >   - Support for booting the system from EL2. Currently, we assume that a
> > >     tests starts running at EL1. This the case when we run with EFI, it's
> > >     not always the case in hardware.
> > >   - Support for reading environment variables and populating __envp.
> > >   - Support for discovering the PCI subsystem using ACPI.
> > >   - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.
> > > 
> > > git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased
> > > 
> > > v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/
> > > 
> > > Changes in v3:
> > >   - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
> > >   - Added support for discovering the GIC through ACPI
> > >   - Added a missing header file (<elf.h>)
> > >   - Added support for correctly parsing the outcome of tests (./run_tests)
> > > 
> > > Thanks,
> > > 
> > > Nikos
> > > 
> > > Alexandru Elisei (1):
> > >    lib: arm: Print test exit status
> > > 
> > > Andrew Jones (3):
> > >    arm/arm64: mmu_disable: Clean and invalidate before disabling
> > >    arm/arm64: Rename etext to _etext
> > >    arm64: Add a new type of memory type flag MR_F_RESERVED
> > > 
> > > Nikos Nikoleris (23):
> > >    lib: Fix style for acpi.{c,h}
> > >    x86: Avoid references to fields of ACPI tables
> > >    lib: Ensure all struct definition for ACPI tables are packed
> > >    lib: Add support for the XSDT ACPI table
> > >    lib: Extend the definition of the ACPI table FADT
> > >    devicetree: Check if fdt is NULL before returning that a DT is
> > >      available
> > >    arm/arm64: Add support for setting up the PSCI conduit through ACPI
> > >    arm/arm64: Add support for discovering the UART through ACPI
> > >    arm/arm64: Add support for timer initialization through ACPI
> > >    arm/arm64: Add support for cpu initialization through ACPI
> > >    arm/arm64: Add support for gic initialization through ACPI
> > >    lib/printf: Support for precision modifier in printf
> > >    lib/printf: Add support for printing wide strings
> > >    lib/efi: Add support for getting the cmdline
> > >    lib: Avoid ms_abi for calls related to EFI on arm64
> > >    arm/arm64: Add a setup sequence for systems that boot through EFI
> > >    arm64: Copy code from GNU-EFI
> > >    arm64: Change GNU-EFI imported file to use defined types
> > >    arm64: Use code from the gnu-efi when booting with EFI
> > >    lib: Avoid external dependency in libelf
> > >    x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
> > >    arm64: Add support for efi in Makefile
> > >    arm64: Add an efi/run script
> > > 
> > >   scripts/runtime.bash        |  14 +-
> > >   arm/efi/run                 |  61 +++++++
> > >   arm/run                     |  14 +-
> > >   configure                   |  15 +-
> > >   Makefile                    |   4 -
> > >   arm/Makefile.arm            |   6 +
> > >   arm/Makefile.arm64          |  18 +-
> > >   arm/Makefile.common         |  48 +++--
> > >   x86/Makefile.x86_64         |   4 +
> > >   lib/linux/efi.h             |  25 +++
> > >   lib/arm/asm/setup.h         |   3 +
> > >   lib/arm/asm/timer.h         |   2 +
> > >   lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
> > >   lib/argv.h                  |   1 +
> > >   lib/elf.h                   |  57 ++++++
> > >   lib/libcflat.h              |   1 +
> > >   lib/acpi.c                  | 129 ++++++++-----
> > >   lib/argv.c                  |   2 +-
> > >   lib/arm/gic.c               | 127 ++++++++++++-
> > >   lib/arm/io.c                |  29 ++-
> > >   lib/arm/mmu.c               |   4 +
> > >   lib/arm/psci.c              |  25 ++-
> > >   lib/arm/setup.c             | 247 ++++++++++++++++++++-----
> > >   lib/arm/timer.c             |  79 ++++++++
> > >   lib/devicetree.c            |   2 +-
> > >   lib/efi.c                   | 102 +++++++++++
> > >   lib/printf.c                | 194 ++++++++++++++++++--
> > >   arm/efi/elf_aarch64_efi.lds |  63 +++++++
> > >   arm/flat.lds                |   2 +-
> > >   arm/cstart.S                |  29 ++-
> > >   arm/cstart64.S              |  28 ++-
> > >   arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
> > >   arm/dummy.c                 |   4 +
> > >   arm/efi/reloc_aarch64.c     |  93 ++++++++++
> > >   arm/micro-bench.c           |   4 +-
> > >   arm/timer.c                 |  10 +-
> > >   x86/s3.c                    |  19 +-
> > >   x86/vmexit.c                |   2 +-
> > >   38 files changed, 1700 insertions(+), 258 deletions(-)
> > >   create mode 100755 arm/efi/run
> > >   create mode 100644 lib/elf.h
> > >   create mode 100644 lib/arm/timer.c
> > >   create mode 100644 arm/efi/elf_aarch64_efi.lds
> > >   create mode 100644 arm/efi/crt0-efi-aarch64.S
> > >   create mode 100644 arm/dummy.c
> > >   create mode 100644 arm/efi/reloc_aarch64.c
> > > 
> > > -- 
> > > 2.25.1
> > > 

Re: [kvm-unit-tests PATCH v3 00/27] EFI and ACPI support for arm64

[Nikos Nikoleris]

On 12/08/2022 15:55, Alexandru Elisei wrote:
> Hi,
> 
> On Tue, Aug 09, 2022 at 05:09:42PM +0100, Nikos Nikoleris wrote:
>> Hi,
>>
>> On 09/08/2022 12:16, Alexandru Elisei wrote:
>>> Hi,
>>>
>>> Adding Sean and Zixuan, as they were involved in the initial x86 UEFI
>>> support.
>>>
>>> This version of the UEFI support for arm64 jumps to lib/efi.c::efi_main
>>> after performing the relocation. I'll post an abbreviated/simplified
>>> version of efi_main() for reference:
>>>
>>> efi_status_t efi_main(efi_handle_t handle, efi_system_table_t *sys_tab)
>>> {
>>> 	/* Get image, cmdline and memory map parameters from UEFI */
>>>
>>>           efi_exit_boot_services(handle, &efi_bootinfo.mem_map);
>>>
>>>           /* Set up arch-specific resources */
>>>           setup_efi(&efi_bootinfo);
>>>
>>>           /* Run the test case */
>>>           ret = main(__argc, __argv, __environ);
>>>
>>>           /* Shutdown the guest VM */
>>>           efi_exit(ret);
>>>
>>>           /* Unreachable */
>>>           return EFI_UNSUPPORTED;
>>> }
>>>
>>> Note that the assumption that efi_main() makes is that setup_efi() doesn't
>>> change the stack from the stack that the UEFI implementation allocated, in
>>> order for setup_efi() to be able to return to efi_main().
>>>
>>> arm64 requires explicit data cache maintenance to keep the contents of the
>>> caches in sync with memory when writing with MMU off/reading with MMU on
>>> and viceversa. More details of what is needed is why here [1] and here [2].
>>> These operations must also be performed for the stack because the stack is
>>> always used when running C code.
>>>
>>> What this means is that if arm64 wants to be able to run C code when the
>>> MMU is disabled and when it is enabled, then it must perform data cache
>>> operations for the stack memory. Which is impossible if the stack has been
>>> allocated by UEFI, as kvm-unit-tests has no way of knowing its size, as it
>>> isn't specified in the UEFI spec*. As a result, either efi_main needs to be
>>> changed such that setup_efi() never returns, or arm64 must implement its
>>> own version of efi_main() (or however it will end up being called).
>>>
>>
>> I think it's possible to know the size of the stack. In 22/27 "arm64: Use
>> code from the gnu-efi when booting with EFI", we change the top of the stack
>> we start executing C code. Then at any point we can get the bottom of the
>> stack.
> 
> For reference, I believe this is the code you are referring to (removed
> some parts because they were irrelevant):
> 
> diff --git a/arm/efi/crt0-efi-aarch64.S b/arm/efi/crt0-efi-aarch64.S
> index d50e78dd1faa..03d29b01cb38 100644
> --- a/arm/efi/crt0-efi-aarch64.S
> +++ b/arm/efi/crt0-efi-aarch64.S
> @@ -111,10 +111,19 @@ section_table:
> 
> 	[..]
> +       /* Align sp; this is necessary due to way we store cpu0's thread_info */
>          mov             x29, sp
> +       and             x29, x29, #THREAD_MASK
> +       mov             x30, sp
> +       mov             sp, x29
> 
> 
> It looks to me like the code doesn't do anything to detect the size and
> top/bottom of the stack. What it looks me, is that the code conveniently
> assumes that 1. the stack is bigger than THREAD_SIZE = THREAD_MASK + 1
> (which is 16K) and 2. the stack is aligned to THREAD_SIZE. As
> under-specified the stack size is, I couldn't find any mention of the stack
> alignment in the spec.
> 

The code assumes that the stack is big enough. You've pointed to manuals 
that support this. It doesn't assume that stack pointer is aligned, it 
aligns the stack pointed as per kvm-unit-tests arm64 requirements.

For the purposes of cleaning the stack, the value of sp computed by this 
code is top of the stack. The bottom of the used stack is the value of 
sp when we call asm_mmu_disable.

> But it's more than about knowing the size and address of the stack. It's
> about reproducibility and having as much control as possible over the
> environment. Today, if the stack overflows/underflows you can detect that
> by comparing the stack pointer with edata/stacktop. What happens if that
> happens to the UEFI stack?  Who knows.
> 

Can you point me to the code where we check for stack 
overflow/underflows? I couldn't find it.

> Also, I think we can agree that it is not inconceivable that there can be a
> bug with how firmware manages the stack for an app. How can that be
> reproduced if the bug manifests itself only on a very particular
> combination of hardware and firmware? How can that be debugged? And how can
> that be fixed? Fixing it in firmware might be impossible, same as expecting
> people to upgrade their firmware, which might not be possible for their
> particular setups. Wouldn't it be much simpler to modify kvm-unit-tests to
> use its own stack?

If we start assuming that edk2/kvm/qemu/kvmtool might be buggy in all 
sort of ways then it will be hard to reason about many more things.

But I am not opposed to the idea of defining our own stack. In the next 
revision, I will switch to a custom stack.

> 
> That's why I think that arm64 doing what x86 did and replace the UEFI stack
> is a good idea.
> 
>>
>> But I doubt cleaning the stack is sufficient. What about the .data segment.
>> Take for example, mem_regions. We populate them with the MMU on. We then use
>> them to create page tables. We will need to clean them too. I won't be
>> surprised if there is more data we would need to clean.
> 
> I agree completely! The setup code should definitely to do dcache
> maintenance.
> 
> If the UEFI setup code does dcache maintenance, I believe it would look
> very strange if the normal setup code (as a kernel inside a VM) doesn't do
> any by relying instead on the cache maintenance that KVM does. That's why I
> think the best way forward would be to share as much of the setup code as
> possible between the two.
> 
> Something like this (this is just an example off the top of my head):
> 
> - setup_efi() does whatever it's needed before disabling the MMU.
> - it branches to the assembly routine that the disables the MMU and
>    whatever else it needs to do (maybe exceptions_init like start does).
> - it calls the C function lib/arm/setup.c::setup(), which can skip some
>    steps under UEFI (like the mem_regions setup).
> - setup_efi() then returns to efi_main().
> 

Yes that a good idea, I will try to merge the two setup() functions.

Thanks,

Nikos

> That way the UEFI code can reuse most of the dcache maintenance that is
> proposed in this series [1], and reuse most of the existing boot code. What
> do you think?
> 
> [1] https://lore.kernel.org/kvmarm/20220809091558.14379-1-alexandru.elisei@arm.com/
> 
> Thanks,
> Alex
> 
>>
>> Thanks,
>>
>> Nikos
>>
>>> One way to get around this is never to run C code with the MMU off. That's
>>> relatively easy to do in the boot code, as the translation tables can be
>>> constructed with the MMU on, and then a fairly small assembly sequence is
>>> required to install them.
>>>
>>> But arm64 also has two mechanisms for disabling the MMU:
>>>
>>> 1. At compile time, the user can request a test to start with the MMU off,
>>> by setting the flag AUXINFO_MMU_OFF. So when booting as an UEFI app,
>>> kvm-unit-tests must disable the MMU.
>>>
>>> 2. A function called mmu_disable() which allows a test to explicitly
>>> disable the MMU.
>>>
>>> If we want to keep the UEFI allocated stack, then both mechanism must be
>>> forbidden when running under UEFI. I dislike this idea, because those two
>>> mechanisms allow kvm-unit-tests to run tests which otherwise wouldn't have
>>> been possible with a normal operating system, which, except for the early
>>> boot code, runs with the MMU enabled.
>>>
>>> Any thoughts or comments about this?
>>>
>>> *UEFI v2.8 states about the stack: "128 KiB or more of available stack
>>> space" (page 35), but EDK2 allocates 64KiB [3]. So without any firmware
>>> call to query the size of the stack, kvm-unit-tests cannot rely on it being
>>> a specific size.
>>>
>>> [1] https://lore.kernel.org/kvm/20220809091558.14379-19-alexandru.elisei@arm.com/
>>> [2] https://lore.kernel.org/kvm/20220809091558.14379-20-alexandru.elisei@arm.com/
>>> [3] https://github.com/tianocore/edk2/blob/master/ArmPlatformPkg/ArmPlatformPkg.dec#L71
>>>
>>> Thanks,
>>> Alex
>>>
>>> On Thu, Jun 30, 2022 at 11:02:57AM +0100, Nikos Nikoleris wrote:
>>>> Hello,
>>>>
>>>> This patch series adds initial support for building arm64 tests as EFI
>>>> tests and running them under QEMU. Much like x86_64 we import external
>>>> dependencies from gnu-efi and adapt them to work with types and other
>>>> assumptions from kvm-unit-tests. In addition, this series adds support
>>>> for enumerating parts of the system using ACPI.
>>>>
>>>> The first set of patches moves the existing ACPI code to the common
>>>> lib path. Then, it extends definitions and functions to allow for more
>>>> robust discovery of ACPI tables. In arm64, we add support for setting
>>>> up the PSCI conduit, discovering the UART, timers and cpus via
>>>> ACPI. The code retains existing behavior and gives priority to
>>>> discovery through DT when one has been provided.
>>>>
>>>> In the second set of patches, we add support for getting the command
>>>> line from the EFI shell. This is a requirement for many of the
>>>> existing arm64 tests.
>>>>
>>>> In the third set of patches, we import code from gnu-efi, make minor
>>>> changes and add an alternative setup sequence from arm64 systems that
>>>> boot through EFI. Finally, we add support in the build system and a
>>>> run script which is used to run an EFI app.
>>>>
>>>> After this set of patches one can build arm64 EFI tests:
>>>>
>>>> $> ./configure --enable-efi
>>>> $> make
>>>>
>>>> And use the run script to run an EFI tests:
>>>>
>>>> $> ./arm/efi/run ./arm/selftest.efi -smp 2 -m 256 -append "setup smp=2 mem=256"
>>>>
>>>> Or all tests:
>>>>
>>>> $> ./run_tests.sh
>>>>
>>>> There are a few items that this series does not address but they would
>>>> be useful to have:
>>>>    - Support for booting the system from EL2. Currently, we assume that a
>>>>      tests starts running at EL1. This the case when we run with EFI, it's
>>>>      not always the case in hardware.
>>>>    - Support for reading environment variables and populating __envp.
>>>>    - Support for discovering the PCI subsystem using ACPI.
>>>>    - Get rid of other assumptions (e.g., vmalloc area) that don't hold on real HW.
>>>>
>>>> git branch: https://github.com/relokin/kvm-unit-tests/pull/new/target-efi-upstream-v3-rebased
>>>>
>>>> v2: https://lore.kernel.org/kvm/20220506205605.359830-1-nikos.nikoleris@arm.com/
>>>>
>>>> Changes in v3:
>>>>    - Addressed feedback from Drew, Alex and Ricardo. Many thanks for the reviews!
>>>>    - Added support for discovering the GIC through ACPI
>>>>    - Added a missing header file (<elf.h>)
>>>>    - Added support for correctly parsing the outcome of tests (./run_tests)
>>>>
>>>> Thanks,
>>>>
>>>> Nikos
>>>>
>>>> Alexandru Elisei (1):
>>>>     lib: arm: Print test exit status
>>>>
>>>> Andrew Jones (3):
>>>>     arm/arm64: mmu_disable: Clean and invalidate before disabling
>>>>     arm/arm64: Rename etext to _etext
>>>>     arm64: Add a new type of memory type flag MR_F_RESERVED
>>>>
>>>> Nikos Nikoleris (23):
>>>>     lib: Fix style for acpi.{c,h}
>>>>     x86: Avoid references to fields of ACPI tables
>>>>     lib: Ensure all struct definition for ACPI tables are packed
>>>>     lib: Add support for the XSDT ACPI table
>>>>     lib: Extend the definition of the ACPI table FADT
>>>>     devicetree: Check if fdt is NULL before returning that a DT is
>>>>       available
>>>>     arm/arm64: Add support for setting up the PSCI conduit through ACPI
>>>>     arm/arm64: Add support for discovering the UART through ACPI
>>>>     arm/arm64: Add support for timer initialization through ACPI
>>>>     arm/arm64: Add support for cpu initialization through ACPI
>>>>     arm/arm64: Add support for gic initialization through ACPI
>>>>     lib/printf: Support for precision modifier in printf
>>>>     lib/printf: Add support for printing wide strings
>>>>     lib/efi: Add support for getting the cmdline
>>>>     lib: Avoid ms_abi for calls related to EFI on arm64
>>>>     arm/arm64: Add a setup sequence for systems that boot through EFI
>>>>     arm64: Copy code from GNU-EFI
>>>>     arm64: Change GNU-EFI imported file to use defined types
>>>>     arm64: Use code from the gnu-efi when booting with EFI
>>>>     lib: Avoid external dependency in libelf
>>>>     x86: Move x86_64-specific EFI CFLAGS to x86_64 Makefile
>>>>     arm64: Add support for efi in Makefile
>>>>     arm64: Add an efi/run script
>>>>
>>>>    scripts/runtime.bash        |  14 +-
>>>>    arm/efi/run                 |  61 +++++++
>>>>    arm/run                     |  14 +-
>>>>    configure                   |  15 +-
>>>>    Makefile                    |   4 -
>>>>    arm/Makefile.arm            |   6 +
>>>>    arm/Makefile.arm64          |  18 +-
>>>>    arm/Makefile.common         |  48 +++--
>>>>    x86/Makefile.x86_64         |   4 +
>>>>    lib/linux/efi.h             |  25 +++
>>>>    lib/arm/asm/setup.h         |   3 +
>>>>    lib/arm/asm/timer.h         |   2 +
>>>>    lib/acpi.h                  | 348 ++++++++++++++++++++++++++++--------
>>>>    lib/argv.h                  |   1 +
>>>>    lib/elf.h                   |  57 ++++++
>>>>    lib/libcflat.h              |   1 +
>>>>    lib/acpi.c                  | 129 ++++++++-----
>>>>    lib/argv.c                  |   2 +-
>>>>    lib/arm/gic.c               | 127 ++++++++++++-
>>>>    lib/arm/io.c                |  29 ++-
>>>>    lib/arm/mmu.c               |   4 +
>>>>    lib/arm/psci.c              |  25 ++-
>>>>    lib/arm/setup.c             | 247 ++++++++++++++++++++-----
>>>>    lib/arm/timer.c             |  79 ++++++++
>>>>    lib/devicetree.c            |   2 +-
>>>>    lib/efi.c                   | 102 +++++++++++
>>>>    lib/printf.c                | 194 ++++++++++++++++++--
>>>>    arm/efi/elf_aarch64_efi.lds |  63 +++++++
>>>>    arm/flat.lds                |   2 +-
>>>>    arm/cstart.S                |  29 ++-
>>>>    arm/cstart64.S              |  28 ++-
>>>>    arm/efi/crt0-efi-aarch64.S  | 143 +++++++++++++++
>>>>    arm/dummy.c                 |   4 +
>>>>    arm/efi/reloc_aarch64.c     |  93 ++++++++++
>>>>    arm/micro-bench.c           |   4 +-
>>>>    arm/timer.c                 |  10 +-
>>>>    x86/s3.c                    |  19 +-
>>>>    x86/vmexit.c                |   2 +-
>>>>    38 files changed, 1700 insertions(+), 258 deletions(-)
>>>>    create mode 100755 arm/efi/run
>>>>    create mode 100644 lib/elf.h
>>>>    create mode 100644 lib/arm/timer.c
>>>>    create mode 100644 arm/efi/elf_aarch64_efi.lds
>>>>    create mode 100644 arm/efi/crt0-efi-aarch64.S
>>>>    create mode 100644 arm/dummy.c
>>>>    create mode 100644 arm/efi/reloc_aarch64.c
>>>>
>>>> -- 
>>>> 2.25.1
>>>>