[RFC 00/13] arm: Allow the user specifying where the RAM is place in the memory

[RFC 00/13] arm: Allow the user specifying where the RAM is place in the memory

[Julien Grall]

Hi all,

At the moment, a user is only able to specify the amount of RAM used by the
guest. Where the RAM will live is left to the software and hardcoded.

It could be useful for testing purpose to move the RAM in different place.
This series adds the possibility for the user to specify multiple RAM region.

The option -m/--mem is extended to specify the address using the following
format: <size>@<addr>. The option needs to be repeated as many times as the
number of RAM region in the guest layout.

For instance, if you want 512MB at 3GB and 512MB 4GB it would look like:
    -m 512 at 0xc0000000 -m 512 at 0x100000000

Note that the memory layout is not yet fully configurable by the user, so the
MMIO region is still living below 2GB. This means RAM cannot live in the
region 0-2GB. This could be changed in the future.

This new version also integrates work from Suzuki to allow the user specifying
the IPA size for the guest. This was previously sent separately on kvmarm [1].

Cheers,

[1] https://lkml.org/lkml/2018/3/27/437

Julien Grall (7):
  kvm__arch_init: Don't pass hugetlbfs_path and ram_size in parameter
  virtio/scsi: Allow to use multiple banks
  Fold kvm__init_ram call in kvm__arch_init
  kvm__arch_sanitize_cfg
  arm: Move anything related to RAM initialization in kvm__init_ram
  Allow the user to specify where the RAM is placed in the memory
  arm: Add support for multi memory regions

Suzuki K Poulose (6):
  arm: Allow use of hugepage with 16K pagesize host
  virtio: Handle aborts using invalid PFN
  kvmtool: Allow backends to run checks on the KVM device fd
  kvmtool: arm64: Add support for guest physical address size
  kvmtool: arm64: Switch memory layout
  kvmtool: arm/arm64: Add support for creating VM with PA size

 arm/aarch32/include/kvm/kvm-arch.h        |   3 +-
 arm/aarch64/include/kvm/kvm-arch.h        |  15 ++-
 arm/aarch64/include/kvm/kvm-config-arch.h |   5 +-
 arm/fdt.c                                 |  17 ++-
 arm/include/arm-common/kvm-arch.h         |  32 ++++--
 arm/include/arm-common/kvm-config-arch.h  |   1 +
 arm/kvm.c                                 | 183 +++++++++++++++++++++++++-----
 builtin-run.c                             |  87 ++++++++++++--
 include/kvm/kvm-config.h                  |  16 ++-
 include/kvm/kvm.h                         |  28 ++++-
 kvm.c                                     |  10 +-
 mips/kvm.c                                |  47 +++++---
 powerpc/kvm.c                             |  33 +++---
 virtio/mmio.c                             |  14 ++-
 virtio/pci.c                              |  10 +-
 virtio/scsi.c                             |  21 ++--
 x86/bios.c                                |   8 +-
 x86/kvm.c                                 |  54 +++++----
 18 files changed, 443 insertions(+), 141 deletions(-)

-- 
2.11.0

[RFC 01/13] arm: Allow use of hugepage with 16K pagesize host

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

With 16K pagesize, the hugepage size is 32M. Align the guest
memory to the hugepagesize for 16K.

Cc: Marc Zyngier <marc.zyngier@arm.com>
Cc: Andre Przywara <andre.przywara@arm.com>
Cc: Will Deacon <will.deacon@arm.com>
Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Julien Grall <julien.grall@arm.com>
---
 arm/kvm.c | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/arm/kvm.c b/arm/kvm.c
index 2ab436e..c4ab5c0 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -59,14 +59,22 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 
 void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
 {
+	unsigned long alignment;
+
 	/*
 	 * Allocate guest memory. We must align our buffer to 64K to
 	 * correlate with the maximum guest page size for virtio-mmio.
 	 * If using THP, then our minimal alignment becomes 2M.
 	 * 2M trumps 64K, so let's go with that.
+	 * If we are running with 16K page size, align the memory to
+	 * 32M, so that we can make use of the THP.
 	 */
+	if (sysconf(_SC_PAGESIZE) == SZ_16K)
+		alignment = SZ_32M;
+	else
+		alignment = SZ_2M;
 	kvm->ram_size = min(ram_size, (u64)ARM_MAX_MEMORY(kvm));
-	kvm->arch.ram_alloc_size = kvm->ram_size + SZ_2M;
+	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
 	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
 						kvm->arch.ram_alloc_size);
 
@@ -75,7 +83,7 @@ void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
 		    kvm->arch.ram_alloc_size, errno);
 
 	kvm->ram_start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
-					SZ_2M);
+					alignment);
 
 	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
 		MADV_MERGEABLE);
-- 
2.11.0

[RFC 02/13] kvm__arch_init: Don't pass hugetlbfs_path and ram_size in parameter

[Julien Grall]

The structure KVM already contains a pointer to the configuration. Both
hugetlbfs_path and ram_size are part of the configuration, so is it not
necessary to path them again in parameter.

Signed-off-by: Julien Grall <julien.grall@arm.com>
---
 arm/kvm.c         | 7 +++++--
 include/kvm/kvm.h | 2 +-
 kvm.c             | 2 +-
 mips/kvm.c        | 5 ++++-
 powerpc/kvm.c     | 5 ++++-
 x86/kvm.c         | 5 ++++-
 6 files changed, 19 insertions(+), 7 deletions(-)

diff --git a/arm/kvm.c b/arm/kvm.c
index c4ab5c0..7424887 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -57,10 +57,13 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 {
 }
 
-void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
+void kvm__arch_init(struct kvm *kvm)
 {
 	unsigned long alignment;
 
+	/* Convenience aliases */
+	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+
 	/*
 	 * Allocate guest memory. We must align our buffer to 64K to
 	 * correlate with the maximum guest page size for virtio-mmio.
@@ -73,7 +76,7 @@ void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
 		alignment = SZ_32M;
 	else
 		alignment = SZ_2M;
-	kvm->ram_size = min(ram_size, (u64)ARM_MAX_MEMORY(kvm));
+	kvm->ram_size = min(kvm->cfg.ram_size, (u64)ARM_MAX_MEMORY(kvm));
 	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
 	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
 						kvm->arch.ram_alloc_size);
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index 90463b8..95bab40 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -104,7 +104,7 @@ int kvm__enumerate_instances(int (*callback)(const char *name, int pid));
 void kvm__remove_socket(const char *name);
 
 void kvm__arch_set_cmdline(char *cmdline, bool video);
-void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size);
+void kvm__arch_init(struct kvm *kvm);
 void kvm__arch_delete_ram(struct kvm *kvm);
 int kvm__arch_setup_firmware(struct kvm *kvm);
 int kvm__arch_free_firmware(struct kvm *kvm);
diff --git a/kvm.c b/kvm.c
index f8f2fdc..98b3cb8 100644
--- a/kvm.c
+++ b/kvm.c
@@ -317,7 +317,7 @@ int kvm__init(struct kvm *kvm)
 		goto err_vm_fd;
 	}
 
-	kvm__arch_init(kvm, kvm->cfg.hugetlbfs_path, kvm->cfg.ram_size);
+	kvm__arch_init(kvm);
 
 	INIT_LIST_HEAD(&kvm->mem_banks);
 	kvm__init_ram(kvm);
diff --git a/mips/kvm.c b/mips/kvm.c
index 24bd650..cdb0007 100644
--- a/mips/kvm.c
+++ b/mips/kvm.c
@@ -57,9 +57,12 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 }
 
 /* Architecture-specific KVM init */
-void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
+void kvm__arch_init(struct kvm *kvm)
 {
 	int ret;
+	/* Convenience aliases */
+	u64 ram_size = kvm->cfg.ram_size;
+	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
 
 	kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram_size);
 	kvm->ram_size = ram_size;
diff --git a/powerpc/kvm.c b/powerpc/kvm.c
index c738c1d..0f2dab3 100644
--- a/powerpc/kvm.c
+++ b/powerpc/kvm.c
@@ -88,10 +88,13 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 }
 
 /* Architecture-specific KVM init */
-void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
+void kvm__arch_init(struct kvm *kvm)
 {
 	int cap_ppc_rma;
 	unsigned long hpt;
+	/* Convenience aliases */
+	u64 ram_size = kvm->cfg.ram_size;
+	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
 
 	kvm->ram_size		= ram_size;
 
diff --git a/x86/kvm.c b/x86/kvm.c
index d8751e9..453922f 100644
--- a/x86/kvm.c
+++ b/x86/kvm.c
@@ -130,10 +130,13 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 }
 
 /* Architecture-specific KVM init */
-void kvm__arch_init(struct kvm *kvm, const char *hugetlbfs_path, u64 ram_size)
+void kvm__arch_init(struct kvm *kvm)
 {
 	struct kvm_pit_config pit_config = { .flags = 0, };
 	int ret;
+	/* Convenience aliases */
+	u64 ram_size = kvm->cfg.ram_size;
+	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
 
 	ret = ioctl(kvm->vm_fd, KVM_SET_TSS_ADDR, 0xfffbd000);
 	if (ret < 0)
-- 
2.11.0

[RFC 03/13] virtio/scsi: Allow to use multiple banks

[Julien Grall]

At the moment, virtio scsi only register a bank starting at 0. On some
architectures this may not be true and the guest may have multiple
memory region.

Register all the memory regions to vhost by browsing kvm->mem_banks. The
code is based on the virtio_net__vhost_init implementation.

Signed-off-by: Julien Grall <julien.grall@arm.com>

---

This code was not tested as I don't have any setup with scsi.
---
 virtio/scsi.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/virtio/scsi.c b/virtio/scsi.c
index 58d2353..ee58e5f 100644
--- a/virtio/scsi.c
+++ b/virtio/scsi.c
@@ -182,24 +182,29 @@ static struct virtio_ops scsi_dev_virtio_ops = {
 
 static void virtio_scsi_vhost_init(struct kvm *kvm, struct scsi_dev *sdev)
 {
+	struct kvm_mem_bank *bank;
 	struct vhost_memory *mem;
 	u64 features;
-	int r;
+	int r, i;
 
 	sdev->vhost_fd = open("/dev/vhost-scsi", O_RDWR);
 	if (sdev->vhost_fd < 0)
 		die_perror("Failed openning vhost-scsi device");
 
-	mem = calloc(1, sizeof(*mem) + sizeof(struct vhost_memory_region));
+	mem = calloc(1, sizeof(*mem) + kvm->mem_slots * sizeof(struct vhost_memory_region));
 	if (mem == NULL)
 		die("Failed allocating memory for vhost memory map");
 
-	mem->nregions = 1;
-	mem->regions[0] = (struct vhost_memory_region) {
-		.guest_phys_addr	= 0,
-		.memory_size		= kvm->ram_size,
-		.userspace_addr		= (unsigned long)kvm->ram_start,
-	};
+	i = 0;
+	list_for_each_entry(bank, &kvm->mem_banks, list) {
+		mem->regions[0] = (struct vhost_memory_region) {
+			.guest_phys_addr	= bank->guest_phys_addr,
+			.memory_size		= bank->size,
+			.userspace_addr		= (unsigned long)bank->host_addr,
+		};
+
+	}
+	mem->nregions = i;
 
 	r = ioctl(sdev->vhost_fd, VHOST_SET_OWNER);
 	if (r != 0)
-- 
2.11.0

[RFC 04/13] Fold kvm__init_ram call in kvm__arch_init

[Julien Grall]

The slipt between kvm__init_ram and kvm__arch_init for setting up the
RAM can be quite subttle. By moving the former call in the later, it
will be possible to make the RAM initialization easier on Arm with multi
RAM banks support.

Note that it is necessary to move the call to kvm__arch_init after the
initialization of the list kvm->mem_banks because kvm__init_ram is
relying on it.

Signed-off-by: Julien Grall <julien.grall@arm.com>
---
 arm/kvm.c         | 4 +++-
 include/kvm/kvm.h | 1 -
 kvm.c             | 4 +---
 mips/kvm.c        | 4 +++-
 powerpc/kvm.c     | 4 +++-
 x86/kvm.c         | 4 +++-
 6 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/arm/kvm.c b/arm/kvm.c
index 7424887..619390a 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -24,7 +24,7 @@ bool kvm__arch_cpu_supports_vm(void)
 	return true;
 }
 
-void kvm__init_ram(struct kvm *kvm)
+static void kvm__init_ram(struct kvm *kvm)
 {
 	int err;
 	u64 phys_start, phys_size;
@@ -97,6 +97,8 @@ void kvm__arch_init(struct kvm *kvm)
 	/* Create the virtual GIC. */
 	if (gic__create(kvm, kvm->cfg.arch.irqchip))
 		die("Failed to create virtual GIC");
+
+	kvm__init_ram(kvm);
 }
 
 #define FDT_ALIGN	SZ_2M
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index 95bab40..6ac5c66 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -79,7 +79,6 @@ int kvm__init(struct kvm *kvm);
 struct kvm *kvm__new(void);
 int kvm__recommended_cpus(struct kvm *kvm);
 int kvm__max_cpus(struct kvm *kvm);
-void kvm__init_ram(struct kvm *kvm);
 int kvm__exit(struct kvm *kvm);
 bool kvm__load_firmware(struct kvm *kvm, const char *firmware_filename);
 bool kvm__load_kernel(struct kvm *kvm, const char *kernel_filename,
diff --git a/kvm.c b/kvm.c
index 98b3cb8..e13d8f2 100644
--- a/kvm.c
+++ b/kvm.c
@@ -317,10 +317,8 @@ int kvm__init(struct kvm *kvm)
 		goto err_vm_fd;
 	}
 
-	kvm__arch_init(kvm);
-
 	INIT_LIST_HEAD(&kvm->mem_banks);
-	kvm__init_ram(kvm);
+	kvm__arch_init(kvm);
 
 	if (!kvm->cfg.firmware_filename) {
 		if (!kvm__load_kernel(kvm, kvm->cfg.kernel_filename,
diff --git a/mips/kvm.c b/mips/kvm.c
index cdb0007..61bf104 100644
--- a/mips/kvm.c
+++ b/mips/kvm.c
@@ -17,7 +17,7 @@ void kvm__arch_read_term(struct kvm *kvm)
 	virtio_console__inject_interrupt(kvm);
 }
 
-void kvm__init_ram(struct kvm *kvm)
+static void kvm__init_ram(struct kvm *kvm)
 {
 	u64	phys_start, phys_size;
 	void	*host_mem;
@@ -75,6 +75,8 @@ void kvm__arch_init(struct kvm *kvm)
 	ret = ioctl(kvm->vm_fd, KVM_CREATE_IRQCHIP);
 	if (ret < 0)
 		die_perror("KVM_CREATE_IRQCHIP ioctl");
+
+	kvm__init_ram(kvm);
 }
 
 void kvm__irq_line(struct kvm *kvm, int irq, int level)
diff --git a/powerpc/kvm.c b/powerpc/kvm.c
index 0f2dab3..2c8dbb7 100644
--- a/powerpc/kvm.c
+++ b/powerpc/kvm.c
@@ -60,7 +60,7 @@ bool kvm__arch_cpu_supports_vm(void)
 	return true;
 }
 
-void kvm__init_ram(struct kvm *kvm)
+static void kvm__init_ram(struct kvm *kvm)
 {
 	u64	phys_start, phys_size;
 	void	*host_mem;
@@ -139,6 +139,8 @@ void kvm__arch_init(struct kvm *kvm)
 			 SPAPR_PCI_MEM_WIN_SIZE,
 			 SPAPR_PCI_IO_WIN_ADDR,
 			 SPAPR_PCI_IO_WIN_SIZE);
+
+	kvm__init_ram(kvm);
 }
 
 void kvm__arch_delete_ram(struct kvm *kvm)
diff --git a/x86/kvm.c b/x86/kvm.c
index 453922f..03b0277 100644
--- a/x86/kvm.c
+++ b/x86/kvm.c
@@ -86,7 +86,7 @@ bool kvm__arch_cpu_supports_vm(void)
  * a gap between 0xe0000000 and 0x100000000 in the guest virtual mem space.
  */
 
-void kvm__init_ram(struct kvm *kvm)
+static void kvm__init_ram(struct kvm *kvm)
 {
 	u64	phys_start, phys_size;
 	void	*host_mem;
@@ -167,6 +167,8 @@ void kvm__arch_init(struct kvm *kvm)
 	ret = ioctl(kvm->vm_fd, KVM_CREATE_IRQCHIP);
 	if (ret < 0)
 		die_perror("KVM_CREATE_IRQCHIP ioctl");
+
+	kvm__init_ram(kvm);
 }
 
 void kvm__arch_delete_ram(struct kvm *kvm)
-- 
2.11.0

[RFC 05/13] kvm__arch_sanitize_cfg

[Julien Grall]

---
 arm/aarch64/include/kvm/kvm-arch.h |  2 +-
 arm/kvm.c                          | 13 ++++++++++++-
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/arm/aarch64/include/kvm/kvm-arch.h b/arm/aarch64/include/kvm/kvm-arch.h
index 9de623a..1b3d0a5 100644
--- a/arm/aarch64/include/kvm/kvm-arch.h
+++ b/arm/aarch64/include/kvm/kvm-arch.h
@@ -5,7 +5,7 @@
 				0x8000				:	\
 				0x80000)
 
-#define ARM_MAX_MEMORY(kvm)	((kvm)->cfg.arch.aarch32_guest	?	\
+#define ARM_MAX_MEMORY(cfg)	((cfg)->arch.aarch32_guest	?	\
 				ARM_LOMAP_MAX_MEMORY		:	\
 				ARM_HIMAP_MAX_MEMORY)
 
diff --git a/arm/kvm.c b/arm/kvm.c
index 619390a..71aad9d 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -57,6 +57,15 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 {
 }
 
+static void kvm__arch_sanitize_cfg(struct kvm_config *cfg)
+{
+	if (cfg->ram_size > ARM_MAX_MEMORY(cfg)) {
+		cfg->ram_size = ARM_MAX_MEMORY(cfg);
+		pr_warning("sanitize: Capping memory to %lluMB",
+			   cfg->ram_size >> 20);
+	}
+}
+
 void kvm__arch_init(struct kvm *kvm)
 {
 	unsigned long alignment;
@@ -64,6 +73,8 @@ void kvm__arch_init(struct kvm *kvm)
 	/* Convenience aliases */
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
 
+	kvm__arch_sanitize_cfg(&kvm->cfg);
+
 	/*
 	 * Allocate guest memory. We must align our buffer to 64K to
 	 * correlate with the maximum guest page size for virtio-mmio.
@@ -76,7 +87,7 @@ void kvm__arch_init(struct kvm *kvm)
 		alignment = SZ_32M;
 	else
 		alignment = SZ_2M;
-	kvm->ram_size = min(kvm->cfg.ram_size, (u64)ARM_MAX_MEMORY(kvm));
+	kvm->ram_size = kvm->cfg.ram_size;
 	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
 	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
 						kvm->arch.ram_alloc_size);
-- 
2.11.0

[RFC 06/13] arm: Move anything related to RAM initialization in kvm__init_ram

[Julien Grall]

---
 arm/kvm.c | 68 +++++++++++++++++++++++++++++++--------------------------------
 1 file changed, 33 insertions(+), 35 deletions(-)

diff --git a/arm/kvm.c b/arm/kvm.c
index 71aad9d..daa7425 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -29,6 +29,39 @@ static void kvm__init_ram(struct kvm *kvm)
 	int err;
 	u64 phys_start, phys_size;
 	void *host_mem;
+	unsigned long alignment;
+	/* Convenience aliases */
+	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+
+	/*
+	 * Allocate guest memory. We must align our buffer to 64K to
+	 * correlate with the maximum guest page size for virtio-mmio.
+	 * If using THP, then our minimal alignment becomes 2M.
+	 * 2M trumps 64K, so let's go with that.
+	 * If we are running with 16K page size, align the memory to
+	 * 32M, so that we can make use of the THP.
+	 */
+	if (sysconf(_SC_PAGESIZE) == SZ_16K)
+		alignment = SZ_32M;
+	else
+		alignment = SZ_2M;
+	kvm->ram_size = kvm->cfg.ram_size;
+	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
+	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
+						kvm->arch.ram_alloc_size);
+
+	if (kvm->arch.ram_alloc_start == MAP_FAILED)
+		die("Failed to map %lld bytes for guest memory (%d)",
+		    kvm->arch.ram_alloc_size, errno);
+
+	kvm->ram_start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
+					SZ_2M);
+
+	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
+		MADV_MERGEABLE);
+
+	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
+		MADV_HUGEPAGE);
 
 	phys_start	= ARM_MEMORY_AREA;
 	phys_size	= kvm->ram_size;
@@ -68,43 +101,8 @@ static void kvm__arch_sanitize_cfg(struct kvm_config *cfg)
 
 void kvm__arch_init(struct kvm *kvm)
 {
-	unsigned long alignment;
-
-	/* Convenience aliases */
-	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
-
 	kvm__arch_sanitize_cfg(&kvm->cfg);
 
-	/*
-	 * Allocate guest memory. We must align our buffer to 64K to
-	 * correlate with the maximum guest page size for virtio-mmio.
-	 * If using THP, then our minimal alignment becomes 2M.
-	 * 2M trumps 64K, so let's go with that.
-	 * If we are running with 16K page size, align the memory to
-	 * 32M, so that we can make use of the THP.
-	 */
-	if (sysconf(_SC_PAGESIZE) == SZ_16K)
-		alignment = SZ_32M;
-	else
-		alignment = SZ_2M;
-	kvm->ram_size = kvm->cfg.ram_size;
-	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
-	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
-						kvm->arch.ram_alloc_size);
-
-	if (kvm->arch.ram_alloc_start == MAP_FAILED)
-		die("Failed to map %lld bytes for guest memory (%d)",
-		    kvm->arch.ram_alloc_size, errno);
-
-	kvm->ram_start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
-					alignment);
-
-	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
-		MADV_MERGEABLE);
-
-	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
-		MADV_HUGEPAGE);
-
 	/* Create the virtual GIC. */
 	if (gic__create(kvm, kvm->cfg.arch.irqchip))
 		die("Failed to create virtual GIC");
-- 
2.11.0

[RFC 07/13] Allow the user to specify where the RAM is placed in the memory

[Julien Grall]

At the moment the user is only able to give the amount of memory used by
the guest. The placement of the RAM in the layout is left to the
software. It would be useful to give more freedom to the user where the
RAM regions will live in the memory layout and the size of them.

The command line to specify the size of the memory (-m/-mem) is now extended
in a compatible way to allow specifiying the base address: <size>[@<addr>].

The <addr> is mandatory if the option is specified multiple (i.e the
user request).

As not all the architecture will support multiple RAM region or even
specificying the address, it is left to the architecture to enable the
feature (see MAX_RAM_BANKS and ARCH_SUPPORT_CFG_RAM_BASE).

At the moment no-one is taking advantages of it, this will change in a
follow-up patch.

Signed-off-by: Julien Grall <julien.grall@arm.com>
---
 arm/fdt.c                |  2 +-
 arm/kvm.c                | 33 +++++++++---------
 builtin-run.c            | 87 +++++++++++++++++++++++++++++++++++++++++++-----
 include/kvm/kvm-config.h | 16 ++++++++-
 include/kvm/kvm.h        | 21 ++++++++++--
 kvm.c                    |  4 +++
 mips/kvm.c               | 40 ++++++++++++----------
 powerpc/kvm.c            | 26 ++++++++-------
 x86/bios.c               |  8 +++--
 x86/kvm.c                | 47 +++++++++++++++-----------
 10 files changed, 204 insertions(+), 80 deletions(-)

diff --git a/arm/fdt.c b/arm/fdt.c
index 980015b..6ac0b33 100644
--- a/arm/fdt.c
+++ b/arm/fdt.c
@@ -116,7 +116,7 @@ static int setup_fdt(struct kvm *kvm)
 	u8 staging_fdt[FDT_MAX_SIZE];
 	u64 mem_reg_prop[]	= {
 		cpu_to_fdt64(kvm->arch.memory_guest_start),
-		cpu_to_fdt64(kvm->ram_size),
+		cpu_to_fdt64(kvm->ram[0].size),
 	};
 	struct psci_fns *fns;
 	void *fdt		= staging_fdt;
diff --git a/arm/kvm.c b/arm/kvm.c
index daa7425..34abe69 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -27,11 +27,11 @@ bool kvm__arch_cpu_supports_vm(void)
 static void kvm__init_ram(struct kvm *kvm)
 {
 	int err;
-	u64 phys_start, phys_size;
-	void *host_mem;
+	u64 phys_start;
 	unsigned long alignment;
 	/* Convenience aliases */
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+	struct kvm_ram_region *ram = &kvm->ram[0];
 
 	/*
 	 * Allocate guest memory. We must align our buffer to 64K to
@@ -45,8 +45,8 @@ static void kvm__init_ram(struct kvm *kvm)
 		alignment = SZ_32M;
 	else
 		alignment = SZ_2M;
-	kvm->ram_size = kvm->cfg.ram_size;
-	kvm->arch.ram_alloc_size = kvm->ram_size + alignment;
+	ram->size = kvm->cfg.ram[0].size;
+	kvm->arch.ram_alloc_size = ram->size + alignment;
 	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
 						kvm->arch.ram_alloc_size);
 
@@ -54,8 +54,8 @@ static void kvm__init_ram(struct kvm *kvm)
 		die("Failed to map %lld bytes for guest memory (%d)",
 		    kvm->arch.ram_alloc_size, errno);
 
-	kvm->ram_start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
-					SZ_2M);
+	ram->start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
+				   SZ_2M);
 
 	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
 		MADV_MERGEABLE);
@@ -64,13 +64,11 @@ static void kvm__init_ram(struct kvm *kvm)
 		MADV_HUGEPAGE);
 
 	phys_start	= ARM_MEMORY_AREA;
-	phys_size	= kvm->ram_size;
-	host_mem	= kvm->ram_start;
 
-	err = kvm__register_mem(kvm, phys_start, phys_size, host_mem);
+	err = kvm__register_mem(kvm, phys_start, ram->size, ram->start);
 	if (err)
 		die("Failed to register %lld bytes of memory at physical "
-		    "address 0x%llx [err %d]", phys_size, phys_start, err);
+		    "address 0x%llx [err %d]", ram->size, phys_start, err);
 
 	kvm->arch.memory_guest_start = phys_start;
 }
@@ -92,10 +90,13 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 
 static void kvm__arch_sanitize_cfg(struct kvm_config *cfg)
 {
-	if (cfg->ram_size > ARM_MAX_MEMORY(cfg)) {
-		cfg->ram_size = ARM_MAX_MEMORY(cfg);
+	/* Convenience aliases */
+	struct kvm_ram_config *bank0 = &cfg->ram[0];
+
+	if (bank0->size > ARM_MAX_MEMORY(cfg)) {
+		bank0->size = ARM_MAX_MEMORY(cfg);
 		pr_warning("sanitize: Capping memory to %lluMB",
-			   cfg->ram_size >> 20);
+			   bank0->size >> 20);
 	}
 }
 
@@ -118,14 +119,16 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	void *pos, *kernel_end, *limit;
 	unsigned long guest_addr;
 	ssize_t file_size;
+	/* Convenience aliases */
+	struct kvm_ram_region *ram0 = &kvm->ram[0];
 
 	/*
 	 * Linux requires the initrd and dtb to be mapped inside lowmem,
 	 * so we can't just place them at the top of memory.
 	 */
-	limit = kvm->ram_start + min(kvm->ram_size, (u64)SZ_256M) - 1;
+	limit = ram0->start + min(ram0->size, (u64)SZ_256M) - 1;
 
-	pos = kvm->ram_start + ARM_KERN_OFFSET(kvm);
+	pos = ram0->start + ARM_KERN_OFFSET(kvm);
 	kvm->arch.kern_guest_start = host_to_guest_flat(kvm, pos);
 	file_size = read_file(fd_kernel, pos, limit - pos);
 	if (file_size < 0) {
diff --git a/builtin-run.c b/builtin-run.c
index b56aea7..5415d70 100644
--- a/builtin-run.c
+++ b/builtin-run.c
@@ -87,6 +87,63 @@ void kvm_run_set_wrapper_sandbox(void)
 	kvm_run_wrapper = KVM_RUN_SANDBOX;
 }
 
+static int mem_parser(const struct option *opt, const char *arg, int unset)
+{
+	struct kvm_config *cfg = opt->value;
+	const char *p = arg;
+	char *next;
+	u64 size, addr;
+	int base;
+
+	/* parse out size */
+	size = strtoll(p, &next, 10);
+
+	if (next == p)
+		die("mem: no size specified.\n");
+
+	if (*next != '@' && *next != '\0')
+		die("mem: unexpected chars after size.\n");
+
+	if (*next == '\0')
+		p = next;
+	else
+		p = next + 1;
+
+	/* parse out guest addr */
+	base = 10;
+	if (strcasestr(p, "0x"))
+		base = 16;
+	addr = strtoll(p, &next, base);
+	if (next == p && addr == 0) {
+		/*
+		 * To keep backward compatibility, the <addr> is not
+		 * mandatory for the first bank.
+		 */
+		if (cfg->nr_ram > 0)
+			die("mem: <addr> must be specified\n");
+		addr = INVALID_RAM_ADDR;
+	}
+
+	if ( cfg->nr_ram == MAX_RAM_BANKS )
+		die("mem: Too many banks\n");
+
+	/*
+	 * Allow the architecture to tell whether it is possible to configure
+	 * the RAM base address.
+	 */
+#ifndef ARCH_SUPPORT_CFG_RAM_BASE
+	if (addr == INVALID_RAM_ADDR)
+		die("mem: specifying <addr> is not allowed\n");
+#endif
+
+	cfg->ram[cfg->nr_ram].base = addr;
+	cfg->ram[cfg->nr_ram].size = size;
+
+	cfg->nr_ram++;
+
+	return 0;
+}
+
 #ifndef OPT_ARCH_RUN
 #define OPT_ARCH_RUN(...)
 #endif
@@ -97,8 +154,11 @@ void kvm_run_set_wrapper_sandbox(void)
 	OPT_STRING('\0', "name", &(cfg)->guest_name, "guest name",	\
 			"A name for the guest"),			\
 	OPT_INTEGER('c', "cpus", &(cfg)->nrcpus, "Number of CPUs"),	\
-	OPT_U64('m', "mem", &(cfg)->ram_size, "Virtual machine memory"	\
-		" size in MiB."),					\
+	OPT_CALLBACK('m', "mem", cfg,					\
+		     "<size>[@<addr>]",					\
+		     "Virtual machine memory region, size in MiB."	\
+		     "<addr> must be specified in case of multiple regions", \
+		     mem_parser, NULL),					\
 	OPT_CALLBACK('\0', "shmem", NULL,				\
 		     "[pci:]<addr>:<size>[:handle=<handle>][:create]",	\
 		     "Share host shmem with guest via pci device",	\
@@ -455,6 +515,8 @@ static struct kvm *kvm_cmd_run_init(int argc, const char **argv)
 	unsigned int nr_online_cpus;
 	struct kvm *kvm = kvm__new();
 	bool video;
+	u64 total_ram = 0;
+	unsigned int i;
 
 	if (IS_ERR(kvm))
 		return kvm;
@@ -521,16 +583,23 @@ static struct kvm *kvm_cmd_run_init(int argc, const char **argv)
 	if (kvm->cfg.nrcpus == 0)
 		kvm->cfg.nrcpus = nr_online_cpus;
 
-	if (!kvm->cfg.ram_size)
-		kvm->cfg.ram_size = get_ram_size(kvm->cfg.nrcpus);
+	/* Deal with user not specifiying memory */
+	if (kvm->cfg.nr_ram == 0 && !kvm->cfg.ram[0].size) {
+		kvm->cfg.ram[0].size = get_ram_size(kvm->cfg.nrcpus);
+		kvm->cfg.nr_ram = 1;
+	}
+
+	for (i = 0; i < kvm->cfg.nr_ram; i++) {
+		total_ram += kvm->cfg.ram[i].size;
+		/* The user input is in MB, but the internal deals with bytes */
+		kvm->cfg.ram[i].size <<= MB_SHIFT;
+	}
 
-	if (kvm->cfg.ram_size > host_ram_size())
+	if (total_ram > host_ram_size())
 		pr_warning("Guest memory size %lluMB exceeds host physical RAM size %lluMB",
-			(unsigned long long)kvm->cfg.ram_size,
+			(unsigned long long)total_ram,
 			(unsigned long long)host_ram_size());
 
-	kvm->cfg.ram_size <<= MB_SHIFT;
-
 	if (!kvm->cfg.dev)
 		kvm->cfg.dev = DEFAULT_KVM_DEV;
 
@@ -636,7 +705,7 @@ static struct kvm *kvm_cmd_run_init(int argc, const char **argv)
 
 	printf("  # %s run -k %s -m %Lu -c %d --name %s\n", KVM_BINARY_NAME,
 		kvm->cfg.kernel_filename,
-		(unsigned long long)kvm->cfg.ram_size / 1024 / 1024,
+		(unsigned long long)kvm->cfg.ram[0].size / 1024 / 1024,
 		kvm->cfg.nrcpus, kvm->cfg.guest_name);
 
 	if (init_list__init(kvm) < 0)
diff --git a/include/kvm/kvm-config.h b/include/kvm/kvm-config.h
index 386fa8c..9e5caf1 100644
--- a/include/kvm/kvm-config.h
+++ b/include/kvm/kvm-config.h
@@ -17,10 +17,24 @@
 #define MIN_RAM_SIZE_MB		(64ULL)
 #define MIN_RAM_SIZE_BYTE	(MIN_RAM_SIZE_MB << MB_SHIFT)
 
+#define INVALID_RAM_ADDR	(~0ULL)
+
+/* By default the user is only able to speficy one RAM bank */
+#ifndef MAX_RAM_BANKS
+#define MAX_RAM_BANKS	1
+#endif
+
+struct kvm_ram_config
+{
+	u64 base;
+	u64 size;
+};
+
 struct kvm_config {
 	struct kvm_config_arch arch;
 	struct disk_image_params disk_image[MAX_DISK_IMAGES];
-	u64 ram_size;
+	struct kvm_ram_config ram[MAX_RAM_BANKS];
+	u8 nr_ram;
 	u8  image_count;
 	u8 num_net_devices;
 	bool virtio_rng;
diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index 6ac5c66..4e21446 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -46,6 +46,11 @@ struct kvm_mem_bank {
 	u64			size;
 };
 
+struct kvm_ram_region {
+	void			*start;
+	u64			size;
+};
+
 struct kvm {
 	struct kvm_arch		arch;
 	struct kvm_config	cfg;
@@ -57,8 +62,8 @@ struct kvm {
 	struct kvm_cpu		**cpus;
 
 	u32			mem_slots;	/* for KVM_SET_USER_MEMORY_REGION */
-	u64			ram_size;
-	void			*ram_start;
+	struct kvm_ram_region	ram[MAX_RAM_BANKS];
+	u32			nr_ram;
 	u64			ram_pagesize;
 	struct list_head	mem_banks;
 
@@ -125,7 +130,17 @@ extern const char *kvm_exit_reasons[];
 
 static inline bool host_ptr_in_ram(struct kvm *kvm, void *p)
 {
-	return kvm->ram_start <= p && p < (kvm->ram_start + kvm->ram_size);
+	unsigned int i;
+
+	for (i = 0; i < kvm->nr_ram; i++ )
+	{
+		struct kvm_ram_region *ram = &kvm->ram[i];
+
+		if (ram->start <= p && p < (ram->start + ram->size))
+			return true;
+	}
+
+	return false;
 }
 
 bool kvm__supports_extension(struct kvm *kvm, unsigned int extension);
diff --git a/kvm.c b/kvm.c
index e13d8f2..9c2cb2f 100644
--- a/kvm.c
+++ b/kvm.c
@@ -151,6 +151,7 @@ static int kvm__check_extensions(struct kvm *kvm)
 
 struct kvm *kvm__new(void)
 {
+	unsigned int i;
 	struct kvm *kvm = calloc(1, sizeof(*kvm));
 	if (!kvm)
 		return ERR_PTR(-ENOMEM);
@@ -158,6 +159,9 @@ struct kvm *kvm__new(void)
 	kvm->sys_fd = -1;
 	kvm->vm_fd = -1;
 
+	for ( i = 0; i < MAX_RAM_BANKS; i++)
+		kvm->cfg.ram[i].base = INVALID_RAM_ADDR;
+
 	return kvm;
 }
 
diff --git a/mips/kvm.c b/mips/kvm.c
index 61bf104..17e7ca5 100644
--- a/mips/kvm.c
+++ b/mips/kvm.c
@@ -21,26 +21,28 @@ static void kvm__init_ram(struct kvm *kvm)
 {
 	u64	phys_start, phys_size;
 	void	*host_mem;
+	/* Convenience aliases */
+	const struct kvm_ram_region *ram = &kvm->ram[0];
 
-	if (kvm->ram_size <= KVM_MMIO_START) {
+	if (ram->size <= KVM_MMIO_START) {
 		/* one region for all memory */
 		phys_start = 0;
-		phys_size  = kvm->ram_size;
-		host_mem   = kvm->ram_start;
+		phys_size  = ram->size;
+		host_mem   = ram->start;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 	} else {
 		/* one region for memory that fits below MMIO range */
 		phys_start = 0;
 		phys_size  = KVM_MMIO_START;
-		host_mem   = kvm->ram_start;
+		host_mem   = ram->start;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 
 		/* one region for rest of memory */
 		phys_start = KVM_MMIO_START + KVM_MMIO_SIZE;
-		phys_size  = kvm->ram_size - KVM_MMIO_START;
-		host_mem   = kvm->ram_start + KVM_MMIO_START;
+		phys_size  = ram->size - KVM_MMIO_START;
+		host_mem   = ram->start + KVM_MMIO_START;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 	}
@@ -48,7 +50,7 @@ static void kvm__init_ram(struct kvm *kvm)
 
 void kvm__arch_delete_ram(struct kvm *kvm)
 {
-	munmap(kvm->ram_start, kvm->ram_size);
+	munmap(kvm->ram[0].start, kvm->ram[0].size);
 }
 
 void kvm__arch_set_cmdline(char *cmdline, bool video)
@@ -61,16 +63,18 @@ void kvm__arch_init(struct kvm *kvm)
 {
 	int ret;
 	/* Convenience aliases */
-	u64 ram_size = kvm->cfg.ram_size;
+	u64 ram_size = kvm->cfg.ram[0].size;
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+	struct kvm_ram_region *ram = &kvm->ram[0];
 
-	kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram_size);
-	kvm->ram_size = ram_size;
+	kvm->nr_ram = 1;
+	ram->start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram_size);
+	ram->size = ram_size;
 
-	if (kvm->ram_start == MAP_FAILED)
+	if (ram->start == MAP_FAILED)
 		die("out of memory");
 
-	madvise(kvm->ram_start, kvm->ram_size, MADV_MERGEABLE);
+	madvise(ram->start, ram->size, MADV_MERGEABLE);
 
 	ret = ioctl(kvm->vm_fd, KVM_CREATE_IRQCHIP);
 	if (ret < 0)
@@ -124,20 +128,22 @@ int kvm__arch_setup_firmware(struct kvm *kvm)
 
 static void kvm__mips_install_cmdline(struct kvm *kvm)
 {
-	char *p = kvm->ram_start;
 	u64 cmdline_offset = 0x2000;
 	u64 argv_start = 0x3000;
 	u64 argv_offset = argv_start;
 	u64 argc = 0;
+	/* Convenience aliases */
+	struct kvm_ram_region *ram = &kvm->ram[0];
 
+	char *p = ram->start;
 
-	if ((u64) kvm->ram_size <= KVM_MMIO_START)
+	if ((u64) ram->size <= KVM_MMIO_START)
 		sprintf(p + cmdline_offset, "mem=0x%llx@0 ",
-			(unsigned long long)kvm->ram_size);
+			(unsigned long long)ram->size);
 	else
 		sprintf(p + cmdline_offset, "mem=0x%llx at 0 mem=0x%llx at 0x%llx ",
 			(unsigned long long)KVM_MMIO_START,
-			(unsigned long long)kvm->ram_size - KVM_MMIO_START,
+			(unsigned long long)ram->size - KVM_MMIO_START,
 			(unsigned long long)(KVM_MMIO_START + KVM_MMIO_SIZE));
 
 	strcat(p + cmdline_offset, kvm->cfg.real_cmdline); /* maximum size is 2K */
@@ -181,7 +187,7 @@ static bool load_flat_binary(struct kvm *kvm, int fd_kernel)
 	p = k_start = guest_flat_to_host(kvm, KERNEL_LOAD_ADDR);
 
 	kernel_size = read_file(fd_kernel, p,
-				kvm->cfg.ram_size - KERNEL_LOAD_ADDR);
+				kvm->cfg.ram[0].size - KERNEL_LOAD_ADDR);
 	if (kernel_size == -1) {
 		if (errno == ENOMEM)
 			die("kernel too big for guest memory");
diff --git a/powerpc/kvm.c b/powerpc/kvm.c
index 2c8dbb7..2800ee7 100644
--- a/powerpc/kvm.c
+++ b/powerpc/kvm.c
@@ -66,8 +66,8 @@ static void kvm__init_ram(struct kvm *kvm)
 	void	*host_mem;
 
 	phys_start = 0;
-	phys_size  = kvm->ram_size;
-	host_mem   = kvm->ram_start;
+	phys_size  = kvm->ram[0].size;
+	host_mem   = kvm->ram[0].start;
 
 	/*
 	 * We put MMIO at PPC_MMIO_START, high up.  Make sure that this doesn't
@@ -93,26 +93,28 @@ void kvm__arch_init(struct kvm *kvm)
 	int cap_ppc_rma;
 	unsigned long hpt;
 	/* Convenience aliases */
-	u64 ram_size = kvm->cfg.ram_size;
+	u64 ram_size = kvm->cfg.ram[0].size;
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+	struct kvm_ram_region *ram = &kvm->ram[0];
 
-	kvm->ram_size		= ram_size;
+	kvm->nr_ram		= 1;
+	ram->size		= ram_size;
 
 	/* Map "default" hugetblfs path to the standard 16M mount point */
 	if (hugetlbfs_path && !strcmp(hugetlbfs_path, "default"))
 		hugetlbfs_path = HUGETLBFS_PATH;
 
-	kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, kvm->ram_size);
+	ram->start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram->size);
 
-	if (kvm->ram_start == MAP_FAILED)
+	if (ram->start == MAP_FAILED)
 		die("Couldn't map %lld bytes for RAM (%d)\n",
-		    kvm->ram_size, errno);
+		    ram->size, errno);
 
 	/* FDT goes at top of memory, RTAS just below */
-	kvm->arch.fdt_gra = kvm->ram_size - FDT_MAX_SIZE;
+	kvm->arch.fdt_gra = ram->size - FDT_MAX_SIZE;
 	/* FIXME: Not all PPC systems have RTAS */
 	kvm->arch.rtas_gra = kvm->arch.fdt_gra - RTAS_MAX_SIZE;
-	madvise(kvm->ram_start, kvm->ram_size, MADV_MERGEABLE);
+	madvise(ram->start, ram->size, MADV_MERGEABLE);
 
 	/* FIXME:  SPAPR-PR specific; allocate a guest HPT. */
 	if (posix_memalign((void **)&hpt, (1<<HPT_ORDER), (1<<HPT_ORDER)))
@@ -145,7 +147,7 @@ void kvm__arch_init(struct kvm *kvm)
 
 void kvm__arch_delete_ram(struct kvm *kvm)
 {
-	munmap(kvm->ram_start, kvm->ram_size);
+	munmap(kvm->ram[0].start, kvm->ram[0].size);
 }
 
 void kvm__irq_trigger(struct kvm *kvm, int irq)
@@ -186,7 +188,7 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd,
 		p = guest_flat_to_host(kvm, INITRD_LOAD_ADDR);
 
 		filesize = read_file(fd_initrd, p,
-			       (kvm->ram_start + kvm->ram_size) - p);
+			       (kvm->ram[0].start + kvm->ram[0].size) - p);
 		if (filesize < 0) {
 			if (errno == ENOMEM)
 				die("initrd too big to contain in guest RAM.\n");
@@ -283,7 +285,7 @@ static void generate_segment_page_sizes(struct kvm_ppc_smmu_info *info, struct f
  */
 static int setup_fdt(struct kvm *kvm)
 {
-	uint64_t 	mem_reg_property[] = { 0, cpu_to_be64(kvm->ram_size) };
+	uint64_t 	mem_reg_property[] = { 0, cpu_to_be64(kvm->ram[0].size) };
 	int 		smp_cpus = kvm->nrcpus;
 	uint32_t	int_server_ranges_prop[] = {0, cpu_to_be32(smp_cpus)};
 	char 		hypertas_prop_kvm[] = "hcall-pft\0hcall-term\0"
diff --git a/x86/bios.c b/x86/bios.c
index 5ac9e24..2efd771 100644
--- a/x86/bios.c
+++ b/x86/bios.c
@@ -59,6 +59,8 @@ static void e820_setup(struct kvm *kvm)
 	struct e820map *e820;
 	struct e820entry *mem_map;
 	unsigned int i = 0;
+	/* Convenience aliases */
+	const struct kvm_ram_region *ram = &kvm->ram[0];
 
 	e820		= guest_flat_to_host(kvm, E820_MAP_START);
 	mem_map		= e820->map;
@@ -78,10 +80,10 @@ static void e820_setup(struct kvm *kvm)
 		.size		= MB_BIOS_END - MB_BIOS_BEGIN,
 		.type		= E820_RESERVED,
 	};
-	if (kvm->ram_size < KVM_32BIT_GAP_START) {
+	if (ram->size < KVM_32BIT_GAP_START) {
 		mem_map[i++]	= (struct e820entry) {
 			.addr		= BZ_KERNEL_START,
-			.size		= kvm->ram_size - BZ_KERNEL_START,
+			.size		= ram->size - BZ_KERNEL_START,
 			.type		= E820_RAM,
 		};
 	} else {
@@ -92,7 +94,7 @@ static void e820_setup(struct kvm *kvm)
 		};
 		mem_map[i++]	= (struct e820entry) {
 			.addr		= KVM_32BIT_MAX_MEM_SIZE,
-			.size		= kvm->ram_size - KVM_32BIT_MAX_MEM_SIZE,
+			.size		= ram->size - KVM_32BIT_MAX_MEM_SIZE,
 			.type		= E820_RAM,
 		};
 	}
diff --git a/x86/kvm.c b/x86/kvm.c
index 03b0277..8faa166 100644
--- a/x86/kvm.c
+++ b/x86/kvm.c
@@ -90,13 +90,15 @@ static void kvm__init_ram(struct kvm *kvm)
 {
 	u64	phys_start, phys_size;
 	void	*host_mem;
+	/* Convenience aliases */
+	const struct kvm_ram_region *ram = &kvm->ram[0];
 
-	if (kvm->ram_size < KVM_32BIT_GAP_START) {
+	if (ram->size < KVM_32BIT_GAP_START) {
 		/* Use a single block of RAM for 32bit RAM */
 
 		phys_start = 0;
-		phys_size  = kvm->ram_size;
-		host_mem   = kvm->ram_start;
+		phys_size  = ram->size;
+		host_mem   = ram->start;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 	} else {
@@ -104,15 +106,15 @@ static void kvm__init_ram(struct kvm *kvm)
 
 		phys_start = 0;
 		phys_size  = KVM_32BIT_GAP_START;
-		host_mem   = kvm->ram_start;
+		host_mem   = ram->start;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 
 		/* Second RAM range from 4GB to the end of RAM: */
 
 		phys_start = KVM_32BIT_MAX_MEM_SIZE;
-		phys_size  = kvm->ram_size - phys_start;
-		host_mem   = kvm->ram_start + phys_start;
+		phys_size  = ram->size - phys_start;
+		host_mem   = ram->start + phys_start;
 
 		kvm__register_mem(kvm, phys_start, phys_size, host_mem);
 	}
@@ -135,8 +137,9 @@ void kvm__arch_init(struct kvm *kvm)
 	struct kvm_pit_config pit_config = { .flags = 0, };
 	int ret;
 	/* Convenience aliases */
-	u64 ram_size = kvm->cfg.ram_size;
+	u64 ram_size = kvm->cfg.ram[0].size;
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
+	struct kvm_ram_region *ram = &kvm->ram[0];
 
 	ret = ioctl(kvm->vm_fd, KVM_SET_TSS_ADDR, 0xfffbd000);
 	if (ret < 0)
@@ -146,23 +149,29 @@ void kvm__arch_init(struct kvm *kvm)
 	if (ret < 0)
 		die_perror("KVM_CREATE_PIT2 ioctl");
 
+	kvm->nr_ram = 1;
 	if (ram_size < KVM_32BIT_GAP_START) {
-		kvm->ram_size = ram_size;
-		kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram_size);
+		ram->size = ram_size;
+		ram->start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
+						    ram_size);
 	} else {
-		kvm->ram_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path, ram_size + KVM_32BIT_GAP_SIZE);
-		kvm->ram_size = ram_size + KVM_32BIT_GAP_SIZE;
-		if (kvm->ram_start != MAP_FAILED)
+		ram->start = mmap_anon_or_hugetlbfs(kvm,
+						    hugetlbfs_path,
+						    ram_size + KVM_32BIT_GAP_SIZE);
+		ram->size = ram_size + KVM_32BIT_GAP_SIZE;
+		if (ram->start != MAP_FAILED)
 			/*
 			 * We mprotect the gap (see kvm__init_ram() for details) PROT_NONE so that
 			 * if we accidently write to it, we will know.
 			 */
-			mprotect(kvm->ram_start + KVM_32BIT_GAP_START, KVM_32BIT_GAP_SIZE, PROT_NONE);
+			mprotect(ram->start + KVM_32BIT_GAP_START,
+				 KVM_32BIT_GAP_SIZE, PROT_NONE);
 	}
-	if (kvm->ram_start == MAP_FAILED)
+	if (ram->start == MAP_FAILED)
 		die("out of memory");
 
-	madvise(kvm->ram_start, kvm->ram_size, MADV_MERGEABLE);
+	madvise(ram->start, ram->size,
+		MADV_MERGEABLE);
 
 	ret = ioctl(kvm->vm_fd, KVM_CREATE_IRQCHIP);
 	if (ret < 0)
@@ -173,7 +182,7 @@ void kvm__arch_init(struct kvm *kvm)
 
 void kvm__arch_delete_ram(struct kvm *kvm)
 {
-	munmap(kvm->ram_start, kvm->ram_size);
+	munmap(kvm->ram[0].start, kvm->ram[0].size);
 }
 
 void kvm__irq_line(struct kvm *kvm, int irq, int level)
@@ -221,7 +230,7 @@ static bool load_flat_binary(struct kvm *kvm, int fd_kernel)
 
 	p = guest_real_to_host(kvm, BOOT_LOADER_SELECTOR, BOOT_LOADER_IP);
 
-	if (read_file(fd_kernel, p, kvm->cfg.ram_size) < 0)
+	if (read_file(fd_kernel, p, kvm->cfg.ram[0].size) < 0)
 		die_perror("read");
 
 	kvm->arch.boot_selector	= BOOT_LOADER_SELECTOR;
@@ -270,7 +279,7 @@ static bool load_bzimage(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	/* read actual kernel image (vmlinux.bin) to BZ_KERNEL_START */
 	p = guest_flat_to_host(kvm, BZ_KERNEL_START);
 	file_size = read_file(fd_kernel, p,
-			      kvm->cfg.ram_size - BZ_KERNEL_START);
+			      kvm->cfg.ram[0].size - BZ_KERNEL_START);
 	if (file_size < 0)
 		die_perror("kernel read");
 
@@ -316,7 +325,7 @@ static bool load_bzimage(struct kvm *kvm, int fd_kernel, int fd_initrd,
 		for (;;) {
 			if (addr < BZ_KERNEL_START)
 				die("Not enough memory for initrd");
-			else if (addr < (kvm->ram_size - initrd_stat.st_size))
+			else if (addr < (kvm->ram[0].size - initrd_stat.st_size))
 				break;
 			addr -= 0x100000;
 		}
-- 
2.11.0

[RFC 08/13] arm: Add support for multi memory regions

[Julien Grall]

Currently, the RAM will always be contiguous and start 2GB. This patch
is giving the possibility to the user to specificy multiple RAM region
and

Note that at the moment it is not possible to place any RAM region below
2GB as the MMIO region is still static.

The implementation for multi memory regions is fairly straight-forward,
althought it the following points are worth to be mention:
    - The banks are sorted by base address, so it is easier to fetch the
    lowest bank later on and do sanity checking
    - If the user does not specify the address of the first bank, the
    old behavior is kept.

Signed-off-by: Julien Grall <julien.grall@arm.com>
---
 arm/aarch32/include/kvm/kvm-arch.h |   2 +-
 arm/aarch64/include/kvm/kvm-arch.h |   4 +-
 arm/fdt.c                          |  17 +++--
 arm/include/arm-common/kvm-arch.h  |  20 ++++--
 arm/kvm.c                          | 125 ++++++++++++++++++++++++++++---------
 5 files changed, 125 insertions(+), 43 deletions(-)

diff --git a/arm/aarch32/include/kvm/kvm-arch.h b/arm/aarch32/include/kvm/kvm-arch.h
index cd31e72..2ee0cb1 100644
--- a/arm/aarch32/include/kvm/kvm-arch.h
+++ b/arm/aarch32/include/kvm/kvm-arch.h
@@ -3,7 +3,7 @@
 
 #define ARM_KERN_OFFSET(...)	0x8000
 
-#define ARM_MAX_MEMORY(...)	ARM_LOMAP_MAX_MEMORY
+#define ARM_MAX_PHYS_SHIFT(...)	32
 
 #include "arm-common/kvm-arch.h"
 
diff --git a/arm/aarch64/include/kvm/kvm-arch.h b/arm/aarch64/include/kvm/kvm-arch.h
index 1b3d0a5..53ac20f 100644
--- a/arm/aarch64/include/kvm/kvm-arch.h
+++ b/arm/aarch64/include/kvm/kvm-arch.h
@@ -5,9 +5,7 @@
 				0x8000				:	\
 				0x80000)
 
-#define ARM_MAX_MEMORY(cfg)	((cfg)->arch.aarch32_guest	?	\
-				ARM_LOMAP_MAX_MEMORY		:	\
-				ARM_HIMAP_MAX_MEMORY)
+#define ARM_MAX_PHYS_SHIFT(cfg)	((cfg)->arch.aarch32_guest ? 32 : 40)
 
 #include "arm-common/kvm-arch.h"
 
diff --git a/arm/fdt.c b/arm/fdt.c
index 6ac0b33..add9dea 100644
--- a/arm/fdt.c
+++ b/arm/fdt.c
@@ -114,10 +114,7 @@ static int setup_fdt(struct kvm *kvm)
 {
 	struct device_header *dev_hdr;
 	u8 staging_fdt[FDT_MAX_SIZE];
-	u64 mem_reg_prop[]	= {
-		cpu_to_fdt64(kvm->arch.memory_guest_start),
-		cpu_to_fdt64(kvm->ram[0].size),
-	};
+	fdt64_t mem_reg_prop[2 * MAX_RAM_BANKS];
 	struct psci_fns *fns;
 	void *fdt		= staging_fdt;
 	void *fdt_dest		= guest_flat_to_host(kvm,
@@ -126,6 +123,8 @@ static int setup_fdt(struct kvm *kvm)
 					void (*)(void *, u8, enum irq_type));
 	void (*generate_cpu_peripheral_fdt_nodes)(void *, struct kvm *)
 					= kvm->cpus[0]->generate_fdt_nodes;
+	unsigned int i;
+	const struct kvm_config *cfg = &kvm->cfg;
 
 	/* Create new tree without a reserve map */
 	_FDT(fdt_create(fdt, FDT_MAX_SIZE));
@@ -158,9 +157,17 @@ static int setup_fdt(struct kvm *kvm)
 	_FDT(fdt_end_node(fdt));
 
 	/* Memory */
+	for (i = 0; i < cfg->nr_ram; i++) {
+		fdt64_t *reg = &mem_reg_prop[i * 2];
+
+		reg[0] = cpu_to_fdt64(cfg->ram[i].base);
+		reg[1] = cpu_to_fdt64(cfg->ram[i].size);
+	}
+
 	_FDT(fdt_begin_node(fdt, "memory"));
 	_FDT(fdt_property_string(fdt, "device_type", "memory"));
-	_FDT(fdt_property(fdt, "reg", mem_reg_prop, sizeof(mem_reg_prop)));
+	_FDT(fdt_property(fdt, "reg", mem_reg_prop,
+			  2 * sizeof(fdt64_t) * cfg->nr_ram));
 	_FDT(fdt_end_node(fdt));
 
 	/* CPU and peripherals (interrupt controller, timers, etc) */
diff --git a/arm/include/arm-common/kvm-arch.h b/arm/include/arm-common/kvm-arch.h
index b9d486d..884dda8 100644
--- a/arm/include/arm-common/kvm-arch.h
+++ b/arm/include/arm-common/kvm-arch.h
@@ -12,9 +12,6 @@
 #define ARM_AXI_AREA		_AC(0x0000000040000000, UL)
 #define ARM_MEMORY_AREA		_AC(0x0000000080000000, UL)
 
-#define ARM_LOMAP_MAX_MEMORY	((1ULL << 32) - ARM_MEMORY_AREA)
-#define ARM_HIMAP_MAX_MEMORY	((1ULL << 40) - ARM_MEMORY_AREA)
-
 #define ARM_GIC_DIST_BASE	(ARM_AXI_AREA - ARM_GIC_DIST_SIZE)
 #define ARM_GIC_CPUI_BASE	(ARM_GIC_DIST_BASE - ARM_GIC_CPUI_SIZE)
 #define ARM_GIC_SIZE		(ARM_GIC_DIST_SIZE + ARM_GIC_CPUI_SIZE)
@@ -34,6 +31,8 @@
 
 #define KVM_IOEVENTFD_HAS_PIO	0
 
+#define ARM_MAX_PHYS_ADDR(cfg)	(1UL << ARM_MAX_PHYS_SHIFT(cfg))
+
 /*
  * On a GICv3 there must be one redistributor per vCPU.
  * The value here is the size for one, we multiply this@runtime with
@@ -56,18 +55,27 @@ static inline bool arm_addr_in_ioport_region(u64 phys_addr)
 	return phys_addr >= KVM_IOPORT_AREA && phys_addr < limit;
 }
 
+#define MAX_RAM_BANKS	8
+
+#define ARCH_SUPPORT_CFG_RAM_BASE 1
+
+struct kvm_arch_ram_region
+{
+	void	*start;
+	u64	size;
+};
+
 struct kvm_arch {
 	/*
 	 * We may have to align the guest memory for virtio, so keep the
 	 * original pointers here for munmap.
 	 */
-	void	*ram_alloc_start;
-	u64	ram_alloc_size;
+	struct kvm_arch_ram_region ram_alloc[MAX_RAM_BANKS];
 
 	/*
 	 * Guest addresses for memory layout.
 	 */
-	u64	memory_guest_start;
+	unsigned int	nr_mem_banks;
 	u64	kern_guest_start;
 	u64	initrd_guest_start;
 	u64	initrd_size;
diff --git a/arm/kvm.c b/arm/kvm.c
index 34abe69..8a7c611 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -11,6 +11,8 @@
 #include <linux/kvm.h>
 #include <linux/sizes.h>
 
+#include <stdlib.h>
+
 struct kvm_ext kvm_req_ext[] = {
 	{ DEFINE_KVM_EXT(KVM_CAP_IRQCHIP) },
 	{ DEFINE_KVM_EXT(KVM_CAP_ONE_REG) },
@@ -24,14 +26,15 @@ bool kvm__arch_cpu_supports_vm(void)
 	return true;
 }
 
-static void kvm__init_ram(struct kvm *kvm)
+static void kvm__init_ram(struct kvm *kvm,
+			  const struct kvm_ram_config *ram_cfg,
+			  struct kvm_ram_region *ram,
+			  struct kvm_arch_ram_region *ram_alloc)
 {
 	int err;
-	u64 phys_start;
 	unsigned long alignment;
 	/* Convenience aliases */
 	const char *hugetlbfs_path = kvm->cfg.hugetlbfs_path;
-	struct kvm_ram_region *ram = &kvm->ram[0];
 
 	/*
 	 * Allocate guest memory. We must align our buffer to 64K to
@@ -45,37 +48,42 @@ static void kvm__init_ram(struct kvm *kvm)
 		alignment = SZ_32M;
 	else
 		alignment = SZ_2M;
-	ram->size = kvm->cfg.ram[0].size;
-	kvm->arch.ram_alloc_size = ram->size + alignment;
-	kvm->arch.ram_alloc_start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
-						kvm->arch.ram_alloc_size);
+	ram->size = ram_cfg->size;
+	ram_alloc->size = ram->size + alignment;
+	ram_alloc->start = mmap_anon_or_hugetlbfs(kvm, hugetlbfs_path,
+						  ram_alloc->size);
 
-	if (kvm->arch.ram_alloc_start == MAP_FAILED)
+	if (ram_alloc->start == MAP_FAILED)
 		die("Failed to map %lld bytes for guest memory (%d)",
-		    kvm->arch.ram_alloc_size, errno);
-
-	ram->start = (void *)ALIGN((unsigned long)kvm->arch.ram_alloc_start,
-				   SZ_2M);
+		    ram_alloc->size, errno);
 
-	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
-		MADV_MERGEABLE);
+	ram->start = (void *)ALIGN((unsigned long)ram_alloc->start, SZ_2M);
 
-	madvise(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size,
-		MADV_HUGEPAGE);
+	madvise(ram_alloc->start, ram_alloc->size, MADV_MERGEABLE);
 
-	phys_start	= ARM_MEMORY_AREA;
+	madvise(ram_alloc->start, ram_alloc->size, MADV_HUGEPAGE);
 
-	err = kvm__register_mem(kvm, phys_start, ram->size, ram->start);
+	err = kvm__register_mem(kvm, ram_cfg->base, ram_cfg->size,
+				ram->start);
 	if (err)
 		die("Failed to register %lld bytes of memory at physical "
-		    "address 0x%llx [err %d]", ram->size, phys_start, err);
-
-	kvm->arch.memory_guest_start = phys_start;
+		    "address 0x%llx [err %d]",
+		    ram_cfg->size,
+		    ram_cfg->base,
+		    err);
+
+	pr_info("Registered memory bank 0x%llx-0x%llx\n",
+		ram_cfg->base,
+		ram_cfg->base + ram_cfg->size);
 }
 
 void kvm__arch_delete_ram(struct kvm *kvm)
 {
-	munmap(kvm->arch.ram_alloc_start, kvm->arch.ram_alloc_size);
+	unsigned int i;
+
+	for (i = 0; i < kvm->nr_ram; i++)
+		munmap(kvm->arch.ram_alloc[i].start,
+		       kvm->arch.ram_alloc[i].size);
 }
 
 void kvm__arch_read_term(struct kvm *kvm)
@@ -88,27 +96,86 @@ void kvm__arch_set_cmdline(char *cmdline, bool video)
 {
 }
 
+/* Only sort the bank by base address */
+static int cmp_bank(const void *p1, const void *p2)
+{
+	const struct kvm_ram_config *bank1 = p1;
+	const struct kvm_ram_config *bank2 = p2;
+
+	if (bank1->base < bank2->base)
+		return -1;
+	else if (bank1->base == bank2->base)
+		return 0;
+	else
+		return 1;
+}
+
 static void kvm__arch_sanitize_cfg(struct kvm_config *cfg)
 {
+	unsigned int i;
 	/* Convenience aliases */
 	struct kvm_ram_config *bank0 = &cfg->ram[0];
 
-	if (bank0->size > ARM_MAX_MEMORY(cfg)) {
-		bank0->size = ARM_MAX_MEMORY(cfg);
-		pr_warning("sanitize: Capping memory to %lluMB",
-			   bank0->size >> 20);
+	/*
+	 * The user may not have set an address for the first bank. (To keep the
+	 * command line backward compatible).
+	 */
+	if (bank0->base == INVALID_RAM_ADDR) {
+		/*
+		 * Impose the user to set address for the first bank when
+		 * multiple banks are specified.
+		 */
+		if (cfg->nr_ram > 1)
+			die("sanitize: Base address should be specified for all the banks\n");
+		bank0->base = ARM_MEMORY_AREA;
+		/*
+		 * Keep compatibility with old KVM command line behavior where
+		 * the memory is capped.
+		 */
+		if ((bank0->base + bank0->size) > ARM_MAX_PHYS_ADDR(cfg)) {
+			bank0->size = ARM_MAX_PHYS_ADDR(cfg) - bank0->base;
+			pr_warning("sanitize: Capping memory to %lluMB",
+				   bank0->size >> 20);
+		}
 	}
+
+	/* Sort banks by address to make easier later on. */
+	qsort(cfg->ram, cfg->nr_ram, sizeof(*cfg->ram), cmp_bank);
+
+	/* Check banks are not overlapping */
+	for (i = 1; i < cfg->nr_ram; i++) {
+		const struct kvm_ram_config *bank1 = &cfg->ram[i - 1];
+		const struct kvm_ram_config *bank2 = &cfg->ram[i];
+		unsigned long long end1 = bank1->base + bank1->size - 1;
+		unsigned long long end2 = bank2->base + bank2->size - 1;
+
+		if (!((end1 < bank2->base) || (bank1->base > end2)))
+			die("Memory bank 0x%llx-0x%llx overlapping with 0x%llx-0x%llx\n",
+			    bank1->base, end1, bank2->base, end2);
+	}
+
+	/* Check the memory is below the IPA size supported */
+	i = cfg->nr_ram - 1;
+	if ((cfg->ram[i].base + cfg->ram[i].size) > ARM_MAX_PHYS_ADDR(cfg))
+		die("Memory bank outside of the maximum IPA\n");
 }
 
 void kvm__arch_init(struct kvm *kvm)
 {
+	unsigned int i;
+	/* Convenience aliases */
+	struct kvm_config *cfg = &kvm->cfg;
+
 	kvm__arch_sanitize_cfg(&kvm->cfg);
 
 	/* Create the virtual GIC. */
 	if (gic__create(kvm, kvm->cfg.arch.irqchip))
 		die("Failed to create virtual GIC");
 
-	kvm__init_ram(kvm);
+	for (i = 0; i < cfg->nr_ram; i++)
+		kvm__init_ram(kvm, &cfg->ram[i], &kvm->ram[i],
+			      &kvm->arch.ram_alloc[i]);
+	kvm->nr_ram = cfg->nr_ram;
 }
 
 #define FDT_ALIGN	SZ_2M
@@ -125,6 +192,8 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	/*
 	 * Linux requires the initrd and dtb to be mapped inside lowmem,
 	 * so we can't just place them at the top of memory.
+	 *
+	 * Let's place it in the first memory bank.
 	 */
 	limit = ram0->start + min(ram0->size, (u64)SZ_256M) - 1;
 
@@ -133,7 +202,7 @@ bool kvm__arch_load_kernel_image(struct kvm *kvm, int fd_kernel, int fd_initrd,
 	file_size = read_file(fd_kernel, pos, limit - pos);
 	if (file_size < 0) {
 		if (errno == ENOMEM)
-			die("kernel image too big to contain in guest memory.");
+			die("kernel image too big to contain in the first memory bank of the guest.");
 
 		die_perror("kernel read");
 	}
-- 
2.11.0

[RFC 09/13] virtio: Handle aborts using invalid PFN

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

When the host fails to complete the shake hand due to various
reasons. e.g, for PCI and MMIO, if 0 is written as the PFN,
it implies the host has given up and simply don't take any
action.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 virtio/mmio.c | 14 ++++++++------
 virtio/pci.c  | 10 ++++++----
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/virtio/mmio.c b/virtio/mmio.c
index 8c0f1cc..b118b8a 100644
--- a/virtio/mmio.c
+++ b/virtio/mmio.c
@@ -188,12 +188,14 @@ static void virtio_mmio_config_out(struct kvm_cpu *vcpu,
 		break;
 	case VIRTIO_MMIO_QUEUE_PFN:
 		val = ioport__read32(data);
-		virtio_mmio_init_ioeventfd(vmmio->kvm, vdev, vmmio->hdr.queue_sel);
-		vdev->ops->init_vq(vmmio->kvm, vmmio->dev,
-				   vmmio->hdr.queue_sel,
-				   vmmio->hdr.guest_page_size,
-				   vmmio->hdr.queue_align,
-				   val);
+		if (val) {
+			virtio_mmio_init_ioeventfd(vmmio->kvm, vdev, vmmio->hdr.queue_sel);
+			vdev->ops->init_vq(vmmio->kvm, vmmio->dev,
+					   vmmio->hdr.queue_sel,
+					   vmmio->hdr.guest_page_size,
+					   vmmio->hdr.queue_align,
+					   val);
+		}
 		break;
 	case VIRTIO_MMIO_QUEUE_NOTIFY:
 		val = ioport__read32(data);
diff --git a/virtio/pci.c b/virtio/pci.c
index 5a5fc6e..403063f 100644
--- a/virtio/pci.c
+++ b/virtio/pci.c
@@ -269,10 +269,12 @@ static bool virtio_pci__io_out(struct ioport *ioport, struct kvm_cpu *vcpu, u16
 		break;
 	case VIRTIO_PCI_QUEUE_PFN:
 		val = ioport__read32(data);
-		virtio_pci__init_ioeventfd(kvm, vdev, vpci->queue_selector);
-		vdev->ops->init_vq(kvm, vpci->dev, vpci->queue_selector,
-				   1 << VIRTIO_PCI_QUEUE_ADDR_SHIFT,
-				   VIRTIO_PCI_VRING_ALIGN, val);
+		if (val) {
+			virtio_pci__init_ioeventfd(kvm, vdev, vpci->queue_selector);
+			vdev->ops->init_vq(kvm, vpci->dev, vpci->queue_selector,
+					   1 << VIRTIO_PCI_QUEUE_ADDR_SHIFT,
+					   VIRTIO_PCI_VRING_ALIGN, val);
+		}
 		break;
 	case VIRTIO_PCI_QUEUE_SEL:
 		vpci->queue_selector = ioport__read16(data);
-- 
2.11.0

[RFC 10/13] kvmtool: Allow backends to run checks on the KVM device fd

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Allow architectures to perform initialisation based on the
KVM device fd ioctls, even before the VM is created.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 include/kvm/kvm.h | 4 ++++
 kvm.c             | 2 ++
 2 files changed, 6 insertions(+)

diff --git a/include/kvm/kvm.h b/include/kvm/kvm.h
index 4e21446..bc916ea 100644
--- a/include/kvm/kvm.h
+++ b/include/kvm/kvm.h
@@ -107,6 +107,10 @@ int kvm__get_sock_by_instance(const char *name);
 int kvm__enumerate_instances(int (*callback)(const char *name, int pid));
 void kvm__remove_socket(const char *name);
 
+#ifndef kvm__arch_init_hyp
+static inline void kvm__arch_init_hyp(struct kvm *kvm) {}
+#endif
+
 void kvm__arch_set_cmdline(char *cmdline, bool video);
 void kvm__arch_init(struct kvm *kvm);
 void kvm__arch_delete_ram(struct kvm *kvm);
diff --git a/kvm.c b/kvm.c
index 9c2cb2f..f40975b 100644
--- a/kvm.c
+++ b/kvm.c
@@ -308,6 +308,8 @@ int kvm__init(struct kvm *kvm)
 		goto err_sys_fd;
 	}
 
+	kvm__arch_init_hyp(kvm);
+
 	kvm->vm_fd = ioctl(kvm->sys_fd, KVM_CREATE_VM, KVM_VM_TYPE);
 	if (kvm->vm_fd < 0) {
 		pr_err("KVM_CREATE_VM ioctl");
-- 
2.11.0

[RFC 11/13] kvmtool: arm64: Add support for guest physical address size

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Add an option to specify the physical address size used by this
VM.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 arm/aarch64/include/kvm/kvm-config-arch.h | 5 ++++-
 arm/include/arm-common/kvm-config-arch.h  | 1 +
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arm/aarch64/include/kvm/kvm-config-arch.h b/arm/aarch64/include/kvm/kvm-config-arch.h
index 04be43d..c4bb207 100644
--- a/arm/aarch64/include/kvm/kvm-config-arch.h
+++ b/arm/aarch64/include/kvm/kvm-config-arch.h
@@ -8,7 +8,10 @@
 			"Create PMUv3 device"),				\
 	OPT_U64('\0', "kaslr-seed", &(cfg)->kaslr_seed,			\
 			"Specify random seed for Kernel Address Space "	\
-			"Layout Randomization (KASLR)"),
+			"Layout Randomization (KASLR)"),		\
+	OPT_UINTEGER('\0', "phys-shift", &(cfg)->phys_shift,		\
+			"Specify maximum physical address size (not "	\
+			"the amount of memory)"),
 
 #include "arm-common/kvm-config-arch.h"
 
diff --git a/arm/include/arm-common/kvm-config-arch.h b/arm/include/arm-common/kvm-config-arch.h
index 6a196f1..d841b0b 100644
--- a/arm/include/arm-common/kvm-config-arch.h
+++ b/arm/include/arm-common/kvm-config-arch.h
@@ -11,6 +11,7 @@ struct kvm_config_arch {
 	bool		has_pmuv3;
 	u64		kaslr_seed;
 	enum irqchip_type irqchip;
+	unsigned int	phys_shift;
 };
 
 int irqchip_parser(const struct option *opt, const char *arg, int unset);
-- 
2.11.0

[RFC 12/13] kvmtool: arm64: Switch memory layout

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

If the guest wants to use a larger physical address space place
the RAM at upper half of the address space. Otherwise, it uses the
default layout.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 arm/aarch32/include/kvm/kvm-arch.h |  1 +
 arm/aarch64/include/kvm/kvm-arch.h | 13 ++++++++++---
 arm/include/arm-common/kvm-arch.h  |  8 ++++++--
 arm/kvm.c                          |  2 +-
 4 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/arm/aarch32/include/kvm/kvm-arch.h b/arm/aarch32/include/kvm/kvm-arch.h
index 2ee0cb1..f0f0943 100644
--- a/arm/aarch32/include/kvm/kvm-arch.h
+++ b/arm/aarch32/include/kvm/kvm-arch.h
@@ -4,6 +4,7 @@
 #define ARM_KERN_OFFSET(...)	0x8000
 
 #define ARM_MAX_PHYS_SHIFT(...)	32
+#define ARM_MEMORY_AREA(...)	ARM32_MEMORY_AREA
 
 #include "arm-common/kvm-arch.h"
 
diff --git a/arm/aarch64/include/kvm/kvm-arch.h b/arm/aarch64/include/kvm/kvm-arch.h
index 53ac20f..9c9009b 100644
--- a/arm/aarch64/include/kvm/kvm-arch.h
+++ b/arm/aarch64/include/kvm/kvm-arch.h
@@ -1,12 +1,19 @@
 #ifndef KVM__KVM_ARCH_H
 #define KVM__KVM_ARCH_H
 
+#include "arm-common/kvm-arch.h"
+
+#define ARM64_MEMORY_AREA(phys_shift)	(1UL << (phys_shift - 1))
+
+#define ARM_MEMORY_AREA(cfg)	((cfg)->arch.aarch32_guest ?			\
+				 ARM32_MEMORY_AREA		:		\
+				 ARM64_MEMORY_AREA((cfg)->arch.phys_shift))
+
 #define ARM_KERN_OFFSET(kvm)	((kvm)->cfg.arch.aarch32_guest	?	\
 				0x8000				:	\
 				0x80000)
 
-#define ARM_MAX_PHYS_SHIFT(cfg)	((cfg)->arch.aarch32_guest ? 32 : 40)
-
-#include "arm-common/kvm-arch.h"
+#define ARM_MAX_PHYS_SHIFT(cfg)	((cfg)->arch.aarch32_guest ? 32 :	\
+				 (cfg)->arch.phys_shift)
 
 #endif /* KVM__KVM_ARCH_H */
diff --git a/arm/include/arm-common/kvm-arch.h b/arm/include/arm-common/kvm-arch.h
index 884dda8..4be9589 100644
--- a/arm/include/arm-common/kvm-arch.h
+++ b/arm/include/arm-common/kvm-arch.h
@@ -10,7 +10,11 @@
 #define ARM_IOPORT_AREA		_AC(0x0000000000000000, UL)
 #define ARM_MMIO_AREA		_AC(0x0000000000010000, UL)
 #define ARM_AXI_AREA		_AC(0x0000000040000000, UL)
-#define ARM_MEMORY_AREA		_AC(0x0000000080000000, UL)
+
+#define ARM32_MEMORY_AREA	_AC(0x0000000080000000, UL)
+#define ARM32_MAX_MEMORY	((1ULL << 32) - ARM32_MEMORY_AREA)
+
+#define ARM_IOMEM_AREA_END	ARM32_MEMORY_AREA
 
 #define ARM_GIC_DIST_BASE	(ARM_AXI_AREA - ARM_GIC_DIST_SIZE)
 #define ARM_GIC_CPUI_BASE	(ARM_GIC_DIST_BASE - ARM_GIC_CPUI_SIZE)
@@ -21,7 +25,7 @@
 #define ARM_IOPORT_SIZE		(ARM_MMIO_AREA - ARM_IOPORT_AREA)
 #define ARM_VIRTIO_MMIO_SIZE	(ARM_AXI_AREA - (ARM_MMIO_AREA + ARM_GIC_SIZE))
 #define ARM_PCI_CFG_SIZE	(1ULL << 24)
-#define ARM_PCI_MMIO_SIZE	(ARM_MEMORY_AREA - \
+#define ARM_PCI_MMIO_SIZE	(ARM_IOMEM_AREA_END - \
 				(ARM_AXI_AREA + ARM_PCI_CFG_SIZE))
 
 #define KVM_IOPORT_AREA		ARM_IOPORT_AREA
diff --git a/arm/kvm.c b/arm/kvm.c
index 8a7c611..5a2bd28 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -127,7 +127,7 @@ static void kvm__arch_sanitize_cfg(struct kvm_config *cfg)
 		 */
 		if (cfg->nr_ram > 1)
 			die("sanitize: Base address should be specified for all the banks\n");
-		bank0->base = ARM_MEMORY_AREA;
+		bank0->base = ARM_MEMORY_AREA(cfg);
 		/*
 		 * Keep compatibility with old KVM command line behavior where
 		 * the memory is capped.
-- 
2.11.0

[RFC 13/13] kvmtool: arm/arm64: Add support for creating VM with PA size

[Julien Grall]

From: Suzuki K Poulose <suzuki.poulose@arm.com>

Specify the physical size for the VM encoded in the vm type.

Signed-off-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Julien Grall <julien.grall@arm.com>
[julien: Don't specify the type of older kernels | fix max_ipa type]
---
 arm/include/arm-common/kvm-arch.h |  6 +++++-
 arm/kvm.c                         | 27 +++++++++++++++++++++++++++
 2 files changed, 32 insertions(+), 1 deletion(-)

diff --git a/arm/include/arm-common/kvm-arch.h b/arm/include/arm-common/kvm-arch.h
index 4be9589..81be262 100644
--- a/arm/include/arm-common/kvm-arch.h
+++ b/arm/include/arm-common/kvm-arch.h
@@ -46,7 +46,11 @@
 
 #define KVM_IRQ_OFFSET		GIC_SPI_IRQ_BASE
 
-#define KVM_VM_TYPE		0
+extern unsigned long		kvm_arm_type;
+extern void kvm__arch_init_hyp(struct kvm *kvm);
+
+#define KVM_VM_TYPE		kvm_arm_type
+#define kvm__arch_init_hyp	kvm__arch_init_hyp
 
 #define VIRTIO_DEFAULT_TRANS(kvm)	\
 	((kvm)->cfg.arch.virtio_trans_pci ? VIRTIO_PCI : VIRTIO_MMIO)
diff --git a/arm/kvm.c b/arm/kvm.c
index 5a2bd28..97c1333 100644
--- a/arm/kvm.c
+++ b/arm/kvm.c
@@ -13,6 +13,8 @@
 
 #include <stdlib.h>
 
+unsigned long kvm_arm_type;
+
 struct kvm_ext kvm_req_ext[] = {
 	{ DEFINE_KVM_EXT(KVM_CAP_IRQCHIP) },
 	{ DEFINE_KVM_EXT(KVM_CAP_ONE_REG) },
@@ -20,6 +22,31 @@ struct kvm_ext kvm_req_ext[] = {
 	{ 0, 0 },
 };
 
+#ifndef KVM_ARM_GET_MAX_VM_PHYS_SHIFT
+#define KVM_ARM_GET_MAX_VM_PHYS_SHIFT		_IO(KVMIO, 0x0a)
+#endif
+
+void kvm__arch_init_hyp(struct kvm *kvm)
+{
+	int max_ipa;
+
+	max_ipa = ioctl(kvm->sys_fd, KVM_ARM_GET_MAX_VM_PHYS_SHIFT);
+	if (max_ipa < 0)
+		max_ipa = 40;
+	if (!kvm->cfg.arch.phys_shift)
+		kvm->cfg.arch.phys_shift = 40;
+	if (kvm->cfg.arch.phys_shift > max_ipa)
+		die("Requested PA size (%u) is not supported by the host (%ubits)\n",
+		    kvm->cfg.arch.phys_shift, max_ipa);
+
+	/*
+	 * Kernel not implementing the ioctl KVM_ARM_GET_MAX_VM_PHYS_SHIFT
+	 * will request to have kvm_arm_type zeroed and assume 40 bits IPA.
+	 */
+	if (kvm->cfg.arch.phys_shift != 40)
+		kvm_arm_type = kvm->cfg.arch.phys_shift;
+}
+
 bool kvm__arch_cpu_supports_vm(void)
 {
 	/* The KVM capability check is enough. */
-- 
2.11.0

[RFC 00/13] arm: Allow the user specifying where the RAM is place in the memory

[Julien Grall]

I forgot to mention in the title that the series is for kvmtools. Sorry 
for the confusion.

Cheers,

On 10/05/18 15:04, Julien Grall wrote:
> Hi all,
> 
> At the moment, a user is only able to specify the amount of RAM used by the
> guest. Where the RAM will live is left to the software and hardcoded.
> 
> It could be useful for testing purpose to move the RAM in different place.
> This series adds the possibility for the user to specify multiple RAM region.
> 
> The option -m/--mem is extended to specify the address using the following
> format: <size>@<addr>. The option needs to be repeated as many times as the
> number of RAM region in the guest layout.
> 
> For instance, if you want 512MB at 3GB and 512MB 4GB it would look like:
>      -m 512 at 0xc0000000 -m 512 at 0x100000000
> 
> Note that the memory layout is not yet fully configurable by the user, so the
> MMIO region is still living below 2GB. This means RAM cannot live in the
> region 0-2GB. This could be changed in the future.
> 
> This new version also integrates work from Suzuki to allow the user specifying
> the IPA size for the guest. This was previously sent separately on kvmarm [1].
> 
> Cheers,
> 
> [1] https://lkml.org/lkml/2018/3/27/437
> 
> Julien Grall (7):
>    kvm__arch_init: Don't pass hugetlbfs_path and ram_size in parameter
>    virtio/scsi: Allow to use multiple banks
>    Fold kvm__init_ram call in kvm__arch_init
>    kvm__arch_sanitize_cfg
>    arm: Move anything related to RAM initialization in kvm__init_ram
>    Allow the user to specify where the RAM is placed in the memory
>    arm: Add support for multi memory regions
> 
> Suzuki K Poulose (6):
>    arm: Allow use of hugepage with 16K pagesize host
>    virtio: Handle aborts using invalid PFN
>    kvmtool: Allow backends to run checks on the KVM device fd
>    kvmtool: arm64: Add support for guest physical address size
>    kvmtool: arm64: Switch memory layout
>    kvmtool: arm/arm64: Add support for creating VM with PA size
> 
>   arm/aarch32/include/kvm/kvm-arch.h        |   3 +-
>   arm/aarch64/include/kvm/kvm-arch.h        |  15 ++-
>   arm/aarch64/include/kvm/kvm-config-arch.h |   5 +-
>   arm/fdt.c                                 |  17 ++-
>   arm/include/arm-common/kvm-arch.h         |  32 ++++--
>   arm/include/arm-common/kvm-config-arch.h  |   1 +
>   arm/kvm.c                                 | 183 +++++++++++++++++++++++++-----
>   builtin-run.c                             |  87 ++++++++++++--
>   include/kvm/kvm-config.h                  |  16 ++-
>   include/kvm/kvm.h                         |  28 ++++-
>   kvm.c                                     |  10 +-
>   mips/kvm.c                                |  47 +++++---
>   powerpc/kvm.c                             |  33 +++---
>   virtio/mmio.c                             |  14 ++-
>   virtio/pci.c                              |  10 +-
>   virtio/scsi.c                             |  21 ++--
>   x86/bios.c                                |   8 +-
>   x86/kvm.c                                 |  54 +++++----
>   18 files changed, 443 insertions(+), 141 deletions(-)
> 

-- 
Julien Grall