[PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

With the availability of a fruity range of arm64 systems, it becomes
obvious that QEMU doesn't deal very well with limited IPA ranges when
used as a front-end for KVM.

This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the
computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the
highmem option

This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

Marc Zyngier (3):
  hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
    scratch VM
  hw/arm/virt: Honor highmem setting when computing highest_gpa
  docs/system/arm/virt: Fix documentation for the 'highmem' option

 docs/system/arm/virt.rst |  6 +++---
 hw/arm/virt.c            | 10 +++++++---
 target/arm/kvm.c         |  7 ++++++-
 3 files changed, 16 insertions(+), 7 deletions(-)

-- 
2.30.2

[PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

With the availability of a fruity range of arm64 systems, it becomes
obvious that QEMU doesn't deal very well with limited IPA ranges when
used as a front-end for KVM.

This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the
computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the
highmem option

This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

Marc Zyngier (3):
  hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
    scratch VM
  hw/arm/virt: Honor highmem setting when computing highest_gpa
  docs/system/arm/virt: Fix documentation for the 'highmem' option

 docs/system/arm/virt.rst |  6 +++---
 hw/arm/virt.c            | 10 +++++++---
 target/arm/kvm.c         |  7 ++++++-
 3 files changed, 16 insertions(+), 7 deletions(-)

-- 
2.30.2

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

With the availability of a fruity range of arm64 systems, it becomes
obvious that QEMU doesn't deal very well with limited IPA ranges when
used as a front-end for KVM.

This short series aims at making usable on such systems:
- the first patch makes the creation of a scratch VM IPA-limit aware
- the second one actually removes the highmem devices from the
computed IPA range when highmem=off
- the last one addresses an imprecision in the documentation for the
highmem option

This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

Marc Zyngier (3):
  hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
    scratch VM
  hw/arm/virt: Honor highmem setting when computing highest_gpa
  docs/system/arm/virt: Fix documentation for the 'highmem' option

 docs/system/arm/virt.rst |  6 +++---
 hw/arm/virt.c            | 10 +++++++---
 target/arm/kvm.c         |  7 ++++++-
 3 files changed, 16 insertions(+), 7 deletions(-)

-- 
2.30.2

[PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Marc Zyngier]

Although we probe for the IPA limits imposed by KVM (and the hardware)
when computing the memory map, we still use the old style '0' when
creating a scratch VM in kvm_arm_create_scratch_host_vcpu().

On systems that are severely IPA challenged (such as the Apple M1),
this results in a failure as KVM cannot use the default 40bit that
'0' represents.

Instead, probe for the extension and use the reported IPA limit
if available.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 target/arm/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index d8381ba224..cc3371a99b 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
                                       struct kvm_vcpu_init *init)
 {
     int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
+    int max_vm_pa_size;
 
     kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
     if (kvmfd < 0) {
         goto err;
     }
-    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
+    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
+    if (max_vm_pa_size < 0) {
+        max_vm_pa_size = 0;
+    }
+    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
     if (vmfd < 0) {
         goto err;
     }
-- 
2.30.2

[PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Marc Zyngier]

Although we probe for the IPA limits imposed by KVM (and the hardware)
when computing the memory map, we still use the old style '0' when
creating a scratch VM in kvm_arm_create_scratch_host_vcpu().

On systems that are severely IPA challenged (such as the Apple M1),
this results in a failure as KVM cannot use the default 40bit that
'0' represents.

Instead, probe for the extension and use the reported IPA limit
if available.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 target/arm/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index d8381ba224..cc3371a99b 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
                                       struct kvm_vcpu_init *init)
 {
     int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
+    int max_vm_pa_size;
 
     kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
     if (kvmfd < 0) {
         goto err;
     }
-    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
+    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
+    if (max_vm_pa_size < 0) {
+        max_vm_pa_size = 0;
+    }
+    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
     if (vmfd < 0) {
         goto err;
     }
-- 
2.30.2

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Marc Zyngier]

Although we probe for the IPA limits imposed by KVM (and the hardware)
when computing the memory map, we still use the old style '0' when
creating a scratch VM in kvm_arm_create_scratch_host_vcpu().

On systems that are severely IPA challenged (such as the Apple M1),
this results in a failure as KVM cannot use the default 40bit that
'0' represents.

Instead, probe for the extension and use the reported IPA limit
if available.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 target/arm/kvm.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index d8381ba224..cc3371a99b 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
                                       struct kvm_vcpu_init *init)
 {
     int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
+    int max_vm_pa_size;
 
     kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
     if (kvmfd < 0) {
         goto err;
     }
-    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
+    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
+    if (max_vm_pa_size < 0) {
+        max_vm_pa_size = 0;
+    }
+    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
     if (vmfd < 0) {
         goto err;
     }
-- 
2.30.2

[PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Marc Zyngier]

Even when the VM is configured with highmem=off, the highest_gpa
field includes devices that are above the 4GiB limit, which is
what highmem=off is supposed to enforce. This leads to failures
in virt_kvm_type() on systems that have a crippled IPA range,
as the reported IPA space is larger than what it should be.

Instead, honor the user-specified limit to only use the devices
at the lowest end of the spectrum.

Note that this doesn't affect memory, which is still allowed to
go beyond 4GiB with highmem=on configurations.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 hw/arm/virt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 81eda46b0b..bc189e30b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 static void virt_set_memmap(VirtMachineState *vms)
 {
     MachineState *ms = MACHINE(vms);
-    hwaddr base, device_memory_base, device_memory_size;
+    hwaddr base, device_memory_base, device_memory_size, ceiling;
     int i;
 
     vms->memmap = extended_memmap;
@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
     device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
 
     /* Base address of the high IO region */
-    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
+    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
     if (base < device_memory_base) {
         error_report("maxmem/slots too huge");
         exit(EXIT_FAILURE);
@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
-    vms->highest_gpa = base - 1;
+    if (vms->highmem) {
+	    /* If we have highmem, move the IPA limit to the top */
+	    ceiling = base;
+    }
+    vms->highest_gpa = ceiling - 1;
     if (device_memory_size > 0) {
         ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
         ms->device_memory->base = device_memory_base;
-- 
2.30.2

[PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Marc Zyngier]

Even when the VM is configured with highmem=off, the highest_gpa
field includes devices that are above the 4GiB limit, which is
what highmem=off is supposed to enforce. This leads to failures
in virt_kvm_type() on systems that have a crippled IPA range,
as the reported IPA space is larger than what it should be.

Instead, honor the user-specified limit to only use the devices
at the lowest end of the spectrum.

Note that this doesn't affect memory, which is still allowed to
go beyond 4GiB with highmem=on configurations.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 hw/arm/virt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 81eda46b0b..bc189e30b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 static void virt_set_memmap(VirtMachineState *vms)
 {
     MachineState *ms = MACHINE(vms);
-    hwaddr base, device_memory_base, device_memory_size;
+    hwaddr base, device_memory_base, device_memory_size, ceiling;
     int i;
 
     vms->memmap = extended_memmap;
@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
     device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
 
     /* Base address of the high IO region */
-    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
+    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
     if (base < device_memory_base) {
         error_report("maxmem/slots too huge");
         exit(EXIT_FAILURE);
@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
-    vms->highest_gpa = base - 1;
+    if (vms->highmem) {
+	    /* If we have highmem, move the IPA limit to the top */
+	    ceiling = base;
+    }
+    vms->highest_gpa = ceiling - 1;
     if (device_memory_size > 0) {
         ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
         ms->device_memory->base = device_memory_base;
-- 
2.30.2

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Marc Zyngier]

Even when the VM is configured with highmem=off, the highest_gpa
field includes devices that are above the 4GiB limit, which is
what highmem=off is supposed to enforce. This leads to failures
in virt_kvm_type() on systems that have a crippled IPA range,
as the reported IPA space is larger than what it should be.

Instead, honor the user-specified limit to only use the devices
at the lowest end of the spectrum.

Note that this doesn't affect memory, which is still allowed to
go beyond 4GiB with highmem=on configurations.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 hw/arm/virt.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 81eda46b0b..bc189e30b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 static void virt_set_memmap(VirtMachineState *vms)
 {
     MachineState *ms = MACHINE(vms);
-    hwaddr base, device_memory_base, device_memory_size;
+    hwaddr base, device_memory_base, device_memory_size, ceiling;
     int i;
 
     vms->memmap = extended_memmap;
@@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
     device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
 
     /* Base address of the high IO region */
-    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
+    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
     if (base < device_memory_base) {
         error_report("maxmem/slots too huge");
         exit(EXIT_FAILURE);
@@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
         vms->memmap[i].size = size;
         base += size;
     }
-    vms->highest_gpa = base - 1;
+    if (vms->highmem) {
+	    /* If we have highmem, move the IPA limit to the top */
+	    ceiling = base;
+    }
+    vms->highest_gpa = ceiling - 1;
     if (device_memory_size > 0) {
         ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
         ms->device_memory->base = device_memory_base;
-- 
2.30.2

[PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

The documentation for the 'highmem' option indicates that it controls
the placement of both devices and RAM. The actual behaviour of QEMU
seems to be that RAM is allowed to go beyond the 4GiB limit, and
that only devices are constraint by this option.

Align the documentation with the actual behaviour.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 docs/system/arm/virt.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index 59acf0eeaf..e206e7565d 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -86,9 +86,9 @@ mte
   Arm Memory Tagging Extensions. The default is ``off``.
 
 highmem
-  Set ``on``/``off`` to enable/disable placing devices and RAM in physical
-  address space above 32 bits. The default is ``on`` for machine types
-  later than ``virt-2.12``.
+  Set ``on``/``off`` to enable/disable placing devices in physical address
+  space above 32 bits. RAM in excess of 3GiB will always be placed above
+  32 bits. The default is ``on`` for machine types later than ``virt-2.12``.
 
 gic-version
   Specify the version of the Generic Interrupt Controller (GIC) to provide.
-- 
2.30.2

[PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

The documentation for the 'highmem' option indicates that it controls
the placement of both devices and RAM. The actual behaviour of QEMU
seems to be that RAM is allowed to go beyond the 4GiB limit, and
that only devices are constraint by this option.

Align the documentation with the actual behaviour.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 docs/system/arm/virt.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index 59acf0eeaf..e206e7565d 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -86,9 +86,9 @@ mte
   Arm Memory Tagging Extensions. The default is ``off``.
 
 highmem
-  Set ``on``/``off`` to enable/disable placing devices and RAM in physical
-  address space above 32 bits. The default is ``on`` for machine types
-  later than ``virt-2.12``.
+  Set ``on``/``off`` to enable/disable placing devices in physical address
+  space above 32 bits. RAM in excess of 3GiB will always be placed above
+  32 bits. The default is ``on`` for machine types later than ``virt-2.12``.
 
 gic-version
   Specify the version of the Generic Interrupt Controller (GIC) to provide.
-- 
2.30.2

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

[PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

The documentation for the 'highmem' option indicates that it controls
the placement of both devices and RAM. The actual behaviour of QEMU
seems to be that RAM is allowed to go beyond the 4GiB limit, and
that only devices are constraint by this option.

Align the documentation with the actual behaviour.

Cc: Andrew Jones <drjones@redhat.com>
Cc: Eric Auger <eric.auger@redhat.com>
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 docs/system/arm/virt.rst | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index 59acf0eeaf..e206e7565d 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -86,9 +86,9 @@ mte
   Arm Memory Tagging Extensions. The default is ``off``.
 
 highmem
-  Set ``on``/``off`` to enable/disable placing devices and RAM in physical
-  address space above 32 bits. The default is ``on`` for machine types
-  later than ``virt-2.12``.
+  Set ``on``/``off`` to enable/disable placing devices in physical address
+  space above 32 bits. RAM in excess of 3GiB will always be placed above
+  32 bits. The default is ``on`` for machine types later than ``virt-2.12``.
 
 gic-version
   Specify the version of the Generic Interrupt Controller (GIC) to provide.
-- 
2.30.2

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

On 2021-08-22 15:44, Marc Zyngier wrote:
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
> 
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
> 
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

I realise I haven't been very clear in my description of the above.
With this series, using 'highmem=off' results in a usable VM, while
sticking to the default 'highmem=on' still generates an error.

         M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

On 2021-08-22 15:44, Marc Zyngier wrote:
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
> 
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
> 
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

I realise I haven't been very clear in my description of the above.
With this series, using 'highmem=off' results in a usable VM, while
sticking to the default 'highmem=on' still generates an error.

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Marc Zyngier]

On 2021-08-22 15:44, Marc Zyngier wrote:
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
> 
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
> 
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.

I realise I haven't been very clear in my description of the above.
With this series, using 'highmem=off' results in a usable VM, while
sticking to the default 'highmem=on' still generates an error.

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Andrew Jones]

On Sun, Aug 22, 2021 at 03:44:39PM +0100, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
> 
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
> 
> Instead, probe for the extension and use the reported IPA limit
> if available.
> 
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }
> -- 
> 2.30.2
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Andrew Jones]

On Sun, Aug 22, 2021 at 03:44:39PM +0100, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
> 
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
> 
> Instead, probe for the extension and use the reported IPA limit
> if available.
> 
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }
> -- 
> 2.30.2
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Andrew Jones]

On Sun, Aug 22, 2021 at 03:44:39PM +0100, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
> 
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
> 
> Instead, probe for the extension and use the reported IPA limit
> if available.
> 
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }
> -- 
> 2.30.2
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> The documentation for the 'highmem' option indicates that it controls
> the placement of both devices and RAM. The actual behaviour of QEMU
> seems to be that RAM is allowed to go beyond the 4GiB limit, and
> that only devices are constraint by this option.
>
> Align the documentation with the actual behaviour.

I think it would be better to align the behaviour with the documentation.

The intent of 'highmem' is to allow a configuration for use with guests
that can't address more than 32 bits (originally, 32-bit guests without
LPAE support compiled in). It seems like a bug that we allow the user
to specify more RAM than will fit into that 32-bit range. We should
instead make QEMU exit with an error if the user tries to specify
both highmem=off and a memory size that's too big to fit.

thanks
-- PMM

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> The documentation for the 'highmem' option indicates that it controls
> the placement of both devices and RAM. The actual behaviour of QEMU
> seems to be that RAM is allowed to go beyond the 4GiB limit, and
> that only devices are constraint by this option.
>
> Align the documentation with the actual behaviour.

I think it would be better to align the behaviour with the documentation.

The intent of 'highmem' is to allow a configuration for use with guests
that can't address more than 32 bits (originally, 32-bit guests without
LPAE support compiled in). It seems like a bug that we allow the user
to specify more RAM than will fit into that 32-bit range. We should
instead make QEMU exit with an error if the user tries to specify
both highmem=off and a memory size that's too big to fit.

thanks
-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> The documentation for the 'highmem' option indicates that it controls
> the placement of both devices and RAM. The actual behaviour of QEMU
> seems to be that RAM is allowed to go beyond the 4GiB limit, and
> that only devices are constraint by this option.
>
> Align the documentation with the actual behaviour.

I think it would be better to align the behaviour with the documentation.

The intent of 'highmem' is to allow a configuration for use with guests
that can't address more than 32 bits (originally, 32-bit guests without
LPAE support compiled in). It seems like a bug that we allow the user
to specify more RAM than will fit into that 32-bit range. We should
instead make QEMU exit with an error if the user tries to specify
both highmem=off and a memory size that's too big to fit.

thanks
-- PMM

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
>
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
>
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
>
> Marc Zyngier (3):
>   hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
>     scratch VM
>   hw/arm/virt: Honor highmem setting when computing highest_gpa
>   docs/system/arm/virt: Fix documentation for the 'highmem' option

I've applied patch 1 to target-arm.next; thanks.

-- PMM

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
>
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
>
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
>
> Marc Zyngier (3):
>   hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
>     scratch VM
>   hw/arm/virt: Honor highmem setting when computing highest_gpa
>   docs/system/arm/virt: Fix documentation for the 'highmem' option

I've applied patch 1 to target-arm.next; thanks.

-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 0/3] target/arm: Reduced-IPA space and highmem=off fixes

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> With the availability of a fruity range of arm64 systems, it becomes
> obvious that QEMU doesn't deal very well with limited IPA ranges when
> used as a front-end for KVM.
>
> This short series aims at making usable on such systems:
> - the first patch makes the creation of a scratch VM IPA-limit aware
> - the second one actually removes the highmem devices from the
> computed IPA range when highmem=off
> - the last one addresses an imprecision in the documentation for the
> highmem option
>
> This has been tested on an M1-based Mac-mini running Linux v5.14-rc6.
>
> Marc Zyngier (3):
>   hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating
>     scratch VM
>   hw/arm/virt: Honor highmem setting when computing highest_gpa
>   docs/system/arm/virt: Fix documentation for the 'highmem' option

I've applied patch 1 to target-arm.next; thanks.

-- PMM

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> Even when the VM is configured with highmem=off, the highest_gpa
> field includes devices that are above the 4GiB limit, which is
> what highmem=off is supposed to enforce. This leads to failures
> in virt_kvm_type() on systems that have a crippled IPA range,
> as the reported IPA space is larger than what it should be.
>
> Instead, honor the user-specified limit to only use the devices
> at the lowest end of the spectrum.
>
> Note that this doesn't affect memory, which is still allowed to
> go beyond 4GiB with highmem=on configurations.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  hw/arm/virt.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 81eda46b0b..bc189e30b8 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>  static void virt_set_memmap(VirtMachineState *vms)
>  {
>      MachineState *ms = MACHINE(vms);
> -    hwaddr base, device_memory_base, device_memory_size;
> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>      int i;
>
>      vms->memmap = extended_memmap;
> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>
>      /* Base address of the high IO region */
> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>      if (base < device_memory_base) {
>          error_report("maxmem/slots too huge");
>          exit(EXIT_FAILURE);
> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>          vms->memmap[i].size = size;
>          base += size;
>      }
> -    vms->highest_gpa = base - 1;
> +    if (vms->highmem) {
> +           /* If we have highmem, move the IPA limit to the top */
> +           ceiling = base;
> +    }
> +    vms->highest_gpa = ceiling - 1;

This doesn't look right to me. If highmem is false and the
high IO region would be above the 4GB mark then we should not
create the high IO region at all, surely? This code looks like
it goes ahead and puts the high IO region above 4GB and then
lies in the highest_gpa value about what the highest used GPA is.

-- PMM

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> Even when the VM is configured with highmem=off, the highest_gpa
> field includes devices that are above the 4GiB limit, which is
> what highmem=off is supposed to enforce. This leads to failures
> in virt_kvm_type() on systems that have a crippled IPA range,
> as the reported IPA space is larger than what it should be.
>
> Instead, honor the user-specified limit to only use the devices
> at the lowest end of the spectrum.
>
> Note that this doesn't affect memory, which is still allowed to
> go beyond 4GiB with highmem=on configurations.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  hw/arm/virt.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 81eda46b0b..bc189e30b8 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>  static void virt_set_memmap(VirtMachineState *vms)
>  {
>      MachineState *ms = MACHINE(vms);
> -    hwaddr base, device_memory_base, device_memory_size;
> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>      int i;
>
>      vms->memmap = extended_memmap;
> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>
>      /* Base address of the high IO region */
> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>      if (base < device_memory_base) {
>          error_report("maxmem/slots too huge");
>          exit(EXIT_FAILURE);
> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>          vms->memmap[i].size = size;
>          base += size;
>      }
> -    vms->highest_gpa = base - 1;
> +    if (vms->highmem) {
> +           /* If we have highmem, move the IPA limit to the top */
> +           ceiling = base;
> +    }
> +    vms->highest_gpa = ceiling - 1;

This doesn't look right to me. If highmem is false and the
high IO region would be above the 4GB mark then we should not
create the high IO region at all, surely? This code looks like
it goes ahead and puts the high IO region above 4GB and then
lies in the highest_gpa value about what the highest used GPA is.

-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Peter Maydell]

On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>
> Even when the VM is configured with highmem=off, the highest_gpa
> field includes devices that are above the 4GiB limit, which is
> what highmem=off is supposed to enforce. This leads to failures
> in virt_kvm_type() on systems that have a crippled IPA range,
> as the reported IPA space is larger than what it should be.
>
> Instead, honor the user-specified limit to only use the devices
> at the lowest end of the spectrum.
>
> Note that this doesn't affect memory, which is still allowed to
> go beyond 4GiB with highmem=on configurations.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  hw/arm/virt.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 81eda46b0b..bc189e30b8 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>  static void virt_set_memmap(VirtMachineState *vms)
>  {
>      MachineState *ms = MACHINE(vms);
> -    hwaddr base, device_memory_base, device_memory_size;
> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>      int i;
>
>      vms->memmap = extended_memmap;
> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>
>      /* Base address of the high IO region */
> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>      if (base < device_memory_base) {
>          error_report("maxmem/slots too huge");
>          exit(EXIT_FAILURE);
> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>          vms->memmap[i].size = size;
>          base += size;
>      }
> -    vms->highest_gpa = base - 1;
> +    if (vms->highmem) {
> +           /* If we have highmem, move the IPA limit to the top */
> +           ceiling = base;
> +    }
> +    vms->highest_gpa = ceiling - 1;

This doesn't look right to me. If highmem is false and the
high IO region would be above the 4GB mark then we should not
create the high IO region at all, surely? This code looks like
it goes ahead and puts the high IO region above 4GB and then
lies in the highest_gpa value about what the highest used GPA is.

-- PMM

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

Hi Peter,

On Tue, 07 Sep 2021 13:51:13 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> >
> > The documentation for the 'highmem' option indicates that it controls
> > the placement of both devices and RAM. The actual behaviour of QEMU
> > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > that only devices are constraint by this option.
> >
> > Align the documentation with the actual behaviour.
> 
> I think it would be better to align the behaviour with the documentation.
> 
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

I'm happy to address this if you are OK with the change in user
visible behaviour.

However, I am still struggling with my original goal, which is to
allow QEMU to create a usable KVM_based VM on systems with a small IPA
space (36 bits on the system I have). What would an acceptable way to
convey this to the code that deals with the virt memory map so that it
falls back to something that actually works?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

Hi Peter,

On Tue, 07 Sep 2021 13:51:13 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> >
> > The documentation for the 'highmem' option indicates that it controls
> > the placement of both devices and RAM. The actual behaviour of QEMU
> > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > that only devices are constraint by this option.
> >
> > Align the documentation with the actual behaviour.
> 
> I think it would be better to align the behaviour with the documentation.
> 
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

I'm happy to address this if you are OK with the change in user
visible behaviour.

However, I am still struggling with my original goal, which is to
allow QEMU to create a usable KVM_based VM on systems with a small IPA
space (36 bits on the system I have). What would an acceptable way to
convey this to the code that deals with the virt memory map so that it
falls back to something that actually works?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

Hi Peter,

On Tue, 07 Sep 2021 13:51:13 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> >
> > The documentation for the 'highmem' option indicates that it controls
> > the placement of both devices and RAM. The actual behaviour of QEMU
> > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > that only devices are constraint by this option.
> >
> > Align the documentation with the actual behaviour.
> 
> I think it would be better to align the behaviour with the documentation.
> 
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

I'm happy to address this if you are OK with the change in user
visible behaviour.

However, I am still struggling with my original goal, which is to
allow QEMU to create a usable KVM_based VM on systems with a small IPA
space (36 bits on the system I have). What would an acceptable way to
convey this to the code that deals with the virt memory map so that it
falls back to something that actually works?

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Peter,
>
> On Tue, 07 Sep 2021 13:51:13 +0100,
> Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > The documentation for the 'highmem' option indicates that it controls
> > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > that only devices are constraint by this option.
> > >
> > > Align the documentation with the actual behaviour.
> >
> > I think it would be better to align the behaviour with the documentation.
> >
> > The intent of 'highmem' is to allow a configuration for use with guests
> > that can't address more than 32 bits (originally, 32-bit guests without
> > LPAE support compiled in). It seems like a bug that we allow the user
> > to specify more RAM than will fit into that 32-bit range. We should
> > instead make QEMU exit with an error if the user tries to specify
> > both highmem=off and a memory size that's too big to fit.
>
> I'm happy to address this if you are OK with the change in user
> visible behaviour.
>
> However, I am still struggling with my original goal, which is to
> allow QEMU to create a usable KVM_based VM on systems with a small IPA
> space (36 bits on the system I have). What would an acceptable way to
> convey this to the code that deals with the virt memory map so that it
> falls back to something that actually works?

Hmm, so at the moment we can either do "fits in 32 bits" or
"assumes at least 40 bits" but not 36 ?

thanks
-- PMM

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Peter,
>
> On Tue, 07 Sep 2021 13:51:13 +0100,
> Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > The documentation for the 'highmem' option indicates that it controls
> > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > that only devices are constraint by this option.
> > >
> > > Align the documentation with the actual behaviour.
> >
> > I think it would be better to align the behaviour with the documentation.
> >
> > The intent of 'highmem' is to allow a configuration for use with guests
> > that can't address more than 32 bits (originally, 32-bit guests without
> > LPAE support compiled in). It seems like a bug that we allow the user
> > to specify more RAM than will fit into that 32-bit range. We should
> > instead make QEMU exit with an error if the user tries to specify
> > both highmem=off and a memory size that's too big to fit.
>
> I'm happy to address this if you are OK with the change in user
> visible behaviour.
>
> However, I am still struggling with my original goal, which is to
> allow QEMU to create a usable KVM_based VM on systems with a small IPA
> space (36 bits on the system I have). What would an acceptable way to
> convey this to the code that deals with the virt memory map so that it
> falls back to something that actually works?

Hmm, so at the moment we can either do "fits in 32 bits" or
"assumes at least 40 bits" but not 36 ?

thanks
-- PMM

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Peter Maydell]

On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Peter,
>
> On Tue, 07 Sep 2021 13:51:13 +0100,
> Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > The documentation for the 'highmem' option indicates that it controls
> > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > that only devices are constraint by this option.
> > >
> > > Align the documentation with the actual behaviour.
> >
> > I think it would be better to align the behaviour with the documentation.
> >
> > The intent of 'highmem' is to allow a configuration for use with guests
> > that can't address more than 32 bits (originally, 32-bit guests without
> > LPAE support compiled in). It seems like a bug that we allow the user
> > to specify more RAM than will fit into that 32-bit range. We should
> > instead make QEMU exit with an error if the user tries to specify
> > both highmem=off and a memory size that's too big to fit.
>
> I'm happy to address this if you are OK with the change in user
> visible behaviour.
>
> However, I am still struggling with my original goal, which is to
> allow QEMU to create a usable KVM_based VM on systems with a small IPA
> space (36 bits on the system I have). What would an acceptable way to
> convey this to the code that deals with the virt memory map so that it
> falls back to something that actually works?

Hmm, so at the moment we can either do "fits in 32 bits" or
"assumes at least 40 bits" but not 36 ?

thanks
-- PMM
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Eric Auger]

Hi
On 9/7/21 2:58 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> Even when the VM is configured with highmem=off, the highest_gpa
>> field includes devices that are above the 4GiB limit, which is
>> what highmem=off is supposed to enforce. This leads to failures
>> in virt_kvm_type() on systems that have a crippled IPA range,
>> as the reported IPA space is larger than what it should be.
>>
>> Instead, honor the user-specified limit to only use the devices
>> at the lowest end of the spectrum.
>>
>> Note that this doesn't affect memory, which is still allowed to
>> go beyond 4GiB with highmem=on configurations.
>>
>> Cc: Andrew Jones <drjones@redhat.com>
>> Cc: Eric Auger <eric.auger@redhat.com>
>> Cc: Peter Maydell <peter.maydell@linaro.org>
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  hw/arm/virt.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index 81eda46b0b..bc189e30b8 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>>  static void virt_set_memmap(VirtMachineState *vms)
>>  {
>>      MachineState *ms = MACHINE(vms);
>> -    hwaddr base, device_memory_base, device_memory_size;
>> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>>      int i;
>>
>>      vms->memmap = extended_memmap;
>> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>>
>>      /* Base address of the high IO region */
>> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>>      if (base < device_memory_base) {
>>          error_report("maxmem/slots too huge");
>>          exit(EXIT_FAILURE);
>> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>>          vms->memmap[i].size = size;
>>          base += size;
>>      }
>> -    vms->highest_gpa = base - 1;
>> +    if (vms->highmem) {
>> +           /* If we have highmem, move the IPA limit to the top */
>> +           ceiling = base;
>> +    }
>> +    vms->highest_gpa = ceiling - 1;
> This doesn't look right to me. If highmem is false and the
> high IO region would be above the 4GB mark then we should not
> create the high IO region at all, surely? This code looks like
> it goes ahead and puts the high IO region above 4GB and then
> lies in the highest_gpa value about what the highest used GPA is.
>
> -- PMM
>
Doesn't the problem come from "if maxram_size is < 255GiB we keep the
legacy memory map" and set base = vms->memmap[VIRT_MEM].base +
LEGACY_RAMLIMIT_BYTES; leading to IO regions allocated above?
Instead shouldn't we condition this to highmem=on only then?

But by the way do we need to added extended_memmap IO regions at all if
highmem=off?
I am not wrong the VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO only are
used if highmem=on. In create_pcie(), base_mmio_high/size_mmio_high are
used if vms->highmem and we have ecam_id =
VIRT_ECAM_ID(vms->highmem_ecam); with vms->highmem_ecam &= vms->highmem
&& (!firmware_loaded || aarch64);

So if I do not miss anything maybe we could skip the allocation of the
extended_memmap IO regions if highmem=off?

And doesn't it look reasonable to limit the number of vcpus if highmem=off?

Thanks

Eric

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Eric Auger]

Hi
On 9/7/21 2:58 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> Even when the VM is configured with highmem=off, the highest_gpa
>> field includes devices that are above the 4GiB limit, which is
>> what highmem=off is supposed to enforce. This leads to failures
>> in virt_kvm_type() on systems that have a crippled IPA range,
>> as the reported IPA space is larger than what it should be.
>>
>> Instead, honor the user-specified limit to only use the devices
>> at the lowest end of the spectrum.
>>
>> Note that this doesn't affect memory, which is still allowed to
>> go beyond 4GiB with highmem=on configurations.
>>
>> Cc: Andrew Jones <drjones@redhat.com>
>> Cc: Eric Auger <eric.auger@redhat.com>
>> Cc: Peter Maydell <peter.maydell@linaro.org>
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  hw/arm/virt.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index 81eda46b0b..bc189e30b8 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>>  static void virt_set_memmap(VirtMachineState *vms)
>>  {
>>      MachineState *ms = MACHINE(vms);
>> -    hwaddr base, device_memory_base, device_memory_size;
>> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>>      int i;
>>
>>      vms->memmap = extended_memmap;
>> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>>
>>      /* Base address of the high IO region */
>> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>>      if (base < device_memory_base) {
>>          error_report("maxmem/slots too huge");
>>          exit(EXIT_FAILURE);
>> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>>          vms->memmap[i].size = size;
>>          base += size;
>>      }
>> -    vms->highest_gpa = base - 1;
>> +    if (vms->highmem) {
>> +           /* If we have highmem, move the IPA limit to the top */
>> +           ceiling = base;
>> +    }
>> +    vms->highest_gpa = ceiling - 1;
> This doesn't look right to me. If highmem is false and the
> high IO region would be above the 4GB mark then we should not
> create the high IO region at all, surely? This code looks like
> it goes ahead and puts the high IO region above 4GB and then
> lies in the highest_gpa value about what the highest used GPA is.
>
> -- PMM
>
Doesn't the problem come from "if maxram_size is < 255GiB we keep the
legacy memory map" and set base = vms->memmap[VIRT_MEM].base +
LEGACY_RAMLIMIT_BYTES; leading to IO regions allocated above?
Instead shouldn't we condition this to highmem=on only then?

But by the way do we need to added extended_memmap IO regions at all if
highmem=off?
I am not wrong the VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO only are
used if highmem=on. In create_pcie(), base_mmio_high/size_mmio_high are
used if vms->highmem and we have ecam_id =
VIRT_ECAM_ID(vms->highmem_ecam); with vms->highmem_ecam &= vms->highmem
&& (!firmware_loaded || aarch64);

So if I do not miss anything maybe we could skip the allocation of the
extended_memmap IO regions if highmem=off?

And doesn't it look reasonable to limit the number of vcpus if highmem=off?

Thanks

Eric

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 2/3] hw/arm/virt: Honor highmem setting when computing highest_gpa

[Eric Auger]

Hi
On 9/7/21 2:58 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> Even when the VM is configured with highmem=off, the highest_gpa
>> field includes devices that are above the 4GiB limit, which is
>> what highmem=off is supposed to enforce. This leads to failures
>> in virt_kvm_type() on systems that have a crippled IPA range,
>> as the reported IPA space is larger than what it should be.
>>
>> Instead, honor the user-specified limit to only use the devices
>> at the lowest end of the spectrum.
>>
>> Note that this doesn't affect memory, which is still allowed to
>> go beyond 4GiB with highmem=on configurations.
>>
>> Cc: Andrew Jones <drjones@redhat.com>
>> Cc: Eric Auger <eric.auger@redhat.com>
>> Cc: Peter Maydell <peter.maydell@linaro.org>
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  hw/arm/virt.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index 81eda46b0b..bc189e30b8 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -1598,7 +1598,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
>>  static void virt_set_memmap(VirtMachineState *vms)
>>  {
>>      MachineState *ms = MACHINE(vms);
>> -    hwaddr base, device_memory_base, device_memory_size;
>> +    hwaddr base, device_memory_base, device_memory_size, ceiling;
>>      int i;
>>
>>      vms->memmap = extended_memmap;
>> @@ -1625,7 +1625,7 @@ static void virt_set_memmap(VirtMachineState *vms)
>>      device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB;
>>
>>      /* Base address of the high IO region */
>> -    base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>> +    ceiling = base = device_memory_base + ROUND_UP(device_memory_size, GiB);
>>      if (base < device_memory_base) {
>>          error_report("maxmem/slots too huge");
>>          exit(EXIT_FAILURE);
>> @@ -1642,7 +1642,11 @@ static void virt_set_memmap(VirtMachineState *vms)
>>          vms->memmap[i].size = size;
>>          base += size;
>>      }
>> -    vms->highest_gpa = base - 1;
>> +    if (vms->highmem) {
>> +           /* If we have highmem, move the IPA limit to the top */
>> +           ceiling = base;
>> +    }
>> +    vms->highest_gpa = ceiling - 1;
> This doesn't look right to me. If highmem is false and the
> high IO region would be above the 4GB mark then we should not
> create the high IO region at all, surely? This code looks like
> it goes ahead and puts the high IO region above 4GB and then
> lies in the highest_gpa value about what the highest used GPA is.
>
> -- PMM
>
Doesn't the problem come from "if maxram_size is < 255GiB we keep the
legacy memory map" and set base = vms->memmap[VIRT_MEM].base +
LEGACY_RAMLIMIT_BYTES; leading to IO regions allocated above?
Instead shouldn't we condition this to highmem=on only then?

But by the way do we need to added extended_memmap IO regions at all if
highmem=off?
I am not wrong the VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO only are
used if highmem=on. In create_pcie(), base_mmio_high/size_mmio_high are
used if vms->highmem and we have ecam_id =
VIRT_ECAM_ID(vms->highmem_ecam); with vms->highmem_ecam &= vms->highmem
&& (!firmware_loaded || aarch64);

So if I do not miss anything maybe we could skip the allocation of the
extended_memmap IO regions if highmem=off?

And doesn't it look reasonable to limit the number of vcpus if highmem=off?

Thanks

Eric

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Eric Auger]

Hi Marc,

On 8/22/21 4:44 PM, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
>
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
>
> Instead, probe for the extension and use the reported IPA limit
> if available.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Eric Auger]

Hi Marc,

On 8/22/21 4:44 PM, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
>
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
>
> Instead, probe for the extension and use the reported IPA limit
> if available.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 1/3] hw/arm/virt: KVM: Probe for KVM_CAP_ARM_VM_IPA_SIZE when creating scratch VM

[Eric Auger]

Hi Marc,

On 8/22/21 4:44 PM, Marc Zyngier wrote:
> Although we probe for the IPA limits imposed by KVM (and the hardware)
> when computing the memory map, we still use the old style '0' when
> creating a scratch VM in kvm_arm_create_scratch_host_vcpu().
>
> On systems that are severely IPA challenged (such as the Apple M1),
> this results in a failure as KVM cannot use the default 40bit that
> '0' represents.
>
> Instead, probe for the extension and use the reported IPA limit
> if available.
>
> Cc: Andrew Jones <drjones@redhat.com>
> Cc: Eric Auger <eric.auger@redhat.com>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  target/arm/kvm.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index d8381ba224..cc3371a99b 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -70,12 +70,17 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        struct kvm_vcpu_init *init)
>  {
>      int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int max_vm_pa_size;
>  
>      kvmfd = qemu_open_old("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
>          goto err;
>      }
> -    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
> +    max_vm_pa_size = ioctl(kvmfd, KVM_CHECK_EXTENSION, KVM_CAP_ARM_VM_IPA_SIZE);
> +    if (max_vm_pa_size < 0) {
> +        max_vm_pa_size = 0;
> +    }
> +    vmfd = ioctl(kvmfd, KVM_CREATE_VM, max_vm_pa_size);
>      if (vmfd < 0) {
>          goto err;
>      }

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Eric Auger]

Hi,

On 9/7/21 2:51 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> The documentation for the 'highmem' option indicates that it controls
>> the placement of both devices and RAM. The actual behaviour of QEMU
>> seems to be that RAM is allowed to go beyond the 4GiB limit, and
>> that only devices are constraint by this option.
>>
>> Align the documentation with the actual behaviour.
> I think it would be better to align the behaviour with the documentation.
>
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

That's my opinion too

Thanks

Eric
>
> thanks
> -- PMM
>

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Eric Auger]

Hi,

On 9/7/21 2:51 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> The documentation for the 'highmem' option indicates that it controls
>> the placement of both devices and RAM. The actual behaviour of QEMU
>> seems to be that RAM is allowed to go beyond the 4GiB limit, and
>> that only devices are constraint by this option.
>>
>> Align the documentation with the actual behaviour.
> I think it would be better to align the behaviour with the documentation.
>
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

That's my opinion too

Thanks

Eric
>
> thanks
> -- PMM
>

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Eric Auger]

Hi,

On 9/7/21 2:51 PM, Peter Maydell wrote:
> On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
>> The documentation for the 'highmem' option indicates that it controls
>> the placement of both devices and RAM. The actual behaviour of QEMU
>> seems to be that RAM is allowed to go beyond the 4GiB limit, and
>> that only devices are constraint by this option.
>>
>> Align the documentation with the actual behaviour.
> I think it would be better to align the behaviour with the documentation.
>
> The intent of 'highmem' is to allow a configuration for use with guests
> that can't address more than 32 bits (originally, 32-bit guests without
> LPAE support compiled in). It seems like a bug that we allow the user
> to specify more RAM than will fit into that 32-bit range. We should
> instead make QEMU exit with an error if the user tries to specify
> both highmem=off and a memory size that's too big to fit.

That's my opinion too

Thanks

Eric
>
> thanks
> -- PMM
>

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

On Tue, 07 Sep 2021 19:25:23 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Peter,
> >
> > On Tue, 07 Sep 2021 13:51:13 +0100,
> > Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > The documentation for the 'highmem' option indicates that it controls
> > > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > > that only devices are constraint by this option.
> > > >
> > > > Align the documentation with the actual behaviour.
> > >
> > > I think it would be better to align the behaviour with the documentation.
> > >
> > > The intent of 'highmem' is to allow a configuration for use with guests
> > > that can't address more than 32 bits (originally, 32-bit guests without
> > > LPAE support compiled in). It seems like a bug that we allow the user
> > > to specify more RAM than will fit into that 32-bit range. We should
> > > instead make QEMU exit with an error if the user tries to specify
> > > both highmem=off and a memory size that's too big to fit.
> >
> > I'm happy to address this if you are OK with the change in user
> > visible behaviour.
> >
> > However, I am still struggling with my original goal, which is to
> > allow QEMU to create a usable KVM_based VM on systems with a small IPA
> > space (36 bits on the system I have). What would an acceptable way to
> > convey this to the code that deals with the virt memory map so that it
> > falls back to something that actually works?
> 
> Hmm, so at the moment we can either do "fits in 32 bits" or
> "assumes at least 40 bits" but not 36 ?

Exactly. I have the gut feeling that we need a 'gpa_bits' option that
would limit the guest physical range and generalise highmem. High IO
ranges would simply not be available if the GPA range isn't big
enough.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

On Tue, 07 Sep 2021 19:25:23 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Peter,
> >
> > On Tue, 07 Sep 2021 13:51:13 +0100,
> > Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > The documentation for the 'highmem' option indicates that it controls
> > > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > > that only devices are constraint by this option.
> > > >
> > > > Align the documentation with the actual behaviour.
> > >
> > > I think it would be better to align the behaviour with the documentation.
> > >
> > > The intent of 'highmem' is to allow a configuration for use with guests
> > > that can't address more than 32 bits (originally, 32-bit guests without
> > > LPAE support compiled in). It seems like a bug that we allow the user
> > > to specify more RAM than will fit into that 32-bit range. We should
> > > instead make QEMU exit with an error if the user tries to specify
> > > both highmem=off and a memory size that's too big to fit.
> >
> > I'm happy to address this if you are OK with the change in user
> > visible behaviour.
> >
> > However, I am still struggling with my original goal, which is to
> > allow QEMU to create a usable KVM_based VM on systems with a small IPA
> > space (36 bits on the system I have). What would an acceptable way to
> > convey this to the code that deals with the virt memory map so that it
> > falls back to something that actually works?
> 
> Hmm, so at the moment we can either do "fits in 32 bits" or
> "assumes at least 40 bits" but not 36 ?

Exactly. I have the gut feeling that we need a 'gpa_bits' option that
would limit the guest physical range and generalise highmem. High IO
ranges would simply not be available if the GPA range isn't big
enough.

	M.

-- 
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [PATCH 3/3] docs/system/arm/virt: Fix documentation for the 'highmem' option

[Marc Zyngier]

On Tue, 07 Sep 2021 19:25:23 +0100,
Peter Maydell <peter.maydell@linaro.org> wrote:
> 
> On Tue, 7 Sept 2021 at 18:10, Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Peter,
> >
> > On Tue, 07 Sep 2021 13:51:13 +0100,
> > Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Sun, 22 Aug 2021 at 15:45, Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > The documentation for the 'highmem' option indicates that it controls
> > > > the placement of both devices and RAM. The actual behaviour of QEMU
> > > > seems to be that RAM is allowed to go beyond the 4GiB limit, and
> > > > that only devices are constraint by this option.
> > > >
> > > > Align the documentation with the actual behaviour.
> > >
> > > I think it would be better to align the behaviour with the documentation.
> > >
> > > The intent of 'highmem' is to allow a configuration for use with guests
> > > that can't address more than 32 bits (originally, 32-bit guests without
> > > LPAE support compiled in). It seems like a bug that we allow the user
> > > to specify more RAM than will fit into that 32-bit range. We should
> > > instead make QEMU exit with an error if the user tries to specify
> > > both highmem=off and a memory size that's too big to fit.
> >
> > I'm happy to address this if you are OK with the change in user
> > visible behaviour.
> >
> > However, I am still struggling with my original goal, which is to
> > allow QEMU to create a usable KVM_based VM on systems with a small IPA
> > space (36 bits on the system I have). What would an acceptable way to
> > convey this to the code that deals with the virt memory map so that it
> > falls back to something that actually works?
> 
> Hmm, so at the moment we can either do "fits in 32 bits" or
> "assumes at least 40 bits" but not 36 ?

Exactly. I have the gut feeling that we need a 'gpa_bits' option that
would limit the guest physical range and generalise highmem. High IO
ranges would simply not be available if the GPA range isn't big
enough.

	M.

-- 
Without deviation from the norm, progress is not possible.