[PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Gavin Shan]

There are three high memory regions, which are VIRT_HIGH_REDIST2,
VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
are floating on highest RAM address. However, they can be disabled
in several cases.

(1) One specific high memory region is disabled by developer by
    toggling vms->highmem_{redists, ecam, mmio}.

(2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
    'virt-2.12' or ealier than it.

(3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
    on 32-bits system.

(4) One specific high memory region is disabled when it breaks the
    PA space limit.

The current implementation of virt_set_memmap() isn't comprehensive
because the space for one specific high memory region is always
reserved from the PA space for case (1), (2) and (3). In the code,
'base' and 'vms->highest_gpa' are always increased for those three
cases. It's unnecessary since the assigned space of the disabled
high memory region won't be used afterwards.

The series intends to improve the address assignment for these
high memory regions and introduces new properties for user to
selectively disable those 3 high memory regions.

PATCH[1-4] preparatory work for the improvment
PATCH[5]   improve high memory region address assignment
PATCH[6]   adds 'compact-highmem' to enable or disable the optimization
PATCH[7]   adds properties so that high memory regions can be disabled

v5: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00280.html
v4: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00067.html
v3: https://lists.nongnu.org/archive/html/qemu-arm/2022-09/msg00258.html
v2: https://lore.kernel.org/all/20220815062958.100366-1-gshan@redhat.com/T/
v1: https://lists.nongnu.org/archive/html/qemu-arm/2022-08/msg00013.html

Changelog
==========
v6:
  * Pick review-by from Connie/Eric                            (Connie/Eric)
  * Make the changes obvious in PATCH[v6 5/7]                  (Eric)
  * Move the example to commit log and describe the legacy
    and compact layout in code's comments in PATCH[v6 6/7]     (Eric)
  * Comment and commit message improvements                    (Connie/Eric)
  * Add 3 properties in PATCH[v6 7/7], allowing user to disable
    those 3 high memory regions                                (Marc)
v5:
  * Pick review-by and tested-by                               (Connie/Zhenyu)
  * Add extra check in PATCH[v5 4/6]                           (Connie)
  * Improve comments about compatibility for disabled regions
    in PATCH[v5 5/6]                                           (Connie)
v4:
  * Add virt_get_high_memmap_enabled() helper                  (Eric)
  * Move 'vms->highmem_compact' and related logic from
    PATCH[v4 6/6] to PATCH[v4 5/6] to avoid git-bisect
    breakage                                                   (Eric)
  * Document the legacy and optimized high memory region
    layout in commit log and source code                       (Eric)
v3:
  * Reorder the patches                                        (Gavin)
  * Add 'highmem-compact' property for backwards compatibility (Eric)
v2:
  * Split the patches for easier review                        (Gavin)
  * Improved changelog                                         (Marc)
  * Use 'bool fits' in virt_set_high_memmap()                  (Eric)

Gavin Shan (7):
  hw/arm/virt: Introduce virt_set_high_memmap() helper
  hw/arm/virt: Rename variable size to region_size in
    virt_set_high_memmap()
  hw/arm/virt: Introduce variable region_base in virt_set_high_memmap()
  hw/arm/virt: Introduce virt_get_high_memmap_enabled() helper
  hw/arm/virt: Improve high memory region address assignment
  hw/arm/virt: Add 'compact-highmem' property
  hw/arm/virt: Add properties to disable high memory regions

 docs/system/arm/virt.rst |  16 ++++
 hw/arm/virt.c            | 182 ++++++++++++++++++++++++++++++++-------
 include/hw/arm/virt.h    |   2 +
 3 files changed, 167 insertions(+), 33 deletions(-)

-- 
2.23.0

[PATCH v6 1/7] hw/arm/virt: Introduce virt_set_high_memmap() helper

[Gavin Shan]

This introduces virt_set_high_memmap() helper. The logic of high
memory region address assignment is moved to the helper. The intention
is to make the subsequent optimization for high memory region address
assignment easier.

No functional change intended.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 hw/arm/virt.c | 74 ++++++++++++++++++++++++++++-----------------------
 1 file changed, 41 insertions(+), 33 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index cda9defe8f..7572c44bda 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1689,6 +1689,46 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
     return arm_cpu_mp_affinity(idx, clustersz);
 }
 
+static void virt_set_high_memmap(VirtMachineState *vms,
+                                 hwaddr base, int pa_bits)
+{
+    int i;
+
+    for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
+        hwaddr size = extended_memmap[i].size;
+        bool fits;
+
+        base = ROUND_UP(base, size);
+        vms->memmap[i].base = base;
+        vms->memmap[i].size = size;
+
+        /*
+         * Check each device to see if they fit in the PA space,
+         * moving highest_gpa as we go.
+         *
+         * For each device that doesn't fit, disable it.
+         */
+        fits = (base + size) <= BIT_ULL(pa_bits);
+        if (fits) {
+            vms->highest_gpa = base + size - 1;
+        }
+
+        switch (i) {
+        case VIRT_HIGH_GIC_REDIST2:
+            vms->highmem_redists &= fits;
+            break;
+        case VIRT_HIGH_PCIE_ECAM:
+            vms->highmem_ecam &= fits;
+            break;
+        case VIRT_HIGH_PCIE_MMIO:
+            vms->highmem_mmio &= fits;
+            break;
+        }
+
+        base += size;
+    }
+}
+
 static void virt_set_memmap(VirtMachineState *vms, int pa_bits)
 {
     MachineState *ms = MACHINE(vms);
@@ -1744,39 +1784,7 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits)
     /* We know for sure that at least the memory fits in the PA space */
     vms->highest_gpa = memtop - 1;
 
-    for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
-        hwaddr size = extended_memmap[i].size;
-        bool fits;
-
-        base = ROUND_UP(base, size);
-        vms->memmap[i].base = base;
-        vms->memmap[i].size = size;
-
-        /*
-         * Check each device to see if they fit in the PA space,
-         * moving highest_gpa as we go.
-         *
-         * For each device that doesn't fit, disable it.
-         */
-        fits = (base + size) <= BIT_ULL(pa_bits);
-        if (fits) {
-            vms->highest_gpa = base + size - 1;
-        }
-
-        switch (i) {
-        case VIRT_HIGH_GIC_REDIST2:
-            vms->highmem_redists &= fits;
-            break;
-        case VIRT_HIGH_PCIE_ECAM:
-            vms->highmem_ecam &= fits;
-            break;
-        case VIRT_HIGH_PCIE_MMIO:
-            vms->highmem_mmio &= fits;
-            break;
-        }
-
-        base += size;
-    }
+    virt_set_high_memmap(vms, base, pa_bits);
 
     if (device_memory_size > 0) {
         ms->device_memory = g_malloc0(sizeof(*ms->device_memory));
-- 
2.23.0

[PATCH v6 2/7] hw/arm/virt: Rename variable size to region_size in virt_set_high_memmap()

[Gavin Shan]

This renames variable 'size' to 'region_size' in virt_set_high_memmap().
Its counterpart ('region_base') will be introduced in next patch.

No functional change intended.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 hw/arm/virt.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 7572c44bda..e2ae88cf8b 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1692,15 +1692,16 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 static void virt_set_high_memmap(VirtMachineState *vms,
                                  hwaddr base, int pa_bits)
 {
+    hwaddr region_size;
+    bool fits;
     int i;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
-        hwaddr size = extended_memmap[i].size;
-        bool fits;
+        region_size = extended_memmap[i].size;
 
-        base = ROUND_UP(base, size);
+        base = ROUND_UP(base, region_size);
         vms->memmap[i].base = base;
-        vms->memmap[i].size = size;
+        vms->memmap[i].size = region_size;
 
         /*
          * Check each device to see if they fit in the PA space,
@@ -1708,9 +1709,9 @@ static void virt_set_high_memmap(VirtMachineState *vms,
          *
          * For each device that doesn't fit, disable it.
          */
-        fits = (base + size) <= BIT_ULL(pa_bits);
+        fits = (base + region_size) <= BIT_ULL(pa_bits);
         if (fits) {
-            vms->highest_gpa = base + size - 1;
+            vms->highest_gpa = base + region_size - 1;
         }
 
         switch (i) {
@@ -1725,7 +1726,7 @@ static void virt_set_high_memmap(VirtMachineState *vms,
             break;
         }
 
-        base += size;
+        base += region_size;
     }
 }
 
-- 
2.23.0

[PATCH v6 3/7] hw/arm/virt: Introduce variable region_base in virt_set_high_memmap()

[Gavin Shan]

This introduces variable 'region_base' for the base address of the
specific high memory region. It's the preparatory work to optimize
high memory region address assignment.

No functional change intended.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 hw/arm/virt.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index e2ae88cf8b..0bf3cb7057 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1692,15 +1692,15 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
 static void virt_set_high_memmap(VirtMachineState *vms,
                                  hwaddr base, int pa_bits)
 {
-    hwaddr region_size;
+    hwaddr region_base, region_size;
     bool fits;
     int i;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
+        region_base = ROUND_UP(base, extended_memmap[i].size);
         region_size = extended_memmap[i].size;
 
-        base = ROUND_UP(base, region_size);
-        vms->memmap[i].base = base;
+        vms->memmap[i].base = region_base;
         vms->memmap[i].size = region_size;
 
         /*
@@ -1709,9 +1709,9 @@ static void virt_set_high_memmap(VirtMachineState *vms,
          *
          * For each device that doesn't fit, disable it.
          */
-        fits = (base + region_size) <= BIT_ULL(pa_bits);
+        fits = (region_base + region_size) <= BIT_ULL(pa_bits);
         if (fits) {
-            vms->highest_gpa = base + region_size - 1;
+            vms->highest_gpa = region_base + region_size - 1;
         }
 
         switch (i) {
@@ -1726,7 +1726,7 @@ static void virt_set_high_memmap(VirtMachineState *vms,
             break;
         }
 
-        base += region_size;
+        base = region_base + region_size;
     }
 }
 
-- 
2.23.0

[PATCH v6 4/7] hw/arm/virt: Introduce virt_get_high_memmap_enabled() helper

[Gavin Shan]

This introduces virt_get_high_memmap_enabled() helper, which returns
the pointer to vms->highmem_{redists, ecam, mmio}. The pointer will
be used in the subsequent patches.

No functional change intended.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 hw/arm/virt.c | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 0bf3cb7057..ee98a8a3b6 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1689,14 +1689,31 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
     return arm_cpu_mp_affinity(idx, clustersz);
 }
 
+static inline bool *virt_get_high_memmap_enabled(VirtMachineState *vms,
+                                                 int index)
+{
+    bool *enabled_array[] = {
+        &vms->highmem_redists,
+        &vms->highmem_ecam,
+        &vms->highmem_mmio,
+    };
+
+    assert(ARRAY_SIZE(extended_memmap) - VIRT_LOWMEMMAP_LAST ==
+           ARRAY_SIZE(enabled_array));
+    assert(index - VIRT_LOWMEMMAP_LAST < ARRAY_SIZE(enabled_array));
+
+    return enabled_array[index - VIRT_LOWMEMMAP_LAST];
+}
+
 static void virt_set_high_memmap(VirtMachineState *vms,
                                  hwaddr base, int pa_bits)
 {
     hwaddr region_base, region_size;
-    bool fits;
+    bool *region_enabled, fits;
     int i;
 
     for (i = VIRT_LOWMEMMAP_LAST; i < ARRAY_SIZE(extended_memmap); i++) {
+        region_enabled = virt_get_high_memmap_enabled(vms, i);
         region_base = ROUND_UP(base, extended_memmap[i].size);
         region_size = extended_memmap[i].size;
 
@@ -1714,18 +1731,7 @@ static void virt_set_high_memmap(VirtMachineState *vms,
             vms->highest_gpa = region_base + region_size - 1;
         }
 
-        switch (i) {
-        case VIRT_HIGH_GIC_REDIST2:
-            vms->highmem_redists &= fits;
-            break;
-        case VIRT_HIGH_PCIE_ECAM:
-            vms->highmem_ecam &= fits;
-            break;
-        case VIRT_HIGH_PCIE_MMIO:
-            vms->highmem_mmio &= fits;
-            break;
-        }
-
+        *region_enabled &= fits;
         base = region_base + region_size;
     }
 }
-- 
2.23.0

[PATCH v6 5/7] hw/arm/virt: Improve high memory region address assignment

[Gavin Shan]

There are three high memory regions, which are VIRT_HIGH_REDIST2,
VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
are floating on highest RAM address. However, they can be disabled
in several cases.

(1) One specific high memory region is likely to be disabled by
    code by toggling vms->highmem_{redists, ecam, mmio}.

(2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
    'virt-2.12' or ealier than it.

(3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
    on 32-bits system.

(4) One specific high memory region is disabled when it breaks the
    PA space limit.

The current implementation of virt_set_{memmap, high_memmap}() isn't
optimized because the high memory region's PA space is always reserved,
regardless of whatever the actual state in the corresponding
vms->highmem_{redists, ecam, mmio} flag. In the code, 'base' and
'vms->highest_gpa' are always increased for case (1), (2) and (3).
It's unnecessary since the assigned PA space for the disabled high
memory region won't be used afterwards.

Improve the address assignment for those three high memory region by
skipping the address assignment for one specific high memory region if
it has been disabled in case (1), (2) and (3). The memory layout may
be changed after the improvement is applied, which leads to potential
migration breakage. So 'vms->highmem_compact' is added to control if
the improvement should be applied. For now, 'vms->highmem_compact' is
set to false, meaning that we don't have memory layout change until it
becomes configurable through property 'compact-highmem' in next patch.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 hw/arm/virt.c         | 15 ++++++++++-----
 include/hw/arm/virt.h |  1 +
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index ee98a8a3b6..4896f600b4 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms,
         vms->memmap[i].size = region_size;
 
         /*
-         * Check each device to see if they fit in the PA space,
-         * moving highest_gpa as we go.
+         * Check each device to see if it fits in the PA space,
+         * moving highest_gpa as we go. For compatibility, move
+         * highest_gpa for disabled fitting devices as well, if
+         * the compact layout has been disabled.
          *
          * For each device that doesn't fit, disable it.
          */
         fits = (region_base + region_size) <= BIT_ULL(pa_bits);
-        if (fits) {
-            vms->highest_gpa = region_base + region_size - 1;
+        *region_enabled &= fits;
+        if (vms->highmem_compact && !*region_enabled) {
+            continue;
         }
 
-        *region_enabled &= fits;
         base = region_base + region_size;
+        if (fits) {
+            vms->highest_gpa = region_base + region_size - 1;
+        }
     }
 }
 
diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
index 6ec479ca2b..709f623741 100644
--- a/include/hw/arm/virt.h
+++ b/include/hw/arm/virt.h
@@ -144,6 +144,7 @@ struct VirtMachineState {
     PFlashCFI01 *flash[2];
     bool secure;
     bool highmem;
+    bool highmem_compact;
     bool highmem_ecam;
     bool highmem_mmio;
     bool highmem_redists;
-- 
2.23.0

[PATCH v6 6/7] hw/arm/virt: Add 'compact-highmem' property

[Gavin Shan]

After the improvement to high memory region address assignment is
applied, the memory layout can be changed, introducing possible
migration breakage. For example, VIRT_HIGH_PCIE_MMIO memory region
is disabled or enabled when the optimization is applied or not, with
the following configuration. The configuration is only achievable by
modifying the source code until more properties are added to allow
users selectively disable those high memory regions.

  pa_bits              = 40;
  vms->highmem_redists = false;
  vms->highmem_ecam    = false;
  vms->highmem_mmio    = true;

  # qemu-system-aarch64 -accel kvm -cpu host    \
    -machine virt-7.2,compact-highmem={on, off} \
    -m 4G,maxmem=511G -monitor stdio

  Region             compact-highmem=off         compact-highmem=on
  ----------------------------------------------------------------
  MEM                [1GB         512GB]        [1GB         512GB]
  HIGH_GIC_REDISTS2  [512GB       512GB+64MB]   [disabled]
  HIGH_PCIE_ECAM     [512GB+256MB 512GB+512MB]  [disabled]
  HIGH_PCIE_MMIO     [disabled]                 [512GB       1TB]

In order to keep backwords compatibility, we need to disable the
optimization on machine, which is virt-7.1 or ealier than it. It
means the optimization is enabled by default from virt-7.2. Besides,
'compact-highmem' property is added so that the optimization can be
explicitly enabled or disabled on all machine types by users.

Signed-off-by: Gavin Shan <gshan@redhat.com>
Reviewed-by: Cornelia Huck <cohuck@redhat.com>
Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
---
 docs/system/arm/virt.rst |  4 ++++
 hw/arm/virt.c            | 32 ++++++++++++++++++++++++++++++++
 include/hw/arm/virt.h    |  1 +
 3 files changed, 37 insertions(+)

diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index 20442ea2c1..4454706392 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -94,6 +94,10 @@ highmem
   address space above 32 bits. The default is ``on`` for machine types
   later than ``virt-2.12``.
 
+compact-highmem
+  Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
+  The default is ``on`` for machine types later than ``virt-7.2``.
+
 gic-version
   Specify the version of the Generic Interrupt Controller (GIC) to provide.
   Valid values are:
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 4896f600b4..11b5685432 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -174,6 +174,12 @@ static const MemMapEntry base_memmap[] = {
  * Note the extended_memmap is sized so that it eventually also includes the
  * base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last
  * index of base_memmap).
+ *
+ * The memory map for these Highmem IO Regions can be in legacy or compact
+ * layout, depending on 'compact-highmem' property. With legacy layout, the
+ * PA space for one specific region is always reserved, even the region has
+ * been disabled or doesn't fit into the PA space. However, the PA space for
+ * the region won't be reserved in these circumstances with compact layout.
  */
 static MemMapEntry extended_memmap[] = {
     /* Additional 64 MB redist region (can contain up to 512 redistributors) */
@@ -2351,6 +2357,20 @@ static void virt_set_highmem(Object *obj, bool value, Error **errp)
     vms->highmem = value;
 }
 
+static bool virt_get_compact_highmem(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return vms->highmem_compact;
+}
+
+static void virt_set_compact_highmem(Object *obj, bool value, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    vms->highmem_compact = value;
+}
+
 static bool virt_get_its(Object *obj, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(obj);
@@ -2969,6 +2989,13 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
                                           "Set on/off to enable/disable using "
                                           "physical address space above 32 bits");
 
+    object_class_property_add_bool(oc, "compact-highmem",
+                                   virt_get_compact_highmem,
+                                   virt_set_compact_highmem);
+    object_class_property_set_description(oc, "compact-highmem",
+                                          "Set on/off to enable/disable compact "
+                                          "layout for high memory regions");
+
     object_class_property_add_str(oc, "gic-version", virt_get_gic_version,
                                   virt_set_gic_version);
     object_class_property_set_description(oc, "gic-version",
@@ -3053,6 +3080,7 @@ static void virt_instance_init(Object *obj)
 
     /* High memory is enabled by default */
     vms->highmem = true;
+    vms->highmem_compact = !vmc->no_highmem_compact;
     vms->gic_version = VIRT_GIC_VERSION_NOSEL;
 
     vms->highmem_ecam = !vmc->no_highmem_ecam;
@@ -3122,8 +3150,12 @@ DEFINE_VIRT_MACHINE_AS_LATEST(7, 2)
 
 static void virt_machine_7_1_options(MachineClass *mc)
 {
+    VirtMachineClass *vmc = VIRT_MACHINE_CLASS(OBJECT_CLASS(mc));
+
     virt_machine_7_2_options(mc);
     compat_props_add(mc->compat_props, hw_compat_7_1, hw_compat_7_1_len);
+    /* Compact layout for high memory regions was introduced with 7.2 */
+    vmc->no_highmem_compact = true;
 }
 DEFINE_VIRT_MACHINE(7, 1)
 
diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
index 709f623741..c7dd59d7f1 100644
--- a/include/hw/arm/virt.h
+++ b/include/hw/arm/virt.h
@@ -125,6 +125,7 @@ struct VirtMachineClass {
     bool no_pmu;
     bool claim_edge_triggered_timers;
     bool smbios_old_sys_ver;
+    bool no_highmem_compact;
     bool no_highmem_ecam;
     bool no_ged;   /* Machines < 4.2 have no support for ACPI GED device */
     bool kvm_no_adjvtime;
-- 
2.23.0

[PATCH v6 7/7] hw/arm/virt: Add properties to disable high memory regions

[Gavin Shan]

These 3 high memory regions are usually enabled by default, but
they may be not used. For example, VIRT_HIGH_GIC_REDIST2 isn't
needed by GICv2. This leads to waste in the PA space.

Add properties to allow users selectively disable them if needed:
"highmem-redists", "highmem-ecam", "highmem-mmio".

Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 docs/system/arm/virt.rst | 12 ++++++++
 hw/arm/virt.c            | 64 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 76 insertions(+)

diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index 4454706392..a1668a969d 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -98,6 +98,18 @@ compact-highmem
   Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
   The default is ``on`` for machine types later than ``virt-7.2``.
 
+highmem-redists
+  Set ``on``/``off`` to enable/disable the high memry region for GICv3/4
+  redistributor. The default is ``on``.
+
+highmem-ecam
+  Set ``on``/``off`` to enable/disable the high memry region for PCI ECAM.
+  The default is ``on`` for machine types later than ``virt-3.0``.
+
+highmem-mmio
+  Set ``on``/``off`` to enable/disable the high memry region for PCI MMIO.
+  The default is ``on``.
+
 gic-version
   Specify the version of the Generic Interrupt Controller (GIC) to provide.
   Valid values are:
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 11b5685432..afafc2d1b8 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -2371,6 +2371,49 @@ static void virt_set_compact_highmem(Object *obj, bool value, Error **errp)
     vms->highmem_compact = value;
 }
 
+static bool virt_get_highmem_redists(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return vms->highmem_redists;
+}
+
+static void virt_set_highmem_redists(Object *obj, bool value, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    vms->highmem_redists = value;
+}
+
+static bool virt_get_highmem_ecam(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return vms->highmem_ecam;
+}
+
+static void virt_set_highmem_ecam(Object *obj, bool value, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    vms->highmem_ecam = value;
+}
+
+static bool virt_get_highmem_mmio(Object *obj, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    return vms->highmem_mmio;
+}
+
+static void virt_set_highmem_mmio(Object *obj, bool value, Error **errp)
+{
+    VirtMachineState *vms = VIRT_MACHINE(obj);
+
+    vms->highmem_mmio = value;
+}
+
+
 static bool virt_get_its(Object *obj, Error **errp)
 {
     VirtMachineState *vms = VIRT_MACHINE(obj);
@@ -2996,6 +3039,27 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
                                           "Set on/off to enable/disable compact "
                                           "layout for high memory regions");
 
+    object_class_property_add_bool(oc, "highmem-redists",
+                                   virt_get_highmem_redists,
+                                   virt_set_highmem_redists);
+    object_class_property_set_description(oc, "highmem-redists",
+                                          "Set on/off to enable/disable high "
+                                          "memory region for GICv3/4 redistributor");
+
+    object_class_property_add_bool(oc, "highmem-ecam",
+                                   virt_get_highmem_ecam,
+                                   virt_set_highmem_ecam);
+    object_class_property_set_description(oc, "highmem-ecam",
+                                          "Set on/off to enable/disable high "
+                                          "memory region for PCI ECAM");
+
+    object_class_property_add_bool(oc, "highmem-mmio",
+                                   virt_get_highmem_mmio,
+                                   virt_set_highmem_mmio);
+    object_class_property_set_description(oc, "highmem-mmio",
+                                          "Set on/off to enable/disable high "
+                                          "memory region for PCI MMIO");
+
     object_class_property_add_str(oc, "gic-version", virt_get_gic_version,
                                   virt_set_gic_version);
     object_class_property_set_description(oc, "gic-version",
-- 
2.23.0

Re: [PATCH v6 6/7] hw/arm/virt: Add 'compact-highmem' property

[Cornelia Huck]

On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:

> After the improvement to high memory region address assignment is
> applied, the memory layout can be changed, introducing possible
> migration breakage. For example, VIRT_HIGH_PCIE_MMIO memory region
> is disabled or enabled when the optimization is applied or not, with
> the following configuration. The configuration is only achievable by
> modifying the source code until more properties are added to allow
> users selectively disable those high memory regions.
>
>   pa_bits              = 40;
>   vms->highmem_redists = false;
>   vms->highmem_ecam    = false;
>   vms->highmem_mmio    = true;
>
>   # qemu-system-aarch64 -accel kvm -cpu host    \
>     -machine virt-7.2,compact-highmem={on, off} \
>     -m 4G,maxmem=511G -monitor stdio
>
>   Region             compact-highmem=off         compact-highmem=on
>   ----------------------------------------------------------------
>   MEM                [1GB         512GB]        [1GB         512GB]
>   HIGH_GIC_REDISTS2  [512GB       512GB+64MB]   [disabled]
>   HIGH_PCIE_ECAM     [512GB+256MB 512GB+512MB]  [disabled]
>   HIGH_PCIE_MMIO     [disabled]                 [512GB       1TB]
>
> In order to keep backwords compatibility, we need to disable the
> optimization on machine, which is virt-7.1 or ealier than it. It
> means the optimization is enabled by default from virt-7.2. Besides,
> 'compact-highmem' property is added so that the optimization can be
> explicitly enabled or disabled on all machine types by users.
>
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
> ---
>  docs/system/arm/virt.rst |  4 ++++
>  hw/arm/virt.c            | 32 ++++++++++++++++++++++++++++++++
>  include/hw/arm/virt.h    |  1 +
>  3 files changed, 37 insertions(+)
>

(...)

> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index 4896f600b4..11b5685432 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -174,6 +174,12 @@ static const MemMapEntry base_memmap[] = {
>   * Note the extended_memmap is sized so that it eventually also includes the
>   * base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last
>   * index of base_memmap).
> + *
> + * The memory map for these Highmem IO Regions can be in legacy or compact
> + * layout, depending on 'compact-highmem' property. With legacy layout, the
> + * PA space for one specific region is always reserved, even the region has

s/even/even if/

> + * been disabled or doesn't fit into the PA space. However, the PA space for
> + * the region won't be reserved in these circumstances with compact layout.
>   */
>  static MemMapEntry extended_memmap[] = {
>      /* Additional 64 MB redist region (can contain up to 512 redistributors) */

Re: [PATCH v6 7/7] hw/arm/virt: Add properties to disable high memory regions

[Cornelia Huck]

On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:

> These 3 high memory regions are usually enabled by default, but

s/These 3/The/ ?

> they may be not used. For example, VIRT_HIGH_GIC_REDIST2 isn't
> needed by GICv2. This leads to waste in the PA space.

When building the command line, do we have enough information on when
the regions provide something useful, and when they just waste space?

>
> Add properties to allow users selectively disable them if needed:
> "highmem-redists", "highmem-ecam", "highmem-mmio".
>
> Suggested-by: Marc Zyngier <maz@kernel.org>
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  docs/system/arm/virt.rst | 12 ++++++++
>  hw/arm/virt.c            | 64 ++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 76 insertions(+)
>
> diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
> index 4454706392..a1668a969d 100644
> --- a/docs/system/arm/virt.rst
> +++ b/docs/system/arm/virt.rst
> @@ -98,6 +98,18 @@ compact-highmem
>    Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
>    The default is ``on`` for machine types later than ``virt-7.2``.
>  
> +highmem-redists
> +  Set ``on``/``off`` to enable/disable the high memry region for GICv3/4

s/memry/memory/

> +  redistributor. The default is ``on``.

Do we need to add a note about what effects setting this to "off" may
have, e.g. "Setting this to ``off`` may limit the maximum number of
cpus." or so? And/or "Setting this to ``off`` when using GICv2 will save
some space."?

> +
> +highmem-ecam
> +  Set ``on``/``off`` to enable/disable the high memry region for PCI ECAM.

s/memry/memory/

> +  The default is ``on`` for machine types later than ``virt-3.0``.
> +
> +highmem-mmio
> +  Set ``on``/``off`` to enable/disable the high memry region for PCI MMIO.

s/memry/memory/

> +  The default is ``on``.
> +
>  gic-version
>    Specify the version of the Generic Interrupt Controller (GIC) to provide.
>    Valid values are:

Re: [PATCH v6 5/7] hw/arm/virt: Improve high memory region address assignment

[Eric Auger]

Hi Gavin,

On 10/24/22 05:54, Gavin Shan wrote:
> There are three high memory regions, which are VIRT_HIGH_REDIST2,
> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
> are floating on highest RAM address. However, they can be disabled
> in several cases.
>
> (1) One specific high memory region is likely to be disabled by
>     code by toggling vms->highmem_{redists, ecam, mmio}.
>
> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>     'virt-2.12' or ealier than it.
>
> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>     on 32-bits system.
>
> (4) One specific high memory region is disabled when it breaks the
>     PA space limit.
>
> The current implementation of virt_set_{memmap, high_memmap}() isn't
> optimized because the high memory region's PA space is always reserved,
> regardless of whatever the actual state in the corresponding
> vms->highmem_{redists, ecam, mmio} flag. In the code, 'base' and
> 'vms->highest_gpa' are always increased for case (1), (2) and (3).
> It's unnecessary since the assigned PA space for the disabled high
> memory region won't be used afterwards.
>
> Improve the address assignment for those three high memory region by
> skipping the address assignment for one specific high memory region if
> it has been disabled in case (1), (2) and (3). The memory layout may
> be changed after the improvement is applied, which leads to potential
> migration breakage. So 'vms->highmem_compact' is added to control if
> the improvement should be applied. For now, 'vms->highmem_compact' is
> set to false, meaning that we don't have memory layout change until it
> becomes configurable through property 'compact-highmem' in next patch.
>
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
the code has quite changed since Connie's R-b
> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
> ---
>  hw/arm/virt.c         | 15 ++++++++++-----
>  include/hw/arm/virt.h |  1 +
>  2 files changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> index ee98a8a3b6..4896f600b4 100644
> --- a/hw/arm/virt.c
> +++ b/hw/arm/virt.c
> @@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms,
>          vms->memmap[i].size = region_size;
>  
>          /*
> -         * Check each device to see if they fit in the PA space,
> -         * moving highest_gpa as we go.
> +         * Check each device to see if it fits in the PA space,
> +         * moving highest_gpa as we go. For compatibility, move
> +         * highest_gpa for disabled fitting devices as well, if
> +         * the compact layout has been disabled.
>           *
>           * For each device that doesn't fit, disable it.
>           */
>          fits = (region_base + region_size) <= BIT_ULL(pa_bits);
> -        if (fits) {
> -            vms->highest_gpa = region_base + region_size - 1;
> +        *region_enabled &= fits;
> +        if (vms->highmem_compact && !*region_enabled) {
> +            continue;
>          }
>  
> -        *region_enabled &= fits;
>          base = region_base + region_size;
> +        if (fits) {
> +            vms->highest_gpa = region_base + region_size - 1;

vms->highest_gpa = base - 1;

> +        }
>      }
>  }
>  
> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
> index 6ec479ca2b..709f623741 100644
> --- a/include/hw/arm/virt.h
> +++ b/include/hw/arm/virt.h
> @@ -144,6 +144,7 @@ struct VirtMachineState {
>      PFlashCFI01 *flash[2];
>      bool secure;
>      bool highmem;
> +    bool highmem_compact;
>      bool highmem_ecam;
>      bool highmem_mmio;
>      bool highmem_redists;
Besides
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [PATCH v6 6/7] hw/arm/virt: Add 'compact-highmem' property

[Eric Auger]

On 10/25/22 12:30, Cornelia Huck wrote:
> On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:
>
>> After the improvement to high memory region address assignment is
>> applied, the memory layout can be changed, introducing possible
>> migration breakage. For example, VIRT_HIGH_PCIE_MMIO memory region
>> is disabled or enabled when the optimization is applied or not, with
>> the following configuration. The configuration is only achievable by
>> modifying the source code until more properties are added to allow
>> users selectively disable those high memory regions.
>>
>>   pa_bits              = 40;
>>   vms->highmem_redists = false;
>>   vms->highmem_ecam    = false;
>>   vms->highmem_mmio    = true;
>>
>>   # qemu-system-aarch64 -accel kvm -cpu host    \
>>     -machine virt-7.2,compact-highmem={on, off} \
>>     -m 4G,maxmem=511G -monitor stdio
>>
>>   Region             compact-highmem=off         compact-highmem=on
>>   ----------------------------------------------------------------
>>   MEM                [1GB         512GB]        [1GB         512GB]
>>   HIGH_GIC_REDISTS2  [512GB       512GB+64MB]   [disabled]
>>   HIGH_PCIE_ECAM     [512GB+256MB 512GB+512MB]  [disabled]
>>   HIGH_PCIE_MMIO     [disabled]                 [512GB       1TB]
>>
>> In order to keep backwords compatibility, we need to disable the
>> optimization on machine, which is virt-7.1 or ealier than it. It
>> means the optimization is enabled by default from virt-7.2. Besides,
>> 'compact-highmem' property is added so that the optimization can be
>> explicitly enabled or disabled on all machine types by users.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
>> ---
>>  docs/system/arm/virt.rst |  4 ++++
>>  hw/arm/virt.c            | 32 ++++++++++++++++++++++++++++++++
>>  include/hw/arm/virt.h    |  1 +
>>  3 files changed, 37 insertions(+)
>>
> (...)
>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index 4896f600b4..11b5685432 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -174,6 +174,12 @@ static const MemMapEntry base_memmap[] = {
>>   * Note the extended_memmap is sized so that it eventually also includes the
>>   * base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last
>>   * index of base_memmap).
>> + *
>> + * The memory map for these Highmem IO Regions can be in legacy or compact
>> + * layout, depending on 'compact-highmem' property. With legacy layout, the
>> + * PA space for one specific region is always reserved, even the region has
> s/even/even if/

Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric
>
>> + * been disabled or doesn't fit into the PA space. However, the PA space for
>> + * the region won't be reserved in these circumstances with compact layout.
>>   */
>>  static MemMapEntry extended_memmap[] = {
>>      /* Additional 64 MB redist region (can contain up to 512 redistributors) */

Re: [PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Gavin Shan]

Hi Peter and Marc,

On 10/24/22 11:54 AM, Gavin Shan wrote:
> There are three high memory regions, which are VIRT_HIGH_REDIST2,
> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
> are floating on highest RAM address. However, they can be disabled
> in several cases.
> 
> (1) One specific high memory region is disabled by developer by
>      toggling vms->highmem_{redists, ecam, mmio}.
> 
> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>      'virt-2.12' or ealier than it.
> 
> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>      on 32-bits system.
> 
> (4) One specific high memory region is disabled when it breaks the
>      PA space limit.
> 
> The current implementation of virt_set_memmap() isn't comprehensive
> because the space for one specific high memory region is always
> reserved from the PA space for case (1), (2) and (3). In the code,
> 'base' and 'vms->highest_gpa' are always increased for those three
> cases. It's unnecessary since the assigned space of the disabled
> high memory region won't be used afterwards.
> 
> The series intends to improve the address assignment for these
> high memory regions and introduces new properties for user to
> selectively disable those 3 high memory regions.
> 
> PATCH[1-4] preparatory work for the improvment
> PATCH[5]   improve high memory region address assignment
> PATCH[6]   adds 'compact-highmem' to enable or disable the optimization
> PATCH[7]   adds properties so that high memory regions can be disabled
> 
> v5: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00280.html
> v4: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00067.html
> v3: https://lists.nongnu.org/archive/html/qemu-arm/2022-09/msg00258.html
> v2: https://lore.kernel.org/all/20220815062958.100366-1-gshan@redhat.com/T/
> v1: https://lists.nongnu.org/archive/html/qemu-arm/2022-08/msg00013.html
> 

Could you help to take a look when getting a chance? I think Connie and
Eric are close to complete the reviews, but v7 is still needed to address
extra comments from them. I hope to make v7 mergeable if possible :)

Thanks,
Gavin

> Changelog
> ==========
> v6:
>    * Pick review-by from Connie/Eric                            (Connie/Eric)
>    * Make the changes obvious in PATCH[v6 5/7]                  (Eric)
>    * Move the example to commit log and describe the legacy
>      and compact layout in code's comments in PATCH[v6 6/7]     (Eric)
>    * Comment and commit message improvements                    (Connie/Eric)
>    * Add 3 properties in PATCH[v6 7/7], allowing user to disable
>      those 3 high memory regions                                (Marc)
> v5:
>    * Pick review-by and tested-by                               (Connie/Zhenyu)
>    * Add extra check in PATCH[v5 4/6]                           (Connie)
>    * Improve comments about compatibility for disabled regions
>      in PATCH[v5 5/6]                                           (Connie)
> v4:
>    * Add virt_get_high_memmap_enabled() helper                  (Eric)
>    * Move 'vms->highmem_compact' and related logic from
>      PATCH[v4 6/6] to PATCH[v4 5/6] to avoid git-bisect
>      breakage                                                   (Eric)
>    * Document the legacy and optimized high memory region
>      layout in commit log and source code                       (Eric)
> v3:
>    * Reorder the patches                                        (Gavin)
>    * Add 'highmem-compact' property for backwards compatibility (Eric)
> v2:
>    * Split the patches for easier review                        (Gavin)
>    * Improved changelog                                         (Marc)
>    * Use 'bool fits' in virt_set_high_memmap()                  (Eric)
> 
> Gavin Shan (7):
>    hw/arm/virt: Introduce virt_set_high_memmap() helper
>    hw/arm/virt: Rename variable size to region_size in
>      virt_set_high_memmap()
>    hw/arm/virt: Introduce variable region_base in virt_set_high_memmap()
>    hw/arm/virt: Introduce virt_get_high_memmap_enabled() helper
>    hw/arm/virt: Improve high memory region address assignment
>    hw/arm/virt: Add 'compact-highmem' property
>    hw/arm/virt: Add properties to disable high memory regions
> 
>   docs/system/arm/virt.rst |  16 ++++
>   hw/arm/virt.c            | 182 ++++++++++++++++++++++++++++++++-------
>   include/hw/arm/virt.h    |   2 +
>   3 files changed, 167 insertions(+), 33 deletions(-)
> 

Re: [PATCH v6 5/7] hw/arm/virt: Improve high memory region address assignment

[Gavin Shan]

Hi Eric,

On 10/26/22 12:29 AM, Eric Auger wrote:
> On 10/24/22 05:54, Gavin Shan wrote:
>> There are three high memory regions, which are VIRT_HIGH_REDIST2,
>> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
>> are floating on highest RAM address. However, they can be disabled
>> in several cases.
>>
>> (1) One specific high memory region is likely to be disabled by
>>      code by toggling vms->highmem_{redists, ecam, mmio}.
>>
>> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>>      'virt-2.12' or ealier than it.
>>
>> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>>      on 32-bits system.
>>
>> (4) One specific high memory region is disabled when it breaks the
>>      PA space limit.
>>
>> The current implementation of virt_set_{memmap, high_memmap}() isn't
>> optimized because the high memory region's PA space is always reserved,
>> regardless of whatever the actual state in the corresponding
>> vms->highmem_{redists, ecam, mmio} flag. In the code, 'base' and
>> 'vms->highest_gpa' are always increased for case (1), (2) and (3).
>> It's unnecessary since the assigned PA space for the disabled high
>> memory region won't be used afterwards.
>>
>> Improve the address assignment for those three high memory region by
>> skipping the address assignment for one specific high memory region if
>> it has been disabled in case (1), (2) and (3). The memory layout may
>> be changed after the improvement is applied, which leads to potential
>> migration breakage. So 'vms->highmem_compact' is added to control if
>> the improvement should be applied. For now, 'vms->highmem_compact' is
>> set to false, meaning that we don't have memory layout change until it
>> becomes configurable through property 'compact-highmem' in next patch.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
> the code has quite changed since Connie's R-b

Right. Connie, could you please check if the changes make sense to you
and I can regain your R-B? :)

>> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
>> ---
>>   hw/arm/virt.c         | 15 ++++++++++-----
>>   include/hw/arm/virt.h |  1 +
>>   2 files changed, 11 insertions(+), 5 deletions(-)
>>
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index ee98a8a3b6..4896f600b4 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms,
>>           vms->memmap[i].size = region_size;
>>   
>>           /*
>> -         * Check each device to see if they fit in the PA space,
>> -         * moving highest_gpa as we go.
>> +         * Check each device to see if it fits in the PA space,
>> +         * moving highest_gpa as we go. For compatibility, move
>> +         * highest_gpa for disabled fitting devices as well, if
>> +         * the compact layout has been disabled.
>>            *
>>            * For each device that doesn't fit, disable it.
>>            */
>>           fits = (region_base + region_size) <= BIT_ULL(pa_bits);
>> -        if (fits) {
>> -            vms->highest_gpa = region_base + region_size - 1;
>> +        *region_enabled &= fits;
>> +        if (vms->highmem_compact && !*region_enabled) {
>> +            continue;
>>           }
>>   
>> -        *region_enabled &= fits;
>>           base = region_base + region_size;
>> +        if (fits) {
>> +            vms->highest_gpa = region_base + region_size - 1;
> 
> vms->highest_gpa = base - 1;
> 

It's personal taste actually. I was thinking of using 'base - 1', but
'region_base + region_size - 1' looks more like a direct way. I don't
have strong sense though and lets use 'base - 1' in next respin.

>> +        }
>>       }
>>   }
>>   
>> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
>> index 6ec479ca2b..709f623741 100644
>> --- a/include/hw/arm/virt.h
>> +++ b/include/hw/arm/virt.h
>> @@ -144,6 +144,7 @@ struct VirtMachineState {
>>       PFlashCFI01 *flash[2];
>>       bool secure;
>>       bool highmem;
>> +    bool highmem_compact;
>>       bool highmem_ecam;
>>       bool highmem_mmio;
>>       bool highmem_redists;
> Besides
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> 

Thanks,
Gavin

Re: [PATCH v6 6/7] hw/arm/virt: Add 'compact-highmem' property

[Gavin Shan]

Hi Connie,

On 10/25/22 6:30 PM, Cornelia Huck wrote:
> On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:
> 
>> After the improvement to high memory region address assignment is
>> applied, the memory layout can be changed, introducing possible
>> migration breakage. For example, VIRT_HIGH_PCIE_MMIO memory region
>> is disabled or enabled when the optimization is applied or not, with
>> the following configuration. The configuration is only achievable by
>> modifying the source code until more properties are added to allow
>> users selectively disable those high memory regions.
>>
>>    pa_bits              = 40;
>>    vms->highmem_redists = false;
>>    vms->highmem_ecam    = false;
>>    vms->highmem_mmio    = true;
>>
>>    # qemu-system-aarch64 -accel kvm -cpu host    \
>>      -machine virt-7.2,compact-highmem={on, off} \
>>      -m 4G,maxmem=511G -monitor stdio
>>
>>    Region             compact-highmem=off         compact-highmem=on
>>    ----------------------------------------------------------------
>>    MEM                [1GB         512GB]        [1GB         512GB]
>>    HIGH_GIC_REDISTS2  [512GB       512GB+64MB]   [disabled]
>>    HIGH_PCIE_ECAM     [512GB+256MB 512GB+512MB]  [disabled]
>>    HIGH_PCIE_MMIO     [disabled]                 [512GB       1TB]
>>
>> In order to keep backwords compatibility, we need to disable the
>> optimization on machine, which is virt-7.1 or ealier than it. It
>> means the optimization is enabled by default from virt-7.2. Besides,
>> 'compact-highmem' property is added so that the optimization can be
>> explicitly enabled or disabled on all machine types by users.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
>> ---
>>   docs/system/arm/virt.rst |  4 ++++
>>   hw/arm/virt.c            | 32 ++++++++++++++++++++++++++++++++
>>   include/hw/arm/virt.h    |  1 +
>>   3 files changed, 37 insertions(+)
>>
> 
> (...)
> 
>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>> index 4896f600b4..11b5685432 100644
>> --- a/hw/arm/virt.c
>> +++ b/hw/arm/virt.c
>> @@ -174,6 +174,12 @@ static const MemMapEntry base_memmap[] = {
>>    * Note the extended_memmap is sized so that it eventually also includes the
>>    * base_memmap entries (VIRT_HIGH_GIC_REDIST2 index is greater than the last
>>    * index of base_memmap).
>> + *
>> + * The memory map for these Highmem IO Regions can be in legacy or compact
>> + * layout, depending on 'compact-highmem' property. With legacy layout, the
>> + * PA space for one specific region is always reserved, even the region has
> 
> s/even/even if/
> 

Thanks, it will be improved as suggested in next respin (v7).

>> + * been disabled or doesn't fit into the PA space. However, the PA space for
>> + * the region won't be reserved in these circumstances with compact layout.
>>    */
>>   static MemMapEntry extended_memmap[] = {
>>       /* Additional 64 MB redist region (can contain up to 512 redistributors) */
> 

Thanks,
Gavin

Re: [PATCH v6 7/7] hw/arm/virt: Add properties to disable high memory regions

[Gavin Shan]

Hi Connie,

On 10/25/22 6:54 PM, Cornelia Huck wrote:
> On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:
> 
>> These 3 high memory regions are usually enabled by default, but
> 
> s/These 3/The/ ?
> 

Ok.

>> they may be not used. For example, VIRT_HIGH_GIC_REDIST2 isn't
>> needed by GICv2. This leads to waste in the PA space.
> 
> When building the command line, do we have enough information on when
> the regions provide something useful, and when they just waste space?
> 

I think the help messages are already indicative. For example, the help
messages for 'highmem-redist2' indicate the region is only needed by
GICv3 or GICv4. 'highmem-ecam' and 'highmem-mmio' are needed by PCI ECAM
and MMIO and the key words 'high' indicates they're the corresponding
second window.

#./qemu-system-aarch64 -M virt,?
highmem-ecam=<bool>    - Set on/off to enable/disable high memory region for PCI ECAM
highmem-mmio=<bool>    - Set on/off to enable/disable high memory region for PCI MMIO
highmem-redists=<bool> - Set on/off to enable/disable high memory region for GICv3 or GICv4 redistributor

>>
>> Add properties to allow users selectively disable them if needed:
>> "highmem-redists", "highmem-ecam", "highmem-mmio".
>>
>> Suggested-by: Marc Zyngier <maz@kernel.org>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   docs/system/arm/virt.rst | 12 ++++++++
>>   hw/arm/virt.c            | 64 ++++++++++++++++++++++++++++++++++++++++
>>   2 files changed, 76 insertions(+)
>>
>> diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
>> index 4454706392..a1668a969d 100644
>> --- a/docs/system/arm/virt.rst
>> +++ b/docs/system/arm/virt.rst
>> @@ -98,6 +98,18 @@ compact-highmem
>>     Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
>>     The default is ``on`` for machine types later than ``virt-7.2``.
>>   
>> +highmem-redists
>> +  Set ``on``/``off`` to enable/disable the high memry region for GICv3/4
> 
> s/memry/memory/
> 

Ok, copy-and-paste error. Will be fixed.

>> +  redistributor. The default is ``on``.
> 
> Do we need to add a note about what effects setting this to "off" may
> have, e.g. "Setting this to ``off`` may limit the maximum number of
> cpus." or so? And/or "Setting this to ``off`` when using GICv2 will save
> some space."?
> 

We may not mention GICv2 since GICv3/v4 are already mentioned. It's a
good idea to mention that the maximum number of CPUs is reduced when
it's turned off. I will have something like below in next respin if
you agree.

highmem-redists
   Set ``on``/``off`` to enable/disable the high memroy region for GICv3 or
   GICv4 redistributor. The default is ``on``. Setting this to ``off`` will
   limit the maximum number of CPUs when GICv3 or GICv4 is used.

Since 'vms->highmem_redists' is changeable, the 'virt_max_cpus' in
machvirt_init() needs to be recalculated based on that. The code change
will be included into next respin. Besides, the follow-up error message
will be improved to something like below.

   error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
                "supported by machine 'mach-virt' (%d). The high memory "
                "region for GICv3 or GICv4 redistributor has been %s",
                max_cpus, virt_max_cpus,
                vms->highmem_redists ? "enabled" : "disabled");

>> +
>> +highmem-ecam
>> +  Set ``on``/``off`` to enable/disable the high memry region for PCI ECAM.
> 
> s/memry/memory/
> 

Ok, copy-and-paste error. Will be fixed.

>> +  The default is ``on`` for machine types later than ``virt-3.0``.
>> +
>> +highmem-mmio
>> +  Set ``on``/``off`` to enable/disable the high memry region for PCI MMIO.
> 
> s/memry/memory/
> 

Ok. copy-and-paste error. Will be fixed.

>> +  The default is ``on``.
>> +
>>   gic-version
>>     Specify the version of the Generic Interrupt Controller (GIC) to provide.
>>     Valid values are:
> 

Thanks,
Gavin

Re: [PATCH v6 5/7] hw/arm/virt: Improve high memory region address assignment

[Cornelia Huck]

On Wed, Oct 26 2022, Gavin Shan <gshan@redhat.com> wrote:

> Hi Eric,
>
> On 10/26/22 12:29 AM, Eric Auger wrote:
>> On 10/24/22 05:54, Gavin Shan wrote:
>>> There are three high memory regions, which are VIRT_HIGH_REDIST2,
>>> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
>>> are floating on highest RAM address. However, they can be disabled
>>> in several cases.
>>>
>>> (1) One specific high memory region is likely to be disabled by
>>>      code by toggling vms->highmem_{redists, ecam, mmio}.
>>>
>>> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>>>      'virt-2.12' or ealier than it.
>>>
>>> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>>>      on 32-bits system.
>>>
>>> (4) One specific high memory region is disabled when it breaks the
>>>      PA space limit.
>>>
>>> The current implementation of virt_set_{memmap, high_memmap}() isn't
>>> optimized because the high memory region's PA space is always reserved,
>>> regardless of whatever the actual state in the corresponding
>>> vms->highmem_{redists, ecam, mmio} flag. In the code, 'base' and
>>> 'vms->highest_gpa' are always increased for case (1), (2) and (3).
>>> It's unnecessary since the assigned PA space for the disabled high
>>> memory region won't be used afterwards.
>>>
>>> Improve the address assignment for those three high memory region by
>>> skipping the address assignment for one specific high memory region if
>>> it has been disabled in case (1), (2) and (3). The memory layout may
>>> be changed after the improvement is applied, which leads to potential
>>> migration breakage. So 'vms->highmem_compact' is added to control if
>>> the improvement should be applied. For now, 'vms->highmem_compact' is
>>> set to false, meaning that we don't have memory layout change until it
>>> becomes configurable through property 'compact-highmem' in next patch.
>>>
>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>> the code has quite changed since Connie's R-b
>
> Right. Connie, could you please check if the changes make sense to you
> and I can regain your R-B? :)

My R-b still holds.

>
>>> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
>>> ---
>>>   hw/arm/virt.c         | 15 ++++++++++-----
>>>   include/hw/arm/virt.h |  1 +
>>>   2 files changed, 11 insertions(+), 5 deletions(-)
>>>
>>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>>> index ee98a8a3b6..4896f600b4 100644
>>> --- a/hw/arm/virt.c
>>> +++ b/hw/arm/virt.c
>>> @@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms,
>>>           vms->memmap[i].size = region_size;
>>>   
>>>           /*
>>> -         * Check each device to see if they fit in the PA space,
>>> -         * moving highest_gpa as we go.
>>> +         * Check each device to see if it fits in the PA space,
>>> +         * moving highest_gpa as we go. For compatibility, move
>>> +         * highest_gpa for disabled fitting devices as well, if
>>> +         * the compact layout has been disabled.
>>>            *
>>>            * For each device that doesn't fit, disable it.
>>>            */
>>>           fits = (region_base + region_size) <= BIT_ULL(pa_bits);
>>> -        if (fits) {
>>> -            vms->highest_gpa = region_base + region_size - 1;
>>> +        *region_enabled &= fits;
>>> +        if (vms->highmem_compact && !*region_enabled) {
>>> +            continue;
>>>           }
>>>   
>>> -        *region_enabled &= fits;
>>>           base = region_base + region_size;
>>> +        if (fits) {
>>> +            vms->highest_gpa = region_base + region_size - 1;
>> 
>> vms->highest_gpa = base - 1;
>> 
>
> It's personal taste actually. I was thinking of using 'base - 1', but
> 'region_base + region_size - 1' looks more like a direct way. I don't
> have strong sense though and lets use 'base - 1' in next respin.

I don't really have a preference for one or the other.

Re: [PATCH v6 7/7] hw/arm/virt: Add properties to disable high memory regions

[Cornelia Huck]

On Wed, Oct 26 2022, Gavin Shan <gshan@redhat.com> wrote:

> Hi Connie,
>
> On 10/25/22 6:54 PM, Cornelia Huck wrote:
>> On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:
>> 
>>> These 3 high memory regions are usually enabled by default, but
>> 
>> s/These 3/The/ ?
>> 
>
> Ok.
>
>>> they may be not used. For example, VIRT_HIGH_GIC_REDIST2 isn't
>>> needed by GICv2. This leads to waste in the PA space.
>> 
>> When building the command line, do we have enough information on when
>> the regions provide something useful, and when they just waste space?
>> 
>
> I think the help messages are already indicative. For example, the help
> messages for 'highmem-redist2' indicate the region is only needed by
> GICv3 or GICv4. 'highmem-ecam' and 'highmem-mmio' are needed by PCI ECAM
> and MMIO and the key words 'high' indicates they're the corresponding
> second window.
>
> #./qemu-system-aarch64 -M virt,?
> highmem-ecam=<bool>    - Set on/off to enable/disable high memory region for PCI ECAM
> highmem-mmio=<bool>    - Set on/off to enable/disable high memory region for PCI MMIO
> highmem-redists=<bool> - Set on/off to enable/disable high memory region for GICv3 or GICv4 redistributor

OK, hopefully this is enough for anyone building a command line
directly. (Do we want to encourage management software like libvirt to
switch off regions that are not needed?)

>
>>>
>>> Add properties to allow users selectively disable them if needed:
>>> "highmem-redists", "highmem-ecam", "highmem-mmio".
>>>
>>> Suggested-by: Marc Zyngier <maz@kernel.org>
>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>> ---
>>>   docs/system/arm/virt.rst | 12 ++++++++
>>>   hw/arm/virt.c            | 64 ++++++++++++++++++++++++++++++++++++++++
>>>   2 files changed, 76 insertions(+)
>>>
>>> diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
>>> index 4454706392..a1668a969d 100644
>>> --- a/docs/system/arm/virt.rst
>>> +++ b/docs/system/arm/virt.rst
>>> @@ -98,6 +98,18 @@ compact-highmem
>>>     Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
>>>     The default is ``on`` for machine types later than ``virt-7.2``.
>>>   
>>> +highmem-redists
>>> +  Set ``on``/``off`` to enable/disable the high memry region for GICv3/4
>> 
>> s/memry/memory/
>> 
>
> Ok, copy-and-paste error. Will be fixed.
>
>>> +  redistributor. The default is ``on``.
>> 
>> Do we need to add a note about what effects setting this to "off" may
>> have, e.g. "Setting this to ``off`` may limit the maximum number of
>> cpus." or so? And/or "Setting this to ``off`` when using GICv2 will save
>> some space."?
>> 
>
> We may not mention GICv2 since GICv3/v4 are already mentioned. It's a
> good idea to mention that the maximum number of CPUs is reduced when
> it's turned off. I will have something like below in next respin if
> you agree.
>
> highmem-redists
>    Set ``on``/``off`` to enable/disable the high memroy region for GICv3 or
>    GICv4 redistributor. The default is ``on``. Setting this to ``off`` will
>    limit the maximum number of CPUs when GICv3 or GICv4 is used.

OK, sounds reasonable to me.

>
> Since 'vms->highmem_redists' is changeable, the 'virt_max_cpus' in
> machvirt_init() needs to be recalculated based on that. The code change
> will be included into next respin. Besides, the follow-up error message
> will be improved to something like below.
>
>    error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
>                 "supported by machine 'mach-virt' (%d). The high memory "
>                 "region for GICv3 or GICv4 redistributor has been %s",
>                 max_cpus, virt_max_cpus,
>                 vms->highmem_redists ? "enabled" : "disabled");

Hm, the doc for error_report() states that "The resulting message should
be a single phrase, with no newline or trailing punctuation." Maybe

    if (max_cpus > virt_max_cpus) {
        error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
                     "supported by machine 'mach-virt' (%d)",
                     max_cpus, virt_max_cpus);
        if (vms->gic_version != VIRT_GIC_VERSION_2 &&
            !vms->higmem_redists) {
            error_printf("Try 'highmem-redists=on' for more CPUs\n");
        }
        exit(1);
    }

Re: [PATCH v6 5/7] hw/arm/virt: Improve high memory region address assignment

[Gavin Shan]

Hi Connie,

On 10/26/22 6:43 PM, Cornelia Huck wrote:
> On Wed, Oct 26 2022, Gavin Shan <gshan@redhat.com> wrote:
>> On 10/26/22 12:29 AM, Eric Auger wrote:
>>> On 10/24/22 05:54, Gavin Shan wrote:
>>>> There are three high memory regions, which are VIRT_HIGH_REDIST2,
>>>> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
>>>> are floating on highest RAM address. However, they can be disabled
>>>> in several cases.
>>>>
>>>> (1) One specific high memory region is likely to be disabled by
>>>>       code by toggling vms->highmem_{redists, ecam, mmio}.
>>>>
>>>> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>>>>       'virt-2.12' or ealier than it.
>>>>
>>>> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>>>>       on 32-bits system.
>>>>
>>>> (4) One specific high memory region is disabled when it breaks the
>>>>       PA space limit.
>>>>
>>>> The current implementation of virt_set_{memmap, high_memmap}() isn't
>>>> optimized because the high memory region's PA space is always reserved,
>>>> regardless of whatever the actual state in the corresponding
>>>> vms->highmem_{redists, ecam, mmio} flag. In the code, 'base' and
>>>> 'vms->highest_gpa' are always increased for case (1), (2) and (3).
>>>> It's unnecessary since the assigned PA space for the disabled high
>>>> memory region won't be used afterwards.
>>>>
>>>> Improve the address assignment for those three high memory region by
>>>> skipping the address assignment for one specific high memory region if
>>>> it has been disabled in case (1), (2) and (3). The memory layout may
>>>> be changed after the improvement is applied, which leads to potential
>>>> migration breakage. So 'vms->highmem_compact' is added to control if
>>>> the improvement should be applied. For now, 'vms->highmem_compact' is
>>>> set to false, meaning that we don't have memory layout change until it
>>>> becomes configurable through property 'compact-highmem' in next patch.
>>>>
>>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>>> Reviewed-by: Cornelia Huck <cohuck@redhat.com>
>>> the code has quite changed since Connie's R-b
>>
>> Right. Connie, could you please check if the changes make sense to you
>> and I can regain your R-B? :)
> 
> My R-b still holds.
> 

Thanks.

>>
>>>> Tested-by: Zhenyu Zhang <zhenyzha@redhat.com>
>>>> ---
>>>>    hw/arm/virt.c         | 15 ++++++++++-----
>>>>    include/hw/arm/virt.h |  1 +
>>>>    2 files changed, 11 insertions(+), 5 deletions(-)
>>>>
>>>> diff --git a/hw/arm/virt.c b/hw/arm/virt.c
>>>> index ee98a8a3b6..4896f600b4 100644
>>>> --- a/hw/arm/virt.c
>>>> +++ b/hw/arm/virt.c
>>>> @@ -1721,18 +1721,23 @@ static void virt_set_high_memmap(VirtMachineState *vms,
>>>>            vms->memmap[i].size = region_size;
>>>>    
>>>>            /*
>>>> -         * Check each device to see if they fit in the PA space,
>>>> -         * moving highest_gpa as we go.
>>>> +         * Check each device to see if it fits in the PA space,
>>>> +         * moving highest_gpa as we go. For compatibility, move
>>>> +         * highest_gpa for disabled fitting devices as well, if
>>>> +         * the compact layout has been disabled.
>>>>             *
>>>>             * For each device that doesn't fit, disable it.
>>>>             */
>>>>            fits = (region_base + region_size) <= BIT_ULL(pa_bits);
>>>> -        if (fits) {
>>>> -            vms->highest_gpa = region_base + region_size - 1;
>>>> +        *region_enabled &= fits;
>>>> +        if (vms->highmem_compact && !*region_enabled) {
>>>> +            continue;
>>>>            }
>>>>    
>>>> -        *region_enabled &= fits;
>>>>            base = region_base + region_size;
>>>> +        if (fits) {
>>>> +            vms->highest_gpa = region_base + region_size - 1;
>>>
>>> vms->highest_gpa = base - 1;
>>>
>>
>> It's personal taste actually. I was thinking of using 'base - 1', but
>> 'region_base + region_size - 1' looks more like a direct way. I don't
>> have strong sense though and lets use 'base - 1' in next respin.
> 
> I don't really have a preference for one or the other.
> 

Ok. Lets use 'base - 1' in next respin.

Thanks,
Gavin

Re: [PATCH v6 7/7] hw/arm/virt: Add properties to disable high memory regions

[Gavin Shan]

Hi Connie,

On 10/26/22 7:10 PM, Cornelia Huck wrote:
> On Wed, Oct 26 2022, Gavin Shan <gshan@redhat.com> wrote:
>>
>> On 10/25/22 6:54 PM, Cornelia Huck wrote:
>>> On Mon, Oct 24 2022, Gavin Shan <gshan@redhat.com> wrote:
>>>
>>>> These 3 high memory regions are usually enabled by default, but
>>>
>>> s/These 3/The/ ?
>>>
>>
>> Ok.
>>
>>>> they may be not used. For example, VIRT_HIGH_GIC_REDIST2 isn't
>>>> needed by GICv2. This leads to waste in the PA space.
>>>
>>> When building the command line, do we have enough information on when
>>> the regions provide something useful, and when they just waste space?
>>>
>>
>> I think the help messages are already indicative. For example, the help
>> messages for 'highmem-redist2' indicate the region is only needed by
>> GICv3 or GICv4. 'highmem-ecam' and 'highmem-mmio' are needed by PCI ECAM
>> and MMIO and the key words 'high' indicates they're the corresponding
>> second window.
>>
>> #./qemu-system-aarch64 -M virt,?
>> highmem-ecam=<bool>    - Set on/off to enable/disable high memory region for PCI ECAM
>> highmem-mmio=<bool>    - Set on/off to enable/disable high memory region for PCI MMIO
>> highmem-redists=<bool> - Set on/off to enable/disable high memory region for GICv3 or GICv4 redistributor
> 
> OK, hopefully this is enough for anyone building a command line
> directly. (Do we want to encourage management software like libvirt to
> switch off regions that are not needed?)
> 

Yeah, to disable those regions aren't needed will make libvirt smart.
I think it's worthy for libvirt to do it. I'm not sure if arbitrary
machine properties have been supported by libvirt or not. For example,
'highmem-redists' can still be turned off from 'vm1.xml' even though
it's not explicitly supported in libvirt's code.

>>
>>>>
>>>> Add properties to allow users selectively disable them if needed:
>>>> "highmem-redists", "highmem-ecam", "highmem-mmio".
>>>>
>>>> Suggested-by: Marc Zyngier <maz@kernel.org>
>>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>>> ---
>>>>    docs/system/arm/virt.rst | 12 ++++++++
>>>>    hw/arm/virt.c            | 64 ++++++++++++++++++++++++++++++++++++++++
>>>>    2 files changed, 76 insertions(+)
>>>>
>>>> diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
>>>> index 4454706392..a1668a969d 100644
>>>> --- a/docs/system/arm/virt.rst
>>>> +++ b/docs/system/arm/virt.rst
>>>> @@ -98,6 +98,18 @@ compact-highmem
>>>>      Set ``on``/``off`` to enable/disable the compact layout for high memory regions.
>>>>      The default is ``on`` for machine types later than ``virt-7.2``.
>>>>    
>>>> +highmem-redists
>>>> +  Set ``on``/``off`` to enable/disable the high memry region for GICv3/4
>>>
>>> s/memry/memory/
>>>
>>
>> Ok, copy-and-paste error. Will be fixed.
>>
>>>> +  redistributor. The default is ``on``.
>>>
>>> Do we need to add a note about what effects setting this to "off" may
>>> have, e.g. "Setting this to ``off`` may limit the maximum number of
>>> cpus." or so? And/or "Setting this to ``off`` when using GICv2 will save
>>> some space."?
>>>
>>
>> We may not mention GICv2 since GICv3/v4 are already mentioned. It's a
>> good idea to mention that the maximum number of CPUs is reduced when
>> it's turned off. I will have something like below in next respin if
>> you agree.
>>
>> highmem-redists
>>     Set ``on``/``off`` to enable/disable the high memroy region for GICv3 or
>>     GICv4 redistributor. The default is ``on``. Setting this to ``off`` will
>>     limit the maximum number of CPUs when GICv3 or GICv4 is used.
> 
> OK, sounds reasonable to me.
> 

Ok, Thanks for your confirm.

>>
>> Since 'vms->highmem_redists' is changeable, the 'virt_max_cpus' in
>> machvirt_init() needs to be recalculated based on that. The code change
>> will be included into next respin. Besides, the follow-up error message
>> will be improved to something like below.
>>
>>     error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
>>                  "supported by machine 'mach-virt' (%d). The high memory "
>>                  "region for GICv3 or GICv4 redistributor has been %s",
>>                  max_cpus, virt_max_cpus,
>>                  vms->highmem_redists ? "enabled" : "disabled");
> 
> Hm, the doc for error_report() states that "The resulting message should
> be a single phrase, with no newline or trailing punctuation." Maybe
> 
>      if (max_cpus > virt_max_cpus) {
>          error_report("Number of SMP CPUs requested (%d) exceeds max CPUs "
>                       "supported by machine 'mach-virt' (%d)",
>                       max_cpus, virt_max_cpus);
>          if (vms->gic_version != VIRT_GIC_VERSION_2 &&
>              !vms->higmem_redists) {
>              error_printf("Try 'highmem-redists=on' for more CPUs\n");
>          }
>          exit(1);
>      }
> 

Ok, I didn't notice the message raised by error_report() has this sort
of requirements. Your suggested snippet looks good to me and it will
be included in next respin. However, I would hold next respin (v7) for
a while to see if I can receive comments from Peter/Marc on v6 :)

Thanks,
Gavin

Re: [PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Peter Maydell]

On Wed, 26 Oct 2022 at 01:30, Gavin Shan <gshan@redhat.com> wrote:
>
> Hi Peter and Marc,
>
> On 10/24/22 11:54 AM, Gavin Shan wrote:
> > There are three high memory regions, which are VIRT_HIGH_REDIST2,
> > VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
> > are floating on highest RAM address. However, they can be disabled
> > in several cases.

> Could you help to take a look when getting a chance? I think Connie and
> Eric are close to complete the reviews, but v7 is still needed to address
> extra comments from them. I hope to make v7 mergeable if possible :)

If Eric and Connie and Marc are happy with it then I don't have
any further issues on top of that.

NB that since softfreeze is next Tuesday, this is going to be
8.0 material at this point, I'm afraid :-( (Softfreeze caught
me by surprise this cycle...)

-- PMM

Re: [PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Marc Zyngier]

On Wed, 26 Oct 2022 01:29:56 +0100,
Gavin Shan <gshan@redhat.com> wrote:
> 
> Hi Peter and Marc,
> 
> On 10/24/22 11:54 AM, Gavin Shan wrote:
> > There are three high memory regions, which are VIRT_HIGH_REDIST2,
> > VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
> > are floating on highest RAM address. However, they can be disabled
> > in several cases.
> > 
> > (1) One specific high memory region is disabled by developer by
> >      toggling vms->highmem_{redists, ecam, mmio}.
> > 
> > (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
> >      'virt-2.12' or ealier than it.
> > 
> > (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
> >      on 32-bits system.
> > 
> > (4) One specific high memory region is disabled when it breaks the
> >      PA space limit.
> > 
> > The current implementation of virt_set_memmap() isn't comprehensive
> > because the space for one specific high memory region is always
> > reserved from the PA space for case (1), (2) and (3). In the code,
> > 'base' and 'vms->highest_gpa' are always increased for those three
> > cases. It's unnecessary since the assigned space of the disabled
> > high memory region won't be used afterwards.
> > 
> > The series intends to improve the address assignment for these
> > high memory regions and introduces new properties for user to
> > selectively disable those 3 high memory regions.
> > 
> > PATCH[1-4] preparatory work for the improvment
> > PATCH[5]   improve high memory region address assignment
> > PATCH[6]   adds 'compact-highmem' to enable or disable the optimization
> > PATCH[7]   adds properties so that high memory regions can be disabled
> > 
> > v5: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00280.html
> > v4: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00067.html
> > v3: https://lists.nongnu.org/archive/html/qemu-arm/2022-09/msg00258.html
> > v2: https://lore.kernel.org/all/20220815062958.100366-1-gshan@redhat.com/T/
> > v1: https://lists.nongnu.org/archive/html/qemu-arm/2022-08/msg00013.html
> > 
> 
> Could you help to take a look when getting a chance? I think Connie and
> Eric are close to complete the reviews, but v7 is still needed to address
> extra comments from them. I hope to make v7 mergeable if possible :)

With the comments from Connie and Eric addressed, this looks good to
me:

Reviewed-by: Marc Zyngier <maz@kernel.org>

Thanks for having gone the extra mile on this one.

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Gavin Shan]

Hi Marc,

On 10/29/22 7:29 PM, Marc Zyngier wrote:
> On Wed, 26 Oct 2022 01:29:56 +0100,
> Gavin Shan <gshan@redhat.com> wrote:
>> On 10/24/22 11:54 AM, Gavin Shan wrote:
>>> There are three high memory regions, which are VIRT_HIGH_REDIST2,
>>> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
>>> are floating on highest RAM address. However, they can be disabled
>>> in several cases.
>>>
>>> (1) One specific high memory region is disabled by developer by
>>>       toggling vms->highmem_{redists, ecam, mmio}.
>>>
>>> (2) VIRT_HIGH_PCIE_ECAM region is disabled on machine, which is
>>>       'virt-2.12' or ealier than it.
>>>
>>> (3) VIRT_HIGH_PCIE_ECAM region is disabled when firmware is loaded
>>>       on 32-bits system.
>>>
>>> (4) One specific high memory region is disabled when it breaks the
>>>       PA space limit.
>>>
>>> The current implementation of virt_set_memmap() isn't comprehensive
>>> because the space for one specific high memory region is always
>>> reserved from the PA space for case (1), (2) and (3). In the code,
>>> 'base' and 'vms->highest_gpa' are always increased for those three
>>> cases. It's unnecessary since the assigned space of the disabled
>>> high memory region won't be used afterwards.
>>>
>>> The series intends to improve the address assignment for these
>>> high memory regions and introduces new properties for user to
>>> selectively disable those 3 high memory regions.
>>>
>>> PATCH[1-4] preparatory work for the improvment
>>> PATCH[5]   improve high memory region address assignment
>>> PATCH[6]   adds 'compact-highmem' to enable or disable the optimization
>>> PATCH[7]   adds properties so that high memory regions can be disabled
>>>
>>> v5: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00280.html
>>> v4: https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00067.html
>>> v3: https://lists.nongnu.org/archive/html/qemu-arm/2022-09/msg00258.html
>>> v2: https://lore.kernel.org/all/20220815062958.100366-1-gshan@redhat.com/T/
>>> v1: https://lists.nongnu.org/archive/html/qemu-arm/2022-08/msg00013.html
>>>
>>
>> Could you help to take a look when getting a chance? I think Connie and
>> Eric are close to complete the reviews, but v7 is still needed to address
>> extra comments from them. I hope to make v7 mergeable if possible :)
> 
> With the comments from Connie and Eric addressed, this looks good to
> me:
> 
> Reviewed-by: Marc Zyngier <maz@kernel.org>
> 
> Thanks for having gone the extra mile on this one.
> 

Thank you for your feedback and reviews. I've posted v7 with your r-b
after resolving comments from Connie/Eric.

https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00693.html (v7)

Thanks,
Gavin

Re: [PATCH v6 0/7] hw/arm/virt: Improve address assignment for high memory regions

[Gavin Shan]

Hi Peter,

On 10/29/22 2:06 AM, Peter Maydell wrote:
> On Wed, 26 Oct 2022 at 01:30, Gavin Shan <gshan@redhat.com> wrote:
>>
>> On 10/24/22 11:54 AM, Gavin Shan wrote:
>>> There are three high memory regions, which are VIRT_HIGH_REDIST2,
>>> VIRT_HIGH_PCIE_ECAM and VIRT_HIGH_PCIE_MMIO. Their base addresses
>>> are floating on highest RAM address. However, they can be disabled
>>> in several cases.
> 
>> Could you help to take a look when getting a chance? I think Connie and
>> Eric are close to complete the reviews, but v7 is still needed to address
>> extra comments from them. I hope to make v7 mergeable if possible :)
> 
> If Eric and Connie and Marc are happy with it then I don't have
> any further issues on top of that.
> 
> NB that since softfreeze is next Tuesday, this is going to be
> 8.0 material at this point, I'm afraid :-( (Softfreeze caught
> me by surprise this cycle...)
> 

Ok. v7 was just posted with Connie/Eric's comments resolved. Marc
also provided his r-b. Nothing is really critical since none of
the patches fixes an existing issue. It would be great if it can
be merged to 7.2 if we are fortunate enough. Otherwise, 8.0 is also
good to me :)

https://lists.nongnu.org/archive/html/qemu-arm/2022-10/msg00693.html (v7)

Thanks,
Gavin