[PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Julien Grall]

From: Julien Grall <jgrall@amazon.com>

Hi all,

At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
only use the first GB of memory.

This is because several devices cannot DMA above 1GB but Xen doesn't
necessarily allocate memory for Dom0 below 1GB.

This small series is trying to address the problem by allowing a
platform to restrict where Dom0 banks are allocated.

This is also a candidate for Xen 4.14. Without it, a user will not be
able to use all the RAM on the Raspberry Pi 4.

This series has only be slighlty tested. I would appreciate more test on
the Rasbperry Pi 4 to confirm this removing the restriction.

Cheers,

Cc: paul@xen.org

Julien Grall (3):
  xen/arm: Allow a platform to override the DMA width
  xen/arm: Take into account the DMA width when allocating Dom0 memory
    banks
  xen/arm: plat: Allocate as much as possible memory below 1GB for dom0
    for RPI

 xen/arch/arm/domain_build.c                | 32 +++++++++++++---------
 xen/arch/arm/platform.c                    |  5 ++++
 xen/arch/arm/platforms/brcm-raspberry-pi.c |  1 +
 xen/include/asm-arm/mm.h                   |  2 ++
 xen/include/asm-arm/numa.h                 |  5 ----
 xen/include/asm-arm/platform.h             |  2 ++
 6 files changed, 29 insertions(+), 18 deletions(-)

-- 
2.17.1

[PATCH for-4.14 1/3] xen/arm: Allow a platform to override the DMA width

[Julien Grall]

From: Julien Grall <jgrall@amazon.com>

At the moment, Xen is assuming that all the devices are at least 32-bit
DMA capable. However, some SoC have devices that may be able to access
a much restricted range. For instance, the RPI has devices that can
only access the first 1GB of RAM.

The structure platform_desc is now extended to allow a platform to
override the DMA width. The new is used to implement
arch_get_dma_bit_size().

The prototype is now moved in asm-arm/mm.h as the function is not NUMA
specific. The implementation is done in platform.c so we don't have to
include platform.h everywhere. This should be fine as the function is
not expected to be called in hotpath.

Signed-off-by: Julien Grall <jgrall@amazon.com>

---

Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: George Dunlap <george.dunlap@citrix.com>

I noticed that arch_get_dma_bit_size() is only called when there is more
than one NUMA node. I am a bit unsure what is the reason behind it.

The goal for Arm is to use arch_get_dma_bit_size() when deciding how low
the first Dom0 bank should be allocated.
---
 xen/arch/arm/platform.c        | 5 +++++
 xen/include/asm-arm/mm.h       | 2 ++
 xen/include/asm-arm/numa.h     | 5 -----
 xen/include/asm-arm/platform.h | 2 ++
 4 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/platform.c b/xen/arch/arm/platform.c
index 8eb0b6e57a5a..4db5bbb4c51d 100644
--- a/xen/arch/arm/platform.c
+++ b/xen/arch/arm/platform.c
@@ -155,6 +155,11 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)
     return (dt_match_node(blacklist, node) != NULL);
 }
 
+unsigned int arch_get_dma_bitsize(void)
+{
+    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h
index 7df91280bc77..f8ba49b1188f 100644
--- a/xen/include/asm-arm/mm.h
+++ b/xen/include/asm-arm/mm.h
@@ -366,6 +366,8 @@ int arch_acquire_resource(struct domain *d, unsigned int type, unsigned int id,
     return -EOPNOTSUPP;
 }
 
+unsigned int arch_get_dma_bitsize(void);
+
 #endif /*  __ARCH_ARM_MM__ */
 /*
  * Local variables:
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 490d1f31aa14..31a6de4e2346 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -25,11 +25,6 @@ extern mfn_t first_valid_mfn;
 #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
 #define __node_distance(a, b) (20)
 
-static inline unsigned int arch_get_dma_bitsize(void)
-{
-    return 32;
-}
-
 #endif /* __ARCH_ARM_NUMA_H */
 /*
  * Local variables:
diff --git a/xen/include/asm-arm/platform.h b/xen/include/asm-arm/platform.h
index ed4d30a1be7c..997eb2521631 100644
--- a/xen/include/asm-arm/platform.h
+++ b/xen/include/asm-arm/platform.h
@@ -38,6 +38,8 @@ struct platform_desc {
      * List of devices which must not pass-through to a guest
      */
     const struct dt_device_match *blacklist_dev;
+    /* Override the DMA width (32-bit by default). */
+    unsigned int dma_bitsize;
 };
 
 /*
-- 
2.17.1

[PATCH for-4.14 2/3] xen/arm: Take into account the DMA width when allocating Dom0 memory banks

[Julien Grall]

From: Julien Grall <jgrall@amazon.com>

At the moment, Xen is assuming that all the devices are at least 32-bit
DMA capable. However, some SoCs have devices that may be able to access
a much restricted range. For instance, the Raspberry PI 4 has devices
that can only access the first GB of RAM.

The function arch_get_dma_bit_size() will return the lowest DMA width on
the platform. Use it to decide what is the limit for the low memory.

Signed-off-by: Julien GralL <jgrall@amazon.com>
---
 xen/arch/arm/domain_build.c | 32 +++++++++++++++++++-------------
 1 file changed, 19 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 430708753642..abc4e463d27c 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -211,10 +211,13 @@ fail:
  *    the ramdisk and DTB must be placed within a certain proximity of
  *    the kernel within RAM.
  * 3. For dom0 we want to place as much of the RAM as we reasonably can
- *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit)
+ *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit).
  *    or when a device assigned to dom0 can only do 32-bit DMA access.
- * 4. For 32-bit dom0 the kernel must be located below 4GB.
- * 5. We want to have a few largers banks rather than many smaller ones.
+ * 4. Some devices assigned to dom0 can only do 32-bit DMA access or
+ *    even be more restricted. We want to allocate as much of the RAM
+ *    as we reasonably can that can be accessed from all the devices..
+ * 5. For 32-bit dom0 the kernel must be located below 4GB.
+ * 6. We want to have a few largers banks rather than many smaller ones.
  *
  * For the first two requirements we need to make sure that the lowest
  * bank is sufficiently large.
@@ -245,9 +248,9 @@ fail:
  * we give up.
  *
  * For 32-bit domain we require that the initial allocation for the
- * first bank is under 4G. For 64-bit domain, the first bank is preferred
- * to be allocated under 4G. Then for the subsequent allocations we
- * initially allocate memory only from below 4GB. Once that runs out
+ * first bank is part of the low mem. For 64-bit, the first bank is preferred
+ * to be allocated in the low mem. Then for subsequent allocation, we
+ * initially allocate memory only from low mem. Once that runs out out
  * (as described above) we allow higher allocations and continue until
  * that runs out (or we have allocated sufficient dom0 memory).
  */
@@ -262,6 +265,7 @@ static void __init allocate_memory_11(struct domain *d,
     int i;
 
     bool lowmem = true;
+    unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());
     unsigned int bits;
 
     /*
@@ -282,7 +286,7 @@ static void __init allocate_memory_11(struct domain *d,
      */
     while ( order >= min_low_order )
     {
-        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+        for ( bits = order ; bits <= lowmem_bitsize; bits++ )
         {
             pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
             if ( pg != NULL )
@@ -296,24 +300,26 @@ static void __init allocate_memory_11(struct domain *d,
         order--;
     }
 
-    /* Failed to allocate bank0 under 4GB */
+    /* Failed to allocate bank0 in the lowmem region. */
     if ( is_32bit_domain(d) )
         panic("Unable to allocate first memory bank\n");
 
-    /* Try to allocate memory from above 4GB */
-    printk(XENLOG_INFO "No bank has been allocated below 4GB.\n");
+    /* Try to allocate memory from above the lowmem region */
+    printk(XENLOG_INFO "No bank has been allocated below %u-bit.\n",
+           lowmem_bitsize);
     lowmem = false;
 
  got_bank0:
 
     /*
-     * If we failed to allocate bank0 under 4GB, continue allocating
-     * memory from above 4GB and fill in banks.
+     * If we failed to allocate bank0 in the lowmem region,
+     * continue allocating from above the lowmem and fill in banks.
      */
     order = get_allocation_size(kinfo->unassigned_mem);
     while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
     {
-        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0);
+        pg = alloc_domheap_pages(d, order,
+                                 lowmem ? MEMF_bits(lowmem_bitsize) : 0);
         if ( !pg )
         {
             order --;
-- 
2.17.1

[PATCH for-4.14 3/3] xen/arm: plat: Allocate as much as possible memory below 1GB for dom0 for RPI

[Julien Grall]

From: Julien Grall <jgrall@amazon.com>

The raspberry PI 4 has devices that can only DMA into the first GB of
the RAM. Therefore we want allocate as much as possible memory below 1GB
for dom0.

Use the recently introduced dma_bitsize field to specify the DMA width
supported.

Signed-off-by: Julien Grall <jgrall@amazon.com>
Reported-by: Corey Minyard <minyard@acm.org>
---
 xen/arch/arm/platforms/brcm-raspberry-pi.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/xen/arch/arm/platforms/brcm-raspberry-pi.c b/xen/arch/arm/platforms/brcm-raspberry-pi.c
index b697fa2c6c0e..ad5483437b31 100644
--- a/xen/arch/arm/platforms/brcm-raspberry-pi.c
+++ b/xen/arch/arm/platforms/brcm-raspberry-pi.c
@@ -43,6 +43,7 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
 PLATFORM_START(rpi4, "Raspberry Pi 4")
     .compatible     = rpi4_dt_compat,
     .blacklist_dev  = rpi4_blacklist_dev,
+    .dma_bitsize    = 10,
 PLATFORM_END
 
 /*
-- 
2.17.1

Re: [PATCH for-4.14 1/3] xen/arm: Allow a platform to override the DMA width

[Volodymyr Babchuk]

Hi Julien,

On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
> From: Julien Grall <jgrall@amazon.com>
> 
> At the moment, Xen is assuming that all the devices are at least 32-bit
> DMA capable. However, some SoC have devices that may be able to access
> a much restricted range. For instance, the RPI has devices that can
> only access the first 1GB of RAM.
> 
> The structure platform_desc is now extended to allow a platform to
> override the DMA width. The new is used to implement
> arch_get_dma_bit_size().
> 
> The prototype is now moved in asm-arm/mm.h as the function is not NUMA
> specific. The implementation is done in platform.c so we don't have to
> include platform.h everywhere. This should be fine as the function is
> not expected to be called in hotpath.
> 
> Signed-off-by: Julien Grall <jgrall@amazon.com>
Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>

> ---
> 
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> Cc: George Dunlap <george.dunlap@citrix.com>
> 
> I noticed that arch_get_dma_bit_size() is only called when there is more
> than one NUMA node. I am a bit unsure what is the reason behind it.
> 
> The goal for Arm is to use arch_get_dma_bit_size() when deciding how low
> the first Dom0 bank should be allocated.
> ---
>  xen/arch/arm/platform.c        | 5 +++++
>  xen/include/asm-arm/mm.h       | 2 ++
>  xen/include/asm-arm/numa.h     | 5 -----
>  xen/include/asm-arm/platform.h | 2 ++
>  4 files changed, 9 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/platform.c b/xen/arch/arm/platform.c
> index 8eb0b6e57a5a..4db5bbb4c51d 100644
> --- a/xen/arch/arm/platform.c
> +++ b/xen/arch/arm/platform.c
> @@ -155,6 +155,11 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)
>      return (dt_match_node(blacklist, node) != NULL);
>  }
>  
> +unsigned int arch_get_dma_bitsize(void)
> +{
> +    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h
> index 7df91280bc77..f8ba49b1188f 100644
> --- a/xen/include/asm-arm/mm.h
> +++ b/xen/include/asm-arm/mm.h
> @@ -366,6 +366,8 @@ int arch_acquire_resource(struct domain *d, unsigned int type, unsigned int id,
>      return -EOPNOTSUPP;
>  }
>  
> +unsigned int arch_get_dma_bitsize(void);
> +
>  #endif /*  __ARCH_ARM_MM__ */
>  /*
>   * Local variables:
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 490d1f31aa14..31a6de4e2346 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -25,11 +25,6 @@ extern mfn_t first_valid_mfn;
>  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
>  #define __node_distance(a, b) (20)
>  
> -static inline unsigned int arch_get_dma_bitsize(void)
> -{
> -    return 32;
> -}
> -
>  #endif /* __ARCH_ARM_NUMA_H */
>  /*
>   * Local variables:
> diff --git a/xen/include/asm-arm/platform.h b/xen/include/asm-arm/platform.h
> index ed4d30a1be7c..997eb2521631 100644
> --- a/xen/include/asm-arm/platform.h
> +++ b/xen/include/asm-arm/platform.h
> @@ -38,6 +38,8 @@ struct platform_desc {
>       * List of devices which must not pass-through to a guest
>       */
>      const struct dt_device_match *blacklist_dev;
> +    /* Override the DMA width (32-bit by default). */
> +    unsigned int dma_bitsize;
>  };
>  
>  /*

Re: [PATCH for-4.14 2/3] xen/arm: Take into account the DMA width when allocating Dom0 memory banks

[Volodymyr Babchuk]

Hi Julien,

On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
> From: Julien Grall <jgrall@amazon.com>
> 
> At the moment, Xen is assuming that all the devices are at least 32-bit
> DMA capable. However, some SoCs have devices that may be able to access
> a much restricted range. For instance, the Raspberry PI 4 has devices
> that can only access the first GB of RAM.
> 
> The function arch_get_dma_bit_size() will return the lowest DMA width on
> the platform. Use it to decide what is the limit for the low memory.
> 
> Signed-off-by: Julien GralL <jgrall@amazon.com>
> ---
>  xen/arch/arm/domain_build.c | 32 +++++++++++++++++++-------------
>  1 file changed, 19 insertions(+), 13 deletions(-)
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 430708753642..abc4e463d27c 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -211,10 +211,13 @@ fail:
>   *    the ramdisk and DTB must be placed within a certain proximity of
>   *    the kernel within RAM.
>   * 3. For dom0 we want to place as much of the RAM as we reasonably can
> - *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit)
> + *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit).
Is full stop really needed there?

>   *    or when a device assigned to dom0 can only do 32-bit DMA access.
> - * 4. For 32-bit dom0 the kernel must be located below 4GB.
> - * 5. We want to have a few largers banks rather than many smaller ones.
> + * 4. Some devices assigned to dom0 can only do 32-bit DMA access or
> + *    even be more restricted. We want to allocate as much of the RAM
> + *    as we reasonably can that can be accessed from all the devices..
> + * 5. For 32-bit dom0 the kernel must be located below 4GB.
> + * 6. We want to have a few largers banks rather than many smaller ones.
>   *
>   * For the first two requirements we need to make sure that the lowest
>   * bank is sufficiently large.
> @@ -245,9 +248,9 @@ fail:
>   * we give up.
>   *
>   * For 32-bit domain we require that the initial allocation for the
> - * first bank is under 4G. For 64-bit domain, the first bank is preferred
> - * to be allocated under 4G. Then for the subsequent allocations we
> - * initially allocate memory only from below 4GB. Once that runs out
> + * first bank is part of the low mem. For 64-bit, the first bank is preferred
> + * to be allocated in the low mem. Then for subsequent allocation, we
> + * initially allocate memory only from low mem. Once that runs out out
>   * (as described above) we allow higher allocations and continue until
>   * that runs out (or we have allocated sufficient dom0 memory).
>   */
> @@ -262,6 +265,7 @@ static void __init allocate_memory_11(struct domain *d,
>      int i;
>  
>      bool lowmem = true;
> +    unsigned int lowmem_bitsize = min(32U, arch_get_dma_bitsize());
>      unsigned int bits;
>  
>      /*
> @@ -282,7 +286,7 @@ static void __init allocate_memory_11(struct domain *d,
>       */
>      while ( order >= min_low_order )
>      {
> -        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
> +        for ( bits = order ; bits <= lowmem_bitsize; bits++ )
>          {
>              pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
>              if ( pg != NULL )
> @@ -296,24 +300,26 @@ static void __init allocate_memory_11(struct domain *d,
>          order--;
>      }
>  
> -    /* Failed to allocate bank0 under 4GB */
> +    /* Failed to allocate bank0 in the lowmem region. */
>      if ( is_32bit_domain(d) )
>          panic("Unable to allocate first memory bank\n");
>  
> -    /* Try to allocate memory from above 4GB */
> -    printk(XENLOG_INFO "No bank has been allocated below 4GB.\n");
> +    /* Try to allocate memory from above the lowmem region */
> +    printk(XENLOG_INFO "No bank has been allocated below %u-bit.\n",
> +           lowmem_bitsize);
>      lowmem = false;
>  
>   got_bank0:
>  
>      /*
> -     * If we failed to allocate bank0 under 4GB, continue allocating
> -     * memory from above 4GB and fill in banks.
> +     * If we failed to allocate bank0 in the lowmem region,
> +     * continue allocating from above the lowmem and fill in banks.
>       */
>      order = get_allocation_size(kinfo->unassigned_mem);
>      while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
>      {
> -        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0);
> +        pg = alloc_domheap_pages(d, order,
> +                                 lowmem ? MEMF_bits(lowmem_bitsize) : 0);
>          if ( !pg )
>          {
>              order --;

Re: [PATCH for-4.14 3/3] xen/arm: plat: Allocate as much as possible memory below 1GB for dom0 for RPI

[Volodymyr Babchuk]

Hi Julien,

On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
> From: Julien Grall <jgrall@amazon.com>
> 
> The raspberry PI 4 has devices that can only DMA into the first GB of
> the RAM. Therefore we want allocate as much as possible memory below 1GB
> for dom0.
> 
> Use the recently introduced dma_bitsize field to specify the DMA width
> supported.
> 
> Signed-off-by: Julien Grall <jgrall@amazon.com>
> Reported-by: Corey Minyard <minyard@acm.org>
> ---
>  xen/arch/arm/platforms/brcm-raspberry-pi.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/xen/arch/arm/platforms/brcm-raspberry-pi.c b/xen/arch/arm/platforms/brcm-raspberry-pi.c
> index b697fa2c6c0e..ad5483437b31 100644
> --- a/xen/arch/arm/platforms/brcm-raspberry-pi.c
> +++ b/xen/arch/arm/platforms/brcm-raspberry-pi.c
> @@ -43,6 +43,7 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
>  PLATFORM_START(rpi4, "Raspberry Pi 4")
>      .compatible     = rpi4_dt_compat,
>      .blacklist_dev  = rpi4_blacklist_dev,
> +    .dma_bitsize    = 10,

I'm confused. Should it be 30?

>  PLATFORM_END
>  
>  /*

Re: [PATCH for-4.14 3/3] xen/arm: plat: Allocate as much as possible memory below 1GB for dom0 for RPI

[Corey Minyard]

On Mon, May 18, 2020 at 08:36:08PM +0000, Volodymyr Babchuk wrote:
> Hi Julien,
> 
> On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
> > From: Julien Grall <jgrall@amazon.com>
> > 
> > The raspberry PI 4 has devices that can only DMA into the first GB of
> > the RAM. Therefore we want allocate as much as possible memory below 1GB
> > for dom0.
> > 
> > Use the recently introduced dma_bitsize field to specify the DMA width
> > supported.
> > 
> > Signed-off-by: Julien Grall <jgrall@amazon.com>
> > Reported-by: Corey Minyard <minyard@acm.org>
> > ---
> >  xen/arch/arm/platforms/brcm-raspberry-pi.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/xen/arch/arm/platforms/brcm-raspberry-pi.c b/xen/arch/arm/platforms/brcm-raspberry-pi.c
> > index b697fa2c6c0e..ad5483437b31 100644
> > --- a/xen/arch/arm/platforms/brcm-raspberry-pi.c
> > +++ b/xen/arch/arm/platforms/brcm-raspberry-pi.c
> > @@ -43,6 +43,7 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
> >  PLATFORM_START(rpi4, "Raspberry Pi 4")
> >      .compatible     = rpi4_dt_compat,
> >      .blacklist_dev  = rpi4_blacklist_dev,
> > +    .dma_bitsize    = 10,
> 
> I'm confused. Should it be 30?

Indeed it should.  I just tested this series, and Linux fails to boot
with this set to 10.  With it set to 30 it works.

With this set to 30, you can have a:

Tested-by: Corey Minyard <cminyard@mvista.com>

for all three patches.

-corey

> 
> >  PLATFORM_END
> >  
> >  /*

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Tamas K Lengyel]

On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>
> From: Julien Grall <jgrall@amazon.com>
>
> Hi all,
>
> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> only use the first GB of memory.
>
> This is because several devices cannot DMA above 1GB but Xen doesn't
> necessarily allocate memory for Dom0 below 1GB.
>
> This small series is trying to address the problem by allowing a
> platform to restrict where Dom0 banks are allocated.
>
> This is also a candidate for Xen 4.14. Without it, a user will not be
> able to use all the RAM on the Raspberry Pi 4.
>
> This series has only be slighlty tested. I would appreciate more test on
> the Rasbperry Pi 4 to confirm this removing the restriction.

Hi Julien,
could you post a git branch somewhere? I can try this on my rpi4 that
already runs 4.13.

Thanks,
Tamas

Re: [PATCH for-4.14 2/3] xen/arm: Take into account the DMA width when allocating Dom0 memory banks

[Julien Grall]

On 18/05/2020 21:34, Volodymyr Babchuk wrote:
> Hi Julien,

Hi Volodymyr,

Thank you for the review.

> 
> On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
>> From: Julien Grall <jgrall@amazon.com>
>>
>> At the moment, Xen is assuming that all the devices are at least 32-bit
>> DMA capable. However, some SoCs have devices that may be able to access
>> a much restricted range. For instance, the Raspberry PI 4 has devices
>> that can only access the first GB of RAM.
>>
>> The function arch_get_dma_bit_size() will return the lowest DMA width on
>> the platform. Use it to decide what is the limit for the low memory.
>>
>> Signed-off-by: Julien GralL <jgrall@amazon.com>
>> ---
>>   xen/arch/arm/domain_build.c | 32 +++++++++++++++++++-------------
>>   1 file changed, 19 insertions(+), 13 deletions(-)
>>
>> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
>> index 430708753642..abc4e463d27c 100644
>> --- a/xen/arch/arm/domain_build.c
>> +++ b/xen/arch/arm/domain_build.c
>> @@ -211,10 +211,13 @@ fail:
>>    *    the ramdisk and DTB must be placed within a certain proximity of
>>    *    the kernel within RAM.
>>    * 3. For dom0 we want to place as much of the RAM as we reasonably can
>> - *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit)
>> + *    below 4GB, so that it can be used by non-LPAE enabled kernels (32-bit).
> Is full stop really needed there?

I was meant to remove the line below as it is now part of 4). I will 
remove it in the next version.

Best regards,

-- 
Julien Grall

Re: [PATCH for-4.14 3/3] xen/arm: plat: Allocate as much as possible memory below 1GB for dom0 for RPI

[Julien Grall]

On 18/05/2020 21:36, Volodymyr Babchuk wrote:
> Hi Julien,

Hi,


> On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
>> From: Julien Grall <jgrall@amazon.com>
>>
>> The raspberry PI 4 has devices that can only DMA into the first GB of
>> the RAM. Therefore we want allocate as much as possible memory below 1GB
>> for dom0.
>>
>> Use the recently introduced dma_bitsize field to specify the DMA width
>> supported.
>>
>> Signed-off-by: Julien Grall <jgrall@amazon.com>
>> Reported-by: Corey Minyard <minyard@acm.org>
>> ---
>>   xen/arch/arm/platforms/brcm-raspberry-pi.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/xen/arch/arm/platforms/brcm-raspberry-pi.c b/xen/arch/arm/platforms/brcm-raspberry-pi.c
>> index b697fa2c6c0e..ad5483437b31 100644
>> --- a/xen/arch/arm/platforms/brcm-raspberry-pi.c
>> +++ b/xen/arch/arm/platforms/brcm-raspberry-pi.c
>> @@ -43,6 +43,7 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
>>   PLATFORM_START(rpi4, "Raspberry Pi 4")
>>       .compatible     = rpi4_dt_compat,
>>       .blacklist_dev  = rpi4_blacklist_dev,
>> +    .dma_bitsize    = 10,
> 
> I'm confused. Should it be 30?

Argh, yes. I computed the number of bits for 1024 and forgot to add 20 :(.

I will fix it in the next revision.

Cheers,

-- 
Julien Grall

Re: [PATCH for-4.14 3/3] xen/arm: plat: Allocate as much as possible memory below 1GB for dom0 for RPI

[Julien Grall]

Hi Corey,

On 19/05/2020 01:02, Corey Minyard wrote:
> On Mon, May 18, 2020 at 08:36:08PM +0000, Volodymyr Babchuk wrote:
>> Hi Julien,
>>
>> On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
>>> From: Julien Grall <jgrall@amazon.com>
>>>
>>> The raspberry PI 4 has devices that can only DMA into the first GB of
>>> the RAM. Therefore we want allocate as much as possible memory below 1GB
>>> for dom0.
>>>
>>> Use the recently introduced dma_bitsize field to specify the DMA width
>>> supported.
>>>
>>> Signed-off-by: Julien Grall <jgrall@amazon.com>
>>> Reported-by: Corey Minyard <minyard@acm.org>
>>> ---
>>>   xen/arch/arm/platforms/brcm-raspberry-pi.c | 1 +
>>>   1 file changed, 1 insertion(+)
>>>
>>> diff --git a/xen/arch/arm/platforms/brcm-raspberry-pi.c b/xen/arch/arm/platforms/brcm-raspberry-pi.c
>>> index b697fa2c6c0e..ad5483437b31 100644
>>> --- a/xen/arch/arm/platforms/brcm-raspberry-pi.c
>>> +++ b/xen/arch/arm/platforms/brcm-raspberry-pi.c
>>> @@ -43,6 +43,7 @@ static const struct dt_device_match rpi4_blacklist_dev[] __initconst =
>>>   PLATFORM_START(rpi4, "Raspberry Pi 4")
>>>       .compatible     = rpi4_dt_compat,
>>>       .blacklist_dev  = rpi4_blacklist_dev,
>>> +    .dma_bitsize    = 10,
>>
>> I'm confused. Should it be 30?
> 
> Indeed it should.  I just tested this series, and Linux fails to boot
> with this set to 10.  With it set to 30 it works.
> 
> With this set to 30, you can have a:
> 
> Tested-by: Corey Minyard <cminyard@mvista.com>
> 
> for all three patches.

Thank you for the testing! I will fix the bug and resend the series.

Cheers,

-- 
Julien Grall

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Julien Grall]

On 19/05/2020 04:08, Tamas K Lengyel wrote:
> On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>>
>> From: Julien Grall <jgrall@amazon.com>
>>
>> Hi all,
>>
>> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> only use the first GB of memory.
>>
>> This is because several devices cannot DMA above 1GB but Xen doesn't
>> necessarily allocate memory for Dom0 below 1GB.
>>
>> This small series is trying to address the problem by allowing a
>> platform to restrict where Dom0 banks are allocated.
>>
>> This is also a candidate for Xen 4.14. Without it, a user will not be
>> able to use all the RAM on the Raspberry Pi 4.
>>
>> This series has only be slighlty tested. I would appreciate more test on
>> the Rasbperry Pi 4 to confirm this removing the restriction.
> 
> Hi Julien,

Hi,

> could you post a git branch somewhere? I can try this on my rpi4 that
> already runs 4.13.

I have pushed a branch based on unstable and the v2 of the series:

git://xenbits.xen.org/people/julieng/xen-unstable.git

branch arm-dma/v2

Thank you in advance for the testing!

Cheers,

-- 
Julien Grall

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Tamas K Lengyel]

On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
>
>
>
> On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> >>
> >> From: Julien Grall <jgrall@amazon.com>
> >>
> >> Hi all,
> >>
> >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> >> only use the first GB of memory.
> >>
> >> This is because several devices cannot DMA above 1GB but Xen doesn't
> >> necessarily allocate memory for Dom0 below 1GB.
> >>
> >> This small series is trying to address the problem by allowing a
> >> platform to restrict where Dom0 banks are allocated.
> >>
> >> This is also a candidate for Xen 4.14. Without it, a user will not be
> >> able to use all the RAM on the Raspberry Pi 4.
> >>
> >> This series has only be slighlty tested. I would appreciate more test on
> >> the Rasbperry Pi 4 to confirm this removing the restriction.
> >
> > Hi Julien,
>
> Hi,
>
> > could you post a git branch somewhere? I can try this on my rpi4 that
> > already runs 4.13.
>
> I have pushed a branch based on unstable and the v2 of the series:
>
> git://xenbits.xen.org/people/julieng/xen-unstable.git
>
> branch arm-dma/v2
>

I've updated my image I built with
https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
defined 2048m as total_mem and Xen seems to be booting fine and passes
execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
it was working. When I increased the mem for dom0 the boot is now
stuck at:

[    1.427788] of_cfs_init
[    1.429667] of_cfs_init: OK
[    1.432561] clk: Not disabling unused clocks
[    1.437239] Waiting for root device /dev/mmcblk0p2...
[    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.546902] random: fast init done
[    1.564590] mmc1: new high speed SDIO card at address 0001

Could this be because the DTB I compiled from a fresh checkout of
https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
kernel itself is from a checkout ~5 months ago? I guess that must be
the cause because even if I decrease the dom0_mem to 512m it still
gets stuck at the same spot whereas it was booting fine before.

Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Stefano Stabellini]

On Tue, 19 May 2020, Tamas K Lengyel wrote:
> On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> >
> >
> >
> > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > >>
> > >> From: Julien Grall <jgrall@amazon.com>
> > >>
> > >> Hi all,
> > >>
> > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > >> only use the first GB of memory.
> > >>
> > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > >> necessarily allocate memory for Dom0 below 1GB.
> > >>
> > >> This small series is trying to address the problem by allowing a
> > >> platform to restrict where Dom0 banks are allocated.
> > >>
> > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > >> able to use all the RAM on the Raspberry Pi 4.
> > >>
> > >> This series has only be slighlty tested. I would appreciate more test on
> > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > >
> > > Hi Julien,
> >
> > Hi,
> >
> > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > already runs 4.13.
> >
> > I have pushed a branch based on unstable and the v2 of the series:
> >
> > git://xenbits.xen.org/people/julieng/xen-unstable.git
> >
> > branch arm-dma/v2
> >
> 
> I've updated my image I built with
> https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> defined 2048m as total_mem and Xen seems to be booting fine and passes
> execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> it was working. When I increased the mem for dom0 the boot is now
> stuck at:
> 
> [    1.427788] of_cfs_init
> [    1.429667] of_cfs_init: OK
> [    1.432561] clk: Not disabling unused clocks
> [    1.437239] Waiting for root device /dev/mmcblk0p2...
> [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.546902] random: fast init done
> [    1.564590] mmc1: new high speed SDIO card at address 0001
> 
> Could this be because the DTB I compiled from a fresh checkout of
> https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> kernel itself is from a checkout ~5 months ago? I guess that must be
> the cause because even if I decrease the dom0_mem to 512m it still
> gets stuck at the same spot whereas it was booting fine before.

Just so that you are aware, there is a known issue with setting dom0_mem
greater than 512M. I have a WIP patch to fix it in Linux that I plan to
send to the list soon.

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Roman Shaposhnik]

On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> >
> >
> >
> > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > >>
> > >> From: Julien Grall <jgrall@amazon.com>
> > >>
> > >> Hi all,
> > >>
> > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > >> only use the first GB of memory.
> > >>
> > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > >> necessarily allocate memory for Dom0 below 1GB.
> > >>
> > >> This small series is trying to address the problem by allowing a
> > >> platform to restrict where Dom0 banks are allocated.
> > >>
> > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > >> able to use all the RAM on the Raspberry Pi 4.
> > >>
> > >> This series has only be slighlty tested. I would appreciate more test on
> > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > >
> > > Hi Julien,
> >
> > Hi,
> >
> > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > already runs 4.13.
> >
> > I have pushed a branch based on unstable and the v2 of the series:
> >
> > git://xenbits.xen.org/people/julieng/xen-unstable.git
> >
> > branch arm-dma/v2
> >
>
> I've updated my image I built with
> https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> defined 2048m as total_mem and Xen seems to be booting fine and passes
> execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> it was working. When I increased the mem for dom0 the boot is now
> stuck at:
>
> [    1.427788] of_cfs_init
> [    1.429667] of_cfs_init: OK
> [    1.432561] clk: Not disabling unused clocks
> [    1.437239] Waiting for root device /dev/mmcblk0p2...
> [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.546902] random: fast init done
> [    1.564590] mmc1: new high speed SDIO card at address 0001
>
> Could this be because the DTB I compiled from a fresh checkout of
> https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> kernel itself is from a checkout ~5 months ago? I guess that must be
> the cause because even if I decrease the dom0_mem to 512m it still
> gets stuck at the same spot whereas it was booting fine before.

Stefano and I are testing the fix right now -- for now just set your
Dom0 mem to less than 512m.

Thanks,
Roman.

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Tamas K Lengyel]

On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
> <tamas.k.lengyel@gmail.com> wrote:
> >
> > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> > >
> > >
> > >
> > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > > >>
> > > >> From: Julien Grall <jgrall@amazon.com>
> > > >>
> > > >> Hi all,
> > > >>
> > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > > >> only use the first GB of memory.
> > > >>
> > > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > > >> necessarily allocate memory for Dom0 below 1GB.
> > > >>
> > > >> This small series is trying to address the problem by allowing a
> > > >> platform to restrict where Dom0 banks are allocated.
> > > >>
> > > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > > >> able to use all the RAM on the Raspberry Pi 4.
> > > >>
> > > >> This series has only be slighlty tested. I would appreciate more test on
> > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > > >
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > > already runs 4.13.
> > >
> > > I have pushed a branch based on unstable and the v2 of the series:
> > >
> > > git://xenbits.xen.org/people/julieng/xen-unstable.git
> > >
> > > branch arm-dma/v2
> > >
> >
> > I've updated my image I built with
> > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> > defined 2048m as total_mem and Xen seems to be booting fine and passes
> > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> > it was working. When I increased the mem for dom0 the boot is now
> > stuck at:
> >
> > [    1.427788] of_cfs_init
> > [    1.429667] of_cfs_init: OK
> > [    1.432561] clk: Not disabling unused clocks
> > [    1.437239] Waiting for root device /dev/mmcblk0p2...
> > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.546902] random: fast init done
> > [    1.564590] mmc1: new high speed SDIO card at address 0001
> >
> > Could this be because the DTB I compiled from a fresh checkout of
> > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> > kernel itself is from a checkout ~5 months ago? I guess that must be
> > the cause because even if I decrease the dom0_mem to 512m it still
> > gets stuck at the same spot whereas it was booting fine before.
>
> Stefano and I are testing the fix right now -- for now just set your
> Dom0 mem to less than 512m.

Actually seems to work after I recompiled the kernel and reinstalled
all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:

xl info:
...
total_memory           : 3956
free_memory            : 1842

cat /proc/meminfo
MemTotal:        1963844 kB

I get an emergency shell during boot on the console complaining about
xenbr0 not coming up but if I just hit continue it boots fine and the
network is up. So AFAICT things are good.

Cheers,
Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Roman Shaposhnik]

[-- Attachment #1: Type: text/plain, Size: 3812 bytes --]

On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com>
wrote:

> On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
> >
> > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
> > <tamas.k.lengyel@gmail.com> wrote:
> > >
> > > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> > > >
> > > >
> > > >
> > > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org>
> wrote:
> > > > >>
> > > > >> From: Julien Grall <jgrall@amazon.com>
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4
> can
> > > > >> only use the first GB of memory.
> > > > >>
> > > > >> This is because several devices cannot DMA above 1GB but Xen
> doesn't
> > > > >> necessarily allocate memory for Dom0 below 1GB.
> > > > >>
> > > > >> This small series is trying to address the problem by allowing a
> > > > >> platform to restrict where Dom0 banks are allocated.
> > > > >>
> > > > >> This is also a candidate for Xen 4.14. Without it, a user will
> not be
> > > > >> able to use all the RAM on the Raspberry Pi 4.
> > > > >>
> > > > >> This series has only be slighlty tested. I would appreciate more
> test on
> > > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > > > >
> > > > > Hi Julien,
> > > >
> > > > Hi,
> > > >
> > > > > could you post a git branch somewhere? I can try this on my rpi4
> that
> > > > > already runs 4.13.
> > > >
> > > > I have pushed a branch based on unstable and the v2 of the series:
> > > >
> > > > git://xenbits.xen.org/people/julieng/xen-unstable.git
> > > >
> > > > branch arm-dma/v2
> > > >
> > >
> > > I've updated my image I built with
> > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> > > defined 2048m as total_mem and Xen seems to be booting fine and passes
> > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> > > it was working. When I increased the mem for dom0 the boot is now
> > > stuck at:
> > >
> > > [    1.427788] of_cfs_init
> > > [    1.429667] of_cfs_init: OK
> > > [    1.432561] clk: Not disabling unused clocks
> > > [    1.437239] Waiting for root device /dev/mmcblk0p2...
> > > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> > > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> > > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.546902] random: fast init done
> > > [    1.564590] mmc1: new high speed SDIO card at address 0001
> > >
> > > Could this be because the DTB I compiled from a fresh checkout of
> > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> > > kernel itself is from a checkout ~5 months ago? I guess that must be
> > > the cause because even if I decrease the dom0_mem to 512m it still
> > > gets stuck at the same spot whereas it was booting fine before.
> >
> > Stefano and I are testing the fix right now -- for now just set your
> > Dom0 mem to less than 512m.
>
> Actually seems to work after I recompiled the kernel and reinstalled
> all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:
>
> xl info:
> ...
> total_memory           : 3956
> free_memory            : 1842
>
> cat /proc/meminfo
> MemTotal:        1963844 kB
>
> I get an emergency shell during boot on the console complaining about
> xenbr0 not coming up but if I just hit continue it boots fine and the
> network is up. So AFAICT things are good.
>

What exact version of the kernel are you using and what did you build it
from?

FWIW: 5.6.x clearly has an issue with DMA.

Thanks,
Roman.

>

[-- Attachment #2: Type: text/html, Size: 6071 bytes --]

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Tamas K Lengyel]

On Tue, May 19, 2020 at 8:28 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:
>>
>> On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
>> >
>> > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
>> > <tamas.k.lengyel@gmail.com> wrote:
>> > >
>> > > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
>> > > >
>> > > >
>> > > >
>> > > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
>> > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>> > > > >>
>> > > > >> From: Julien Grall <jgrall@amazon.com>
>> > > > >>
>> > > > >> Hi all,
>> > > > >>
>> > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> > > > >> only use the first GB of memory.
>> > > > >>
>> > > > >> This is because several devices cannot DMA above 1GB but Xen doesn't
>> > > > >> necessarily allocate memory for Dom0 below 1GB.
>> > > > >>
>> > > > >> This small series is trying to address the problem by allowing a
>> > > > >> platform to restrict where Dom0 banks are allocated.
>> > > > >>
>> > > > >> This is also a candidate for Xen 4.14. Without it, a user will not be
>> > > > >> able to use all the RAM on the Raspberry Pi 4.
>> > > > >>
>> > > > >> This series has only be slighlty tested. I would appreciate more test on
>> > > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
>> > > > >
>> > > > > Hi Julien,
>> > > >
>> > > > Hi,
>> > > >
>> > > > > could you post a git branch somewhere? I can try this on my rpi4 that
>> > > > > already runs 4.13.
>> > > >
>> > > > I have pushed a branch based on unstable and the v2 of the series:
>> > > >
>> > > > git://xenbits.xen.org/people/julieng/xen-unstable.git
>> > > >
>> > > > branch arm-dma/v2
>> > > >
>> > >
>> > > I've updated my image I built with
>> > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
>> > > defined 2048m as total_mem and Xen seems to be booting fine and passes
>> > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
>> > > it was working. When I increased the mem for dom0 the boot is now
>> > > stuck at:
>> > >
>> > > [    1.427788] of_cfs_init
>> > > [    1.429667] of_cfs_init: OK
>> > > [    1.432561] clk: Not disabling unused clocks
>> > > [    1.437239] Waiting for root device /dev/mmcblk0p2...
>> > > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
>> > > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
>> > > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.546902] random: fast init done
>> > > [    1.564590] mmc1: new high speed SDIO card at address 0001
>> > >
>> > > Could this be because the DTB I compiled from a fresh checkout of
>> > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
>> > > kernel itself is from a checkout ~5 months ago? I guess that must be
>> > > the cause because even if I decrease the dom0_mem to 512m it still
>> > > gets stuck at the same spot whereas it was booting fine before.
>> >
>> > Stefano and I are testing the fix right now -- for now just set your
>> > Dom0 mem to less than 512m.
>>
>> Actually seems to work after I recompiled the kernel and reinstalled
>> all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:
>>
>> xl info:
>> ...
>> total_memory           : 3956
>> free_memory            : 1842
>>
>> cat /proc/meminfo
>> MemTotal:        1963844 kB
>>
>> I get an emergency shell during boot on the console complaining about
>> xenbr0 not coming up but if I just hit continue it boots fine and the
>> network is up. So AFAICT things are good.
>
>
> What exact version of the kernel are you using and what did you build it from?
>
> FWIW: 5.6.x clearly has an issue with DMA.

As I said above: https://github.com/raspberrypi/linux.git branch
rpi-4.19.y, I applied the Linux patches from the xen-rpi4-builder
repo, just changing the dom0_mem option in patch 1. I reverted the
xen-rpi4-builder a couple revisions as to not build using the DTB
overlay.

Tamas

Re: [PATCH for-4.14 1/3] xen/arm: Allow a platform to override the DMA width

[Stefano Stabellini]

On Mon, 18 May 2020, Volodymyr Babchuk wrote:
> Hi Julien,
> 
> On Mon, 2020-05-18 at 12:30 +0100, Julien Grall wrote:
> > From: Julien Grall <jgrall@amazon.com>
> > 
> > At the moment, Xen is assuming that all the devices are at least 32-bit
> > DMA capable. However, some SoC have devices that may be able to access
> > a much restricted range. For instance, the RPI has devices that can
> > only access the first 1GB of RAM.
> > 
> > The structure platform_desc is now extended to allow a platform to
> > override the DMA width. The new is used to implement
> > arch_get_dma_bit_size().
> > 
> > The prototype is now moved in asm-arm/mm.h as the function is not NUMA
> > specific. The implementation is done in platform.c so we don't have to
> > include platform.h everywhere. This should be fine as the function is
> > not expected to be called in hotpath.
> > 
> > Signed-off-by: Julien Grall <jgrall@amazon.com>
> Reviewed-by: Volodymyr Babchuk <volodymyr_babchuk@epam.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> > ---
> > 
> > Cc: Jan Beulich <jbeulich@suse.com>
> > Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> > Cc: George Dunlap <george.dunlap@citrix.com>
> > 
> > I noticed that arch_get_dma_bit_size() is only called when there is more
> > than one NUMA node. I am a bit unsure what is the reason behind it.
> > 
> > The goal for Arm is to use arch_get_dma_bit_size() when deciding how low
> > the first Dom0 bank should be allocated.
> > ---
> >  xen/arch/arm/platform.c        | 5 +++++
> >  xen/include/asm-arm/mm.h       | 2 ++
> >  xen/include/asm-arm/numa.h     | 5 -----
> >  xen/include/asm-arm/platform.h | 2 ++
> >  4 files changed, 9 insertions(+), 5 deletions(-)
> > 
> > diff --git a/xen/arch/arm/platform.c b/xen/arch/arm/platform.c
> > index 8eb0b6e57a5a..4db5bbb4c51d 100644
> > --- a/xen/arch/arm/platform.c
> > +++ b/xen/arch/arm/platform.c
> > @@ -155,6 +155,11 @@ bool platform_device_is_blacklisted(const struct dt_device_node *node)
> >      return (dt_match_node(blacklist, node) != NULL);
> >  }
> >  
> > +unsigned int arch_get_dma_bitsize(void)
> > +{
> > +    return ( platform && platform->dma_bitsize ) ? platform->dma_bitsize : 32;
> > +}
> > +
> >  /*
> >   * Local variables:
> >   * mode: C
> > diff --git a/xen/include/asm-arm/mm.h b/xen/include/asm-arm/mm.h
> > index 7df91280bc77..f8ba49b1188f 100644
> > --- a/xen/include/asm-arm/mm.h
> > +++ b/xen/include/asm-arm/mm.h
> > @@ -366,6 +366,8 @@ int arch_acquire_resource(struct domain *d, unsigned int type, unsigned int id,
> >      return -EOPNOTSUPP;
> >  }
> >  
> > +unsigned int arch_get_dma_bitsize(void);
> > +
> >  #endif /*  __ARCH_ARM_MM__ */
> >  /*
> >   * Local variables:
> > diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> > index 490d1f31aa14..31a6de4e2346 100644
> > --- a/xen/include/asm-arm/numa.h
> > +++ b/xen/include/asm-arm/numa.h
> > @@ -25,11 +25,6 @@ extern mfn_t first_valid_mfn;
> >  #define node_start_pfn(nid) (mfn_x(first_valid_mfn))
> >  #define __node_distance(a, b) (20)
> >  
> > -static inline unsigned int arch_get_dma_bitsize(void)
> > -{
> > -    return 32;
> > -}
> > -
> >  #endif /* __ARCH_ARM_NUMA_H */
> >  /*
> >   * Local variables:
> > diff --git a/xen/include/asm-arm/platform.h b/xen/include/asm-arm/platform.h
> > index ed4d30a1be7c..997eb2521631 100644
> > --- a/xen/include/asm-arm/platform.h
> > +++ b/xen/include/asm-arm/platform.h
> > @@ -38,6 +38,8 @@ struct platform_desc {
> >       * List of devices which must not pass-through to a guest
> >       */
> >      const struct dt_device_match *blacklist_dev;
> > +    /* Override the DMA width (32-bit by default). */
> > +    unsigned int dma_bitsize;
> >  };
> >  
> >  /*
> 

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Stefano Stabellini]

On Mon, 18 May 2020, Julien Grall wrote:
> From: Julien Grall <jgrall@amazon.com>
> 
> Hi all,
> 
> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> only use the first GB of memory.
> 
> This is because several devices cannot DMA above 1GB but Xen doesn't
> necessarily allocate memory for Dom0 below 1GB.
> 
> This small series is trying to address the problem by allowing a
> platform to restrict where Dom0 banks are allocated.
> 
> This is also a candidate for Xen 4.14. Without it, a user will not be
> able to use all the RAM on the Raspberry Pi 4.

The series looks good to me aside from the couple of minor issues being
discussed


> This series has only be slighlty tested. I would appreciate more test on
> the Rasbperry Pi 4 to confirm this removing the restriction.
> 
> Cheers,
> 
> Cc: paul@xen.org
> 
> Julien Grall (3):
>   xen/arm: Allow a platform to override the DMA width
>   xen/arm: Take into account the DMA width when allocating Dom0 memory
>     banks
>   xen/arm: plat: Allocate as much as possible memory below 1GB for dom0
>     for RPI
> 
>  xen/arch/arm/domain_build.c                | 32 +++++++++++++---------
>  xen/arch/arm/platform.c                    |  5 ++++
>  xen/arch/arm/platforms/brcm-raspberry-pi.c |  1 +
>  xen/include/asm-arm/mm.h                   |  2 ++
>  xen/include/asm-arm/numa.h                 |  5 ----
>  xen/include/asm-arm/platform.h             |  2 ++
>  6 files changed, 29 insertions(+), 18 deletions(-)
> 
> -- 
> 2.17.1
> 

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Julien Grall]

Hi Stefano,

On 20/05/2020 23:13, Stefano Stabellini wrote:
> On Mon, 18 May 2020, Julien Grall wrote:
>> From: Julien Grall <jgrall@amazon.com>
>>
>> Hi all,
>>
>> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> only use the first GB of memory.
>>
>> This is because several devices cannot DMA above 1GB but Xen doesn't
>> necessarily allocate memory for Dom0 below 1GB.
>>
>> This small series is trying to address the problem by allowing a
>> platform to restrict where Dom0 banks are allocated.
>>
>> This is also a candidate for Xen 4.14. Without it, a user will not be
>> able to use all the RAM on the Raspberry Pi 4.
> 
> The series looks good to me aside from the couple of minor issues being
> discussed

Thanks, I have sent the v2 yesterday but forgot to call 
add_maintainers.pl ([1]). Do you want me to resend it with you CCed?

Cheers,

[1] <20200519172028.31169-1-julien@xen.org>

-- 
Julien Grall

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

[Stefano Stabellini]

On Thu, 21 May 2020, Julien Grall wrote:
> Hi Stefano,
> 
> On 20/05/2020 23:13, Stefano Stabellini wrote:
> > On Mon, 18 May 2020, Julien Grall wrote:
> > > From: Julien Grall <jgrall@amazon.com>
> > > 
> > > Hi all,
> > > 
> > > At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > > only use the first GB of memory.
> > > 
> > > This is because several devices cannot DMA above 1GB but Xen doesn't
> > > necessarily allocate memory for Dom0 below 1GB.
> > > 
> > > This small series is trying to address the problem by allowing a
> > > platform to restrict where Dom0 banks are allocated.
> > > 
> > > This is also a candidate for Xen 4.14. Without it, a user will not be
> > > able to use all the RAM on the Raspberry Pi 4.
> > 
> > The series looks good to me aside from the couple of minor issues being
> > discussed
> 
> Thanks, I have sent the v2 yesterday but forgot to call add_maintainers.pl
> ([1]). Do you want me to resend it with you CCed?
> 
> Cheers,
> 
> [1] <20200519172028.31169-1-julien@xen.org>

No worries I found them now