Re: Xen on RP4

Re: Xen on RP4

[Stefano Stabellini]

+ xen-devel

On Fri, 23 Oct 2020, Elliott Mitchell wrote:
> On Thu, Oct 22, 2020 at 06:02:46PM -0700, Stefano Stabellini wrote:
> > On Thu, 22 Oct 2020, Elliott Mitchell wrote:
> > > On Thu, Oct 22, 2020 at 04:27:23PM -0700, Stefano Stabellini wrote:
> > > > On Wed, 21 Oct 2020, Elliott Mitchell wrote:
> > > > > Due to experimenting with "proper" console on serial port, I ended up
> > > > > getting output.  Apparently domain 0 was panicing when trying to setup
> > > > > xen-blkback due to the swiotlb code being unable to allocate a bounce
> > > > > buffer.
> > > > > 
> > > > > Stefano, what is the status of swiotlb in the 5.8 kernel series?
> > > > 
> > > > The swiotlb fixes for RPi4 are not in 5.8. Linux 5.9 has just been
> > > > released, and it should come with everything you need.
> > > 
> > > I had 13 patches applied to Debian's 5.8 kernel source.  Two of the
> > > batch I had against 5.6 had gotten into mainline.  No issues were visible
> > > during normal operation.
> > > 
> > > Problem showed up when trying to start a domain.  By using Xen's console
> > > device I managed to get the messages:
> > > 
> > > xen-blkback: backend/vbd/3/51712: using 2 queues, protocol 1 (arm-abi) persistent grants
> > > Kernel panic - not syncing: Can not allocate SWIOTLB buffer earlier and can't now provide you with the DMA bounce buffer
> > > 
> > > Worth noting that by the time when I was starting this domain, the device
> > > had an uptime of more than an hour.  There could be a problem of swiotlb
> > > needing the ability to claim DMA-viable pages after they've been in use
> > > for other purposes.
> >  
> > I'll have a look
> 
> Finally came up with one detail of a change I'd made in the right time
> frame to trigger this issue.  As such I can now control this behavior and
> get it to occur or not.
> 
> I have some suspicion my planned workload approach differs from others.
> 
> During the runs where I was able to successfully boot a child domain,
> domain 0 had been allocated 512MB of memory.  During the unsuccessful run
> where the above message occurred, domain 0 had been allocated 2GB of
> memory.  This appears to reliably control the occurrence of this bug.

This is what is going on. kernel/dma/swiotlb.c:swiotlb_init gets called
and tries to allocate a buffer for the swiotlb. It does so by calling

  memblock_alloc_low(PAGE_ALIGN(bytes), PAGE_SIZE);

In your case, the allocation must fail, no_iotlb_memory is set, and I
expect you see this warning among the boot messages:

  Cannot allocate buffer

Later during initialization swiotlb-xen comes in
(drivers/xen/swiotlb-xen.c:xen_swiotlb_init) and given that io_tlb_start
is != 0 it thinks the memory is ready to use when actually it is not.

When the swiotlb is actually needed, swiotlb_tbl_map_single gets called
and since no_iotlb_memory is set the kernel panics.


The reason why you are only seeing it with a 2G dom0 is because
swiotlb_init is only called when:

  max_pfn > PFN_DOWN(arm64_dma_phys_limit ? : arm64_dma32_phys_limit))

see arch/arm64/mm/init.c:mem_init. So when dom0 is 512MB swiotlb_init is
not called at all. swiotlb-xen does the allocation itself with
memblock_alloc and it succeeds.

Note that I tried to repro the issue here at my end but it works for me
with device tree. So the swiotlb_init memory allocation failure probably
only shows on ACPI, maybe because ACPI is reserving too much low memory.

In any case, I think the issue might be "fixed" by this patch:



diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
index c19379fabd20..84e15e7d3929 100644
--- a/kernel/dma/swiotlb.c
+++ b/kernel/dma/swiotlb.c
@@ -231,6 +231,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
 		io_tlb_orig_addr[i] = INVALID_PHYS_ADDR;
 	}
 	io_tlb_index = 0;
+	no_iotlb_memory = false;
 
 	if (verbose)
 		swiotlb_print_info();
@@ -263,8 +264,11 @@ swiotlb_init(int verbose)
 		return;
 
 	if (io_tlb_start)
+	{
 		memblock_free_early(io_tlb_start,
 				    PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT));
+		io_tlb_start = 0;
+	}
 	pr_warn("Cannot allocate buffer");
 	no_iotlb_memory = true;
 }
@@ -362,6 +366,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
 		io_tlb_orig_addr[i] = INVALID_PHYS_ADDR;
 	}
 	io_tlb_index = 0;
+	no_iotlb_memory = false;
 
 	swiotlb_print_info();

Re: Xen on RP4

[Elliott Mitchell]

On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote:
> This is what is going on. kernel/dma/swiotlb.c:swiotlb_init gets called
> and tries to allocate a buffer for the swiotlb. It does so by calling
> 
>   memblock_alloc_low(PAGE_ALIGN(bytes), PAGE_SIZE);
> 
> In your case, the allocation must fail, no_iotlb_memory is set, and I
> expect you see this warning among the boot messages:
> 
>   Cannot allocate buffer
> 
> Later during initialization swiotlb-xen comes in
> (drivers/xen/swiotlb-xen.c:xen_swiotlb_init) and given that io_tlb_start
> is != 0 it thinks the memory is ready to use when actually it is not.
> 
> When the swiotlb is actually needed, swiotlb_tbl_map_single gets called
> and since no_iotlb_memory is set the kernel panics.
> 
> 
> The reason why you are only seeing it with a 2G dom0 is because
> swiotlb_init is only called when:
> 
>   max_pfn > PFN_DOWN(arm64_dma_phys_limit ? : arm64_dma32_phys_limit))
> 
> see arch/arm64/mm/init.c:mem_init. So when dom0 is 512MB swiotlb_init is
> not called at all. swiotlb-xen does the allocation itself with
> memblock_alloc and it succeeds.
> 
> Note that I tried to repro the issue here at my end but it works for me
> with device tree. So the swiotlb_init memory allocation failure probably
> only shows on ACPI, maybe because ACPI is reserving too much low memory.
> 
> In any case, I think the issue might be "fixed" by this patch:
> 
> 
> 
> diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c
> index c19379fabd20..84e15e7d3929 100644
> --- a/kernel/dma/swiotlb.c
> +++ b/kernel/dma/swiotlb.c
> @@ -231,6 +231,7 @@ int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
>  		io_tlb_orig_addr[i] = INVALID_PHYS_ADDR;
>  	}
>  	io_tlb_index = 0;
> +	no_iotlb_memory = false;
>  
>  	if (verbose)
>  		swiotlb_print_info();
> @@ -263,8 +264,11 @@ swiotlb_init(int verbose)
>  		return;
>  
>  	if (io_tlb_start)
> +	{
>  		memblock_free_early(io_tlb_start,
>  				    PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT));
> +		io_tlb_start = 0;
> +	}
>  	pr_warn("Cannot allocate buffer");
>  	no_iotlb_memory = true;
>  }
> @@ -362,6 +366,7 @@ swiotlb_late_init_with_tbl(char *tlb, unsigned long nslabs)
>  		io_tlb_orig_addr[i] = INVALID_PHYS_ADDR;
>  	}
>  	io_tlb_index = 0;
> +	no_iotlb_memory = false;
>  
>  	swiotlb_print_info();

This does indeed appear to take care of the domain 0 panic.

Last issue is the framebuffer and this project has reached usability.
My impression was framebuffer was an issue for device-tree as well.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Elliott Mitchell]

On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote:
> This is what is going on. kernel/dma/swiotlb.c:swiotlb_init gets called
> and tries to allocate a buffer for the swiotlb. It does so by calling
> 
>   memblock_alloc_low(PAGE_ALIGN(bytes), PAGE_SIZE);
> 
> In your case, the allocation must fail, no_iotlb_memory is set, and I
> expect you see this warning among the boot messages:
> 
>   Cannot allocate buffer
> 
> Later during initialization swiotlb-xen comes in
> (drivers/xen/swiotlb-xen.c:xen_swiotlb_init) and given that io_tlb_start
> is != 0 it thinks the memory is ready to use when actually it is not.

Forgot to respond to this portion even though I'd noticed on my first
read.  I haven't ever observed that message.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Elliott Mitchell]

On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote:
> On Fri, 23 Oct 2020, Elliott Mitchell wrote:
> > Finally came up with one detail of a change I'd made in the right time
> > frame to trigger this issue.  As such I can now control this behavior and
> > get it to occur or not.
> > 
> > I have some suspicion my planned workload approach differs from others.
> > 
> > During the runs where I was able to successfully boot a child domain,
> > domain 0 had been allocated 512MB of memory.  During the unsuccessful run
> > where the above message occurred, domain 0 had been allocated 2GB of
> > memory.  This appears to reliably control the occurrence of this bug.

> In your case, the allocation must fail, no_iotlb_memory is set, and I
> expect you see this warning among the boot messages:
> 
>   Cannot allocate buffer
> 
> Later during initialization swiotlb-xen comes in
> (drivers/xen/swiotlb-xen.c:xen_swiotlb_init) and given that io_tlb_start
> is != 0 it thinks the memory is ready to use when actually it is not.

Then I look at more copies of `dmesg` logs and discover I did have one
where that message occurred.  Then start playing a bit more and a
matching pattern emerges.

dom0_mem=1023M
=> "software IO TLB: mapped [mem 0x2c000000-0x30000000] (64MB)"

dom0_mem=1024M
=> "software IO TLB: Cannot allocate buffer"

That looks suspicious.  Really suspicious.  I don't know how many bugs
are combined together, nor where they are, but more data for you.

I see a possibility Tianocore marks a smaller region of memory as being
DMA-capable.  This though is speculation.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Elliott Mitchell]

On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote:
> Note that I tried to repro the issue here at my end but it works for me
> with device tree. So the swiotlb_init memory allocation failure probably
> only shows on ACPI, maybe because ACPI is reserving too much low memory.

Found it.  Take a look at 437b0aa06a014ce174e24c0d3530b3e9ab19b18b

 PLATFORM_START(rpi4, "Raspberry Pi 4")
     .compatible     = rpi4_dt_compat,
     .blacklist_dev  = rpi4_blacklist_dev,
+    .dma_bitsize    = 30,
 PLATFORM_END

Where this is used to match against a *device-tree*.  ACPI has a distinct
means of specifying a limited DMA-width; the above fails, because it
assumes a *device-tree*.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Julien Grall]

Hi Elliott,

On 24/10/2020 06:35, Elliott Mitchell wrote:
> On Fri, Oct 23, 2020 at 04:59:30PM -0700, Stefano Stabellini wrote:
>> Note that I tried to repro the issue here at my end but it works for me
>> with device tree. So the swiotlb_init memory allocation failure probably
>> only shows on ACPI, maybe because ACPI is reserving too much low memory.
> 
> Found it.  Take a look at 437b0aa06a014ce174e24c0d3530b3e9ab19b18b
> 
>   PLATFORM_START(rpi4, "Raspberry Pi 4")
>       .compatible     = rpi4_dt_compat,
>       .blacklist_dev  = rpi4_blacklist_dev,
> +    .dma_bitsize    = 30,
>   PLATFORM_END
> 
> Where this is used to match against a *device-tree*.

Right. When we introduced ACPI in Xen, we made the assumption there 
would be no need for per-platform workaround.

> ACPI has a distinct
> means of specifying a limited DMA-width; the above fails, because it
> assumes a *device-tree*.

Do you know if it would be possible to infer from the ACPI static table 
the DMA-width?

If not, is there a way to uniquely identify the platform?


Cheers,

-- 
Julien Grall

Re: Xen on RP4

[Elliott Mitchell]

On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> On 24/10/2020 06:35, Elliott Mitchell wrote:
> > ACPI has a distinct
> > means of specifying a limited DMA-width; the above fails, because it
> > assumes a *device-tree*.
> 
> Do you know if it would be possible to infer from the ACPI static table 
> the DMA-width?

Yes, and it is.  Due to not knowing much about ACPI tables I don't know
what the C code would look like though (problem is which documentation
should I be looking at first?).

Handy bit of information is in the RP4 Tianocore table source:
https://github.com/tianocore/edk2-platforms/blob/d492639638eee331ac3389e6cf53ea266c3c84b3/Platform/RaspberryPi/AcpiTables/Dsdt.asl

      Name (_DMA, ResourceTemplate() {
        //
        // Only the first GB is available.
        // Bus 0xC0000000 -> CPU 0x00000000.
        //
        QWordMemory (ResourceConsumer,
          ,
          MinFixed,
          MaxFixed,
          NonCacheable,
          ReadWrite,
          0x0,
          0x00000000C0000000, // MIN
          0x00000000FFFFFFFF, // MAX
          0xFFFFFFFF40000000, // TRA
          0x0000000040000000, // LEN
          ,
          ,
          )
      })

There should be some corresponding code in the Linux 5.9 kernels.  From
the look of that, it might even be possible to specify a memory range
which didn't start at address 0.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Julien Grall]

Hi Elliott,

On 26/10/2020 16:03, Elliott Mitchell wrote:
> On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
>> On 24/10/2020 06:35, Elliott Mitchell wrote:
>>> ACPI has a distinct
>>> means of specifying a limited DMA-width; the above fails, because it
>>> assumes a *device-tree*.
>>
>> Do you know if it would be possible to infer from the ACPI static table
>> the DMA-width?
> 
> Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> what the C code would look like though (problem is which documentation
> should I be looking at first?).

What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
is written in AML. So far the shortest implementation I have seen for 
the AML parser is around 5000 lines (see [1]). It might be possible to 
strip some the code, although I think this will still probably too big 
for a single workaround.

What I meant by "static table" is a table that looks like a structure 
and can be parsed in a few lines. If we can't find on contain the DMA 
window, then the next best solution is to find a way to identity the 
platform.

I don't know enough ACPI to know if this solution is possible. A good 
starter would probably be the ACPI spec [2].

> 
> Handy bit of information is in the RP4 Tianocore table source:
> https://github.com/tianocore/edk2-platforms/blob/d492639638eee331ac3389e6cf53ea266c3c84b3/Platform/RaspberryPi/AcpiTables/Dsdt.asl
> 
>        Name (_DMA, ResourceTemplate() {
>          //
>          // Only the first GB is available.
>          // Bus 0xC0000000 -> CPU 0x00000000.
>          //
>          QWordMemory (ResourceConsumer,
>            ,
>            MinFixed,
>            MaxFixed,
>            NonCacheable,
>            ReadWrite,
>            0x0,
>            0x00000000C0000000, // MIN
>            0x00000000FFFFFFFF, // MAX
>            0xFFFFFFFF40000000, // TRA
>            0x0000000040000000, // LEN
>            ,
>            ,
>            )
>        })
> 
> There should be some corresponding code in the Linux 5.9 kernels.  From
> the look of that, it might even be possible to specify a memory range
> which didn't start at address 0.
> 
> 

Cheers,

[1] https://github.com/openbsd/src/blob/master/sys/dev/acpi/dsdt.c
[2] https://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf

-- 
Julien Grall

Re: Xen on RP4

[Elliott Mitchell]

On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> Hi Elliott,
> 
> On 26/10/2020 16:03, Elliott Mitchell wrote:
> > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> >>> ACPI has a distinct
> >>> means of specifying a limited DMA-width; the above fails, because it
> >>> assumes a *device-tree*.
> >>
> >> Do you know if it would be possible to infer from the ACPI static table
> >> the DMA-width?
> > 
> > Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> > what the C code would look like though (problem is which documentation
> > should I be looking at first?).
> 
> What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> is written in AML. So far the shortest implementation I have seen for 
> the AML parser is around 5000 lines (see [1]). It might be possible to 
> strip some the code, although I think this will still probably too big 
> for a single workaround.
> 
> What I meant by "static table" is a table that looks like a structure 
> and can be parsed in a few lines. If we can't find on contain the DMA 
> window, then the next best solution is to find a way to identity the 
> platform.
> 
> I don't know enough ACPI to know if this solution is possible. A good 
> starter would probably be the ACPI spec [2].

Be assured, you likely know more about ACPI than I do.  :-)

A crucial point though is the mentions of dealing with DMA on the
Raspberry PI 4B using ACPI pointed at that "_DMA" string.  What is there
is Good Enough(tm) to make a 5.8 Linux kernel successfully operate
using ACPI.

Looking at the 5.8 source, apparently _DMA is an ACPI method.  That
almost looks straightforward enough for me to tackle for Xen...
Good news is looks like only supportting a single DMA window...


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Elliott Mitchell]

On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> On 26/10/2020 16:03, Elliott Mitchell wrote:
> > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> >>> ACPI has a distinct
> >>> means of specifying a limited DMA-width; the above fails, because it
> >>> assumes a *device-tree*.
> >>
> >> Do you know if it would be possible to infer from the ACPI static table
> >> the DMA-width?
> > 
> > Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> > what the C code would look like though (problem is which documentation
> > should I be looking at first?).
> 
> What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> is written in AML. So far the shortest implementation I have seen for 
> the AML parser is around 5000 lines (see [1]). It might be possible to 
> strip some the code, although I think this will still probably too big 
> for a single workaround.
> 
> What I meant by "static table" is a table that looks like a structure 
> and can be parsed in a few lines. If we can't find on contain the DMA 
> window, then the next best solution is to find a way to identity the 
> platform.
> 
> I don't know enough ACPI to know if this solution is possible. A good 
> starter would probably be the ACPI spec [2].

Okay, that is worse than I had thought (okay, ACPI is impressively
complex for something nominally firmware-level).

There are strings in the present Tianocore implementation for Raspberry
PI 4B which could be targeted.  Notably included in the output during
boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
unsure how kosher these are as this wsn't implemented nor blessed by the
Raspberry PI Foundation).

I strongly dislike this approach as you soon turn the Xen project into a
database of hardware.  This is already occurring with
xen/arch/arm/platforms and I would love to do something about this.  On
that thought, how about utilizing Xen's command-line for this purpose?

Have a procedure of during installation/updates retrieve DMA limitation
information from the running OS and the following boot Xen will follow
the appropriate setup.  By its nature, Domain 0 will have the information
needed, just becomes an issue of how hard that is to retrieve...


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Stefano Stabellini]

On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> > >>> ACPI has a distinct
> > >>> means of specifying a limited DMA-width; the above fails, because it
> > >>> assumes a *device-tree*.
> > >>
> > >> Do you know if it would be possible to infer from the ACPI static table
> > >> the DMA-width?
> > > 
> > > Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> > > what the C code would look like though (problem is which documentation
> > > should I be looking at first?).
> > 
> > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> > is written in AML. So far the shortest implementation I have seen for 
> > the AML parser is around 5000 lines (see [1]). It might be possible to 
> > strip some the code, although I think this will still probably too big 
> > for a single workaround.
> > 
> > What I meant by "static table" is a table that looks like a structure 
> > and can be parsed in a few lines. If we can't find on contain the DMA 
> > window, then the next best solution is to find a way to identity the 
> > platform.
> > 
> > I don't know enough ACPI to know if this solution is possible. A good 
> > starter would probably be the ACPI spec [2].
> 
> Okay, that is worse than I had thought (okay, ACPI is impressively
> complex for something nominally firmware-level).
>
> There are strings in the present Tianocore implementation for Raspberry
> PI 4B which could be targeted.  Notably included in the output during
> boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> unsure how kosher these are as this wsn't implemented nor blessed by the
> Raspberry PI Foundation).
> 
> I strongly dislike this approach as you soon turn the Xen project into a
> database of hardware.  This is already occurring with
> xen/arch/arm/platforms and I would love to do something about this.  On
> that thought, how about utilizing Xen's command-line for this purpose?

I don't think that a command line option is a good idea: basically it is
punting to users the task of platform detection. Also, it means that
users will be necessarily forced to edit the uboot script or grub
configuration file to boot.

Note that even if we introduced a new command line, we wouldn't take
away the need for xen/arch/arm/platforms anyway.

It would be far better for Xen to autodetect the platform if we can.


> Have a procedure of during installation/updates retrieve DMA limitation
> information from the running OS and the following boot Xen will follow
> the appropriate setup.  By its nature, Domain 0 will have the information
> needed, just becomes an issue of how hard that is to retrieve...

Historically that is what we used to do for many things related to ACPI,
but unfortunately it leads to a pretty bad architecture where Xen
depends on Dom0 for booting rather than the other way around. (Dom0
should be the one requiring Xen for booting, given that Xen is higher
privilege and boots first.)


I think the best compromise is still to use an ACPI string to detect the
platform. For instance, would it be possible to use the OEMID fields in
RSDT, XSDT, FADT?  Possibly even a combination of them?

Another option might be to get the platform name from UEFI somehow. 

Re: Xen on RP4

[Jürgen Groß]

On 29.10.20 01:37, Stefano Stabellini wrote:
> On Tue, 27 Oct 2020, Elliott Mitchell wrote:
>> On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
>>> On 26/10/2020 16:03, Elliott Mitchell wrote:
>>>> On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
>>>>> On 24/10/2020 06:35, Elliott Mitchell wrote:
>>>>>> ACPI has a distinct
>>>>>> means of specifying a limited DMA-width; the above fails, because it
>>>>>> assumes a *device-tree*.
>>>>>
>>>>> Do you know if it would be possible to infer from the ACPI static table
>>>>> the DMA-width?
>>>>
>>>> Yes, and it is.  Due to not knowing much about ACPI tables I don't know
>>>> what the C code would look like though (problem is which documentation
>>>> should I be looking at first?).
>>>
>>> What you provided below is an excerpt of the DSDT. AFAIK, DSDT content
>>> is written in AML. So far the shortest implementation I have seen for
>>> the AML parser is around 5000 lines (see [1]). It might be possible to
>>> strip some the code, although I think this will still probably too big
>>> for a single workaround.
>>>
>>> What I meant by "static table" is a table that looks like a structure
>>> and can be parsed in a few lines. If we can't find on contain the DMA
>>> window, then the next best solution is to find a way to identity the
>>> platform.
>>>
>>> I don't know enough ACPI to know if this solution is possible. A good
>>> starter would probably be the ACPI spec [2].
>>
>> Okay, that is worse than I had thought (okay, ACPI is impressively
>> complex for something nominally firmware-level).
>>
>> There are strings in the present Tianocore implementation for Raspberry
>> PI 4B which could be targeted.  Notably included in the output during
>> boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
>> unsure how kosher these are as this wsn't implemented nor blessed by the
>> Raspberry PI Foundation).
>>
>> I strongly dislike this approach as you soon turn the Xen project into a
>> database of hardware.  This is already occurring with
>> xen/arch/arm/platforms and I would love to do something about this.  On
>> that thought, how about utilizing Xen's command-line for this purpose?
> 
> I don't think that a command line option is a good idea: basically it is
> punting to users the task of platform detection. Also, it means that
> users will be necessarily forced to edit the uboot script or grub
> configuration file to boot.
> 
> Note that even if we introduced a new command line, we wouldn't take
> away the need for xen/arch/arm/platforms anyway.
> 
> It would be far better for Xen to autodetect the platform if we can.
> 
> 
>> Have a procedure of during installation/updates retrieve DMA limitation
>> information from the running OS and the following boot Xen will follow
>> the appropriate setup.  By its nature, Domain 0 will have the information
>> needed, just becomes an issue of how hard that is to retrieve...
> 
> Historically that is what we used to do for many things related to ACPI,
> but unfortunately it leads to a pretty bad architecture where Xen
> depends on Dom0 for booting rather than the other way around. (Dom0
> should be the one requiring Xen for booting, given that Xen is higher
> privilege and boots first.)
> 
> 
> I think the best compromise is still to use an ACPI string to detect the
> platform. For instance, would it be possible to use the OEMID fields in
> RSDT, XSDT, FADT?  Possibly even a combination of them?
> 
> Another option might be to get the platform name from UEFI somehow.

What about having a small domain parsing the ACPI booting first and use
that information for booting dom0?

That dom would be part of the Xen build and the hypervisor wouldn't need
to gain all the ACPI AML logic.


Juergen

Re: Xen on RP4

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 4338 bytes --]

On Thu, 29 Oct 2020, Jürgen Groß wrote:
> On 29.10.20 01:37, Stefano Stabellini wrote:
> > On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> > > On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > > > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > > > > > On 24/10/2020 06:35, Elliott Mitchell wrote:
> > > > > > > ACPI has a distinct
> > > > > > > means of specifying a limited DMA-width; the above fails, because
> > > > > > > it
> > > > > > > assumes a *device-tree*.
> > > > > > 
> > > > > > Do you know if it would be possible to infer from the ACPI static
> > > > > > table
> > > > > > the DMA-width?
> > > > > 
> > > > > Yes, and it is.  Due to not knowing much about ACPI tables I don't
> > > > > know
> > > > > what the C code would look like though (problem is which documentation
> > > > > should I be looking at first?).
> > > > 
> > > > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content
> > > > is written in AML. So far the shortest implementation I have seen for
> > > > the AML parser is around 5000 lines (see [1]). It might be possible to
> > > > strip some the code, although I think this will still probably too big
> > > > for a single workaround.
> > > > 
> > > > What I meant by "static table" is a table that looks like a structure
> > > > and can be parsed in a few lines. If we can't find on contain the DMA
> > > > window, then the next best solution is to find a way to identity the
> > > > platform.
> > > > 
> > > > I don't know enough ACPI to know if this solution is possible. A good
> > > > starter would probably be the ACPI spec [2].
> > > 
> > > Okay, that is worse than I had thought (okay, ACPI is impressively
> > > complex for something nominally firmware-level).
> > > 
> > > There are strings in the present Tianocore implementation for Raspberry
> > > PI 4B which could be targeted.  Notably included in the output during
> > > boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> > > unsure how kosher these are as this wsn't implemented nor blessed by the
> > > Raspberry PI Foundation).
> > > 
> > > I strongly dislike this approach as you soon turn the Xen project into a
> > > database of hardware.  This is already occurring with
> > > xen/arch/arm/platforms and I would love to do something about this.  On
> > > that thought, how about utilizing Xen's command-line for this purpose?
> > 
> > I don't think that a command line option is a good idea: basically it is
> > punting to users the task of platform detection. Also, it means that
> > users will be necessarily forced to edit the uboot script or grub
> > configuration file to boot.
> > 
> > Note that even if we introduced a new command line, we wouldn't take
> > away the need for xen/arch/arm/platforms anyway.
> > 
> > It would be far better for Xen to autodetect the platform if we can.
> > 
> > 
> > > Have a procedure of during installation/updates retrieve DMA limitation
> > > information from the running OS and the following boot Xen will follow
> > > the appropriate setup.  By its nature, Domain 0 will have the information
> > > needed, just becomes an issue of how hard that is to retrieve...
> > 
> > Historically that is what we used to do for many things related to ACPI,
> > but unfortunately it leads to a pretty bad architecture where Xen
> > depends on Dom0 for booting rather than the other way around. (Dom0
> > should be the one requiring Xen for booting, given that Xen is higher
> > privilege and boots first.)
> > 
> > 
> > I think the best compromise is still to use an ACPI string to detect the
> > platform. For instance, would it be possible to use the OEMID fields in
> > RSDT, XSDT, FADT?  Possibly even a combination of them?
> > 
> > Another option might be to get the platform name from UEFI somehow.
> 
> What about having a small domain parsing the ACPI booting first and use
> that information for booting dom0?
> 
> That dom would be part of the Xen build and the hypervisor wouldn't need
> to gain all the ACPI AML logic.

That could work, but in practice we don't have such a domain today --
the infrastructure is missing. I wonder whether the bootloader (uboot or
grub) would know about the platform and might be able to pass that
information to Xen somehow.

Re: Xen on RP4

[Elliott Mitchell]

On Thu, Oct 29, 2020 at 12:57:58PM -0700, Stefano Stabellini wrote:
> On Thu, 29 Oct 2020, J??rgen Gro?? wrote:
> > What about having a small domain parsing the ACPI booting first and use
> > that information for booting dom0?
> > 
> > That dom would be part of the Xen build and the hypervisor wouldn't need
> > to gain all the ACPI AML logic.
> 
> That could work, but in practice we don't have such a domain today --
> the infrastructure is missing. I wonder whether the bootloader (uboot or
> grub) would know about the platform and might be able to pass that
> information to Xen somehow.

How long would such likely take to implement?  This reads like a
complicated project, and likely to take a while...


Then would be the issue of efifb.



I've been pondering allocate_memory_11() and coming up with a rather
complicated potential problem.  ACPI appears to potentially allow for
non-power of 2 DMA ranges; I'm unaware of any such device, but the code
should allow for such.

I can imagine a device which has multiple DMA ranges.  The ranges could
be fully contained within each other, the ranges could partially overlap,
or the ranges could be disjoint.

Someone might wish to allocate all DMA-capable memory to domain 0,
someone might wish to allocate less.  Additionally if all DMA-capable
memory is allocated to domain 0, some non-DMA-capable memory could be
desired.

Ideally Xen would move to non-DMA memory.  This would protect Xen against
a malicious domain 0 and allow allocating more DMA-capable memory to
domain 0.

This interacts with ballooning.  If memory is removed from domain 0,
non-DMA memory should be removed first.  If domain 0 is allocated more
memory, DMA memory should be added first (if any isn't allocated to
domain 0).

Then again I may be severely overthinking things.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Christopher Clark]

On Thu, Oct 29, 2020 at 12:58 PM Stefano Stabellini
<sstabellini@kernel.org> wrote:
>
> On Thu, 29 Oct 2020, Jürgen Groß wrote:
> > On 29.10.20 01:37, Stefano Stabellini wrote:
> > > On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> > > > On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > > > > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > > > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > > > > > > On 24/10/2020 06:35, Elliott Mitchell wrote:
> > > > > > > > ACPI has a distinct
> > > > > > > > means of specifying a limited DMA-width; the above fails, because
> > > > > > > > it
> > > > > > > > assumes a *device-tree*.
> > > > > > >
> > > > > > > Do you know if it would be possible to infer from the ACPI static
> > > > > > > table
> > > > > > > the DMA-width?
> > > > > >
> > > > > > Yes, and it is.  Due to not knowing much about ACPI tables I don't
> > > > > > know
> > > > > > what the C code would look like though (problem is which documentation
> > > > > > should I be looking at first?).
> > > > >
> > > > > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content
> > > > > is written in AML. So far the shortest implementation I have seen for
> > > > > the AML parser is around 5000 lines (see [1]). It might be possible to
> > > > > strip some the code, although I think this will still probably too big
> > > > > for a single workaround.
> > > > >
> > > > > What I meant by "static table" is a table that looks like a structure
> > > > > and can be parsed in a few lines. If we can't find on contain the DMA
> > > > > window, then the next best solution is to find a way to identity the
> > > > > platform.
> > > > >
> > > > > I don't know enough ACPI to know if this solution is possible. A good
> > > > > starter would probably be the ACPI spec [2].
> > > >
> > > > Okay, that is worse than I had thought (okay, ACPI is impressively
> > > > complex for something nominally firmware-level).
> > > >
> > > > There are strings in the present Tianocore implementation for Raspberry
> > > > PI 4B which could be targeted.  Notably included in the output during
> > > > boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> > > > unsure how kosher these are as this wsn't implemented nor blessed by the
> > > > Raspberry PI Foundation).
> > > >
> > > > I strongly dislike this approach as you soon turn the Xen project into a
> > > > database of hardware.  This is already occurring with
> > > > xen/arch/arm/platforms and I would love to do something about this.  On
> > > > that thought, how about utilizing Xen's command-line for this purpose?
> > >
> > > I don't think that a command line option is a good idea: basically it is
> > > punting to users the task of platform detection. Also, it means that
> > > users will be necessarily forced to edit the uboot script or grub
> > > configuration file to boot.
> > >
> > > Note that even if we introduced a new command line, we wouldn't take
> > > away the need for xen/arch/arm/platforms anyway.
> > >
> > > It would be far better for Xen to autodetect the platform if we can.
> > >
> > >
> > > > Have a procedure of during installation/updates retrieve DMA limitation
> > > > information from the running OS and the following boot Xen will follow
> > > > the appropriate setup.  By its nature, Domain 0 will have the information
> > > > needed, just becomes an issue of how hard that is to retrieve...
> > >
> > > Historically that is what we used to do for many things related to ACPI,
> > > but unfortunately it leads to a pretty bad architecture where Xen
> > > depends on Dom0 for booting rather than the other way around. (Dom0
> > > should be the one requiring Xen for booting, given that Xen is higher
> > > privilege and boots first.)
> > >
> > >
> > > I think the best compromise is still to use an ACPI string to detect the
> > > platform. For instance, would it be possible to use the OEMID fields in
> > > RSDT, XSDT, FADT?  Possibly even a combination of them?
> > >
> > > Another option might be to get the platform name from UEFI somehow.
> >
> > What about having a small domain parsing the ACPI booting first and use
> > that information for booting dom0?
> >
> > That dom would be part of the Xen build and the hypervisor wouldn't need
> > to gain all the ACPI AML logic.
>
> That could work, but in practice we don't have such a domain today --
> the infrastructure is missing. I wonder whether the bootloader (uboot or
> grub) would know about the platform and might be able to pass that
> information to Xen somehow.

This is one of the functions envisioned for the Boot Domain in the
proposal to add DomB mode to dom0less[1], in the work developed by
Daniel Smith and I.

With DomB as much or as little of platform detection and initial
domain configuration will be possible from the Boot Domain. ACPI was
specifically discussed in the second DomB Design Session at this
year's Xen Summit[2].

Further information on this is available on the Xen wiki[3], thanks to
Rich Persaud, including links to the Summit presentation on DomB for
Xen System Boot, and the following discussion of that in the Design
Session[4], and specific questions can be raised in this thread.

thanks,

Christopher

[1] https://lists.xenproject.org/archives/html/xen-devel/2020-05/msg00233.html
[2] https://www.youtube.com/watch?v=jlyWfdW6D-E&t=15m26s
[3] https://wiki.xenproject.org/wiki/DomB_mode_of_dom0less
[4] https://lists.archive.carbon60.com/xen/devel/592140

Re: Xen on RP4

[Elliott Mitchell]

On Wed, Oct 28, 2020 at 05:37:02PM -0700, Stefano Stabellini wrote:
> On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> > On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > > >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> > > >>> ACPI has a distinct
> > > >>> means of specifying a limited DMA-width; the above fails, because it
> > > >>> assumes a *device-tree*.
> > > >>
> > > >> Do you know if it would be possible to infer from the ACPI static table
> > > >> the DMA-width?
> > > > 
> > > > Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> > > > what the C code would look like though (problem is which documentation
> > > > should I be looking at first?).
> > > 
> > > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> > > is written in AML. So far the shortest implementation I have seen for 
> > > the AML parser is around 5000 lines (see [1]). It might be possible to 
> > > strip some the code, although I think this will still probably too big 
> > > for a single workaround.
> > > 
> > > What I meant by "static table" is a table that looks like a structure 
> > > and can be parsed in a few lines. If we can't find on contain the DMA 
> > > window, then the next best solution is to find a way to identity the 
> > > platform.
> > > 
> > > I don't know enough ACPI to know if this solution is possible. A good 
> > > starter would probably be the ACPI spec [2].
> > 
> > Okay, that is worse than I had thought (okay, ACPI is impressively
> > complex for something nominally firmware-level).
> >
> > There are strings in the present Tianocore implementation for Raspberry
> > PI 4B which could be targeted.  Notably included in the output during
> > boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> > unsure how kosher these are as this wsn't implemented nor blessed by the
> > Raspberry PI Foundation).
> > 
> > I strongly dislike this approach as you soon turn the Xen project into a
> > database of hardware.  This is already occurring with
> > xen/arch/arm/platforms and I would love to do something about this.  On
> > that thought, how about utilizing Xen's command-line for this purpose?
> 
> I don't think that a command line option is a good idea: basically it is
> punting to users the task of platform detection. Also, it means that
> users will be necessarily forced to edit the uboot script or grub
> configuration file to boot.

-EINVAL

Many Linux installations (near universal on desktop/server, but may be
uncommon on ARM servers) Xen's command-line comes from grub.cfg.
grub.cfg is in turn created by a series of scripts with several places
for users to modify configuration without breaking things.

The scripts which create grub.cfg could add a "dma_mem=" option to Xen's
command-line based upon what the running kernel reports.  If the kernel
is running on top of Xen, it will still be able to retrieve this
information out of ACPI.

This does mean distributions would need to modify scripts, but that is
doable.  This also means a dumb user could potentially jump in, modify
the value and thus cause unrecoverable breakage.  Yet on the flip side
this also allows for the short-term stop-gap of smart users modifying
the option as appropriate for new hardware.

Certainly it may not be the greatest approach, but it isn't as bad as
you're claiming.


> Note that even if we introduced a new command line, we wouldn't take
> away the need for xen/arch/arm/platforms anyway.

Perhaps, but it could allow for this setting at least to be moved to
somewhere outside of Xen.

I'm inclined to agree with Juergen Gro??, this reads kind of like having
an extra domain run during Xen's initialization which can talk ACPI.


> > Have a procedure of during installation/updates retrieve DMA limitation
> > information from the running OS and the following boot Xen will follow
> > the appropriate setup.  By its nature, Domain 0 will have the information
> > needed, just becomes an issue of how hard that is to retrieve...
> 
> Historically that is what we used to do for many things related to ACPI,
> but unfortunately it leads to a pretty bad architecture where Xen
> depends on Dom0 for booting rather than the other way around. (Dom0
> should be the one requiring Xen for booting, given that Xen is higher
> privilege and boots first.)
> 
> 
> I think the best compromise is still to use an ACPI string to detect the
> platform. For instance, would it be possible to use the OEMID fields in
> RSDT, XSDT, FADT?  Possibly even a combination of them?
> 
> Another option might be to get the platform name from UEFI somehow. 

I included appropriate strings in e-mail.  Suitable strings do appear in
`dmesg`.

Problem is this feels like you're hard-coding a fixed list of platforms
Xen can run on.  Instead values like these should be provided by
firmware.  ACPI includes a method for encoding DMA limitations,
device-tree really should have one added.  Only challenge for device-tree
is getting everyone to agree on names and parameters.



Looking at it, there are really issues with the allocate_memory_11()
function in xen/arch/arm/domain_build.c.  Two tasks have been merged and
I'm unsure they were merged correctly.

I'm unaware of any Xen-capable platforms with such, but DMA can have
distinct restrictions outside of what allocate_memory_11() provides for.
ACPI allows for explicit address ranges and in the past many devices have
used addresses that didn't start at zero.

Additionally a device might have several devices with restricted DMA and
these could have differing non-overlapping ranges.  Domain 0 might need
memory in several DMA ranges.  Luckily In the past few years I haven't
read about any potentially Xen-capable devices where DMA-capable memory
had differing capabilities/performance from non-DMA-capable memory.

Meanwhile for a platform which does have DMA limitations, the kernel and
boot information for domain 0 shouldn't be placed in DMA-capable memory
if domain 0 has any memory which isn't DMA-capable.  Yet it appears
allocate_memory_11() will cause kernel/initrd/boot information to be
placed in DMA-capable memory.

If my understanding is correct (this is a BIG IF), as a last step
allocate_memory_11() should reverse the order of memory banks.  Another
trick is DMA-capable banks need to be subject to ballooning AFTER
non-DMA-capable banks (I'm unsure how often ballooning is used on ARM).


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Stefano Stabellini]

On Thu, 29 Oct 2020, Elliott Mitchell wrote:
> On Wed, Oct 28, 2020 at 05:37:02PM -0700, Stefano Stabellini wrote:
> > On Tue, 27 Oct 2020, Elliott Mitchell wrote:
> > > On Mon, Oct 26, 2020 at 06:44:27PM +0000, Julien Grall wrote:
> > > > On 26/10/2020 16:03, Elliott Mitchell wrote:
> > > > > On Mon, Oct 26, 2020 at 01:31:42PM +0000, Julien Grall wrote:
> > > > >> On 24/10/2020 06:35, Elliott Mitchell wrote:
> > > > >>> ACPI has a distinct
> > > > >>> means of specifying a limited DMA-width; the above fails, because it
> > > > >>> assumes a *device-tree*.
> > > > >>
> > > > >> Do you know if it would be possible to infer from the ACPI static table
> > > > >> the DMA-width?
> > > > > 
> > > > > Yes, and it is.  Due to not knowing much about ACPI tables I don't know
> > > > > what the C code would look like though (problem is which documentation
> > > > > should I be looking at first?).
> > > > 
> > > > What you provided below is an excerpt of the DSDT. AFAIK, DSDT content 
> > > > is written in AML. So far the shortest implementation I have seen for 
> > > > the AML parser is around 5000 lines (see [1]). It might be possible to 
> > > > strip some the code, although I think this will still probably too big 
> > > > for a single workaround.
> > > > 
> > > > What I meant by "static table" is a table that looks like a structure 
> > > > and can be parsed in a few lines. If we can't find on contain the DMA 
> > > > window, then the next best solution is to find a way to identity the 
> > > > platform.
> > > > 
> > > > I don't know enough ACPI to know if this solution is possible. A good 
> > > > starter would probably be the ACPI spec [2].
> > > 
> > > Okay, that is worse than I had thought (okay, ACPI is impressively
> > > complex for something nominally firmware-level).
> > >
> > > There are strings in the present Tianocore implementation for Raspberry
> > > PI 4B which could be targeted.  Notably included in the output during
> > > boot listing the tables, "RPIFDN", "RPIFDN RPI" and "RPIFDN RPI4" (I'm
> > > unsure how kosher these are as this wsn't implemented nor blessed by the
> > > Raspberry PI Foundation).
> > > 
> > > I strongly dislike this approach as you soon turn the Xen project into a
> > > database of hardware.  This is already occurring with
> > > xen/arch/arm/platforms and I would love to do something about this.  On
> > > that thought, how about utilizing Xen's command-line for this purpose?
> > 
> > I don't think that a command line option is a good idea: basically it is
> > punting to users the task of platform detection. Also, it means that
> > users will be necessarily forced to edit the uboot script or grub
> > configuration file to boot.
> 
> -EINVAL
> 
> Many Linux installations (near universal on desktop/server, but may be
> uncommon on ARM servers) Xen's command-line comes from grub.cfg.
> grub.cfg is in turn created by a series of scripts with several places
> for users to modify configuration without breaking things.
> 
> The scripts which create grub.cfg could add a "dma_mem=" option to Xen's
> command-line based upon what the running kernel reports.  If the kernel
> is running on top of Xen, it will still be able to retrieve this
> information out of ACPI.
> 
> This does mean distributions would need to modify scripts, but that is
> doable.  This also means a dumb user could potentially jump in, modify
> the value and thus cause unrecoverable breakage.  Yet on the flip side
> this also allows for the short-term stop-gap of smart users modifying
> the option as appropriate for new hardware.
> 
> Certainly it may not be the greatest approach, but it isn't as bad as
> you're claiming.

From what I have seen in previous years getting all the distros to
update their scripts is actually a lot of work. It would take a long
time to see it through. A few years. I think it is likely we would solve
the problem more quickly with a different solution.

 
> > Note that even if we introduced a new command line, we wouldn't take
> > away the need for xen/arch/arm/platforms anyway.
> 
> Perhaps, but it could allow for this setting at least to be moved to
> somewhere outside of Xen.

I was trying to say that it is not just about dma_mem=. There are other
things under xen/arch/arm/platforms that are sometimes required.  In the
specific case of the RPi4, if we added a general dma_mem= parameter, we
would still need to detect the platform to be able to do a platform
reboot, unless it is guaranteed that when ACPI is present ATF is also
present?

More on this below.


> I'm inclined to agree with Juergen Gro??, this reads kind of like having
> an extra domain run during Xen's initialization which can talk ACPI.
> 
> > > Have a procedure of during installation/updates retrieve DMA limitation
> > > information from the running OS and the following boot Xen will follow
> > > the appropriate setup.  By its nature, Domain 0 will have the information
> > > needed, just becomes an issue of how hard that is to retrieve...
> > 
> > Historically that is what we used to do for many things related to ACPI,
> > but unfortunately it leads to a pretty bad architecture where Xen
> > depends on Dom0 for booting rather than the other way around. (Dom0
> > should be the one requiring Xen for booting, given that Xen is higher
> > privilege and boots first.)
> > 
> > 
> > I think the best compromise is still to use an ACPI string to detect the
> > platform. For instance, would it be possible to use the OEMID fields in
> > RSDT, XSDT, FADT?  Possibly even a combination of them?
> > 
> > Another option might be to get the platform name from UEFI somehow. 
> 
> I included appropriate strings in e-mail.  Suitable strings do appear in
> `dmesg`.

Do you mean the ones mentioned in the previous email or did you forget
an attachment here? I don't think you added the value of OEMID anywhere?


> Problem is this feels like you're hard-coding a fixed list of platforms
> Xen can run on.  Instead values like these should be provided by
> firmware.  ACPI includes a method for encoding DMA limitations,

I see where you are coming from: DMA limitations is a general problem,
so it looks like there should be a general solution.

Yes, I agree that it would be good to have a general solution to
automatically detect DMA limitations, no matter if we do ACPI or device
tree.

However, xen/arch/arm/platforms is not just for DMA limitations. There
are per-platform quirks and workarounds that sometimes are required for
booting. Even if we solved the DMA limitations problem generically
(which would be good if we find a way), we are still likely to have to
find another way to detect the platform name.


> device-tree really should have one added.  Only challenge for
> device-tree is getting everyone to agree on names and parameters.

Device tree has ways to describe dma limitations, but they are not
really taken into account when allocating dom0 memory today
unfortunately :-/


> Looking at it, there are really issues with the allocate_memory_11()
> function in xen/arch/arm/domain_build.c.  Two tasks have been merged and
> I'm unsure they were merged correctly.
> 
> I'm unaware of any Xen-capable platforms with such, but DMA can have
> distinct restrictions outside of what allocate_memory_11() provides for.
> ACPI allows for explicit address ranges and in the past many devices have
> used addresses that didn't start at zero.
> 
> Additionally a device might have several devices with restricted DMA and
> these could have differing non-overlapping ranges.  Domain 0 might need
> memory in several DMA ranges.  Luckily In the past few years I haven't
> read about any potentially Xen-capable devices where DMA-capable memory
> had differing capabilities/performance from non-DMA-capable memory.

What you described is possible, at least theoretically. Any patches to
improve allocate_memory_11() in that respect would be great to have.


> Meanwhile for a platform which does have DMA limitations, the kernel and
> boot information for domain 0 shouldn't be placed in DMA-capable memory
> if domain 0 has any memory which isn't DMA-capable.  Yet it appears
> allocate_memory_11() will cause kernel/initrd/boot information to be
> placed in DMA-capable memory.

The dom0 kernel (if zImage) needs to be loaded in the first 128Mb of
RAM. In any case, the kernel will relocate those modules in high memory
and free the original memory during boot, so it shouldn't make a lot of
difference?


> If my understanding is correct (this is a BIG IF), as a last step
> allocate_memory_11() should reverse the order of memory banks.  Another
> trick is DMA-capable banks need to be subject to ballooning AFTER
> non-DMA-capable banks (I'm unsure how often ballooning is used on ARM).

Why reverse the order?

Re: Xen on RP4

[Ash Wilding]

Hi,


>> I think the best compromise is still to use an ACPI string to detect
>> the platform. For instance, would it be possible to use the OEMID
>> fields in RSDT, XSDT, FADT?  Possibly even a combination of them?
>>
>> Another option might be to get the platform name from UEFI somehow. 
>
> I included appropriate strings in e-mail.  Suitable strings do appear
> in `dmesg`.


Just as a heads-up, SMCCC does define the optional SMCCC_ARCH_SOC_ID [1]
function and this is listed as mandatory in the Server Base Boot Reqs
(SBBR); see pp 15 of ARM DEN 0044F [2].

Unfortunately it looks like RPi 4's firmware doesn't currently support
this, or at least the rpi4-uefi project [3] didn't think so as of FW
version 1.6 [4], but I couldn't find equivalent SBBR feature tracking
pages on that site for FW versions 1.7 or 1.8 to confirm, nor could I
find any reference to SMCCC_ARCH_SOC_ID in the RPi 4 FW sources [5].

On the bright side, while not very helpful in the short-term, note that
Arm's recently announced SystemReady [6] program is an evolution of
ServerReady (SBSA+SBBR) but for other segments and applications incl.
Embedded, IoT, and general Linux Boot.

That means in future we should see more platform firmware supporting
SMCCC_ARCH_SOC_ID, as the SiPs will (hopefully) want their platforms to
be SystemReady certified.

Hope that's useful info.

Thanks,
Ash.

[1] https://developer.arm.com/documentation/den0028/c
[2] https://developer.arm.com/documentation/den0044/latest
[3] https://rpi4-uefi.dev/about/
[4] https://rpi4-uefi.dev/status-v1-6-firmware/
[5] https://github.com/pftf/RPi4/tree/master
[6] https://developer.arm.com/architectures/system-architectures/arm-systemready

Re: Xen on RP4

[Stefano Stabellini]

On Sun, 1 Nov 2020, Ash Wilding wrote:
> >> I think the best compromise is still to use an ACPI string to detect
> >> the platform. For instance, would it be possible to use the OEMID
> >> fields in RSDT, XSDT, FADT?  Possibly even a combination of them?
> >>
> >> Another option might be to get the platform name from UEFI somehow. 
> >
> > I included appropriate strings in e-mail.  Suitable strings do appear
> > in `dmesg`.
> 
> 
> Just as a heads-up, SMCCC does define the optional SMCCC_ARCH_SOC_ID [1]
> function and this is listed as mandatory in the Server Base Boot Reqs
> (SBBR); see pp 15 of ARM DEN 0044F [2].

Thanks for sharing, it is good to know there is a "proper" way to do it.


> Unfortunately it looks like RPi 4's firmware doesn't currently support
> this, or at least the rpi4-uefi project [3] didn't think so as of FW
> version 1.6 [4], but I couldn't find equivalent SBBR feature tracking
> pages on that site for FW versions 1.7 or 1.8 to confirm, nor could I
> find any reference to SMCCC_ARCH_SOC_ID in the RPi 4 FW sources [5].

Well, call me an optimist but maybe it is just one patch away from
happening :-)


> On the bright side, while not very helpful in the short-term, note that
> Arm's recently announced SystemReady [6] program is an evolution of
> ServerReady (SBSA+SBBR) but for other segments and applications incl.
> Embedded, IoT, and general Linux Boot.
> 
> That means in future we should see more platform firmware supporting
> SMCCC_ARCH_SOC_ID, as the SiPs will (hopefully) want their platforms to
> be SystemReady certified.
> 
> Hope that's useful info.
> 
> Thanks,
> Ash.
> 
> [1] https://developer.arm.com/documentation/den0028/c
> [2] https://developer.arm.com/documentation/den0044/latest
> [3] https://rpi4-uefi.dev/about/
> [4] https://rpi4-uefi.dev/status-v1-6-firmware/
> [5] https://github.com/pftf/RPi4/tree/master
> [6] https://developer.arm.com/architectures/system-architectures/arm-systemready
> 

Re: Xen on RP4

[Elliott Mitchell]

I finally have U-Boot -> GRUB -> Xen sort-of operational as an
alternative to Tianocore -> GRUB -> Xen on a Raspberry PI 4B.

Stefano Stabellini, how much of the Raspberry PI 4B hardware have you
observed being operational under Linux on Xen?  In particular, have you
ever observed operational graphical output from a Raspberry Pi 4B running
Xen?

Presently I'm using a 5.8 kernel with your patches and haven't seen
graphical output under Xen with either boot stack.  I've confirmed fully
operational graphics without Xen on Tianocore, I've confirmed operational
virtual terminals with U-Boot and not Xen.

I had been planning to wait a bit before moving to 5.9, but if that is
the crucial ingredient I'll move early.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Roman Shaposhnik]

On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> I finally have U-Boot -> GRUB -> Xen sort-of operational as an

Yup -- same as what we're using in EVE -- we're now on the same page ;-)

> alternative to Tianocore -> GRUB -> Xen on a Raspberry PI 4B.
>
> Stefano Stabellini, how much of the Raspberry PI 4B hardware have you
> observed being operational under Linux on Xen?  In particular, have you
> ever observed operational graphical output from a Raspberry Pi 4B running
> Xen?

Pretty much everything is fully operational including graphics (but
not native one -- the default LK one)

> Presently I'm using a 5.8 kernel with your patches and haven't seen
> graphical output under Xen with either boot stack.  I've confirmed fully
> operational graphics without Xen on Tianocore, I've confirmed operational
> virtual terminals with U-Boot and not Xen.
>
> I had been planning to wait a bit before moving to 5.9, but if that is
> the crucial ingredient I'll move early.

We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.

I can bet $10 whatever it is -- it is DT related ;-)

Thanks,
Roman.

Re: Xen on RP4

[Elliott Mitchell]

On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > graphical output under Xen with either boot stack.  I've confirmed fully
> > operational graphics without Xen on Tianocore, I've confirmed operational
> > virtual terminals with U-Boot and not Xen.
> >
> > I had been planning to wait a bit before moving to 5.9, but if that is
> > the crucial ingredient I'll move early.
> 
> We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.
> 
> I can bet $10 whatever it is -- it is DT related ;-)

Given how many of the pieces I'm assembling are alpha or beta level, I
estimate a 50:50 chance on that.  Good odds it is device-tree, but good
odds I grabbed a bad version of $something.

I mostly wanted to know whether I was in completely uncharted territory
and needed to wait for others to catch up, versus merely working in a
situation where support is funky and I'm at an unknown location in
charted territory.

I'll be keeping the Tianocore setup available since Xen on ARM really
/should/ allow ACPI...


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Roman Shaposhnik]

On Tue, Nov 24, 2020 at 8:41 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> > On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > > graphical output under Xen with either boot stack.  I've confirmed fully
> > > operational graphics without Xen on Tianocore, I've confirmed operational
> > > virtual terminals with U-Boot and not Xen.
> > >
> > > I had been planning to wait a bit before moving to 5.9, but if that is
> > > the crucial ingredient I'll move early.
> >
> > We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.
> >
> > I can bet $10 whatever it is -- it is DT related ;-)
>
> Given how many of the pieces I'm assembling are alpha or beta level, I
> estimate a 50:50 chance on that.  Good odds it is device-tree, but good
> odds I grabbed a bad version of $something.
>
> I mostly wanted to know whether I was in completely uncharted territory
> and needed to wait for others to catch up, versus merely working in a
> situation where support is funky and I'm at an unknown location in
> charted territory.
>
> I'll be keeping the Tianocore setup available since Xen on ARM really
> /should/ allow ACPI...

I don't think you're in the uncharted -- so perhaps a bit of debugging left.

And, of course, alway feel free to compare what we do -- the image is
docker pull away.

Thanks,
Roman.

Re: Xen on RP4

[Elliott Mitchell]

On Tue, Nov 24, 2020 at 08:45:32PM -0800, Roman Shaposhnik wrote:
> On Tue, Nov 24, 2020 at 8:41 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> >
> > On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> > > On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > > > graphical output under Xen with either boot stack.  I've confirmed fully
> > > > operational graphics without Xen on Tianocore, I've confirmed operational
> > > > virtual terminals with U-Boot and not Xen.
> > > >
> > > > I had been planning to wait a bit before moving to 5.9, but if that is
> > > > the crucial ingredient I'll move early.
> > >
> > > We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.
> > >
> > > I can bet $10 whatever it is -- it is DT related ;-)
> >
> > Given how many of the pieces I'm assembling are alpha or beta level, I
> > estimate a 50:50 chance on that.  Good odds it is device-tree, but good
> > odds I grabbed a bad version of $something.
> >
> > I mostly wanted to know whether I was in completely uncharted territory
> > and needed to wait for others to catch up, versus merely working in a
> > situation where support is funky and I'm at an unknown location in
> > charted territory.
> >
> > I'll be keeping the Tianocore setup available since Xen on ARM really
> > /should/ allow ACPI...
> 
> I don't think you're in the uncharted -- so perhaps a bit of debugging left.
> 
> And, of course, alway feel free to compare what we do -- the image is
> docker pull away.

Actually, since device-tree is very much on my list of concerns, what is
your Xen boot process setup like?

Presently as previously mentioned I'm trying for
U-Boot -> GRUB/EFI -> Xen.  According to the information I currently have
the device-trees are often tied pretty closely to the kernel.  I'm also
using GRUB 2.04 since that has proper support for loading Xen on ARM.

The section of grub.cfg for Linux is roughly:
    linux /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
    devicetree /boot/dtb-5.8.10-2rp4-6.1.7
    initrd /boot/initrd.img-5.8.10-2rp4-6.1.7

My testing section for Xen is:
    xen_hypervisor /boot/xen-4.14-arm64.efi
    xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
    devicetree /boot/dtb-5.8.10-2rp4-6.1.7
    xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7

I've frankly got no idea how to ensure the correct device-tree is passed
to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
device-tree matched to the Linux kernel appropriate for Xen?

(I'm guessing the second is "yes", but the first I don't have a clue)


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Stefano Stabellini]

On Tue, 24 Nov 2020, Elliott Mitchell wrote:
> On Tue, Nov 24, 2020 at 08:45:32PM -0800, Roman Shaposhnik wrote:
> > On Tue, Nov 24, 2020 at 8:41 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > >
> > > On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> > > > On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > > > > graphical output under Xen with either boot stack.  I've confirmed fully
> > > > > operational graphics without Xen on Tianocore, I've confirmed operational
> > > > > virtual terminals with U-Boot and not Xen.
> > > > >
> > > > > I had been planning to wait a bit before moving to 5.9, but if that is
> > > > > the crucial ingredient I'll move early.
> > > >
> > > > We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.
> > > >
> > > > I can bet $10 whatever it is -- it is DT related ;-)
> > >
> > > Given how many of the pieces I'm assembling are alpha or beta level, I
> > > estimate a 50:50 chance on that.  Good odds it is device-tree, but good
> > > odds I grabbed a bad version of $something.
> > >
> > > I mostly wanted to know whether I was in completely uncharted territory
> > > and needed to wait for others to catch up, versus merely working in a
> > > situation where support is funky and I'm at an unknown location in
> > > charted territory.
> > >
> > > I'll be keeping the Tianocore setup available since Xen on ARM really
> > > /should/ allow ACPI...
> > 
> > I don't think you're in the uncharted -- so perhaps a bit of debugging left.
> > 
> > And, of course, alway feel free to compare what we do -- the image is
> > docker pull away.
> 
> Actually, since device-tree is very much on my list of concerns, what is
> your Xen boot process setup like?
> 
> Presently as previously mentioned I'm trying for
> U-Boot -> GRUB/EFI -> Xen.  According to the information I currently have
> the device-trees are often tied pretty closely to the kernel.  I'm also
> using GRUB 2.04 since that has proper support for loading Xen on ARM.
> 
> The section of grub.cfg for Linux is roughly:
>     linux /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     initrd /boot/initrd.img-5.8.10-2rp4-6.1.7
> 
> My testing section for Xen is:
>     xen_hypervisor /boot/xen-4.14-arm64.efi
>     xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7
> 
> I've frankly got no idea how to ensure the correct device-tree is passed
> to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> device-tree matched to the Linux kernel appropriate for Xen?
> 
> (I'm guessing the second is "yes", but the first I don't have a clue)

Yes, devicetree is correct. I have not used the graphical output, so I
cannot help you there but yes the best bet is to use the devicetree that
comes with the kernel.

One thing I noticed is that you are missing some of the command line
arguments for Xen and Linux in your grub config. For instance on the Xen
line you want to have something like:

    dom0_mem=1024M console=dtuart sync_console

And on the Linux line you might want to have:

    console=tty0 console=hvc0

Re: Xen on RP4

[Elliott Mitchell]

On Wed, Nov 25, 2020 at 10:57:31AM -0800, Stefano Stabellini wrote:
> On Tue, 24 Nov 2020, Elliott Mitchell wrote:
> > My testing section for Xen is:
> >     xen_hypervisor /boot/xen-4.14-arm64.efi
> >     xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
> >     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
> >     xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7
> > 
> > I've frankly got no idea how to ensure the correct device-tree is passed
> > to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> > device-tree matched to the Linux kernel appropriate for Xen?
> > 
> > (I'm guessing the second is "yes", but the first I don't have a clue)
> 
> Yes, devicetree is correct. I have not used the graphical output, so I
> cannot help you there but yes the best bet is to use the devicetree that
> comes with the kernel.

Well, I've now got everything together for a "proper" Debian Raspberry PI
installation.  Apparently since 5.2 (perhaps earlier, but 5.2 is
confirmed), Debian's kernel source packages have had their Raspberry PI
device-trees garbled.  I do have full untouched Linux kernel source
handy, but I tend to stick with the distribution until that proves
untenable.


> One thing I noticed is that you are missing some of the command line
> arguments for Xen and Linux in your grub config. For instance on the Xen
> line you want to have something like:
> 
>     dom0_mem=1024M console=dtuart sync_console
> 
> And on the Linux line you might want to have:
> 
>     console=tty0 console=hvc0

This plus adding the devicetree setting now turns into a report
elsewhere...


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Roman Shaposhnik]

On Wed, Nov 25, 2020 at 7:36 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Wed, Nov 25, 2020 at 10:57:31AM -0800, Stefano Stabellini wrote:
> > On Tue, 24 Nov 2020, Elliott Mitchell wrote:
> > > My testing section for Xen is:
> > >     xen_hypervisor /boot/xen-4.14-arm64.efi
> > >     xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
> > >     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
> > >     xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7
> > >
> > > I've frankly got no idea how to ensure the correct device-tree is passed
> > > to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> > > device-tree matched to the Linux kernel appropriate for Xen?
> > >
> > > (I'm guessing the second is "yes", but the first I don't have a clue)
> >
> > Yes, devicetree is correct. I have not used the graphical output, so I
> > cannot help you there but yes the best bet is to use the devicetree that
> > comes with the kernel.
>
> Well, I've now got everything together for a "proper" Debian Raspberry PI
> installation.  Apparently since 5.2 (perhaps earlier, but 5.2 is
> confirmed), Debian's kernel source packages have had their Raspberry PI
> device-trees garbled.  I do have full untouched Linux kernel source
> handy, but I tend to stick with the distribution until that proves
> untenable.

Yup. Same here. I started with upstream kernel, wasted a lot of time,
threw in the towel
and imported all of the RPi Foundation patches wholesale :-(

Thanks,
Roman.

Re: Xen on RP4

[Elliott Mitchell]

On Fri, Nov 27, 2020 at 11:59:10PM -0800, Roman Shaposhnik wrote:
> On Wed, Nov 25, 2020 at 7:36 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > Well, I've now got everything together for a "proper" Debian Raspberry PI
> > installation.  Apparently since 5.2 (perhaps earlier, but 5.2 is
> > confirmed), Debian's kernel source packages have had their Raspberry PI
> > device-trees garbled.  I do have full untouched Linux kernel source
> > handy, but I tend to stick with the distribution until that proves
> > untenable.
> 
> Yup. Same here. I started with upstream kernel, wasted a lot of time,
> threw in the towel
> and imported all of the RPi Foundation patches wholesale :-(

Any chance you could send e-mail to 939633@bugs.debian.org and state you
have also observed corrupt device-trees with Debian's kernel somewhere
in the >=5.2 range?

I'm a half step away from marking that bug "critical" (breaks most/all
Raspberry PI variants, this breaks Debian's Xen build).  I just tend to
be reluctant...

I can almost believe Intel/Tianocore are trying to create a FUD situation
for device-trees in order to push UEFI.  Debian has been cited by the
Tianocore folks as an example of a Linux distribution which can readily
install without modification.  Just a small push to kill device-trees.

Presently I have no evidence of this, just a niggling suspicion.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Elliott Mitchell]

On Wed, Nov 25, 2020 at 10:57:31AM -0800, Stefano Stabellini wrote:
> On Tue, 24 Nov 2020, Elliott Mitchell wrote:
> > I've frankly got no idea how to ensure the correct device-tree is passed
> > to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> > device-tree matched to the Linux kernel appropriate for Xen?
> > 
> > (I'm guessing the second is "yes", but the first I don't have a clue)
> 
> Yes, devicetree is correct. I have not used the graphical output, so I
> cannot help you there but yes the best bet is to use the devicetree that
> comes with the kernel.
> 
> One thing I noticed is that you are missing some of the command line
> arguments for Xen and Linux in your grub config. For instance on the Xen
> line you want to have something like:
> 
>     dom0_mem=1024M console=dtuart sync_console
> 
> And on the Linux line you might want to have:
> 
>     console=tty0 console=hvc0

I was sending the bare minimum.  Some of the known bits were filtered
out.  After having spent several hours pounding on this and building
multiple kernels, I'm headed towards odd theories...

I'm wondering how Debian's kernel source trees have managed to remain
broken for the Raspberry PIs for over a year:
https://bugs.debian.org/939633

Right now that feels like conspiracy theory territory, but my mind is
wandering in odd directions...   Is someone at Intel trying to sabotage
device-trees so everyone moves to UEFI?

Could simply be Debian's kernel maintainers are very busy and the
original reporter of the bug failed to draw enough attention to a large
problem.  If that odd suspicion is true though, getting EFI to supported
status on ARM might be a major concern.

Alternatively I've been exploring an incorrect path, I should simply
stick with the device-trees which come from the Raspberry PI Foundation,
and not try to follow the Linux kernel's device-trees.


-- 
(\___(\___(\______          --=> 8-) EHM <=--          ______/)___/)___/)
 \BS (    |         ehem+sigmsg@m5p.com  PGP 87145445         |    )   /
  \_CS\   |  _____  -O #include <stddisclaimer.h> O-   _____  |   /  _/
8A19\___\_|_/58D2 7E3D DDF4 7BA6 <-PGP-> 41D1 B375 37D0 8714\_|_/___/5445

Re: Xen on RP4

[Roman Shaposhnik]

On Tue, Nov 24, 2020 at 9:17 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
>
> On Tue, Nov 24, 2020 at 08:45:32PM -0800, Roman Shaposhnik wrote:
> > On Tue, Nov 24, 2020 at 8:41 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > >
> > > On Tue, Nov 24, 2020 at 08:01:32PM -0800, Roman Shaposhnik wrote:
> > > > On Tue, Nov 24, 2020 at 7:37 PM Elliott Mitchell <ehem+xen@m5p.com> wrote:
> > > > > Presently I'm using a 5.8 kernel with your patches and haven't seen
> > > > > graphical output under Xen with either boot stack.  I've confirmed fully
> > > > > operational graphics without Xen on Tianocore, I've confirmed operational
> > > > > virtual terminals with U-Boot and not Xen.
> > > > >
> > > > > I had been planning to wait a bit before moving to 5.9, but if that is
> > > > > the crucial ingredient I'll move early.
> > > >
> > > > We're still using 5.4 -- but it seems that the next LTS 5.10 is also functional.
> > > >
> > > > I can bet $10 whatever it is -- it is DT related ;-)
> > >
> > > Given how many of the pieces I'm assembling are alpha or beta level, I
> > > estimate a 50:50 chance on that.  Good odds it is device-tree, but good
> > > odds I grabbed a bad version of $something.
> > >
> > > I mostly wanted to know whether I was in completely uncharted territory
> > > and needed to wait for others to catch up, versus merely working in a
> > > situation where support is funky and I'm at an unknown location in
> > > charted territory.
> > >
> > > I'll be keeping the Tianocore setup available since Xen on ARM really
> > > /should/ allow ACPI...
> >
> > I don't think you're in the uncharted -- so perhaps a bit of debugging left.
> >
> > And, of course, alway feel free to compare what we do -- the image is
> > docker pull away.
>
> Actually, since device-tree is very much on my list of concerns, what is
> your Xen boot process setup like?
>
> Presently as previously mentioned I'm trying for
> U-Boot -> GRUB/EFI -> Xen.

Exactly the same. Here's what we put on a vfat partition:
     https://github.com/lf-edge/eve/tree/master/pkg/u-boot/rpi
and here's how u-boot is built:
     https://github.com/lf-edge/eve/blob/master/pkg/u-boot/Dockerfile

> According to the information I currently have
> the device-trees are often tied pretty closely to the kernel.  I'm also
> using GRUB 2.04 since that has proper support for loading Xen on ARM.

Yes. Our DT here:
    https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/bcm2711-rpi-4-b.dtb
came from an honest build of our kernel (our build is still in flux -- hence
a quick hack of keeping a blob):
    https://github.com/lf-edge/eve/blob/master/pkg/kernel/Dockerfile#L154

> The section of grub.cfg for Linux is roughly:
>     linux /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     initrd /boot/initrd.img-5.8.10-2rp4-6.1.7
>
> My testing section for Xen is:
>     xen_hypervisor /boot/xen-4.14-arm64.efi
>     xen_module /boot/vmlinuz-5.8.10-2rp4-6.1.7 root=UUID=01234567-dead-beef-d13d-456789abcdef ro
>     devicetree /boot/dtb-5.8.10-2rp4-6.1.7
>     xen_module --nounzip /boot/initrd.img-5.8.10-2rp4-6.1.7

Roughly the same -- but see Stefano's comment. More here:
    https://github.com/lf-edge/eve/blob/master/pkg/grub/rootfs.cfg

> I've frankly got no idea how to ensure the correct device-tree is passed
> to Xen.  Is GRUB's `devicetree` command correct when loading Xen?  Is a
> device-tree matched to the Linux kernel appropriate for Xen?
>
> (I'm guessing the second is "yes", but the first I don't have a clue)

While you can use `devicetree`  in GRUB. E.g.:
    https://github.com/lf-edge/eve/blob/master/pkg/grub/rootfs.cfg#L207
on EVE side we've only ever used it as an emergency override.

The happy path boot sequence preserves the DT that RPi bootloader
makes available to u-boot and it gets passed down the chain without
anybody doing anything.

Hope this helps.

Thanks,
Roman.