[Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Jorge Pereira]

[-- Attachment #1.1: Type: text/plain, Size: 656 bytes --]

Hi Guys,

I'm currently using XEN in order to run side-by-side a DOM-0 with a DOM-U guest. My use-case scenario requires in the DOM-U direct access to some dma-capable devices such ethernet and some GPUs.

Since our target platform (i.MX8MM) does not support IOMMU, we can't assign dma-capable devices to the DOM-U guest because XEN does not create 1:1 mapping for that guest in the 2nd stage MMU. So, guest-virtual addresses are different than the physical ones.

Is it possible to have 1:1 mapping for DOM-U guests? If not, I'm interested to know what would be the estimated effort to support this feature?

Thanks in advance,

Cheers,
Jorge

[-- Attachment #1.2: Type: text/html, Size: 2909 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Julien Grall]

On 14/01/2020 21:39, Jorge Pereira wrote:
> Hi Guys,

Hello,

> 
> I’m currently using XEN in order to run side-by-side a DOM-0 with a 
> DOM-U guest. My use-case scenario requires in the DOM-U direct access to 
> some dma-capable devices such ethernet and some GPUs.
> 
> Since our target platform (i.MX8MM) does not support IOMMU, we can’t 
> assign dma-capable devices to the DOM-U guest because XEN does not 
> create 1:1 mapping for that guest in the 2^nd stage MMU. So, 
> guest-virtual addresses are different than the physical ones.

Bear in mind this setup is going to be insecure unless you have another 
way to prevent your passthrough-ed device to access memory it should not 
(e.g an MPU).

> Is it possible to have 1:1 mapping for DOM-U guests?

It is not possible at the moment. There are been various effort to try 
to do it, but I have always push back as this is actively defeating the 
purposing of an hypervisor.

This would be a different story if we had support for MPU in Xen.

> If not, I’m 
> interested to know what would be the estimated effort to support this 
> feature?

I think you have someone else in NXP looking at 1:1 mapping for Xen (in 
CC). I provided to Andrei some tips how to get 1:1 mapping for DomU 
using dom0less in December (see [1]). So you may want to sync-up with 
him here.

If you are looking at 1:1 DomU using xl, then it is going to require 
more work as the hypercall allocating memory is based on guest frame 
number. There was a thread on the ML a few years ago, I can try to dig 
it down if you are interested.

Cheers,

[1] 
https://lists.xenproject.org/archives/html/xen-devel/2019-12/msg01364.html

> 
> Thanks in advance,
> 
> Cheers,
> 
> Jorge
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 3016 bytes --]

On Wed, 15 Jan 2020, Julien Grall wrote:
> On 14/01/2020 21:39, Jorge Pereira wrote:
> > Hi Guys,
> 
> Hello,
> 
> > 
> > I’m currently using XEN in order to run side-by-side a DOM-0 with a DOM-U
> > guest. My use-case scenario requires in the DOM-U direct access to some
> > dma-capable devices such ethernet and some GPUs.
> > 
> > Since our target platform (i.MX8MM) does not support IOMMU, we can’t assign
> > dma-capable devices to the DOM-U guest because XEN does not create 1:1
> > mapping for that guest in the 2^nd stage MMU. So, guest-virtual addresses
> > are different than the physical ones.
> 
> Bear in mind this setup is going to be insecure unless you have another way to
> prevent your passthrough-ed device to access memory it should not (e.g an
> MPU).
> 
> > Is it possible to have 1:1 mapping for DOM-U guests?

I have a patch that enables 1:1 mapping of memory for dom0-less DomUs,
see attached. It introduces a new option called "direct-map" in the domU
specific device tree section. When direct-map is found, the memory gets
allocated 1:1 for the DomU. direct-map can be used in conjuction with
the existing xen,force-assign-without-iommu parameter to enable device
assignment to domUs without a SMMU.

Note that direct-map is supposed to go under /chosen in the main device
tree, while xen,force-assign-without-iommu is supposed to go under
/passthrough in the partial device tree for device assignment.


The patch is only lightly tested and might not work on your platform.
Also, in addition to the security concerns Julien pointed out, it is
very possible to run into trouble with other static addresses.
Currently, we have a number of resources with fixed addresses in the
DomU address space. Give a look at xen/include/public/arch-arm.h: the
domU GICD is fixed at 0x03001000 for example. If the 1:1 memory
allocation selects a memory region that conflicts with any of those
fixed addresses, you are going to have a problem. I am not even sure
that Xen will be able to detect the error and fail explicitly: it could
just silently fail to boot.



> It is not possible at the moment. There are been various effort to try to do
> it, but I have always push back as this is actively defeating the purposing of
> an hypervisor.
> 
> This would be a different story if we had support for MPU in Xen.

A MPU is a requirement to make 1:1 secure, however, it would be
difficult to add support for it in Xen: on Xilinx platforms, the MPU is
not only for VM/VM protection but also for Cortex-Rs/Cortex-As
protection so it is typically programmed beforehand by somebody with a
system wide view (Xen has only a Cortex-A view of the system.) Xen might
not be the right place to configure the MPU at least on Xilinx boards.

This makes me realize that maybe we need to be able to allow the user to
specify not just that she wants direct mapping (like I did in the
attached patch), but also that she wants a specific range of memory for
each DomU, so that she can go and configure the MPU herself.

[-- Attachment #2: Type: text/plain, Size: 4137 bytes --]

diff --git a/docs/misc/arm/passthrough-noiommu.txt b/docs/misc/arm/passthrough-noiommu.txt
new file mode 100644
index 0000000000..d9b7a9b109
--- /dev/null
+++ b/docs/misc/arm/passthrough-noiommu.txt
@@ -0,0 +1,29 @@
+Disable IOMMU
+=============
+
+Add status = "disabled"; under the smmu node:
+
+		smmu@fd800000 {
+			compatible = "arm,mmu-500";
+			status = "disabled";
+
+
+Request Device Assignment without IOMMU support
+===============================================
+
+Add xen,force-assign-without-iommu; to the device tree snippet
+
+		ethernet: ethernet@ff0e0000 {
+			compatible = "cdns,zynqmp-gem";
+			xen,path = "/amba/ethernet@ff0e0000";
+			xen,reg = <0x0 0xff0e0000 0x1000 0x0 0xff0e0000>;
+			xen,force-assign-without-iommu;
+
+
+Request 1:1 memory mapping for the domain
+=========================================
+
+Add direct-map under the appropriate /chosen/domU node.
+If the user is using imagebuilder, you can add to boot.source:
+
+fdt set /chosen/domU0 direct-map <0x1>
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index dd9c3b73ba..51ec7d76db 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -269,9 +269,9 @@ static void __init allocate_memory_11(struct domain *d,
      */
     BUG_ON(!is_domain_direct_mapped(d));
 
-    printk("Allocating 1:1 mappings totalling %ldMB for dom0:\n",
+    printk("Allocating 1:1 mappings totalling %ldMB for dom%u:\n",
            /* Don't want format this as PRIpaddr (16 digit hex) */
-           (unsigned long)(kinfo->unassigned_mem >> 20));
+           (unsigned long)(kinfo->unassigned_mem >> 20), d->domain_id);
 
     kinfo->mem.nr_banks = 0;
 
@@ -2434,7 +2434,11 @@ static int __init construct_domU(struct domain *d,
     /* type must be set before allocate memory */
     d->arch.type = kinfo.type;
 #endif
-    allocate_memory(d, &kinfo);
+
+    if ( !is_domain_direct_mapped(d) )
+        allocate_memory(d, &kinfo);
+    else
+        allocate_memory_11(d, &kinfo);
 
     rc = prepare_dtb_domU(d, &kinfo);
     if ( rc < 0 )
@@ -2454,6 +2458,8 @@ void __init create_domUs(void)
 {
     struct dt_device_node *node;
     const struct dt_device_node *chosen = dt_find_node_by_path("/chosen");
+    int rc;
+    u32 direct_map = 0;
 
     BUG_ON(chosen == NULL);
     dt_for_each_child_node(chosen, node)
@@ -2474,7 +2480,9 @@ void __init create_domUs(void)
             panic("Missing property 'cpus' for domain %s\n",
                   dt_node_name(node));
 
-        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
+        rc = dt_property_read_u32(node, "direct-map", &direct_map);
+        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+             (!rc || !direct_map) )
             d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
 
         if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
@@ -2495,6 +2503,7 @@ void __init create_domUs(void)
             panic("Error creating domain %s\n", dt_node_name(node));
 
         d->is_console = true;
+        d->arch.direct_map = (rc && direct_map != 0);
 
         if ( construct_domU(d, node) != 0 )
             panic("Could not set up domain %s\n", dt_node_name(node));
@@ -2524,6 +2533,7 @@ int __init construct_dom0(struct domain *d)
 
     iommu_hwdom_init(d);
 
+    d->arch.direct_map = true;
     d->max_pages = ~0U;
 
     kinfo.unassigned_mem = dom0_mem;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index f3f3fb7d7f..8b5e1542e5 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -32,7 +32,7 @@ enum domain_type {
 #endif
 
 /* The hardware domain has always its memory direct mapped. */
-#define is_domain_direct_mapped(d) ((d) == hardware_domain)
+#define is_domain_direct_mapped(d) ((d)->arch.direct_map != false)
 
 struct vtimer {
     struct vcpu *v;
@@ -101,6 +101,8 @@ struct arch_domain
 #ifdef CONFIG_TEE
     void *tee;
 #endif
+
+    bool direct_map;
 }  __cacheline_aligned;
 
 struct arch_vcpu

[-- Attachment #3: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Stefano Stabellini]

On Wed, 1 Apr 2020, Andrei Cherechesu wrote:
> > I cannot see any issues with the driver. Can you paste the device tree as dom0
> > sees it? You can access it from /proc/device-tree (you can convert it to dts
> > with dtc -I fs -O dts). And also the host device tree?
> > 
> > We should see something like the following (this example is taken from a Xilinx
> > ZynqMP):
> > 
> > 1) host device tree
> > 
> >         serial@ff000000 {
> >                 u-boot,dm-pre-reloc;
> >                 compatible = "cdns,uart-r1p12", "xlnx,xuartps";
> >                 status = "okay";
> >                 interrupt-parent = <0x4>;
> >                 interrupts = <0x0 0x15 0x4>;
> >                 reg = <0x0 0xff000000 0x0 0x1000>;
> >                 clock-names = "uart_clk", "pclk";
> >                 power-domains = <0x26 0x21>;
> >                 clocks = <0x3 0x38 0x3 0x1f>;
> >                 pinctrl-names = "default";
> >                 pinctrl-0 = <0x37>;
> >                 cts-override;
> >                 device_type = "serial";
> >                 port-number = <0x0>;
> >         };
> > 
> > 
> > 2) dom0 device tree
> > 
> >     The node is missing
> > 
> > 
> > The key is that dom0 should not see the UART node in device tree at all.
> > 
> > If Dom0 is seeing the node, then it is a problem with Xen that somehow is not
> > hiding it (see "Skip nodes used by Xen" under
> > xen/arch/arm/domain_build.c:handle_node)
> > 
> > If Dom0 is *not* seeing the node, that what is the underlying device tree node
> > that the dom0 driver is using to bring up /dev/ttyLF0?
> > 
> 
> I looked under /proc/device-tree and Dom0 actually sees the node corresponding
> to the serial device. That means Xen is not hiding it. 
> 
> I've pasted the host device tree at [0] and the Dom0 device tree at [1]
> [0] https://pastebin.com/ir7VkfES
> [1] https://pastebin.com/UqRDycHF
> 
> So this is the device in the host device tree:
> 
>     uart0:serial@401C8000 {
>         compatible = "fsl,s32-linflexuart";
>         reg = <0x0 0x401C8000 0x0 0x3000>;
>         interrupts = <0 82 1>;
>         clocks = <&clks S32GEN1_CLK_LIN_BAUD>,
>                 <&clks S32GEN1_CLK_LIN>;
>         clock-names = "lin", "ipg";
>         dmas = <&edma0 0 4>,
>                 <&edma0 0 3>;
>         dma-names = "rx", "tx";
>     };
> 
> 
> And this is the device in the Dom0 device tree:
> 
>     serial@401C8000 {
>         clock-names = "lin\0ipg";
>         interrupts = <0x00 0x52 0x01>;
>         clocks = <0x04 0x2a 0x04 0x2b>;
>         dma-names = "rx\0tx";
>         compatible = "fsl,s32-linflexuart";
>         reg = <0x00 0x401c8000 0x00 0x3000>;
>         dmas = <0x0e 0x00 0x04 0x0e 0x00 0x03>;
>     };

All right, now we know what the problem is :-)

If you are unsure, I would enable CONFIG_DEVICE_TREE_DEBUG and/or add a
printk or two to figure out why Xen is not filtering out the node:


diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 4307087536..fa6a108de8 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -1486,6 +1486,7 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
     /* Skip nodes used by Xen */
     if ( dt_device_used_by(node) == DOMID_XEN )
     {
+        printk("DEBUG skipping %s\n",path);
         dt_dprintk("  Skip it (used by Xen)\n");
         return 0;
     }

You might also want to add a printk just before the call to
dt_device_set_used_by(dev, DOMID_XEN) in your uart driver.



> From this story I figured out something else: that I was not passing to Xen's
> bootargs "console=dtuart dtuart=serial0". Maybe that's why Xen is not hiding
> the node from Dom0 (serial0 is an alias to uart0).

Yes, it is worth double-checking that Xen is using the right driver for
the right device and the right device tree node description. Print in
your driver not only that is getting called but also that is getting
called for the right device node.

But console=dtuart dtuart=serial0 seems correct with your device tree.


> So I went on and I added that to bootargs. And now my Xen boot stops here
> (DEBUG print on):
> ...
> ...
> (XEN) fixed up name for siul2_reg@0x4401170C -> siul2_reg
> (XEN) fixed up name for siul2_reg@0x4401174C -> siul2_reg
> (XEN) fixed up name for usbmisc@44064200 -> usbmisc
> (XEN) fixed up name for usb@44064000 -> usb
> (XEN) fixed up name for memory_DDR0@80000000 -> memory_DDR0
> (XEN) fixed up name for memory_DDR1@c0000000 -> memory_DDR1
> (XEN) fixed up name for memory_DDR2@880000000 -> memory_DDR2
> (XEN)  <- unflatten_device_tree()
> (XEN) adding DT alias:can0: stem=can id=0 node=/flexcan@401B4000
> (XEN) adding DT alias:can1: stem=can id=1 node=/flexcan@401BE000
> (XEN) adding DT alias:can2: stem=can id=2 node=/flexcan@402A8000
> (XEN) adding DT alias:can3: stem=can id=3 node=/flexcan@402B2000
> (XEN) adding DT alias:serial0: stem=serial id=0 node=/serial@401C8000
> (XEN) adding DT alias:serial1: stem=serial id=1 node=/serial@401CC000
> (XEN) adding DT alias:serial2: stem=serial id=2 node=/serial@402BC000
> (XEN) Platform: Generic System
> (XEN) Looking for dtuart at "serial0", options ""
> (XEN) DT: ** translation for device /serial@401C8000 **
> (XEN) DT: bus is default (na=2, ns=2) on /
> (XEN) DT: translating address:<3> 00000000<3> 401c8000<3>
> (XEN) DT: reached root node
> (XEN) dt_device_get_raw_irq: dev=/serial@401C8000, index=0
> (XEN)  intspec=0 intlen=3
> (XEN)  intsize=3 intlen=3
> (XEN) dt_irq_map_raw: par=/interrupt-controller@50800000,intspec=[0x00000000 0x00000052...],ointsize=3
> (XEN) dt_irq_map_raw: ipar=/interrupt-controller@50800000, size=3
> (XEN)  -> addrsize=2
> (XEN)  -> got it??? 
> 
> I started debugging and I found out that it hangs in:
> console_init_preirq() -> __putstr(xen_banner()) -> sercon_puts() -> serial_puts() -> __serial_putc(),
> where it spins at line 178:         
>         /* Synchronous finite-capacity transmitter. */
>         while ( !(n = port->driver->tx_ready(port)) )
>             cpu_relax();
>  
> Which is a bit strange, considering that my serial device is asynchronous,
> I think it should not get there. But it gets on that "else" branch because
> port->txbuf is actually NULL at line 120 when it performs the check, and
> it does not enter the branch for asynchronous transmitters.
 
What Xen base version are you using? It is not line 178 on master.

I would look a bit more closely into that driver. For instance, one
question I would ask myself is why is tx_ready not returning 1?
Something is probably not quite right in the initialization of that
driver.

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Julien Grall]

Hi,

On 01/04/2020 20:58, Andrei Cherechesu wrote:
>> I cannot see any issues with the driver. Can you paste the device tree as dom0
>> sees it? You can access it from /proc/device-tree (you can convert it to dts
>> with dtc -I fs -O dts). And also the host device tree?
>>
>> We should see something like the following (this example is taken from a Xilinx
>> ZynqMP):
>>
>> 1) host device tree
>>
>>          serial@ff000000 {
>>                  u-boot,dm-pre-reloc;
>>                  compatible = "cdns,uart-r1p12", "xlnx,xuartps";
>>                  status = "okay";
>>                  interrupt-parent = <0x4>;
>>                  interrupts = <0x0 0x15 0x4>;
>>                  reg = <0x0 0xff000000 0x0 0x1000>;
>>                  clock-names = "uart_clk", "pclk";
>>                  power-domains = <0x26 0x21>;
>>                  clocks = <0x3 0x38 0x3 0x1f>;
>>                  pinctrl-names = "default";
>>                  pinctrl-0 = <0x37>;
>>                  cts-override;
>>                  device_type = "serial";
>>                  port-number = <0x0>;
>>          };
>>
>>
>> 2) dom0 device tree
>>
>>      The node is missing
>>
>>
>> The key is that dom0 should not see the UART node in device tree at all.
>>
>> If Dom0 is seeing the node, then it is a problem with Xen that somehow is not
>> hiding it (see "Skip nodes used by Xen" under
>> xen/arch/arm/domain_build.c:handle_node)
>>
>> If Dom0 is *not* seeing the node, that what is the underlying device tree node
>> that the dom0 driver is using to bring up /dev/ttyLF0?
>>
> 
> I looked under /proc/device-tree and Dom0 actually sees the node corresponding
> to the serial device. That means Xen is not hiding it.
> 
> I've pasted the host device tree at [0] and the Dom0 device tree at [1]
> [0] https://pastebin.com/ir7VkfES
> [1] https://pastebin.com/UqRDycHF
> 
> So this is the device in the host device tree:
> 
>      uart0:serial@401C8000 {
>          compatible = "fsl,s32-linflexuart";
>          reg = <0x0 0x401C8000 0x0 0x3000>;
>          interrupts = <0 82 1>;
>          clocks = <&clks S32GEN1_CLK_LIN_BAUD>,
>                  <&clks S32GEN1_CLK_LIN>;
>          clock-names = "lin", "ipg";
>          dmas = <&edma0 0 4>,
>                  <&edma0 0 3>;
>          dma-names = "rx", "tx";
>      };
> 
> 
> And this is the device in the Dom0 device tree:
> 
>      serial@401C8000 {
>          clock-names = "lin\0ipg";
>          interrupts = <0x00 0x52 0x01>;
>          clocks = <0x04 0x2a 0x04 0x2b>;
>          dma-names = "rx\0tx";
>          compatible = "fsl,s32-linflexuart";
>          reg = <0x00 0x401c8000 0x00 0x3000>;
>          dmas = <0x0e 0x00 0x04 0x0e 0x00 0x03>;
>      };
> 
>  From this story I figured out something else: that I was not passing to Xen's
> bootargs "console=dtuart dtuart=serial0". Maybe that's why Xen is not hiding
> the node from Dom0 (serial0 is an alias to uart0).

Xen is not going to hide the UART from dom0 if it is not used by the 
console driver.

> 
> So I went on and I added that to bootargs. And now my Xen boot stops here
> (DEBUG print on):
> ...
> ...
> (XEN) fixed up name for siul2_reg@0x4401170C -> siul2_reg
> (XEN) fixed up name for siul2_reg@0x4401174C -> siul2_reg
> (XEN) fixed up name for usbmisc@44064200 -> usbmisc
> (XEN) fixed up name for usb@44064000 -> usb
> (XEN) fixed up name for memory_DDR0@80000000 -> memory_DDR0
> (XEN) fixed up name for memory_DDR1@c0000000 -> memory_DDR1
> (XEN) fixed up name for memory_DDR2@880000000 -> memory_DDR2
> (XEN)  <- unflatten_device_tree()
> (XEN) adding DT alias:can0: stem=can id=0 node=/flexcan@401B4000
> (XEN) adding DT alias:can1: stem=can id=1 node=/flexcan@401BE000
> (XEN) adding DT alias:can2: stem=can id=2 node=/flexcan@402A8000
> (XEN) adding DT alias:can3: stem=can id=3 node=/flexcan@402B2000
> (XEN) adding DT alias:serial0: stem=serial id=0 node=/serial@401C8000
> (XEN) adding DT alias:serial1: stem=serial id=1 node=/serial@401CC000
> (XEN) adding DT alias:serial2: stem=serial id=2 node=/serial@402BC000
> (XEN) Platform: Generic System
> (XEN) Looking for dtuart at "serial0", options ""
> (XEN) DT: ** translation for device /serial@401C8000 **
> (XEN) DT: bus is default (na=2, ns=2) on /
> (XEN) DT: translating address:<3> 00000000<3> 401c8000<3>
> (XEN) DT: reached root node
> (XEN) dt_device_get_raw_irq: dev=/serial@401C8000, index=0
> (XEN)  intspec=0 intlen=3
> (XEN)  intsize=3 intlen=3
> (XEN) dt_irq_map_raw: par=/interrupt-controller@50800000,intspec=[0x00000000 0x00000052...],ointsize=3
> (XEN) dt_irq_map_raw: ipar=/interrupt-controller@50800000, size=3
> (XEN)  -> addrsize=2
> (XEN)  -> got it�
> 
> I started debugging and I found out that it hangs in:
> console_init_preirq() -> __putstr(xen_banner()) -> sercon_puts() -> serial_puts() -> __serial_putc(),
> where it spins at line 178:
>          /* Synchronous finite-capacity transmitter. */
>          while ( !(n = port->driver->tx_ready(port)) )
>              cpu_relax();
>   
> Which is a bit strange, considering that my serial device is asynchronous,
> I think it should not get there. But it gets on that "else" branch because
> port->txbuf is actually NULL at line 120 when it performs the check, and
> it does not enter the branch for asynchronous transmitters.

The console is initialized much before the interrupt subsystem is 
initialized. Therefore you need synchronous support for some part of the 
boot.

Also, looking at your driver, you don't seem to call 
serial_async_transmist() in console_init_postirq() (see ns16550.c) so I 
am not sure how asynchronous would work.

Cheers,

-- 
Julien Grall

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Andrei Cherechesu]

> I cannot see any issues with the driver. Can you paste the device tree as dom0
> sees it? You can access it from /proc/device-tree (you can convert it to dts
> with dtc -I fs -O dts). And also the host device tree?
> 
> We should see something like the following (this example is taken from a Xilinx
> ZynqMP):
> 
> 1) host device tree
> 
>         serial@ff000000 {
>                 u-boot,dm-pre-reloc;
>                 compatible = "cdns,uart-r1p12", "xlnx,xuartps";
>                 status = "okay";
>                 interrupt-parent = <0x4>;
>                 interrupts = <0x0 0x15 0x4>;
>                 reg = <0x0 0xff000000 0x0 0x1000>;
>                 clock-names = "uart_clk", "pclk";
>                 power-domains = <0x26 0x21>;
>                 clocks = <0x3 0x38 0x3 0x1f>;
>                 pinctrl-names = "default";
>                 pinctrl-0 = <0x37>;
>                 cts-override;
>                 device_type = "serial";
>                 port-number = <0x0>;
>         };
> 
> 
> 2) dom0 device tree
> 
>     The node is missing
> 
> 
> The key is that dom0 should not see the UART node in device tree at all.
> 
> If Dom0 is seeing the node, then it is a problem with Xen that somehow is not
> hiding it (see "Skip nodes used by Xen" under
> xen/arch/arm/domain_build.c:handle_node)
> 
> If Dom0 is *not* seeing the node, that what is the underlying device tree node
> that the dom0 driver is using to bring up /dev/ttyLF0?
> 

I looked under /proc/device-tree and Dom0 actually sees the node corresponding
to the serial device. That means Xen is not hiding it. 

I've pasted the host device tree at [0] and the Dom0 device tree at [1]
[0] https://pastebin.com/ir7VkfES
[1] https://pastebin.com/UqRDycHF

So this is the device in the host device tree:

    uart0:serial@401C8000 {
        compatible = "fsl,s32-linflexuart";
        reg = <0x0 0x401C8000 0x0 0x3000>;
        interrupts = <0 82 1>;
        clocks = <&clks S32GEN1_CLK_LIN_BAUD>,
                <&clks S32GEN1_CLK_LIN>;
        clock-names = "lin", "ipg";
        dmas = <&edma0 0 4>,
                <&edma0 0 3>;
        dma-names = "rx", "tx";
    };


And this is the device in the Dom0 device tree:

    serial@401C8000 {
        clock-names = "lin\0ipg";
        interrupts = <0x00 0x52 0x01>;
        clocks = <0x04 0x2a 0x04 0x2b>;
        dma-names = "rx\0tx";
        compatible = "fsl,s32-linflexuart";
        reg = <0x00 0x401c8000 0x00 0x3000>;
        dmas = <0x0e 0x00 0x04 0x0e 0x00 0x03>;
    };

From this story I figured out something else: that I was not passing to Xen's
bootargs "console=dtuart dtuart=serial0". Maybe that's why Xen is not hiding
the node from Dom0 (serial0 is an alias to uart0).

So I went on and I added that to bootargs. And now my Xen boot stops here
(DEBUG print on):
...
...
(XEN) fixed up name for siul2_reg@0x4401170C -> siul2_reg
(XEN) fixed up name for siul2_reg@0x4401174C -> siul2_reg
(XEN) fixed up name for usbmisc@44064200 -> usbmisc
(XEN) fixed up name for usb@44064000 -> usb
(XEN) fixed up name for memory_DDR0@80000000 -> memory_DDR0
(XEN) fixed up name for memory_DDR1@c0000000 -> memory_DDR1
(XEN) fixed up name for memory_DDR2@880000000 -> memory_DDR2
(XEN)  <- unflatten_device_tree()
(XEN) adding DT alias:can0: stem=can id=0 node=/flexcan@401B4000
(XEN) adding DT alias:can1: stem=can id=1 node=/flexcan@401BE000
(XEN) adding DT alias:can2: stem=can id=2 node=/flexcan@402A8000
(XEN) adding DT alias:can3: stem=can id=3 node=/flexcan@402B2000
(XEN) adding DT alias:serial0: stem=serial id=0 node=/serial@401C8000
(XEN) adding DT alias:serial1: stem=serial id=1 node=/serial@401CC000
(XEN) adding DT alias:serial2: stem=serial id=2 node=/serial@402BC000
(XEN) Platform: Generic System
(XEN) Looking for dtuart at "serial0", options ""
(XEN) DT: ** translation for device /serial@401C8000 **
(XEN) DT: bus is default (na=2, ns=2) on /
(XEN) DT: translating address:<3> 00000000<3> 401c8000<3>
(XEN) DT: reached root node
(XEN) dt_device_get_raw_irq: dev=/serial@401C8000, index=0
(XEN)  intspec=0 intlen=3
(XEN)  intsize=3 intlen=3
(XEN) dt_irq_map_raw: par=/interrupt-controller@50800000,intspec=[0x00000000 0x00000052...],ointsize=3
(XEN) dt_irq_map_raw: ipar=/interrupt-controller@50800000, size=3
(XEN)  -> addrsize=2
(XEN)  -> got it� 

I started debugging and I found out that it hangs in:
console_init_preirq() -> __putstr(xen_banner()) -> sercon_puts() -> serial_puts() -> __serial_putc(),
where it spins at line 178:         
        /* Synchronous finite-capacity transmitter. */
        while ( !(n = port->driver->tx_ready(port)) )
            cpu_relax();
 
Which is a bit strange, considering that my serial device is asynchronous,
I think it should not get there. But it gets on that "else" branch because
port->txbuf is actually NULL at line 120 when it performs the check, and
it does not enter the branch for asynchronous transmitters.

Thanks,
Andrei Cherechesu,
NXP Semiconductors

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Stefano Stabellini]

On Wed, 1 Apr 2020, Andrei Cherechesu wrote:
> Hi,
> 
> And thanks for your help.
> 
> > Great to hear!
> > 
> > 
> > > However, I am encountering another problem now: in Dom0 and in 
> > > dom0less-booted DomUs, I cannot use /dev/hvc0.
> > 
> > For dom0less-booted DomUs it is normal because they don't get a PV console,
> > they get an emulated PL011 UART instead.  Make sure to have a "vpl011" tag in
> > device tree to enable it (ImageBuilder generates it by default.) The device
> > name is usually ttyAMA0.
> 
> Ok, understood. I had my vpl011 tag in the dom0less DomUs nodes in the DT,
> so that's not the problem. I am able to see DomUs boot prompt when booting
> dom0less. The problem is after DomU boots, that I am not able to switch to
> its console from Dom0, in order to be able to log in.

It looks like the UART driver in Xen is not working properly; it doesn't
seem to be a dom0less issue.


> > > Even though I'm specifying "console=hvc0" in dom0-bootargs, when dom0 
> > > finishes booting, it looks like I cannot use the getty spawned on /dev/hvc0.
> > >
> > > This is the end of the boot log:
> > > [    2.947845] random: rngd: uninitialized urandom read (4 bytes read)
> > > [    2.958415] random: rngd: uninitialized urandom read (4 bytes read)
> > > [    2.965452] random: rngd: uninitialized urandom read (2500 bytes read)
> > > .
> > > [    2.972410] random: crng init done
> > > Starting OpenBSD Secure Shell server: sshd done.
> > > Starting /usr/sbin/xenstored...
> > > Setting domain 0 name, domid and JSON config...
> > > Done setting up Dom0
> > > Starting xenconsoled...
> > > Starting QEMU as disk backend for dom0 Starting domain watchdog 
> > > daemon: xenwatchdogd startup
> > >
> > > [done]
> > >
> > > Auto Linux BSP 1.0 s32g274aevb /dev/hvc0
> > >
> > > s32g274aevb login:
> > > Auto Linux BSP 1.0 s32g274aevb /dev/ttyLF0
> > >
> > > s32g274aevb login:
> > >
> > > ----- END -----
> > >
> > > It seems that the getty spawned on /dev/ttyLF0 overwrites the one 
> > > spawned on /dev/hvc0. Which I do not understand, since Dom0 should not 
> > > have access (?) directly to ttyLF0 (the serial console device on our 
> > > boards). If I remove the line which spawns the getty on ttyLF0 from 
> > > /etc/inittab, the system hangs when waiting for the username, and it does not let me type in any characters. For the record, hvc0 is added to /etc/securetty.
> > >
> > > In a system where I boot DomU via xl from Dom0, I can switch to its 
> > > console with xl console, and hvc0 works there.
> > >
> > > The problem that comes with this is that I can not use the CTRL-AAA 
> > > command to switch between Dom0 console and DomU console in a dom0less 
> > > case, and I cannot therefore test that the passthrough works. But at least Dom0 does not have an entry for it under /dev, anymore, and DomU boot prompt tells that the driver has been registered.
> > 
> > It looks like there is some kind of interference between the dom0 ttyLF0 driver and the Xen serial driver.
> > 
> > Is your Xen UART driver marking the device as "used by Xen"? See for instance the pl011 driver, at the end of
> > xen/drivers/char/pl011.c:pl011_dt_uart_init:
> > 
> >     dt_device_set_used_by(dev, DOMID_XEN);
> > 
> > Devices that are marked as "used by Xen" are not exposed to dom0, so you shouldn't see the ttyLF0 device come up in Linux at all.
> 
> I've checked my Xen UART Driver and that call is there. So ttyLF0 should be
> marked for Xen to use.
> 
> I don't have any ideas why this happens.
> 
> I've attached the driver [0], if you want to take a look.
> 
> [0] https://pastebin.com/PuXi50H0

I cannot see any issues with the driver. Can you paste the device tree
as dom0 sees it? You can access it from /proc/device-tree (you can
convert it to dts with dtc -I fs -O dts). And also the host device tree?

We should see something like the following (this example is taken from a
Xilinx ZynqMP):

1) host device tree

	serial@ff000000 {
		u-boot,dm-pre-reloc;
		compatible = "cdns,uart-r1p12", "xlnx,xuartps";
		status = "okay";
		interrupt-parent = <0x4>;
		interrupts = <0x0 0x15 0x4>;
		reg = <0x0 0xff000000 0x0 0x1000>;
		clock-names = "uart_clk", "pclk";
		power-domains = <0x26 0x21>;
		clocks = <0x3 0x38 0x3 0x1f>;
		pinctrl-names = "default";
		pinctrl-0 = <0x37>;
		cts-override;
		device_type = "serial";
		port-number = <0x0>;
	};


2) dom0 device tree

    The node is missing
     

The key is that dom0 should not see the UART node in device tree at all.

If Dom0 is seeing the node, then it is a problem with Xen that somehow
is not hiding it (see "Skip nodes used by Xen" under
xen/arch/arm/domain_build.c:handle_node)

If Dom0 is *not* seeing the node, that what is the underlying device
tree node that the dom0 driver is using to bring up /dev/ttyLF0?

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Andrei Cherechesu]

Hi,

And thanks for your help.

> Great to hear!
> 
> 
> > However, I am encountering another problem now: in Dom0 and in 
> > dom0less-booted DomUs, I cannot use /dev/hvc0.
> 
> For dom0less-booted DomUs it is normal because they don't get a PV console,
> they get an emulated PL011 UART instead.  Make sure to have a "vpl011" tag in
> device tree to enable it (ImageBuilder generates it by default.) The device
> name is usually ttyAMA0.

Ok, understood. I had my vpl011 tag in the dom0less DomUs nodes in the DT,
so that's not the problem. I am able to see DomUs boot prompt when booting
dom0less. The problem is after DomU boots, that I am not able to switch to
its console from Dom0, in order to be able to log in.

> > Even though I'm specifying "console=hvc0" in dom0-bootargs, when dom0 
> > finishes booting, it looks like I cannot use the getty spawned on /dev/hvc0.
> >
> > This is the end of the boot log:
> > [    2.947845] random: rngd: uninitialized urandom read (4 bytes read)
> > [    2.958415] random: rngd: uninitialized urandom read (4 bytes read)
> > [    2.965452] random: rngd: uninitialized urandom read (2500 bytes read)
> > .
> > [    2.972410] random: crng init done
> > Starting OpenBSD Secure Shell server: sshd done.
> > Starting /usr/sbin/xenstored...
> > Setting domain 0 name, domid and JSON config...
> > Done setting up Dom0
> > Starting xenconsoled...
> > Starting QEMU as disk backend for dom0 Starting domain watchdog 
> > daemon: xenwatchdogd startup
> >
> > [done]
> >
> > Auto Linux BSP 1.0 s32g274aevb /dev/hvc0
> >
> > s32g274aevb login:
> > Auto Linux BSP 1.0 s32g274aevb /dev/ttyLF0
> >
> > s32g274aevb login:
> >
> > ----- END -----
> >
> > It seems that the getty spawned on /dev/ttyLF0 overwrites the one 
> > spawned on /dev/hvc0. Which I do not understand, since Dom0 should not 
> > have access (?) directly to ttyLF0 (the serial console device on our 
> > boards). If I remove the line which spawns the getty on ttyLF0 from 
> > /etc/inittab, the system hangs when waiting for the username, and it does not let me type in any characters. For the record, hvc0 is added to /etc/securetty.
> >
> > In a system where I boot DomU via xl from Dom0, I can switch to its 
> > console with xl console, and hvc0 works there.
> >
> > The problem that comes with this is that I can not use the CTRL-AAA 
> > command to switch between Dom0 console and DomU console in a dom0less 
> > case, and I cannot therefore test that the passthrough works. But at least Dom0 does not have an entry for it under /dev, anymore, and DomU boot prompt tells that the driver has been registered.
> 
> It looks like there is some kind of interference between the dom0 ttyLF0 driver and the Xen serial driver.
> 
> Is your Xen UART driver marking the device as "used by Xen"? See for instance the pl011 driver, at the end of
> xen/drivers/char/pl011.c:pl011_dt_uart_init:
> 
>     dt_device_set_used_by(dev, DOMID_XEN);
> 
> Devices that are marked as "used by Xen" are not exposed to dom0, so you shouldn't see the ttyLF0 device come up in Linux at all.

I've checked my Xen UART Driver and that call is there. So ttyLF0 should be
marked for Xen to use.

I don't have any ideas why this happens.

I've attached the driver [0], if you want to take a look.

[0] https://pastebin.com/PuXi50H0

Thanks,
Andrei Cherechesu,
NXP Semiconductors

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Stefano Stabellini]

On Thu, 13 Feb 2020, Andrei Cherechesu wrote:
> Hello,
> 
> I used the Xen from Stefano's tree and made the updates to the partial
> dtb that he specified.
> 
> > This is mostly likely because Linux is trying to access a region
> > that is not mapped in stage-2. You can rebuild Xen with debug enabled
> > and you should see a message "traps.c:..." telling the exact physical
> > address accessed.
> > 
> > In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
> > 
> > Cheers,
> > 
> > --
> > Julien Grall
> 
> I enabled debug config and gave it another try. But I'm still
> getting the same unhandled fault error, that seems to match what
> Julien described above.
> 
> It is indeed a stage-2 abort caused by the guest.
> 
> I attached the DomU1 crash log at [0].
> 
> [0] https://pastebin.com/BSHVFQiK
> 
> How should I proceed in this case?

Looking at the logs, you can see:

(XEN) traps.c:1973:d1v0 HSR=0x939f0046 pc=0xffffff80083ac864 gva=0xffffff800800d048 gpa=0x000000402f0048

So the guest was accessing address 0x402f0048, however, the MMIO address
range of the device that you are remapping (looking at
https://pastebin.com/j0NS4x5Z) is 0x4002f000-0x40030000.

I spotted the mistake now: looking at the partial DTB again, the
address of the device is:

  reg = <0x0 0x402f0000 0x1000>;

but the address that you are remapping is:

  xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;

They are not the same! :-)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Julien Grall]

Hi,

On 13/02/2020 18:27, Andrei Cherechesu wrote:
> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: Thursday, February 13, 2020 00:03
> To: Stefano Stabellini <sstabellini@kernel.org>; Andrei Cherechesu <andrei.cherechesu@nxp.com>
> Cc: Jorge Pereira <jorge.pereira@nxp.com>; xen-devel@lists.xenproject.org
> Subject: Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.
> 
> Hello,
> 
> I used the Xen from Stefano's tree and made the updates to the partial
> dtb that he specified.
> 
>> This is mostly likely because Linux is trying to access a region
>> that is not mapped in stage-2. You can rebuild Xen with debug enabled
>> and you should see a message "traps.c:..." telling the exact physical
>> address accessed.
>>
>> In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
>>
>> Cheers,
>>
>> --
>> Julien Grall
> 
> I enabled debug config and gave it another try. But I'm still
> getting the same unhandled fault error, that seems to match what
> Julien described above.
> 
> It is indeed a stage-2 abort caused by the guest.
> 
> I attached the DomU1 crash log at [0].
> 
> [0] https://pastebin.com/BSHVFQiK

 From the log:

(XEN) traps.c:1973:d1v0 HSR=0x939f0046 pc=0xffffff80083ac864 
gva=0xffffff800800d048 gpa=0x000000402f0048

So the guest is trying to access the guest physical address 
0x000000402f0048. Looking at the partial device tree you provided, you 
are requesting Xen to map the range 0x4002f000 - 0x40030000.

The address does not belong to the range. Could you check whether you 
did passthrough the correct range? (It seems like a zero is missing).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Andrei Cherechesu]

-----Original Message-----
From: Julien Grall <julien@xen.org> 
Sent: Thursday, February 13, 2020 00:03
To: Stefano Stabellini <sstabellini@kernel.org>; Andrei Cherechesu <andrei.cherechesu@nxp.com>
Cc: Jorge Pereira <jorge.pereira@nxp.com>; xen-devel@lists.xenproject.org
Subject: Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

Hello,

I used the Xen from Stefano's tree and made the updates to the partial
dtb that he specified.

> This is mostly likely because Linux is trying to access a region
> that is not mapped in stage-2. You can rebuild Xen with debug enabled
> and you should see a message "traps.c:..." telling the exact physical
> address accessed.
> 
> In general I would recommend to build Xen with debug enabled during development as the hypervisor will give you more information of what's going on.
> 
> Cheers,
> 
> --
> Julien Grall

I enabled debug config and gave it another try. But I'm still
getting the same unhandled fault error, that seems to match what
Julien described above.

It is indeed a stage-2 abort caused by the guest.

I attached the DomU1 crash log at [0].

[0] https://pastebin.com/BSHVFQiK

How should I proceed in this case?

Thank you again for your support,
Andrei
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Julien Grall]

On 12/02/2020 22:36, Stefano Stabellini wrote:
> On Wed, 12 Feb 2020, Andrei Cherechesu wrote:
>> (XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
>> (XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
>> (XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
>> (XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
>> (XEN) DOM1: [    3.909021] Mem abort info:
>> (XEN) DOM1: [    3.912863]   ESR = 0x96000000
>> (XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
>> (XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
>> (XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
>> (XEN) DOM1: [    3.932457] Data abort info:
>> (XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
>> (XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
>> (XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
>> (XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
>> (XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
>> (XEN) DOM1: [    3.973546] Modules linked in:
>> (XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
>> (XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
>> (XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
>> (XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
>> (XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
>> (XEN) DOM1: [    4.008955] sp : ffffff800800bba0
>> (XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
>> (XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
>> (XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
>> (XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
>> (XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
>> (XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
>> (XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
>> (XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
>> (XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
>> (XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
>> (XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
>> (XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
>> (XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
>> (XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
>> (XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
>> (XEN) DOM1: [    4.109729] Call trace:
>> (XEN) DOM1: [    4.113290]  0xffffff80083ac864
>> (XEN) DOM1: [    4.117541]  0xffffff800832e3b8
>> (XEN) DOM1: [    4.121795]  0xffffff800832c49c
>> (XEN) DOM1: [    4.126047]  0xffffff800832c6bc
>> (XEN) DOM1: [    4.130301]  0xffffff800832c808
>> (XEN) DOM1: [    4.134554]  0xffffff800832a208
>> (XEN) DOM1: [    4.138807]  0xffffff800832bd38
>> (XEN) DOM1: [    4.143060]  0xffffff800832b5d8
>> (XEN) DOM1: [    4.147314]  0xffffff800832d1f0
>> (XEN) DOM1: [    4.151567]  0xffffff800832e318
>> (XEN) DOM1: [    4.155820]  0xffffff800861d5f8
>> (XEN) DOM1: [    4.160073]  0xffffff800808397c
>> (XEN) DOM1: [    4.164326]  0xffffff8008600db4
>> (XEN) DOM1: [    4.168580]  0xffffff80085078c0
>> (XEN) DOM1: [    4.172833]  0xffffff8008084c30
>> (XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
>> (XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
>> (XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>>   
>>
>> I uploaded on pastebin.com the u-boot env settings [0], my device
>> passthrough partial dts [1], and the whole log of boot messages
>> from xen, Dom0 and DomU [2].
> 
> I don't know for sure what caused the kernel panic you are seeing,

This is mostly likely because Linux is trying to access a region that is 
not mapped in stage-2. You can rebuild Xen with debug enabled and you 
should see a message "traps.c:..." telling the exact physical address 
accessed.

In general I would recommend to build Xen with debug enabled during 
development as the hypervisor will give you more information of what's 
going on.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 8373 bytes --]

On Wed, 12 Feb 2020, Andrei Cherechesu wrote:
> Hello,
>  
> 
> I applied your direct-map patch, Stefano, on top of RELEASE-4.13.0
> Xen.

FYI I am working on a direct-map patch series but it is still
work-in-progress:

  http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git direct-map-1
  
There are a few fixes on top of the original direct-map patch I sent.

 
> I also took your advice and used the Imagebuilder tool to setup my
> u-boot environment. I modified the tool to allow SDCard booting and
> tweaked the parameters a little to fit our platforms, also introducing
> support to add “direct-map” parameter in specific /chosen/DomU node
> and “xen,passthrough” in the host dts. The tool is very helpful and
> allows me to quickly change the u-boot environment without manually
> entering all the fdt formatting commands.

That's great to hear :-)

For your information, if you have any changes that are worth
upstreaming, I'd be happy to take patches for imagebuilder. The mailing
list for that is viryaos-discuss@lists.sourceforge.net.

 
> The dom0less booting is successful, 

Good! I know it is not easy to setup a dom0less system. I am trying to
build tools and features to make it easier going forward.


> however, when I try to passthrough any device (I tried with ethernet
> card and uSDHC) I get a kernel panic in DomU when it tries to probe
> the driver, because of an unhandled
> 
> fault:
> 
> (XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
> (XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
> (XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
> (XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
> (XEN) DOM1: [    3.909021] Mem abort info:
> (XEN) DOM1: [    3.912863]   ESR = 0x96000000
> (XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
> (XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
> (XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
> (XEN) DOM1: [    3.932457] Data abort info:
> (XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
> (XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
> (XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
> (XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
> (XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
> (XEN) DOM1: [    3.973546] Modules linked in:
> (XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
> (XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
> (XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
> (XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
> (XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
> (XEN) DOM1: [    4.008955] sp : ffffff800800bba0
> (XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
> (XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
> (XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
> (XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
> (XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
> (XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
> (XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
> (XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
> (XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
> (XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
> (XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
> (XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
> (XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
> (XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
> (XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
> (XEN) DOM1: [    4.109729] Call trace:
> (XEN) DOM1: [    4.113290]  0xffffff80083ac864
> (XEN) DOM1: [    4.117541]  0xffffff800832e3b8
> (XEN) DOM1: [    4.121795]  0xffffff800832c49c
> (XEN) DOM1: [    4.126047]  0xffffff800832c6bc
> (XEN) DOM1: [    4.130301]  0xffffff800832c808
> (XEN) DOM1: [    4.134554]  0xffffff800832a208
> (XEN) DOM1: [    4.138807]  0xffffff800832bd38
> (XEN) DOM1: [    4.143060]  0xffffff800832b5d8
> (XEN) DOM1: [    4.147314]  0xffffff800832d1f0
> (XEN) DOM1: [    4.151567]  0xffffff800832e318
> (XEN) DOM1: [    4.155820]  0xffffff800861d5f8
> (XEN) DOM1: [    4.160073]  0xffffff800808397c
> (XEN) DOM1: [    4.164326]  0xffffff8008600db4
> (XEN) DOM1: [    4.168580]  0xffffff80085078c0
> (XEN) DOM1: [    4.172833]  0xffffff8008084c30
> (XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
> (XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
> (XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
>  
> 
> I uploaded on pastebin.com the u-boot env settings [0], my device
> passthrough partial dts [1], and the whole log of boot messages
> from xen, Dom0 and DomU [2].

I don't know for sure what caused the kernel panic you are seeing,
however, I spotted a couple of issues:

1) A missing feature in the original direct-map patch, specifically it
   couldn't handle interrupts. Please give a try at the updated branch.
2) missing information in the partial device tree.

You have:

        usdhc@402F0000 {
            xen,force-assign-without-iommu;
            #address-cells = <1>;
            #size-cells = <0>;
            compatible = "fsl,s32gen1-usdhc";
            status = "okay";
            reg = <0x0 0x402f0000 0x1000>;
            interrupt-parent = <&gic>;
            interrupts = <0 36 4>;
            clocks = <&misc_clk &misc_clk1 &misc_clk2>;
            clock-names = "ipg", "ahb", "per";
            bus-width = <8>;
            xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;
        };

You also need to specify xen,path so that the interrupts are properly
remapped (with my latest direct-map patch series.) Something like:

            xen,path = "/amba/usdhc@402F0000";
            xen,reg = <0x0 0x402f0000 0x10000 0x0 0x402f0000>;
            xen,force-assign-without-iommu;

Assuming that /amba/usdhc@402F0000 is the right path on the host device
tree. Also you shouldn't need the following under usdhc@402F0000:

            #address-cells = <1>;
            #size-cells = <0>;

So overall I'd use:

        usdhc@402F0000 {
            compatible = "fsl,s32gen1-usdhc";
            status = "okay";
            reg = <0x0 0x402f0000 0x1000>;
            interrupt-parent = <&gic>;
            interrupts = <0 36 4>;
            clocks = <&misc_clk &misc_clk1 &misc_clk2>;
            clock-names = "ipg", "ahb", "per";
            bus-width = <8>;
            xen,path = "/amba/usdhc@402F0000";
            xen,reg = <0x0 0x4002f000 0x1000 0x0 0x4002f000>;
            xen,force-assign-without-iommu;
        };


> I also modified the guest address
> layout and mapped the PL011 UART and GICv3 addresses to match
> the physical ones, as well as setting the GUEST_GNTTAB_BASE and
> GUEST_MAGIC_BASE to addresses before our board's RAM start address.
> I updated the GUEST_RAM0_BASE and GUEST_RAM0_SIZE to match the
> physical ones.

Well done! FYI one of the new things in my updated patch series is the
ability to set emulated devices addresses based on the corresponding
physical addresses automatically. Not everything is done yet, but it is
a start.


> Maybe you could check if I did anything wrong, because I couldn't
> figure it out.

Let me know how it goes with the updated partial device tree and
direct-map branch. The changes I suggested should fix the interrupts
setup. However, the kernel panic you are seeing might be caused by
something else -- there might be also another bug.

  
 
> [0] https://pastebin.com/As6PgVFf
> 
> [1] https://pastebin.com/j0NS4x5Z
> 
> [2] https://pastebin.com/TaZR8pii
> 
>  
> 
> Thank you once again for your support,
> 
> Andrei

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Having a DOM-U guest with 1:1 mapping in the second stage MMU.

[Andrei Cherechesu]

[-- Attachment #1.1: Type: text/plain, Size: 5134 bytes --]

[[ Sorry. Needed to send another mail because I forgot to add xen-devel lists to CC. ]]

Hello,

I applied your direct-map patch, Stefano, on top of RELEASE-4.13.0
Xen.

I also took your advice and used the Imagebuilder tool to setup my
u-boot environment. I modified the tool to allow SDCard booting and
tweaked the parameters a little to fit our platforms, also introducing
support to add "direct-map" parameter in specific /chosen/DomU node
and "xen,passthrough" in the host dts. The tool is very helpful and
allows me to quickly change the u-boot environment without manually
entering all the fdt formatting commands.

The dom0less booting is successful, however, when I try to passthrough
any device (I tried with ethernet card and uSDHC) I get a kernel panic
in DomU when it tries to probe the driver, because of an unhandled
fault:
(XEN) DOM1: [    3.883482] sdhci: Secure Digital Host Controller Interface driver
(XEN) DOM1: [    3.891021] sdhci: Copyright(c) Pierre Ossman
(XEN) DOM1: [    3.896389] sdhci-pltfm: SDHCI platform and OF driver helper
(XEN) DOM1: [    3.903298] Unhandled fault at 0xffffff800800d048
(XEN) DOM1: [    3.909021] Mem abort info:
(XEN) DOM1: [    3.912863]   ESR = 0x96000000
(XEN) DOM1: [    3.917019]   Exception class = DABT (current EL), IL = 32 bits
(XEN) DOM1: [    3.924115]   SET = 0, FnV = 0
(XEN) DOM1: [    3.928206]   EA = 0, S1PTW = 0
(XEN) DOM1: [    3.932457] Data abort info:
(XEN) DOM1: [    3.936514]   ISV = 0, ISS = 0x00000000
(XEN) DOM1: [    3.941398]   CM = 0, WnR = 0
(XEN) DOM1: [    3.945481] swapper pgtable: 4k pages, 39-bit VAs, pgdp = (____ptrval____)
(XEN) DOM1: [    3.953532] [ffffff800800d048] pgd=00000000bfffe803, pud=00000000bfffe803, pmd=00000000bfffd803, pte=00e80000402f0f07
(XEN) DOM1: [    3.965278] Internal error: ttbr address size fault: 96000000 [#1] PREEMPT SMP
(XEN) DOM1: [    3.973546] Modules linked in:
(XEN) DOM1: [    3.977709] Process swapper/0 (pid: 1, stack limit = 0x(____ptrval____))
(XEN) DOM1: [    3.985525] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.59-rt24+g00334f2 #1
(XEN) DOM1: [    3.993855] pstate: 60000005 (nZCv daif -PAN -UAO)
(XEN) DOM1: [    3.999755] pc : 0xffffff80083ac864
(XEN) DOM1: [    4.004354] lr : 0xffffff80083ac810
(XEN) DOM1: [    4.008955] sp : ffffff800800bba0
(XEN) DOM1: [    4.013382] x29: ffffff800800bba0 x28: 0000000000000000
(XEN) DOM1: [    4.019805] x27: ffffff800864f068 x26: ffffff80086ba000
(XEN) DOM1: [    4.026228] x25: ffffffc031564980 x24: ffffff800856e0c0
(XEN) DOM1: [    4.032651] x23: ffffffc03e8eec00 x22: ffffffc03e8eec10
(XEN) DOM1: [    4.039074] x21: ffffffc03e8bf500 x20: ffffffc03e8bf800
(XEN) DOM1: [    4.045497] x19: 0000000000000000 x18: ffffffffffffffff
(XEN) DOM1: [    4.051921] x17: 0000000000000000 x16: 0000000000000000
(XEN) DOM1: [    4.058344] x15: ffffff8008678548 x14: ffffffffffffffff
(XEN) DOM1: [    4.064767] x13: 0000000000000018 x12: 0101010101010101
(XEN) DOM1: [    4.071190] x11: 0000000000000020 x10: 0101010101010101
(XEN) DOM1: [    4.077613] x9 : 0000000000000000 x8 : ffffffc031564c00
(XEN) DOM1: [    4.084036] x7 : 0000000000000000 x6 : 000000000000003f
(XEN) DOM1: [    4.090459] x5 : 0000000000000002 x4 : ffffffc03e83b4c0
(XEN) DOM1: [    4.096883] x3 : 0000000000000000 x2 : 0000000000000000
(XEN) DOM1: [    4.103306] x1 : ffffffc03e8bf000 x0 : ffffff800800d048
(XEN) DOM1: [    4.109729] Call trace:
(XEN) DOM1: [    4.113290]  0xffffff80083ac864
(XEN) DOM1: [    4.117541]  0xffffff800832e3b8
(XEN) DOM1: [    4.121795]  0xffffff800832c49c
(XEN) DOM1: [    4.126047]  0xffffff800832c6bc
(XEN) DOM1: [    4.130301]  0xffffff800832c808
(XEN) DOM1: [    4.134554]  0xffffff800832a208
(XEN) DOM1: [    4.138807]  0xffffff800832bd38
(XEN) DOM1: [    4.143060]  0xffffff800832b5d8
(XEN) DOM1: [    4.147314]  0xffffff800832d1f0
(XEN) DOM1: [    4.151567]  0xffffff800832e318
(XEN) DOM1: [    4.155820]  0xffffff800861d5f8
(XEN) DOM1: [    4.160073]  0xffffff800808397c
(XEN) DOM1: [    4.164326]  0xffffff8008600db4
(XEN) DOM1: [    4.168580]  0xffffff80085078c0
(XEN) DOM1: [    4.172833]  0xffffff8008084c30
(XEN) DOM1: [    4.177091] Code: b9000ea0 d5033e9f f9400ea0 91012000 (b900001f)
(XEN) DOM1: [    4.184298] ---[ end trace 7dc5f6b878cccbfa ]---
(XEN) DOM1: [    4.191546] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b

I uploaded on pastebin.com the u-boot env settings [0], my device
passthrough partial dts [1], and the whole log of boot messages
from xen, Dom0 and DomU [2]. I also modified the guest address
layout and mapped the PL011 UART and GICv3 addresses to match
the physical ones, as well as setting the GUEST_GNTTAB_BASE and
GUEST_MAGIC_BASE to addresses before our board's RAM start address.
I updated the GUEST_RAM0_BASE and GUEST_RAM0_SIZE to match the
physical ones.

Maybe you could check if I did anything wrong, because I couldn't
figure it out.

[0] https://pastebin.com/As6PgVFf
[1] https://pastebin.com/j0NS4x5Z
[2] https://pastebin.com/TaZR8pii

Thank you once again for your support,
Andrei

[-- Attachment #1.2: Type: text/html, Size: 11635 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel