Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Andrei Cherechesu]

> On Mon, 16 Dec 2019, Julien Grall wrote:
> > On 16/12/2019 23:05, Stefano Stabellini wrote:
> > > On Mon, 16 Dec 2019, Julien Grall wrote:
> > > > On 16/12/2019 18:02, Andrei Cherechesu wrote:
> > > > But even with this patch, RAM in DomU is not direct mapped (i.e Guest
> > > > Physical
> > > > Address == Host Physical Address). This means that DMA-capable device
> > > > would
> > > > not work properly in DomU.
> > > >
> > > > We could theoritically map DomU direct mapped, but this would break the
> > > > isolation provided by the hypervisor.
> > >
> > > Yes, being able to map the DomU memory 1:1 can be pretty useful for some
> > > very embedded dom0less configurations, in fact I was surprised that a
> > > couple of Xilinx users asked me for that recently. Typically, the users
> > > are aware of the consequences but they still find them better than the
> > > alternative (i.e. the lack of isolation is bad but is tolerable in their
> > > configuration.)
> > This does not make much sense... The whole point of a hypervisor is to isolate
> > guest between each other... So if you are happy with the lack of isolation,
> > then why are you using an hypervisor at the first place?
>
> There are a number of reasons, although they are all variation of the
> same theme. In all these cases the IOMMU cannot be used for one reason
> or the other (a device is not behind the IOMMU, or due to an errata,
> etc.)
>
> - multiple baremetal apps
> The user wants to run two or more baremetal (unikernel-like)
> applications. The user owns both applications and she is not much
> concerned about isolation (although it is always desirable when
> possible.)
>
> - multiple OSes
> This is similar to the one before, however, instead of multiple
> baremetal apps, we are talking about multiple full OSes. For instance,
> Linux and Android or Linux and VxWorks. Again, they are both maintained
> by the same user (no multi-tenancy) so isolation is desirable but it is
> not the top concern.
>
> - real-time / no real-time
> The user wants to run a real-time OS or real-time baremetal app and a
> non real-time OS. For instance a tiny baremetal app controlling one
> specific device and Linux. Again, the user is responsible for both
> systems so isolation is not a concern.
>
> In all these cases the users has to run multiple OSes or baremetal apps
> so she needs a hypervisor. However, it is tolerable that the apps are
> not actually fully isolated from each others because they are both
> developed and deployed together by the same "owner".
>

Basically, since we do not have an IOMMU, we would be able
to ensure memory isolation via a NXP IP named xRDC (Extended
Resource Domain Controller) that our boards have, which supervises
the access to memory buses.

But before we get to think about isolation, we need to enable
basic passthrough functionality (via 1:1 mapping, since no IOMMU).

Firstly, a good step forward would be to get any non-DMA-capable
device passed-through and working. 
I rebased onto upstream/staging branch and applied the hack
that skips the setting of XEN_DOMCTL_CDF_iommu flag,
that Julien specified.

Then I tried to passthrough the eMMC, but I got the following
error:
(XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request region for resource [mem 0x4005d000-0x4005dfff]
(XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc: sdhci_pltfm_init failed -16
(XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed with error -16

Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the DT.
It seems that the DomU1 kernel does not have access to that memory zone.

I'm trying to passthrough the eMMC in order to mount DomU1's root
on a SDCard partition, because I couldn't get to DomU1's Linux prompt
when I tried to boot with a ramdisk module. I always get this error:
(XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image starting at 0.

Could this be because the ramdisk is too big? The smallest I've tried with
Is approximately 60MB in size. What size are the ramdisks that you
are using in your dom0less booting demos?

> > >  From an implementation perspective, it should be a matter of calling
> > > allocate_memory_11 instead of allocate_memory from construct_domU. I
> > > wanted to experiment with it myself but I haven't had the time. If
> > > nothing else, it would be useful to have a patch around to do it if
> > > needed.
> > This is not that simple. You at least also need to:
> >     - Update the code to generate the DT based on the new 1:1 address
> >     - Modify the various emulation in Xen because they rely on Xen guest
> > memory layout
> >     - Modify is_domain_direct_mapped() to deal with guest
> >
> > I probably missed other bits. Anyway, this is not something I am willing to
> > accept upstream as this break the core idea of an hypervisor.
>
> If you prefer not to have it upstream, I would be happy to maintain it
> downstream in Xilinx/Xen or another tree, and take it as a contribution
> from Andrei if he volunteers to write and test the patch.
>
> Andrei, if you are going to write the patch, thanks in advance :-)
> Otherwise, I might get to it at some point but it might me a while.
>
> Cheers,
>
> Stefano

I'll gladly write the patch if you give me some basic
instructions regarding it, because I'm not that familiar with
all the Xen internal mechanisms, and I wouldn't know where
to look in order to ensure everything is properly done.

Thank you very much for your help,
Andrei


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Julien Grall]

Hi Andrei,

On 17/12/2019 17:20, Andrei Cherechesu wrote:
>> On Mon, 16 Dec 2019, Julien Grall wrote:
>>> On 16/12/2019 23:05, Stefano Stabellini wrote:
>>>> On Mon, 16 Dec 2019, Julien Grall wrote:
>>>>> On 16/12/2019 18:02, Andrei Cherechesu wrote:
>>>>> But even with this patch, RAM in DomU is not direct mapped (i.e Guest
>>>>> Physical
>>>>> Address == Host Physical Address). This means that DMA-capable device
>>>>> would
>>>>> not work properly in DomU.
>>>>>
>>>>> We could theoritically map DomU direct mapped, but this would break the
>>>>> isolation provided by the hypervisor.
>>>>
>>>> Yes, being able to map the DomU memory 1:1 can be pretty useful for some
>>>> very embedded dom0less configurations, in fact I was surprised that a
>>>> couple of Xilinx users asked me for that recently. Typically, the users
>>>> are aware of the consequences but they still find them better than the
>>>> alternative (i.e. the lack of isolation is bad but is tolerable in their
>>>> configuration.)
>>> This does not make much sense... The whole point of a hypervisor is to isolate
>>> guest between each other... So if you are happy with the lack of isolation,
>>> then why are you using an hypervisor at the first place?
>>
>> There are a number of reasons, although they are all variation of the
>> same theme. In all these cases the IOMMU cannot be used for one reason
>> or the other (a device is not behind the IOMMU, or due to an errata,
>> etc.)
>>
>> - multiple baremetal apps
>> The user wants to run two or more baremetal (unikernel-like)
>> applications. The user owns both applications and she is not much
>> concerned about isolation (although it is always desirable when
>> possible.)
>>
>> - multiple OSes
>> This is similar to the one before, however, instead of multiple
>> baremetal apps, we are talking about multiple full OSes. For instance,
>> Linux and Android or Linux and VxWorks. Again, they are both maintained
>> by the same user (no multi-tenancy) so isolation is desirable but it is
>> not the top concern.
>>
>> - real-time / no real-time
>> The user wants to run a real-time OS or real-time baremetal app and a
>> non real-time OS. For instance a tiny baremetal app controlling one
>> specific device and Linux. Again, the user is responsible for both
>> systems so isolation is not a concern.
>>
>> In all these cases the users has to run multiple OSes or baremetal apps
>> so she needs a hypervisor. However, it is tolerable that the apps are
>> not actually fully isolated from each others because they are both
>> developed and deployed together by the same "owner".
>>
> 
> Basically, since we do not have an IOMMU, we would be able
> to ensure memory isolation via a NXP IP named xRDC (Extended
> Resource Domain Controller) that our boards have, which supervises
> the access to memory buses.

Ok, so you have some sort of MPU. I assume this will be between the 
devices and the memory, am I right?
> 
> But before we get to think about isolation, we need to enable
> basic passthrough functionality (via 1:1 mapping, since no IOMMU).

So you are in better place than what Stefano described. Your use case is 
probably the only place where a 1:1 mapping would be warrant as 
isolation is still provided by the HW.

> 
> Firstly, a good step forward would be to get any non-DMA-capable
> device passed-through and working.
> I rebased onto upstream/staging branch and applied the hack
> that skips the setting of XEN_DOMCTL_CDF_iommu flag,
> that Julien specified.
> 
> Then I tried to passthrough the eMMC, but I got the following
> error:
> (XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request region for resource [mem 0x4005d000-0x4005dfff]
> (XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc: sdhci_pltfm_init failed -16
> (XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed with error -16
> 
> Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the DT.
> It seems that the DomU1 kernel does not have access to that memory zone.

Could you paste your partial Device-Tree and domain node?

> 
> I'm trying to passthrough the eMMC in order to mount DomU1's root
> on a SDCard partition, because I couldn't get to DomU1's Linux prompt
> when I tried to boot with a ramdisk module. I always get this error:
> (XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image starting at 0.

How did you pass the ramdisk to dom1?

> 
> Could this be because the ramdisk is too big? The smallest I've tried with
> Is approximately 60MB in size. What size are the ramdisks that you
> are using in your dom0less booting demos?

How much memory did you give to your guest?

[...]

> I'll gladly write the patch if you give me some basic
> instructions regarding it, because I'm not that familiar with
> all the Xen internal mechanisms, and I wouldn't know where
> to look in order to ensure everything is properly done.

I am going to suggest a quick and dirty way but it should get you to the 
point where 1:1 mapping will work in basic use case:

1) Update the guest memory map in xen/include/public/arch-arm.h (see 
GUEST_*) so all the regions don't overlap your RAM. The best way would 
be to re-use the same address for the GIC and UART as the host.

2) Allocate the memory bank 1:1. The patch below should do the job (not 
compiled nor tested):

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index dd9c3b73ba..9155728640 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -374,6 +374,7 @@ static void __init allocate_memory_11(struct domain *d,
      }
  }

+#if 0
  static bool __init allocate_bank_memory(struct domain *d,
                                          struct kernel_info *kinfo,
                                          gfn_t sgfn,
@@ -472,6 +473,7 @@ fail:
            " %ldKB unallocated. Fix the VMs configurations.\n",
            (unsigned long)kinfo->unassigned_mem >> 10);
  }
+#endif

  static int __init write_properties(struct domain *d, struct 
kernel_info *kinfo,
                                     const struct dt_device_node *node)
@@ -2434,7 +2436,7 @@ static int __init construct_domU(struct domain *d,
      /* type must be set before allocate memory */
      d->arch.type = kinfo.type;
  #endif
-    allocate_memory(d, &kinfo);
+    allocate_memory_11(d, &kinfo);

      rc = prepare_dtb_domU(d, &kinfo);
      if ( rc < 0 )

If you want to use Xen hypercalls such as for ballooning, you can look 
at the macro is_domain_direct_mapped(). This will most likely require 
change in your guest OS because any hypercall modifying memory will not 
keep the 1:1 assumption. Therefore the OS has to keep track of it. This 
is a bit tedious but already exists in Linux as we use it for Dom0.

For a more generic approach, you would need to replace the static 
version suggested 1) with a more dynamic layout that could be based on 
the host memory layout.

Best regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Stefano Stabellini]

On Tue, 17 Dec 2019, Andrei Cherechesu wrote:
> > On Mon, 16 Dec 2019, Julien Grall wrote:
> > > On 16/12/2019 23:05, Stefano Stabellini wrote:
> > > > On Mon, 16 Dec 2019, Julien Grall wrote:
> > > > > On 16/12/2019 18:02, Andrei Cherechesu wrote:
> > > > > But even with this patch, RAM in DomU is not direct mapped (i.e Guest
> > > > > Physical
> > > > > Address == Host Physical Address). This means that DMA-capable device
> > > > > would
> > > > > not work properly in DomU.
> > > > >
> > > > > We could theoritically map DomU direct mapped, but this would break the
> > > > > isolation provided by the hypervisor.
> > > >
> > > > Yes, being able to map the DomU memory 1:1 can be pretty useful for some
> > > > very embedded dom0less configurations, in fact I was surprised that a
> > > > couple of Xilinx users asked me for that recently. Typically, the users
> > > > are aware of the consequences but they still find them better than the
> > > > alternative (i.e. the lack of isolation is bad but is tolerable in their
> > > > configuration.)
> > > This does not make much sense... The whole point of a hypervisor is to isolate
> > > guest between each other... So if you are happy with the lack of isolation,
> > > then why are you using an hypervisor at the first place?
> >
> > There are a number of reasons, although they are all variation of the
> > same theme. In all these cases the IOMMU cannot be used for one reason
> > or the other (a device is not behind the IOMMU, or due to an errata,
> > etc.)
> >
> > - multiple baremetal apps
> > The user wants to run two or more baremetal (unikernel-like)
> > applications. The user owns both applications and she is not much
> > concerned about isolation (although it is always desirable when
> > possible.)
> >
> > - multiple OSes
> > This is similar to the one before, however, instead of multiple
> > baremetal apps, we are talking about multiple full OSes. For instance,
> > Linux and Android or Linux and VxWorks. Again, they are both maintained
> > by the same user (no multi-tenancy) so isolation is desirable but it is
> > not the top concern.
> >
> > - real-time / no real-time
> > The user wants to run a real-time OS or real-time baremetal app and a
> > non real-time OS. For instance a tiny baremetal app controlling one
> > specific device and Linux. Again, the user is responsible for both
> > systems so isolation is not a concern.
> >
> > In all these cases the users has to run multiple OSes or baremetal apps
> > so she needs a hypervisor. However, it is tolerable that the apps are
> > not actually fully isolated from each others because they are both
> > developed and deployed together by the same "owner".
> >
> 
> Basically, since we do not have an IOMMU, we would be able
> to ensure memory isolation via a NXP IP named xRDC (Extended
> Resource Domain Controller) that our boards have, which supervises
> the access to memory buses.
> 
> But before we get to think about isolation, we need to enable
> basic passthrough functionality (via 1:1 mapping, since no IOMMU).
> 
> Firstly, a good step forward would be to get any non-DMA-capable
> device passed-through and working. 
> I rebased onto upstream/staging branch and applied the hack
> that skips the setting of XEN_DOMCTL_CDF_iommu flag,
> that Julien specified.
> 
> Then I tried to passthrough the eMMC, but I got the following
> error:
> (XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request region for resource [mem 0x4005d000-0x4005dfff]
> (XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc: sdhci_pltfm_init failed -16
> (XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed with error -16
> 
> Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the DT.
> It seems that the DomU1 kernel does not have access to that memory zone.

It looks like drivers/mmc/host/sdhci-pltfm.c:sdhci_pltfm_init failed,
but I cannot see a simple reason why it would. As Julien mentioned the
device tree snippet would be useful. Also the domU config and the full
device tree would be useful. i.e. did you add "xen,passthrough;" under
the related uSDHC node on the host device tree?


> I'm trying to passthrough the eMMC in order to mount DomU1's root
> on a SDCard partition, because I couldn't get to DomU1's Linux prompt
> when I tried to boot with a ramdisk module. I always get this error:
> (XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image starting at 0.
> 
> Could this be because the ramdisk is too big? The smallest I've tried with
> Is approximately 60MB in size. What size are the ramdisks that you
> are using in your dom0less booting demos?

I don't think so, I could boot with ramdisk 120MB in size or even
larger. It is probably an address calculation error: it is easy to make
a small mistake in the addresses so that they end up overlapping.
Sometimes it is even U-Boot that causes the overlaps.

I would suggest to use ImageBuilder to create the U-Boot boot script to
load all the binaries and boot the system. Have a look at
uboot-script-gen in particular:

https://gitlab.com/ViryaOS/imagebuilder/blob/master/scripts/uboot-script-gen

See this wikipage on the subject:

https://wiki.xenproject.org/wiki/ImageBuilder

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Julien Grall]

Hi Stefano,

On 17/12/2019 18:28, Stefano Stabellini wrote:
>> Then I tried to passthrough the eMMC, but I got the following
>> error:
>> (XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request region for resource [mem 0x4005d000-0x4005dfff]
>> (XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc: sdhci_pltfm_init failed -16
>> (XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed with error -16
>>
>> Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the DT.
>> It seems that the DomU1 kernel does not have access to that memory zone.
> 
> It looks like drivers/mmc/host/sdhci-pltfm.c:sdhci_pltfm_init failed,
> but I cannot see a simple reason why it would. As Julien mentioned the
> device tree snippet would be useful. Also the domU config and the full
> device tree would be useful. i.e. did you add "xen,passthrough;" under
> the related uSDHC node on the host device tree?

The only purpose of "xen,passthrough" is to mark the device as disabled 
in Dom0 DT. It will not affect how device will be passthrough to a guest.

In this case, I don't believe the problem is DT related because Linux is 
able to find the regions. If the region were not mapped to the guest, 
then it would be likely result to a data abort later on.

Looking at Andrei's e-mail again, he doesn't mention anything about the 
1:1 mapping. So I assume, he is still using the guest memory layout. The 
physical address 0x4005d000 which is roughly 372KB into the first RAM 
bank for the guest.

>> I'm trying to passthrough the eMMC in order to mount DomU1's root
>> on a SDCard partition, because I couldn't get to DomU1's Linux prompt
>> when I tried to boot with a ramdisk module. I always get this error:
>> (XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image starting at 0.
>>
>> Could this be because the ramdisk is too big? The smallest I've tried with
>> Is approximately 60MB in size. What size are the ramdisks that you
>> are using in your dom0less booting demos?
> 
> I don't think so, I could boot with ramdisk 120MB in size or even
> larger. It is probably an address calculation error: it is easy to make
> a small mistake in the addresses so that they end up overlapping.
> Sometimes it is even U-Boot that causes the overlaps.
> 
> I would suggest to use ImageBuilder to create the U-Boot boot script to
> load all the binaries and boot the system. Have a look at
> uboot-script-gen in particular:
> 
> https://gitlab.com/ViryaOS/imagebuilder/blob/master/scripts/uboot-script-gen

Nice script, but it seems to contain hardcoded value (see offset and 
memaddr override), does not take into account reserved region and assume 
where U-boot/ATF may be loaded. So it may require some work before it 
can be used on NXP board...

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Stefano Stabellini]

On Wed, 18 Dec 2019, Julien Grall wrote:
> Hi Stefano,
> 
> On 17/12/2019 18:28, Stefano Stabellini wrote:
> > > Then I tried to passthrough the eMMC, but I got the following
> > > error:
> > > (XEN) DOM1: [    0.879151] sdhci-esdhc-imx 4005d000.usdhc: can't request
> > > region for resource [mem 0x4005d000-0x4005dfff]
> > > (XEN) DOM1: [    0.891137] sdhci-esdhc-imx 4005d000.usdhc:
> > > sdhci_pltfm_init failed -16
> > > (XEN) DOM1: [    0.900249] sdhci-esdhc-imx: probe of 4005d000.usdhc failed
> > > with error -16
> > > 
> > > Where 0x4005d000 is the physical address of the uSDHC(eMMC) node in the
> > > DT.
> > > It seems that the DomU1 kernel does not have access to that memory zone.
> > 
> > It looks like drivers/mmc/host/sdhci-pltfm.c:sdhci_pltfm_init failed,
> > but I cannot see a simple reason why it would. As Julien mentioned the
> > device tree snippet would be useful. Also the domU config and the full
> > device tree would be useful. i.e. did you add "xen,passthrough;" under
> > the related uSDHC node on the host device tree?
> 
> The only purpose of "xen,passthrough" is to mark the device as disabled in
> Dom0 DT. It will not affect how device will be passthrough to a guest.
> 
> In this case, I don't believe the problem is DT related because Linux is able
> to find the regions. If the region were not mapped to the guest, then it would
> be likely result to a data abort later on.
> 
> Looking at Andrei's e-mail again, he doesn't mention anything about the 1:1
> mapping. So I assume, he is still using the guest memory layout. The physical
> address 0x4005d000 which is roughly 372KB into the first RAM bank for the
> guest.
> 
> > > I'm trying to passthrough the eMMC in order to mount DomU1's root
> > > on a SDCard partition, because I couldn't get to DomU1's Linux prompt
> > > when I tried to boot with a ramdisk module. I always get this error:
> > > (XEN) DOM1: [    1.544199] RAMDISK: Couldn't find valid RAM disk image
> > > starting at 0.
> > > 
> > > Could this be because the ramdisk is too big? The smallest I've tried with
> > > Is approximately 60MB in size. What size are the ramdisks that you
> > > are using in your dom0less booting demos?
> > 
> > I don't think so, I could boot with ramdisk 120MB in size or even
> > larger. It is probably an address calculation error: it is easy to make
> > a small mistake in the addresses so that they end up overlapping.
> > Sometimes it is even U-Boot that causes the overlaps.
> > 
> > I would suggest to use ImageBuilder to create the U-Boot boot script to
> > load all the binaries and boot the system. Have a look at
> > uboot-script-gen in particular:
> > 
> > https://gitlab.com/ViryaOS/imagebuilder/blob/master/scripts/uboot-script-gen
> 
> Nice script, but it seems to contain hardcoded value (see offset and memaddr
> override), does not take into account reserved region and assume where
> U-boot/ATF may be loaded. So it may require some work before it can be used on
> NXP board...

Yes, you are right about that. The script doesn't understand
reserved-memory today and it will just start loading binaries at 2MB
after "MEMORY_START" as specified in the config file, assuming that it
is safe to do so.

Andrei, if you end up using it and it doesn't work, please let me know.
I am interested in understanding any failures and might be able to
improve the script or take patches for it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Julien Grall]

On 17/12/2019 01:18, Stefano Stabellini wrote:
> On Mon, 16 Dec 2019, Julien Grall wrote:
>> On 16/12/2019 23:05, Stefano Stabellini wrote:
>>> On Mon, 16 Dec 2019, Julien Grall wrote:
>>>> On 16/12/2019 18:02, Andrei Cherechesu wrote:
>>>> But even with this patch, RAM in DomU is not direct mapped (i.e Guest
>>>> Physical
>>>> Address == Host Physical Address). This means that DMA-capable device
>>>> would
>>>> not work properly in DomU.
>>>>
>>>> We could theoritically map DomU direct mapped, but this would break the
>>>> isolation provided by the hypervisor.
>>>
>>> Yes, being able to map the DomU memory 1:1 can be pretty useful for some
>>> very embedded dom0less configurations, in fact I was surprised that a
>>> couple of Xilinx users asked me for that recently. Typically, the users
>>> are aware of the consequences but they still find them better than the
>>> alternative (i.e. the lack of isolation is bad but is tolerable in their
>>> configuration.)
>> This does not make much sense... The whole point of a hypervisor is to isolate
>> guest between each other... So if you are happy with the lack of isolation,
>> then why are you using an hypervisor at the first place?
> 
> There are a number of reasons, although they are all variation of the
> same theme. In all these cases the IOMMU cannot be used for one reason
> or the other (a device is not behind the IOMMU, or due to an errata,
> etc.)
> 
> - multiple baremetal apps
> The user wants to run two or more baremetal (unikernel-like)
> applications. The user owns both applications and she is not much
> concerned about isolation (although it is always desirable when
> possible.)
> 
> - multiple OSes
> This is similar to the one before, however, instead of multiple
> baremetal apps, we are talking about multiple full OSes. For instance,
> Linux and Android or Linux and VxWorks. Again, they are both maintained
> by the same user (no multi-tenancy) so isolation is desirable but it is
> not the top concern.
> 
> - real-time / no real-time
> The user wants to run a real-time OS or real-time baremetal app and a
> non real-time OS. For instance a tiny baremetal app controlling one
> specific device and Linux. Again, the user is responsible for both
> systems so isolation is not a concern.
> 
> In all these cases the users has to run multiple OSes or baremetal apps
> so she needs a hypervisor. However, it is tolerable that the apps are
> not actually fully isolated from each others because they are both
> developed and deployed together by the same "owner".

I don't think "maintained and deployed by the same owner" is enough to 
justify it is somewhat safe to use it. You also need to trust your 
users. For instance, if you allow your users to interact with the OS 
(e.g installing app...) or have internet, then you will be one day or 
another prone to a vulnerability.

A good example that come into mind is the blackhat talk in 2017 about 
BluePill (see [1]). This is one case where isolation was dismissed, yet 
it bite them after.

This was only one OS. I let you imagine with a system multiple OSes...

Cheers,

[1] 
https://www.blackhat.com/docs/us-17/wednesday/us-17-Bazhaniuk-BluePill-For-Your-Phone.pdf

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Stefano Stabellini]

On Mon, 16 Dec 2019, Julien Grall wrote:
> On 16/12/2019 23:05, Stefano Stabellini wrote:
> > On Mon, 16 Dec 2019, Julien Grall wrote:
> > > On 16/12/2019 18:02, Andrei Cherechesu wrote:
> > > But even with this patch, RAM in DomU is not direct mapped (i.e Guest
> > > Physical
> > > Address == Host Physical Address). This means that DMA-capable device
> > > would
> > > not work properly in DomU.
> > > 
> > > We could theoritically map DomU direct mapped, but this would break the
> > > isolation provided by the hypervisor.
> > 
> > Yes, being able to map the DomU memory 1:1 can be pretty useful for some
> > very embedded dom0less configurations, in fact I was surprised that a
> > couple of Xilinx users asked me for that recently. Typically, the users
> > are aware of the consequences but they still find them better than the
> > alternative (i.e. the lack of isolation is bad but is tolerable in their
> > configuration.)
> This does not make much sense... The whole point of a hypervisor is to isolate
> guest between each other... So if you are happy with the lack of isolation,
> then why are you using an hypervisor at the first place?

There are a number of reasons, although they are all variation of the
same theme. In all these cases the IOMMU cannot be used for one reason
or the other (a device is not behind the IOMMU, or due to an errata,
etc.)

- multiple baremetal apps
The user wants to run two or more baremetal (unikernel-like)
applications. The user owns both applications and she is not much
concerned about isolation (although it is always desirable when
possible.)

- multiple OSes
This is similar to the one before, however, instead of multiple
baremetal apps, we are talking about multiple full OSes. For instance,
Linux and Android or Linux and VxWorks. Again, they are both maintained
by the same user (no multi-tenancy) so isolation is desirable but it is
not the top concern.

- real-time / no real-time
The user wants to run a real-time OS or real-time baremetal app and a
non real-time OS. For instance a tiny baremetal app controlling one
specific device and Linux. Again, the user is responsible for both
systems so isolation is not a concern.

In all these cases the users has to run multiple OSes or baremetal apps
so she needs a hypervisor. However, it is tolerable that the apps are
not actually fully isolated from each others because they are both
developed and deployed together by the same "owner".


> >  From an implementation perspective, it should be a matter of calling
> > allocate_memory_11 instead of allocate_memory from construct_domU. I
> > wanted to experiment with it myself but I haven't had the time. If
> > nothing else, it would be useful to have a patch around to do it if
> > needed.
> This is not that simple. You at least also need to:
>     - Update the code to generate the DT based on the new 1:1 address
>     - Modify the various emulation in Xen because they rely on Xen guest
> memory layout
>     - Modify is_domain_direct_mapped() to deal with guest
> 
> I probably missed other bits. Anyway, this is not something I am willing to
> accept upstream as this break the core idea of an hypervisor.

If you prefer not to have it upstream, I would be happy to maintain it
downstream in Xilinx/Xen or another tree, and take it as a contribution
from Andrei if he volunteers to write and test the patch.

Andrei, if you are going to write the patch, thanks in advance :-)
Otherwise, I might get to it at some point but it might me a while.

Cheers,

Stefano

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Julien Grall]

On 16/12/2019 23:05, Stefano Stabellini wrote:
> On Mon, 16 Dec 2019, Julien Grall wrote:
>> On 16/12/2019 18:02, Andrei Cherechesu wrote:
>> But even with this patch, RAM in DomU is not direct mapped (i.e Guest Physical
>> Address == Host Physical Address). This means that DMA-capable device would
>> not work properly in DomU.
>>
>> We could theoritically map DomU direct mapped, but this would break the
>> isolation provided by the hypervisor.
> 
> Yes, being able to map the DomU memory 1:1 can be pretty useful for some
> very embedded dom0less configurations, in fact I was surprised that a
> couple of Xilinx users asked me for that recently. Typically, the users
> are aware of the consequences but they still find them better than the
> alternative (i.e. the lack of isolation is bad but is tolerable in their
> configuration.)
This does not make much sense... The whole point of a hypervisor is to 
isolate guest between each other... So if you are happy with the lack of 
isolation, then why are you using an hypervisor at the first place?

> 
>  From an implementation perspective, it should be a matter of calling
> allocate_memory_11 instead of allocate_memory from construct_domU. I
> wanted to experiment with it myself but I haven't had the time. If
> nothing else, it would be useful to have a patch around to do it if
> needed.
This is not that simple. You at least also need to:
     - Update the code to generate the DT based on the new 1:1 address
     - Modify the various emulation in Xen because they rely on Xen 
guest memory layout
     - Modify is_domain_direct_mapped() to deal with guest

I probably missed other bits. Anyway, this is not something I am willing 
to accept upstream as this break the core idea of an hypervisor.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5404 bytes --]

On Mon, 16 Dec 2019, Julien Grall wrote:
> On 16/12/2019 18:02, Andrei Cherechesu wrote:
> > Hello,
> 
> Hello,
> 
> > My name is Andrei Cherechesu and I'm a Software Engineer at NXP
> > 
> > Semiconductors in the Automotive department, Linux BSP Team.
> > 
> > I would like to tell you have done a great job so far with Xen.
> 
> Thank you for your interest in Xen on Arm.

Yes, thank you!!


> > Thus, we have ported and integrated Xen ARM in the Linux BSP for our
> > 
> > boards.
> > 
> > Currently, we are trying to passthrough devices in order to statically
> > 
> > assign them to different DomUs at boot time (Dom0less configuration).
> > 
> > Our boards do not have an IOMMU and so our work is only possible
> > 
> > with the help of the patches submitted by Stefano Stabellini that
> > 
> > implemented "xen,force-assign-without-iommu" property in the
> > 
> > partial device trees used for passthrough.
> > 
> > I made a configuration in order to setup the dtb /chosen
> > 
> > node to boot a Dom0less VM in parallel with Dom0, with a
> > 
> > peripherical device statically assigned to it.
> > 
> > However, only if I advertise the passthrough dtb node in u-boot,
> > 
> > when I boot Xen, I am given the following error:
> > 
> > (XEN) *** LOADING DOMAIN 0 ***
> > 
> > (XEN) Loading d0 kernel from boot module @ 00000000be800000
> > 
> > (XEN) Loading ramdisk from boot module @ 00000000b1000000
> > 
> > (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> > 
> > (XEN) BANK[0] 0x000000d0000000-0x000000f0000000 (512MB)
> > 
> > (XEN) Grant table range: 0x00000080200000-0x00000080240000
> > 
> > (XEN) Allocating PPI 16 for event channel interrupt
> > 
> > (XEN) Loading zImage from 00000000be800000 to
> > 00000000d0080000-00000000d0930008
> > 
> > (XEN) Loading dom0 initrd from 00000000b1000000 to
> > 0x00000000d8200000-0x00000000e01fffff
> > 
> > (XEN) Loading dom0 DTB to 0x00000000d8000000-0x00000000d800596a
> > 
> > (XEN) Initial low memory virq threshold set at 0x4000 pages.
> > 
> > (XEN) Scrubbing Free RAM in background
> > 
> > (XEN) Std. Loglevel: All
> > 
> > (XEN) Guest Loglevel: All
> > 
> > (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> > 
> > (XEN) domain.c:322: IOMMU is not enabled
> > 
> > (XEN)
> > 
> > (XEN) ****************************************
> > 
> > (XEN) Panic on CPU 0:
> > 
> > (XEN) Error creating domain domU1
> > 
> > (XEN) ****************************************
> > 
> > My Xen is taken from Stefano’s tree
> > (http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git)
> > 
> > that he specified in this
> > <https://lists.xenproject.org/archives/html/xen-devel/2019-10/msg00414.html>
> > thread index, and only has the additional commits
> > 
> > that add support for our boards applied by myself.
> 
> This series as been merged in the upcoming release of Xen. I would therefore
> recommend to use the upstream staging/master in case bugs may have been fixed
> there.
> 
> > 
> >  From what I've read on the mailing lists, the Dom0less Device
> > 
> > Passthrough Without IOMMU feature is supported, at least,
> > 
> > on boards that do have a physical IOMMU but have some devices that
> > 
> > are not behind it.
> > 
> > My question is: In the way that it is currently implemented,
> > 
> > is the Dom0less Device Passthrough feature currently supported
> > 
> > on boards that do not have an IOMMU at all?
> 
> Looking at the code, there are a few issues in current Xen that would prevent
> passthrough without IOMMU to work.
> 
> For a first, Xen is requesting to create instiate an IOMMU context for the
> domain if a partial DT is present (see create_domUs()). The following path
> should get rid of the panic above (/!\ it is a gross hack and not for
> upstream/production):
> 
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index dd9c3b73ba..5700b992df 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -2474,9 +2474,6 @@ void __init create_domUs(void)
>              panic("Missing property 'cpus' for domain %s\n",
>                    dt_node_name(node));
> 
> -        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
> -            d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
> -
>          if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
>          {
>              d_cfg.arch.nr_spis = gic_number_lines() - 32;
> 
> But even with this patch, RAM in DomU is not direct mapped (i.e Guest Physical
> Address == Host Physical Address). This means that DMA-capable device would
> not work properly in DomU.
> 
> We could theoritically map DomU direct mapped, but this would break the
> isolation provided by the hypervisor.

Yes, being able to map the DomU memory 1:1 can be pretty useful for some
very embedded dom0less configurations, in fact I was surprised that a
couple of Xilinx users asked me for that recently. Typically, the users
are aware of the consequences but they still find them better than the
alternative (i.e. the lack of isolation is bad but is tolerable in their
configuration.)

From an implementation perspective, it should be a matter of calling
allocate_memory_11 instead of allocate_memory from construct_domU. I
wanted to experiment with it myself but I haven't had the time. If
nothing else, it would be useful to have a patch around to do it if
needed.

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Julien Grall]

On 16/12/2019 18:02, Andrei Cherechesu wrote:
> Hello,

Hello,

> My name is Andrei Cherechesu and I'm a Software Engineer at NXP
> 
> Semiconductors in the Automotive department, Linux BSP Team.
> 
> I would like to tell you have done a great job so far with Xen.

Thank you for your interest in Xen on Arm.

> 
> Thus, we have ported and integrated Xen ARM in the Linux BSP for our
> 
> boards.
> 
> Currently, we are trying to passthrough devices in order to statically
> 
> assign them to different DomUs at boot time (Dom0less configuration).
> 
> Our boards do not have an IOMMU and so our work is only possible
> 
> with the help of the patches submitted by Stefano Stabellini that
> 
> implemented "xen,force-assign-without-iommu" property in the
> 
> partial device trees used for passthrough.
> 
> I made a configuration in order to setup the dtb /chosen
> 
> node to boot a Dom0less VM in parallel with Dom0, with a
> 
> peripherical device statically assigned to it.
> 
> However, only if I advertise the passthrough dtb node in u-boot,
> 
> when I boot Xen, I am given the following error:
> 
> (XEN) *** LOADING DOMAIN 0 ***
> 
> (XEN) Loading d0 kernel from boot module @ 00000000be800000
> 
> (XEN) Loading ramdisk from boot module @ 00000000b1000000
> 
> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
> 
> (XEN) BANK[0] 0x000000d0000000-0x000000f0000000 (512MB)
> 
> (XEN) Grant table range: 0x00000080200000-0x00000080240000
> 
> (XEN) Allocating PPI 16 for event channel interrupt
> 
> (XEN) Loading zImage from 00000000be800000 to 
> 00000000d0080000-00000000d0930008
> 
> (XEN) Loading dom0 initrd from 00000000b1000000 to 
> 0x00000000d8200000-0x00000000e01fffff
> 
> (XEN) Loading dom0 DTB to 0x00000000d8000000-0x00000000d800596a
> 
> (XEN) Initial low memory virq threshold set at 0x4000 pages.
> 
> (XEN) Scrubbing Free RAM in background
> 
> (XEN) Std. Loglevel: All
> 
> (XEN) Guest Loglevel: All
> 
> (XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
> 
> (XEN) domain.c:322: IOMMU is not enabled
> 
> (XEN)
> 
> (XEN) ****************************************
> 
> (XEN) Panic on CPU 0:
> 
> (XEN) Error creating domain domU1
> 
> (XEN) ****************************************
> 
> My Xen is taken from Stefano’s tree 
> (http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git)
> 
> that he specified in this 
> <https://lists.xenproject.org/archives/html/xen-devel/2019-10/msg00414.html> 
> thread index, and only has the additional commits
> 
> that add support for our boards applied by myself.

This series as been merged in the upcoming release of Xen. I would 
therefore recommend to use the upstream staging/master in case bugs may 
have been fixed there.

> 
>  From what I've read on the mailing lists, the Dom0less Device
> 
> Passthrough Without IOMMU feature is supported, at least,
> 
> on boards that do have a physical IOMMU but have some devices that
> 
> are not behind it.
> 
> My question is: In the way that it is currently implemented,
> 
> is the Dom0less Device Passthrough feature currently supported
> 
> on boards that do not have an IOMMU at all?

Looking at the code, there are a few issues in current Xen that would 
prevent passthrough without IOMMU to work.

For a first, Xen is requesting to create instiate an IOMMU context for 
the domain if a partial DT is present (see create_domUs()). The 
following path should get rid of the panic above (/!\ it is a gross hack 
and not for upstream/production):

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index dd9c3b73ba..5700b992df 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -2474,9 +2474,6 @@ void __init create_domUs(void)
              panic("Missing property 'cpus' for domain %s\n",
                    dt_node_name(node));

-        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") )
-            d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
-
          if ( !dt_property_read_u32(node, "nr_spis", &d_cfg.arch.nr_spis) )
          {
              d_cfg.arch.nr_spis = gic_number_lines() - 32;

But even with this patch, RAM in DomU is not direct mapped (i.e Guest 
Physical Address == Host Physical Address). This means that DMA-capable 
device would not work properly in DomU.

We could theoritically map DomU direct mapped, but this would break the 
isolation provided by the hypervisor.

What sort of device do you plan to assign?

Best regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Xen ARM Dom0less passthrough without IOMMU

[Andrei Cherechesu]

[-- Attachment #1.1: Type: text/plain, Size: 2852 bytes --]

Hello,

My name is Andrei Cherechesu and I'm a Software Engineer at NXP
Semiconductors in the Automotive department, Linux BSP Team.
I would like to tell you have done a great job so far with Xen.

Thus, we have ported and integrated Xen ARM in the Linux BSP for our
boards.

Currently, we are trying to passthrough devices in order to statically
assign them to different DomUs at boot time (Dom0less configuration).

Our boards do not have an IOMMU and so our work is only possible
with the help of the patches submitted by Stefano Stabellini that
implemented "xen,force-assign-without-iommu" property in the
partial device trees used for passthrough.

I made a configuration in order to setup the dtb /chosen
node to boot a Dom0less VM in parallel with Dom0, with a
peripherical device statically assigned to it.

However, only if I advertise the passthrough dtb node in u-boot,
when I boot Xen, I am given the following error:

(XEN) *** LOADING DOMAIN 0 ***
(XEN) Loading d0 kernel from boot module @ 00000000be800000
(XEN) Loading ramdisk from boot module @ 00000000b1000000
(XEN) Allocating 1:1 mappings totalling 512MB for dom0:
(XEN) BANK[0] 0x000000d0000000-0x000000f0000000 (512MB)
(XEN) Grant table range: 0x00000080200000-0x00000080240000
(XEN) Allocating PPI 16 for event channel interrupt
(XEN) Loading zImage from 00000000be800000 to 00000000d0080000-00000000d0930008
(XEN) Loading dom0 initrd from 00000000b1000000 to 0x00000000d8200000-0x00000000e01fffff
(XEN) Loading dom0 DTB to 0x00000000d8000000-0x00000000d800596a
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Scrubbing Free RAM in background
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) domain.c:322: IOMMU is not enabled
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Error creating domain domU1
(XEN) ****************************************

My Xen is taken from Stefano's tree (http://xenbits.xenproject.org/git-http/people/sstabellini/xen-unstable.git)
that he specified in this<https://lists.xenproject.org/archives/html/xen-devel/2019-10/msg00414.html> thread index, and only has the additional commits
that add support for our boards applied by myself.

From what I've read on the mailing lists, the Dom0less Device
Passthrough Without IOMMU feature is supported, at least,
on boards that do have a physical IOMMU but have some devices that
are not behind it.

My question is: In the way that it is currently implemented,
is the Dom0less Device Passthrough feature currently supported
on boards that do not have an IOMMU at all?

I can provide my complete u-boot environment config commands and
the Xen boot messages.

Thank you for your support,
Andrei Cherechesu

[-- Attachment #1.2: Type: text/html, Size: 6967 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel