Device pass through on XEN on ARM

Device pass through on XEN on ARM

[Khandelwal, Shubham]

[-- Attachment #1.1: Type: text/plain, Size: 264 bytes --]

Hello,

I would like to understand how is pass through achieved on XEN on ARM for devices which do not have a PCI bus controller, like for example embedded devices. Any pointers in this direction will be appreciated.

Thanks & Regards

Shubham Khandelwal


[-- Attachment #1.2: Type: text/html, Size: 2175 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Device pass through on XEN on ARM

[Julien Grall]

On 04/29/2014 12:35 PM, Khandelwal, Shubham wrote:
> Hello,

Hello,

I've dropped the xen-arm list as it has been archived few months ago.

> I would like to understand how is pass through achieved on XEN on ARM
> for devices which do not have a PCI bus controller, like for example
> embedded devices. Any pointers in this direction will be appreciated.

For the moment, Xen 4.5 doesn't support device passthrough. We are
working on 2 distinct solutions to allow non-pci device passthrough.

Arianna has sent a patch series [1] to use "iomem" in the configuration
file for xl. Her solution allows you to passthrough a range of MMIO to a
specific guest. There is no device tree support (i.e xl won't create the
node for your DT node in the device tree generate for the guest), and it
won't work if the device is protected by an IOMMU.

I'm actually working to add support for IRQ passthrough in a similar way.

In a near future (once the latter item is done), I plan to add support
for non-pci passthrough via the device tree.

> Thanks & Regards

Regards,

[1] http://lists.xen.org/archives/html/xen-devel/2014-04/msg02646.html

-- 
Julien Grall

Re: Device pass through on XEN on ARM

[Ian Campbell]

On Thu, 2014-05-01 at 15:44 +0100, Julien Grall wrote:
> On 04/29/2014 12:35 PM, Khandelwal, Shubham wrote:
> > Hello,
> 
> Hello,
> 
> I've dropped the xen-arm list as it has been archived few months ago.
> 
> > I would like to understand how is pass through achieved on XEN on ARM
> > for devices which do not have a PCI bus controller, like for example
> > embedded devices. Any pointers in this direction will be appreciated.
> 
> For the moment, Xen 4.5 doesn't support device passthrough.

People who are using 4.4 have been hardcoding mmio passthrough for
specific platforms though, essentially by hacking the hypervisor code to
do whatever magic was needed.

The method is somewhat similar to the platform specific mappings
callback which allows MMIO regions which are undescribed in the DT to be
passed to dom0. xgene_storm_specific_mapping() is probably the best
example.

Essentially you just need to arrange for map_mmio_region() and
gic_route_irq_to_guest() to be called for the appropriate regions when
constructing the guest domain. Doing this statically for e.g. dom1 is
pretty trivial, just hack some calls in somewhere on the domain create
path with he appropriate checks.

Allowing this to happen for any domain (useful if you want to reboot the
guest domain) is a bit trickier, I'd probably be inclined to create a
custom domctl and arrange for it to be called during domain build for
the appropriate domain.

Obviously this is all very hacky and as Julien says there are various
things in the pipeline for 4.5 which will allow this to be done
properly.

Ian.

Re: Device pass through on XEN on ARM

[Khandelwal, Shubham]

On 05/01/2014 08:26 PM, Ian Campbell wrote:
> On Thu, 2014-05-01 at 15:44 +0100, Julien Grall wrote:
> > On 04/29/2014 12:35 PM, Khandelwal, Shubham wrote:
> > > Hello,
> > 
> > Hello,
> > 
> > I've dropped the xen-arm list as it has been archived few months ago.
> > 
> > > I would like to understand how is pass through achieved on XEN on ARM
> > > for devices which do not have a PCI bus controller, like for example
> > > embedded devices. Any pointers in this direction will be appreciated.
> > 
> > For the moment, Xen 4.5 doesn't support device passthrough.
>
> People who are using 4.4 have been hardcoding mmio passthrough for
> specific platforms though, essentially by hacking the hypervisor code to
> do whatever magic was needed.
>
> The method is somewhat similar to the platform specific mappings
> callback which allows MMIO regions which are undescribed in the DT to be
> passed to dom0. xgene_storm_specific_mapping() is probably the best
> example.
>
> Essentially you just need to arrange for map_mmio_region() and
> gic_route_irq_to_guest() to be called for the appropriate regions when
> constructing the guest domain. Doing this statically for e.g. dom1 is
> pretty trivial, just hack some calls in somewhere on the domain create
> path with he appropriate checks.
>
> Allowing this to happen for any domain (useful if you want to reboot the
> guest domain) is a bit trickier, I'd probably be inclined to create a
> custom domctl and arrange for it to be called during domain build for
> the appropriate domain.
>
> Obviously this is all very hacky and as Julien says there are various
> things in the pipeline for 4.5 which will allow this to be done
> properly.
>
> Ian.

Hello Ian, Julien,

Just to try out the passthrough using the hacks you mentioned, I made changes in xen to assign a GPIO controlled LED to domU by calling map_mmio_regions() in the domain creation path. After domU boots, when I try to access the LED using ioremap, ioread and iowrite from kernel space I see that I am not able to control the LED, also when I try to control LED using sysfs from user space it shows 'no such device' error.

I am using xen 4.5 unstable and I have made the following changes in xen/common/domctl.c :: do_domctl(), after line 473:



int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000, 0x4805B000);
if (passthrough)
        printk("Failed to map passthrough_LED @ 0x4805B000 to dom%d\n", d->domain_id); 



While booting domU, in the logs I can see there is no error in map_mmio_regions(). Am I missing something here.

Thanks and Regards

Shubham Khandelwal

Re: Device pass through on XEN on ARM

[Ian Campbell]

On Wed, 2014-06-11 at 11:16 +0200, Khandelwal, Shubham wrote:
> Just to try out the passthrough using the hacks you mentioned, I made
> changes in xen to assign a GPIO controlled LED to domU by calling
> map_mmio_regions() in the domain creation path. After domU boots, when
> I try to access the LED using ioremap, ioread and iowrite from kernel
> space I see that I am not able to control the LED,

How does it fail? Does it give a fault in the hypervisor or the guest
kernel or just silently eat the MMIO writes and not do anything?

>  also when I try to control LED using sysfs from user space it shows
> 'no such device' error.
> 
> I am using xen 4.5 unstable and I have made the following changes in
> xen/common/domctl.c :: do_domctl(), after line 473:
> 
> 
> 
> int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000, 0x4805B000);
> if (passthrough)
>         printk("Failed to map passthrough_LED @ 0x4805B000 to dom%d\n", d->domain_id); 
> 
> 
> 
> While booting domU, in the logs I can see there is no error in
> map_mmio_regions(). Am I missing something here.

0x4xxxxxxx is a RAM region on 4.5 (see the end of
xen/include/public/arch-arm.h), I suspect your mmio mapping is being
clobbered by a RAM mapping (and therefore that your answer above is
"silently does nothing".

Do you need this to be a 1:1 mapping or could you move it to some free
space? Most of the low GB is free (but do check arch-arm.h) to be used
for MMIO mappings.

Ian.

Re: Device pass through on XEN on ARM

[Gordan Bobic]

On 2014-06-11 10:42, Ian Campbell wrote:
> On Wed, 2014-06-11 at 11:16 +0200, Khandelwal, Shubham wrote:
>> Just to try out the passthrough using the hacks you mentioned, I made
>> changes in xen to assign a GPIO controlled LED to domU by calling
>> map_mmio_regions() in the domain creation path. After domU boots, when
>> I try to access the LED using ioremap, ioread and iowrite from kernel
>> space I see that I am not able to control the LED,
> 
> How does it fail? Does it give a fault in the hypervisor or the guest
> kernel or just silently eat the MMIO writes and not do anything?
> 
>>  also when I try to control LED using sysfs from user space it shows
>> 'no such device' error.
>> 
>> I am using xen 4.5 unstable and I have made the following changes in
>> xen/common/domctl.c :: do_domctl(), after line 473:
>> 
>> 
>> 
>> int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000, 
>> 0x4805B000);
>> if (passthrough)
>>         printk("Failed to map passthrough_LED @ 0x4805B000 to 
>> dom%d\n", d->domain_id);
>> 
>> 
>> 
>> While booting domU, in the logs I can see there is no error in
>> map_mmio_regions(). Am I missing something here.
> 
> 0x4xxxxxxx is a RAM region on 4.5 (see the end of
> xen/include/public/arch-arm.h), I suspect your mmio mapping is being
> clobbered by a RAM mapping (and therefore that your answer above is
> "silently does nothing".
> 
> Do you need this to be a 1:1 mapping or could you move it to some free
> space? Most of the low GB is free (but do check arch-arm.h) to be used
> for MMIO mappings.

I have a sneaky suspicion that part of the problem is that the
offending BAR is 16GB in size.

Gordan

Re: Device pass through on XEN on ARM

[Ian Campbell]

On Wed, 2014-06-11 at 10:53 +0100, Gordan Bobic wrote:
> On 2014-06-11 10:42, Ian Campbell wrote:
> > On Wed, 2014-06-11 at 11:16 +0200, Khandelwal, Shubham wrote:
> >> Just to try out the passthrough using the hacks you mentioned, I made
> >> changes in xen to assign a GPIO controlled LED to domU by calling
> >> map_mmio_regions() in the domain creation path. After domU boots, when
> >> I try to access the LED using ioremap, ioread and iowrite from kernel
> >> space I see that I am not able to control the LED,
> > 
> > How does it fail? Does it give a fault in the hypervisor or the guest
> > kernel or just silently eat the MMIO writes and not do anything?
> > 
> >>  also when I try to control LED using sysfs from user space it shows
> >> 'no such device' error.
> >> 
> >> I am using xen 4.5 unstable and I have made the following changes in
> >> xen/common/domctl.c :: do_domctl(), after line 473:
> >> 
> >> 
> >> 
> >> int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000, 
> >> 0x4805B000);
> >> if (passthrough)
> >>         printk("Failed to map passthrough_LED @ 0x4805B000 to 
> >> dom%d\n", d->domain_id);
> >> 
> >> 
> >> 
> >> While booting domU, in the logs I can see there is no error in
> >> map_mmio_regions(). Am I missing something here.
> > 
> > 0x4xxxxxxx is a RAM region on 4.5 (see the end of
> > xen/include/public/arch-arm.h), I suspect your mmio mapping is being
> > clobbered by a RAM mapping (and therefore that your answer above is
> > "silently does nothing".
> > 
> > Do you need this to be a 1:1 mapping or could you move it to some free
> > space? Most of the low GB is free (but do check arch-arm.h) to be used
> > for MMIO mappings.
> 
> I have a sneaky suspicion that part of the problem is that the
> offending BAR is 16GB in size.

What makes you think that? The region is 4K according to the lines shown
above.

Since you mention BAR (a PCI thing) I've a sneaking suspicious this
isn't the thread you think it is ;-)

Ian.

Re: Device pass through on XEN on ARM

[Gordan Bobic]

On 2014-06-11 11:00, Ian Campbell wrote:
> On Wed, 2014-06-11 at 10:53 +0100, Gordan Bobic wrote:
>> On 2014-06-11 10:42, Ian Campbell wrote:
>> > On Wed, 2014-06-11 at 11:16 +0200, Khandelwal, Shubham wrote:
>> >> Just to try out the passthrough using the hacks you mentioned, I made
>> >> changes in xen to assign a GPIO controlled LED to domU by calling
>> >> map_mmio_regions() in the domain creation path. After domU boots, when
>> >> I try to access the LED using ioremap, ioread and iowrite from kernel
>> >> space I see that I am not able to control the LED,
>> >
>> > How does it fail? Does it give a fault in the hypervisor or the guest
>> > kernel or just silently eat the MMIO writes and not do anything?
>> >
>> >>  also when I try to control LED using sysfs from user space it shows
>> >> 'no such device' error.
>> >>
>> >> I am using xen 4.5 unstable and I have made the following changes in
>> >> xen/common/domctl.c :: do_domctl(), after line 473:
>> >>
>> >>
>> >>
>> >> int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000,
>> >> 0x4805B000);
>> >> if (passthrough)
>> >>         printk("Failed to map passthrough_LED @ 0x4805B000 to
>> >> dom%d\n", d->domain_id);
>> >>
>> >>
>> >>
>> >> While booting domU, in the logs I can see there is no error in
>> >> map_mmio_regions(). Am I missing something here.
>> >
>> > 0x4xxxxxxx is a RAM region on 4.5 (see the end of
>> > xen/include/public/arch-arm.h), I suspect your mmio mapping is being
>> > clobbered by a RAM mapping (and therefore that your answer above is
>> > "silently does nothing".
>> >
>> > Do you need this to be a 1:1 mapping or could you move it to some free
>> > space? Most of the low GB is free (but do check arch-arm.h) to be used
>> > for MMIO mappings.
>> 
>> I have a sneaky suspicion that part of the problem is that the
>> offending BAR is 16GB in size.
> 
> What makes you think that? The region is 4K according to the lines 
> shown
> above.
> 
> Since you mention BAR (a PCI thing) I've a sneaking suspicious this
> isn't the thread you think it is ;-)

D'oh! My mail-foo is weak today. I was reading one thread but somehow
managed to post on a different one. Apologies for the noise. :(

Gordan

Re: Device pass through on XEN on ARM

[Khandelwal, Shubham]

Hello Ian,

On 06/11/2014 03:13 PM, Ian Campbell wrote:
> > Just to try out the passthrough using the hacks you mentioned, I made
> > changes in xen to assign a GPIO controlled LED to domU by calling
> > map_mmio_regions() in the domain creation path. After domU boots, when
> > I try to access the LED using ioremap, ioread and iowrite from kernel
> > space I see that I am not able to control the LED,

> How does it fail? Does it give a fault in the hypervisor or the guest
> kernel or just silently eat the MMIO writes and not do anything?

There is a panic in the guest kernel


> 0x4xxxxxxx is a RAM region on 4.5 (see the end of
> xen/include/public/arch-arm.h), I suspect your mmio mapping is being
> clobbered by a RAM mapping (and therefore that your answer above is
> "silently does nothing".

I just checked xen/include/public/arch-arm.h Line 372 is 

#define GUEST_RAM_BASE    0x80000000ULL

So as per my understanding 0x4xxxxxxx is not a ram region and thus there should not be any clobbering.


> Do you need this to be a 1:1 mapping or could you move it to some free
> space? Most of the low GB is free (but do check arch-arm.h) to be used
> for MMIO mappings.

I do not necessarily need it to be a 1:1 mapping.

> Ian.

Regards

Shubham Khandelwal

Re: Device pass through on XEN on ARM

[Ian Campbell]

On Thu, 2014-06-12 at 10:36 +0200, Khandelwal, Shubham wrote:
> Hello Ian,
> 
> On 06/11/2014 03:13 PM, Ian Campbell wrote:
> > > Just to try out the passthrough using the hacks you mentioned, I made
> > > changes in xen to assign a GPIO controlled LED to domU by calling
> > > map_mmio_regions() in the domain creation path. After domU boots, when
> > > I try to access the LED using ioremap, ioread and iowrite from kernel
> > > space I see that I am not able to control the LED,
> 
> > How does it fail? Does it give a fault in the hypervisor or the guest
> > kernel or just silently eat the MMIO writes and not do anything?
> 
> There is a panic in the guest kernel

Can you post it please.

You might also want to add debug to xen/arch/arm/traps.c inject_* to see
if this is a trap which is going via Xen and being redirected to the
guest or if it is generated directly by the MMU.

If it is going via Xen then you could call dump_p2m_lookup on the guest
IPA and see if it looks valid.

> > 0x4xxxxxxx is a RAM region on 4.5 (see the end of
> > xen/include/public/arch-arm.h), I suspect your mmio mapping is being
> > clobbered by a RAM mapping (and therefore that your answer above is
> > "silently does nothing".
> 
> I just checked xen/include/public/arch-arm.h Line 372 is 
> 
> #define GUEST_RAM_BASE    0x80000000ULL
> 
> So as per my understanding 0x4xxxxxxx is not a ram region and thus there should not be any clobbering.

In current 4.5 it is:
 #define GUEST_RAM0_BASE   0x40000000ULL /* 3GB of low RAM @ 1GB */

I suppose you are using 4.5 from before commit 2d72d29bd432 or so.

> > Do you need this to be a 1:1 mapping or could you move it to some free
> > space? Most of the low GB is free (but do check arch-arm.h) to be used
> > for MMIO mappings.
> 
> I do not necessarily need it to be a 1:1 mapping.

If you were clashing with RAM then you could move it to wherever you
wanted then. But given that your rambase is 0x80000000 that doesn't
sound like the issue.

> 
> > Ian.
> 
> Regards
> 
> Shubham Khandelwal

Re: Device pass through on XEN on ARM

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2196 bytes --]

On mer, 2014-06-11 at 11:16 +0200, Khandelwal, Shubham wrote:
> On 05/01/2014 08:26 PM, Ian Campbell wrote:

> > Obviously this is all very hacky and as Julien says there are various
> > things in the pipeline for 4.5 which will allow this to be done
> > properly.
> >
> > Ian.
> 
> Hello Ian, Julien,
> 
> Just to try out the passthrough using the hacks you mentioned, I made changes in xen to assign a GPIO controlled LED to domU by calling map_mmio_regions() in the domain creation path. After domU boots, when I try to access the LED using ioremap, ioread and iowrite from kernel space I see that I am not able to control the LED, also when I try to control LED using sysfs from user space it shows 'no such device' error.
> 
> I am using xen 4.5 unstable and I have made the following changes in xen/common/domctl.c :: do_domctl(), after line 473:
> 
> 
> 
> int passthrough = map_mmio_regions(d, 0x4805B000, 0x4805B000 + 0x1000, 0x4805B000);
> if (passthrough)
>         printk("Failed to map passthrough_LED @ 0x4805B000 to dom%d\n", d->domain_id); 
> 
> 
> 
> While booting domU, in the logs I can see there is no error in map_mmio_regions(). Am I missing something here.
> 
May I suggest trying to apply Arianna's series, and see whether that
helps? IIRC, there should even be the possibility of setting up non 1:1
mappings, directly from the config file (uless that got dropped in the
latest versions, which I could not follow closely).

In any case, it look like both less work, and a more tested path (given
the amount of attention and review the series is getting). It also would
be greeat to have some more testing of the series itself, not to mention
that, if it works, you'll be all set for when the series will go
upstream! :-P

I believe the last version of Arianna's series is this one:
  http://markmail.org/thread/wtevxxizyxjhvmg2

Just 2 cents... :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Device pass through on XEN on ARM

[Julien Grall]

(Adding back Xen devel)

Please don't drop Xen devel.

On 22/05/14 13:24, Khandelwal, Shubham wrote:
> Hello Julien,

Hello,

> Thank you for this information.
>
> So is this patch merged with mainline xen. From where can I download xen containing these patches already.

The patch series from Arianna (implement memory mapping DOMCTL) is in 
process review. IIRC, she is currently preparing a new version of the 
series.

For the other series (IRQ passthrough & common DT passthrough). I'm 
actually working on it. I don't have any idea when I will be able to 
send an RFC on the mailing list.

Regards,

-- 
Julien Grall