[Qemu-devel] Direct guest device access from nested guest

[Qemu-devel] Direct guest device access from nested guest

[Lluís Vilanova]

Hi,

I want to get the following setup, but don't know how (or if it's even
possible):

* A guest VM with two AHCI controllers, with one device each. One of the AHCI
  controllers provides the VM's disk ("system"), while the other provides
  another disk ("nested") and uses a different emulation driver in QEMU (ahci2):

  host$ qemu-system-x86_64 -enable-kvm \
            -drive id=system,file=system.img,if=none \
            -device ahci,id=ahci \
            -device ide-drive,drive=system,bus=ahci.0 \
            -drive id=nested,file=nested.img,if=none \
            -device ahci2,id=ahci2 \
            -device ide-drive,drive=nested,bus=ahci2.0

* A nested guest VM using the guest's (its host) AHCI2 controller.

  I've tried assigning the AHCI2 device to the nested guest using "pci-assign"
  and "vfio", but without any luck.

  The culprit of the problem seems to be I cannot get the nested guest to have
  an IOMMU.

Is it possible to give a nested guest direct access to a device on the guest?
(more specifically, an AHCI controller).


Thanks a lot,
  Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Direct guest device access from nested guest

[Jan Kiszka]

On 2013-08-28 16:28, Lluís Vilanova wrote:
> Hi,
> 
> I want to get the following setup, but don't know how (or if it's even
> possible):
> 
> * A guest VM with two AHCI controllers, with one device each. One of the AHCI
>   controllers provides the VM's disk ("system"), while the other provides
>   another disk ("nested") and uses a different emulation driver in QEMU (ahci2):
> 
>   host$ qemu-system-x86_64 -enable-kvm \
>             -drive id=system,file=system.img,if=none \
>             -device ahci,id=ahci \
>             -device ide-drive,drive=system,bus=ahci.0 \
>             -drive id=nested,file=nested.img,if=none \
>             -device ahci2,id=ahci2 \
>             -device ide-drive,drive=nested,bus=ahci2.0
> 
> * A nested guest VM using the guest's (its host) AHCI2 controller.
> 
>   I've tried assigning the AHCI2 device to the nested guest using "pci-assign"
>   and "vfio", but without any luck.
> 
>   The culprit of the problem seems to be I cannot get the nested guest to have
>   an IOMMU.
> 
> Is it possible to give a nested guest direct access to a device on the guest?
> (more specifically, an AHCI controller).

Nope, we are lacking support for emulating or (securely) forwarding
VT-d/IOMMU features to the first level guest. Would be cool to have,
just not yet there. But I've talked to Intel people recently, and they
are considering to support some nested VT-d with KVM.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Direct guest device access from nested guest

[Jan Kiszka]

On 2013-08-28 16:28, Lluís Vilanova wrote:
> Hi,
> 
> I want to get the following setup, but don't know how (or if it's even
> possible):
> 
> * A guest VM with two AHCI controllers, with one device each. One of the AHCI
>   controllers provides the VM's disk ("system"), while the other provides
>   another disk ("nested") and uses a different emulation driver in QEMU (ahci2):
> 
>   host$ qemu-system-x86_64 -enable-kvm \
>             -drive id=system,file=system.img,if=none \
>             -device ahci,id=ahci \
>             -device ide-drive,drive=system,bus=ahci.0 \
>             -drive id=nested,file=nested.img,if=none \
>             -device ahci2,id=ahci2 \
>             -device ide-drive,drive=nested,bus=ahci2.0
> 
> * A nested guest VM using the guest's (its host) AHCI2 controller.
> 
>   I've tried assigning the AHCI2 device to the nested guest using "pci-assign"
>   and "vfio", but without any luck.
> 
>   The culprit of the problem seems to be I cannot get the nested guest to have
>   an IOMMU.
> 
> Is it possible to give a nested guest direct access to a device on the guest?
> (more specifically, an AHCI controller).

Nope, we are lacking support for emulating or (securely) forwarding
VT-d/IOMMU features to the first level guest. Would be cool to have,
just not yet there. But I've talked to Intel people recently, and they
are considering to support some nested VT-d with KVM.

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: Direct guest device access from nested guest

[Lluís Vilanova]

Jan Kiszka writes:
[...]
>> Is it possible to give a nested guest direct access to a device on the guest?
>> (more specifically, an AHCI controller).

> Nope, we are lacking support for emulating or (securely) forwarding
> VT-d/IOMMU features to the first level guest. Would be cool to have,
> just not yet there. But I've talked to Intel people recently, and they
> are considering to support some nested VT-d with KVM.

Thanks a lot. I've been told there's some patches floating around to add such
support, but I suppose they've been long outdated and only work as POCs.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: [Qemu-devel] Direct guest device access from nested guest

[Lluís Vilanova]

Jan Kiszka writes:
[...]
>> Is it possible to give a nested guest direct access to a device on the guest?
>> (more specifically, an AHCI controller).

> Nope, we are lacking support for emulating or (securely) forwarding
> VT-d/IOMMU features to the first level guest. Would be cool to have,
> just not yet there. But I've talked to Intel people recently, and they
> are considering to support some nested VT-d with KVM.

Thanks a lot. I've been told there's some patches floating around to add such
support, but I suppose they've been long outdated and only work as POCs.


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Direct guest device access from nested guest

[Jan Kiszka]

On 2013-08-28 20:12, Lluís Vilanova wrote:
> Jan Kiszka writes:
> [...]
>>> Is it possible to give a nested guest direct access to a device on the guest?
>>> (more specifically, an AHCI controller).
> 
>> Nope, we are lacking support for emulating or (securely) forwarding
>> VT-d/IOMMU features to the first level guest. Would be cool to have,
>> just not yet there. But I've talked to Intel people recently, and they
>> are considering to support some nested VT-d with KVM.
> 
> Thanks a lot. I've been told there's some patches floating around to add such
> support, but I suppose they've been long outdated and only work as POCs.

I haven't seen anything in public.

Jan

PS: You have Mail-Followup-To set in your answers - people will drop you
from CC this way.

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Direct guest device access from nested guest

[Jan Kiszka]

On 2013-08-28 20:12, Lluís Vilanova wrote:
> Jan Kiszka writes:
> [...]
>>> Is it possible to give a nested guest direct access to a device on the guest?
>>> (more specifically, an AHCI controller).
> 
>> Nope, we are lacking support for emulating or (securely) forwarding
>> VT-d/IOMMU features to the first level guest. Would be cool to have,
>> just not yet there. But I've talked to Intel people recently, and they
>> are considering to support some nested VT-d with KVM.
> 
> Thanks a lot. I've been told there's some patches floating around to add such
> support, but I suppose they've been long outdated and only work as POCs.

I haven't seen anything in public.

Jan

PS: You have Mail-Followup-To set in your answers - people will drop you
from CC this way.

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Qemu-devel] Direct guest device access from nested guest

[Lluís Vilanova]

Jan Kiszka writes:

> On 2013-08-28 20:12, Lluís Vilanova wrote:
>> Jan Kiszka writes:
>> [...]
>>>> Is it possible to give a nested guest direct access to a device on the guest?
>>>> (more specifically, an AHCI controller).
>> 
>>> Nope, we are lacking support for emulating or (securely) forwarding
>>> VT-d/IOMMU features to the first level guest. Would be cool to have,
>>> just not yet there. But I've talked to Intel people recently, and they
>>> are considering to support some nested VT-d with KVM.
>> 
>> Thanks a lot. I've been told there's some patches floating around to add such
>> support, but I suppose they've been long outdated and only work as POCs.

> I haven't seen anything in public.

This is what I've found:

  https://lists.nongnu.org/archive/html/qemu-devel/2011-04/msg01970.html


> PS: You have Mail-Followup-To set in your answers - people will drop you
> from CC this way.

Yes, my mail client tried to be clever but didn't know I'm not subscribed to the
KVM list :)


Lluis

-- 
 "And it's much the same thing with knowledge, for whenever you learn
 something new, the whole world becomes that much richer."
 -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
 Tollbooth

Re: Direct guest device access from nested guest

[Aaron Fabbri]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

Has anyone considered a paravirt approach?  That is:

Guest kernel:  Write a new IOMMU API back end which does KVM hypercalls.
 Exposes VFIO to guest user processes (nested VMs) as usual.

Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
host_pa} mappings to {guest_va -> host_pa}, and call through to underlying
IOMMU.

Opinions?



On Wed, Aug 28, 2013 at 12:18 PM, Lluís Vilanova <vilanova@ac.upc.edu>wrote:

> Jan Kiszka writes:
>
> > On 2013-08-28 20:12, Lluís Vilanova wrote:
> >> Jan Kiszka writes:
> >> [...]
> >>>> Is it possible to give a nested guest direct access to a device on
> the guest?
> >>>> (more specifically, an AHCI controller).
> >>
> >>> Nope, we are lacking support for emulating or (securely) forwarding
> >>> VT-d/IOMMU features to the first level guest. Would be cool to have,
> >>> just not yet there. But I've talked to Intel people recently, and they
> >>> are considering to support some nested VT-d with KVM.
> >>
> >> Thanks a lot. I've been told there's some patches floating around to
> add such
> >> support, but I suppose they've been long outdated and only work as POCs.
>
> > I haven't seen anything in public.
>
> This is what I've found:
>
>   https://lists.nongnu.org/archive/html/qemu-devel/2011-04/msg01970.html
>
>
> > PS: You have Mail-Followup-To set in your answers - people will drop you
> > from CC this way.
>
> Yes, my mail client tried to be clever but didn't know I'm not subscribed
> to the
> KVM list :)
>
>
> Lluis
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>
>

[-- Attachment #2: Type: text/html, Size: 2588 bytes --]

Re: [Qemu-devel] Direct guest device access from nested guest

[Aaron Fabbri]

[-- Attachment #1: Type: text/plain, Size: 1782 bytes --]

Has anyone considered a paravirt approach?  That is:

Guest kernel:  Write a new IOMMU API back end which does KVM hypercalls.
 Exposes VFIO to guest user processes (nested VMs) as usual.

Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
host_pa} mappings to {guest_va -> host_pa}, and call through to underlying
IOMMU.

Opinions?



On Wed, Aug 28, 2013 at 12:18 PM, Lluís Vilanova <vilanova@ac.upc.edu>wrote:

> Jan Kiszka writes:
>
> > On 2013-08-28 20:12, Lluís Vilanova wrote:
> >> Jan Kiszka writes:
> >> [...]
> >>>> Is it possible to give a nested guest direct access to a device on
> the guest?
> >>>> (more specifically, an AHCI controller).
> >>
> >>> Nope, we are lacking support for emulating or (securely) forwarding
> >>> VT-d/IOMMU features to the first level guest. Would be cool to have,
> >>> just not yet there. But I've talked to Intel people recently, and they
> >>> are considering to support some nested VT-d with KVM.
> >>
> >> Thanks a lot. I've been told there's some patches floating around to
> add such
> >> support, but I suppose they've been long outdated and only work as POCs.
>
> > I haven't seen anything in public.
>
> This is what I've found:
>
>   https://lists.nongnu.org/archive/html/qemu-devel/2011-04/msg01970.html
>
>
> > PS: You have Mail-Followup-To set in your answers - people will drop you
> > from CC this way.
>
> Yes, my mail client tried to be clever but didn't know I'm not subscribed
> to the
> KVM list :)
>
>
> Lluis
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>
>

[-- Attachment #2: Type: text/html, Size: 2588 bytes --]

Fwd: [Qemu-devel] Direct guest device access from nested guest

[Aaron Fabbri]

Sorry. Resending in plain text.  (Gmail).

---------- Forwarded message ----------

Has anyone considered a paravirt approach?  That is:

Guest kernel:  Write a new IOMMU API back end which does KVM
hypercalls.  Exposes VFIO to guest user processes (nested VMs) as
usual.

Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
host_pa} mappings to {guest_va -> host_pa}, and call through to
underlying IOMMU.

Opinions?



On Wed, Aug 28, 2013 at 12:18 PM, Lluís Vilanova <vilanova@ac.upc.edu> wrote:
>
> Jan Kiszka writes:
>
> > On 2013-08-28 20:12, Lluís Vilanova wrote:
> >> Jan Kiszka writes:
> >> [...]
> >>>> Is it possible to give a nested guest direct access to a device on the guest?
> >>>> (more specifically, an AHCI controller).
> >>
> >>> Nope, we are lacking support for emulating or (securely) forwarding
> >>> VT-d/IOMMU features to the first level guest. Would be cool to have,
> >>> just not yet there. But I've talked to Intel people recently, and they
> >>> are considering to support some nested VT-d with KVM.
> >>
> >> Thanks a lot. I've been told there's some patches floating around to add such
> >> support, but I suppose they've been long outdated and only work as POCs.
>
> > I haven't seen anything in public.
>
> This is what I've found:
>
>   https://lists.nongnu.org/archive/html/qemu-devel/2011-04/msg01970.html
>
>
> > PS: You have Mail-Followup-To set in your answers - people will drop you
> > from CC this way.
>
> Yes, my mail client tried to be clever but didn't know I'm not subscribed to the
> KVM list :)
>
>
> Lluis
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>

[Qemu-devel] Fwd: Direct guest device access from nested guest

[Aaron Fabbri]

Sorry. Resending in plain text.  (Gmail).

---------- Forwarded message ----------

Has anyone considered a paravirt approach?  That is:

Guest kernel:  Write a new IOMMU API back end which does KVM
hypercalls.  Exposes VFIO to guest user processes (nested VMs) as
usual.

Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
host_pa} mappings to {guest_va -> host_pa}, and call through to
underlying IOMMU.

Opinions?



On Wed, Aug 28, 2013 at 12:18 PM, Lluís Vilanova <vilanova@ac.upc.edu> wrote:
>
> Jan Kiszka writes:
>
> > On 2013-08-28 20:12, Lluís Vilanova wrote:
> >> Jan Kiszka writes:
> >> [...]
> >>>> Is it possible to give a nested guest direct access to a device on the guest?
> >>>> (more specifically, an AHCI controller).
> >>
> >>> Nope, we are lacking support for emulating or (securely) forwarding
> >>> VT-d/IOMMU features to the first level guest. Would be cool to have,
> >>> just not yet there. But I've talked to Intel people recently, and they
> >>> are considering to support some nested VT-d with KVM.
> >>
> >> Thanks a lot. I've been told there's some patches floating around to add such
> >> support, but I suppose they've been long outdated and only work as POCs.
>
> > I haven't seen anything in public.
>
> This is what I've found:
>
>   https://lists.nongnu.org/archive/html/qemu-devel/2011-04/msg01970.html
>
>
> > PS: You have Mail-Followup-To set in your answers - people will drop you
> > from CC this way.
>
> Yes, my mail client tried to be clever but didn't know I'm not subscribed to the
> KVM list :)
>
>
> Lluis
>
> --
>  "And it's much the same thing with knowledge, for whenever you learn
>  something new, the whole world becomes that much richer."
>  -- The Princess of Pure Reason, as told by Norton Juster in The Phantom
>  Tollbooth
>

Re: [Qemu-devel] Direct guest device access from nested guest

[Muli Ben-Yehuda]

On Thu, Aug 29, 2013 at 03:55:20PM -0700, Aaron Fabbri wrote:

> Has anyone considered a paravirt approach?  That is:
> 
> Guest kernel:  Write a new IOMMU API back end which does KVM hypercalls.
>  Exposes VFIO to guest user processes (nested VMs) as usual.
> 
> Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
> host_pa} mappings to {guest_va -> host_pa}, and call through to
> underlying IOMMU.
> 
> Opinions?

The paravirt approach can certainly work but has a couple of
drawbacks. First, you need to modify the guest kernel
(obviously). Second, frequent map/unmap calls can be very expensive,
so you probably want to relax protection somewhat and cache DMA
mappings, preferably at the guest level or at the host level[1].

Personally, I think that emulating an IOMMU is the right way to go
with current generation hardware, and can provide very reasonable
performance if you are willing to relax protection[2].

[1] http://www.mulix.org/pubs/iommu/dmamapping.pdf
[2] http://www.mulix.org/pubs/iommu/viommu.pdf

Cheers,
Muli

Re: [Qemu-devel] Direct guest device access from nested guest

[Muli Ben-Yehuda]

On Thu, Aug 29, 2013 at 03:55:20PM -0700, Aaron Fabbri wrote:

> Has anyone considered a paravirt approach?  That is:
> 
> Guest kernel:  Write a new IOMMU API back end which does KVM hypercalls.
>  Exposes VFIO to guest user processes (nested VMs) as usual.
> 
> Host kernel: KVM does things like collapse {guest_va -> guest_pa ->
> host_pa} mappings to {guest_va -> host_pa}, and call through to
> underlying IOMMU.
> 
> Opinions?

The paravirt approach can certainly work but has a couple of
drawbacks. First, you need to modify the guest kernel
(obviously). Second, frequent map/unmap calls can be very expensive,
so you probably want to relax protection somewhat and cache DMA
mappings, preferably at the guest level or at the host level[1].

Personally, I think that emulating an IOMMU is the right way to go
with current generation hardware, and can provide very reasonable
performance if you are willing to relax protection[2].

[1] http://www.mulix.org/pubs/iommu/dmamapping.pdf
[2] http://www.mulix.org/pubs/iommu/viommu.pdf

Cheers,
Muli