Query on VFIO in Virtual machine

Query on VFIO in Virtual machine

[Nitin Saxena]

Hi,

I have a PCI device connected as an endpoint to Intel host machine.
The requirement is to run dpdk like user space data path application
in VM using PCI PF passthrough (SRIOV disabled). This application
works fine on host kernel and uses VFIO to get MSIX interrupts from
PCI device. We are trying to run this existing application in VM using
PCI passthrough. This application has capability to use
VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU.

On Intel host machine VT-d has been enabled and using virt-manager PCI
device PF is assigned to the VM. This makes virt-manager to implicitly
binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel
was booted with intel_iommu=on as boot parameter.

My question: Is it possible that vfio can coexist in host (by
virt-manager) as well as VM (by application)? If yes, does application
running inside VM needs to configure VFIO with iommu_type=IOMMU or
iommu_type=no-iommu.

In VM I tried inserting vfio_iommu_type1.ko kernel module which failed
with "No such device error". Thats why I am confused whether my
requirement is legitimate or not. What could be the best solution?

Thanks in advance
Nitin

Re: Query on VFIO in Virtual machine

[Alex Williamson]

[cc +qemu-devel, +peterx]

On Thu, 22 Jun 2017 22:18:06 +0530
Nitin Saxena <nitin.lnx@gmail.com> wrote:

> Hi,
> 
> I have a PCI device connected as an endpoint to Intel host machine.
> The requirement is to run dpdk like user space data path application
> in VM using PCI PF passthrough (SRIOV disabled). This application
> works fine on host kernel and uses VFIO to get MSIX interrupts from
> PCI device. We are trying to run this existing application in VM using
> PCI passthrough. This application has capability to use
> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU.
> 
> On Intel host machine VT-d has been enabled and using virt-manager PCI
> device PF is assigned to the VM. This makes virt-manager to implicitly
> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel
> was booted with intel_iommu=on as boot parameter.
> 
> My question: Is it possible that vfio can coexist in host (by
> virt-manager) as well as VM (by application)? If yes, does application
> running inside VM needs to configure VFIO with iommu_type=IOMMU or
> iommu_type=no-iommu.
> 
> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed
> with "No such device error". Thats why I am confused whether my
> requirement is legitimate or not. What could be the best solution?

This is really more of a QEMU question.  In order to use
vfio_iommu_type1 in the guest, you need an iommu in the guest.  The
most recent release of QEMU supports this with an emulated VT-d
device.  Therefore if you create a VM with emulated VT-d and a device
assigned through vfio-pci, you can expose it to userspace in the VM with
physical iommu protection.  Without an iommu in the VM, you'd be
limited to no-iommu support for VM userspace, the physical iommu would
only protect the device to the extent of VM memory, no to specific
userspace mappings within the VM.  Thanks,

Alex

Re: [Qemu-devel] Query on VFIO in Virtual machine

[Alex Williamson]

[cc +qemu-devel, +peterx]

On Thu, 22 Jun 2017 22:18:06 +0530
Nitin Saxena <nitin.lnx@gmail.com> wrote:

> Hi,
> 
> I have a PCI device connected as an endpoint to Intel host machine.
> The requirement is to run dpdk like user space data path application
> in VM using PCI PF passthrough (SRIOV disabled). This application
> works fine on host kernel and uses VFIO to get MSIX interrupts from
> PCI device. We are trying to run this existing application in VM using
> PCI passthrough. This application has capability to use
> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU.
> 
> On Intel host machine VT-d has been enabled and using virt-manager PCI
> device PF is assigned to the VM. This makes virt-manager to implicitly
> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel
> was booted with intel_iommu=on as boot parameter.
> 
> My question: Is it possible that vfio can coexist in host (by
> virt-manager) as well as VM (by application)? If yes, does application
> running inside VM needs to configure VFIO with iommu_type=IOMMU or
> iommu_type=no-iommu.
> 
> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed
> with "No such device error". Thats why I am confused whether my
> requirement is legitimate or not. What could be the best solution?

This is really more of a QEMU question.  In order to use
vfio_iommu_type1 in the guest, you need an iommu in the guest.  The
most recent release of QEMU supports this with an emulated VT-d
device.  Therefore if you create a VM with emulated VT-d and a device
assigned through vfio-pci, you can expose it to userspace in the VM with
physical iommu protection.  Without an iommu in the VM, you'd be
limited to no-iommu support for VM userspace, the physical iommu would
only protect the device to the extent of VM memory, no to specific
userspace mappings within the VM.  Thanks,

Alex

Re: Query on VFIO in Virtual machine

[Nitin Saxena]

Thanks Alex.

>> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,
So are you trying to say VFIO NO-IOMMU should work inside VM. Does
that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
is a legitimate configuration? I did tried this configuration and the
application (in VM) seems to get container_fd, group_fd, device_fd
successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
from VM as well as from host. This could be specific to PCI endpoint
device which I can dig.

I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
device is legitimate configuration.

Thanks
Nitin

On Thu, Jun 22, 2017 at 10:29 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> [cc +qemu-devel, +peterx]
>
> On Thu, 22 Jun 2017 22:18:06 +0530
> Nitin Saxena <nitin.lnx@gmail.com> wrote:
>
>> Hi,
>>
>> I have a PCI device connected as an endpoint to Intel host machine.
>> The requirement is to run dpdk like user space data path application
>> in VM using PCI PF passthrough (SRIOV disabled). This application
>> works fine on host kernel and uses VFIO to get MSIX interrupts from
>> PCI device. We are trying to run this existing application in VM using
>> PCI passthrough. This application has capability to use
>> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU.
>>
>> On Intel host machine VT-d has been enabled and using virt-manager PCI
>> device PF is assigned to the VM. This makes virt-manager to implicitly
>> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel
>> was booted with intel_iommu=on as boot parameter.
>>
>> My question: Is it possible that vfio can coexist in host (by
>> virt-manager) as well as VM (by application)? If yes, does application
>> running inside VM needs to configure VFIO with iommu_type=IOMMU or
>> iommu_type=no-iommu.
>>
>> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed
>> with "No such device error". Thats why I am confused whether my
>> requirement is legitimate or not. What could be the best solution?
>
> This is really more of a QEMU question.  In order to use
> vfio_iommu_type1 in the guest, you need an iommu in the guest.  The
> most recent release of QEMU supports this with an emulated VT-d
> device.  Therefore if you create a VM with emulated VT-d and a device
> assigned through vfio-pci, you can expose it to userspace in the VM with
> physical iommu protection.  Without an iommu in the VM, you'd be
> limited to no-iommu support for VM userspace, the physical iommu would
> only protect the device to the extent of VM memory, no to specific
> userspace mappings within the VM.  Thanks,
>
> Alex

Re: [Qemu-devel] Query on VFIO in Virtual machine

[Nitin Saxena]

Thanks Alex.

>> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,
So are you trying to say VFIO NO-IOMMU should work inside VM. Does
that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
is a legitimate configuration? I did tried this configuration and the
application (in VM) seems to get container_fd, group_fd, device_fd
successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
from VM as well as from host. This could be specific to PCI endpoint
device which I can dig.

I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
device is legitimate configuration.

Thanks
Nitin

On Thu, Jun 22, 2017 at 10:29 PM, Alex Williamson
<alex.williamson@redhat.com> wrote:
> [cc +qemu-devel, +peterx]
>
> On Thu, 22 Jun 2017 22:18:06 +0530
> Nitin Saxena <nitin.lnx@gmail.com> wrote:
>
>> Hi,
>>
>> I have a PCI device connected as an endpoint to Intel host machine.
>> The requirement is to run dpdk like user space data path application
>> in VM using PCI PF passthrough (SRIOV disabled). This application
>> works fine on host kernel and uses VFIO to get MSIX interrupts from
>> PCI device. We are trying to run this existing application in VM using
>> PCI passthrough. This application has capability to use
>> VFIO_IOMMU_TYPE1 as wells as VFIO_NOIOMMU.
>>
>> On Intel host machine VT-d has been enabled and using virt-manager PCI
>> device PF is assigned to the VM. This makes virt-manager to implicitly
>> binds PCI device PF to vfio with vfio_iommu_type1. The VM LINUX kernel
>> was booted with intel_iommu=on as boot parameter.
>>
>> My question: Is it possible that vfio can coexist in host (by
>> virt-manager) as well as VM (by application)? If yes, does application
>> running inside VM needs to configure VFIO with iommu_type=IOMMU or
>> iommu_type=no-iommu.
>>
>> In VM I tried inserting vfio_iommu_type1.ko kernel module which failed
>> with "No such device error". Thats why I am confused whether my
>> requirement is legitimate or not. What could be the best solution?
>
> This is really more of a QEMU question.  In order to use
> vfio_iommu_type1 in the guest, you need an iommu in the guest.  The
> most recent release of QEMU supports this with an emulated VT-d
> device.  Therefore if you create a VM with emulated VT-d and a device
> assigned through vfio-pci, you can expose it to userspace in the VM with
> physical iommu protection.  Without an iommu in the VM, you'd be
> limited to no-iommu support for VM userspace, the physical iommu would
> only protect the device to the extent of VM memory, no to specific
> userspace mappings within the VM.  Thanks,
>
> Alex

Re: Query on VFIO in Virtual machine

[Alex Williamson]

On Thu, 22 Jun 2017 22:42:19 +0530
Nitin Saxena <nitin.lnx@gmail.com> wrote:

> Thanks Alex.
> 
> >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,  
> So are you trying to say VFIO NO-IOMMU should work inside VM. Does
> that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
> is a legitimate configuration? I did tried this configuration and the
> application (in VM) seems to get container_fd, group_fd, device_fd
> successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
> from VM as well as from host. This could be specific to PCI endpoint
> device which I can dig.
> 
> I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
> device is legitimate configuration.

Using no-iommu in the guest should work in that configuration, however
there's no isolation from the user to the rest of VM memory, so the VM
kernel will be tainted.  Host memory does have iommu isolation.  Device
reset from VM userspace sounds like another bug to investigate.  Thanks,

Alex

Re: [Qemu-devel] Query on VFIO in Virtual machine

[Alex Williamson]

On Thu, 22 Jun 2017 22:42:19 +0530
Nitin Saxena <nitin.lnx@gmail.com> wrote:

> Thanks Alex.
> 
> >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,  
> So are you trying to say VFIO NO-IOMMU should work inside VM. Does
> that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
> is a legitimate configuration? I did tried this configuration and the
> application (in VM) seems to get container_fd, group_fd, device_fd
> successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
> from VM as well as from host. This could be specific to PCI endpoint
> device which I can dig.
> 
> I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
> device is legitimate configuration.

Using no-iommu in the guest should work in that configuration, however
there's no isolation from the user to the rest of VM memory, so the VM
kernel will be tainted.  Host memory does have iommu isolation.  Device
reset from VM userspace sounds like another bug to investigate.  Thanks,

Alex

Re: Query on VFIO in Virtual machine

[Peter Xu]

On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote:
> On Thu, 22 Jun 2017 22:42:19 +0530
> Nitin Saxena <nitin.lnx@gmail.com> wrote:
> 
> > Thanks Alex.
> > 
> > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,  
> > So are you trying to say VFIO NO-IOMMU should work inside VM. Does
> > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
> > is a legitimate configuration? I did tried this configuration and the
> > application (in VM) seems to get container_fd, group_fd, device_fd
> > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
> > from VM as well as from host. This could be specific to PCI endpoint
> > device which I can dig.
> > 
> > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
> > device is legitimate configuration.
> 
> Using no-iommu in the guest should work in that configuration, however
> there's no isolation from the user to the rest of VM memory, so the VM
> kernel will be tainted.  Host memory does have iommu isolation.  Device
> reset from VM userspace sounds like another bug to investigate.  Thanks,
> 
> Alex

Besides what Alex has mentioned, there is a wiki page for the usage.
The command line will be slightly different on QEMU side comparing to
without vIOMMU:

  http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices

One more thing to mention is that, when vfio-pci devices in the guest
are used with emulated VT-d, huge performance degradation will be
expected for dynamic allocations at least for now. While for mostly
static allocations (like DPDK) the performance should be merely the
same as no-IOMMU mode. It's just a hint on performance, and I believe
for your own case it should mostly depend on how the application is
managing DMA map/unmaps.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] Query on VFIO in Virtual machine

[Peter Xu]

On Thu, Jun 22, 2017 at 11:27:09AM -0600, Alex Williamson wrote:
> On Thu, 22 Jun 2017 22:42:19 +0530
> Nitin Saxena <nitin.lnx@gmail.com> wrote:
> 
> > Thanks Alex.
> > 
> > >> Without an iommu in the VM, you'd be limited to no-iommu support for VM userspace,  
> > So are you trying to say VFIO NO-IOMMU should work inside VM. Does
> > that mean VFIO NO-IOMMU in VM and VFIO IOMMU in host for same device
> > is a legitimate configuration? I did tried this configuration and the
> > application (in VM) seems to get container_fd, group_fd, device_fd
> > successfully but after VFIO_DEVICE_RESET ioctl the PCI link breaks
> > from VM as well as from host. This could be specific to PCI endpoint
> > device which I can dig.
> > 
> > I will be happy if VFIO NO-IOMMU in VM and IOMMU in host for same
> > device is legitimate configuration.
> 
> Using no-iommu in the guest should work in that configuration, however
> there's no isolation from the user to the rest of VM memory, so the VM
> kernel will be tainted.  Host memory does have iommu isolation.  Device
> reset from VM userspace sounds like another bug to investigate.  Thanks,
> 
> Alex

Besides what Alex has mentioned, there is a wiki page for the usage.
The command line will be slightly different on QEMU side comparing to
without vIOMMU:

  http://wiki.qemu.org/Features/VT-d#With_Assigned_Devices

One more thing to mention is that, when vfio-pci devices in the guest
are used with emulated VT-d, huge performance degradation will be
expected for dynamic allocations at least for now. While for mostly
static allocations (like DPDK) the performance should be merely the
same as no-IOMMU mode. It's just a hint on performance, and I believe
for your own case it should mostly depend on how the application is
managing DMA map/unmaps.

Thanks,

-- 
Peter Xu