[driver question] IOMMU concept in Xen

[driver question] IOMMU concept in Xen

[Maciek]

Hi all,

During development of linux kernel PCI driver with SR-IOV I meet some 
difficulty and I wanted to make sure that I understand correctly Xen 
concepts.

Things that confuse me is build in function: iommu_present. Is usually 
used in driver that are using iommu to work.
My problem is that this function work perfectly fine on bare-metal 
linux, but when I move to Xen (or XenServer) then this function return 
always false.

I know that Xen concept move IOMMU from linux kernel (which is on xen 
just VM) to the hyperviosr. In XenServer is done using command: 
xen-cmdline --set-xen iommu=1. After this command (and restart) iommu 
should work.
I can also add this parameter to kernel cmd line (intel_iommu=on). Using 
that in kernel messages I can see message: Intel-IOMMU: enabled. But no 
matter which way I follow, iommu_present will always return me false.

I just commented out this check block and run whole driver, there wasn't 
any issue and driver works perfectly.

After that I if there are other drivers in kernel that have the same 
issue, but I didn't find anything that can solve this issue. One driver 
check has conditional compilation for iommu_present block: #if 
IS_ENABLED(CONFIG_IOMMU_API), but in XenServer (I checked XS 6.5) this 
option is set to yes in kernel configuration. (This driver source is: 
drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c)

Using grep I found other drivers that can run into same trouble (doesn't 
start or throw warning log) inside Xen as dom0:
## drivers that are uisng iommu_present
drivers/gpu/drm/tegra/drm.c
drivers/infiniband/hw/usnic/usnic_uiom.c
drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
drivers/vfio/vfio.c
drivers/gpu/drm/msm/msm_gem.c
drivers/gpu/drm/msm/msm_drv.c
drivers/gpu/drm/mediatek/mtk_drm_drv.c
drivers/crypto/qat/qat_common/adf_sriov.c

So there are two options: there drivers are not designed for Xen or I 
totally mess up something.

In my driver I started thinking to use #if IS_ENABLED(CONFIG_XEN_DOM0) 
to handle different Hypervisor, but maybe is not right solution.


I will appreciate any help and clarification of IOMMU concept

Thanks

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [driver question] IOMMU concept in Xen

[Jan Beulich]

>>> On 10.10.16 at 02:19, <gotoc64@gmail.com> wrote:
> During development of linux kernel PCI driver with SR-IOV I meet some 
> difficulty and I wanted to make sure that I understand correctly Xen 
> concepts.
> 
> Things that confuse me is build in function: iommu_present. Is usually 
> used in driver that are using iommu to work.
> My problem is that this function work perfectly fine on bare-metal 
> linux, but when I move to Xen (or XenServer) then this function return 
> always false.
> 
> I know that Xen concept move IOMMU from linux kernel (which is on xen 
> just VM) to the hyperviosr. In XenServer is done using command: 
> xen-cmdline --set-xen iommu=1. After this command (and restart) iommu 
> should work.
> I can also add this parameter to kernel cmd line (intel_iommu=on). Using 
> that in kernel messages I can see message: Intel-IOMMU: enabled. But no 
> matter which way I follow, iommu_present will always return me false.
> 
> I just commented out this check block and run whole driver, there wasn't 
> any issue and driver works perfectly.
> 
> After that I if there are other drivers in kernel that have the same 
> issue, but I didn't find anything that can solve this issue. One driver 
> check has conditional compilation for iommu_present block: #if 
> IS_ENABLED(CONFIG_IOMMU_API), but in XenServer (I checked XS 6.5) this 
> option is set to yes in kernel configuration. (This driver source is: 
> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c)
> 
> Using grep I found other drivers that can run into same trouble (doesn't 
> start or throw warning log) inside Xen as dom0:
> ## drivers that are uisng iommu_present
> drivers/gpu/drm/tegra/drm.c
> drivers/infiniband/hw/usnic/usnic_uiom.c
> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
> drivers/vfio/vfio.c
> drivers/gpu/drm/msm/msm_gem.c
> drivers/gpu/drm/msm/msm_drv.c
> drivers/gpu/drm/mediatek/mtk_drm_drv.c
> drivers/crypto/qat/qat_common/adf_sriov.c
> 
> So there are two options: there drivers are not designed for Xen or I 
> totally mess up something.
> 
> In my driver I started thinking to use #if IS_ENABLED(CONFIG_XEN_DOM0) 
> to handle different Hypervisor, but maybe is not right solution.

What you don't clarify in all of the above explanations and analysis
is - why do you need to know in the driver in the first place. To a
guest the presence of an IOMMU should be transparent. There may
be certain cases where Dom0 drivers need to know, but such then
need to be made interact with Xen (as the entity driving the IOMMU),
i.e. can't normally work on Xen without some minimal adjustments.
And this should really be the exception, not the rule.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [driver question] IOMMU concept in Xen

[Maciek]

Yes you are right,

I was too focused on my issue that I didn't specify what I am trying to 
achieve :)
My company current driver can work as normal PF pci device, and also we 
support SR-IOV.
In driver initialization we can call sriov_configure to turn on SRIOV 
and enable VFs.
To be able to run on VM using VF we need IOMMU enabled (currently we 
recommend our
customers to add parameter to kernel intel_iommu=on).
But because we have many cases when people were trying to run driver 
without iommu,
caused fail, that VFs were visible but driver not working we decided to 
handle inside sriov_configure
call to handle this case by using simple check by iommu_present.
That solve our issue till we moved to Xen.
As I pointed out on Xen we didn't see iommu on Dom0, and that cause our 
check to don't enable sriov.
After we removed this check driver works perfectly fine, but we are in 
the same place.
So my intention is to leave this check in case of other hypervisor, and 
also meet Xen requirements about iommu handling.

Is that put more sense on my investigation?

Thanks
Maciej


On 10/10/16 00:44, Jan Beulich wrote:
>>>> On 10.10.16 at 02:19, <gotoc64@gmail.com> wrote:
>> During development of linux kernel PCI driver with SR-IOV I meet some
>> difficulty and I wanted to make sure that I understand correctly Xen
>> concepts.
>>
>> Things that confuse me is build in function: iommu_present. Is usually
>> used in driver that are using iommu to work.
>> My problem is that this function work perfectly fine on bare-metal
>> linux, but when I move to Xen (or XenServer) then this function return
>> always false.
>>
>> I know that Xen concept move IOMMU from linux kernel (which is on xen
>> just VM) to the hyperviosr. In XenServer is done using command:
>> xen-cmdline --set-xen iommu=1. After this command (and restart) iommu
>> should work.
>> I can also add this parameter to kernel cmd line (intel_iommu=on). Using
>> that in kernel messages I can see message: Intel-IOMMU: enabled. But no
>> matter which way I follow, iommu_present will always return me false.
>>
>> I just commented out this check block and run whole driver, there wasn't
>> any issue and driver works perfectly.
>>
>> After that I if there are other drivers in kernel that have the same
>> issue, but I didn't find anything that can solve this issue. One driver
>> check has conditional compilation for iommu_present block: #if
>> IS_ENABLED(CONFIG_IOMMU_API), but in XenServer (I checked XS 6.5) this
>> option is set to yes in kernel configuration. (This driver source is:
>> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c)
>>
>> Using grep I found other drivers that can run into same trouble (doesn't
>> start or throw warning log) inside Xen as dom0:
>> ## drivers that are uisng iommu_present
>> drivers/gpu/drm/tegra/drm.c
>> drivers/infiniband/hw/usnic/usnic_uiom.c
>> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
>> drivers/vfio/vfio.c
>> drivers/gpu/drm/msm/msm_gem.c
>> drivers/gpu/drm/msm/msm_drv.c
>> drivers/gpu/drm/mediatek/mtk_drm_drv.c
>> drivers/crypto/qat/qat_common/adf_sriov.c
>>
>> So there are two options: there drivers are not designed for Xen or I
>> totally mess up something.
>>
>> In my driver I started thinking to use #if IS_ENABLED(CONFIG_XEN_DOM0)
>> to handle different Hypervisor, but maybe is not right solution.
> What you don't clarify in all of the above explanations and analysis
> is - why do you need to know in the driver in the first place. To a
> guest the presence of an IOMMU should be transparent. There may
> be certain cases where Dom0 drivers need to know, but such then
> need to be made interact with Xen (as the entity driving the IOMMU),
> i.e. can't normally work on Xen without some minimal adjustments.
> And this should really be the exception, not the rule.
>
> Jan
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [driver question] IOMMU concept in Xen

[Jan Beulich]

>>> Maciek <gotoc64@gmail.com> 10/10/16 10:17 PM >>>

First of all - please don't top post.

>In driver initialization we can call sriov_configure to turn on SRIOV 
>and enable VFs.
>To be able to run on VM using VF we need IOMMU enabled (currently we 
>recommend our
>customers to add parameter to kernel intel_iommu=on).
>But because we have many cases when people were trying to run driver 
>without iommu,
>caused fail, that VFs were visible but driver not working we decided to 
>handle inside sriov_configure
>call to handle this case by using simple check by iommu_present.
>That solve our issue till we moved to Xen.
>As I pointed out on Xen we didn't see iommu on Dom0, and that cause our 
>check to don't enable sriov.
>After we removed this check driver works perfectly fine, but we are in 
>the same place.
>So my intention is to leave this check in case of other hypervisor, and 
>also meet Xen requirements about iommu handling.

Hmm, as indicated the Dom0 kernel doesn't and isn't supposed to know
of IOMMUs and their enabled state, so there's a conceptual problem here.
I'm afraid your users will need to learn not to disable the IOMMU on Xen (it's
enabled by default). The tool stack (which normally runs in Dom0 user
space) can know, and iirc it also prevents pass-through of devices when
the IOMMU is not enabled. Wouldn't the resulting error message not be
sufficient for your purposes?

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel