Re: [driver question] IOMMU concept in Xen

[Maciek <gotoc64@gmail.com>]

Yes you are right,

I was too focused on my issue that I didn't specify what I am trying to 
achieve :)
My company current driver can work as normal PF pci device, and also we 
support SR-IOV.
In driver initialization we can call sriov_configure to turn on SRIOV 
and enable VFs.
To be able to run on VM using VF we need IOMMU enabled (currently we 
recommend our
customers to add parameter to kernel intel_iommu=on).
But because we have many cases when people were trying to run driver 
without iommu,
caused fail, that VFs were visible but driver not working we decided to 
handle inside sriov_configure
call to handle this case by using simple check by iommu_present.
That solve our issue till we moved to Xen.
As I pointed out on Xen we didn't see iommu on Dom0, and that cause our 
check to don't enable sriov.
After we removed this check driver works perfectly fine, but we are in 
the same place.
So my intention is to leave this check in case of other hypervisor, and 
also meet Xen requirements about iommu handling.

Is that put more sense on my investigation?

Thanks
Maciej


On 10/10/16 00:44, Jan Beulich wrote:
>>>> On 10.10.16 at 02:19, <gotoc64@gmail.com> wrote:
>> During development of linux kernel PCI driver with SR-IOV I meet some
>> difficulty and I wanted to make sure that I understand correctly Xen
>> concepts.
>>
>> Things that confuse me is build in function: iommu_present. Is usually
>> used in driver that are using iommu to work.
>> My problem is that this function work perfectly fine on bare-metal
>> linux, but when I move to Xen (or XenServer) then this function return
>> always false.
>>
>> I know that Xen concept move IOMMU from linux kernel (which is on xen
>> just VM) to the hyperviosr. In XenServer is done using command:
>> xen-cmdline --set-xen iommu=1. After this command (and restart) iommu
>> should work.
>> I can also add this parameter to kernel cmd line (intel_iommu=on). Using
>> that in kernel messages I can see message: Intel-IOMMU: enabled. But no
>> matter which way I follow, iommu_present will always return me false.
>>
>> I just commented out this check block and run whole driver, there wasn't
>> any issue and driver works perfectly.
>>
>> After that I if there are other drivers in kernel that have the same
>> issue, but I didn't find anything that can solve this issue. One driver
>> check has conditional compilation for iommu_present block: #if
>> IS_ENABLED(CONFIG_IOMMU_API), but in XenServer (I checked XS 6.5) this
>> option is set to yes in kernel configuration. (This driver source is:
>> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c)
>>
>> Using grep I found other drivers that can run into same trouble (doesn't
>> start or throw warning log) inside Xen as dom0:
>> ## drivers that are uisng iommu_present
>> drivers/gpu/drm/tegra/drm.c
>> drivers/infiniband/hw/usnic/usnic_uiom.c
>> drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
>> drivers/vfio/vfio.c
>> drivers/gpu/drm/msm/msm_gem.c
>> drivers/gpu/drm/msm/msm_drv.c
>> drivers/gpu/drm/mediatek/mtk_drm_drv.c
>> drivers/crypto/qat/qat_common/adf_sriov.c
>>
>> So there are two options: there drivers are not designed for Xen or I
>> totally mess up something.
>>
>> In my driver I started thinking to use #if IS_ENABLED(CONFIG_XEN_DOM0)
>> to handle different Hypervisor, but maybe is not right solution.
> What you don't clarify in all of the above explanations and analysis
> is - why do you need to know in the driver in the first place. To a
> guest the presence of an IOMMU should be transparent. There may
> be certain cases where Dom0 drivers need to know, but such then
> need to be made interact with Xen (as the entity driving the IOMMU),
> i.e. can't normally work on Xen without some minimal adjustments.
> And this should really be the exception, not the rule.
>
> Jan
>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel