* Disable IOMMU in Dom0
@ 2021-08-31 19:54 Roman Skakun
2021-08-31 21:50 ` Stefano Stabellini
0 siblings, 1 reply; 6+ messages in thread
From: Roman Skakun @ 2021-08-31 19:54 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, xen-devel, Bertrand Marquis, Andrii Anisov,
Roman Skakun, Roman Skakun
Hi, Stefano!
I have seen your negotiation of disabling xen-swiotlb for devices which are controlled by IOMMU in Dom0:
https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/
As I was thinking to create a common implementation because I have the issue
when device controlled by IOMMU using xen-swiotlb fops and bounce buffer as the result.
https://lore.kernel.org/xen-devel/060b5741-922c-115c-7e8c-97d8aa5f46f4@xen.org/T/
Do you have any future plans to finish implementation for upstream?
Cheers!
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disable IOMMU in Dom0
2021-08-31 19:54 Disable IOMMU in Dom0 Roman Skakun
@ 2021-08-31 21:50 ` Stefano Stabellini
2021-09-01 9:59 ` Roman Skakun
0 siblings, 1 reply; 6+ messages in thread
From: Stefano Stabellini @ 2021-08-31 21:50 UTC (permalink / raw)
To: Roman Skakun
Cc: Stefano Stabellini, Julien Grall, xen-devel, Bertrand Marquis,
Andrii Anisov, Roman Skakun
On Tue, 31 Aug 2021, Roman Skakun wrote:
> Hi, Stefano!
>
> I have seen your negotiation of disabling xen-swiotlb for devices which are controlled by IOMMU in Dom0:
> https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/
>
> As I was thinking to create a common implementation because I have the issue
> when device controlled by IOMMU using xen-swiotlb fops and bounce buffer as the result.
> https://lore.kernel.org/xen-devel/060b5741-922c-115c-7e8c-97d8aa5f46f4@xen.org/T/
>
> Do you have any future plans to finish implementation for upstream?
Hi Roman,
The feature is already upstream in Linux as f5079a9a2, and the new
feature flags are XENFEAT_direct_mapped and XENFEAT_not_direct_mapped.
If you have a setup where Dom0 is not 1:1 mapped (which is not currently
possible with upstream Xen but it is possible with cache coloring) and
uses the IOMMU to make device DMA work like regular DomUs, then passing
XENFEAT_not_direct_mapped to Linux would make it work. Without
XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
code that relies on Linux being 1:1 mapped to work properly.
Is that the same problem that you have, or is dom0 1:1 mapped in your
case? If dom0 is 1:1 mapped then swiotlb-xen should work regardless of
whether the IOMMU is enabled or disabled.
I hope this helps.
Cheers,
Stefano
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disable IOMMU in Dom0
2021-08-31 21:50 ` Stefano Stabellini
@ 2021-09-01 9:59 ` Roman Skakun
2021-09-01 10:22 ` Julien Grall
0 siblings, 1 reply; 6+ messages in thread
From: Roman Skakun @ 2021-09-01 9:59 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Julien Grall, xen-devel, Bertrand Marquis, Andrii Anisov,
Roman Skakun, Roman Skakun
[-- Attachment #1: Type: text/plain, Size: 3321 bytes --]
Hi, Stefano!
Thanks for responding!
> If you have a setup where Dom0 is not 1:1 mapped (which is not currently
> possible with upstream Xen but it is possible with cache coloring) and
> uses the IOMMU to make device DMA work like regular DomUs, then passing
> XENFEAT_not_direct_mapped to Linux would make it work. Without
> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
> code that relies on Linux being 1:1 mapped to work properly.
I'm using 1:1 Dom0.
According to your patch series, xen-swiotlb fops will be applied for all devices
because XENFEAT_direct_mapped is active, as shown here:
https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c#L56
I agreed, that xen-swiotlb should work correctly, but in my case, I retrieved MFN here:
https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c#L366
which is greater than 32bit and xen-swiotlb tries to use bounce buffer as expected.
It can lead to decrease a performance because I have a long buffer ~4MB.
I thought, that we can disable swiotlb fops for devices which are controlled by IOMMU.
Cheers,
________________________________
From: Stefano Stabellini <sstabellini@kernel.org>
Sent: Wednesday, September 1, 2021 12:50 AM
To: Roman Skakun <Roman_Skakun@epam.com>
Cc: Stefano Stabellini <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Roman Skakun <rm.skakun@gmail.com>
Subject: Re: Disable IOMMU in Dom0
On Tue, 31 Aug 2021, Roman Skakun wrote:
> Hi, Stefano!
>
> I have seen your negotiation of disabling xen-swiotlb for devices which are controlled by IOMMU in Dom0:
> https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/__;!!GF_29dbcQIUBPA!hl4_2pWlI3NQmAkwh4uvJFu7NhcOay9O9bIJRr7eOOMKD_I2z8MPbwnSXpNmqU6F$ [patchwork[.]kernel[.]org]
>
> As I was thinking to create a common implementation because I have the issue
> when device controlled by IOMMU using xen-swiotlb fops and bounce buffer as the result.
> https://urldefense.com/v3/__https://lore.kernel.org/xen-devel/060b5741-922c-115c-7e8c-97d8aa5f46f4@xen.org/T/__;!!GF_29dbcQIUBPA!hl4_2pWlI3NQmAkwh4uvJFu7NhcOay9O9bIJRr7eOOMKD_I2z8MPbwnSXp1TdIa7$ [lore[.]kernel[.]org]
>
> Do you have any future plans to finish implementation for upstream?
Hi Roman,
The feature is already upstream in Linux as f5079a9a2, and the new
feature flags are XENFEAT_direct_mapped and XENFEAT_not_direct_mapped.
If you have a setup where Dom0 is not 1:1 mapped (which is not currently
possible with upstream Xen but it is possible with cache coloring) and
uses the IOMMU to make device DMA work like regular DomUs, then passing
XENFEAT_not_direct_mapped to Linux would make it work. Without
XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
code that relies on Linux being 1:1 mapped to work properly.
Is that the same problem that you have, or is dom0 1:1 mapped in your
case? If dom0 is 1:1 mapped then swiotlb-xen should work regardless of
whether the IOMMU is enabled or disabled.
I hope this helps.
Cheers,
Stefano
[-- Attachment #2: Type: text/html, Size: 12524 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disable IOMMU in Dom0
2021-09-01 9:59 ` Roman Skakun
@ 2021-09-01 10:22 ` Julien Grall
2021-09-09 14:25 ` Roman Skakun
0 siblings, 1 reply; 6+ messages in thread
From: Julien Grall @ 2021-09-01 10:22 UTC (permalink / raw)
To: Roman Skakun, Stefano Stabellini
Cc: xen-devel, Bertrand Marquis, Andrii Anisov, Roman Skakun,
Oleksandr Tyshchenko
Hi Roman
On 01/09/2021 10:59, Roman Skakun wrote:
>> If you have a setup where Dom0 is not 1:1 mapped (which is not currently
>> possible with upstream Xen but it is possible with cache coloring) and
>> uses the IOMMU to make device DMA work like regular DomUs, then passing
>> XENFEAT_not_direct_mapped to Linux would make it work. Without
>> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
>> code that relies on Linux being 1:1 mapped to work properly.
>
> I'm using 1:1 Dom0.
> According to your patch series, xen-swiotlb fops will be applied for all
> devices
> because XENFEAT_direct_mapped is active, as shown here:
> https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c#L56
> <https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c#L56>
>
> I agreed, that xen-swiotlb should work correctly, but in my case, I
> retrieved MFN here:
> https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c#L366
> <https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c#L366>
> which is greater than 32bit and xen-swiotlb tries to use bounce buffer
> as expected.
> It can lead to decrease a performance because I have a long buffer ~4MB.
>
> I thought, that we can disable swiotlb fops for devices which are
> controlled by IOMMU.
Yes you can disable swiotlb for devices which are controlled by the
IOMMU. But this will not make your problem disappear, it simply hides
until it bites you in a more subttle way.
From what you wrote, you have a 32-bit DMA capable. So you always need
to have an address below 4GB. For foreign mapping, there is no guarantee
the Guest Physical Address will actually be below 4GB.
Today, the ballooning code only ask Linux to steal *a* RAM page for
mapping the foreign page. This may or may not be below 4GB depending on
how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB).
But that's the current behavior. One of your work colleague is looking
at avoid to steal RAM page to avoid exhausting the memory. So foreign
mapping may end up to be a lot higher in memory.
IOW, you will need to be able to bounce the DMA buffer for your device.
If you want to avoid bouncing, the proper way would be to rework the
ballonning code so pages are allocated below 4GB.
Cheers,
--
Julien Grall
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disable IOMMU in Dom0
2021-09-01 10:22 ` Julien Grall
@ 2021-09-09 14:25 ` Roman Skakun
2021-09-09 19:53 ` Stefano Stabellini
0 siblings, 1 reply; 6+ messages in thread
From: Roman Skakun @ 2021-09-09 14:25 UTC (permalink / raw)
To: Julien Grall
Cc: xen-devel, Bertrand Marquis, Andrii Anisov, Roman Skakun,
Oleksandr Tyshchenko, Stefano Stabellini
[-- Attachment #1: Type: text/plain, Size: 4041 bytes --]
Hi Julien,
Thanks for the clarification!
I aim towards to prepare implementation for upstream to disable SWIOTLB for IOMMU-protected devices in Dom0.
To provide this functionality need to add additional binding for each protected device in device-tree.
After this step, I will also prepare the patch to make ensure that ballooning code prepares all allocations below 4GB.
We are going to prepare this functionality only for device-tree based system configurations.
We don't have resources to support ACPI configuration.
Would you be ok with upstreaming only device-tree configuration?
Cheers,
Roman
________________________________
From: Julien Grall <julien@xen.org>
Sent: Wednesday, September 1, 2021 1:22 PM
To: Roman Skakun <Roman_Skakun@epam.com>; Stefano Stabellini <sstabellini@kernel.org>
Cc: xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Roman Skakun <rm.skakun@gmail.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>
Subject: Re: Disable IOMMU in Dom0
Hi Roman
On 01/09/2021 10:59, Roman Skakun wrote:
>> If you have a setup where Dom0 is not 1:1 mapped (which is not currently
>> possible with upstream Xen but it is possible with cache coloring) and
>> uses the IOMMU to make device DMA work like regular DomUs, then passing
>> XENFEAT_not_direct_mapped to Linux would make it work. Without
>> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
>> code that relies on Linux being 1:1 mapped to work properly.
>
> I'm using 1:1 Dom0.
> According to your patch series, xen-swiotlb fops will be applied for all
> devices
> because XENFEAT_direct_mapped is active, as shown here:
> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com]
> <https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com]>
>
> I agreed, that xen-swiotlb should work correctly, but in my case, I
> retrieved MFN here:
> https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]
> <https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]>
> which is greater than 32bit and xen-swiotlb tries to use bounce buffer
> as expected.
> It can lead to decrease a performance because I have a long buffer ~4MB.
>
> I thought, that we can disable swiotlb fops for devices which are
> controlled by IOMMU.
Yes you can disable swiotlb for devices which are controlled by the
IOMMU. But this will not make your problem disappear, it simply hides
until it bites you in a more subttle way.
From what you wrote, you have a 32-bit DMA capable. So you always need
to have an address below 4GB. For foreign mapping, there is no guarantee
the Guest Physical Address will actually be below 4GB.
Today, the ballooning code only ask Linux to steal *a* RAM page for
mapping the foreign page. This may or may not be below 4GB depending on
how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB).
But that's the current behavior. One of your work colleague is looking
at avoid to steal RAM page to avoid exhausting the memory. So foreign
mapping may end up to be a lot higher in memory.
IOW, you will need to be able to bounce the DMA buffer for your device.
If you want to avoid bouncing, the proper way would be to rework the
ballonning code so pages are allocated below 4GB.
Cheers,
--
Julien Grall
[-- Attachment #2: Type: text/html, Size: 6017 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Disable IOMMU in Dom0
2021-09-09 14:25 ` Roman Skakun
@ 2021-09-09 19:53 ` Stefano Stabellini
0 siblings, 0 replies; 6+ messages in thread
From: Stefano Stabellini @ 2021-09-09 19:53 UTC (permalink / raw)
To: Roman Skakun
Cc: Julien Grall, xen-devel, Bertrand Marquis, Andrii Anisov,
Roman Skakun, Oleksandr Tyshchenko, Stefano Stabellini
[-- Attachment #1: Type: text/plain, Size: 5369 bytes --]
I am fine with adding this functionality only to device tree initially.
It is certainly true that if a DMA-capable device is behind an IOMMU,
then we can skip swiotlb-xen for foreign pages address transactions.
Those addresses will be translated just fine thanks to the IOMMU.
Skipping swiotlb-xen could provide a non-negligible performance
improvement, which is good.
However, you should be aware that your proposal should not be needed for
correctness or to make devices with a 4GB DMA mask work.
Thanks to commit 04085ec1a "xen/arm: fix gnttab_need_iommu_mapping"
foreign pages are added to the Dom0 p2m when mapped to Dom0. swiotlb-xen
translates foreign pages gfn to mfn then uses the mfn to program the
device DMA. The DMA transaction will be accepted by the IOMMU thanks to
the 1:1 GFN<->MFN entry added to Dom0 at the time of mapping.
If the mfn is >4GB and the device requires an address <4GB, then the
dma_capable check at the beginning of xen_swiotlb_map_page fails and the
DMA transaction gets bounced on a swiotlb-xen buffer lower than 4GB.
Am I missing anything?
On Thu, 9 Sep 2021, Roman Skakun wrote:
> Hi Julien,
> Thanks for the clarification!
>
> I aim towards to prepare implementation for upstream to disable SWIOTLB for IOMMU-protected devices in Dom0.
> To provide this functionality need to add additional binding for each protected device in device-tree.
> After this step, I will also prepare the patch to make ensure that ballooning code prepares all allocations below 4GB.
>
> We are going to prepare this functionality only for device-tree based system configurations.
> We don't have resources to support ACPI configuration.
>
> Would you be ok with upstreaming only device-tree configuration?
>
> Cheers,
> Roman
>
> ___________________________________________________________________________________________________________________________________________
> From: Julien Grall <julien@xen.org>
> Sent: Wednesday, September 1, 2021 1:22 PM
> To: Roman Skakun <Roman_Skakun@epam.com>; Stefano Stabellini <sstabellini@kernel.org>
> Cc: xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov
> <Andrii_Anisov@epam.com>; Roman Skakun <rm.skakun@gmail.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>
> Subject: Re: Disable IOMMU in Dom0
> Hi Roman
>
> On 01/09/2021 10:59, Roman Skakun wrote:
> >> If you have a setup where Dom0 is not 1:1 mapped (which is not currently
> >> possible with upstream Xen but it is possible with cache coloring) and
> >> uses the IOMMU to make device DMA work like regular DomUs, then passing
> >> XENFEAT_not_direct_mapped to Linux would make it work. Without
> >> XENFEAT_not_direct_mapped, Linux would try to use swiotlb-xen which has
> >> code that relies on Linux being 1:1 mapped to work properly.
> >
> > I'm using 1:1 Dom0.
> > According to your patch series, xen-swiotlb fops will be applied for all
> > devices
> > because XENFEAT_direct_mapped is active, as shown here:
> >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4i
> bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com]
> ><https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/arch/arm64/mm/dma-mapping.c*L56__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4
> ibLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_Iv_mvn3O$ [elixir[.]bootlin[.]com]>
> >
> > I agreed, that xen-swiotlb should work correctly, but in my case, I
> > retrieved MFN here:
> >https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4ib
> LDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]
> ><https://urldefense.com/v3/__https://elixir.bootlin.com/linux/v5.14/source/drivers/xen/swiotlb-xen.c*L366__;Iw!!GF_29dbcQIUBPA!i7I0DxCbP4i
> bLDwzRkeFkgRQbKh-fVD9McLqabG1TzZs9smOVBeowPS_IgZgXPjC$ [elixir[.]bootlin[.]com]>
> > which is greater than 32bit and xen-swiotlb tries to use bounce buffer
> > as expected.
> > It can lead to decrease a performance because I have a long buffer ~4MB.
> >
> > I thought, that we can disable swiotlb fops for devices which are
> > controlled by IOMMU.
>
> Yes you can disable swiotlb for devices which are controlled by the
> IOMMU. But this will not make your problem disappear, it simply hides
> until it bites you in a more subttle way.
>
> From what you wrote, you have a 32-bit DMA capable. So you always need
> to have an address below 4GB. For foreign mapping, there is no guarantee
> the Guest Physical Address will actually be below 4GB.
>
> Today, the ballooning code only ask Linux to steal *a* RAM page for
> mapping the foreign page. This may or may not be below 4GB depending on
> how you assigned the RAM to dom0 (IIRC you had some RAM above 4GB).
>
> But that's the current behavior. One of your work colleague is looking
> at avoid to steal RAM page to avoid exhausting the memory. So foreign
> mapping may end up to be a lot higher in memory.
>
> IOW, you will need to be able to bounce the DMA buffer for your device.
> If you want to avoid bouncing, the proper way would be to rework the
> ballonning code so pages are allocated below 4GB.
>
> Cheers,
>
> --
> Julien Grall
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-09-09 19:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-31 19:54 Disable IOMMU in Dom0 Roman Skakun
2021-08-31 21:50 ` Stefano Stabellini
2021-09-01 9:59 ` Roman Skakun
2021-09-01 10:22 ` Julien Grall
2021-09-09 14:25 ` Roman Skakun
2021-09-09 19:53 ` Stefano Stabellini
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.