IOMMU software bounce buffers and their overhead

IOMMU software bounce buffers and their overhead

[Chathura Rajapaksha]

Hi All,

I have a few questions about IOMMU software bounce buffering.
Description of the kernel config IOMMU_DEFAULT_DMA_STRICT mentions that,
"Untrusted devices always use this mode, with an additional layer of
bounce-buffering such that they cannot gain access to any unrelated data
within a mapped page.".

What I am trying to do:
What I understood from the above description is that the STRICT mode
provides an additional layer of security for the hardware IOMMU.
I want to measure the overhead of this additional layer of protection
(software bounce buffers).

My questions are:
1. Is my understanding correct regarding the software bounce buffers
being used alongside hardware IOMMU, or do software bounce buffers replace
the hardware IOMMU?

2. Are untrusted PCI devices use SW bounce buffers by default in the upstream
kernel?

3. Are there already available numbers for the performance/memory overhead
of software bounce buffers?

3. How can I enable/disable software bounce buffers without
enabling/disabling strict IOTLB invalidations?
Details:
Since I want to measure the overhead of SW bounce buffers, I want to
know if there is a way to enable/disable only the bounce buffers while
keeping all other configurations the same. Any suggestions on how to
properly measure the overhead is welcome.

I tried to find answers to these questions in kernel mailing lists and
documentation but I could not find the exact answers.
References I looked at so far:
https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt
https://www.kernel.org/doc/html/v6.3/admin-guide/kernel-parameters.html
https://lore.kernel.org/lkml/20190730045229.3826-9-baolu.lu@linux.intel.com/T/
https://lwn.net/Articles/786558/

I really appreciate any help on this. Thank you in advance.

Regards,
Chathura

Re: IOMMU software bounce buffers and their overhead

[Baolu Lu]

On 2023/6/4 0:49, Chathura Rajapaksha wrote:
> Hi All,
> 
> I have a few questions about IOMMU software bounce buffering.
> Description of the kernel config IOMMU_DEFAULT_DMA_STRICT mentions that,
> "Untrusted devices always use this mode, with an additional layer of
> bounce-buffering such that they cannot gain access to any unrelated data
> within a mapped page.".
> 
> What I am trying to do:
> What I understood from the above description is that the STRICT mode
> provides an additional layer of security for the hardware IOMMU.
> I want to measure the overhead of this additional layer of protection
> (software bounce buffers).
> 
> My questions are:
> 1. Is my understanding correct regarding the software bounce buffers
> being used alongside hardware IOMMU, or do software bounce buffers replace
> the hardware IOMMU?

No.

The strict mode represents that every time a DMA buffer unmaps, the
IOMMU caches (IOTLB and possibly device TLB) will be invalidated and the
cache invalidation will be completed before the unmap is returned.

The IOMMU DMA bounce buffering occurs only when the device has been
marked by the PCI subsystem as an "untrusted device". You can get more
details about "PCI untrusted device" and how IOMMU subsystem addresses
them by reading this:

https://lwn.net/Articles/786558/

> 2. Are untrusted PCI devices use SW bounce buffers by default in the upstream
> kernel?

Yes.

> 
> 3. Are there already available numbers for the performance/memory overhead
> of software bounce buffers?

It's case by case. I haven't seen a complete study on this aspect yet.

> 3. How can I enable/disable software bounce buffers without
> enabling/disabling strict IOTLB invalidations?

You can't disable it. Perhaps you can hack one for your test or
evaluation purpose.

Another option is to use ACPI/DT overriding and tell the OS if a device
is trusted according to your wishes if you are playing with an out-of-
box Linux kernel.

> Details:
> Since I want to measure the overhead of SW bounce buffers, I want to
> know if there is a way to enable/disable only the bounce buffers while
> keeping all other configurations the same. Any suggestions on how to
> properly measure the overhead is welcome.
> 
> I tried to find answers to these questions in kernel mailing lists and
> documentation but I could not find the exact answers.
> References I looked at so far:
> https://www.kernel.org/doc/Documentation/x86/x86_64/boot-options.txt
> https://www.kernel.org/doc/html/v6.3/admin-guide/kernel-parameters.html
> https://lore.kernel.org/lkml/20190730045229.3826-9-baolu.lu@linux.intel.com/T/
> https://lwn.net/Articles/786558/
> 
> I really appreciate any help on this. Thank you in advance.
> 
> Regards,
> Chathura
> 

Best regards,
baolu

Re: IOMMU software bounce buffers and their overhead

[Chathura Rajapaksha]

Thank you for the response and resources.
I was able to mark a device as external and untrusted by
editing the kernel source code which enabled SW bounce
buffers for that device.

Best regards,
Chathura