Re: Disable swiotlb for Dom0

[Roman Skakun <Roman_Skakun@epam.com>]

[-- Attachment #1: Type: text/plain, Size: 4439 bytes --]


>> But I think, we cannot use64af97000 address in the swiotlb_bounce()
>> directly.
>>
>I am not sure to understand what you mean. Can you clarify?

I thought, that the address 64af97000 can be used directly here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572
and I was confused about it.
After some diggings I realized that this code is not reachable because
DMA_ATTR_SKIP_CPU_SYNC is active.

>> Swiotlb did not fit requested slots because the maximum slot size equals
>> IO_TLB_SEGSIZE=128 by default.
>
>> Ok. So it sounds like your problem is the have a DMA buffer that is too
>> large for the default swiotlb. Did you try to bump the value from the
>> command line?

Yes, you are right. I checked the implementation more detailed and figured out that
swiotlb tries to find a contiguous segment that is a big enough, as shown here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528,

But the maximum TLB segment equals 128K by default in accordance with
https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25

This means that the max size of TLB segment equals:
128 * 2048 = 256K

After this, I tried to increase the TLB segment size like this:

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index fbdc65782195..85f32865bb6c 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -22,7 +22,7 @@ enum swiotlb_force {
  * must be a power of 2.  What is the appropriate value ?
  * The complexity of {map,unmap}_single is linearly dependent on this value.
  */
-#define IO_TLB_SEGSIZE 128
+#define IO_TLB_SEGSIZE 2048

 /*
  * log of the size of each IO TLB slab.  The number of slabs is command line

and the problem is gone.

Also, I found the article https://www.xilinx.com/support/answers/72694.html,
where I believe the same issue was mentioned.

Thank you so much for your time and help!

________________________________
From: Julien Grall <julien@xen.org>
Sent: Friday, August 13, 2021 1:51 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0



On 13/08/2021 10:38, Roman Skakun wrote:
> Hi Julien,

Hi Roman,

>> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>>
>> swiotlb-xen on Arm will convert it to the MFN because it is not aware
>> whether the device is behind an IOMMU.
>
> If I'm understand right, it seems like that swiotlb-xen is not ready to
> work properly in case
> when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
> Maybe you know some ideas to overcome this condition?

swiotlb-xen work as intended. You have a DMA buffer at an address that
your device cannot deal with. So it will try to bounce it.

>
>>  As the address is too high to be handled by the device, swiotlb will try
>>  to bounce it. I think it is correct to bounce the page but I am not sure
>>  why it can't. What the size of the DMA transaction?
>
> The DMA map size is 3686400 bytes.

So that's a 3MB buffer. I am slightly confused because in an earlier
message you wrote that the memory is coming from the guest. How did you
map it?

>
> I've added several logs to swiotlb map_single() and see:
> [  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr:
> 64af97000, needed: 708,
> avail: 7fc0, stride: 2, index: 4160

I am not sure how to read the logs... Are the number in hexadecimal or
decimal? It might be useful if you post the diff of your changes.

[...]

> Swiotlb did not fit requested slots because the maximum slot size equals
> IO_TLB_SEGSIZE=128 by default.

Ok. So it sounds like your problem is the have a DMA buffer that is too
large for the default swiotlb. Did you try to bump the value from the
command line?

> But I think, we cannot use64af97000 address in the swiotlb_bounce()
> directly.

I am not sure to understand what you mean. Can you clarify?

Cheers,

[...]

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 13972 bytes --]