Disable swiotlb for Dom0

Disable swiotlb for Dom0

[Roman Skakun]

Hi, Stefano!

I have observed your patch here:
https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/

And I collided with the same issue, when Dom0 device trying to use
swiotlb fops for devices which are controlled by IOMMU.

Prerequisites:
https://github.com/xen-project/xen/tree/stable-4.15
https://github.com/torvalds/linux/tree/v5.10

Issue caused in xen_swiotlb_map_page():
```
 dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, 
xen_phys_to_dma(phys): 64b1d0000
```

There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0
swiotlb couldn't proceed to this address and throws the log:

```
[   99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots)
```

Temporary, I resolved this issue by disabling swiotlb for dom0 at all
because sure that all devices goes through IOMMU, but this mention can
be true only for me.

But, I think of a more reliable way is to declare a special IOMMU
property in xen dts for each device. If the device controlled by IOMMU
not need to set swiotlb fops in arch_setup_dma_ops.
What do you think about it?

Re: Disable swiotlb for Dom0

[Julien Grall]

On 10/08/2021 16:38, Roman Skakun wrote:
> Hi, Stefano!

Hi,

> I have observed your patch here:
> https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/
> 
> And I collided with the same issue, when Dom0 device trying to use
> swiotlb fops for devices which are controlled by IOMMU.

The issue Stefano reported was when the dom0 is not direct mapped. 
However...

> 
> Prerequisites:
> https://github.com/xen-project/xen/tree/stable-4.15

... if you are really using 4.15, then the domain will always be direct 
mapped. So I think this is a different one.

> https://github.com/torvalds/linux/tree/v5.10

Any reason to not use the stable branch for 5.10? I don't know whether 
your issue will be fixed there, but the stable branch usually contains a 
lot of bug fixes (including security one). So it is a good idea to use 
it over the first release of a kernel version.

> Issue caused in xen_swiotlb_map_page():
> ```
>   dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000,
> xen_phys_to_dma(phys): 64b1d0000
> ```

I can't seem to find this printk in Linux 5.10. Did you add it yourself?

> 
> There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0
> swiotlb couldn't proceed to this address and throws the log:
> 
> ```
> [   99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots)
> ```

This line suggests that the SWIOTLB tried to bounce the DMA buffer. In 
general, the use of the bounce buffer should be rare. So I would suggest 
to find out why this is used.

Looking at the code, this suggests that one of the following check is false:

         /*
          * If the address happens to be in the device's DMA window,
          * we can safely return the device addr and not worry about bounce
          * buffering it.
          */
         if (dma_capable(dev, dev_addr, size, true) &&
             !range_straddles_page_boundary(phys, size) &&
                 !xen_arch_need_swiotlb(dev, phys, dev_addr) &&
                 swiotlb_force != SWIOTLB_FORCE)
                 goto done;

> 
> Temporary, I resolved this issue by disabling swiotlb for dom0 at all
> because sure that all devices goes through IOMMU, but this mention can
> be true only for me.
> 
> But, I think of a more reliable way is to declare a special IOMMU
> property in xen dts for each device. If the device controlled by IOMMU
> not need to set swiotlb fops in arch_setup_dma_ops.
> What do you think about it?

Let me start with that I agree we should disable swiotlb when we know 
the device is protected. However, from what you describe, it sounds like 
the same issue would appear if the IOMMU was disabled.

Therefore, I think we should first find out why Linux wants to bounce 
the DMA buffer. Does your device have any DMA restriction?

Cheers,

-- 
Julien Grall

Re: Disable swiotlb for Dom0

[Roman Skakun]

[-- Attachment #1: Type: text/plain, Size: 8389 bytes --]

Hi, Julien!

Thanks for the answer!

>> I have observed your patch here:
>> https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$<https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$> [patchwork[.]kernel[.]org]
>>
>> And I collided with the same issue, when Dom0 device trying to use
>> swiotlb fops for devices which are controlled by IOMMU.
>
>The issue Stefano reported was when the dom0 is not direct mapped.
>However...

I applied these patches:
https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578
https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e
to check this more pragmatically.

Also, I added the log in xen_swiotlb_detect() and can see that swiotlb still used (other devices within dom0 used too), when dom0 is direct mapped:

[    1.870363] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb
[    1.878352] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb
[    1.886309] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb

This means, that all devices are using swiotlb-xen DMA fops.
By the way, before applying this patches, dom0 always used swiotlb-xen fops for initial domain by design.


> Any reason to not use the stable branch for 5.10? I don't know whether
> your issue will be fixed there, but the stable branch usually contains a
> lot of bug fixes (including security one). So it is a good idea to use
> it over the first release of a kernel version.

Yes, sure, current BSP release based on 5.10 kernel:
https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1
based on https://github.com/renesas-rcar/linux-bsp<https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2>
BTW, I specified the wrong kernel URL in the previous massage, sorry.

>> Issue caused in xen_swiotlb_map_page():
>> ```
>>   dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000,
>> xen_phys_to_dma(phys): 64b1d0000
>> ```
>
>I can't seem to find this printk in Linux 5.10. Did you add it yourself?

Yes, it's my own log.


> This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
> general, the use of the bounce buffer should be rare. So I would suggest
> to find out why this is used.
>
> Looking at the code, this suggests that one of the following check is false:
>
>         /*
>          * If the address happens to be in the device's DMA window,
>           * we can safely return the device addr and not worry about bounce
>           * buffering it.
>           */
>         if (dma_capable(dev, dev_addr, size, true) &&
>             !range_straddles_page_boundary(phys, size) &&
>                 !xen_arch_need_swiotlb(dev, phys, dev_addr) &&
>                 swiotlb_force != SWIOTLB_FORCE)
>                 goto done;

I checked this earlier and saw that dma_capable(dev, dev_addr, size, true)  returns false as expected because
we got dev_addr equals 64b1d0000 and according to this expression under dma_capable():

```
dma_addr_t end = dev_addr + size - 1;
return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
```
As result, DMA mask more than 32bit.

> Let me start with that I agree we should disable swiotlb when we know
> the device is protected. However, from what you describe, it sounds like
> the same issue would appear if the IOMMU was disabled.

Yes, it looks like a potential issue. This means that swiotlb should be worked correctly, when it's needed, agreed.
But this is also potential improvement, and I presented this idea to discuss and create some patches.

> Therefore, I think we should first find out why Linux wants to bounce
> the DMA buffer.

We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver wants to use only 32 bit boundary address, but that's consequence.
I think, the main reason of using bounce buffer is MFN address, not DMA phys address.

Cheers!
________________________________
From: Julien Grall <julien@xen.org>
Sent: Tuesday, August 10, 2021 7:16 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0



On 10/08/2021 16:38, Roman Skakun wrote:
> Hi, Stefano!

Hi,

> I have observed your patch here:
> https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ [patchwork[.]kernel[.]org]
>
> And I collided with the same issue, when Dom0 device trying to use
> swiotlb fops for devices which are controlled by IOMMU.

The issue Stefano reported was when the dom0 is not direct mapped.
However...

>
> Prerequisites:
> https://urldefense.com/v3/__https://github.com/xen-project/xen/tree/stable-4.15__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B6EVrRb_$ [github[.]com]

... if you are really using 4.15, then the domain will always be direct
mapped. So I think this is a different one.

> https://urldefense.com/v3/__https://github.com/torvalds/linux/tree/v5.10__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B1n8el4F$ [github[.]com]

Any reason to not use the stable branch for 5.10? I don't know whether
your issue will be fixed there, but the stable branch usually contains a
lot of bug fixes (including security one). So it is a good idea to use
it over the first release of a kernel version.

> Issue caused in xen_swiotlb_map_page():
> ```
>   dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000,
> xen_phys_to_dma(phys): 64b1d0000
> ```

I can't seem to find this printk in Linux 5.10. Did you add it yourself?

>
> There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0
> swiotlb couldn't proceed to this address and throws the log:
>
> ```
> [   99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots)
> ```

This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
general, the use of the bounce buffer should be rare. So I would suggest
to find out why this is used.

Looking at the code, this suggests that one of the following check is false:

         /*
          * If the address happens to be in the device's DMA window,
          * we can safely return the device addr and not worry about bounce
          * buffering it.
          */
         if (dma_capable(dev, dev_addr, size, true) &&
             !range_straddles_page_boundary(phys, size) &&
                 !xen_arch_need_swiotlb(dev, phys, dev_addr) &&
                 swiotlb_force != SWIOTLB_FORCE)
                 goto done;

>
> Temporary, I resolved this issue by disabling swiotlb for dom0 at all
> because sure that all devices goes through IOMMU, but this mention can
> be true only for me.
>
> But, I think of a more reliable way is to declare a special IOMMU
> property in xen dts for each device. If the device controlled by IOMMU
> not need to set swiotlb fops in arch_setup_dma_ops.
> What do you think about it?

Let me start with that I agree we should disable swiotlb when we know
the device is protected. However, from what you describe, it sounds like
the same issue would appear if the IOMMU was disabled.

Therefore, I think we should first find out why Linux wants to bounce
the DMA buffer. Does your device have any DMA restriction?

Cheers,

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 38221 bytes --]

Re: Disable swiotlb for Dom0

[Julien Grall]

On 11/08/2021 09:49, Roman Skakun wrote:
> Hi, Julien!

Hi Roman,

>> > I have observed your patch here:
>> >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org]
>> >
>> > And I collided with the same issue, when Dom0 device trying to use
>> > swiotlb fops for devices which are controlled by IOMMU.
>>
>>The issue Stefano reported was when the dom0 is not direct mapped.
>>However...
> 
> I applied these patches:
> https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578 
> <https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578>
> https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e 
> <https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e>
> to check this more pragmatically.
> 
> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb 
> still used (other devices within dom0 used too), when dom0 is direct mapped:
> 
> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp, 
> XENFEAT_direct_mapped, use swiotlb
> 
> This means, that all devices are using swiotlb-xen DMA fops.
> By the way, before applying this patches, dom0 always used swiotlb-xen 
> fops for initial domain by design.

This is expected because your domain is direct mapped.

> 
> 
>> Any reason to not use  the stable branch for 5.10? I don't know whether
>> your issue will be  fixed there, but the stable branch usually contains a
>> lot of bug fixes (including  security one). So it is a good idea to use
>> it over the first release  of a kernel version.
> 
> Yes, sure, current BSP release based on 5.10 kernel:
> https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1 
> <https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1>
> based on https://github.com/renesas-rcar/linux-bsp 
> <https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2>
> BTW, I specified the wrong kernel URL in the previous massage, sorry.
> 
>> > Issue caused in xen_swiotlb_map_page():
>> > ```
>> > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, 
> page_to_phys: 64b1d0000,
>> > xen_phys_to_dma(phys): 64b1d0000
>> > ```
>>
>>I can't seem to find this printk in Linux 5.10. Did you add it yourself?
> 
> Yes, it's my own log.

Ok. Would you be able to provide more information on where the dom0 
memory is allocated and the list of host RAM?

> 
> 
>>This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
>>general, the use of the bounce buffer should be rare. So I would suggest
>>to find out why this is used.
>>
>>Looking at the code, this suggests that one of the following check is false:
>>
>>/*
>>        * If the address happens to be in the device's DMA window,
>>* we can safely return the device addr and not worry about bounce
>>* buffering it.
>>*/
>>if (dma_capable(dev, dev_addr, size, true) &&
>>!range_straddles_page_boundary(phys, size) &&
>>!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
>>swiotlb_force != SWIOTLB_FORCE)
>>goto done;
> 
> I checked this earlier and saw that dma_capable(dev, dev_addr, size, 
> true)returns false as expected because
> we got dev_addr equals 64b1d0000 and according to this expression under 
> dma_capable():
> 
> ```
> dma_addr_t end = dev_addr + size - 1;
> return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
> ```
> As result, DMA mask more than 32bit.
>> Let me start with that I agree we should disable swiotlb when we know
>> the device is protected. However, from what you describe, it sounds like
>> the same issue would appear if the IOMMU was disabled.
> 
> Yes, it looks like a potential issue. This means that swiotlb should be 
> worked correctly, when it's needed, agreed.
> But this is also potential improvement, and I presented this idea to 
> discuss and create some patches.

You might be able to remove the Xen swiotlb but I am not sure you will 
be able to remove the swiotlb completely if you have a device that only 
supports 32-bit DMA.

> 
>> Therefore, I think we should first find out why Linux wants to bounce
>> the DMA buffer.
> 
> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver 
> wants to use only 32 bit boundary address, but that's consequence.

Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

> I think, the main reason of using bounce buffer is MFN address, not DMA 
> phys address.

I don't understand this sentence. Can you clarify it?

Cheers,

-- 
Julien Grall

Re: Disable swiotlb for Dom0

[Roman Skakun]

[-- Attachment #1: Type: text/plain, Size: 9249 bytes --]

> Hi Roman,
Hi Julien,

>> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
>> still used (other devices within dom0 used too), when dom0 is direct mapped:
>>
>> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>>
>> This means, that all devices are using swiotlb-xen DMA fops.
>> By the way, before applying this patches, dom0 always used swiotlb-xen
>> fops for initial domain by design.

>> This is expected because your domain is direct mapped.

May be, I don't understand right, Stefano reported the same issue when dom0 is not direct mapped,
but I have direct mapped dom0 and problem still exists.

>Ok. Would you be able to provide more information on where the dom0
>memory is allocated and the list of host RAM?

Host memory:
DRAM:  7.9 GiB
Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB
Bank #1: 0x500000000 - 0x57fffffff, 2 GiB
Bank #2: 0x600000000 - 0x67fffffff, 2 GiB
Bank #3: 0x700000000 - 0x77fffffff, 2 GiB

dom0 memory map:
(XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
(XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
(XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB)
(XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB)


>> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver
>> wants to use only 32 bit boundary address, but that's consequence.
>>
> Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

Yes.

>> I think, the main reason of using bounce buffer is MFN address, not DMA
>> phys address.
>>
>I don't understand this sentence. Can you clarify it?

This address looks like the MFN because I'm using mapped grant tables from domU.

I've added the log and see the following:
with swiotlb:
[   78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000

without swiotlb (worked fine):
[   74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: b6000000

I guess, need to figure out why we got a normal dom0 DMA address (b6000000) and why 64b1d0000 when using swiotlb.

________________________________
From: Julien Grall <julien@xen.org>
Sent: Wednesday, August 11, 2021 1:30 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0



On 11/08/2021 09:49, Roman Skakun wrote:
> Hi, Julien!

Hi Roman,

>> > I have observed your patch here:
>> >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org]
>> >
>> > And I collided with the same issue, when Dom0 device trying to use
>> > swiotlb fops for devices which are controlled by IOMMU.
>>
>>The issue Stefano reported was when the dom0 is not direct mapped.
>>However...
>
> I applied these patches:
> https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com]>
> https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com]>
> to check this more pragmatically.
>
> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
> still used (other devices within dom0 used too), when dom0 is direct mapped:
>
> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
>
> This means, that all devices are using swiotlb-xen DMA fops.
> By the way, before applying this patches, dom0 always used swiotlb-xen
> fops for initial domain by design.

This is expected because your domain is direct mapped.

>
>
>> Any reason to not use  the stable branch for 5.10? I don't know whether
>> your issue will be  fixed there, but the stable branch usually contains a
>> lot of bug fixes (including  security one). So it is a good idea to use
>> it over the first release  of a kernel version.
>
> Yes, sure, current BSP release based on 5.10 kernel:
> https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com]>
> based on https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPWPOIbVJ$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPQXMt_yV$ [github[.]com]>
> BTW, I specified the wrong kernel URL in the previous massage, sorry.
>
>> > Issue caused in xen_swiotlb_map_page():
>> > ```
>> > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400,
> page_to_phys: 64b1d0000,
>> > xen_phys_to_dma(phys): 64b1d0000
>> > ```
>>
>>I can't seem to find this printk in Linux 5.10. Did you add it yourself?
>
> Yes, it's my own log.

Ok. Would you be able to provide more information on where the dom0
memory is allocated and the list of host RAM?

>
>
>>This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
>>general, the use of the bounce buffer should be rare. So I would suggest
>>to find out why this is used.
>>
>>Looking at the code, this suggests that one of the following check is false:
>>
>>/*
>>        * If the address happens to be in the device's DMA window,
>>* we can safely return the device addr and not worry about bounce
>>* buffering it.
>>*/
>>if (dma_capable(dev, dev_addr, size, true) &&
>>!range_straddles_page_boundary(phys, size) &&
>>!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
>>swiotlb_force != SWIOTLB_FORCE)
>>goto done;
>
> I checked this earlier and saw that dma_capable(dev, dev_addr, size,
> true)returns false as expected because
> we got dev_addr equals 64b1d0000 and according to this expression under
> dma_capable():
>
> ```
> dma_addr_t end = dev_addr + size - 1;
> return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
> ```
> As result, DMA mask more than 32bit.
>> Let me start with that I agree we should disable swiotlb when we know
>> the device is protected. However, from what you describe, it sounds like
>> the same issue would appear if the IOMMU was disabled.
>
> Yes, it looks like a potential issue. This means that swiotlb should be
> worked correctly, when it's needed, agreed.
> But this is also potential improvement, and I presented this idea to
> discuss and create some patches.

You might be able to remove the Xen swiotlb but I am not sure you will
be able to remove the swiotlb completely if you have a device that only
supports 32-bit DMA.

>
>> Therefore, I think we should first find out why Linux wants to bounce
>> the DMA buffer.
>
> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver
> wants to use only 32 bit boundary address, but that's consequence.

Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

> I think, the main reason of using bounce buffer is MFN address, not DMA
> phys address.

I don't understand this sentence. Can you clarify it?

Cheers,

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 27816 bytes --]

Re: Disable swiotlb for Dom0

[Julien Grall]

Hi,

On 11/08/2021 15:13, Roman Skakun wrote:
>> > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
>> > still used (other devices within dom0 used too), when dom0 is direct 
> mapped:
>> >
>> > [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> >
>>> This means, that all  devices are using swiotlb-xen DMA fops.
>> > By the way, before applying this patches, dom0 always used swiotlb-xen
>> > fops for initial domain by design.
> 
>> >This is expected because your domain is direct mapped.
> 
> May be, I don't understand right, Stefano reported the same issue when 
> dom0 is not direct mapped,
> but I have direct mapped dom0 and problem still exists.

I am not entirely sure why you think this is the same problem as 
Stefano. He asked to bypass the swiotlb, but AFAIK, this is not because 
the buffer get bounced.

Instead, it is because swiotlb-xen on Arm has been relying on its RAM to 
be direct-mapped (GFN == MFN). With cache coloring, the memory will not 
be direct-mapped, hence it will be broken.

> 
>>Ok. Would you be able to provide more information on where the dom0
>>memory is allocated  and the list of host RAM?
> 
> Host memory:
> DRAM:  7.9 GiB
> Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB
> Bank #1: 0x500000000 - 0x57fffffff, 2 GiB
> Bank #2: 0x600000000 - 0x67fffffff, 2 GiB
> Bank #3: 0x700000000 - 0x77fffffff, 2 GiB
> 
> dom0 memory map:
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
> (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB)
> (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB)

Thanks! So you have some memory assigned above 4GB to dom0 as well.

>>> We retrieved dev_addr(64b1d0000)  + size > 32bit mask, but fcp driver
>>> wants to use only  32 bit boundary address, but that's consequence.
>>>
>> Ok. So your device is only capable to do a 32-bit DMA. Is that correct?
> 
> Yes.
> 
>> > I think, the main reason of using bounce buffer is MFN address, not DMA
>> > phys address.
>> >
>>I don't understand this sentence. Can you clarify it?
> 
> This address looks like theMFN because I'm using mapped grant tables 
> from domU.
> 
> I've added the log and see the following:
> with swiotlb:
> [   78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: 
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000
> 
> without swiotlb (worked fine):
> [   74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: 
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma:b6000000
> 
> I guess, need to figure out why we got a normal dom0 DMA address 
> (b6000000) and why 64b1d0000 when using swiotlb.

So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.

swiotlb-xen on Arm will convert it to the MFN because it is not aware 
whether the device is behind an IOMMU.

As the address is too high to be handled by the device, swiotlb will try 
to bounce it. I think it is correct to bounce the page but I am not sure 
why it can't. What the size of the DMA transaction?

However, even if you disable xen-swiotlb, you are likely going to face 
the same issue sooner or later because the grant can be mapped anywhere 
in the memory of dom0 (the balloon code doesn't look to restrict where 
the memory can be allocated). So it is possible for the grant to be 
mapped in the dom0 memory above 4GB.

Oleksandr is also looking to provide a safe range which would be outside 
of the existing RAM. So, I believe, you will have to bounce the DMA 
buffer unless we always force the grant/foreign mapping to be mapped 
below 4GB.

Cheers,

-- 
Julien Grall

Re: Disable swiotlb for Dom0

[Roman Skakun]

[-- Attachment #1: Type: text/plain, Size: 12593 bytes --]

Hi Julien,

> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>
> swiotlb-xen on Arm will convert it to the MFN because it is not aware
> whether the device is behind an IOMMU.

If I'm understand right, it seems like that swiotlb-xen is not ready to work properly in case
when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
Maybe you know some ideas to overcome this condition?

>  As the address is too high to be handled by the device, swiotlb will try
>  to bounce it. I think it is correct to bounce the page but I am not sure
>  why it can't. What the size of the DMA transaction?

The DMA map size is 3686400 bytes.

I've added several logs to swiotlb map_single() and see:
[  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: 64af97000, needed: 708,
avail: 7fc0, stride: 2, index: 4160

It's expected because:
....
[  259.468006] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31742]=2 slots < 708 nslots. Continue...
[  259.477070] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31744]=80 slots < 708 nslots. Continue...
[  259.486229] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31746]=7e slots < 708 nslots. Continue...
[  259.495387] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31748]=7c slots < 708 nslots. Continue...
[  259.504546] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31750]=7a slots < 708 nslots. Continue...
[  259.513704] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31752]=78 slots < 708 nslots. Continue...
[  259.522863] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31754]=76 slots < 708 nslots. Continue...
[  259.532021] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31756]=74 slots < 708 nslots. Continue...
[  259.541179] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31758]=72 slots < 708 nslots. Continue...
[  259.550338] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31760]=70 slots < 708 nslots. Continue...
[  259.559496] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31762]=6e slots < 708 nslots. Continue...
[  259.568660] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31764]=6c slots < 708 nslots. Continue...
[  259.577813] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31766]=6a slots < 708 nslots. Continue...
[  259.586972] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31768]=68 slots < 708 nslots. Continue...
[  259.596130] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31770]=66 slots < 708 nslots. Continue...
[  259.605289] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31772]=64 slots < 708 nslots. Continue...
[  259.614447] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31774]=62 slots < 708 nslots. Continue...
[  259.623606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31776]=60 slots < 708 nslots. Continue...
[  259.632764] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31778]=5e slots < 708 nslots. Continue...
[  259.641922] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31780]=5c slots < 708 nslots. Continue...
[  259.651081] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31782]=5a slots < 708 nslots. Continue...
[  259.660239] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31784]=58 slots < 708 nslots. Continue...
[  259.669398] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31786]=56 slots < 708 nslots. Continue...
[  259.678563] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31788]=54 slots < 708 nslots. Continue...
[  259.687714] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31790]=52 slots < 708 nslots. Continue...
[  259.696873] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31792]=50 slots < 708 nslots. Continue...
[  259.706032] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31794]=4e slots < 708 nslots. Continue...
[  259.715190] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31796]=4c slots < 708 nslots. Continue...
[  259.724348] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31798]=4a slots < 708 nslots. Continue...
[  259.733507] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31800]=48 slots < 708 nslots. Continue...
[  259.742665] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31802]=46 slots < 708 nslots. Continue...
[  259.751824] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31804]=44 slots < 708 nslots. Continue...
[  259.760982] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31806]=42 slots < 708 nslots. Continue...
[  259.770141] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31808]=40 slots < 708 nslots. Continue...
[  259.779299] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31810]=3e slots < 708 nslots. Continue...
[  259.788466] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31812]=3c slots < 708 nslots. Continue...
[  259.797615] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31814]=3a slots < 708 nslots. Continue...
[  259.806774] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31816]=38 slots < 708 nslots. Continue...
[  259.815933] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31818]=36 slots < 708 nslots. Continue...
[  259.825091] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31820]=34 slots < 708 nslots. Continue...
[  259.834249] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31822]=32 slots < 708 nslots. Continue...
[  259.843408] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31824]=30 slots < 708 nslots. Continue...
[  259.852567] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31826]=2e slots < 708 nslots. Continue...
[  259.861725] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31828]=2c slots < 708 nslots. Continue...
[  259.870883] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31830]=2a slots < 708 nslots. Continue...
[  259.880042] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31832]=28 slots < 708 nslots. Continue...
[  259.889200] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31834]=26 slots < 708 nslots. Continue...
[  259.898365] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31836]=24 slots < 708 nslots. Continue...
[  259.907516] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31838]=22 slots < 708 nslots. Continue...
[  259.916676] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31840]=20 slots < 708 nslots. Continue...
[  259.925834] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31842]=1e slots < 708 nslots. Continue...
[  259.934992] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31844]=1c slots < 708 nslots. Continue...
[  259.944151] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31846]=1a slots < 708 nslots. Continue...
[  259.953309] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31848]=18 slots < 708 nslots. Continue...
[  259.962468] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31850]=16 slots < 708 nslots. Continue...
[  259.971626] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31852]=14 slots < 708 nslots. Continue...
[  259.980784] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31854]=12 slots < 708 nslots. Continue...
[  259.989943] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31856]=10 slots < 708 nslots. Continue...
[  259.999102] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31858]=e slots < 708 nslots. Continue...
[  260.008181] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31860]=c slots < 708 nslots. Continue...
[  260.017245] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31862]=a slots < 708 nslots. Continue...
[  260.026318] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31864]=8 slots < 708 nslots. Continue...
[  260.035389] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31866]=6 slots < 708 nslots. Continue...
[  260.044461] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31868]=4 slots < 708 nslots. Continue...
[  260.053533] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31870]=2 slots < 708 nslots. Continue...
[  260.062606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31872]=80 slots < 708 nslots. Continue...
....

Swiotlb did not fit requested slots because the maximum slot size equals IO_TLB_SEGSIZE=128 by default.
But I think, we cannot use 64af97000 address in the swiotlb_bounce() directly.

________________________________
From: Julien Grall <julien@xen.org>
Sent: Wednesday, August 11, 2021 6:20 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0

Hi,

On 11/08/2021 15:13, Roman Skakun wrote:
>> > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
>> > still used (other devices within dom0 used too), when dom0 is direct
> mapped:
>> >
>> > [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> > [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
>> > XENFEAT_direct_mapped, use swiotlb
>> >
>>> This means, that all  devices are using swiotlb-xen DMA fops.
>> > By the way, before applying this patches, dom0 always used swiotlb-xen
>> > fops for initial domain by design.
>
>> >This is expected because your domain is direct mapped.
>
> May be, I don't understand right, Stefano reported the same issue when
> dom0 is not direct mapped,
> but I have direct mapped dom0 and problem still exists.

I am not entirely sure why you think this is the same problem as
Stefano. He asked to bypass the swiotlb, but AFAIK, this is not because
the buffer get bounced.

Instead, it is because swiotlb-xen on Arm has been relying on its RAM to
be direct-mapped (GFN == MFN). With cache coloring, the memory will not
be direct-mapped, hence it will be broken.

>
>>Ok. Would you be able to provide more information on where the dom0
>>memory is allocated  and the list of host RAM?
>
> Host memory:
> DRAM:  7.9 GiB
> Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB
> Bank #1: 0x500000000 - 0x57fffffff, 2 GiB
> Bank #2: 0x600000000 - 0x67fffffff, 2 GiB
> Bank #3: 0x700000000 - 0x77fffffff, 2 GiB
>
> dom0 memory map:
> (XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
> (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
> (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB)
> (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB)

Thanks! So you have some memory assigned above 4GB to dom0 as well.

>>> We retrieved dev_addr(64b1d0000)  + size > 32bit mask, but fcp driver
>>> wants to use only  32 bit boundary address, but that's consequence.
>>>
>> Ok. So your device is only capable to do a 32-bit DMA. Is that correct?
>
> Yes.
>
>> > I think, the main reason of using bounce buffer is MFN address, not DMA
>> > phys address.
>> >
>>I don't understand this sentence. Can you clarify it?
>
> This address looks like theMFN because I'm using mapped grant tables
> from domU.
>
> I've added the log and see the following:
> with swiotlb:
> [   78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page:
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000
>
> without swiotlb (worked fine):
> [   74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page:
> fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma:b6000000
>
> I guess, need to figure out why we got a normal dom0 DMA address
> (b6000000) and why 64b1d0000 when using swiotlb.

So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.

swiotlb-xen on Arm will convert it to the MFN because it is not aware
whether the device is behind an IOMMU.

As the address is too high to be handled by the device, swiotlb will try
to bounce it. I think it is correct to bounce the page but I am not sure
why it can't. What the size of the DMA transaction?

However, even if you disable xen-swiotlb, you are likely going to face
the same issue sooner or later because the grant can be mapped anywhere
in the memory of dom0 (the balloon code doesn't look to restrict where
the memory can be allocated). So it is possible for the grant to be
mapped in the dom0 memory above 4GB.

Oleksandr is also looking to provide a safe range which would be outside
of the existing RAM. So, I believe, you will have to bounce the DMA
buffer unless we always force the grant/foreign mapping to be mapped
below 4GB.

Cheers,

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 20263 bytes --]

Re: Disable swiotlb for Dom0

[Julien Grall]

On 13/08/2021 10:38, Roman Skakun wrote:
> Hi Julien,

Hi Roman,

>> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>>
>> swiotlb-xen on Arm will convert it to the MFN because it is not aware
>> whether the device is behind an IOMMU.
> 
> If I'm understand right, it seems like that swiotlb-xen is not ready to 
> work properly in case
> when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
> Maybe you know some ideas to overcome this condition?

swiotlb-xen work as intended. You have a DMA buffer at an address that 
your device cannot deal with. So it will try to bounce it.

> 
>>  As the address is too high to be handled by the device, swiotlb will try
>>  to bounce it. I think it is correct to bounce the page but I am not sure
>>  why it can't. What the size of the DMA transaction?
> 
> The DMA map size is 3686400 bytes.

So that's a 3MB buffer. I am slightly confused because in an earlier 
message you wrote that the memory is coming from the guest. How did you 
map it?

> 
> I've added several logs to swiotlb map_single() and see:
> [  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: 
> 64af97000, needed: 708,
> avail: 7fc0, stride: 2, index: 4160

I am not sure how to read the logs... Are the number in hexadecimal or 
decimal? It might be useful if you post the diff of your changes.

[...]

> Swiotlb did not fit requested slots because the maximum slot size equals 
> IO_TLB_SEGSIZE=128 by default.

Ok. So it sounds like your problem is the have a DMA buffer that is too 
large for the default swiotlb. Did you try to bump the value from the 
command line?

> But I think, we cannot use64af97000 address in the swiotlb_bounce() 
> directly.

I am not sure to understand what you mean. Can you clarify?

Cheers,

[...]

-- 
Julien Grall

Re: Disable swiotlb for Dom0

[Roman Skakun]

[-- Attachment #1: Type: text/plain, Size: 4439 bytes --]


>> But I think, we cannot use64af97000 address in the swiotlb_bounce()
>> directly.
>>
>I am not sure to understand what you mean. Can you clarify?

I thought, that the address 64af97000 can be used directly here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572
and I was confused about it.
After some diggings I realized that this code is not reachable because
DMA_ATTR_SKIP_CPU_SYNC is active.

>> Swiotlb did not fit requested slots because the maximum slot size equals
>> IO_TLB_SEGSIZE=128 by default.
>
>> Ok. So it sounds like your problem is the have a DMA buffer that is too
>> large for the default swiotlb. Did you try to bump the value from the
>> command line?

Yes, you are right. I checked the implementation more detailed and figured out that
swiotlb tries to find a contiguous segment that is a big enough, as shown here:
https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528,

But the maximum TLB segment equals 128K by default in accordance with
https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25

This means that the max size of TLB segment equals:
128 * 2048 = 256K

After this, I tried to increase the TLB segment size like this:

diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h
index fbdc65782195..85f32865bb6c 100644
--- a/include/linux/swiotlb.h
+++ b/include/linux/swiotlb.h
@@ -22,7 +22,7 @@ enum swiotlb_force {
  * must be a power of 2.  What is the appropriate value ?
  * The complexity of {map,unmap}_single is linearly dependent on this value.
  */
-#define IO_TLB_SEGSIZE 128
+#define IO_TLB_SEGSIZE 2048

 /*
  * log of the size of each IO TLB slab.  The number of slabs is command line

and the problem is gone.

Also, I found the article https://www.xilinx.com/support/answers/72694.html,
where I believe the same issue was mentioned.

Thank you so much for your time and help!

________________________________
From: Julien Grall <julien@xen.org>
Sent: Friday, August 13, 2021 1:51 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0



On 13/08/2021 10:38, Roman Skakun wrote:
> Hi Julien,

Hi Roman,

>> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU.
>>
>> swiotlb-xen on Arm will convert it to the MFN because it is not aware
>> whether the device is behind an IOMMU.
>
> If I'm understand right, it seems like that swiotlb-xen is not ready to
> work properly in case
> when we retrieved MFN instead of proper GFN mapped to Dom0 memory.
> Maybe you know some ideas to overcome this condition?

swiotlb-xen work as intended. You have a DMA buffer at an address that
your device cannot deal with. So it will try to bounce it.

>
>>  As the address is too high to be handled by the device, swiotlb will try
>>  to bounce it. I think it is correct to bounce the page but I am not sure
>>  why it can't. What the size of the DMA transaction?
>
> The DMA map size is 3686400 bytes.

So that's a 3MB buffer. I am slightly confused because in an earlier
message you wrote that the memory is coming from the guest. How did you
map it?

>
> I've added several logs to swiotlb map_single() and see:
> [  151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr:
> 64af97000, needed: 708,
> avail: 7fc0, stride: 2, index: 4160

I am not sure how to read the logs... Are the number in hexadecimal or
decimal? It might be useful if you post the diff of your changes.

[...]

> Swiotlb did not fit requested slots because the maximum slot size equals
> IO_TLB_SEGSIZE=128 by default.

Ok. So it sounds like your problem is the have a DMA buffer that is too
large for the default swiotlb. Did you try to bump the value from the
command line?

> But I think, we cannot use64af97000 address in the swiotlb_bounce()
> directly.

I am not sure to understand what you mean. Can you clarify?

Cheers,

[...]

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 13972 bytes --]