Re: Disable swiotlb for Dom0

[Roman Skakun <Roman_Skakun@epam.com>]

[-- Attachment #1: Type: text/plain, Size: 9249 bytes --]

> Hi Roman,
Hi Julien,

>> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
>> still used (other devices within dom0 used too), when dom0 is direct mapped:
>>
>> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
>> XENFEAT_direct_mapped, use swiotlb
>>
>> This means, that all devices are using swiotlb-xen DMA fops.
>> By the way, before applying this patches, dom0 always used swiotlb-xen
>> fops for initial domain by design.

>> This is expected because your domain is direct mapped.

May be, I don't understand right, Stefano reported the same issue when dom0 is not direct mapped,
but I have direct mapped dom0 and problem still exists.

>Ok. Would you be able to provide more information on where the dom0
>memory is allocated and the list of host RAM?

Host memory:
DRAM:  7.9 GiB
Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB
Bank #1: 0x500000000 - 0x57fffffff, 2 GiB
Bank #2: 0x600000000 - 0x67fffffff, 2 GiB
Bank #3: 0x700000000 - 0x77fffffff, 2 GiB

dom0 memory map:
(XEN) Allocating 1:1 mappings totalling 2048MB for dom0:
(XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB)
(XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB)
(XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB)


>> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver
>> wants to use only 32 bit boundary address, but that's consequence.
>>
> Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

Yes.

>> I think, the main reason of using bounce buffer is MFN address, not DMA
>> phys address.
>>
>I don't understand this sentence. Can you clarify it?

This address looks like the MFN because I'm using mapped grant tables from domU.

I've added the log and see the following:
with swiotlb:
[   78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000

without swiotlb (worked fine):
[   74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: b6000000

I guess, need to figure out why we got a normal dom0 DMA address (b6000000) and why 64b1d0000 when using swiotlb.

________________________________
From: Julien Grall <julien@xen.org>
Sent: Wednesday, August 11, 2021 1:30 PM
To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org>
Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com>
Subject: Re: Disable swiotlb for Dom0



On 11/08/2021 09:49, Roman Skakun wrote:
> Hi, Julien!

Hi Roman,

>> > I have observed your patch here:
>> >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org]
>> >
>> > And I collided with the same issue, when Dom0 device trying to use
>> > swiotlb fops for devices which are controlled by IOMMU.
>>
>>The issue Stefano reported was when the dom0 is not direct mapped.
>>However...
>
> I applied these patches:
> https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com]>
> https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com]>
> to check this more pragmatically.
>
> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb
> still used (other devices within dom0 used too), when dom0 is direct mapped:
>
> [    1.870363] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
> [    1.878352] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
> [    1.886309] xen_swiotlb_detect() dev: rcar-fcp,
> XENFEAT_direct_mapped, use swiotlb
>
> This means, that all devices are using swiotlb-xen DMA fops.
> By the way, before applying this patches, dom0 always used swiotlb-xen
> fops for initial domain by design.

This is expected because your domain is direct mapped.

>
>
>> Any reason to not use  the stable branch for 5.10? I don't know whether
>> your issue will be  fixed there, but the stable branch usually contains a
>> lot of bug fixes (including  security one). So it is a good idea to use
>> it over the first release  of a kernel version.
>
> Yes, sure, current BSP release based on 5.10 kernel:
> https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com]>
> based on https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPWPOIbVJ$ [github[.]com]
> <https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPQXMt_yV$ [github[.]com]>
> BTW, I specified the wrong kernel URL in the previous massage, sorry.
>
>> > Issue caused in xen_swiotlb_map_page():
>> > ```
>> > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400,
> page_to_phys: 64b1d0000,
>> > xen_phys_to_dma(phys): 64b1d0000
>> > ```
>>
>>I can't seem to find this printk in Linux 5.10. Did you add it yourself?
>
> Yes, it's my own log.

Ok. Would you be able to provide more information on where the dom0
memory is allocated and the list of host RAM?

>
>
>>This line suggests that the SWIOTLB tried to bounce the DMA buffer. In
>>general, the use of the bounce buffer should be rare. So I would suggest
>>to find out why this is used.
>>
>>Looking at the code, this suggests that one of the following check is false:
>>
>>/*
>>        * If the address happens to be in the device's DMA window,
>>* we can safely return the device addr and not worry about bounce
>>* buffering it.
>>*/
>>if (dma_capable(dev, dev_addr, size, true) &&
>>!range_straddles_page_boundary(phys, size) &&
>>!xen_arch_need_swiotlb(dev, phys, dev_addr) &&
>>swiotlb_force != SWIOTLB_FORCE)
>>goto done;
>
> I checked this earlier and saw that dma_capable(dev, dev_addr, size,
> true)returns false as expected because
> we got dev_addr equals 64b1d0000 and according to this expression under
> dma_capable():
>
> ```
> dma_addr_t end = dev_addr + size - 1;
> return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit);
> ```
> As result, DMA mask more than 32bit.
>> Let me start with that I agree we should disable swiotlb when we know
>> the device is protected. However, from what you describe, it sounds like
>> the same issue would appear if the IOMMU was disabled.
>
> Yes, it looks like a potential issue. This means that swiotlb should be
> worked correctly, when it's needed, agreed.
> But this is also potential improvement, and I presented this idea to
> discuss and create some patches.

You might be able to remove the Xen swiotlb but I am not sure you will
be able to remove the swiotlb completely if you have a device that only
supports 32-bit DMA.

>
>> Therefore, I think we should first find out why Linux wants to bounce
>> the DMA buffer.
>
> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver
> wants to use only 32 bit boundary address, but that's consequence.

Ok. So your device is only capable to do a 32-bit DMA. Is that correct?

> I think, the main reason of using bounce buffer is MFN address, not DMA
> phys address.

I don't understand this sentence. Can you clarify it?

Cheers,

--
Julien Grall

[-- Attachment #2: Type: text/html, Size: 27816 bytes --]