* Disable swiotlb for Dom0 @ 2021-08-10 15:38 Roman Skakun 2021-08-10 16:16 ` Julien Grall 0 siblings, 1 reply; 9+ messages in thread From: Roman Skakun @ 2021-08-10 15:38 UTC (permalink / raw) To: sstabellini Cc: Bertrand Marquis, Andrii Anisov, Julien Grall, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Roman Skakun, Jan Beulich Hi, Stefano! I have observed your patch here: https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/ And I collided with the same issue, when Dom0 device trying to use swiotlb fops for devices which are controlled by IOMMU. Prerequisites: https://github.com/xen-project/xen/tree/stable-4.15 https://github.com/torvalds/linux/tree/v5.10 Issue caused in xen_swiotlb_map_page(): ``` dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, xen_phys_to_dma(phys): 64b1d0000 ``` There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0 swiotlb couldn't proceed to this address and throws the log: ``` [ 99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots) ``` Temporary, I resolved this issue by disabling swiotlb for dom0 at all because sure that all devices goes through IOMMU, but this mention can be true only for me. But, I think of a more reliable way is to declare a special IOMMU property in xen dts for each device. If the device controlled by IOMMU not need to set swiotlb fops in arch_setup_dma_ops. What do you think about it? ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-10 15:38 Disable swiotlb for Dom0 Roman Skakun @ 2021-08-10 16:16 ` Julien Grall 2021-08-11 8:49 ` Roman Skakun 0 siblings, 1 reply; 9+ messages in thread From: Julien Grall @ 2021-08-10 16:16 UTC (permalink / raw) To: Roman Skakun, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich On 10/08/2021 16:38, Roman Skakun wrote: > Hi, Stefano! Hi, > I have observed your patch here: > https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/ > > And I collided with the same issue, when Dom0 device trying to use > swiotlb fops for devices which are controlled by IOMMU. The issue Stefano reported was when the dom0 is not direct mapped. However... > > Prerequisites: > https://github.com/xen-project/xen/tree/stable-4.15 ... if you are really using 4.15, then the domain will always be direct mapped. So I think this is a different one. > https://github.com/torvalds/linux/tree/v5.10 Any reason to not use the stable branch for 5.10? I don't know whether your issue will be fixed there, but the stable branch usually contains a lot of bug fixes (including security one). So it is a good idea to use it over the first release of a kernel version. > Issue caused in xen_swiotlb_map_page(): > ``` > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, > xen_phys_to_dma(phys): 64b1d0000 > ``` I can't seem to find this printk in Linux 5.10. Did you add it yourself? > > There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0 > swiotlb couldn't proceed to this address and throws the log: > > ``` > [ 99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots) > ``` This line suggests that the SWIOTLB tried to bounce the DMA buffer. In general, the use of the bounce buffer should be rare. So I would suggest to find out why this is used. Looking at the code, this suggests that one of the following check is false: /* * If the address happens to be in the device's DMA window, * we can safely return the device addr and not worry about bounce * buffering it. */ if (dma_capable(dev, dev_addr, size, true) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && swiotlb_force != SWIOTLB_FORCE) goto done; > > Temporary, I resolved this issue by disabling swiotlb for dom0 at all > because sure that all devices goes through IOMMU, but this mention can > be true only for me. > > But, I think of a more reliable way is to declare a special IOMMU > property in xen dts for each device. If the device controlled by IOMMU > not need to set swiotlb fops in arch_setup_dma_ops. > What do you think about it? Let me start with that I agree we should disable swiotlb when we know the device is protected. However, from what you describe, it sounds like the same issue would appear if the IOMMU was disabled. Therefore, I think we should first find out why Linux wants to bounce the DMA buffer. Does your device have any DMA restriction? Cheers, -- Julien Grall ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-10 16:16 ` Julien Grall @ 2021-08-11 8:49 ` Roman Skakun 2021-08-11 10:30 ` Julien Grall 0 siblings, 1 reply; 9+ messages in thread From: Roman Skakun @ 2021-08-11 8:49 UTC (permalink / raw) To: Julien Grall, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich, Roman Skakun [-- Attachment #1: Type: text/plain, Size: 8389 bytes --] Hi, Julien! Thanks for the answer! >> I have observed your patch here: >> https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$<https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$> [patchwork[.]kernel[.]org] >> >> And I collided with the same issue, when Dom0 device trying to use >> swiotlb fops for devices which are controlled by IOMMU. > >The issue Stefano reported was when the dom0 is not direct mapped. >However... I applied these patches: https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578 https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e to check this more pragmatically. Also, I added the log in xen_swiotlb_detect() and can see that swiotlb still used (other devices within dom0 used too), when dom0 is direct mapped: [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, XENFEAT_direct_mapped, use swiotlb This means, that all devices are using swiotlb-xen DMA fops. By the way, before applying this patches, dom0 always used swiotlb-xen fops for initial domain by design. > Any reason to not use the stable branch for 5.10? I don't know whether > your issue will be fixed there, but the stable branch usually contains a > lot of bug fixes (including security one). So it is a good idea to use > it over the first release of a kernel version. Yes, sure, current BSP release based on 5.10 kernel: https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1 based on https://github.com/renesas-rcar/linux-bsp<https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2> BTW, I specified the wrong kernel URL in the previous massage, sorry. >> Issue caused in xen_swiotlb_map_page(): >> ``` >> dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, >> xen_phys_to_dma(phys): 64b1d0000 >> ``` > >I can't seem to find this printk in Linux 5.10. Did you add it yourself? Yes, it's my own log. > This line suggests that the SWIOTLB tried to bounce the DMA buffer. In > general, the use of the bounce buffer should be rare. So I would suggest > to find out why this is used. > > Looking at the code, this suggests that one of the following check is false: > > /* > * If the address happens to be in the device's DMA window, > * we can safely return the device addr and not worry about bounce > * buffering it. > */ > if (dma_capable(dev, dev_addr, size, true) && > !range_straddles_page_boundary(phys, size) && > !xen_arch_need_swiotlb(dev, phys, dev_addr) && > swiotlb_force != SWIOTLB_FORCE) > goto done; I checked this earlier and saw that dma_capable(dev, dev_addr, size, true) returns false as expected because we got dev_addr equals 64b1d0000 and according to this expression under dma_capable(): ``` dma_addr_t end = dev_addr + size - 1; return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit); ``` As result, DMA mask more than 32bit. > Let me start with that I agree we should disable swiotlb when we know > the device is protected. However, from what you describe, it sounds like > the same issue would appear if the IOMMU was disabled. Yes, it looks like a potential issue. This means that swiotlb should be worked correctly, when it's needed, agreed. But this is also potential improvement, and I presented this idea to discuss and create some patches. > Therefore, I think we should first find out why Linux wants to bounce > the DMA buffer. We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver wants to use only 32 bit boundary address, but that's consequence. I think, the main reason of using bounce buffer is MFN address, not DMA phys address. Cheers! ________________________________ From: Julien Grall <julien@xen.org> Sent: Tuesday, August 10, 2021 7:16 PM To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org> Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com> Subject: Re: Disable swiotlb for Dom0 On 10/08/2021 16:38, Roman Skakun wrote: > Hi, Stefano! Hi, > I have observed your patch here: > https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad-T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ [patchwork[.]kernel[.]org] > > And I collided with the same issue, when Dom0 device trying to use > swiotlb fops for devices which are controlled by IOMMU. The issue Stefano reported was when the dom0 is not direct mapped. However... > > Prerequisites: > https://urldefense.com/v3/__https://github.com/xen-project/xen/tree/stable-4.15__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B6EVrRb_$ [github[.]com] ... if you are really using 4.15, then the domain will always be direct mapped. So I think this is a different one. > https://urldefense.com/v3/__https://github.com/torvalds/linux/tree/v5.10__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B1n8el4F$ [github[.]com] Any reason to not use the stable branch for 5.10? I don't know whether your issue will be fixed there, but the stable branch usually contains a lot of bug fixes (including security one). So it is a good idea to use it over the first release of a kernel version. > Issue caused in xen_swiotlb_map_page(): > ``` > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, page_to_phys: 64b1d0000, > xen_phys_to_dma(phys): 64b1d0000 > ``` I can't seem to find this printk in Linux 5.10. Did you add it yourself? > > There is retrieved MFN(0x64b1d0000), which belongs to DomU. Dom0 > swiotlb couldn't proceed to this address and throws the log: > > ``` > [ 99.504990] rcar-fcp fea2f000.fcp: swiotlb buffer is full (sz: 3686400 bytes), total 32768 (slots), used 64 (slots) > ``` This line suggests that the SWIOTLB tried to bounce the DMA buffer. In general, the use of the bounce buffer should be rare. So I would suggest to find out why this is used. Looking at the code, this suggests that one of the following check is false: /* * If the address happens to be in the device's DMA window, * we can safely return the device addr and not worry about bounce * buffering it. */ if (dma_capable(dev, dev_addr, size, true) && !range_straddles_page_boundary(phys, size) && !xen_arch_need_swiotlb(dev, phys, dev_addr) && swiotlb_force != SWIOTLB_FORCE) goto done; > > Temporary, I resolved this issue by disabling swiotlb for dom0 at all > because sure that all devices goes through IOMMU, but this mention can > be true only for me. > > But, I think of a more reliable way is to declare a special IOMMU > property in xen dts for each device. If the device controlled by IOMMU > not need to set swiotlb fops in arch_setup_dma_ops. > What do you think about it? Let me start with that I agree we should disable swiotlb when we know the device is protected. However, from what you describe, it sounds like the same issue would appear if the IOMMU was disabled. Therefore, I think we should first find out why Linux wants to bounce the DMA buffer. Does your device have any DMA restriction? Cheers, -- Julien Grall [-- Attachment #2: Type: text/html, Size: 38221 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-11 8:49 ` Roman Skakun @ 2021-08-11 10:30 ` Julien Grall 2021-08-11 14:13 ` Roman Skakun 0 siblings, 1 reply; 9+ messages in thread From: Julien Grall @ 2021-08-11 10:30 UTC (permalink / raw) To: Roman Skakun, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich On 11/08/2021 09:49, Roman Skakun wrote: > Hi, Julien! Hi Roman, >> > I have observed your patch here: >> >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org] >> > >> > And I collided with the same issue, when Dom0 device trying to use >> > swiotlb fops for devices which are controlled by IOMMU. >> >>The issue Stefano reported was when the dom0 is not direct mapped. >>However... > > I applied these patches: > https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578 > <https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578> > https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e > <https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e> > to check this more pragmatically. > > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb > still used (other devices within dom0 used too), when dom0 is direct mapped: > > [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > > This means, that all devices are using swiotlb-xen DMA fops. > By the way, before applying this patches, dom0 always used swiotlb-xen > fops for initial domain by design. This is expected because your domain is direct mapped. > > >> Any reason to not use the stable branch for 5.10? I don't know whether >> your issue will be fixed there, but the stable branch usually contains a >> lot of bug fixes (including security one). So it is a good idea to use >> it over the first release of a kernel version. > > Yes, sure, current BSP release based on 5.10 kernel: > https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1 > <https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1> > based on https://github.com/renesas-rcar/linux-bsp > <https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2> > BTW, I specified the wrong kernel URL in the previous massage, sorry. > >> > Issue caused in xen_swiotlb_map_page(): >> > ``` >> > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, > page_to_phys: 64b1d0000, >> > xen_phys_to_dma(phys): 64b1d0000 >> > ``` >> >>I can't seem to find this printk in Linux 5.10. Did you add it yourself? > > Yes, it's my own log. Ok. Would you be able to provide more information on where the dom0 memory is allocated and the list of host RAM? > > >>This line suggests that the SWIOTLB tried to bounce the DMA buffer. In >>general, the use of the bounce buffer should be rare. So I would suggest >>to find out why this is used. >> >>Looking at the code, this suggests that one of the following check is false: >> >>/* >> * If the address happens to be in the device's DMA window, >>* we can safely return the device addr and not worry about bounce >>* buffering it. >>*/ >>if (dma_capable(dev, dev_addr, size, true) && >>!range_straddles_page_boundary(phys, size) && >>!xen_arch_need_swiotlb(dev, phys, dev_addr) && >>swiotlb_force != SWIOTLB_FORCE) >>goto done; > > I checked this earlier and saw that dma_capable(dev, dev_addr, size, > true)returns false as expected because > we got dev_addr equals 64b1d0000 and according to this expression under > dma_capable(): > > ``` > dma_addr_t end = dev_addr + size - 1; > return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit); > ``` > As result, DMA mask more than 32bit. >> Let me start with that I agree we should disable swiotlb when we know >> the device is protected. However, from what you describe, it sounds like >> the same issue would appear if the IOMMU was disabled. > > Yes, it looks like a potential issue. This means that swiotlb should be > worked correctly, when it's needed, agreed. > But this is also potential improvement, and I presented this idea to > discuss and create some patches. You might be able to remove the Xen swiotlb but I am not sure you will be able to remove the swiotlb completely if you have a device that only supports 32-bit DMA. > >> Therefore, I think we should first find out why Linux wants to bounce >> the DMA buffer. > > We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver > wants to use only 32 bit boundary address, but that's consequence. Ok. So your device is only capable to do a 32-bit DMA. Is that correct? > I think, the main reason of using bounce buffer is MFN address, not DMA > phys address. I don't understand this sentence. Can you clarify it? Cheers, -- Julien Grall ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-11 10:30 ` Julien Grall @ 2021-08-11 14:13 ` Roman Skakun 2021-08-11 15:20 ` Julien Grall 0 siblings, 1 reply; 9+ messages in thread From: Roman Skakun @ 2021-08-11 14:13 UTC (permalink / raw) To: Julien Grall, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich [-- Attachment #1: Type: text/plain, Size: 9249 bytes --] > Hi Roman, Hi Julien, >> Also, I added the log in xen_swiotlb_detect() and can see that swiotlb >> still used (other devices within dom0 used too), when dom0 is direct mapped: >> >> [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, >> XENFEAT_direct_mapped, use swiotlb >> [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, >> XENFEAT_direct_mapped, use swiotlb >> [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, >> XENFEAT_direct_mapped, use swiotlb >> >> This means, that all devices are using swiotlb-xen DMA fops. >> By the way, before applying this patches, dom0 always used swiotlb-xen >> fops for initial domain by design. >> This is expected because your domain is direct mapped. May be, I don't understand right, Stefano reported the same issue when dom0 is not direct mapped, but I have direct mapped dom0 and problem still exists. >Ok. Would you be able to provide more information on where the dom0 >memory is allocated and the list of host RAM? Host memory: DRAM: 7.9 GiB Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB Bank #1: 0x500000000 - 0x57fffffff, 2 GiB Bank #2: 0x600000000 - 0x67fffffff, 2 GiB Bank #3: 0x700000000 - 0x77fffffff, 2 GiB dom0 memory map: (XEN) Allocating 1:1 mappings totalling 2048MB for dom0: (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB) (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB) (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB) >> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver >> wants to use only 32 bit boundary address, but that's consequence. >> > Ok. So your device is only capable to do a 32-bit DMA. Is that correct? Yes. >> I think, the main reason of using bounce buffer is MFN address, not DMA >> phys address. >> >I don't understand this sentence. Can you clarify it? This address looks like the MFN because I'm using mapped grant tables from domU. I've added the log and see the following: with swiotlb: [ 78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000 without swiotlb (worked fine): [ 74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: b6000000 I guess, need to figure out why we got a normal dom0 DMA address (b6000000) and why 64b1d0000 when using swiotlb. ________________________________ From: Julien Grall <julien@xen.org> Sent: Wednesday, August 11, 2021 1:30 PM To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org> Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com> Subject: Re: Disable swiotlb for Dom0 On 11/08/2021 09:49, Roman Skakun wrote: > Hi, Julien! Hi Roman, >> > I have observed your patch here: >> >https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->>T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$ <https://urldefense.com/v3/__https://patchwork.kernel.org/project/xen-devel/patch/alpine.DEB.2.21.2102161333090.3234@sstabellini-ThinkPad->T480s/__;!!GF_29dbcQIUBPA!kH5gzG1mxcIgDqMu2cVjTD3ggN9LiPN4OVinOnqrhLQrNr-mRb72udp2B5XBqZlW$>[patchwork[.]kernel[.]org] >> > >> > And I collided with the same issue, when Dom0 device trying to use >> > swiotlb fops for devices which are controlled by IOMMU. >> >>The issue Stefano reported was when the dom0 is not direct mapped. >>However... > > I applied these patches: > https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com] > <https://urldefense.com/v3/__https://github.com/torvalds/linux/commit/f5079a9a2a31607a2343e544e9182ce35b030578__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPUgqgI9K$ [github[.]com]> > https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com] > <https://urldefense.com/v3/__https://github.com/xen-project/xen/commit/d66bf122c0ab79063a607d6cf68edf5e91d17d5e__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPcdadGTB$ [github[.]com]> > to check this more pragmatically. > > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb > still used (other devices within dom0 used too), when dom0 is direct mapped: > > [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, > XENFEAT_direct_mapped, use swiotlb > > This means, that all devices are using swiotlb-xen DMA fops. > By the way, before applying this patches, dom0 always used swiotlb-xen > fops for initial domain by design. This is expected because your domain is direct mapped. > > >> Any reason to not use the stable branch for 5.10? I don't know whether >> your issue will be fixed there, but the stable branch usually contains a >> lot of bug fixes (including security one). So it is a good idea to use >> it over the first release of a kernel version. > > Yes, sure, current BSP release based on 5.10 kernel: > https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com] > <https://urldefense.com/v3/__https://github.com/xen-troops/linux/tree/v5.10/rcar-5.0.0.rc4-xt0.1__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPVkpyAnu$ [github[.]com]> > based on https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPWPOIbVJ$ [github[.]com] > <https://urldefense.com/v3/__https://github.com/renesas-rcar/linux-bsp/tree/v5.10.41/rcar-5.1.0.rc2__;!!GF_29dbcQIUBPA!mZHMTZ8iSAfPg9D1VyO4mamUWVxP7K-H26d1jSf4qMsWkB3l92muGUIJPQXMt_yV$ [github[.]com]> > BTW, I specified the wrong kernel URL in the previous massage, sorry. > >> > Issue caused in xen_swiotlb_map_page(): >> > ``` >> > dev: rcar-fcp, cap: 0, dma_mask: ffffffff, page: fffffe00180c7400, > page_to_phys: 64b1d0000, >> > xen_phys_to_dma(phys): 64b1d0000 >> > ``` >> >>I can't seem to find this printk in Linux 5.10. Did you add it yourself? > > Yes, it's my own log. Ok. Would you be able to provide more information on where the dom0 memory is allocated and the list of host RAM? > > >>This line suggests that the SWIOTLB tried to bounce the DMA buffer. In >>general, the use of the bounce buffer should be rare. So I would suggest >>to find out why this is used. >> >>Looking at the code, this suggests that one of the following check is false: >> >>/* >> * If the address happens to be in the device's DMA window, >>* we can safely return the device addr and not worry about bounce >>* buffering it. >>*/ >>if (dma_capable(dev, dev_addr, size, true) && >>!range_straddles_page_boundary(phys, size) && >>!xen_arch_need_swiotlb(dev, phys, dev_addr) && >>swiotlb_force != SWIOTLB_FORCE) >>goto done; > > I checked this earlier and saw that dma_capable(dev, dev_addr, size, > true)returns false as expected because > we got dev_addr equals 64b1d0000 and according to this expression under > dma_capable(): > > ``` > dma_addr_t end = dev_addr + size - 1; > return end <= min_not_zero(*dev->dma_mask, dev->bus_dma_limit); > ``` > As result, DMA mask more than 32bit. >> Let me start with that I agree we should disable swiotlb when we know >> the device is protected. However, from what you describe, it sounds like >> the same issue would appear if the IOMMU was disabled. > > Yes, it looks like a potential issue. This means that swiotlb should be > worked correctly, when it's needed, agreed. > But this is also potential improvement, and I presented this idea to > discuss and create some patches. You might be able to remove the Xen swiotlb but I am not sure you will be able to remove the swiotlb completely if you have a device that only supports 32-bit DMA. > >> Therefore, I think we should first find out why Linux wants to bounce >> the DMA buffer. > > We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver > wants to use only 32 bit boundary address, but that's consequence. Ok. So your device is only capable to do a 32-bit DMA. Is that correct? > I think, the main reason of using bounce buffer is MFN address, not DMA > phys address. I don't understand this sentence. Can you clarify it? Cheers, -- Julien Grall [-- Attachment #2: Type: text/html, Size: 27816 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-11 14:13 ` Roman Skakun @ 2021-08-11 15:20 ` Julien Grall 2021-08-13 9:38 ` Roman Skakun 0 siblings, 1 reply; 9+ messages in thread From: Julien Grall @ 2021-08-11 15:20 UTC (permalink / raw) To: Roman Skakun, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich Hi, On 11/08/2021 15:13, Roman Skakun wrote: >> > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb >> > still used (other devices within dom0 used too), when dom0 is direct > mapped: >> > >> > [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > >>> This means, that all devices are using swiotlb-xen DMA fops. >> > By the way, before applying this patches, dom0 always used swiotlb-xen >> > fops for initial domain by design. > >> >This is expected because your domain is direct mapped. > > May be, I don't understand right, Stefano reported the same issue when > dom0 is not direct mapped, > but I have direct mapped dom0 and problem still exists. I am not entirely sure why you think this is the same problem as Stefano. He asked to bypass the swiotlb, but AFAIK, this is not because the buffer get bounced. Instead, it is because swiotlb-xen on Arm has been relying on its RAM to be direct-mapped (GFN == MFN). With cache coloring, the memory will not be direct-mapped, hence it will be broken. > >>Ok. Would you be able to provide more information on where the dom0 >>memory is allocated and the list of host RAM? > > Host memory: > DRAM: 7.9 GiB > Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB > Bank #1: 0x500000000 - 0x57fffffff, 2 GiB > Bank #2: 0x600000000 - 0x67fffffff, 2 GiB > Bank #3: 0x700000000 - 0x77fffffff, 2 GiB > > dom0 memory map: > (XEN) Allocating 1:1 mappings totalling 2048MB for dom0: > (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB) > (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB) > (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB) Thanks! So you have some memory assigned above 4GB to dom0 as well. >>> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver >>> wants to use only 32 bit boundary address, but that's consequence. >>> >> Ok. So your device is only capable to do a 32-bit DMA. Is that correct? > > Yes. > >> > I think, the main reason of using bounce buffer is MFN address, not DMA >> > phys address. >> > >>I don't understand this sentence. Can you clarify it? > > This address looks like theMFN because I'm using mapped grant tables > from domU. > > I've added the log and see the following: > with swiotlb: > [ 78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: > fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000 > > without swiotlb (worked fine): > [ 74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: > fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma:b6000000 > > I guess, need to figure out why we got a normal dom0 DMA address > (b6000000) and why 64b1d0000 when using swiotlb. So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. swiotlb-xen on Arm will convert it to the MFN because it is not aware whether the device is behind an IOMMU. As the address is too high to be handled by the device, swiotlb will try to bounce it. I think it is correct to bounce the page but I am not sure why it can't. What the size of the DMA transaction? However, even if you disable xen-swiotlb, you are likely going to face the same issue sooner or later because the grant can be mapped anywhere in the memory of dom0 (the balloon code doesn't look to restrict where the memory can be allocated). So it is possible for the grant to be mapped in the dom0 memory above 4GB. Oleksandr is also looking to provide a safe range which would be outside of the existing RAM. So, I believe, you will have to bounce the DMA buffer unless we always force the grant/foreign mapping to be mapped below 4GB. Cheers, -- Julien Grall ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-11 15:20 ` Julien Grall @ 2021-08-13 9:38 ` Roman Skakun 2021-08-13 10:51 ` Julien Grall 0 siblings, 1 reply; 9+ messages in thread From: Roman Skakun @ 2021-08-13 9:38 UTC (permalink / raw) To: Julien Grall, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich, Roman Skakun [-- Attachment #1: Type: text/plain, Size: 12593 bytes --] Hi Julien, > So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. > > swiotlb-xen on Arm will convert it to the MFN because it is not aware > whether the device is behind an IOMMU. If I'm understand right, it seems like that swiotlb-xen is not ready to work properly in case when we retrieved MFN instead of proper GFN mapped to Dom0 memory. Maybe you know some ideas to overcome this condition? > As the address is too high to be handled by the device, swiotlb will try > to bounce it. I think it is correct to bounce the page but I am not sure > why it can't. What the size of the DMA transaction? The DMA map size is 3686400 bytes. I've added several logs to swiotlb map_single() and see: [ 151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: 64af97000, needed: 708, avail: 7fc0, stride: 2, index: 4160 It's expected because: .... [ 259.468006] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31742]=2 slots < 708 nslots. Continue... [ 259.477070] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31744]=80 slots < 708 nslots. Continue... [ 259.486229] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31746]=7e slots < 708 nslots. Continue... [ 259.495387] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31748]=7c slots < 708 nslots. Continue... [ 259.504546] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31750]=7a slots < 708 nslots. Continue... [ 259.513704] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31752]=78 slots < 708 nslots. Continue... [ 259.522863] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31754]=76 slots < 708 nslots. Continue... [ 259.532021] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31756]=74 slots < 708 nslots. Continue... [ 259.541179] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31758]=72 slots < 708 nslots. Continue... [ 259.550338] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31760]=70 slots < 708 nslots. Continue... [ 259.559496] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31762]=6e slots < 708 nslots. Continue... [ 259.568660] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31764]=6c slots < 708 nslots. Continue... [ 259.577813] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31766]=6a slots < 708 nslots. Continue... [ 259.586972] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31768]=68 slots < 708 nslots. Continue... [ 259.596130] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31770]=66 slots < 708 nslots. Continue... [ 259.605289] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31772]=64 slots < 708 nslots. Continue... [ 259.614447] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31774]=62 slots < 708 nslots. Continue... [ 259.623606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31776]=60 slots < 708 nslots. Continue... [ 259.632764] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31778]=5e slots < 708 nslots. Continue... [ 259.641922] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31780]=5c slots < 708 nslots. Continue... [ 259.651081] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31782]=5a slots < 708 nslots. Continue... [ 259.660239] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31784]=58 slots < 708 nslots. Continue... [ 259.669398] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31786]=56 slots < 708 nslots. Continue... [ 259.678563] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31788]=54 slots < 708 nslots. Continue... [ 259.687714] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31790]=52 slots < 708 nslots. Continue... [ 259.696873] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31792]=50 slots < 708 nslots. Continue... [ 259.706032] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31794]=4e slots < 708 nslots. Continue... [ 259.715190] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31796]=4c slots < 708 nslots. Continue... [ 259.724348] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31798]=4a slots < 708 nslots. Continue... [ 259.733507] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31800]=48 slots < 708 nslots. Continue... [ 259.742665] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31802]=46 slots < 708 nslots. Continue... [ 259.751824] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31804]=44 slots < 708 nslots. Continue... [ 259.760982] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31806]=42 slots < 708 nslots. Continue... [ 259.770141] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31808]=40 slots < 708 nslots. Continue... [ 259.779299] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31810]=3e slots < 708 nslots. Continue... [ 259.788466] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31812]=3c slots < 708 nslots. Continue... [ 259.797615] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31814]=3a slots < 708 nslots. Continue... [ 259.806774] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31816]=38 slots < 708 nslots. Continue... [ 259.815933] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31818]=36 slots < 708 nslots. Continue... [ 259.825091] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31820]=34 slots < 708 nslots. Continue... [ 259.834249] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31822]=32 slots < 708 nslots. Continue... [ 259.843408] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31824]=30 slots < 708 nslots. Continue... [ 259.852567] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31826]=2e slots < 708 nslots. Continue... [ 259.861725] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31828]=2c slots < 708 nslots. Continue... [ 259.870883] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31830]=2a slots < 708 nslots. Continue... [ 259.880042] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31832]=28 slots < 708 nslots. Continue... [ 259.889200] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31834]=26 slots < 708 nslots. Continue... [ 259.898365] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31836]=24 slots < 708 nslots. Continue... [ 259.907516] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31838]=22 slots < 708 nslots. Continue... [ 259.916676] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31840]=20 slots < 708 nslots. Continue... [ 259.925834] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31842]=1e slots < 708 nslots. Continue... [ 259.934992] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31844]=1c slots < 708 nslots. Continue... [ 259.944151] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31846]=1a slots < 708 nslots. Continue... [ 259.953309] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31848]=18 slots < 708 nslots. Continue... [ 259.962468] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31850]=16 slots < 708 nslots. Continue... [ 259.971626] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31852]=14 slots < 708 nslots. Continue... [ 259.980784] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31854]=12 slots < 708 nslots. Continue... [ 259.989943] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31856]=10 slots < 708 nslots. Continue... [ 259.999102] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31858]=e slots < 708 nslots. Continue... [ 260.008181] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31860]=c slots < 708 nslots. Continue... [ 260.017245] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31862]=a slots < 708 nslots. Continue... [ 260.026318] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31864]=8 slots < 708 nslots. Continue... [ 260.035389] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31866]=6 slots < 708 nslots. Continue... [ 260.044461] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31868]=4 slots < 708 nslots. Continue... [ 260.053533] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31870]=2 slots < 708 nslots. Continue... [ 260.062606] <SWIOTLB> swiotlb_tbl_map_single() io_tlb_list[31872]=80 slots < 708 nslots. Continue... .... Swiotlb did not fit requested slots because the maximum slot size equals IO_TLB_SEGSIZE=128 by default. But I think, we cannot use 64af97000 address in the swiotlb_bounce() directly. ________________________________ From: Julien Grall <julien@xen.org> Sent: Wednesday, August 11, 2021 6:20 PM To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org> Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com> Subject: Re: Disable swiotlb for Dom0 Hi, On 11/08/2021 15:13, Roman Skakun wrote: >> > Also, I added the log in xen_swiotlb_detect() and can see that swiotlb >> > still used (other devices within dom0 used too), when dom0 is direct > mapped: >> > >> > [ 1.870363] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > [ 1.878352] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > [ 1.886309] xen_swiotlb_detect() dev: rcar-fcp, >> > XENFEAT_direct_mapped, use swiotlb >> > >>> This means, that all devices are using swiotlb-xen DMA fops. >> > By the way, before applying this patches, dom0 always used swiotlb-xen >> > fops for initial domain by design. > >> >This is expected because your domain is direct mapped. > > May be, I don't understand right, Stefano reported the same issue when > dom0 is not direct mapped, > but I have direct mapped dom0 and problem still exists. I am not entirely sure why you think this is the same problem as Stefano. He asked to bypass the swiotlb, but AFAIK, this is not because the buffer get bounced. Instead, it is because swiotlb-xen on Arm has been relying on its RAM to be direct-mapped (GFN == MFN). With cache coloring, the memory will not be direct-mapped, hence it will be broken. > >>Ok. Would you be able to provide more information on where the dom0 >>memory is allocated and the list of host RAM? > > Host memory: > DRAM: 7.9 GiB > Bank #0: 0x048000000 - 0x0bfffffff, 1.9 GiB > Bank #1: 0x500000000 - 0x57fffffff, 2 GiB > Bank #2: 0x600000000 - 0x67fffffff, 2 GiB > Bank #3: 0x700000000 - 0x77fffffff, 2 GiB > > dom0 memory map: > (XEN) Allocating 1:1 mappings totalling 2048MB for dom0: > (XEN) BANK[0] 0x00000048000000-0x00000050000000 (128MB) > (XEN) BANK[1] 0x00000058000000-0x000000c0000000 (1664MB) > (XEN) BANK[2] 0x00000510000000-0x00000520000000 (256MB) Thanks! So you have some memory assigned above 4GB to dom0 as well. >>> We retrieved dev_addr(64b1d0000) + size > 32bit mask, but fcp driver >>> wants to use only 32 bit boundary address, but that's consequence. >>> >> Ok. So your device is only capable to do a 32-bit DMA. Is that correct? > > Yes. > >> > I think, the main reason of using bounce buffer is MFN address, not DMA >> > phys address. >> > >>I don't understand this sentence. Can you clarify it? > > This address looks like theMFN because I'm using mapped grant tables > from domU. > > I've added the log and see the following: > with swiotlb: > [ 78.620386] dma_map_sg_attrs() dev: rcar-du swiotlb, sg_page: > fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma: 64b1d0000 > > without swiotlb (worked fine): > [ 74.456426] dma_map_sg_attrs() dev: rcar-du direct map, sg_page: > fffffe0001b80000, page_to_phy: b6000000, xen_phys_to_dma:b6000000 > > I guess, need to figure out why we got a normal dom0 DMA address > (b6000000) and why 64b1d0000 when using swiotlb. So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. swiotlb-xen on Arm will convert it to the MFN because it is not aware whether the device is behind an IOMMU. As the address is too high to be handled by the device, swiotlb will try to bounce it. I think it is correct to bounce the page but I am not sure why it can't. What the size of the DMA transaction? However, even if you disable xen-swiotlb, you are likely going to face the same issue sooner or later because the grant can be mapped anywhere in the memory of dom0 (the balloon code doesn't look to restrict where the memory can be allocated). So it is possible for the grant to be mapped in the dom0 memory above 4GB. Oleksandr is also looking to provide a safe range which would be outside of the existing RAM. So, I believe, you will have to bounce the DMA buffer unless we always force the grant/foreign mapping to be mapped below 4GB. Cheers, -- Julien Grall [-- Attachment #2: Type: text/html, Size: 20263 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-13 9:38 ` Roman Skakun @ 2021-08-13 10:51 ` Julien Grall 2021-08-13 15:59 ` Roman Skakun 0 siblings, 1 reply; 9+ messages in thread From: Julien Grall @ 2021-08-13 10:51 UTC (permalink / raw) To: Roman Skakun, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich On 13/08/2021 10:38, Roman Skakun wrote: > Hi Julien, Hi Roman, >> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. >> >> swiotlb-xen on Arm will convert it to the MFN because it is not aware >> whether the device is behind an IOMMU. > > If I'm understand right, it seems like that swiotlb-xen is not ready to > work properly in case > when we retrieved MFN instead of proper GFN mapped to Dom0 memory. > Maybe you know some ideas to overcome this condition? swiotlb-xen work as intended. You have a DMA buffer at an address that your device cannot deal with. So it will try to bounce it. > >> As the address is too high to be handled by the device, swiotlb will try >> to bounce it. I think it is correct to bounce the page but I am not sure >> why it can't. What the size of the DMA transaction? > > The DMA map size is 3686400 bytes. So that's a 3MB buffer. I am slightly confused because in an earlier message you wrote that the memory is coming from the guest. How did you map it? > > I've added several logs to swiotlb map_single() and see: > [ 151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: > 64af97000, needed: 708, > avail: 7fc0, stride: 2, index: 4160 I am not sure how to read the logs... Are the number in hexadecimal or decimal? It might be useful if you post the diff of your changes. [...] > Swiotlb did not fit requested slots because the maximum slot size equals > IO_TLB_SEGSIZE=128 by default. Ok. So it sounds like your problem is the have a DMA buffer that is too large for the default swiotlb. Did you try to bump the value from the command line? > But I think, we cannot use64af97000 address in the swiotlb_bounce() > directly. I am not sure to understand what you mean. Can you clarify? Cheers, [...] -- Julien Grall ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Disable swiotlb for Dom0 2021-08-13 10:51 ` Julien Grall @ 2021-08-13 15:59 ` Roman Skakun 0 siblings, 0 replies; 9+ messages in thread From: Roman Skakun @ 2021-08-13 15:59 UTC (permalink / raw) To: Julien Grall, sstabellini Cc: Bertrand Marquis, Andrii Anisov, Volodymyr Babchuk, Oleksandr Tyshchenko, Oleksandr Andrushchenko, xen-devel, Roman Skakun, Jan Beulich, Roman Skakun [-- Attachment #1: Type: text/plain, Size: 4439 bytes --] >> But I think, we cannot use64af97000 address in the swiotlb_bounce() >> directly. >> >I am not sure to understand what you mean. Can you clarify? I thought, that the address 64af97000 can be used directly here: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L572 and I was confused about it. After some diggings I realized that this code is not reachable because DMA_ATTR_SKIP_CPU_SYNC is active. >> Swiotlb did not fit requested slots because the maximum slot size equals >> IO_TLB_SEGSIZE=128 by default. > >> Ok. So it sounds like your problem is the have a DMA buffer that is too >> large for the default swiotlb. Did you try to bump the value from the >> command line? Yes, you are right. I checked the implementation more detailed and figured out that swiotlb tries to find a contiguous segment that is a big enough, as shown here: https://elixir.bootlin.com/linux/v5.10/source/kernel/dma/swiotlb.c#L528, But the maximum TLB segment equals 128K by default in accordance with https://elixir.bootlin.com/linux/v5.10/source/include/linux/swiotlb.h#L25 This means that the max size of TLB segment equals: 128 * 2048 = 256K After this, I tried to increase the TLB segment size like this: diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index fbdc65782195..85f32865bb6c 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -22,7 +22,7 @@ enum swiotlb_force { * must be a power of 2. What is the appropriate value ? * The complexity of {map,unmap}_single is linearly dependent on this value. */ -#define IO_TLB_SEGSIZE 128 +#define IO_TLB_SEGSIZE 2048 /* * log of the size of each IO TLB slab. The number of slabs is command line and the problem is gone. Also, I found the article https://www.xilinx.com/support/answers/72694.html, where I believe the same issue was mentioned. Thank you so much for your time and help! ________________________________ From: Julien Grall <julien@xen.org> Sent: Friday, August 13, 2021 1:51 PM To: Roman Skakun <Roman_Skakun@epam.com>; sstabellini@kernel.org <sstabellini@kernel.org> Cc: Bertrand Marquis <bertrand.marquis@arm.com>; Andrii Anisov <Andrii_Anisov@epam.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>; Oleksandr Tyshchenko <Oleksandr_Tyshchenko@epam.com>; Oleksandr Andrushchenko <Oleksandr_Andrushchenko@epam.com>; xen-devel@lists.xenproject.org <xen-devel@lists.xenproject.org>; Roman Skakun <rm.skakun@gmail.com>; Jan Beulich <jbeulich@suse.com> Subject: Re: Disable swiotlb for Dom0 On 13/08/2021 10:38, Roman Skakun wrote: > Hi Julien, Hi Roman, >> So 0xb6000000 is most likely the GFN used to mapped the grant from the domU. >> >> swiotlb-xen on Arm will convert it to the MFN because it is not aware >> whether the device is behind an IOMMU. > > If I'm understand right, it seems like that swiotlb-xen is not ready to > work properly in case > when we retrieved MFN instead of proper GFN mapped to Dom0 memory. > Maybe you know some ideas to overcome this condition? swiotlb-xen work as intended. You have a DMA buffer at an address that your device cannot deal with. So it will try to bounce it. > >> As the address is too high to be handled by the device, swiotlb will try >> to bounce it. I think it is correct to bounce the page but I am not sure >> why it can't. What the size of the DMA transaction? > > The DMA map size is 3686400 bytes. So that's a 3MB buffer. I am slightly confused because in an earlier message you wrote that the memory is coming from the guest. How did you map it? > > I've added several logs to swiotlb map_single() and see: > [ 151.298455] <SWIOTLB> swiotlb_tbl_map_single() origin_addr: > 64af97000, needed: 708, > avail: 7fc0, stride: 2, index: 4160 I am not sure how to read the logs... Are the number in hexadecimal or decimal? It might be useful if you post the diff of your changes. [...] > Swiotlb did not fit requested slots because the maximum slot size equals > IO_TLB_SEGSIZE=128 by default. Ok. So it sounds like your problem is the have a DMA buffer that is too large for the default swiotlb. Did you try to bump the value from the command line? > But I think, we cannot use64af97000 address in the swiotlb_bounce() > directly. I am not sure to understand what you mean. Can you clarify? Cheers, [...] -- Julien Grall [-- Attachment #2: Type: text/html, Size: 13972 bytes --] ^ permalink raw reply related [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-08-13 16:00 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-10 15:38 Disable swiotlb for Dom0 Roman Skakun 2021-08-10 16:16 ` Julien Grall 2021-08-11 8:49 ` Roman Skakun 2021-08-11 10:30 ` Julien Grall 2021-08-11 14:13 ` Roman Skakun 2021-08-11 15:20 ` Julien Grall 2021-08-13 9:38 ` Roman Skakun 2021-08-13 10:51 ` Julien Grall 2021-08-13 15:59 ` Roman Skakun
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).