RE: How to resolve an issue in swiotlb environment?

[Biju Das <biju.das@bp.renesas.com>]

Hi All,

Any update on the below issue. I am seeing similar issue on RZ/G2M board with Linux version 5.2.0-rc3.

root@hihope-rz-g2m:~# [   35.414177] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci-hcd
[   35.449402] usb-storage 2-1:1.0: USB Mass Storage device detected
[   35.455915] scsi host0: usb-storage 2-1:1.0
[   36.482585] scsi 0:0:0:0: Direct-Access     SanDisk  Extreme          0001 PQ: 0 ANSI: 6
[   36.491260] sd 0:0:0:0: [sda] 125045424 512-byte logical blocks: (64.0 GB/59.6 GiB)
[   36.499823] sd 0:0:0:0: [sda] Write Protect is off
[   36.505474] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   36.518074]  sda: sda1
[   36.523163] sd 0:0:0:0: [sda] Attached SCSI disk

root@hihope-rz-g2m:~# mkdir -p /tmp/rmnt/sda1
root@hihope-rz-g2m:~# mount -t auto /dev/sda1 /tmp/rmnt/sda1
root@hihope-rz-g2m:~# dd if=/dev/urandom of=/tmp/sda1-random bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.187696 s, 55.9 MB/s
root@hihope-rz-g2m:~# cp /tmp/sda1-random /tmp/rmnt/sda1/sda1-random
root@hihope-rz-g2m:~# [  218.861212] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  218.871885] xhci-hcd ee000000.usb: overflow 0x000000067430b000+1003520 of DMA mask ffffffff bus mask 0
[  218.881233] WARNING: CPU: 0 PID: 258 at kernel/dma/direct.c:43 report_addr+0x38/0xa8
[  218.888974] Modules linked in: renesas_usb3 usb_dmac phy_rcar_gen3_usb3
[  218.895594] CPU: 0 PID: 258 Comm: usb-storage Not tainted 5.2.0-rc3-00017-gc80b083-dirty #5
[  218.903940] Hardware name: HopeRun HiHope RZ/G2M with sub board (DT)
[  218.910291] pstate: 40000005 (nZcv daif -PAN -UAO)
[  218.915078] pc : report_addr+0x38/0xa8
[  218.918821] lr : report_addr+0xa0/0xa8
[  218.922564] sp : ffff0000125fb970
[  218.925872] x29: ffff0000125fb970 x28: 0000000000000000
[  218.931180] x27: 0000000000000000 x26: 000000001f020280
[  218.936487] x25: ffff8006394a82a8 x24: 0000000000000000
[  218.941794] x23: 0000000000000001 x22: 0000000000000000
[  218.947101] x21: 00000000000f5000 x20: ffff000011309000
[  218.952408] x19: ffff80063a600010 x18: ffffffffffffffff
[  218.957714] x17: 0000000000000000 x16: 0000000000000000
[  218.963023] x15: ffff0000113096c8 x14: 4d4420666f203032
[  218.968331] x13: 35333030312b3030 x12: 3062303334373630
[  218.973638] x11: 3030303030307830 x10: ffff000011309f20
[  218.978945] x9 : ffff0000112e3018 x8 : 0000000000000123
[  218.984252] x7 : 0000000000000005 x6 : ffff80063b578180
[  218.989559] x5 : ffff80063b578180 x4 : 0000000000000000
[  218.994865] x3 : ffff80063b57ef10 x2 : eed25f279b69f300
[  219.000172] x1 : eed25f279b69f300 x0 : 0000000000000000
[  219.005481] Call trace:
[  219.007923]  report_addr+0x38/0xa8
[  219.011321]  dma_direct_map_page+0x148/0x158
[  219.015586]  dma_direct_map_sg+0x78/0xe0
[  219.019510]  usb_hcd_map_urb_for_dma+0x2fc/0x468
[  219.024124]  xhci_map_urb_for_dma+0x54/0x68
[  219.028303]  usb_hcd_submit_urb+0x88/0x968
[  219.032394]  usb_submit_urb+0x3b0/0x570
[  219.036226]  usb_sg_wait+0x98/0x158
[  219.039711]  usb_stor_bulk_transfer_sglist.part.3+0x94/0x128
[  219.045366]  usb_stor_bulk_srb+0x48/0x88
[  219.049283]  usb_stor_Bulk_transport+0x10c/0x390
[  219.053896]  usb_stor_invoke_transport+0x3c/0x500
[  219.058595]  usb_stor_transparent_scsi_command+0xc/0x18
[  219.063816]  usb_stor_control_thread+0x1c4/0x260
[  219.068431]  kthread+0x124/0x128
[  219.071660]  ret_from_fork+0x10/0x18
[  219.075229] ---[ end trace dd9ef2a6b7fef860 ]---
[  219.080087] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.090810] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.101510] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.112209] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.122901] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.133591] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.144283] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.154973] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  219.165674] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.861717] swiotlb_tbl_map_single: 67451 callbacks suppressed
[  223.861721] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.878249] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.888940] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.899630] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.910318] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.921005] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.931695] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.942387] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.953077] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  223.963765] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.865664] swiotlb_tbl_map_single: 70409 callbacks suppressed
[  228.865668] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.882188] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.892878] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.903567] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.914256] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.924944] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.935636] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.946326] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.957015] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)
[  228.967705] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 1003520 bytes), total 32768 (slots), used 1088 (slots)

Regards,
Biju

> Subject: RE: How to resolve an issue in swiotlb environment?
> 
> Hi Christoph,
> 
> I think we should continue to discuss on this email thread instead of the fixed
> DMA-API.txt patch [1]
> 
> [1]
> https://marc.info/?t=155989412200001&r=1&w=2
> 
> > From: Yoshihiro Shimoda, Sent: Monday, June 3, 2019 3:42 PM
> >
> > Hi linux-block and iommu mailing lists,
> >
> > I have an issue that a USB SSD with xHCI on R-Car H3 causes "swiotlb is full"
> like below.
> >
> >     [   36.745286] xhci-hcd ee000000.usb: swiotlb buffer is full (sz: 524288
> bytes), total 32768 (slots), used 1338 (slots)
> >
> > I have investigated this issue by using git bisect, and then I found the
> following commit:
> >
> > ---
> > commit 09324d32d2a0843e66652a087da6f77924358e62
> > Author: Christoph Hellwig <hch@lst.de>
> > Date:   Tue May 21 09:01:41 2019 +0200
> >
> >     block: force an unlimited segment size on queues with a virt
> > boundary
> > ---
> 
> Thank you for your comment on other email thread [2] like below:
> ---
> Turns out it isn't as simple as I thought, as there doesn't seem to be an easy
> way to get to the struct device used for DMA mapping from USB drivers.  I'll
> need to think a bit more how to handle that best.
> ---
> 
> [2]
> https://marc.info/?l=linux-doc&m=155989651620473&w=2
> 
> I'm not sure this is a correct way, but the issue disappears if I applied a patch
> below to USB storage driver. Especially, WARNING happened on
> blk_queue_max_segment_size().
> Maybe we need to expand the argument "struct device *" of
> blk_queue_virt_boundary() to call dma_max_mapping_size()?
> ---
> diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
> index 59190d8..fa37b39 100644
> --- a/drivers/usb/storage/scsiglue.c
> +++ b/drivers/usb/storage/scsiglue.c
> @@ -28,6 +28,7 @@
>   * status of a command.
>   */
> 
> +#include <linux/dma-mapping.h>
>  #include <linux/module.h>
>  #include <linux/mutex.h>
> 
> @@ -83,6 +84,15 @@ static int slave_alloc (struct scsi_device *sdev)
>  	maxp = usb_maxpacket(us->pusb_dev, us->recv_bulk_pipe, 0);
>  	blk_queue_virt_boundary(sdev->request_queue, maxp - 1);
> 
> +{
> +	struct device *dev = us->pusb_dev->bus->controller;
> +
> +	dev_info(dev, "%s: size = %zu\n", __func__,
> dma_max_mapping_size(dev));
> +	blk_queue_max_segment_size(sdev->request_queue,
> +				   dma_max_mapping_size(dev));
> +}
> +
> +
>  	/*
>  	 * Some host controllers may have alignment requirements.
>  	 * We'll play it safe by requiring 512-byte alignment always.
> ---
> 
> Best regards,
> Yoshihiro Shimoda