multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

Hello

Phase 2 of the testing for EDR100 and IB/srp means testing multipath fail-over and recovery during controller reboots.
Running 40 parallel tasks to 40 mpath devices will consistently land up in a stack dump when calling swiotlb_alloc_coherent, during reconnect attempts waiting for the controller to return.

Most of the time the system will recover paths when the controller returns but will flood the logs during the reconnects.

I am wondering we should disable this as its supposed to be a warning so looking for opinions here.

Notes
-----
This is initiated from mlx5_core

The dump stack seems to have been pulled in with this commit - e2172d8fd500a51a3845bc2294cdf4feaa388dab

Specifically 
  swiotlb: Warn on allocation failure in swiotlb_alloc_coherent()

From: Joerg Roedel <jroedel-l3A5Bk7waGM@public.gmane.org>

Print a warning when all allocation tries have been failed
and the function is about to return NULL. This prepares for
calling the function with __GFP_NOWARN to suppress
allocation failure warnings before all fall-backs have
failed.

Looking at the code here:

We call __get_free_pages(flags, order) and we cannot DMA to the ConnectX-4 and we land up in 

err_warn:
        pr_warn("swiotlb: coherent allocation failed for device %s size=%zu\n",
                dev_name(hwdev), size);
        dump_stack();

        return NULL;
}




Jun  8 10:12:52 jumpclient kernel: device-mapper: multipath: Failing path 68:240.
Jun  8 10:12:52 jumpclient kernel: device-mapper: multipath: Failing path 69:16.
Jun  8 10:12:52 jumpclient kernel: device-mapper: multipath: Failing path 68:160.
Jun  8 10:12:52 jumpclient kernel: device-mapper: multipath: Failing path 68:224.
Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed for device 0000:08:00.1 size=266240
Jun  8 10:12:52 jumpclient kernel: CPU: 4 PID: 22125 Comm: kworker/4:1 Tainted: G          I     4.7.0-rc1.bart+ #1
Jun  8 10:12:52 jumpclient kernel: Hardware name: HP ProLiant DL380 G7, BIOS P67 08/16/2015
Jun  8 10:12:52 jumpclient kernel: Workqueue: events_long srp_reconnect_work [scsi_transport_srp]
Jun  8 10:12:52 jumpclient kernel: 0000000000000286 000000009fe8136d ffff8801027ffa10 ffffffff8134514f
Jun  8 10:12:52 jumpclient kernel: 0000000000041000 ffff88060ba1f0a0 ffff8801027ffa50 ffffffff8136eab9
Jun  8 10:12:52 jumpclient kernel: ffffffff00000007 00000000024082c0 ffff88060ba1f0a0 0000000000041000
Jun  8 10:12:52 jumpclient kernel: Call Trace:
Jun  8 10:12:52 jumpclient kernel: [<ffffffff8134514f>] dump_stack+0x63/0x84
Jun  8 10:12:52 jumpclient kernel: [<ffffffff8136eab9>] swiotlb_alloc_coherent+0x149/0x160
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810655e3>] x86_swiotlb_alloc_coherent+0x43/0x50
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa01e5874>] mlx5_dma_zalloc_coherent_node+0xa4/0x100 [mlx5_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa01e5bdd>] mlx5_buf_alloc_node+0x4d/0xc0 [mlx5_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa01e5c64>] mlx5_buf_alloc+0x14/0x20 [mlx5_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0187db5>] create_kernel_qp.isra.46+0x285/0x7a0 [mlx5_ib]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa018934b>] ? mlx5_ib_create_qp+0xdb/0x490 [mlx5_ib]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0188ede>] create_qp_common+0xc0e/0xdc0 [mlx5_ib]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa018934b>] ? mlx5_ib_create_qp+0xdb/0x490 [mlx5_ib]
Jun  8 10:12:52 jumpclient kernel: [<ffffffff811f32a8>] ? kmem_cache_alloc_trace+0x1f8/0x210
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0189373>] mlx5_ib_create_qp+0x103/0x490 [mlx5_ib]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0425cf9>] ? ib_alloc_cq+0x89/0x160 [ib_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0425cf9>] ? ib_alloc_cq+0x89/0x160 [ib_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa042583f>] ib_create_qp+0x3f/0x240 [ib_core]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa06650e3>] srp_create_ch_ib+0x133/0x530 [ib_srp]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0662833>] ? srp_finish_req+0x93/0xb0 [ib_srp]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa06669aa>] srp_rport_reconnect+0xea/0x1d0 [ib_srp]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa06272a3>] srp_reconnect_rport+0xc3/0x230 [scsi_transport_srp]
Jun  8 10:12:52 jumpclient kernel: [<ffffffffa0627454>] srp_reconnect_work+0x44/0xd4 [scsi_transport_srp]
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810a2e82>] process_one_work+0x152/0x400
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810a3775>] worker_thread+0x125/0x4b0
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810a3650>] ? rescuer_thread+0x380/0x380
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810a92b8>] kthread+0xd8/0xf0
Jun  8 10:12:52 jumpclient kernel: [<ffffffff816c43bf>] ret_from_fork+0x1f/0x40
Jun  8 10:12:52 jumpclient kernel: [<ffffffff810a91e0>] ? kthread_park+0x60/0x60
Jun  8 10:12:52 jumpclient kernel: scsi host2: reconnect attempt 2 failed (-12)
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Bart Van Assche]

On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
> Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed for device 0000:08:00.1 size=266240

Hello,

I think the above means that the coherent memory allocation succeeded 
but that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can 
someone from Mellanox tell us whether or not it would be safe to set 
coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?

Thanks,

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
> Cc: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, June 13, 2016 2:32:53 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> > Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb buffer
> > is full (sz: 266240 bytes)
> > Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed for
> > device 0000:08:00.1 size=266240
> 
> Hello,
> 
> I think the above means that the coherent memory allocation succeeded
> but that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can
> someone from Mellanox tell us whether or not it would be safe to set
> coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?
> 
> Thanks,
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Leon

Are you able to answer Bart's question here re the possibility of using DMA_BIT_MASK(64)
For now I have disabled this warning and dump stack while I continue to test.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

On Sun, Jun 12, 2016 at 11:32:53PM -0700, Bart Van Assche wrote:
> On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> >Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
> >Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed for device 0000:08:00.1 size=266240
> 
> Hello,
> 
> I think the above means that the coherent memory allocation succeeded but
> that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can someone
> from Mellanox tell us whether or not it would be safe to set
> coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?

Bart and Laurence,
We are actually doing it For mlx5 driver.

926 static int mlx5_pci_init(struct mlx5_core_dev *dev, struct  mlx5_priv *priv)
<...>
961         err = set_dma_caps(pdev);

187 static int set_dma_caps(struct pci_dev *pdev)
<...>
201         err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
202         if (err) {
203                 dev_warn(&pdev->dev,
204                          "Warning: couldn't set 64-bit consistent PCI DMA mask\n");
205                 err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32));
206                 if (err) {
207                         dev_err(&pdev->dev,
208                                 "Can't set consistent PCI DMA mask, aborting\n");
209                         return err;
210                 }
211         }

118 static inline int pci_set_consistent_dma_mask(struct pci_dev *dev,u64 mask)
119 {
120         return dma_set_coherent_mask(&dev->dev, mask);
121 }

> 
> Thanks,
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, June 13, 2016 10:07:47 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On Sun, Jun 12, 2016 at 11:32:53PM -0700, Bart Van Assche wrote:
> > On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> > >Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb buffer
> > >is full (sz: 266240 bytes)
> > >Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed for
> > >device 0000:08:00.1 size=266240
> > 
> > Hello,
> > 
> > I think the above means that the coherent memory allocation succeeded but
> > that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can someone
> > from Mellanox tell us whether or not it would be safe to set
> > coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?
> 
> Bart and Laurence,
> We are actually doing it For mlx5 driver.
> 
> 926 static int mlx5_pci_init(struct mlx5_core_dev *dev, struct  mlx5_priv
> *priv)
> <...>
> 961         err = set_dma_caps(pdev);
> 
> 187 static int set_dma_caps(struct pci_dev *pdev)
> <...>
> 201         err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
> 202         if (err) {
> 203                 dev_warn(&pdev->dev,
> 204                          "Warning: couldn't set 64-bit consistent PCI DMA
> mask\n");
> 205                 err = pci_set_consistent_dma_mask(pdev,
> DMA_BIT_MASK(32));
> 206                 if (err) {
> 207                         dev_err(&pdev->dev,
> 208                                 "Can't set consistent PCI DMA mask,
> aborting\n");
> 209                         return err;
> 210                 }
> 211         }
> 
> 118 static inline int pci_set_consistent_dma_mask(struct pci_dev *dev,u64
> mask)
> 119 {
> 120         return dma_set_coherent_mask(&dev->dev, mask);
> 121 }
> 
> > 
> > Thanks,
> > 
> > Bart.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Hi Leon,

OK I see it now

static int set_dma_caps(struct pci_dev *pdev)
{
        int err;

        err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
        if (err) {

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, June 13, 2016 10:19:57 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Monday, June 13, 2016 10:07:47 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > On Sun, Jun 12, 2016 at 11:32:53PM -0700, Bart Van Assche wrote:
> > > On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> > > >Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb
> > > >buffer
> > > >is full (sz: 266240 bytes)
> > > >Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed
> > > >for
> > > >device 0000:08:00.1 size=266240
> > > 
> > > Hello,
> > > 
> > > I think the above means that the coherent memory allocation succeeded but
> > > that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can someone
> > > from Mellanox tell us whether or not it would be safe to set
> > > coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?
> > 
> > Bart and Laurence,
> > We are actually doing it For mlx5 driver.
> > 
> > 926 static int mlx5_pci_init(struct mlx5_core_dev *dev, struct  mlx5_priv
> > *priv)
> > <...>
> > 961         err = set_dma_caps(pdev);
> > 
> > 187 static int set_dma_caps(struct pci_dev *pdev)
> > <...>
> > 201         err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
> > 202         if (err) {
> > 203                 dev_warn(&pdev->dev,
> > 204                          "Warning: couldn't set 64-bit consistent PCI
> > DMA
> > mask\n");
> > 205                 err = pci_set_consistent_dma_mask(pdev,
> > DMA_BIT_MASK(32));
> > 206                 if (err) {
> > 207                         dev_err(&pdev->dev,
> > 208                                 "Can't set consistent PCI DMA mask,
> > aborting\n");
> > 209                         return err;
> > 210                 }
> > 211         }
> > 
> > 118 static inline int pci_set_consistent_dma_mask(struct pci_dev *dev,u64
> > mask)
> > 119 {
> > 120         return dma_set_coherent_mask(&dev->dev, mask);
> > 121 }
> > 
> > > 
> > > Thanks,
> > > 
> > > Bart.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> Hi Leon,
> 
> OK I see it now
> 
> static int set_dma_caps(struct pci_dev *pdev)
> {
>         int err;
> 
>         err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
>         if (err) {
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Replying to my own email.
Leon, what is the implication of the mapping failure.
Its only in the reconnect stack when I am restarting controllers with the messaging and stack dump masked I still see the failure but it seems transparent in that all the paths come back.

[ 1595.167812] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1595.379133] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1595.460627] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1598.121096] scsi host1: reconnect attempt 3 failed (-48)
[ 1608.187869] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1615.911705] scsi host1: reconnect attempt 4 failed (-12)
[ 1641.446017] scsi host1: ib_srp: Got failed path rec status -110
[ 1641.482947] scsi host1: ib_srp: Path record query failed
[ 1641.513454] scsi host1: reconnect attempt 5 failed (-110)
[ 1662.330883] scsi host1: ib_srp: Got failed path rec status -110
[ 1662.361224] scsi host1: ib_srp: Path record query failed
[ 1662.390768] scsi host1: reconnect attempt 6 failed (-110)
[ 1683.892311] scsi host1: ib_srp: Got failed path rec status -110
[ 1683.922653] scsi host1: ib_srp: Path record query failed
[ 1683.952717] scsi host1: reconnect attempt 7 failed (-110)
SM port is up

Entering MASTER state

[ 1705.254048] scsi host1:   REJ reason 0x8
[ 1705.274869] scsi host1: reconnect attempt 8 failed (-104)
[ 1723.264914] scsi host1:   REJ reason 0x8
[ 1723.285193] scsi host1: reconnect attempt 9 failed (-104)
[ 1743.658091] scsi host1:   REJ reason 0x8
[ 1743.678562] scsi host1: reconnect attempt 10 failed (-104)
[ 1761.911512] scsi host1:   REJ reason 0x8
[ 1761.932006] scsi host1: reconnect attempt 11 failed (-104)
[ 1782.209020] scsi host1: ib_srp: reconnect succeeded

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, June 13, 2016 11:22:19 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas"
> > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Monday, June 13, 2016 10:19:57 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > > <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Monday, June 13, 2016 10:07:47 AM
> > > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > > swiotlb_alloc_coherent()
> > > 
> > > On Sun, Jun 12, 2016 at 11:32:53PM -0700, Bart Van Assche wrote:
> > > > On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> > > > >Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb
> > > > >buffer
> > > > >is full (sz: 266240 bytes)
> > > > >Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation failed
> > > > >for
> > > > >device 0000:08:00.1 size=266240
> > > > 
> > > > Hello,
> > > > 
> > > > I think the above means that the coherent memory allocation succeeded
> > > > but
> > > > that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can
> > > > someone
> > > > from Mellanox tell us whether or not it would be safe to set
> > > > coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?
> > > 
> > > Bart and Laurence,
> > > We are actually doing it For mlx5 driver.
> > > 
> > > 926 static int mlx5_pci_init(struct mlx5_core_dev *dev, struct  mlx5_priv
> > > *priv)
> > > <...>
> > > 961         err = set_dma_caps(pdev);
> > > 
> > > 187 static int set_dma_caps(struct pci_dev *pdev)
> > > <...>
> > > 201         err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
> > > 202         if (err) {
> > > 203                 dev_warn(&pdev->dev,
> > > 204                          "Warning: couldn't set 64-bit consistent PCI
> > > DMA
> > > mask\n");
> > > 205                 err = pci_set_consistent_dma_mask(pdev,
> > > DMA_BIT_MASK(32));
> > > 206                 if (err) {
> > > 207                         dev_err(&pdev->dev,
> > > 208                                 "Can't set consistent PCI DMA mask,
> > > aborting\n");
> > > 209                         return err;
> > > 210                 }
> > > 211         }
> > > 
> > > 118 static inline int pci_set_consistent_dma_mask(struct pci_dev *dev,u64
> > > mask)
> > > 119 {
> > > 120         return dma_set_coherent_mask(&dev->dev, mask);
> > > 121 }
> > > 
> > > > 
> > > > Thanks,
> > > > 
> > > > Bart.
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > > in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > Hi Leon,
> > 
> > OK I see it now
> > 
> > static int set_dma_caps(struct pci_dev *pdev)
> > {
> >         int err;
> > 
> >         err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
> >         if (err) {
> > 
> > Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Replying to my own email.
> Leon, what is the implication of the mapping failure.
> Its only in the reconnect stack when I am restarting controllers with the
> messaging and stack dump masked I still see the failure but it seems
> transparent in that all the paths come back.
> 
> [ 1595.167812] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1595.379133] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1595.460627] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1598.121096] scsi host1: reconnect attempt 3 failed (-48)
> [ 1608.187869] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1615.911705] scsi host1: reconnect attempt 4 failed (-12)
> [ 1641.446017] scsi host1: ib_srp: Got failed path rec status -110
> [ 1641.482947] scsi host1: ib_srp: Path record query failed
> [ 1641.513454] scsi host1: reconnect attempt 5 failed (-110)
> [ 1662.330883] scsi host1: ib_srp: Got failed path rec status -110
> [ 1662.361224] scsi host1: ib_srp: Path record query failed
> [ 1662.390768] scsi host1: reconnect attempt 6 failed (-110)
> [ 1683.892311] scsi host1: ib_srp: Got failed path rec status -110
> [ 1683.922653] scsi host1: ib_srp: Path record query failed
> [ 1683.952717] scsi host1: reconnect attempt 7 failed (-110)
> SM port is up
> 
> Entering MASTER state
> 
> [ 1705.254048] scsi host1:   REJ reason 0x8
> [ 1705.274869] scsi host1: reconnect attempt 8 failed (-104)
> [ 1723.264914] scsi host1:   REJ reason 0x8
> [ 1723.285193] scsi host1: reconnect attempt 9 failed (-104)
> [ 1743.658091] scsi host1:   REJ reason 0x8
> [ 1743.678562] scsi host1: reconnect attempt 10 failed (-104)
> [ 1761.911512] scsi host1:   REJ reason 0x8
> [ 1761.932006] scsi host1: reconnect attempt 11 failed (-104)
> [ 1782.209020] scsi host1: ib_srp: reconnect succeeded
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Leon

Calling relationship looks like this.

swiotlb_alloc_coherent 
     u64 dma_mask = DMA_BIT_MASK(32);
          
      May get overwritten here, I assume as a 64 bit mask right ?

       if (hwdev && hwdev->coherent_dma_mask)
		dma_mask = hwdev->coherent_dma_mask;          ***** Is this now DMA_BIT_MASK(64)


We fail here and then try single but we fail so we see the warning
 ret = (void *)__get_free_pages(flags, order);

It seems to be a non-critical event in that when we are able to reconnect we do.
I am missing how we recover here. I assume on next try we pass.
I will add some instrumentation to figure this out.


if (!ret) {
		/*
		 * We are either out of memory or the device can't DMA to
		 * GFP_DMA memory; fall back on map_single(), which
		 * will grab memory from the lowest available address range.
		 */
		phys_addr_t paddr = map_single(hwdev, 0, size, DMA_FROM_DEVICE);
		if (paddr == SWIOTLB_MAP_ERROR)
			goto err_warn;

		ret = phys_to_virt(paddr);
		dev_addr = phys_to_dma(hwdev, paddr);

		/* Confirm address can be DMA'd by device */
		if (dev_addr + size - 1 > dma_mask) {
			printk("hwdev DMA mask = 0x%016Lx, dev_addr = 0x%016Lx\n",
			       (unsigned long long)dma_mask,
			       (unsigned long long)dev_addr);

			/* DMA_TO_DEVICE to avoid memcpy in unmap_single */
			swiotlb_tbl_unmap_single(hwdev, paddr,
						 size, DMA_TO_DEVICE);
			goto err_warn;
		}
	}
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Monday, June 13, 2016 6:30:04 PM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas"
> > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Monday, June 13, 2016 11:22:19 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> > > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas"
> > > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Monday, June 13, 2016 10:19:57 AM
> > > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > > swiotlb_alloc_coherent()
> > > 
> > > 
> > > 
> > > ----- Original Message -----
> > > > From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > > Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, "Laurence Oberman"
> > > > <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > Sent: Monday, June 13, 2016 10:07:47 AM
> > > > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack
> > > > in
> > > > swiotlb_alloc_coherent()
> > > > 
> > > > On Sun, Jun 12, 2016 at 11:32:53PM -0700, Bart Van Assche wrote:
> > > > > On 06/12/2016 03:40 PM, Laurence Oberman wrote:
> > > > > >Jun  8 10:12:52 jumpclient kernel: mlx5_core 0000:08:00.1: swiotlb
> > > > > >buffer
> > > > > >is full (sz: 266240 bytes)
> > > > > >Jun  8 10:12:52 jumpclient kernel: swiotlb: coherent allocation
> > > > > >failed
> > > > > >for
> > > > > >device 0000:08:00.1 size=266240
> > > > > 
> > > > > Hello,
> > > > > 
> > > > > I think the above means that the coherent memory allocation succeeded
> > > > > but
> > > > > that the test dev_addr + size - 1 <= DMA_BIT_MASK(32) failed. Can
> > > > > someone
> > > > > from Mellanox tell us whether or not it would be safe to set
> > > > > coherent_dma_mask to DMA_BIT_MASK(64) for the mlx4 and mlx5 drivers?
> > > > 
> > > > Bart and Laurence,
> > > > We are actually doing it For mlx5 driver.
> > > > 
> > > > 926 static int mlx5_pci_init(struct mlx5_core_dev *dev, struct
> > > > mlx5_priv
> > > > *priv)
> > > > <...>
> > > > 961         err = set_dma_caps(pdev);
> > > > 
> > > > 187 static int set_dma_caps(struct pci_dev *pdev)
> > > > <...>
> > > > 201         err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
> > > > 202         if (err) {
> > > > 203                 dev_warn(&pdev->dev,
> > > > 204                          "Warning: couldn't set 64-bit consistent
> > > > PCI
> > > > DMA
> > > > mask\n");
> > > > 205                 err = pci_set_consistent_dma_mask(pdev,
> > > > DMA_BIT_MASK(32));
> > > > 206                 if (err) {
> > > > 207                         dev_err(&pdev->dev,
> > > > 208                                 "Can't set consistent PCI DMA mask,
> > > > aborting\n");
> > > > 209                         return err;
> > > > 210                 }
> > > > 211         }
> > > > 
> > > > 118 static inline int pci_set_consistent_dma_mask(struct pci_dev
> > > > *dev,u64
> > > > mask)
> > > > 119 {
> > > > 120         return dma_set_coherent_mask(&dev->dev, mask);
> > > > 121 }
> > > > 
> > > > > 
> > > > > Thanks,
> > > > > 
> > > > > Bart.
> > > > > --
> > > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > > > in
> > > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > > Hi Leon,
> > > 
> > > OK I see it now
> > > 
> > > static int set_dma_caps(struct pci_dev *pdev)
> > > {
> > >         int err;
> > > 
> > >         err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
> > >         if (err) {
> > > 
> > > Thanks
> > > Laurence
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > 
> > Replying to my own email.
> > Leon, what is the implication of the mapping failure.
> > Its only in the reconnect stack when I am restarting controllers with the
> > messaging and stack dump masked I still see the failure but it seems
> > transparent in that all the paths come back.
> > 
> > [ 1595.167812] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> > bytes)
> > [ 1595.379133] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> > bytes)
> > [ 1595.460627] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> > bytes)
> > [ 1598.121096] scsi host1: reconnect attempt 3 failed (-48)
> > [ 1608.187869] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240
> > bytes)
> > [ 1615.911705] scsi host1: reconnect attempt 4 failed (-12)
> > [ 1641.446017] scsi host1: ib_srp: Got failed path rec status -110
> > [ 1641.482947] scsi host1: ib_srp: Path record query failed
> > [ 1641.513454] scsi host1: reconnect attempt 5 failed (-110)
> > [ 1662.330883] scsi host1: ib_srp: Got failed path rec status -110
> > [ 1662.361224] scsi host1: ib_srp: Path record query failed
> > [ 1662.390768] scsi host1: reconnect attempt 6 failed (-110)
> > [ 1683.892311] scsi host1: ib_srp: Got failed path rec status -110
> > [ 1683.922653] scsi host1: ib_srp: Path record query failed
> > [ 1683.952717] scsi host1: reconnect attempt 7 failed (-110)
> > SM port is up
> > 
> > Entering MASTER state
> > 
> > [ 1705.254048] scsi host1:   REJ reason 0x8
> > [ 1705.274869] scsi host1: reconnect attempt 8 failed (-104)
> > [ 1723.264914] scsi host1:   REJ reason 0x8
> > [ 1723.285193] scsi host1: reconnect attempt 9 failed (-104)
> > [ 1743.658091] scsi host1:   REJ reason 0x8
> > [ 1743.678562] scsi host1: reconnect attempt 10 failed (-104)
> > [ 1761.911512] scsi host1:   REJ reason 0x8
> > [ 1761.932006] scsi host1: reconnect attempt 11 failed (-104)
> > [ 1782.209020] scsi host1: ib_srp: reconnect succeeded
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Hi Leon
> 
> Calling relationship looks like this.
> 
> swiotlb_alloc_coherent
>      u64 dma_mask = DMA_BIT_MASK(32);
>           
>       May get overwritten here, I assume as a 64 bit mask right ?
> 
>        if (hwdev && hwdev->coherent_dma_mask)
> 		dma_mask = hwdev->coherent_dma_mask;          ***** Is this now
> 		DMA_BIT_MASK(64)
> 
> 
> We fail here and then try single but we fail so we see the warning
>  ret = (void *)__get_free_pages(flags, order);
> 
> It seems to be a non-critical event in that when we are able to reconnect we
> do.
> I am missing how we recover here. I assume on next try we pass.
> I will add some instrumentation to figure this out.
> 
> 
> if (!ret) {
> 		/*
> 		 * We are either out of memory or the device can't DMA to
> 		 * GFP_DMA memory; fall back on map_single(), which
> 		 * will grab memory from the lowest available address range.
> 		 */
> 		phys_addr_t paddr = map_single(hwdev, 0, size, DMA_FROM_DEVICE);
> 		if (paddr == SWIOTLB_MAP_ERROR)
> 			goto err_warn;
> 
> 		ret = phys_to_virt(paddr);
> 		dev_addr = phys_to_dma(hwdev, paddr);
> 
> 		/* Confirm address can be DMA'd by device */
> 		if (dev_addr + size - 1 > dma_mask) {
> 			printk("hwdev DMA mask = 0x%016Lx, dev_addr = 0x%016Lx\n",
> 			       (unsigned long long)dma_mask,
> 			       (unsigned long long)dev_addr);
> 
> 			/* DMA_TO_DEVICE to avoid memcpy in unmap_single */
> 			swiotlb_tbl_unmap_single(hwdev, paddr,
> 						 size, DMA_TO_DEVICE);
> 			goto err_warn;
> 		}
> 	}
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Tracing

        do {
                while (iommu_is_span_boundary(index, nslots, offset_slots,
                                              max_slots)) {
                        index += stride;
                        if (index >= io_tlb_nslabs)
                                index = 0;
                        if (index == wrap)
                                goto not_found; ------------------------> take this jump
                }


[  986.838508] RHDEBUG: wrap=56 index=56

not_found:
        spin_unlock_irqrestore(&io_tlb_lock, flags);
        if (printk_ratelimit()) {
                dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes)\n", size);
                printk("RHDEBUG: wrap=%u index=%u\n",wrap,index);
        }
        return SWIOTLB_MAP_ERROR;

[  990.484449] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff

Then

We take this branch to get to err_warn label.

                phys_addr_t paddr = map_single(hwdev, 0, size, DMA_FROM_DEVICE);
                if (paddr == SWIOTLB_MAP_ERROR) {
                        printk("RHDEBUG: SWIOTLB_MAP_ERROR %llx\n",paddr);
                        goto err_warn;
                }
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Bart Van Assche]

On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> Tracing [ ... ]

Hello Laurence,

Can you try to set the kernel command line parameter swiotlb to a value 
that is a multiple of its default (64MB), reboot and see whether that 
helps? See also Documentation/kernel-parameters.txt in the kernel tree.

Thanks,

Bart.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 484 bytes --]

On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> >Tracing [ ... ]
> 
> Hello Laurence,
> 
> Can you try to set the kernel command line parameter swiotlb to a value that
> is a multiple of its default (64MB), reboot and see whether that helps? See
> also Documentation/kernel-parameters.txt in the kernel tree.

Do you think that 64MB is not enough for this scenario?

> 
> Thanks,
> 
> Bart.
> 
> 

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Bart Van Assche]

On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
>> On 06/14/2016 03:56 AM, Laurence Oberman wrote:
>>> Tracing [ ... ]
>>
>> Can you try to set the kernel command line parameter swiotlb to a value that
>> is a multiple of its default (64MB), reboot and see whether that helps? See
>> also Documentation/kernel-parameters.txt in the kernel tree.
>
> Do you think that 64MB is not enough for this scenario?

Hello Leon,

Does this mean that you think that the "swiotlb buffer is full" messages 
reported by Laurence could be caused by something else than hitting the 
swiotlb limit?

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, June 14, 2016 8:25:15 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> > On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> >> On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> >>> Tracing [ ... ]
> >>
> >> Can you try to set the kernel command line parameter swiotlb to a value
> >> that
> >> is a multiple of its default (64MB), reboot and see whether that helps?
> >> See
> >> also Documentation/kernel-parameters.txt in the kernel tree.
> >
> > Do you think that 64MB is not enough for this scenario?
> 
> Hello Leon,
> 
> Does this mean that you think that the "swiotlb buffer is full" messages
> reported by Laurence could be caused by something else than hitting the
> swiotlb limit?
> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Trying this today and will report back.
Its interesting to me that I am burying this host in I/O testing, both reads and writes and never have issues with the swiotlb space.
Its only on the reconnect that this happens when I test the path loss and controller reboots.

Back later with results of the tuning of the swiotlb buffers.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1192 bytes --]

On Tue, Jun 14, 2016 at 02:25:15PM +0200, Bart Van Assche wrote:
> On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> >On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> >>On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> >>>Tracing [ ... ]
> >>
> >>Can you try to set the kernel command line parameter swiotlb to a value that
> >>is a multiple of its default (64MB), reboot and see whether that helps? See
> >>also Documentation/kernel-parameters.txt in the kernel tree.
> >
> >Do you think that 64MB is not enough for this scenario?
> 
> Hello Leon,
> 
> Does this mean that you think that the "swiotlb buffer is full" messages
> reported by Laurence could be caused by something else than hitting the
> swiotlb limit?

I don't have enough knowledge in that area to answer if message "swiotlb
buffer is full" is correctly explain current situation and I will be
glad to get input from you about it.

Thanks.

> 
> Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, June 14, 2016 9:15:52 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On Tue, Jun 14, 2016 at 02:25:15PM +0200, Bart Van Assche wrote:
> > On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> > >On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> > >>On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> > >>>Tracing [ ... ]
> > >>
> > >>Can you try to set the kernel command line parameter swiotlb to a value
> > >>that
> > >>is a multiple of its default (64MB), reboot and see whether that helps?
> > >>See
> > >>also Documentation/kernel-parameters.txt in the kernel tree.
> > >
> > >Do you think that 64MB is not enough for this scenario?
> > 
> > Hello Leon,
> > 
> > Does this mean that you think that the "swiotlb buffer is full" messages
> > reported by Laurence could be caused by something else than hitting the
> > swiotlb limit?
> 
> I don't have enough knowledge in that area to answer if message "swiotlb
> buffer is full" is correctly explain current situation and I will be
> glad to get input from you about it.
> 
> Thanks.
> 
> > 
> > Bart.
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Hello Bart and Leon

Set to swiotlb=4, I no longer have the swiotlb issue. 
Its no longer taking the error path so I confirm that increasing the swiotlb slab count solves this issue.

BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=4

I am in the process of preparing a Doc patch for IB/srp and IB/srpt where I have captured all the tuning work I have done with Bart.
I will add this to the Doc patch priot to sending it.

Many Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, June 14, 2016 9:57:14 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Cc: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Yishai Hadas"
> > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Tuesday, June 14, 2016 9:15:52 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > On Tue, Jun 14, 2016 at 02:25:15PM +0200, Bart Van Assche wrote:
> > > On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> > > >On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> > > >>On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> > > >>>Tracing [ ... ]
> > > >>
> > > >>Can you try to set the kernel command line parameter swiotlb to a value
> > > >>that
> > > >>is a multiple of its default (64MB), reboot and see whether that helps?
> > > >>See
> > > >>also Documentation/kernel-parameters.txt in the kernel tree.
> > > >
> > > >Do you think that 64MB is not enough for this scenario?
> > > 
> > > Hello Leon,
> > > 
> > > Does this mean that you think that the "swiotlb buffer is full" messages
> > > reported by Laurence could be caused by something else than hitting the
> > > swiotlb limit?
> > 
> > I don't have enough knowledge in that area to answer if message "swiotlb
> > buffer is full" is correctly explain current situation and I will be
> > glad to get input from you about it.
> > 
> > Thanks.
> > 
> > > 
> > > Bart.
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> Hello Bart and Leon
> 
> Set to swiotlb=4, I no longer have the swiotlb issue.
> Its no longer taking the error path so I confirm that increasing the swiotlb
> slab count solves this issue.
> 
> BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro
> crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap
> console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=4
> 
> I am in the process of preparing a Doc patch for IB/srp and IB/srpt where I
> have captured all the tuning work I have done with Bart.
> I will add this to the Doc patch priot to sending it.
> 
> Many Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Seems I may have spoken too soon here. :(

I increased the ib_srp params to max values and I see it again
options ib_srp cmd_sg_entries=256 indirect_sg_entries=2048

[ 1344.111449] sd 2:0:0:27: rejecting I/O to offline device
[ 1345.815918] scsi host2: reconnect attempt 3 failed (-12)
[ 1363.744587] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
[ 1363.781476] RHDEBUG: wrap=56 index=56
[ 1363.802643] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1365.361446] scsi host2: reconnect attempt 4 failed (-12)
[ 1375.468042] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
[ 1375.506694] RHDEBUG: wrap=56 index=56
[ 1375.526821] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1375.820655] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240 bytes)
[ 1375.857679] RHDEBUG: wrap=56 index=56
[ 1375.877973] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1384.729797] scsi host2: reconnect attempt 5 failed (-24)
[ 1406.100966] scsi host2: ib_srp: Got failed path rec status -110
[ 1406.133565] scsi host2: ib_srp: Path record query failed
[ 1406.163410] scsi host2: reconnect attempt 6 failed (-110)
[ 1424.229937] scsi host2:   REJ reason 0x8
[ 1424.251503] scsi host2: reconnect attempt 7 failed (-104)
[ 1443.257233] scsi host2:   REJ reason 0x8
[ 1443.280298] scsi host2: reconnect attempt 8 failed (-104)
[ 1466.691538] scsi host2:   REJ reason 0x8
[ 1466.712377] scsi host2: reconnect attempt 9 failed (-104)

I will increase the swiotlb to 8 from 4 and see what happens, but maybe I just got lucky prior.
Will update the thread when its consistent either way.

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Tuesday, June 14, 2016 1:40:35 PM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> > Cc: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>, "Yishai Hadas"
> > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Tuesday, June 14, 2016 9:57:14 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Leon Romanovsky" <leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org>
> > > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > Cc: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, "Yishai Hadas"
> > > <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Tuesday, June 14, 2016 9:15:52 AM
> > > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > > swiotlb_alloc_coherent()
> > > 
> > > On Tue, Jun 14, 2016 at 02:25:15PM +0200, Bart Van Assche wrote:
> > > > On 06/14/2016 02:08 PM, Leon Romanovsky wrote:
> > > > >On Tue, Jun 14, 2016 at 11:24:28AM +0200, Bart Van Assche wrote:
> > > > >>On 06/14/2016 03:56 AM, Laurence Oberman wrote:
> > > > >>>Tracing [ ... ]
> > > > >>
> > > > >>Can you try to set the kernel command line parameter swiotlb to a
> > > > >>value
> > > > >>that
> > > > >>is a multiple of its default (64MB), reboot and see whether that
> > > > >>helps?
> > > > >>See
> > > > >>also Documentation/kernel-parameters.txt in the kernel tree.
> > > > >
> > > > >Do you think that 64MB is not enough for this scenario?
> > > > 
> > > > Hello Leon,
> > > > 
> > > > Does this mean that you think that the "swiotlb buffer is full"
> > > > messages
> > > > reported by Laurence could be caused by something else than hitting the
> > > > swiotlb limit?
> > > 
> > > I don't have enough knowledge in that area to answer if message "swiotlb
> > > buffer is full" is correctly explain current situation and I will be
> > > glad to get input from you about it.
> > > 
> > > Thanks.
> > > 
> > > > 
> > > > Bart.
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe linux-rdma"
> > > > in
> > > > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > 
> > Hello Bart and Leon
> > 
> > Set to swiotlb=4, I no longer have the swiotlb issue.
> > Its no longer taking the error path so I confirm that increasing the
> > swiotlb
> > slab count solves this issue.
> > 
> > BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro
> > crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap
> > console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=4
> > 
> > I am in the process of preparing a Doc patch for IB/srp and IB/srpt where I
> > have captured all the tuning work I have done with Bart.
> > I will add this to the Doc patch priot to sending it.
> > 
> > Many Thanks
> > Laurence
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> Seems I may have spoken too soon here. :(
> 
> I increased the ib_srp params to max values and I see it again
> options ib_srp cmd_sg_entries=256 indirect_sg_entries=2048
> 
> [ 1344.111449] sd 2:0:0:27: rejecting I/O to offline device
> [ 1345.815918] scsi host2: reconnect attempt 3 failed (-12)
> [ 1363.744587] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1363.781476] RHDEBUG: wrap=56 index=56
> [ 1363.802643] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
> [ 1365.361446] scsi host2: reconnect attempt 4 failed (-12)
> [ 1375.468042] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1375.506694] RHDEBUG: wrap=56 index=56
> [ 1375.526821] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
> [ 1375.820655] mlx5_core 0000:08:00.1: swiotlb buffer is full (sz: 266240
> bytes)
> [ 1375.857679] RHDEBUG: wrap=56 index=56
> [ 1375.877973] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
> [ 1384.729797] scsi host2: reconnect attempt 5 failed (-24)
> [ 1406.100966] scsi host2: ib_srp: Got failed path rec status -110
> [ 1406.133565] scsi host2: ib_srp: Path record query failed
> [ 1406.163410] scsi host2: reconnect attempt 6 failed (-110)
> [ 1424.229937] scsi host2:   REJ reason 0x8
> [ 1424.251503] scsi host2: reconnect attempt 7 failed (-104)
> [ 1443.257233] scsi host2:   REJ reason 0x8
> [ 1443.280298] scsi host2: reconnect attempt 8 failed (-104)
> [ 1466.691538] scsi host2:   REJ reason 0x8
> [ 1466.712377] scsi host2: reconnect attempt 9 failed (-104)
> 
> I will increase the swiotlb to 8 from 4 and see what happens, but maybe I
> just got lucky prior.
> Will update the thread when its consistent either way.
> 
> Thanks
> Laurence
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

This may be a data point here
After each change I have rebooted the host as its required.
I am at swiotlb=16 and after the first reboot with maxed out tuning I had no alerts.
On the second controller restart without a system reboot I got them again.

Again, I never see these other than when I am in  the reconnect loop, and they seem to be non-intrusive as each time I recover fully.

When I first changed to 4 and had not increased the ib_srp paramaters I had two restarts with no messages so that was what led me to report that this seems to have worked.
I can see now that this was not the case and already mentioned, the claim that the change fixed this was wrong.
Apologies for that.

I am continuing to research and debug now.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Bart Van Assche]

On 06/14/2016 08:41 PM, Laurence Oberman wrote:
> This may be a data point here
> After each change I have rebooted the host as its required.
> I am at swiotlb=16 and after the first reboot with maxed out tuning I had no alerts.
> On the second controller restart without a system reboot I got them again.
>
> Again, I never see these other than when I am in  the reconnect loop, and they seem to be non-intrusive as each time I recover fully.
>
> When I first changed to 4 and had not increased the ib_srp paramaters I had two restarts with no messages so that was what led me to report that this seems to have worked.
> I can see now that this was not the case and already mentioned, the claim that the change fixed this was wrong.
> Apologies for that.
>
> I am continuing to research and debug now.

Hello Laurence,

In the kernel source tree I found the following:

 From include/linux/swiotlb.h:

#define IO_TLB_SHIFT 11

 From lib/swiotlb.c:

#define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
[ ... ]
#define IO_TLB_DEFAULT_SIZE (64UL<<20)
[ ... ]
static int __init
setup_io_tlb_npages(char *str)
{
	if (isdigit(*str)) {
		io_tlb_nslabs = simple_strtoul(str, &str, 0);
		/* avoid tail segment of size < IO_TLB_SEGSIZE */
		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
	}
         [ ... ]
}
early_param("swiotlb", setup_io_tlb_npages);
[ ... ]
void  __init
swiotlb_init(int verbose)
{
	size_t default_size = IO_TLB_DEFAULT_SIZE;
	[ ... ]

	if (!io_tlb_nslabs) {
		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
	}
	[ ... ]
}

I think this means that the swiotlb parameter has to be set to a value 
above 32768 to increase the number of swiotlb buffers above the default.

Bart.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, June 15, 2016 3:40:23 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On 06/14/2016 08:41 PM, Laurence Oberman wrote:
> > This may be a data point here
> > After each change I have rebooted the host as its required.
> > I am at swiotlb=16 and after the first reboot with maxed out tuning I had
> > no alerts.
> > On the second controller restart without a system reboot I got them again.
> >
> > Again, I never see these other than when I am in  the reconnect loop, and
> > they seem to be non-intrusive as each time I recover fully.
> >
> > When I first changed to 4 and had not increased the ib_srp paramaters I had
> > two restarts with no messages so that was what led me to report that this
> > seems to have worked.
> > I can see now that this was not the case and already mentioned, the claim
> > that the change fixed this was wrong.
> > Apologies for that.
> >
> > I am continuing to research and debug now.
> 
> Hello Laurence,
> 
> In the kernel source tree I found the following:
> 
>  From include/linux/swiotlb.h:
> 
> #define IO_TLB_SHIFT 11
> 
>  From lib/swiotlb.c:
> 
> #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
> [ ... ]
> #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> [ ... ]
> static int __init
> setup_io_tlb_npages(char *str)
> {
> 	if (isdigit(*str)) {
> 		io_tlb_nslabs = simple_strtoul(str, &str, 0);
> 		/* avoid tail segment of size < IO_TLB_SEGSIZE */
> 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
> 	}
>          [ ... ]
> }
> early_param("swiotlb", setup_io_tlb_npages);
> [ ... ]
> void  __init
> swiotlb_init(int verbose)
> {
> 	size_t default_size = IO_TLB_DEFAULT_SIZE;
> 	[ ... ]
> 
> 	if (!io_tlb_nslabs) {
> 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
> 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
> 	}
> 	[ ... ]
> }
> 
> I think this means that the swiotlb parameter has to be set to a value
> above 32768 to increase the number of swiotlb buffers above the default.
> 
> Bart.
> 
> 
> 
Hello Bart

I will try that.
When I looked at the code I saw it being set to 1 as a default, and read the Doc comments as a slab count so figured its an int and would be calculated as n x slabs.
I guess that's another Document update needed for kernel docs.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, June 15, 2016 6:57:59 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>, leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org
> > Cc: "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, June 15, 2016 3:40:23 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > On 06/14/2016 08:41 PM, Laurence Oberman wrote:
> > > This may be a data point here
> > > After each change I have rebooted the host as its required.
> > > I am at swiotlb=16 and after the first reboot with maxed out tuning I had
> > > no alerts.
> > > On the second controller restart without a system reboot I got them
> > > again.
> > >
> > > Again, I never see these other than when I am in  the reconnect loop, and
> > > they seem to be non-intrusive as each time I recover fully.
> > >
> > > When I first changed to 4 and had not increased the ib_srp paramaters I
> > > had
> > > two restarts with no messages so that was what led me to report that this
> > > seems to have worked.
> > > I can see now that this was not the case and already mentioned, the claim
> > > that the change fixed this was wrong.
> > > Apologies for that.
> > >
> > > I am continuing to research and debug now.
> > 
> > Hello Laurence,
> > 
> > In the kernel source tree I found the following:
> > 
> >  From include/linux/swiotlb.h:
> > 
> > #define IO_TLB_SHIFT 11
> > 
> >  From lib/swiotlb.c:
> > 
> > #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
> > [ ... ]
> > #define IO_TLB_DEFAULT_SIZE (64UL<<20)
> > [ ... ]
> > static int __init
> > setup_io_tlb_npages(char *str)
> > {
> > 	if (isdigit(*str)) {
> > 		io_tlb_nslabs = simple_strtoul(str, &str, 0);
> > 		/* avoid tail segment of size < IO_TLB_SEGSIZE */
> > 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
> > 	}
> >          [ ... ]
> > }
> > early_param("swiotlb", setup_io_tlb_npages);
> > [ ... ]
> > void  __init
> > swiotlb_init(int verbose)
> > {
> > 	size_t default_size = IO_TLB_DEFAULT_SIZE;
> > 	[ ... ]
> > 
> > 	if (!io_tlb_nslabs) {
> > 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
> > 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
> > 	}
> > 	[ ... ]
> > }
> > 
> > I think this means that the swiotlb parameter has to be set to a value
> > above 32768 to increase the number of swiotlb buffers above the default.
> > 
> > Bart.
> > 
> > 
> > 
> Hello Bart
> 
> I will try that.
> When I looked at the code I saw it being set to 1 as a default, and read the
> Doc comments as a slab count so figured its an int and would be calculated
> as n x slabs.
> I guess that's another Document update needed for kernel docs.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

We are missing something here

I set it to double 266240 given the message below where its says we are full at 266240.

I will instrument kernel and see what it gets set to to make sure we see whats happening.

BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=532480

dmesg | grep -i swio
[    0.000000] Linux version 4.7.0-rc1.bart.swiotlb+ (loberman@jumptest1) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #5 SMP Mon Jun 13 21:09:50 EDT 2016
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=532480
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.7.0-rc1.bart.swiotlb+ root=/dev/mapper/rhel-root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap console=ttyS1,115200n8 scsi_mod.use_blk_mq=1 swiotlb=532480

[    4.663794] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)           **** Note

[    4.917998] usb usb1: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ ehci_hcd
[    4.954666] usb usb2: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ uhci_hcd
[    5.083110] usb usb3: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ uhci_hcd
[    5.111634] usb usb4: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ uhci_hcd
[    5.240089] usb usb5: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ uhci_hcd
[    5.373986] usb usb6: Manufacturer: Linux 4.7.0-rc1.bart.swiotlb+ uhci_hcd
[ 1403.045092] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1403.045095] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1403.075632] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1403.075634] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1404.091624] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1404.091627] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1404.207057] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1404.207060] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1404.673154] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1404.673157] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1414.717610] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1414.779978] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1415.016524] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1415.073408] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1415.143262] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1415.204337] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff


[ 1414.717610] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1414.758355] RHDEBUG: wrap=56 index=56
[ 1414.779978] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1415.016524] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1415.053908] RHDEBUG: wrap=56 index=56
[ 1415.073408] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
[ 1415.143262] mlx5_core 0000:08:00.0: swiotlb buffer is full (sz: 266240 bytes)
[ 1415.183465] RHDEBUG: wrap=56 index=56
[ 1415.204337] RHDEBUG: SWIOTLB_MAP_ERROR ffffffffffffffff
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Bart Van Assche]

On 06/15/2016 02:02 PM, Laurence Oberman wrote:
> We are missing something here

The source code excerpts in my previous e-mail came from the latest 
Linux kernel (v4.7-rc3). Maybe older kernels behave in a different way.

BTW, did you run into the "swiotlb buffer is full" error messages while 
testing 4MB I/O? Have you already considered to reduce the memory that 
is needed for RDMA queues by reducing the queue depth? I ran my SRP 
tests with default swiotlb buffer size and with the following in 
srp_daemon.conf:

a queue_size=32,max_cmd_per_lun=32,max_sect=8192

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, June 15, 2016 8:51:18 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> On 06/15/2016 02:02 PM, Laurence Oberman wrote:
> > We are missing something here
> 
> The source code excerpts in my previous e-mail came from the latest
> Linux kernel (v4.7-rc3). Maybe older kernels behave in a different way.
> 
> BTW, did you run into the "swiotlb buffer is full" error messages while
> testing 4MB I/O? Have you already considered to reduce the memory that
> is needed for RDMA queues by reducing the queue depth? I ran my SRP
> tests with default swiotlb buffer size and with the following in
> srp_daemon.conf:
> 
> a queue_size=32,max_cmd_per_lun=32,max_sect=8192
> 
> Bart.
> 

Hi Bart 

All my testing here has been 4MB I/O while restarting controllers.
This is a customer requirement to be doing large sequential 4MB, buffered and O_DIRECT.

I have 128, but will reduce to 32 and test it.

My config is as follows per customer requirements.

[root@jumptest1 ~]# cat /etc/ddn/srp_daemon.conf 
a      queue_size=128,max_cmd_per_lun=128,max_sect=8192

Interestingly, I have absolutely no issue with ib_srp and testing all types of I/O on this very large array.
Its rock solid upstream now since all the fixes we have now in ib_srp.

The swiotlb seems, as already mentioned, to be only in reconnects and does NOT affect behavior of regular I/O.

I will make this observation in the patch I will be sending for ib_srp* Documentation

Thanks!!
Laurence

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, June 15, 2016 9:19:32 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, June 15, 2016 8:51:18 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > On 06/15/2016 02:02 PM, Laurence Oberman wrote:
> > > We are missing something here
> > 
> > The source code excerpts in my previous e-mail came from the latest
> > Linux kernel (v4.7-rc3). Maybe older kernels behave in a different way.
> > 
> > BTW, did you run into the "swiotlb buffer is full" error messages while
> > testing 4MB I/O? Have you already considered to reduce the memory that
> > is needed for RDMA queues by reducing the queue depth? I ran my SRP
> > tests with default swiotlb buffer size and with the following in
> > srp_daemon.conf:
> > 
> > a queue_size=32,max_cmd_per_lun=32,max_sect=8192
> > 
> > Bart.
> > 
> 
> Hi Bart
> 
> All my testing here has been 4MB I/O while restarting controllers.
> This is a customer requirement to be doing large sequential 4MB, buffered and
> O_DIRECT.
> 
> I have 128, but will reduce to 32 and test it.
> 
> My config is as follows per customer requirements.
> 
> [root@jumptest1 ~]# cat /etc/ddn/srp_daemon.conf
> a      queue_size=128,max_cmd_per_lun=128,max_sect=8192
> 
> Interestingly, I have absolutely no issue with ib_srp and testing all types
> of I/O on this very large array.
> Its rock solid upstream now since all the fixes we have now in ib_srp.
> 
> The swiotlb seems, as already mentioned, to be only in reconnects and does
> NOT affect behavior of regular I/O.
> 
> I will make this observation in the patch I will be sending for ib_srp*
> Documentation
> 
> Thanks!!
> Laurence
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Hi Bart,

I should add that it does not even affect the reconnects, they increment each time until the SM sees the controller come back and the controller is ready to receive the reconnects.
All paths are successfully recovered.

I am thinking we should remove the dump_stack and leave the message in as a warning
Customers seeing messages as warnings will be less concerend than seeing kernel stack dumps.

Let me know if you want me to submit a patch for that.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()

[Laurence Oberman]

----- Original Message -----
> From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>, linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Sent: Wednesday, June 15, 2016 9:23:58 AM
> Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in swiotlb_alloc_coherent()
> 
> 
> 
> ----- Original Message -----
> > From: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > To: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > Sent: Wednesday, June 15, 2016 9:19:32 AM
> > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > swiotlb_alloc_coherent()
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Bart Van Assche" <bart.vanassche-XdAiOPVOjttBDgjK7y7TUQ@public.gmane.org>
> > > To: "Laurence Oberman" <loberman-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> > > Cc: leon-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org, "Yishai Hadas" <yishaih-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>,
> > > linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > > Sent: Wednesday, June 15, 2016 8:51:18 AM
> > > Subject: Re: multipath IB/srp fail-over testing lands up in dump stack in
> > > swiotlb_alloc_coherent()
> > > 
> > > On 06/15/2016 02:02 PM, Laurence Oberman wrote:
> > > > We are missing something here
> > > 
> > > The source code excerpts in my previous e-mail came from the latest
> > > Linux kernel (v4.7-rc3). Maybe older kernels behave in a different way.
> > > 
> > > BTW, did you run into the "swiotlb buffer is full" error messages while
> > > testing 4MB I/O? Have you already considered to reduce the memory that
> > > is needed for RDMA queues by reducing the queue depth? I ran my SRP
> > > tests with default swiotlb buffer size and with the following in
> > > srp_daemon.conf:
> > > 
> > > a queue_size=32,max_cmd_per_lun=32,max_sect=8192
> > > 
> > > Bart.
> > > 
> > 
> > Hi Bart
> > 
> > All my testing here has been 4MB I/O while restarting controllers.
> > This is a customer requirement to be doing large sequential 4MB, buffered
> > and
> > O_DIRECT.
> > 
> > I have 128, but will reduce to 32 and test it.
> > 
> > My config is as follows per customer requirements.
> > 
> > [root@jumptest1 ~]# cat /etc/ddn/srp_daemon.conf
> > a      queue_size=128,max_cmd_per_lun=128,max_sect=8192
> > 
> > Interestingly, I have absolutely no issue with ib_srp and testing all types
> > of I/O on this very large array.
> > Its rock solid upstream now since all the fixes we have now in ib_srp.
> > 
> > The swiotlb seems, as already mentioned, to be only in reconnects and does
> > NOT affect behavior of regular I/O.
> > 
> > I will make this observation in the patch I will be sending for ib_srp*
> > Documentation
> > 
> > Thanks!!
> > Laurence
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> Hi Bart,
> 
> I should add that it does not even affect the reconnects, they increment each
> time until the SM sees the controller come back and the controller is ready
> to receive the reconnects.
> All paths are successfully recovered.
> 
> I am thinking we should remove the dump_stack and leave the message in as a
> warning
> Customers seeing messages as warnings will be less concerend than seeing
> kernel stack dumps.
> 
> Let me know if you want me to submit a patch for that.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hello Bart

Confirming reducing queue_depth max to 32 prevents the swiotlb errors.

I will document this

Thanks
Laurence
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html