kernel spew from nouveau/ swiotlb

kernel spew from nouveau/ swiotlb

[Mike Galbraith]

Greetings,

When box is earning its keep, nouveau/swiotlb grumble.. a LOT.  The
below is from master.today.

[12594.640959] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12594.693000] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12594.713787] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12594.743413] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12594.796740] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12607.000774] swiotlb_tbl_map_single: 54 callbacks suppressed
[12607.000776] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12607.347941] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
[12608.677038] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
homer:/novell/ssh # dmesg|grep 'swiotlb buffer is full'|wc -l
2052
homer:/novell/ssh # dmesg|grep 'callbacks suppressed'|wc -l
171

lib/swiotlb.c:
 573 not_found:
 574         spin_unlock_irqrestore(&io_tlb_lock, flags);
 575         if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit())
 576                 dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes)\n", size);

Does nouveau perhaps want one of those DMA_ATTR_NO_WARN thingies?

	-Mike

Re: kernel spew from nouveau/ swiotlb

[Mike Galbraith]

On Thu, 2018-05-10 at 11:10 +0200, Mike Galbraith wrote:
> Greetings,
> 
> When box is earning its keep, nouveau/swiotlb grumble.. a LOT.  The
> below is from master.today.
> 
> [12594.640959] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12594.693000] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12594.713787] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12594.743413] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12594.796740] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12607.000774] swiotlb_tbl_map_single: 54 callbacks suppressed
> [12607.000776] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12607.347941] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> [12608.677038] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> homer:/novell/ssh # dmesg|grep 'swiotlb buffer is full'|wc -l
> 2052
> homer:/novell/ssh # dmesg|grep 'callbacks suppressed'|wc -l
> 171
> 
> lib/swiotlb.c:
>  573 not_found:
>  574         spin_unlock_irqrestore(&io_tlb_lock, flags);
>  575         if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit())
>  576                 dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes)\n", size);
> 
> Does nouveau perhaps want one of those DMA_ATTR_NO_WARN thingies?

Or should ttm perhaps always use the one on hand?  (seems to work)

---
 drivers/gpu/drm/ttm/ttm_page_alloc_dma.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c
+++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c
@@ -342,7 +342,7 @@ static struct dma_page *__ttm_dma_alloc_
 	if (!d_page)
 		return NULL;
 
-	if (pool->type & IS_HUGE)
+	if (1 || pool->type & IS_HUGE)
 		attrs = DMA_ATTR_NO_WARN;
 
 	vaddr = dma_alloc_attrs(pool->dev, pool->size, &d_page->dma,

Re: [Nouveau] kernel spew from nouveau/ swiotlb

[Jerome Glisse]

> Greetings,
> 
> When box is earning its keep, nouveau/swiotlb grumble.. a LOT.  The
> below is from master.today.
> 
> [12594.640959] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12594.693000] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12594.713787] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12594.743413] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12594.796740] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12607.000774] swiotlb_tbl_map_single: 54 callbacks suppressed
> [12607.000776] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12607.347941] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> [12608.677038] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152
> bytes)
> homer:/novell/ssh # dmesg|grep 'swiotlb buffer is full'|wc -l
> 2052
> homer:/novell/ssh # dmesg|grep 'callbacks suppressed'|wc -l
> 171
> 
> lib/swiotlb.c:
>  573 not_found:
>  574         spin_unlock_irqrestore(&io_tlb_lock, flags);
>  575         if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit())
>  576                 dev_warn(hwdev, "swiotlb buffer is full (sz: %zd
>  bytes)\n", size);
> 
> Does nouveau perhaps want one of those DMA_ATTR_NO_WARN thingies?
> 

Could you bisect ? I would love to point finger upstream to the DMA
folk who made changes to that API without testing with GPU.

Cheers,
Jérôme

Re: [Nouveau] kernel spew from nouveau/ swiotlb

[Mike Galbraith]

On Thu, 2018-05-10 at 10:31 -0400, Jerome Glisse wrote:
> 
> Could you bisect ? I would love to point finger upstream to the DMA
> folk who made changes to that API without testing with GPU.

Rummaging a bit, it might be...

nouveau_bo_new()
...
ttm_dma_pool_alloc_new_pages()
  dma_alloc_attrs()
    ops->alloc() == x86_swiotlb_alloc_coherent()
    x86_swiotlb_alloc_coherent() flags |= __GFP_NOWARN;
      swiotlb_alloc_coherent(..flags)
        swiotlb_alloc_coherent(..flags) attrs = (flags & __GFP_NOWARN) ? DMA_ATTR_NO_WARN : 0;
          swiotlb_alloc_buffer(..attr)
            swiotlb_alloc_buffer(..0)  <== hm, pass zero instead of attr?
              swiotlb_tbl_map_single() gripeage

...that?

	-Mike

Re: [Nouveau] kernel spew from nouveau/ swiotlb

[Mike Galbraith]

On Thu, 2018-05-10 at 17:31 +0200, Mike Galbraith wrote:
> On Thu, 2018-05-10 at 10:31 -0400, Jerome Glisse wrote:
> > 
> > Could you bisect ? I would love to point finger upstream to the DMA
> > folk who made changes to that API without testing with GPU.
> 
> Rummaging a bit, it might be...
> 

(unsend, whack duplicate line, munge, send;)

> nouveau_bo_new()
> ...
> ttm_dma_pool_alloc_new_pages()
>   dma_alloc_attrs()
>     ops->alloc() == x86_swiotlb_alloc_coherent()
>     x86_swiotlb_alloc_coherent() flags |= __GFP_NOWARN;
>       swiotlb_alloc_coherent(..flags)
>         swiotlb_alloc_coherent(..flags) attrs = (flags & __GFP_NOWARN) ? DMA_ATTR_NO_WARN : 0;
>           swiotlb_alloc_buffer(..attrs)
*              swiotlb_tbl_map_single(..0) passed 0 vs attrs, gripeage follows

Or something like that.

Re: kernel spew from nouveau/ swiotlb

[Mike Galbraith]

On Thu, 2018-05-10 at 12:28 +0200, Mike Galbraith wrote:
> On Thu, 2018-05-10 at 11:10 +0200, Mike Galbraith wrote:
> > Greetings,
> > 
> > When box is earning its keep, nouveau/swiotlb grumble.. a LOT.  The
> > below is from master.today.
> > 
> > [12594.640959] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12594.693000] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12594.713787] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12594.743413] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12594.796740] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12607.000774] swiotlb_tbl_map_single: 54 callbacks suppressed
> > [12607.000776] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12607.347941] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > [12608.677038] nouveau 0000:01:00.0: swiotlb buffer is full (sz: 2097152 bytes)
> > homer:/novell/ssh # dmesg|grep 'swiotlb buffer is full'|wc -l
> > 2052
> > homer:/novell/ssh # dmesg|grep 'callbacks suppressed'|wc -l
> > 171
> > 
> > lib/swiotlb.c:
> >  573 not_found:
> >  574         spin_unlock_irqrestore(&io_tlb_lock, flags);
> >  575         if (!(attrs & DMA_ATTR_NO_WARN) && printk_ratelimit())
> >  576                 dev_warn(hwdev, "swiotlb buffer is full (sz: %zd bytes)\n", size);
> > 
> > Does nouveau perhaps want one of those DMA_ATTR_NO_WARN thingies?
> 
> Or should ttm perhaps always use the one on hand?  (seems to work)

No it didn't, I just didn't wait long enough for spew to start...

> ---
>  drivers/gpu/drm/ttm/ttm_page_alloc_dma.c |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> --- a/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c
> +++ b/drivers/gpu/drm/ttm/ttm_page_alloc_dma.c
> @@ -342,7 +342,7 @@ static struct dma_page *__ttm_dma_alloc_
>  	if (!d_page)
>  		return NULL;
>  
> -	if (pool->type & IS_HUGE)
> +	if (1 || pool->type & IS_HUGE)
>  		attrs = DMA_ATTR_NO_WARN;
>  
>  	vaddr = dma_alloc_attrs(pool->dev, pool->size, &d_page->dma,

While IS_HUGE is indeed false on my box, it just doesn't matter,
because when we get to either the old or the new alloc(), it calls
swiotlb_alloc_buffer(), which drops attrs passed to it on the floor,
making it unlikely that alloc() caller wishes are granted.

	-Mike

[patch] swiotlb: fix ignored DMA_ATTR_NO_WARN request

[Mike Galbraith]

In the trace below, swiotlb_alloc() is called with __GFP_NOWARN, it ors
attrs with DMA_ATTR_NO_WARN and passes it to swiotlb_alloc_buffer(),
which does NOT pass it on to swiotlb_tbl_map_single(), leading to an
ever repeating warning that the caller of swiotlb_alloc() explicitly
asked to be squelched.  Pass the caller's request for silence onward.

 Xorg-3170  [006] ....   963.866098: swiotlb_alloc+0x1d/0x1a0: gfp & __GFP_NOWARN
 Xorg-3170  [006] ....   963.866101: <stack trace>
 => ttm_dma_populate+0x250/0x310 [ttm]
 => ttm_tt_populate+0x28/0x70 [ttm]
 => ttm_tt_bind+0x26/0x60 [ttm]
 => ttm_bo_handle_move_mem+0x51a/0x580 [ttm]
 => ttm_bo_validate+0xfa/0x110 [ttm]
 => ttm_bo_init_reserved+0x296/0x450 [ttm]
 => ttm_bo_init+0x73/0xd0 [ttm]
 => nouveau_bo_new+0x3eb/0x5c0 [nouveau]
 => nouveau_gem_new+0x66/0x110 [nouveau]
 => nouveau_gem_ioctl_new+0x48/0xc0 [nouveau]
 => drm_ioctl_kernel+0x66/0xb0 [drm]
 => drm_ioctl+0x28d/0x340 [drm]
 => nouveau_drm_ioctl+0x50/0xb0 [nouveau]
 => do_vfs_ioctl+0x92/0x5e0
 => ksys_ioctl+0x3a/0x70
 => __x64_sys_ioctl+0x16/0x20
 => do_syscall_64+0x5b/0x180
 => entry_SYSCALL_64_after_hwframe+0x44/0xa9
 Xorg-3170  [006] ....   963.866917: swiotlb_tbl_map_single+0x29b/0x2d0: swiotlb buffer is full (sz: 2097152 bytes)

Signed-off-by: Mike Galbraith <efault@gmx.de>
---
 lib/swiotlb.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -714,7 +714,7 @@ swiotlb_alloc_buffer(struct device *dev,
 
 	phys_addr = swiotlb_tbl_map_single(dev,
 			__phys_to_dma(dev, io_tlb_start),
-			0, size, DMA_FROM_DEVICE, 0);
+			0, size, DMA_FROM_DEVICE, attrs);
 	if (phys_addr == SWIOTLB_MAP_ERROR)
 		goto out_warn;
 

Re: [patch] swiotlb: fix ignored DMA_ATTR_NO_WARN request

[Mike Galbraith]

To conclude to this snail like thread (/me=walking wounded), with the
v4.16.8 hunk below, traces showing that swiotlb_alloc_coherent() was
being asked to not bother warning started showing up after the box had
been flogged for a while.

Whatever finally happens with swiotlb (seems to be in flux), other
folks meeting annoying gripeage can find bandaids in the interim.

The End

v4.16.8 !DMA_DIRECT_OPS
Xorg-3105  [001] ....  2156.711471: swiotlb_alloc_coherent+0xa7/0x1e0: yup
Xorg-3105  [001] ....  2156.711497: <stack trace>
 => ttm_dma_populate+0x23c/0x310 [ttm]
 => ttm_tt_bind+0x31/0x60 [ttm]
 => ttm_bo_handle_move_mem+0x527/0x580 [ttm]
 => ttm_bo_validate+0xfb/0x110 [ttm]
 => ttm_bo_init_reserved+0x289/0x450 [ttm]
 => ttm_bo_init+0x77/0xd0 [ttm]
 => nouveau_bo_new+0x3fc/0x5e0 [nouveau]
 => nouveau_gem_new+0x66/0x110 [nouveau]
 => nouveau_gem_ioctl_new+0x48/0xc0 [nouveau]
 => drm_ioctl_kernel+0x66/0xb0 [drm]
 => drm_ioctl+0x2a4/0x360 [drm]
 => nouveau_drm_ioctl+0x50/0xb0 [nouveau]
 => do_vfs_ioctl+0x92/0x5e0
 => SyS_ioctl+0x3b/0x70
 => do_syscall_64+0x74/0x1a0
 => entry_SYSCALL_64_after_hwframe+0x3d/0xa2

--- a/arch/x86/kernel/pci-swiotlb.c
+++ b/arch/x86/kernel/pci-swiotlb.c
@@ -28,10 +28,8 @@ void *x86_swiotlb_alloc_coherent(struct
 	 * swiotlb_alloc_coherent() will print a warning when the DMA
 	 * memory allocation ultimately failed.
 	 */
-	flags |= __GFP_NOWARN;
-
-	vaddr = dma_generic_alloc_coherent(hwdev, size, dma_handle, flags,
-					   attrs);
+	vaddr = dma_generic_alloc_coherent(hwdev, size, dma_handle,
+					   flags | __GFP_NOWARN, attrs);
 	if (vaddr)
 		return vaddr;