dma api errors with swiotlb

dma api errors with swiotlb

[Thomas Hellström (VMware)]

Hi, Cristoph.

Another DMA related question before I start to post patches in this area 
again..

Our virtual SCSI device (which BTW is fully DMA compliant) has a large 
queue depth and therefore runs out of SWIOTLB space => The scsi middle 
layer behaves nicely and asks the driver to retry the dma mapping 
operation. All fine.

The problem is that while this happens, the kernel log spits out a 
number of dma- and SWIOTLB errors, which are really harmless.

While one could probably just say go and increase the SWIOTLB size, but 
on some systems that might not be doable.

We could reduce the queue depth when SEV is active, which is probably 
one of those hacky solutions you dislike.

We could silence the dma- and swiotlb errors? At least selectively for 
some devices?

Do you have any guidance into the best solution here?

Thanks,
Thomas.

Re: dma api errors with swiotlb

[Christoph Hellwig]

A call to dma_max_mapping_size() to limit the maximum I/O size solves
that problem.  With the latest kernel that should actually be done
automatically by the SCSI midlayer for you.

Re: dma api errors with swiotlb

[Thomas Hellström (VMware)]

On 9/4/19 2:17 PM, Christoph Hellwig wrote:
> A call to dma_max_mapping_size() to limit the maximum I/O size solves
> that problem.  With the latest kernel that should actually be done
> automatically by the SCSI midlayer for you.

Hmm, OK. I guess with a sufficient queue depth and many mappings waiting 
for DMA completion, the SWIOTLB may fill up anyway...

I'll see if I can come up with something.

Thanks,

Thomas

Re: dma api errors with swiotlb

[Christoph Hellwig]

On Wed, Sep 04, 2019 at 02:54:26PM +0200, Thomas Hellström (VMware) wrote:
> On 9/4/19 2:17 PM, Christoph Hellwig wrote:
> > A call to dma_max_mapping_size() to limit the maximum I/O size solves
> > that problem.  With the latest kernel that should actually be done
> > automatically by the SCSI midlayer for you.
> 
> Hmm, OK. I guess with a sufficient queue depth and many mappings waiting for
> DMA completion, the SWIOTLB may fill up anyway...
> 
> I'll see if I can come up with something.

You are supposed to return SCSI_MLQUEUE_HOST_BUSY in that case,
which means that the kernel won't send more commands until another
one completed.

Re: dma api errors with swiotlb

[Thomas Hellström (VMware)]

On 9/4/19 2:55 PM, Christoph Hellwig wrote:
> On Wed, Sep 04, 2019 at 02:54:26PM +0200, Thomas Hellström (VMware) wrote:
>> On 9/4/19 2:17 PM, Christoph Hellwig wrote:
>>> A call to dma_max_mapping_size() to limit the maximum I/O size solves
>>> that problem.  With the latest kernel that should actually be done
>>> automatically by the SCSI midlayer for you.
>> Hmm, OK. I guess with a sufficient queue depth and many mappings waiting for
>> DMA completion, the SWIOTLB may fill up anyway...
>>
>> I'll see if I can come up with something.
> You are supposed to return SCSI_MLQUEUE_HOST_BUSY in that case,
> which means that the kernel won't send more commands until another
> one completed.

It looks like it does that, although when we send it, the SWIOTLB error 
has already occured and been printed out, and then the sequence starts 
again.

Seems like the most effective way to stop it is to decrease the queue depth.

/Thomas

Re: dma api errors with swiotlb

[Christoph Hellwig]

On Wed, Sep 04, 2019 at 03:02:14PM +0200, Thomas Hellström (VMware) wrote:
> It looks like it does that, although when we send it, the SWIOTLB error has
> already occured and been printed out, and then the sequence starts again.

Well, the only way to really find out is to try.  We also have a
backoff algorithm in the SCSI midlayer, so it should not be much of
an inpact.  If you care about the warnings they can be disabled using
DMA_ATTR_NO_WARN.  I have to admit that I'm not even sure the warnings
are all that useful, but that's something you need to take up with
the swiotlb maintainer.

Re: dma api errors with swiotlb

[Thomas Hellström (VMware)]

On 9/4/19 3:16 PM, Christoph Hellwig wrote:
> On Wed, Sep 04, 2019 at 03:02:14PM +0200, Thomas Hellström (VMware) wrote:
>> It looks like it does that, although when we send it, the SWIOTLB error has
>> already occured and been printed out, and then the sequence starts again.
> Well, the only way to really find out is to try.  We also have a
> backoff algorithm in the SCSI midlayer, so it should not be much of
> an inpact.  If you care about the warnings they can be disabled using
> DMA_ATTR_NO_WARN.  I have to admit that I'm not even sure the warnings
> are all that useful, but that's something you need to take up with
> the swiotlb maintainer.

Yes, I guess so. I actually think the warnings make sense, since while 
the scsi midlayer handles this nicely, other drivers competing for the 
space might not.

Thanks!
Thomas