[PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[John Pittman]

When scsi_dma_map() fails by returning a sges_left value less than
zero, the amount of logging can be extremely high.  In a recent
end-user environment, 1200 messages per second were being sent to
the log buffer.  This eventually overwhelmed the system and it
stalled.  As the messages are almost all identical, use
pr_err_ratelimited() instead of sdev_printk() to print the
scsi_dma_map failure messages.

Signed-off-by: John Pittman <jpittman@redhat.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 511726f92d9a..ac9ccde6f3f8 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -2594,9 +2594,8 @@ _base_check_pcie_native_sgl(struct MPT3SAS_ADAPTER *ioc,
 	/* Get the SG list pointer and info. */
 	sges_left = scsi_dma_map(scmd);
 	if (sges_left < 0) {
-		sdev_printk(KERN_ERR, scmd->device,
-			"scsi_dma_map failed: request for %d bytes!\n",
-			scsi_bufflen(scmd));
+		pr_err_ratelimited("sd %s: scsi_dma_map failed: request for %d bytes!\n",
+			dev_name(&scmd->device->sdev_gendev), scsi_bufflen(scmd));
 		return 1;
 	}
 
@@ -2706,9 +2705,8 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER *ioc,
 	sg_scmd = scsi_sglist(scmd);
 	sges_left = scsi_dma_map(scmd);
 	if (sges_left < 0) {
-		sdev_printk(KERN_ERR, scmd->device,
-		 "scsi_dma_map failed: request for %d bytes!\n",
-		 scsi_bufflen(scmd));
+		pr_err_ratelimited("sd %s: scsi_dma_map failed: request for %d bytes!\n",
+			dev_name(&scmd->device->sdev_gendev), scsi_bufflen(scmd));
 		return -ENOMEM;
 	}
 
@@ -2854,9 +2852,8 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER *ioc,
 	sg_scmd = scsi_sglist(scmd);
 	sges_left = scsi_dma_map(scmd);
 	if (sges_left < 0) {
-		sdev_printk(KERN_ERR, scmd->device,
-			"scsi_dma_map failed: request for %d bytes!\n",
-			scsi_bufflen(scmd));
+		pr_err_ratelimited("sd %s: scsi_dma_map failed: request for %d bytes!\n",
+			dev_name(&scmd->device->sdev_gendev), scsi_bufflen(scmd));
 		return -ENOMEM;
 	}
 
-- 
2.17.2

Re: [PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[Laurence Oberman]

On Tue, 2022-02-22 at 10:03 -0500, John Pittman wrote:
> When scsi_dma_map() fails by returning a sges_left value less than
> zero, the amount of logging can be extremely high.  In a recent
> end-user environment, 1200 messages per second were being sent to
> the log buffer.  This eventually overwhelmed the system and it
> stalled.  As the messages are almost all identical, use
> pr_err_ratelimited() instead of sdev_printk() to print the
> scsi_dma_map failure messages.
> 
> Signed-off-by: John Pittman <jpittman@redhat.com>
> ---
>  drivers/scsi/mpt3sas/mpt3sas_base.c | 15 ++++++---------
>  1 file changed, 6 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c
> b/drivers/scsi/mpt3sas/mpt3sas_base.c
> index 511726f92d9a..ac9ccde6f3f8 100644
> --- a/drivers/scsi/mpt3sas/mpt3sas_base.c
> +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
> @@ -2594,9 +2594,8 @@ _base_check_pcie_native_sgl(struct
> MPT3SAS_ADAPTER *ioc,
>  	/* Get the SG list pointer and info. */
>  	sges_left = scsi_dma_map(scmd);
>  	if (sges_left < 0) {
> -		sdev_printk(KERN_ERR, scmd->device,
> -			"scsi_dma_map failed: request for %d bytes!\n",
> -			scsi_bufflen(scmd));
> +		pr_err_ratelimited("sd %s: scsi_dma_map failed: request
> for %d bytes!\n",
> +			dev_name(&scmd->device->sdev_gendev),
> scsi_bufflen(scmd));
>  		return 1;
>  	}
>  
> @@ -2706,9 +2705,8 @@ _base_build_sg_scmd(struct MPT3SAS_ADAPTER
> *ioc,
>  	sg_scmd = scsi_sglist(scmd);
>  	sges_left = scsi_dma_map(scmd);
>  	if (sges_left < 0) {
> -		sdev_printk(KERN_ERR, scmd->device,
> -		 "scsi_dma_map failed: request for %d bytes!\n",
> -		 scsi_bufflen(scmd));
> +		pr_err_ratelimited("sd %s: scsi_dma_map failed: request
> for %d bytes!\n",
> +			dev_name(&scmd->device->sdev_gendev),
> scsi_bufflen(scmd));
>  		return -ENOMEM;
>  	}
>  
> @@ -2854,9 +2852,8 @@ _base_build_sg_scmd_ieee(struct MPT3SAS_ADAPTER
> *ioc,
>  	sg_scmd = scsi_sglist(scmd);
>  	sges_left = scsi_dma_map(scmd);
>  	if (sges_left < 0) {
> -		sdev_printk(KERN_ERR, scmd->device,
> -			"scsi_dma_map failed: request for %d bytes!\n",
> -			scsi_bufflen(scmd));
> +		pr_err_ratelimited("sd %s: scsi_dma_map failed: request
> for %d bytes!\n",
> +			dev_name(&scmd->device->sdev_gendev),
> scsi_bufflen(scmd));
>  		return -ENOMEM;
>  	}
>  
This high message rate triggered a double completion at a customer and
this patch will help prevent the issue of the latency causing the race
window for the double completion. This was triaged by David Jeffery

Reviewed-by: Laurence Oberman <loberman@redhat.com>

Re: [PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[Christoph Hellwig]

On Tue, Feb 22, 2022 at 10:03:19AM -0500, John Pittman wrote:
> When scsi_dma_map() fails by returning a sges_left value less than
> zero, the amount of logging can be extremely high.  In a recent
> end-user environment, 1200 messages per second were being sent to
> the log buffer.  This eventually overwhelmed the system and it
> stalled.  As the messages are almost all identical, use
> pr_err_ratelimited() instead of sdev_printk() to print the
> scsi_dma_map failure messages.

I'd remove the message entirely.

Re: [PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[Martin K. Petersen]

> ACK by BRCM.

Please resubmit with message removed as requested by Christoph.

-- 
Martin K. Petersen	Oracle Linux Engineering

Re: [PATCH] scsi: mpt3sas: decrease potential frequency of scsi_dma_map errors

[Sreekanth Reddy]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

On Wed, Mar 2, 2022 at 9:54 AM Martin K. Petersen
<martin.petersen@oracle.com> wrote:
>
>
> > ACK by BRCM.
>
> Please resubmit with message removed as requested by Christoph.

Posted below patch to remove these error messages,
mpt3sas: Remove scsi_dma_map errors messages
https://patchwork.kernel.org/project/linux-scsi/patch/20220303140203.12642-1-sreekanth.reddy@broadcom.com/

>
> --
> Martin K. Petersen      Oracle Linux Engineering

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]