[PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Suganath Prabu]

From: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>

Although SAS3 & SAS3.5 IT HBA controllers support
64-bit DMA addressing, as per hardware design,
if DMA able range contains all 64-bits set (0xFFFFFFFF-FFFFFFFF) then
it results in a firmware fault.

e.g. SGE's start address is 0xFFFFFFFF-FFFF000 and
data length is 0x1000 bytes. when HBA tries to DMA the data
at 0xFFFFFFFF-FFFFFFFF location then HBA will
fault the firmware.

Fix:
Driver will set 63-bit DMA mask to ensure the above address
will not be used.

Cc: <stable@vger.kernel.org> # 4.4.186, # 4.9.186
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
---
Note:
This Patch is for stable kernel 4.4.186 and 4.9.186.
Original patch is applied to 5.3/scsi-fixes.
commit ID:  df9a606184bfdb5ae3ca9d226184e9489f5c24f7

 drivers/scsi/mpt3sas/mpt3sas_base.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 9b53672..7af7a08 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -1686,9 +1686,11 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 {
 	struct sysinfo s;
 	u64 consistent_dma_mask;
+	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
+	int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64;
 
 	if (ioc->dma_mask)
-		consistent_dma_mask = DMA_BIT_MASK(64);
+		consistent_dma_mask = DMA_BIT_MASK(dma_mask);
 	else
 		consistent_dma_mask = DMA_BIT_MASK(32);
 
@@ -1696,11 +1698,11 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 		const uint64_t required_mask =
 		    dma_get_required_mask(&pdev->dev);
 		if ((required_mask > DMA_BIT_MASK(32)) &&
-		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) &&
+		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(dma_mask)) &&
 		    !pci_set_consistent_dma_mask(pdev, consistent_dma_mask)) {
 			ioc->base_add_sg_single = &_base_add_sg_single_64;
 			ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-			ioc->dma_mask = 64;
+			ioc->dma_mask = dma_mask;
 			goto out;
 		}
 	}
@@ -1726,7 +1728,7 @@ static int
 _base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc,
 				      struct pci_dev *pdev)
 {
-	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {
+	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) {
 		if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)))
 			return -ENODEV;
 	}
@@ -3325,7 +3327,7 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc,  int sleep_flag)
 		total_sz += sz;
 	} while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count));
 
-	if (ioc->dma_mask == 64) {
+	if (ioc->dma_mask > 32) {
 		if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) {
 			pr_warn(MPT3SAS_FMT
 			    "no suitable consistent DMA mask for %s\n",
-- 
2.16.3

[PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Suganath Prabu]

From: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>

Although SAS3 & SAS3.5 IT HBA controllers support
64-bit DMA addressing, as per hardware design,
if DMA able range contains all 64-bits set (0xFFFFFFFF-FFFFFFFF) then
it results in a firmware fault.

e.g. SGE's start address is 0xFFFFFFFF-FFFF000 and
data length is 0x1000 bytes. when HBA tries to DMA the data
at 0xFFFFFFFF-FFFFFFFF location then HBA will
fault the firmware.

Fix:
Driver will set 63-bit DMA mask to ensure the above address
will not be used.

Cc: <stable@vger.kernel.org> # 4.19.63
Signed-off-by: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
---
Note:
This Patch is for stable kernel 4.19.63.
Original patch is applied to 5.3/scsi-fixes.
commit ID:  df9a606184bfdb5ae3ca9d226184e9489f5c24f7

 drivers/scsi/mpt3sas/mpt3sas_base.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 8776330..d2ab520 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -2565,12 +2565,14 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 {
 	struct sysinfo s;
 	u64 consistent_dma_mask;
+	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
+	int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64;
 
 	if (ioc->is_mcpu_endpoint)
 		goto try_32bit;
 
 	if (ioc->dma_mask)
-		consistent_dma_mask = DMA_BIT_MASK(64);
+		consistent_dma_mask = DMA_BIT_MASK(dma_mask);
 	else
 		consistent_dma_mask = DMA_BIT_MASK(32);
 
@@ -2578,11 +2580,11 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 		const uint64_t required_mask =
 		    dma_get_required_mask(&pdev->dev);
 		if ((required_mask > DMA_BIT_MASK(32)) &&
-		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) &&
+		    !pci_set_dma_mask(pdev, DMA_BIT_MASK(dma_mask)) &&
 		    !pci_set_consistent_dma_mask(pdev, consistent_dma_mask)) {
 			ioc->base_add_sg_single = &_base_add_sg_single_64;
 			ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-			ioc->dma_mask = 64;
+			ioc->dma_mask = dma_mask;
 			goto out;
 		}
 	}
@@ -2609,7 +2611,7 @@ static int
 _base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc,
 				      struct pci_dev *pdev)
 {
-	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {
+	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) {
 		if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)))
 			return -ENODEV;
 	}
@@ -4545,7 +4547,7 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 		total_sz += sz;
 	} while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count));
 
-	if (ioc->dma_mask == 64) {
+	if (ioc->dma_mask > 32) {
 		if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) {
 			pr_warn(MPT3SAS_FMT
 			    "no suitable consistent DMA mask for %s\n",
-- 
2.16.3

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Greg KH]

On Wed, Jul 31, 2019 at 05:58:49AM -0400, Suganath Prabu wrote:
> From: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
> 
> Although SAS3 & SAS3.5 IT HBA controllers support
> 64-bit DMA addressing, as per hardware design,
> if DMA able range contains all 64-bits set (0xFFFFFFFF-FFFFFFFF) then
> it results in a firmware fault.
> 
> e.g. SGE's start address is 0xFFFFFFFF-FFFF000 and
> data length is 0x1000 bytes. when HBA tries to DMA the data
> at 0xFFFFFFFF-FFFFFFFF location then HBA will
> fault the firmware.
> 
> Fix:
> Driver will set 63-bit DMA mask to ensure the above address
> will not be used.
> 
> Cc: <stable@vger.kernel.org> # 4.19.63
> Signed-off-by: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
> ---
> Note:
> This Patch is for stable kernel 4.19.63.
> Original patch is applied to 5.3/scsi-fixes.
> commit ID:  df9a606184bfdb5ae3ca9d226184e9489f5c24f7

Found this one.  Now queued up,t hanks.

greg k-h

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Greg KH]

On Wed, Jul 31, 2019 at 05:58:48AM -0400, Suganath Prabu wrote:
> From: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
> 
> Although SAS3 & SAS3.5 IT HBA controllers support
> 64-bit DMA addressing, as per hardware design,
> if DMA able range contains all 64-bits set (0xFFFFFFFF-FFFFFFFF) then
> it results in a firmware fault.
> 
> e.g. SGE's start address is 0xFFFFFFFF-FFFF000 and
> data length is 0x1000 bytes. when HBA tries to DMA the data
> at 0xFFFFFFFF-FFFFFFFF location then HBA will
> fault the firmware.
> 
> Fix:
> Driver will set 63-bit DMA mask to ensure the above address
> will not be used.
> 
> Cc: <stable@vger.kernel.org> # 4.4.186, # 4.9.186
> Signed-off-by: Suganath Prabu S <suganath-prabu.subramani@broadcom.com>
> ---
> Note:
> This Patch is for stable kernel 4.4.186 and 4.9.186.
> Original patch is applied to 5.3/scsi-fixes.
> commit ID:  df9a606184bfdb5ae3ca9d226184e9489f5c24f7
> 

We can't do anything with this for a stable release until it hits
Linus's kernel tree.  Please wait until then to send backports.

thanks,

greg k-h

[PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Suganath Prabu]

Although SAS3 & SAS3.5 IT HBA controllers support
64-bit DMA addressing, as per hardware design,
DMA address with all 64-bits set (0xFFFFFFFF-FFFFFFFF)
results in a firmware fault.

Fix:
Driver will set 63-bit DMA mask to ensure the above address
will not be used.

Signed-off-by: Suganath Prabu <suganath-prabu.subramani@broadcom.com>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index 6846628..050c0f0 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -2703,6 +2703,8 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 {
 	u64 required_mask, coherent_mask;
 	struct sysinfo s;
+	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
+	int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64;
 
 	if (ioc->is_mcpu_endpoint)
 		goto try_32bit;
@@ -2712,17 +2714,17 @@ _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 		goto try_32bit;
 
 	if (ioc->dma_mask)
-		coherent_mask = DMA_BIT_MASK(64);
+		coherent_mask = DMA_BIT_MASK(dma_mask);
 	else
 		coherent_mask = DMA_BIT_MASK(32);
 
-	if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)) ||
+	if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(dma_mask)) ||
 	    dma_set_coherent_mask(&pdev->dev, coherent_mask))
 		goto try_32bit;
 
 	ioc->base_add_sg_single = &_base_add_sg_single_64;
 	ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-	ioc->dma_mask = 64;
+	ioc->dma_mask = dma_mask;
 	goto out;
 
  try_32bit:
@@ -2744,7 +2746,7 @@ static int
 _base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc,
 				      struct pci_dev *pdev)
 {
-	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64))) {
+	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) {
 		if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)))
 			return -ENODEV;
 	}
@@ -4989,7 +4991,7 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 		total_sz += sz;
 	} while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count));
 
-	if (ioc->dma_mask == 64) {
+	if (ioc->dma_mask > 32) {
 		if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) {
 			ioc_warn(ioc, "no suitable consistent DMA mask for %s\n",
 				 pci_name(ioc->pdev));
-- 
1.8.3.1

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Greg KH]

On Fri, Jul 26, 2019 at 06:00:57AM -0400, Suganath Prabu wrote:
> Although SAS3 & SAS3.5 IT HBA controllers support
> 64-bit DMA addressing, as per hardware design,
> DMA address with all 64-bits set (0xFFFFFFFF-FFFFFFFF)
> results in a firmware fault.
> 
> Fix:
> Driver will set 63-bit DMA mask to ensure the above address
> will not be used.
> 
> Signed-off-by: Suganath Prabu <suganath-prabu.subramani@broadcom.com>
> ---
>  drivers/scsi/mpt3sas/mpt3sas_base.c | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)

<formletter>

This is not the correct way to submit patches for inclusion in the
stable kernel tree.  Please read:
    https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html
for how to do this properly.

</formletter>

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Christoph Hellwig]

On Fri, Jul 26, 2019 at 06:00:57AM -0400, Suganath Prabu wrote:
> Although SAS3 & SAS3.5 IT HBA controllers support
> 64-bit DMA addressing, as per hardware design,
> DMA address with all 64-bits set (0xFFFFFFFF-FFFFFFFF)
> results in a firmware fault.

Linux will never send a dma address with all bits set anyway, as that
is our magic escape for the dma_addr_t error value.  Additionally to
generate that address you'd need a 1-byte sized, 1-byte aligned buffer,
which we never use.

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Sreekanth Reddy]

On Fri, Jul 26, 2019 at 7:57 PM Christoph Hellwig <hch@infradead.org> wrote:
>
> On Fri, Jul 26, 2019 at 06:00:57AM -0400, Suganath Prabu wrote:
> > Although SAS3 & SAS3.5 IT HBA controllers support
> > 64-bit DMA addressing, as per hardware design,
> > DMA address with all 64-bits set (0xFFFFFFFF-FFFFFFFF)
> > results in a firmware fault.
>
> Linux will never send a dma address with all bits set anyway, as that
> is our magic escape for the dma_addr_t error value.  Additionally to
> generate that address you'd need a 1-byte sized, 1-byte aligned buffer,
> which we never use.

I agree with your above statement. But it is also possible that
0xFFFFFFFF-FFFFFFFF falls under the DMA able range, e.g. SGE's start
address is 0xFFFFFFFF-FFFF000 and data length is 0x1000 bytes. So when
HBA tries to DMA the data at 0xFFFFFFFF-FFFFFFFF location then it will
faults the firmware due to it's hardware design.

We have observed above example's SGE address and length on AMD systems
with SME & IOMMU enabled.

Thanks,
Sreekanth

Re: [PATCH] mpt3sas: Use 63-bit DMA addressing on SAS35 HBA

[Christoph Hellwig]

On Mon, Jul 29, 2019 at 05:25:35PM +0530, Sreekanth Reddy wrote:
> 
> I agree with your above statement. But it is also possible that
> 0xFFFFFFFF-FFFFFFFF falls under the DMA able range, e.g. SGE's start
> address is 0xFFFFFFFF-FFFF000 and data length is 0x1000 bytes. So when
> HBA tries to DMA the data at 0xFFFFFFFF-FFFFFFFF location then it will
> faults the firmware due to it's hardware design.
> 
> We have observed above example's SGE address and length on AMD systems
> with SME & IOMMU enabled.

Ok.  Please slightly update the changelog to say dma ranges instead
of a dma address, as that implicies the addr field in the sg to me.

Otherwise looks ok:

Reviewed-by: Christoph Hellwig <hch@lst.de>