Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

[Christoph Hellwig]

Hi Abdul,

I think the problem is that mpt3sas has some convoluted logic to do
some DMA allocations with a 32-bit coherent mask, and then switches
to a 63 or 64 bit mask, which is not supported by the DMA API.

Can you try the patch below?

---
From 0738b1704ed528497b41b0408325f6828a8e51f6 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <hch@lst.de>
Date: Thu, 16 Jan 2020 18:31:38 +0100
Subject: mpt3sas: don't change the dma coherent mask after allocations

The DMA layer does not allow changing the DMA coherent mask after
there are outstanding allocations.  Stop doing that and always
use a 32-bit coherent DMA mask in mpt3sas.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 drivers/scsi/mpt3sas/mpt3sas_base.c | 67 ++++++++---------------------
 drivers/scsi/mpt3sas/mpt3sas_base.h |  2 -
 2 files changed, 19 insertions(+), 50 deletions(-)

diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
index fea3cb6a090b..3b51bed05008 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
@@ -2706,58 +2706,38 @@ _base_build_sg_ieee(struct MPT3SAS_ADAPTER *ioc, void *psge,
 static int
 _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
 {
-	u64 required_mask, coherent_mask;
 	struct sysinfo s;
-	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
-	int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64;
-
-	if (ioc->is_mcpu_endpoint)
-		goto try_32bit;
+	int dma_mask;
 
-	required_mask = dma_get_required_mask(&pdev->dev);
-	if (sizeof(dma_addr_t) == 4 || required_mask == 32)
-		goto try_32bit;
-
-	if (ioc->dma_mask)
-		coherent_mask = DMA_BIT_MASK(dma_mask);
+	if (ioc->is_mcpu_endpoint ||
+	    sizeof(dma_addr_t) == 4 ||
+	    dma_get_required_mask(&pdev->dev) <= 32)
+		dma_mask = 32;
+	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
+	else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)
+		dma_mask = 63;
 	else
-		coherent_mask = DMA_BIT_MASK(32);
+		dma_mask = 64;
 
 	if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(dma_mask)) ||
-	    dma_set_coherent_mask(&pdev->dev, coherent_mask))
-		goto try_32bit;
-
-	ioc->base_add_sg_single = &_base_add_sg_single_64;
-	ioc->sge_size = sizeof(Mpi2SGESimple64_t);
-	ioc->dma_mask = dma_mask;
-	goto out;
-
- try_32bit:
-	if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)))
+	    dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32)))
 		return -ENODEV;
 
-	ioc->base_add_sg_single = &_base_add_sg_single_32;
-	ioc->sge_size = sizeof(Mpi2SGESimple32_t);
-	ioc->dma_mask = 32;
- out:
+	if (dma_mask > 32) {
+		ioc->base_add_sg_single = &_base_add_sg_single_64;
+		ioc->sge_size = sizeof(Mpi2SGESimple64_t);
+	} else {
+		ioc->base_add_sg_single = &_base_add_sg_single_32;
+		ioc->sge_size = sizeof(Mpi2SGESimple32_t);
+	}
+
 	si_meminfo(&s);
 	ioc_info(ioc, "%d BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (%ld kB)\n",
-		 ioc->dma_mask, convert_to_kb(s.totalram));
+		 dma_mask, convert_to_kb(s.totalram));
 
 	return 0;
 }
 
-static int
-_base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc,
-				      struct pci_dev *pdev)
-{
-	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) {
-		if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)))
-			return -ENODEV;
-	}
-	return 0;
-}
-
 /**
  * _base_check_enable_msix - checks MSIX capabable.
  * @ioc: per adapter object
@@ -5030,14 +5010,6 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
 		total_sz += sz;
 	} while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count));
 
-	if (ioc->dma_mask > 32) {
-		if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) {
-			ioc_warn(ioc, "no suitable consistent DMA mask for %s\n",
-				 pci_name(ioc->pdev));
-			goto out;
-		}
-	}
-
 	ioc->scsiio_depth = ioc->hba_queue_depth -
 	    ioc->hi_priority_depth - ioc->internal_depth;
 
@@ -6965,7 +6937,6 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc)
 	ioc->smp_affinity_enable = smp_affinity_enable;
 
 	ioc->rdpq_array_enable_assigned = 0;
-	ioc->dma_mask = 0;
 	if (ioc->is_aero_ioc)
 		ioc->base_readl = &_base_readl_aero;
 	else
diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
index faca0a5e71f8..e57cade1155c 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_base.h
+++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
@@ -1011,7 +1011,6 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
  * @ir_firmware: IR firmware present
  * @bars: bitmask of BAR's that must be configured
  * @mask_interrupts: ignore interrupt
- * @dma_mask: used to set the consistent dma mask
  * @pci_access_mutex: Mutex to synchronize ioctl, sysfs show path and
  *			pci resource handling
  * @fault_reset_work_q_name: fw fault work queue
@@ -1185,7 +1184,6 @@ struct MPT3SAS_ADAPTER {
 	u8		ir_firmware;
 	int		bars;
 	u8		mask_interrupts;
-	int		dma_mask;
 
 	/* fw fault handler */
 	char		fault_reset_work_q_name[20];
-- 
2.24.1

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

[Abdul Haleem]

On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote:
> Hi Abdul,
> 
> I think the problem is that mpt3sas has some convoluted logic to do
> some DMA allocations with a 32-bit coherent mask, and then switches
> to a 63 or 64 bit mask, which is not supported by the DMA API.
> 
> Can you try the patch below?

Thank you Christoph, with the given patch applied the bug is not seen.

rmmod of mpt3sas driver is successful, no kernel Oops

Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>

> 
> ---
> From 0738b1704ed528497b41b0408325f6828a8e51f6 Mon Sep 17 00:00:00 2001
> From: Christoph Hellwig <hch@lst.de>
> Date: Thu, 16 Jan 2020 18:31:38 +0100
> Subject: mpt3sas: don't change the dma coherent mask after allocations
> 
> The DMA layer does not allow changing the DMA coherent mask after
> there are outstanding allocations.  Stop doing that and always
> use a 32-bit coherent DMA mask in mpt3sas.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  drivers/scsi/mpt3sas/mpt3sas_base.c | 67 ++++++++---------------------
>  drivers/scsi/mpt3sas/mpt3sas_base.h |  2 -
>  2 files changed, 19 insertions(+), 50 deletions(-)
> 
> diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c
> index fea3cb6a090b..3b51bed05008 100644
> --- a/drivers/scsi/mpt3sas/mpt3sas_base.c
> +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c
> @@ -2706,58 +2706,38 @@ _base_build_sg_ieee(struct MPT3SAS_ADAPTER *ioc, void *psge,
>  static int
>  _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev)
>  {
> -	u64 required_mask, coherent_mask;
>  	struct sysinfo s;
> -	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
> -	int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64;
> -
> -	if (ioc->is_mcpu_endpoint)
> -		goto try_32bit;
> +	int dma_mask;
> 
> -	required_mask = dma_get_required_mask(&pdev->dev);
> -	if (sizeof(dma_addr_t) == 4 || required_mask == 32)
> -		goto try_32bit;
> -
> -	if (ioc->dma_mask)
> -		coherent_mask = DMA_BIT_MASK(dma_mask);
> +	if (ioc->is_mcpu_endpoint ||
> +	    sizeof(dma_addr_t) == 4 ||
> +	    dma_get_required_mask(&pdev->dev) <= 32)
> +		dma_mask = 32;
> +	/* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */
> +	else if (ioc->hba_mpi_version_belonged > MPI2_VERSION)
> +		dma_mask = 63;
>  	else
> -		coherent_mask = DMA_BIT_MASK(32);
> +		dma_mask = 64;
> 
>  	if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(dma_mask)) ||
> -	    dma_set_coherent_mask(&pdev->dev, coherent_mask))
> -		goto try_32bit;
> -
> -	ioc->base_add_sg_single = &_base_add_sg_single_64;
> -	ioc->sge_size = sizeof(Mpi2SGESimple64_t);
> -	ioc->dma_mask = dma_mask;
> -	goto out;
> -
> - try_32bit:
> -	if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32)))
> +	    dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32)))
>  		return -ENODEV;
> 
> -	ioc->base_add_sg_single = &_base_add_sg_single_32;
> -	ioc->sge_size = sizeof(Mpi2SGESimple32_t);
> -	ioc->dma_mask = 32;
> - out:
> +	if (dma_mask > 32) {
> +		ioc->base_add_sg_single = &_base_add_sg_single_64;
> +		ioc->sge_size = sizeof(Mpi2SGESimple64_t);
> +	} else {
> +		ioc->base_add_sg_single = &_base_add_sg_single_32;
> +		ioc->sge_size = sizeof(Mpi2SGESimple32_t);
> +	}
> +
>  	si_meminfo(&s);
>  	ioc_info(ioc, "%d BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (%ld kB)\n",
> -		 ioc->dma_mask, convert_to_kb(s.totalram));
> +		 dma_mask, convert_to_kb(s.totalram));
> 
>  	return 0;
>  }
> 
> -static int
> -_base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc,
> -				      struct pci_dev *pdev)
> -{
> -	if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) {
> -		if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32)))
> -			return -ENODEV;
> -	}
> -	return 0;
> -}
> -
>  /**
>   * _base_check_enable_msix - checks MSIX capabable.
>   * @ioc: per adapter object
> @@ -5030,14 +5010,6 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc)
>  		total_sz += sz;
>  	} while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count));
> 
> -	if (ioc->dma_mask > 32) {
> -		if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) {
> -			ioc_warn(ioc, "no suitable consistent DMA mask for %s\n",
> -				 pci_name(ioc->pdev));
> -			goto out;
> -		}
> -	}
> -
>  	ioc->scsiio_depth = ioc->hba_queue_depth -
>  	    ioc->hi_priority_depth - ioc->internal_depth;
> 
> @@ -6965,7 +6937,6 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc)
>  	ioc->smp_affinity_enable = smp_affinity_enable;
> 
>  	ioc->rdpq_array_enable_assigned = 0;
> -	ioc->dma_mask = 0;
>  	if (ioc->is_aero_ioc)
>  		ioc->base_readl = &_base_readl_aero;
>  	else
> diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h
> index faca0a5e71f8..e57cade1155c 100644
> --- a/drivers/scsi/mpt3sas/mpt3sas_base.h
> +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h
> @@ -1011,7 +1011,6 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc);
>   * @ir_firmware: IR firmware present
>   * @bars: bitmask of BAR's that must be configured
>   * @mask_interrupts: ignore interrupt
> - * @dma_mask: used to set the consistent dma mask
>   * @pci_access_mutex: Mutex to synchronize ioctl, sysfs show path and
>   *			pci resource handling
>   * @fault_reset_work_q_name: fw fault work queue
> @@ -1185,7 +1184,6 @@ struct MPT3SAS_ADAPTER {
>  	u8		ir_firmware;
>  	int		bars;
>  	u8		mask_interrupts;
> -	int		dma_mask;
> 
>  	/* fw fault handler */
>  	char		fault_reset_work_q_name[20];


-- 
Regard's

Abdul Haleem
IBM Linux Technology Centre



_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

[Abdul Haleem]

On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote:
> On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote:
> > Hi Abdul,
> > 
> > I think the problem is that mpt3sas has some convoluted logic to do
> > some DMA allocations with a 32-bit coherent mask, and then switches
> > to a 63 or 64 bit mask, which is not supported by the DMA API.
> > 
> > Can you try the patch below?
> 
> Thank you Christoph, with the given patch applied the bug is not seen.
> 
> rmmod of mpt3sas driver is successful, no kernel Oops
> 
> Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>

Hi Christoph,

I see the patch is under discussion, will this be merged upstream any
time soon ? as boot is broken on our machines with out your patch.

-- 
Regard's

Abdul Haleem
IBM Linux Technology Centre



_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

[Sreekanth Reddy via iommu]

On Tue, Feb 25, 2020 at 11:51 AM Abdul Haleem
<abdhalee@linux.vnet.ibm.com> wrote:
>
> On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote:
> > On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote:
> > > Hi Abdul,
> > >
> > > I think the problem is that mpt3sas has some convoluted logic to do
> > > some DMA allocations with a 32-bit coherent mask, and then switches
> > > to a 63 or 64 bit mask, which is not supported by the DMA API.
> > >
> > > Can you try the patch below?
> >
> > Thank you Christoph, with the given patch applied the bug is not seen.
> >
> > rmmod of mpt3sas driver is successful, no kernel Oops
> >
> > Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>
>
> Hi Christoph,
>
> I see the patch is under discussion, will this be merged upstream any
> time soon ? as boot is broken on our machines with out your patch.
>

Hi Abdul,

We have posted a new set of patches to fix this issue. This patch set
won't change the DMA Mask on the fly and also won't hardcode the DMA
mask to 32 bit.

[PATCH 0/5] mpt3sas: Fix changing coherent mask after allocation.

This patchset will have below patches, Please review and try with this
patch set.

Suganath Prabu S (5):
  mpt3sas: Don't change the dma coherent mask after      allocations
  mpt3sas: Rename function name is_MSB_are_same
  mpt3sas: Code Refactoring.
  mpt3sas: Handle RDPQ DMA allocation in same 4g region
  mpt3sas: Update version to 33.101.00.00

Regards,
Sreekanth

> --
> Regard's
>
> Abdul Haleem
> IBM Linux Technology Centre
>
>
>
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu

Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver

[Abdul Haleem]

On Tue, 2020-02-25 at 12:23 +0530, Sreekanth Reddy wrote:
> On Tue, Feb 25, 2020 at 11:51 AM Abdul Haleem
> <abdhalee@linux.vnet.ibm.com> wrote:
> >
> > On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote:
> > > On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote:
> > > > Hi Abdul,
> > > >
> > > > I think the problem is that mpt3sas has some convoluted logic to do
> > > > some DMA allocations with a 32-bit coherent mask, and then switches
> > > > to a 63 or 64 bit mask, which is not supported by the DMA API.
> > > >
> > > > Can you try the patch below?
> > >
> > > Thank you Christoph, with the given patch applied the bug is not seen.
> > >
> > > rmmod of mpt3sas driver is successful, no kernel Oops
> > >
> > > Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>
> >
> > Hi Christoph,
> >
> > I see the patch is under discussion, will this be merged upstream any
> > time soon ? as boot is broken on our machines with out your patch.
> >
> 
> Hi Abdul,
> 
> We have posted a new set of patches to fix this issue. This patch set
> won't change the DMA Mask on the fly and also won't hardcode the DMA
> mask to 32 bit.
> 
> [PATCH 0/5] mpt3sas: Fix changing coherent mask after allocation.
> 
> This patchset will have below patches, Please review and try with this
> patch set.
> 
> Suganath Prabu S (5):
>   mpt3sas: Don't change the dma coherent mask after      allocations
>   mpt3sas: Rename function name is_MSB_are_same
>   mpt3sas: Code Refactoring.
>   mpt3sas: Handle RDPQ DMA allocation in same 4g region
>   mpt3sas: Update version to 33.101.00.00

Hi Suganath, 

The above patch fixes the issue, driver is loading and unloading with no
kernel oops. 

Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com>

-- 
Regard's

Abdul Haleem
IBM Linux Technology Centre



_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu