* Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver [not found] ` <1578980874.11996.3.camel@abdul.in.ibm.com> @ 2020-01-16 17:44 ` Christoph Hellwig 2020-01-17 12:51 ` Abdul Haleem 0 siblings, 1 reply; 5+ messages in thread From: Christoph Hellwig @ 2020-01-16 17:44 UTC (permalink / raw) To: Abdul Haleem Cc: sachinp, Chaitra P B, linux-scsi, MPT-FusionLinux.pdl, manvanth, Sathya Prakash, jcmvbkbc, iommu, linux-next, Oliver, aneesh.kumar, Suganath Prabu Subramani, Brian King, linuxppc-dev Hi Abdul, I think the problem is that mpt3sas has some convoluted logic to do some DMA allocations with a 32-bit coherent mask, and then switches to a 63 or 64 bit mask, which is not supported by the DMA API. Can you try the patch below? --- From 0738b1704ed528497b41b0408325f6828a8e51f6 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig <hch@lst.de> Date: Thu, 16 Jan 2020 18:31:38 +0100 Subject: mpt3sas: don't change the dma coherent mask after allocations The DMA layer does not allow changing the DMA coherent mask after there are outstanding allocations. Stop doing that and always use a 32-bit coherent DMA mask in mpt3sas. Signed-off-by: Christoph Hellwig <hch@lst.de> --- drivers/scsi/mpt3sas/mpt3sas_base.c | 67 ++++++++--------------------- drivers/scsi/mpt3sas/mpt3sas_base.h | 2 - 2 files changed, 19 insertions(+), 50 deletions(-) diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c index fea3cb6a090b..3b51bed05008 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.c +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c @@ -2706,58 +2706,38 @@ _base_build_sg_ieee(struct MPT3SAS_ADAPTER *ioc, void *psge, static int _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev) { - u64 required_mask, coherent_mask; struct sysinfo s; - /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */ - int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64; - - if (ioc->is_mcpu_endpoint) - goto try_32bit; + int dma_mask; - required_mask = dma_get_required_mask(&pdev->dev); - if (sizeof(dma_addr_t) == 4 || required_mask == 32) - goto try_32bit; - - if (ioc->dma_mask) - coherent_mask = DMA_BIT_MASK(dma_mask); + if (ioc->is_mcpu_endpoint || + sizeof(dma_addr_t) == 4 || + dma_get_required_mask(&pdev->dev) <= 32) + dma_mask = 32; + /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */ + else if (ioc->hba_mpi_version_belonged > MPI2_VERSION) + dma_mask = 63; else - coherent_mask = DMA_BIT_MASK(32); + dma_mask = 64; if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(dma_mask)) || - dma_set_coherent_mask(&pdev->dev, coherent_mask)) - goto try_32bit; - - ioc->base_add_sg_single = &_base_add_sg_single_64; - ioc->sge_size = sizeof(Mpi2SGESimple64_t); - ioc->dma_mask = dma_mask; - goto out; - - try_32bit: - if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32))) + dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32))) return -ENODEV; - ioc->base_add_sg_single = &_base_add_sg_single_32; - ioc->sge_size = sizeof(Mpi2SGESimple32_t); - ioc->dma_mask = 32; - out: + if (dma_mask > 32) { + ioc->base_add_sg_single = &_base_add_sg_single_64; + ioc->sge_size = sizeof(Mpi2SGESimple64_t); + } else { + ioc->base_add_sg_single = &_base_add_sg_single_32; + ioc->sge_size = sizeof(Mpi2SGESimple32_t); + } + si_meminfo(&s); ioc_info(ioc, "%d BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (%ld kB)\n", - ioc->dma_mask, convert_to_kb(s.totalram)); + dma_mask, convert_to_kb(s.totalram)); return 0; } -static int -_base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc, - struct pci_dev *pdev) -{ - if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) { - if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32))) - return -ENODEV; - } - return 0; -} - /** * _base_check_enable_msix - checks MSIX capabable. * @ioc: per adapter object @@ -5030,14 +5010,6 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc) total_sz += sz; } while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count)); - if (ioc->dma_mask > 32) { - if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) { - ioc_warn(ioc, "no suitable consistent DMA mask for %s\n", - pci_name(ioc->pdev)); - goto out; - } - } - ioc->scsiio_depth = ioc->hba_queue_depth - ioc->hi_priority_depth - ioc->internal_depth; @@ -6965,7 +6937,6 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) ioc->smp_affinity_enable = smp_affinity_enable; ioc->rdpq_array_enable_assigned = 0; - ioc->dma_mask = 0; if (ioc->is_aero_ioc) ioc->base_readl = &_base_readl_aero; else diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h index faca0a5e71f8..e57cade1155c 100644 --- a/drivers/scsi/mpt3sas/mpt3sas_base.h +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h @@ -1011,7 +1011,6 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc); * @ir_firmware: IR firmware present * @bars: bitmask of BAR's that must be configured * @mask_interrupts: ignore interrupt - * @dma_mask: used to set the consistent dma mask * @pci_access_mutex: Mutex to synchronize ioctl, sysfs show path and * pci resource handling * @fault_reset_work_q_name: fw fault work queue @@ -1185,7 +1184,6 @@ struct MPT3SAS_ADAPTER { u8 ir_firmware; int bars; u8 mask_interrupts; - int dma_mask; /* fw fault handler */ char fault_reset_work_q_name[20]; -- 2.24.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver 2020-01-16 17:44 ` [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver Christoph Hellwig @ 2020-01-17 12:51 ` Abdul Haleem 2020-02-25 6:20 ` Abdul Haleem 0 siblings, 1 reply; 5+ messages in thread From: Abdul Haleem @ 2020-01-17 12:51 UTC (permalink / raw) To: Christoph Hellwig Cc: sachinp, Chaitra P B, linux-scsi, MPT-FusionLinux.pdl, manvanth, Sathya Prakash, jcmvbkbc, iommu, linux-next, Oliver, aneesh.kumar, Suganath Prabu Subramani, Brian King, linuxppc-dev On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote: > Hi Abdul, > > I think the problem is that mpt3sas has some convoluted logic to do > some DMA allocations with a 32-bit coherent mask, and then switches > to a 63 or 64 bit mask, which is not supported by the DMA API. > > Can you try the patch below? Thank you Christoph, with the given patch applied the bug is not seen. rmmod of mpt3sas driver is successful, no kernel Oops Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com> > > --- > From 0738b1704ed528497b41b0408325f6828a8e51f6 Mon Sep 17 00:00:00 2001 > From: Christoph Hellwig <hch@lst.de> > Date: Thu, 16 Jan 2020 18:31:38 +0100 > Subject: mpt3sas: don't change the dma coherent mask after allocations > > The DMA layer does not allow changing the DMA coherent mask after > there are outstanding allocations. Stop doing that and always > use a 32-bit coherent DMA mask in mpt3sas. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > drivers/scsi/mpt3sas/mpt3sas_base.c | 67 ++++++++--------------------- > drivers/scsi/mpt3sas/mpt3sas_base.h | 2 - > 2 files changed, 19 insertions(+), 50 deletions(-) > > diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.c b/drivers/scsi/mpt3sas/mpt3sas_base.c > index fea3cb6a090b..3b51bed05008 100644 > --- a/drivers/scsi/mpt3sas/mpt3sas_base.c > +++ b/drivers/scsi/mpt3sas/mpt3sas_base.c > @@ -2706,58 +2706,38 @@ _base_build_sg_ieee(struct MPT3SAS_ADAPTER *ioc, void *psge, > static int > _base_config_dma_addressing(struct MPT3SAS_ADAPTER *ioc, struct pci_dev *pdev) > { > - u64 required_mask, coherent_mask; > struct sysinfo s; > - /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */ > - int dma_mask = (ioc->hba_mpi_version_belonged > MPI2_VERSION) ? 63 : 64; > - > - if (ioc->is_mcpu_endpoint) > - goto try_32bit; > + int dma_mask; > > - required_mask = dma_get_required_mask(&pdev->dev); > - if (sizeof(dma_addr_t) == 4 || required_mask == 32) > - goto try_32bit; > - > - if (ioc->dma_mask) > - coherent_mask = DMA_BIT_MASK(dma_mask); > + if (ioc->is_mcpu_endpoint || > + sizeof(dma_addr_t) == 4 || > + dma_get_required_mask(&pdev->dev) <= 32) > + dma_mask = 32; > + /* Set 63 bit DMA mask for all SAS3 and SAS35 controllers */ > + else if (ioc->hba_mpi_version_belonged > MPI2_VERSION) > + dma_mask = 63; > else > - coherent_mask = DMA_BIT_MASK(32); > + dma_mask = 64; > > if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(dma_mask)) || > - dma_set_coherent_mask(&pdev->dev, coherent_mask)) > - goto try_32bit; > - > - ioc->base_add_sg_single = &_base_add_sg_single_64; > - ioc->sge_size = sizeof(Mpi2SGESimple64_t); > - ioc->dma_mask = dma_mask; > - goto out; > - > - try_32bit: > - if (dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32))) > + dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(32))) > return -ENODEV; > > - ioc->base_add_sg_single = &_base_add_sg_single_32; > - ioc->sge_size = sizeof(Mpi2SGESimple32_t); > - ioc->dma_mask = 32; > - out: > + if (dma_mask > 32) { > + ioc->base_add_sg_single = &_base_add_sg_single_64; > + ioc->sge_size = sizeof(Mpi2SGESimple64_t); > + } else { > + ioc->base_add_sg_single = &_base_add_sg_single_32; > + ioc->sge_size = sizeof(Mpi2SGESimple32_t); > + } > + > si_meminfo(&s); > ioc_info(ioc, "%d BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (%ld kB)\n", > - ioc->dma_mask, convert_to_kb(s.totalram)); > + dma_mask, convert_to_kb(s.totalram)); > > return 0; > } > > -static int > -_base_change_consistent_dma_mask(struct MPT3SAS_ADAPTER *ioc, > - struct pci_dev *pdev) > -{ > - if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(ioc->dma_mask))) { > - if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(32))) > - return -ENODEV; > - } > - return 0; > -} > - > /** > * _base_check_enable_msix - checks MSIX capabable. > * @ioc: per adapter object > @@ -5030,14 +5010,6 @@ _base_allocate_memory_pools(struct MPT3SAS_ADAPTER *ioc) > total_sz += sz; > } while (ioc->rdpq_array_enable && (++i < ioc->reply_queue_count)); > > - if (ioc->dma_mask > 32) { > - if (_base_change_consistent_dma_mask(ioc, ioc->pdev) != 0) { > - ioc_warn(ioc, "no suitable consistent DMA mask for %s\n", > - pci_name(ioc->pdev)); > - goto out; > - } > - } > - > ioc->scsiio_depth = ioc->hba_queue_depth - > ioc->hi_priority_depth - ioc->internal_depth; > > @@ -6965,7 +6937,6 @@ mpt3sas_base_attach(struct MPT3SAS_ADAPTER *ioc) > ioc->smp_affinity_enable = smp_affinity_enable; > > ioc->rdpq_array_enable_assigned = 0; > - ioc->dma_mask = 0; > if (ioc->is_aero_ioc) > ioc->base_readl = &_base_readl_aero; > else > diff --git a/drivers/scsi/mpt3sas/mpt3sas_base.h b/drivers/scsi/mpt3sas/mpt3sas_base.h > index faca0a5e71f8..e57cade1155c 100644 > --- a/drivers/scsi/mpt3sas/mpt3sas_base.h > +++ b/drivers/scsi/mpt3sas/mpt3sas_base.h > @@ -1011,7 +1011,6 @@ typedef void (*MPT3SAS_FLUSH_RUNNING_CMDS)(struct MPT3SAS_ADAPTER *ioc); > * @ir_firmware: IR firmware present > * @bars: bitmask of BAR's that must be configured > * @mask_interrupts: ignore interrupt > - * @dma_mask: used to set the consistent dma mask > * @pci_access_mutex: Mutex to synchronize ioctl, sysfs show path and > * pci resource handling > * @fault_reset_work_q_name: fw fault work queue > @@ -1185,7 +1184,6 @@ struct MPT3SAS_ADAPTER { > u8 ir_firmware; > int bars; > u8 mask_interrupts; > - int dma_mask; > > /* fw fault handler */ > char fault_reset_work_q_name[20]; -- Regard's Abdul Haleem IBM Linux Technology Centre _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver 2020-01-17 12:51 ` Abdul Haleem @ 2020-02-25 6:20 ` Abdul Haleem 2020-02-25 6:53 ` Sreekanth Reddy via iommu 0 siblings, 1 reply; 5+ messages in thread From: Abdul Haleem @ 2020-02-25 6:20 UTC (permalink / raw) To: Christoph Hellwig Cc: sachinp, Chaitra P B, linux-scsi, MPT-FusionLinux.pdl, manvanth, Sathya Prakash, jcmvbkbc, iommu, linux-next, Oliver, aneesh.kumar, Suganath Prabu Subramani, Brian King, linuxppc-dev On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote: > On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote: > > Hi Abdul, > > > > I think the problem is that mpt3sas has some convoluted logic to do > > some DMA allocations with a 32-bit coherent mask, and then switches > > to a 63 or 64 bit mask, which is not supported by the DMA API. > > > > Can you try the patch below? > > Thank you Christoph, with the given patch applied the bug is not seen. > > rmmod of mpt3sas driver is successful, no kernel Oops > > Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com> Hi Christoph, I see the patch is under discussion, will this be merged upstream any time soon ? as boot is broken on our machines with out your patch. -- Regard's Abdul Haleem IBM Linux Technology Centre _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver 2020-02-25 6:20 ` Abdul Haleem @ 2020-02-25 6:53 ` Sreekanth Reddy via iommu 2020-03-20 9:59 ` Abdul Haleem 0 siblings, 1 reply; 5+ messages in thread From: Sreekanth Reddy via iommu @ 2020-02-25 6:53 UTC (permalink / raw) To: Abdul Haleem Cc: sachinp, Chaitra P B, linux-scsi, PDL-MPT-FUSIONLINUX, manvanth, Sathya Prakash, jcmvbkbc, iommu, linux-next, Oliver, aneesh.kumar, Suganath Prabu Subramani, Brian King, linuxppc-dev On Tue, Feb 25, 2020 at 11:51 AM Abdul Haleem <abdhalee@linux.vnet.ibm.com> wrote: > > On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote: > > On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote: > > > Hi Abdul, > > > > > > I think the problem is that mpt3sas has some convoluted logic to do > > > some DMA allocations with a 32-bit coherent mask, and then switches > > > to a 63 or 64 bit mask, which is not supported by the DMA API. > > > > > > Can you try the patch below? > > > > Thank you Christoph, with the given patch applied the bug is not seen. > > > > rmmod of mpt3sas driver is successful, no kernel Oops > > > > Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com> > > Hi Christoph, > > I see the patch is under discussion, will this be merged upstream any > time soon ? as boot is broken on our machines with out your patch. > Hi Abdul, We have posted a new set of patches to fix this issue. This patch set won't change the DMA Mask on the fly and also won't hardcode the DMA mask to 32 bit. [PATCH 0/5] mpt3sas: Fix changing coherent mask after allocation. This patchset will have below patches, Please review and try with this patch set. Suganath Prabu S (5): mpt3sas: Don't change the dma coherent mask after allocations mpt3sas: Rename function name is_MSB_are_same mpt3sas: Code Refactoring. mpt3sas: Handle RDPQ DMA allocation in same 4g region mpt3sas: Update version to 33.101.00.00 Regards, Sreekanth > -- > Regard's > > Abdul Haleem > IBM Linux Technology Centre > > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver 2020-02-25 6:53 ` Sreekanth Reddy via iommu @ 2020-03-20 9:59 ` Abdul Haleem 0 siblings, 0 replies; 5+ messages in thread From: Abdul Haleem @ 2020-03-20 9:59 UTC (permalink / raw) To: Sreekanth Reddy Cc: sachinp, Chaitra P B, linux-scsi, PDL-MPT-FUSIONLINUX, manvanth, Sathya Prakash, jcmvbkbc, iommu, linux-next, Oliver, aneesh.kumar, Suganath Prabu Subramani, Brian King, linuxppc-dev On Tue, 2020-02-25 at 12:23 +0530, Sreekanth Reddy wrote: > On Tue, Feb 25, 2020 at 11:51 AM Abdul Haleem > <abdhalee@linux.vnet.ibm.com> wrote: > > > > On Fri, 2020-01-17 at 18:21 +0530, Abdul Haleem wrote: > > > On Thu, 2020-01-16 at 09:44 -0800, Christoph Hellwig wrote: > > > > Hi Abdul, > > > > > > > > I think the problem is that mpt3sas has some convoluted logic to do > > > > some DMA allocations with a 32-bit coherent mask, and then switches > > > > to a 63 or 64 bit mask, which is not supported by the DMA API. > > > > > > > > Can you try the patch below? > > > > > > Thank you Christoph, with the given patch applied the bug is not seen. > > > > > > rmmod of mpt3sas driver is successful, no kernel Oops > > > > > > Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com> > > > > Hi Christoph, > > > > I see the patch is under discussion, will this be merged upstream any > > time soon ? as boot is broken on our machines with out your patch. > > > > Hi Abdul, > > We have posted a new set of patches to fix this issue. This patch set > won't change the DMA Mask on the fly and also won't hardcode the DMA > mask to 32 bit. > > [PATCH 0/5] mpt3sas: Fix changing coherent mask after allocation. > > This patchset will have below patches, Please review and try with this > patch set. > > Suganath Prabu S (5): > mpt3sas: Don't change the dma coherent mask after allocations > mpt3sas: Rename function name is_MSB_are_same > mpt3sas: Code Refactoring. > mpt3sas: Handle RDPQ DMA allocation in same 4g region > mpt3sas: Update version to 33.101.00.00 Hi Suganath, The above patch fixes the issue, driver is loading and unloading with no kernel oops. Reported-and-tested-by: Abdul Haleem <abdhalee@linux.vnet.ibm.com> -- Regard's Abdul Haleem IBM Linux Technology Centre _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-03-20 10:00 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <1578489498.29952.11.camel@abdul> [not found] ` <1578560245.30409.0.camel@abdul.in.ibm.com> [not found] ` <20200109142218.GA16477@infradead.org> [not found] ` <1578980874.11996.3.camel@abdul.in.ibm.com> 2020-01-16 17:44 ` [linux-next/mainline][bisected 3acac06][ppc] Oops when unloading mpt3sas driver Christoph Hellwig 2020-01-17 12:51 ` Abdul Haleem 2020-02-25 6:20 ` Abdul Haleem 2020-02-25 6:53 ` Sreekanth Reddy via iommu 2020-03-20 9:59 ` Abdul Haleem
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).