* IOMMU breakage on arm64 (was: Re: dma-direct: implement complete bus_dma_mask handling) [not found] <git-mailbomb-linux-master-b4ebe6063204da58e48600b810a97c29ae9e5d12@kernel.org> @ 2018-11-06 19:44 ` Geert Uytterhoeven 0 siblings, 0 replies; 6+ messages in thread From: Geert Uytterhoeven @ 2018-11-06 19:44 UTC (permalink / raw) To: Christoph Hellwig, Will Deacon, Robin Murphy, Joerg Roedel, Magnus Damm Cc: Linux-Renesas, Linux ARM, Linux Kernel Mailing List, Linux IOMMU Hi Christoph et al, On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List <linux-kernel@vger.kernel.org> wrote: > Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 > Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced > Refname: refs/heads/master > Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 > Author: Christoph Hellwig <hch@lst.de> > AuthorDate: Thu Sep 20 14:04:08 2018 +0200 > Committer: Christoph Hellwig <hch@lst.de> > CommitDate: Mon Oct 1 07:28:03 2018 -0700 > > dma-direct: implement complete bus_dma_mask handling > > Instead of rejecting devices with a too small bus_dma_mask we can handle > by taking the bus dma_mask into account for allocations and bounce > buffering decisions. > > Signed-off-by: Christoph Hellwig <hch@lst.de> I have bisected the following crash to the above commit: ipmmu-vmsa e67b0000.mmu: Cannot accommodate DMA translation for IOMMU page tables sata_rcar ee300000.sata: Unable to initialize IPMMU context iommu: Failed to add device ee300000.sata to group 0: -22 Unable to handle kernel NULL pointer dereference at virtual address 0000000000000038 Mem abort info: ESR = 0x96000004 Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 Data abort info: ISV = 0, ISS = 0x00000004 CM = 0, WnR = 0 [0000000000000038] user address but active_mm is swapper Internal error: Oops: 96000004 [#1] PREEMPT SMP CPU: 2 PID: 51 Comm: kworker/2:1 Not tainted 4.20.0-rc1-arm64-renesas-dirty #74 Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT) Workqueue: events deferred_probe_work_func pstate: 60000005 (nZCv daif -PAN -UAO) pc : ipmmu_domain_free+0x1c/0xa0 lr : ipmmu_domain_free+0x14/0xa0 sp : ffff000009c9b990 x29: ffff000009c9b990 x28: 0000000000000000 x27: ffff000008dff000 x26: 0000000000000000 x25: ffff000008dd9000 x24: 0000000000000014 x23: ffff8006fffdbb20 x22: ffff000008dff000 x21: ffff8006f898e680 x20: ffff8006f8fa6c00 x19: ffff8006f8fa6c08 x18: 0000000000000037 x17: 0000000000000020 x16: ffff000008a8f780 x15: ffff8006fb096f10 x14: ffff8006f93731c8 x13: 0000000000000000 x12: ffff8006fb096f10 x11: ffff8006f9372fe8 x10: ffff000008dff708 x9 : ffff8006fb096f50 x8 : ffff000008dff708 x7 : ffff0000089f1858 x6 : 0000000000000000 x5 : 0000000000000000 x4 : ffff8006f89a3000 x3 : 00008006f7131000 x2 : ffff8006fb2b5700 x1 : 0000000000000000 x0 : 0000000000000028 Process kworker/2:1 (pid: 51, stack limit = 0x(____ptrval____)) Call trace: ipmmu_domain_free+0x1c/0xa0 iommu_group_release+0x48/0x68 kobject_put+0x74/0xe8 kobject_del.part.0+0x3c/0x50 kobject_put+0x60/0xe8 iommu_group_get_for_dev+0xa8/0x1f0 ipmmu_add_device+0x1c/0x40 of_iommu_configure+0x118/0x190 of_dma_configure+0xcc/0x1f0 platform_dma_configure+0x18/0x28 really_probe+0x94/0x2a8 driver_probe_device+0x58/0x100 __device_attach_driver+0x90/0xd0 bus_for_each_drv+0x64/0xc8 __device_attach+0xd8/0x130 device_initial_probe+0x10/0x18 bus_probe_device+0x98/0xa0 deferred_probe_work_func+0x74/0xb0 process_one_work+0x294/0x6f0 worker_thread+0x238/0x460 kthread+0x120/0x128 ret_from_fork+0x10/0x1c Code: aa0003f3 97ffeb1a d1002274 f85f8261 (f9401c20) ---[ end trace 4c46c7fd7cd07245 ]--- Reproducing on v4.20-rc1 requires CONFIG_IPMMU_VMSA=y, and whitelisting SATA for IOMMU use (https://patchwork.kernel.org/patch/10544059/). For older versions you also have to backport commit 3a0832d093693ede ("arm64: dts: renesas: salvator-xs: enable SATA"). Actually there are two issues at hand: 1) drivers/iommu/io-pgtable-arm.c:__arm_lpae_alloc_pages() allocates memory (above 4 GiB) and maps it for DMA use, but it is rejected due to: dma_map_single(dev, pages, size, DMA_TO_DEVICE) != virt_to_phys(pages) > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -27,7 +27,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) > if (!dev->dma_mask) > return false; > > - return addr + size - 1 <= *dev->dma_mask; > + return addr + size - 1 <= > + min_not_zero(*dev->dma_mask, dev->bus_dma_mask); *dev->dma_mask = 0xffffffffff (40-bit) dev->bus_dma_mask = 0xffffffff (32-bit) Hence before, we had (in __arm_lpae_alloc_pages()): arm-lpae io-pgtable: pages = ffff8006f88f3000 arm-lpae io-pgtable: dma = 0x00000007388f3000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x00000007388f3000 After this change, we have: arm-lpae io-pgtable: pages = ffff8006f882b000 arm-lpae io-pgtable: dma = 0x0000000074009000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x000000073882b000 And SATA runs without using the IOMMU. 2) The Renesas IPMMU driver doesn't handle the above failure well, leading to a NULL pointer dereference. This can be fixed using the (gmail-whitespace-damaged) patch below: --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -635,7 +635,8 @@ static void ipmmu_domain_free(struct iommu_domain *io_domain) * been detached. */ iommu_put_dma_cookie(io_domain); - ipmmu_domain_destroy_context(domain); + if (domain->mmu) + ipmmu_domain_destroy_context(domain); free_io_pgtable_ops(domain->iop); kfree(domain); } I expect drivers/iommu/arm-smmu-v3.c and drivers/iommu/arm-smmu.c need similar fixes. I didn't check all drivers, but e.g. drivers/iommu/amd_iommu.c has a similar check. Does the IOMMU work on other arm64 platforms in v4.20-rc1? Thanks for your comments! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 6+ messages in thread
* IOMMU breakage on arm64 (was: Re: dma-direct: implement complete bus_dma_mask handling) @ 2018-11-06 19:44 ` Geert Uytterhoeven 0 siblings, 0 replies; 6+ messages in thread From: Geert Uytterhoeven @ 2018-11-06 19:44 UTC (permalink / raw) To: linux-arm-kernel Hi Christoph et al, On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List <linux-kernel@vger.kernel.org> wrote: > Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 > Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced > Refname: refs/heads/master > Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 > Author: Christoph Hellwig <hch@lst.de> > AuthorDate: Thu Sep 20 14:04:08 2018 +0200 > Committer: Christoph Hellwig <hch@lst.de> > CommitDate: Mon Oct 1 07:28:03 2018 -0700 > > dma-direct: implement complete bus_dma_mask handling > > Instead of rejecting devices with a too small bus_dma_mask we can handle > by taking the bus dma_mask into account for allocations and bounce > buffering decisions. > > Signed-off-by: Christoph Hellwig <hch@lst.de> I have bisected the following crash to the above commit: ipmmu-vmsa e67b0000.mmu: Cannot accommodate DMA translation for IOMMU page tables sata_rcar ee300000.sata: Unable to initialize IPMMU context iommu: Failed to add device ee300000.sata to group 0: -22 Unable to handle kernel NULL pointer dereference at virtual address 0000000000000038 Mem abort info: ESR = 0x96000004 Exception class = DABT (current EL), IL = 32 bits SET = 0, FnV = 0 EA = 0, S1PTW = 0 Data abort info: ISV = 0, ISS = 0x00000004 CM = 0, WnR = 0 [0000000000000038] user address but active_mm is swapper Internal error: Oops: 96000004 [#1] PREEMPT SMP CPU: 2 PID: 51 Comm: kworker/2:1 Not tainted 4.20.0-rc1-arm64-renesas-dirty #74 Hardware name: Renesas Salvator-X 2nd version board based on r8a7795 ES2.0+ (DT) Workqueue: events deferred_probe_work_func pstate: 60000005 (nZCv daif -PAN -UAO) pc : ipmmu_domain_free+0x1c/0xa0 lr : ipmmu_domain_free+0x14/0xa0 sp : ffff000009c9b990 x29: ffff000009c9b990 x28: 0000000000000000 x27: ffff000008dff000 x26: 0000000000000000 x25: ffff000008dd9000 x24: 0000000000000014 x23: ffff8006fffdbb20 x22: ffff000008dff000 x21: ffff8006f898e680 x20: ffff8006f8fa6c00 x19: ffff8006f8fa6c08 x18: 0000000000000037 x17: 0000000000000020 x16: ffff000008a8f780 x15: ffff8006fb096f10 x14: ffff8006f93731c8 x13: 0000000000000000 x12: ffff8006fb096f10 x11: ffff8006f9372fe8 x10: ffff000008dff708 x9 : ffff8006fb096f50 x8 : ffff000008dff708 x7 : ffff0000089f1858 x6 : 0000000000000000 x5 : 0000000000000000 x4 : ffff8006f89a3000 x3 : 00008006f7131000 x2 : ffff8006fb2b5700 x1 : 0000000000000000 x0 : 0000000000000028 Process kworker/2:1 (pid: 51, stack limit = 0x(____ptrval____)) Call trace: ipmmu_domain_free+0x1c/0xa0 iommu_group_release+0x48/0x68 kobject_put+0x74/0xe8 kobject_del.part.0+0x3c/0x50 kobject_put+0x60/0xe8 iommu_group_get_for_dev+0xa8/0x1f0 ipmmu_add_device+0x1c/0x40 of_iommu_configure+0x118/0x190 of_dma_configure+0xcc/0x1f0 platform_dma_configure+0x18/0x28 really_probe+0x94/0x2a8 driver_probe_device+0x58/0x100 __device_attach_driver+0x90/0xd0 bus_for_each_drv+0x64/0xc8 __device_attach+0xd8/0x130 device_initial_probe+0x10/0x18 bus_probe_device+0x98/0xa0 deferred_probe_work_func+0x74/0xb0 process_one_work+0x294/0x6f0 worker_thread+0x238/0x460 kthread+0x120/0x128 ret_from_fork+0x10/0x1c Code: aa0003f3 97ffeb1a d1002274 f85f8261 (f9401c20) ---[ end trace 4c46c7fd7cd07245 ]--- Reproducing on v4.20-rc1 requires CONFIG_IPMMU_VMSA=y, and whitelisting SATA for IOMMU use (https://patchwork.kernel.org/patch/10544059/). For older versions you also have to backport commit 3a0832d093693ede ("arm64: dts: renesas: salvator-xs: enable SATA"). Actually there are two issues at hand: 1) drivers/iommu/io-pgtable-arm.c:__arm_lpae_alloc_pages() allocates memory (above 4 GiB) and maps it for DMA use, but it is rejected due to: dma_map_single(dev, pages, size, DMA_TO_DEVICE) != virt_to_phys(pages) > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -27,7 +27,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) > if (!dev->dma_mask) > return false; > > - return addr + size - 1 <= *dev->dma_mask; > + return addr + size - 1 <= > + min_not_zero(*dev->dma_mask, dev->bus_dma_mask); *dev->dma_mask = 0xffffffffff (40-bit) dev->bus_dma_mask = 0xffffffff (32-bit) Hence before, we had (in __arm_lpae_alloc_pages()): arm-lpae io-pgtable: pages = ffff8006f88f3000 arm-lpae io-pgtable: dma = 0x00000007388f3000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x00000007388f3000 After this change, we have: arm-lpae io-pgtable: pages = ffff8006f882b000 arm-lpae io-pgtable: dma = 0x0000000074009000 arm-lpae io-pgtable: virt_to_phys(pages) = 0x000000073882b000 And SATA runs without using the IOMMU. 2) The Renesas IPMMU driver doesn't handle the above failure well, leading to a NULL pointer dereference. This can be fixed using the (gmail-whitespace-damaged) patch below: --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -635,7 +635,8 @@ static void ipmmu_domain_free(struct iommu_domain *io_domain) * been detached. */ iommu_put_dma_cookie(io_domain); - ipmmu_domain_destroy_context(domain); + if (domain->mmu) + ipmmu_domain_destroy_context(domain); free_io_pgtable_ops(domain->iop); kfree(domain); } I expect drivers/iommu/arm-smmu-v3.c and drivers/iommu/arm-smmu.c need similar fixes. I didn't check all drivers, but e.g. drivers/iommu/amd_iommu.c has a similar check. Does the IOMMU work on other arm64 platforms in v4.20-rc1? Thanks for your comments! Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: IOMMU breakage on arm64 2018-11-06 19:44 ` Geert Uytterhoeven @ 2018-11-06 20:20 ` Robin Murphy -1 siblings, 0 replies; 6+ messages in thread From: Robin Murphy @ 2018-11-06 20:20 UTC (permalink / raw) To: Geert Uytterhoeven, Christoph Hellwig, Will Deacon, Joerg Roedel, Magnus Damm Cc: Linux-Renesas, Linux ARM, Linux Kernel Mailing List, Linux IOMMU Hi Geert, On 2018-11-06 7:44 pm, Geert Uytterhoeven wrote: > Hi Christoph et al, > > On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List > <linux-kernel@vger.kernel.org> wrote: >> Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 >> Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced >> Refname: refs/heads/master >> Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 >> Author: Christoph Hellwig <hch@lst.de> >> AuthorDate: Thu Sep 20 14:04:08 2018 +0200 >> Committer: Christoph Hellwig <hch@lst.de> >> CommitDate: Mon Oct 1 07:28:03 2018 -0700 >> >> dma-direct: implement complete bus_dma_mask handling >> >> Instead of rejecting devices with a too small bus_dma_mask we can handle >> by taking the bus dma_mask into account for allocations and bounce >> buffering decisions. >> >> Signed-off-by: Christoph Hellwig <hch@lst.de> > > I have bisected the following crash to the above commit: I think that commit mostly just changes the presentation of my underlying cockup - see here for what should fix it: https://patchwork.kernel.org/patch/10670177/ I have a feeling we've ironed out crash-on-early-domain-free bugs in the SMMU drivers already - arm-smmu certainly has an early return in arm_smmu_destroy_domain_context() which should behave exactly like your diff below, while I think arm-smmu-v3 gets away with it by virtue of smmu_domain->cfg being unset, but I'll double-check that when I'm fresh tomorrow (Jean-Philippe reported SMMUv3 hitting the DMA thing to me internally, but didn't mention any crash). Thanks for the report, Robin. > ipmmu-vmsa e67b0000.mmu: Cannot accommodate DMA translation for > IOMMU page tables > sata_rcar ee300000.sata: Unable to initialize IPMMU context > iommu: Failed to add device ee300000.sata to group 0: -22 > Unable to handle kernel NULL pointer dereference at virtual > address 0000000000000038 > Mem abort info: > ESR = 0x96000004 > Exception class = DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > Data abort info: > ISV = 0, ISS = 0x00000004 > CM = 0, WnR = 0 > [0000000000000038] user address but active_mm is swapper > Internal error: Oops: 96000004 [#1] PREEMPT SMP > CPU: 2 PID: 51 Comm: kworker/2:1 Not tainted > 4.20.0-rc1-arm64-renesas-dirty #74 > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Workqueue: events deferred_probe_work_func > pstate: 60000005 (nZCv daif -PAN -UAO) > pc : ipmmu_domain_free+0x1c/0xa0 > lr : ipmmu_domain_free+0x14/0xa0 > sp : ffff000009c9b990 > x29: ffff000009c9b990 x28: 0000000000000000 > x27: ffff000008dff000 x26: 0000000000000000 > x25: ffff000008dd9000 x24: 0000000000000014 > x23: ffff8006fffdbb20 x22: ffff000008dff000 > x21: ffff8006f898e680 x20: ffff8006f8fa6c00 > x19: ffff8006f8fa6c08 x18: 0000000000000037 > x17: 0000000000000020 x16: ffff000008a8f780 > x15: ffff8006fb096f10 x14: ffff8006f93731c8 > x13: 0000000000000000 x12: ffff8006fb096f10 > x11: ffff8006f9372fe8 x10: ffff000008dff708 > x9 : ffff8006fb096f50 x8 : ffff000008dff708 > x7 : ffff0000089f1858 x6 : 0000000000000000 > x5 : 0000000000000000 x4 : ffff8006f89a3000 > x3 : 00008006f7131000 x2 : ffff8006fb2b5700 > x1 : 0000000000000000 x0 : 0000000000000028 > Process kworker/2:1 (pid: 51, stack limit = 0x(____ptrval____)) > Call trace: > ipmmu_domain_free+0x1c/0xa0 > iommu_group_release+0x48/0x68 > kobject_put+0x74/0xe8 > kobject_del.part.0+0x3c/0x50 > kobject_put+0x60/0xe8 > iommu_group_get_for_dev+0xa8/0x1f0 > ipmmu_add_device+0x1c/0x40 > of_iommu_configure+0x118/0x190 > of_dma_configure+0xcc/0x1f0 > platform_dma_configure+0x18/0x28 > really_probe+0x94/0x2a8 > driver_probe_device+0x58/0x100 > __device_attach_driver+0x90/0xd0 > bus_for_each_drv+0x64/0xc8 > __device_attach+0xd8/0x130 > device_initial_probe+0x10/0x18 > bus_probe_device+0x98/0xa0 > deferred_probe_work_func+0x74/0xb0 > process_one_work+0x294/0x6f0 > worker_thread+0x238/0x460 > kthread+0x120/0x128 > ret_from_fork+0x10/0x1c > Code: aa0003f3 97ffeb1a d1002274 f85f8261 (f9401c20) > ---[ end trace 4c46c7fd7cd07245 ]--- > > Reproducing on v4.20-rc1 requires CONFIG_IPMMU_VMSA=y, and whitelisting > SATA for IOMMU use (https://patchwork.kernel.org/patch/10544059/). > For older versions you also have to backport commit 3a0832d093693ede ("arm64: > dts: renesas: salvator-xs: enable SATA"). > > Actually there are two issues at hand: > > 1) drivers/iommu/io-pgtable-arm.c:__arm_lpae_alloc_pages() allocates > memory (above 4 GiB) and maps it for DMA use, but it is rejected due > to: > > dma_map_single(dev, pages, size, DMA_TO_DEVICE) != virt_to_phys(pages) > >> --- a/include/linux/dma-direct.h >> +++ b/include/linux/dma-direct.h >> @@ -27,7 +27,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) >> if (!dev->dma_mask) >> return false; >> >> - return addr + size - 1 <= *dev->dma_mask; >> + return addr + size - 1 <= >> + min_not_zero(*dev->dma_mask, dev->bus_dma_mask); > > *dev->dma_mask = 0xffffffffff (40-bit) > dev->bus_dma_mask = 0xffffffff (32-bit) > > Hence before, we had (in __arm_lpae_alloc_pages()): > > arm-lpae io-pgtable: pages = ffff8006f88f3000 > arm-lpae io-pgtable: dma = 0x00000007388f3000 > arm-lpae io-pgtable: virt_to_phys(pages) = 0x00000007388f3000 > > After this change, we have: > > arm-lpae io-pgtable: pages = ffff8006f882b000 > arm-lpae io-pgtable: dma = 0x0000000074009000 > arm-lpae io-pgtable: virt_to_phys(pages) = 0x000000073882b000 > > And SATA runs without using the IOMMU. > > 2) The Renesas IPMMU driver doesn't handle the above failure well, > leading to a NULL pointer dereference. > This can be fixed using the (gmail-whitespace-damaged) patch below: > > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -635,7 +635,8 @@ static void ipmmu_domain_free(struct iommu_domain > *io_domain) > * been detached. > */ > iommu_put_dma_cookie(io_domain); > - ipmmu_domain_destroy_context(domain); > + if (domain->mmu) > + ipmmu_domain_destroy_context(domain); > free_io_pgtable_ops(domain->iop); > kfree(domain); > } > > I expect drivers/iommu/arm-smmu-v3.c and drivers/iommu/arm-smmu.c > need similar fixes. > I didn't check all drivers, but e.g. drivers/iommu/amd_iommu.c has > a similar check. > > Does the IOMMU work on other arm64 platforms in v4.20-rc1? > > Thanks for your comments! > > Gr{oetje,eeting}s, > > Geert > ^ permalink raw reply [flat|nested] 6+ messages in thread
* IOMMU breakage on arm64 @ 2018-11-06 20:20 ` Robin Murphy 0 siblings, 0 replies; 6+ messages in thread From: Robin Murphy @ 2018-11-06 20:20 UTC (permalink / raw) To: linux-arm-kernel Hi Geert, On 2018-11-06 7:44 pm, Geert Uytterhoeven wrote: > Hi Christoph et al, > > On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List > <linux-kernel@vger.kernel.org> wrote: >> Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 >> Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced >> Refname: refs/heads/master >> Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 >> Author: Christoph Hellwig <hch@lst.de> >> AuthorDate: Thu Sep 20 14:04:08 2018 +0200 >> Committer: Christoph Hellwig <hch@lst.de> >> CommitDate: Mon Oct 1 07:28:03 2018 -0700 >> >> dma-direct: implement complete bus_dma_mask handling >> >> Instead of rejecting devices with a too small bus_dma_mask we can handle >> by taking the bus dma_mask into account for allocations and bounce >> buffering decisions. >> >> Signed-off-by: Christoph Hellwig <hch@lst.de> > > I have bisected the following crash to the above commit: I think that commit mostly just changes the presentation of my underlying cockup - see here for what should fix it: https://patchwork.kernel.org/patch/10670177/ I have a feeling we've ironed out crash-on-early-domain-free bugs in the SMMU drivers already - arm-smmu certainly has an early return in arm_smmu_destroy_domain_context() which should behave exactly like your diff below, while I think arm-smmu-v3 gets away with it by virtue of smmu_domain->cfg being unset, but I'll double-check that when I'm fresh tomorrow (Jean-Philippe reported SMMUv3 hitting the DMA thing to me internally, but didn't mention any crash). Thanks for the report, Robin. > ipmmu-vmsa e67b0000.mmu: Cannot accommodate DMA translation for > IOMMU page tables > sata_rcar ee300000.sata: Unable to initialize IPMMU context > iommu: Failed to add device ee300000.sata to group 0: -22 > Unable to handle kernel NULL pointer dereference at virtual > address 0000000000000038 > Mem abort info: > ESR = 0x96000004 > Exception class = DABT (current EL), IL = 32 bits > SET = 0, FnV = 0 > EA = 0, S1PTW = 0 > Data abort info: > ISV = 0, ISS = 0x00000004 > CM = 0, WnR = 0 > [0000000000000038] user address but active_mm is swapper > Internal error: Oops: 96000004 [#1] PREEMPT SMP > CPU: 2 PID: 51 Comm: kworker/2:1 Not tainted > 4.20.0-rc1-arm64-renesas-dirty #74 > Hardware name: Renesas Salvator-X 2nd version board based on > r8a7795 ES2.0+ (DT) > Workqueue: events deferred_probe_work_func > pstate: 60000005 (nZCv daif -PAN -UAO) > pc : ipmmu_domain_free+0x1c/0xa0 > lr : ipmmu_domain_free+0x14/0xa0 > sp : ffff000009c9b990 > x29: ffff000009c9b990 x28: 0000000000000000 > x27: ffff000008dff000 x26: 0000000000000000 > x25: ffff000008dd9000 x24: 0000000000000014 > x23: ffff8006fffdbb20 x22: ffff000008dff000 > x21: ffff8006f898e680 x20: ffff8006f8fa6c00 > x19: ffff8006f8fa6c08 x18: 0000000000000037 > x17: 0000000000000020 x16: ffff000008a8f780 > x15: ffff8006fb096f10 x14: ffff8006f93731c8 > x13: 0000000000000000 x12: ffff8006fb096f10 > x11: ffff8006f9372fe8 x10: ffff000008dff708 > x9 : ffff8006fb096f50 x8 : ffff000008dff708 > x7 : ffff0000089f1858 x6 : 0000000000000000 > x5 : 0000000000000000 x4 : ffff8006f89a3000 > x3 : 00008006f7131000 x2 : ffff8006fb2b5700 > x1 : 0000000000000000 x0 : 0000000000000028 > Process kworker/2:1 (pid: 51, stack limit = 0x(____ptrval____)) > Call trace: > ipmmu_domain_free+0x1c/0xa0 > iommu_group_release+0x48/0x68 > kobject_put+0x74/0xe8 > kobject_del.part.0+0x3c/0x50 > kobject_put+0x60/0xe8 > iommu_group_get_for_dev+0xa8/0x1f0 > ipmmu_add_device+0x1c/0x40 > of_iommu_configure+0x118/0x190 > of_dma_configure+0xcc/0x1f0 > platform_dma_configure+0x18/0x28 > really_probe+0x94/0x2a8 > driver_probe_device+0x58/0x100 > __device_attach_driver+0x90/0xd0 > bus_for_each_drv+0x64/0xc8 > __device_attach+0xd8/0x130 > device_initial_probe+0x10/0x18 > bus_probe_device+0x98/0xa0 > deferred_probe_work_func+0x74/0xb0 > process_one_work+0x294/0x6f0 > worker_thread+0x238/0x460 > kthread+0x120/0x128 > ret_from_fork+0x10/0x1c > Code: aa0003f3 97ffeb1a d1002274 f85f8261 (f9401c20) > ---[ end trace 4c46c7fd7cd07245 ]--- > > Reproducing on v4.20-rc1 requires CONFIG_IPMMU_VMSA=y, and whitelisting > SATA for IOMMU use (https://patchwork.kernel.org/patch/10544059/). > For older versions you also have to backport commit 3a0832d093693ede ("arm64: > dts: renesas: salvator-xs: enable SATA"). > > Actually there are two issues at hand: > > 1) drivers/iommu/io-pgtable-arm.c:__arm_lpae_alloc_pages() allocates > memory (above 4 GiB) and maps it for DMA use, but it is rejected due > to: > > dma_map_single(dev, pages, size, DMA_TO_DEVICE) != virt_to_phys(pages) > >> --- a/include/linux/dma-direct.h >> +++ b/include/linux/dma-direct.h >> @@ -27,7 +27,8 @@ static inline bool dma_capable(struct device *dev, dma_addr_t addr, size_t size) >> if (!dev->dma_mask) >> return false; >> >> - return addr + size - 1 <= *dev->dma_mask; >> + return addr + size - 1 <= >> + min_not_zero(*dev->dma_mask, dev->bus_dma_mask); > > *dev->dma_mask = 0xffffffffff (40-bit) > dev->bus_dma_mask = 0xffffffff (32-bit) > > Hence before, we had (in __arm_lpae_alloc_pages()): > > arm-lpae io-pgtable: pages = ffff8006f88f3000 > arm-lpae io-pgtable: dma = 0x00000007388f3000 > arm-lpae io-pgtable: virt_to_phys(pages) = 0x00000007388f3000 > > After this change, we have: > > arm-lpae io-pgtable: pages = ffff8006f882b000 > arm-lpae io-pgtable: dma = 0x0000000074009000 > arm-lpae io-pgtable: virt_to_phys(pages) = 0x000000073882b000 > > And SATA runs without using the IOMMU. > > 2) The Renesas IPMMU driver doesn't handle the above failure well, > leading to a NULL pointer dereference. > This can be fixed using the (gmail-whitespace-damaged) patch below: > > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -635,7 +635,8 @@ static void ipmmu_domain_free(struct iommu_domain > *io_domain) > * been detached. > */ > iommu_put_dma_cookie(io_domain); > - ipmmu_domain_destroy_context(domain); > + if (domain->mmu) > + ipmmu_domain_destroy_context(domain); > free_io_pgtable_ops(domain->iop); > kfree(domain); > } > > I expect drivers/iommu/arm-smmu-v3.c and drivers/iommu/arm-smmu.c > need similar fixes. > I didn't check all drivers, but e.g. drivers/iommu/amd_iommu.c has > a similar check. > > Does the IOMMU work on other arm64 platforms in v4.20-rc1? > > Thanks for your comments! > > Gr{oetje,eeting}s, > > Geert > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: IOMMU breakage on arm64 2018-11-06 20:20 ` Robin Murphy @ 2018-11-07 8:42 ` Geert Uytterhoeven -1 siblings, 0 replies; 6+ messages in thread From: Geert Uytterhoeven @ 2018-11-07 8:42 UTC (permalink / raw) To: Robin Murphy Cc: Christoph Hellwig, Will Deacon, Joerg Roedel, Magnus Damm, Linux-Renesas, Linux ARM, Linux Kernel Mailing List, Linux IOMMU Hi Robin, On Tue, Nov 6, 2018 at 9:20 PM Robin Murphy <robin.murphy@arm.com> wrote: > On 2018-11-06 7:44 pm, Geert Uytterhoeven wrote: > > On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List > > <linux-kernel@vger.kernel.org> wrote: > >> Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 > >> Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced > >> Refname: refs/heads/master > >> Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 > >> Author: Christoph Hellwig <hch@lst.de> > >> AuthorDate: Thu Sep 20 14:04:08 2018 +0200 > >> Committer: Christoph Hellwig <hch@lst.de> > >> CommitDate: Mon Oct 1 07:28:03 2018 -0700 > >> > >> dma-direct: implement complete bus_dma_mask handling > >> > >> Instead of rejecting devices with a too small bus_dma_mask we can handle > >> by taking the bus dma_mask into account for allocations and bounce > >> buffering decisions. > >> > >> Signed-off-by: Christoph Hellwig <hch@lst.de> > > > > I have bisected the following crash to the above commit: > > I think that commit mostly just changes the presentation of my > underlying cockup - see here for what should fix it: > https://patchwork.kernel.org/patch/10670177/ Thanks, that fixed the issue. > I have a feeling we've ironed out crash-on-early-domain-free bugs in the > SMMU drivers already - arm-smmu certainly has an early return in > arm_smmu_destroy_domain_context() which should behave exactly like your > diff below, while I think arm-smmu-v3 gets away with it by virtue of > smmu_domain->cfg being unset, but I'll double-check that when I'm fresh > tomorrow (Jean-Philippe reported SMMUv3 hitting the DMA thing to me > internally, but didn't mention any crash). OK, I'll send a proper patch for the ipmmu-vmsa driver. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 6+ messages in thread
* IOMMU breakage on arm64 @ 2018-11-07 8:42 ` Geert Uytterhoeven 0 siblings, 0 replies; 6+ messages in thread From: Geert Uytterhoeven @ 2018-11-07 8:42 UTC (permalink / raw) To: linux-arm-kernel Hi Robin, On Tue, Nov 6, 2018 at 9:20 PM Robin Murphy <robin.murphy@arm.com> wrote: > On 2018-11-06 7:44 pm, Geert Uytterhoeven wrote: > > On Tue, Oct 23, 2018 at 1:40 AM Linux Kernel Mailing List > > <linux-kernel@vger.kernel.org> wrote: > >> Commit: b4ebe6063204da58e48600b810a97c29ae9e5d12 > >> Parent: 7d21ee4c719f00896767ce19c4c01a56374c2ced > >> Refname: refs/heads/master > >> Web: https://git.kernel.org/torvalds/c/b4ebe6063204da58e48600b810a97c29ae9e5d12 > >> Author: Christoph Hellwig <hch@lst.de> > >> AuthorDate: Thu Sep 20 14:04:08 2018 +0200 > >> Committer: Christoph Hellwig <hch@lst.de> > >> CommitDate: Mon Oct 1 07:28:03 2018 -0700 > >> > >> dma-direct: implement complete bus_dma_mask handling > >> > >> Instead of rejecting devices with a too small bus_dma_mask we can handle > >> by taking the bus dma_mask into account for allocations and bounce > >> buffering decisions. > >> > >> Signed-off-by: Christoph Hellwig <hch@lst.de> > > > > I have bisected the following crash to the above commit: > > I think that commit mostly just changes the presentation of my > underlying cockup - see here for what should fix it: > https://patchwork.kernel.org/patch/10670177/ Thanks, that fixed the issue. > I have a feeling we've ironed out crash-on-early-domain-free bugs in the > SMMU drivers already - arm-smmu certainly has an early return in > arm_smmu_destroy_domain_context() which should behave exactly like your > diff below, while I think arm-smmu-v3 gets away with it by virtue of > smmu_domain->cfg being unset, but I'll double-check that when I'm fresh > tomorrow (Jean-Philippe reported SMMUv3 hitting the DMA thing to me > internally, but didn't mention any crash). OK, I'll send a proper patch for the ipmmu-vmsa driver. Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert at linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2018-11-07 8:42 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <git-mailbomb-linux-master-b4ebe6063204da58e48600b810a97c29ae9e5d12@kernel.org>
2018-11-06 19:44 ` IOMMU breakage on arm64 (was: Re: dma-direct: implement complete bus_dma_mask handling) Geert Uytterhoeven
2018-11-06 19:44 ` Geert Uytterhoeven
2018-11-06 20:20 ` IOMMU breakage on arm64 Robin Murphy
2018-11-06 20:20 ` Robin Murphy
2018-11-07 8:42 ` Geert Uytterhoeven
2018-11-07 8:42 ` Geert Uytterhoeven
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.