* RFC on Kdump and PCIe on ARM64 @ 2018-03-01 17:44 Sinan Kaya 2018-03-01 19:05 ` Bjorn Helgaas 0 siblings, 1 reply; 10+ messages in thread From: Sinan Kaya @ 2018-03-01 17:44 UTC (permalink / raw) To: Lorenzo Pieralisi, Will Deacon, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec Hi, We are seeing IOMMU faults when booting the kdump kernel on ARM64. [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 This is Nate's interpretation of the fault: "The PCI device is sending transactions just after the SMMU was reset/reinitialized which is problematic because the device has not yet been added to the SMMU and thus should not be doing *any* DMA. DMA from the PCI devices should be quiesced prior to starting the crashdump kernel or you risk overwriting portions of memory you meant to preserve. In this case the SMMU was actually doing you a favor by blocking these errant DMA operations!!" I think this makes sense especially for the IOMMU enabled case on the host where an IOVA can overlap with the region of memory kdump reserved for itself. Apparently, there has been similar concerns in the past. https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf and was not addressed globally due to IOMMU+PCI driver ordering issues and bugs in HW due to hot reset. https://lkml.org/lkml/2012/8/3/160 Hot reset as mentioned is destructive and may not be the best implementation choice. However, most of the modern endpoints support PCIE function level reset. One other solution is for SMMUv3 driver to reserve the kdump used IOVA addresses. Another solution is for the SMMUv3 driver to disable PCIe devices behind the SMMU if it see SMMU is already enabled. Appreciate the feedback, Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-01 17:44 RFC on Kdump and PCIe on ARM64 Sinan Kaya @ 2018-03-01 19:05 ` Bjorn Helgaas 2018-03-01 19:19 ` Sinan Kaya 2018-03-02 1:44 ` Baoquan He 0 siblings, 2 replies; 10+ messages in thread From: Bjorn Helgaas @ 2018-03-01 19:05 UTC (permalink / raw) To: Sinan Kaya Cc: Lorenzo Pieralisi, Vikram Sethi, Linux PCI, Joerg Roedel, Will Deacon, iommu, shankerd, linux-arm Mailing List, Goel, Sameer, David Woodhouse, kexec, Nate Watterson [+cc Joerg, David, iommu list] On Thu, Mar 01, 2018 at 12:44:26PM -0500, Sinan Kaya wrote: > Hi, > > We are seeing IOMMU faults when booting the kdump kernel on ARM64. > > [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: > [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 > [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > This is Nate's interpretation of the fault: > > "The PCI device is sending transactions just after the SMMU was > reset/reinitialized which is problematic because the device has not > yet been added to the SMMU and thus should not be doing *any* DMA. > DMA from the PCI devices should be quiesced prior to starting the > crashdump kernel or you risk overwriting portions of memory you > meant to preserve. In this case the SMMU was actually doing you a > favor by blocking these errant DMA operations!!" > > I think this makes sense especially for the IOMMU enabled case on > the host where an IOVA can overlap with the region of memory kdump > reserved for itself. > > Apparently, there has been similar concerns in the past. > > https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf > > and was not addressed globally due to IOMMU+PCI driver ordering > issues and bugs in HW due to hot reset. > > https://lkml.org/lkml/2012/8/3/160 > > Hot reset as mentioned is destructive and may not be the best > implementation choice. However, most of the modern endpoints > support PCIE function level reset. > > One other solution is for SMMUv3 driver to reserve the kdump used > IOVA addresses. > > Another solution is for the SMMUv3 driver to disable PCIe devices > behind the SMMU if it see SMMU is already enabled. What problem are you trying to solve? If the IOMMU is blocking DMA after the kdump kernel starts up, that sounds like the desired behavior. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-01 19:05 ` Bjorn Helgaas @ 2018-03-01 19:19 ` Sinan Kaya 2018-03-02 0:03 ` Bjorn Helgaas 2018-03-02 1:44 ` Baoquan He 1 sibling, 1 reply; 10+ messages in thread From: Sinan Kaya @ 2018-03-01 19:19 UTC (permalink / raw) To: Bjorn Helgaas Cc: Lorenzo Pieralisi, Will Deacon, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse On 3/1/2018 2:05 PM, Bjorn Helgaas wrote: > [+cc Joerg, David, iommu list] > > On Thu, Mar 01, 2018 at 12:44:26PM -0500, Sinan Kaya wrote: >> Hi, >> >> We are seeing IOMMU faults when booting the kdump kernel on ARM64. >> >> [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: >> [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 >> [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 >> [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 >> [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 >> >> This is Nate's interpretation of the fault: >> >> "The PCI device is sending transactions just after the SMMU was >> reset/reinitialized which is problematic because the device has not >> yet been added to the SMMU and thus should not be doing *any* DMA. >> DMA from the PCI devices should be quiesced prior to starting the >> crashdump kernel or you risk overwriting portions of memory you >> meant to preserve. In this case the SMMU was actually doing you a >> favor by blocking these errant DMA operations!!" >> >> I think this makes sense especially for the IOMMU enabled case on >> the host where an IOVA can overlap with the region of memory kdump >> reserved for itself. >> >> Apparently, there has been similar concerns in the past. >> >> https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf >> >> and was not addressed globally due to IOMMU+PCI driver ordering >> issues and bugs in HW due to hot reset. >> >> https://lkml.org/lkml/2012/8/3/160 >> >> Hot reset as mentioned is destructive and may not be the best >> implementation choice. However, most of the modern endpoints >> support PCIE function level reset. >> >> One other solution is for SMMUv3 driver to reserve the kdump used >> IOVA addresses. >> >> Another solution is for the SMMUv3 driver to disable PCIe devices >> behind the SMMU if it see SMMU is already enabled. > > What problem are you trying to solve? If the IOMMU is blocking DMA > after the kdump kernel starts up, that sounds like the desired > behavior. > Three issues: 1. I'm seeing a flood of SMMUv3 faults due to adapter using addresses from the previous kernel. This might be OK. 2. When the SMMUv3 driver sees that it is enabled, it resets itself and configures it one more time. [ 7.018304] arm-smmu-v3 arm-smmu-v3.0.auto: ias 44-bit, oas 44-bit (features 0x00001fef) [ 7.026379] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting... >From the moment IOMMU is disabled to the point where IOMMU get enabled again, there is a potential for the PCIE device to corrupt the kdump kernel memory as the bus master and memory enable bits are left enabled. [ 0.000000] crashkernel reserved: 0x000000007fe00000 - 0x00000000ffe00000 (2048 MB) This region happens to overlap with the IOVA addresses that SMMUv3 driver on the main kernel is allocating. IOVA addresses start from 0xFFFFFFFF and get decremented on each allocation. 3. The last one is adapter gets into fuzzy state due to not coming out of clean state in the second time init and being rejected by SMMUv3 multiple times. [ 16.093441] pci 0000:01:00.0: aer_status: 0x00040000, aer_mask: 0x00000000 [ 16.099356] pci 0000:01:00.0: Malformed TLP [ 16.103522] pci 0000:01:00.0: aer_layer=Transaction Layer, aer_agent=Receiver ID [ 16.110900] pci 0000:01:00.0: aer_uncor_severity: 0x00062011 [ 16.116543] pci 0000:01:00.0: TLP Header: 0a00a000 00008100 01010100 00000000 -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-01 19:19 ` Sinan Kaya @ 2018-03-02 0:03 ` Bjorn Helgaas 2018-03-02 10:30 ` Will Deacon 2018-03-02 14:20 ` Sinan Kaya 0 siblings, 2 replies; 10+ messages in thread From: Bjorn Helgaas @ 2018-03-02 0:03 UTC (permalink / raw) To: Sinan Kaya Cc: Lorenzo Pieralisi, Vikram Sethi, Linux PCI, Joerg Roedel, Will Deacon, iommu, shankerd, linux-arm Mailing List, Goel, Sameer, David Woodhouse, kexec, Nate Watterson On Thu, Mar 01, 2018 at 02:19:09PM -0500, Sinan Kaya wrote: > On 3/1/2018 2:05 PM, Bjorn Helgaas wrote: > > On Thu, Mar 01, 2018 at 12:44:26PM -0500, Sinan Kaya wrote: > >> Hi, > >> > >> We are seeing IOMMU faults when booting the kdump kernel on ARM64. > >> > >> [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: > >> [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 > >> [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > >> [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > >> [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > >> > >> This is Nate's interpretation of the fault: > >> > >> "The PCI device is sending transactions just after the SMMU was > >> reset/reinitialized which is problematic because the device has not > >> yet been added to the SMMU and thus should not be doing *any* DMA. > >> DMA from the PCI devices should be quiesced prior to starting the > >> crashdump kernel or you risk overwriting portions of memory you > >> meant to preserve. In this case the SMMU was actually doing you a > >> favor by blocking these errant DMA operations!!" > >> > >> I think this makes sense especially for the IOMMU enabled case on > >> the host where an IOVA can overlap with the region of memory kdump > >> reserved for itself. > >> > >> Apparently, there has been similar concerns in the past. > >> > >> https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf > >> > >> and was not addressed globally due to IOMMU+PCI driver ordering > >> issues and bugs in HW due to hot reset. > >> > >> https://lkml.org/lkml/2012/8/3/160 > >> > >> Hot reset as mentioned is destructive and may not be the best > >> implementation choice. However, most of the modern endpoints > >> support PCIE function level reset. > >> > >> One other solution is for SMMUv3 driver to reserve the kdump used > >> IOVA addresses. > >> > >> Another solution is for the SMMUv3 driver to disable PCIe devices > >> behind the SMMU if it see SMMU is already enabled. > > > > What problem are you trying to solve? If the IOMMU is blocking DMA > > after the kdump kernel starts up, that sounds like the desired > > behavior. > > > > Three issues: > 1. I'm seeing a flood of SMMUv3 faults due to adapter using > addresses from the previous kernel. This might be OK. Yep. That's cosmetic and we could suppress the messages if they were a problem. Isn't part of the point of an IOMMU protection against malicious devices and drivers? If so, we should be able to withstand an arbitrary number of faults. > 2. When the SMMUv3 driver sees that it is enabled, it resets itself > and configures it one more time. > > [ 7.018304] arm-smmu-v3 arm-smmu-v3.0.auto: ias 44-bit, oas 44-bit (features 0x00001fef) > [ 7.026379] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting... > > From the moment IOMMU is disabled to the point where IOMMU get > enabled again, there is a potential for the PCIE device to corrupt > the kdump kernel memory as the bus master and memory enable bits are > left enabled. Do you really have to reset the IOMMU? Can you just give it new page tables that start out with all IOVAs from all devices being invalid, then add valid mappings as drivers need them (presumably after the driver has done whatever it needs to so the device stops using the old DMA addresses)? > [ 0.000000] crashkernel reserved: 0x000000007fe00000 - 0x00000000ffe00000 (2048 MB) > > This region happens to overlap with the IOVA addresses that SMMUv3 > driver on the main kernel is allocating. > > IOVA addresses start from 0xFFFFFFFF and get decremented on each > allocation. > > 3. The last one is adapter gets into fuzzy state due to not coming > out of clean state in the second time init and being rejected by > SMMUv3 multiple times. > > [ 16.093441] pci 0000:01:00.0: aer_status: 0x00040000, aer_mask: 0x00000000 > [ 16.099356] pci 0000:01:00.0: Malformed TLP > [ 16.103522] pci 0000:01:00.0: aer_layer=Transaction Layer, aer_agent=Receiver ID > [ 16.110900] pci 0000:01:00.0: aer_uncor_severity: 0x00062011 > [ 16.116543] pci 0000:01:00.0: TLP Header: 0a00a000 00008100 01010100 00000000 I'm not clear on this. I don't remember what an IOMMU fault looks like to an Endpoint. Are you saying that if an Endpoint sees too many of those faults, it gets into this "fuzzy state" (whatever that is :))? Is this a hardware defect? Do we care (this is a kdump kernel, after all)? If we do care, can we fix the device by resetting it? Bjorn _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-02 0:03 ` Bjorn Helgaas @ 2018-03-02 10:30 ` Will Deacon 2018-03-02 14:12 ` Sinan Kaya 2018-03-02 14:20 ` Sinan Kaya 1 sibling, 1 reply; 10+ messages in thread From: Will Deacon @ 2018-03-02 10:30 UTC (permalink / raw) To: Bjorn Helgaas Cc: Sinan Kaya, Lorenzo Pieralisi, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse On Thu, Mar 01, 2018 at 06:03:03PM -0600, Bjorn Helgaas wrote: > On Thu, Mar 01, 2018 at 02:19:09PM -0500, Sinan Kaya wrote: > > On 3/1/2018 2:05 PM, Bjorn Helgaas wrote: > > > On Thu, Mar 01, 2018 at 12:44:26PM -0500, Sinan Kaya wrote: > > >> Hi, > > >> > > >> We are seeing IOMMU faults when booting the kdump kernel on ARM64. > > >> > > >> [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: > > >> [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 > > >> [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > >> [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > >> [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > >> > > >> This is Nate's interpretation of the fault: > > >> > > >> "The PCI device is sending transactions just after the SMMU was > > >> reset/reinitialized which is problematic because the device has not > > >> yet been added to the SMMU and thus should not be doing *any* DMA. > > >> DMA from the PCI devices should be quiesced prior to starting the > > >> crashdump kernel or you risk overwriting portions of memory you > > >> meant to preserve. In this case the SMMU was actually doing you a > > >> favor by blocking these errant DMA operations!!" > > >> > > >> I think this makes sense especially for the IOMMU enabled case on > > >> the host where an IOVA can overlap with the region of memory kdump > > >> reserved for itself. > > >> > > >> Apparently, there has been similar concerns in the past. > > >> > > >> https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf > > >> > > >> and was not addressed globally due to IOMMU+PCI driver ordering > > >> issues and bugs in HW due to hot reset. > > >> > > >> https://lkml.org/lkml/2012/8/3/160 > > >> > > >> Hot reset as mentioned is destructive and may not be the best > > >> implementation choice. However, most of the modern endpoints > > >> support PCIE function level reset. > > >> > > >> One other solution is for SMMUv3 driver to reserve the kdump used > > >> IOVA addresses. > > >> > > >> Another solution is for the SMMUv3 driver to disable PCIe devices > > >> behind the SMMU if it see SMMU is already enabled. > > > > > > What problem are you trying to solve? If the IOMMU is blocking DMA > > > after the kdump kernel starts up, that sounds like the desired > > > behavior. > > > > > > > Three issues: > > 1. I'm seeing a flood of SMMUv3 faults due to adapter using > > addresses from the previous kernel. This might be OK. > > Yep. That's cosmetic and we could suppress the messages if they were > a problem. > > Isn't part of the point of an IOMMU protection against malicious > devices and drivers? If so, we should be able to withstand an > arbitrary number of faults. > > > 2. When the SMMUv3 driver sees that it is enabled, it resets itself > > and configures it one more time. > > > > [ 7.018304] arm-smmu-v3 arm-smmu-v3.0.auto: ias 44-bit, oas 44-bit (features 0x00001fef) > > [ 7.026379] arm-smmu-v3 arm-smmu-v3.0.auto: SMMU currently enabled! Resetting... > > > > From the moment IOMMU is disabled to the point where IOMMU get > > enabled again, there is a potential for the PCIE device to corrupt > > the kdump kernel memory as the bus master and memory enable bits are > > left enabled. > > Do you really have to reset the IOMMU? Can you just give it new page > tables that start out with all IOVAs from all devices being invalid, > then add valid mappings as drivers need them (presumably after the > driver has done whatever it needs to so the device stops using the old > DMA addresses)? We already have the option to do that via the command line using the disable_bypass option, so it just sounds like we need to take this into account when resetting the SMMU to take care that GBPA is configured so that transactions are terminated when SMMUEN=0. Will ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-02 10:30 ` Will Deacon @ 2018-03-02 14:12 ` Sinan Kaya 2018-03-02 18:02 ` Will Deacon 0 siblings, 1 reply; 10+ messages in thread From: Sinan Kaya @ 2018-03-02 14:12 UTC (permalink / raw) To: Will Deacon, Bjorn Helgaas Cc: Lorenzo Pieralisi, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse Hi Will, On 3/2/2018 5:30 AM, Will Deacon wrote: >> Do you really have to reset the IOMMU? Can you just give it new page >> tables that start out with all IOVAs from all devices being invalid, >> then add valid mappings as drivers need them (presumably after the >> driver has done whatever it needs to so the device stops using the old >> DMA addresses)? > We already have the option to do that via the command line using the > disable_bypass option, so it just sounds like we need to take this into > account when resetting the SMMU to take care that GBPA is configured so > that transactions are terminated when SMMUEN=0. How about the points that Baoquan highlighted in his email regarding the solution from AMD and X86? I have not read the entire thread but, is this just a matter of following what Bjorn recommended or there is more to it? Sinan -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-02 14:12 ` Sinan Kaya @ 2018-03-02 18:02 ` Will Deacon 2018-03-02 18:37 ` Sinan Kaya 0 siblings, 1 reply; 10+ messages in thread From: Will Deacon @ 2018-03-02 18:02 UTC (permalink / raw) To: Sinan Kaya Cc: Bjorn Helgaas, Lorenzo Pieralisi, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse On Fri, Mar 02, 2018 at 09:12:29AM -0500, Sinan Kaya wrote: > On 3/2/2018 5:30 AM, Will Deacon wrote: > >> Do you really have to reset the IOMMU? Can you just give it new page > >> tables that start out with all IOVAs from all devices being invalid, > >> then add valid mappings as drivers need them (presumably after the > >> driver has done whatever it needs to so the device stops using the old > >> DMA addresses)? > > We already have the option to do that via the command line using the > > disable_bypass option, so it just sounds like we need to take this into > > account when resetting the SMMU to take care that GBPA is configured so > > that transactions are terminated when SMMUEN=0. > > How about the points that Baoquan highlighted in his email regarding the > solution from AMD and X86? Which specific points do you think this proposal doesn't address? > I have not read the entire thread but, is this just a matter of following > what Bjorn recommended or there is more to it? I'm trying to say how I think Bjorn's idea can be implemented for SMMUv3. I basically want to avoid a situation where the SMMU driver tries to walk the in-memory data structures left by a previous kernel and infer the setup from that. Will ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-02 18:02 ` Will Deacon @ 2018-03-02 18:37 ` Sinan Kaya 0 siblings, 0 replies; 10+ messages in thread From: Sinan Kaya @ 2018-03-02 18:37 UTC (permalink / raw) To: Will Deacon Cc: Bjorn Helgaas, Lorenzo Pieralisi, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse On 3/2/2018 1:02 PM, Will Deacon wrote: >> How about the points that Baoquan highlighted in his email regarding the >> solution from AMD and X86? > Which specific points do you think this proposal doesn't address? > No specific concerns at this moment. >> I have not read the entire thread but, is this just a matter of following >> what Bjorn recommended or there is more to it? > I'm trying to say how I think Bjorn's idea can be implemented for SMMUv3. > I basically want to avoid a situation where the SMMU driver tries to walk > the in-memory data structures left by a previous kernel and infer the > setup from that. Let me know if you have something to test. We have a test case with the crash. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-02 0:03 ` Bjorn Helgaas 2018-03-02 10:30 ` Will Deacon @ 2018-03-02 14:20 ` Sinan Kaya 1 sibling, 0 replies; 10+ messages in thread From: Sinan Kaya @ 2018-03-02 14:20 UTC (permalink / raw) To: Bjorn Helgaas Cc: Lorenzo Pieralisi, Will Deacon, Linux PCI, linux-arm Mailing List, Nate Watterson, shankerd, Vikram Sethi, Goel, Sameer, kexec, Joerg Roedel, iommu, David Woodhouse On 3/1/2018 7:03 PM, Bjorn Helgaas wrote: >> 3. The last one is adapter gets into fuzzy state due to not coming >> out of clean state in the second time init and being rejected by >> SMMUv3 multiple times. >> >> [ 16.093441] pci 0000:01:00.0: aer_status: 0x00040000, aer_mask: 0x00000000 >> [ 16.099356] pci 0000:01:00.0: Malformed TLP >> [ 16.103522] pci 0000:01:00.0: aer_layer=Transaction Layer, aer_agent=Receiver ID >> [ 16.110900] pci 0000:01:00.0: aer_uncor_severity: 0x00062011 >> [ 16.116543] pci 0000:01:00.0: TLP Header: 0a00a000 00008100 01010100 00000000 > I'm not clear on this. I don't remember what an IOMMU fault looks > like to an Endpoint. Are you saying that if an Endpoint sees too many > of those faults, it gets into this "fuzzy state" (whatever that is :))? > Is this a hardware defect? Do we care (this is a kdump kernel, after > all)? If we do care, can we fix the device by resetting it? fuzzy=funky=funny=wierd Regardless of what we do in the IOMMU driver, I think we still have to reset the endpoint in order to have a clean initialization. I'm not sure if all endpoint drivers can recover an adapter from a live state. I wasn't expecting to see a Malformed TLP error. I was guessing that this was caused by SMMU giving a CA or UR to the endpoint or having a live adapter in the middle of driver initialization. I think we do care about the adapter coming up properly otherwise how would you collect the dumps from the system? I was expecting to come through the network interface and download it from the target. That's why, I was suggesting FLR/PM reset etc. when we know that we are booting a kdump kernel. -- Sinan Kaya Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc. Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RFC on Kdump and PCIe on ARM64 2018-03-01 19:05 ` Bjorn Helgaas 2018-03-01 19:19 ` Sinan Kaya @ 2018-03-02 1:44 ` Baoquan He 1 sibling, 0 replies; 10+ messages in thread From: Baoquan He @ 2018-03-02 1:44 UTC (permalink / raw) To: Bjorn Helgaas Cc: Sinan Kaya, Lorenzo Pieralisi, Vikram Sethi, Linux PCI, Joerg Roedel, Will Deacon, iommu, shankerd, linux-arm Mailing List, Goel, Sameer, David Woodhouse, kexec, Nate Watterson On 03/01/18 at 01:05pm, Bjorn Helgaas wrote: > [+cc Joerg, David, iommu list] > > On Thu, Mar 01, 2018 at 12:44:26PM -0500, Sinan Kaya wrote: > > Hi, > > > > We are seeing IOMMU faults when booting the kdump kernel on ARM64. > > > > [ 7.220162] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x02 received: > > [ 7.226123] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000010000000002 > > [ 7.232023] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > [ 7.237925] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > [ 7.243827] arm-smmu-v3 arm-smmu-v3.0.auto: 0x0000000000000000 > > > > This is Nate's interpretation of the fault: > > > > "The PCI device is sending transactions just after the SMMU was > > reset/reinitialized which is problematic because the device has not > > yet been added to the SMMU and thus should not be doing *any* DMA. > > DMA from the PCI devices should be quiesced prior to starting the > > crashdump kernel or you risk overwriting portions of memory you > > meant to preserve. In this case the SMMU was actually doing you a > > favor by blocking these errant DMA operations!!" This seems an known issue which existed on x86 arch with intel vt-d or amd-vi iommu deployed. Both of them have been fixed on x86. The root cause is that kexec/kdump jumping is a warm reboot, it skips bios/firmware. That left behind on-flight DMA which is started in 1st kernel, and on-going during kdump kernel bootup. Then iommu devices init will cause the on-flight DMA being stray and access those memory region violently until pci devices initialization. On x86, for intel vt-d iommu, patches and discussion can be found here: https://lists.onap.org/pipermail/iommu/2015-May/012948.html Finally, Joerg made a formal fix to make it. On amd-iommu, I made a patchset with Joerg's help. http://linux.kernel.narkive.com/Przi0Xaf/patch-v10-00-12-fix-the-on-flight-dma-issue-on-system-with-amd-iommu On arm64, not sure how different the smmu is, you might need to do the similar thing. Personal opinion, just for reference. Thanks Baoquan > > > > I think this makes sense especially for the IOMMU enabled case on > > the host where an IOVA can overlap with the region of memory kdump > > reserved for itself. > > > > Apparently, there has been similar concerns in the past. > > > > https://www.fujitsu.com/jp/documents/products/software/os/linux/catalog/LinuxConJapan2013-Indoh.pdf > > > > and was not addressed globally due to IOMMU+PCI driver ordering > > issues and bugs in HW due to hot reset. > > > > https://lkml.org/lkml/2012/8/3/160 > > > > Hot reset as mentioned is destructive and may not be the best > > implementation choice. However, most of the modern endpoints > > support PCIE function level reset. > > > > One other solution is for SMMUv3 driver to reserve the kdump used > > IOVA addresses. > > > > Another solution is for the SMMUv3 driver to disable PCIe devices > > behind the SMMU if it see SMMU is already enabled. > > What problem are you trying to solve? If the IOMMU is blocking DMA > after the kdump kernel starts up, that sounds like the desired > behavior. > > _______________________________________________ > kexec mailing list > kexec@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/kexec ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-03-02 18:37 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-03-01 17:44 RFC on Kdump and PCIe on ARM64 Sinan Kaya 2018-03-01 19:05 ` Bjorn Helgaas 2018-03-01 19:19 ` Sinan Kaya 2018-03-02 0:03 ` Bjorn Helgaas 2018-03-02 10:30 ` Will Deacon 2018-03-02 14:12 ` Sinan Kaya 2018-03-02 18:02 ` Will Deacon 2018-03-02 18:37 ` Sinan Kaya 2018-03-02 14:20 ` Sinan Kaya 2018-03-02 1:44 ` Baoquan He
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).