* arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 @ 2020-02-14 20:13 Jerry Snitselaar 2020-02-14 20:58 ` Robin Murphy 0 siblings, 1 reply; 5+ messages in thread From: Jerry Snitselaar @ 2020-02-14 20:13 UTC (permalink / raw) To: iommu, Will Deacon Hi Will, On a gigabyte system with Cavium CN8xx, when doing a fio test against an nvme drive we are seeing the following: [ 637.161194] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x8010003f6000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.174329] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x801000036000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.186887] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x8010002ee000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.199275] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x8010003c7000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.211885] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x801000392000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.224580] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x801000018000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.237241] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x801000360000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.249657] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x8010000ba000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.262120] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x80100003e000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 [ 637.274468] arm-smmu arm-smmu.1.auto: Unhandled context fault: fsr=0x80000402, iova=0x801000304000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 I also reproduced with 5.5-rc7, and will check 5.6-rc1 later today. I couldn't narrow it down further into 5.4-rc1. I don't know smmu or the code well, any thoughts on where to start digging into this? fio test that is being run is: #fio -filename=/dev/nvme0n1 -iodepth=64 -thread -rw=randwrite -ioengine=libaio -bs=4k -runtime=43200 -size=-group_reporting -name=mytest -numjobs=32 Regards, Jerry _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 2020-02-14 20:13 arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 Jerry Snitselaar @ 2020-02-14 20:58 ` Robin Murphy 2020-02-16 22:11 ` Jerry Snitselaar 0 siblings, 1 reply; 5+ messages in thread From: Robin Murphy @ 2020-02-14 20:58 UTC (permalink / raw) To: Jerry Snitselaar, iommu, Will Deacon Hi Jerry, On 2020-02-14 8:13 pm, Jerry Snitselaar wrote: > Hi Will, > > On a gigabyte system with Cavium CN8xx, when doing a fio test against > an nvme drive we are seeing the following: > > [ 637.161194] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x8010003f6000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.174329] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x801000036000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.186887] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x8010002ee000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.199275] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x8010003c7000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.211885] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x801000392000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.224580] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x801000018000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.237241] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x801000360000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.249657] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x8010000ba000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.262120] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x80100003e000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 > [ 637.274468] arm-smmu arm-smmu.1.auto: Unhandled context fault: > fsr=0x80000402, iova=0x801000304000, fsynr=0x70091, cbfrsynra=0x9000, cb=7 Those "IOVAs" don't look much like IOVAs from the DMA allocator - if they were physical addresses, would they correspond to an expected region of the physical memory map? I would suspect that this is most likely misbehaviour in the NVMe driver (issuing a write to a non-DMA-mapped address), and the SMMU is just doing its job in blocking and reporting it. > I also reproduced with 5.5-rc7, and will check 5.6-rc1 later today. I > couldn't narrow it down further into 5.4-rc1. > I don't know smmu or the code well, any thoughts on where to start > digging into this? > > fio test that is being run is: > > #fio -filename=/dev/nvme0n1 -iodepth=64 -thread -rw=randwrite > -ioengine=libaio -bs=4k -runtime=43200 -size=-group_reporting > -name=mytest -numjobs=32 Just to clarify, do other tests work OK on the same device? Thanks, Robin. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 2020-02-14 20:58 ` Robin Murphy @ 2020-02-16 22:11 ` Jerry Snitselaar 2020-02-17 13:08 ` Robin Murphy 0 siblings, 1 reply; 5+ messages in thread From: Jerry Snitselaar @ 2020-02-16 22:11 UTC (permalink / raw) To: Robin Murphy; +Cc: iommu, Will Deacon On Fri Feb 14 20, Robin Murphy wrote: >Hi Jerry, > >On 2020-02-14 8:13 pm, Jerry Snitselaar wrote: >>Hi Will, >> >>On a gigabyte system with Cavium CN8xx, when doing a fio test against >>an nvme drive we are seeing the following: >> >>[ 637.161194] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x8010003f6000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.174329] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x801000036000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.186887] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x8010002ee000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.199275] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x8010003c7000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.211885] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x801000392000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.224580] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x801000018000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.237241] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x801000360000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.249657] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x8010000ba000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.262120] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x80100003e000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 >>[ 637.274468] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>fsr=0x80000402, iova=0x801000304000, fsynr=0x70091, >>cbfrsynra=0x9000, cb=7 > >Those "IOVAs" don't look much like IOVAs from the DMA allocator - if >they were physical addresses, would they correspond to an expected >region of the physical memory map? > >I would suspect that this is most likely misbehaviour in the NVMe >driver (issuing a write to a non-DMA-mapped address), and the SMMU is >just doing its job in blocking and reporting it. > >>I also reproduced with 5.5-rc7, and will check 5.6-rc1 later today. >>I couldn't narrow it down further into 5.4-rc1. >>I don't know smmu or the code well, any thoughts on where to start >>digging into this? >> >>fio test that is being run is: >> >>#fio -filename=/dev/nvme0n1 -iodepth=64 -thread -rw=randwrite >>-ioengine=libaio -bs=4k -runtime=43200 -size=-group_reporting >>-name=mytest -numjobs=32 > >Just to clarify, do other tests work OK on the same device? > >Thanks, >Robin. > I was able to get back on the system today. I think I know what the problem is: [ 0.036189] iommu: Gigabyte R120-T34-00 detected, force iommu passthrough mode [ 6.324282] iommu: Default domain type: Translated So the new default domain code in 5.4 overrides the iommu quirk code setting default passthrough. Testing a quick patch that tracks whether the default domain was set in the quirk code, and leaves it alone if it was. So far it seems to be working. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 2020-02-16 22:11 ` Jerry Snitselaar @ 2020-02-17 13:08 ` Robin Murphy 2020-02-17 14:58 ` Jerry Snitselaar 0 siblings, 1 reply; 5+ messages in thread From: Robin Murphy @ 2020-02-17 13:08 UTC (permalink / raw) To: Jerry Snitselaar; +Cc: iommu, Will Deacon On 16/02/2020 10:11 pm, Jerry Snitselaar wrote: > On Fri Feb 14 20, Robin Murphy wrote: >> Hi Jerry, >> >> On 2020-02-14 8:13 pm, Jerry Snitselaar wrote: >>> Hi Will, >>> >>> On a gigabyte system with Cavium CN8xx, when doing a fio test against >>> an nvme drive we are seeing the following: >>> >>> [ 637.161194] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x8010003f6000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.174329] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x801000036000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.186887] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x8010002ee000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.199275] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x8010003c7000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.211885] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x801000392000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.224580] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x801000018000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.237241] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x801000360000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.249657] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x8010000ba000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.262120] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x80100003e000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >>> [ 637.274468] arm-smmu arm-smmu.1.auto: Unhandled context fault: >>> fsr=0x80000402, iova=0x801000304000, fsynr=0x70091, cbfrsynra=0x9000, >>> cb=7 >> >> Those "IOVAs" don't look much like IOVAs from the DMA allocator - if >> they were physical addresses, would they correspond to an expected >> region of the physical memory map? >> >> I would suspect that this is most likely misbehaviour in the NVMe >> driver (issuing a write to a non-DMA-mapped address), and the SMMU is >> just doing its job in blocking and reporting it. >> >>> I also reproduced with 5.5-rc7, and will check 5.6-rc1 later today. I >>> couldn't narrow it down further into 5.4-rc1. >>> I don't know smmu or the code well, any thoughts on where to start >>> digging into this? >>> >>> fio test that is being run is: >>> >>> #fio -filename=/dev/nvme0n1 -iodepth=64 -thread -rw=randwrite >>> -ioengine=libaio -bs=4k -runtime=43200 -size=-group_reporting >>> -name=mytest -numjobs=32 >> >> Just to clarify, do other tests work OK on the same device? >> >> Thanks, >> Robin. >> > > I was able to get back on the system today. I think I know what the > problem is: > > [ 0.036189] iommu: Gigabyte R120-T34-00 detected, force iommu > passthrough mode > [ 6.324282] iommu: Default domain type: Translated > > So the new default domain code in 5.4 overrides the iommu quirk code > setting default > passthrough. Testing a quick patch that tracks whether the default > domain was set > in the quirk code, and leaves it alone if it was. So far it seems to be > working. Ah, OK. Could you point me at that quirk code? I can't seem to track it down in mainline, and seeing this much leaves me dubious that it's even correct - matching a particular board implies that it's a firmware issue (as far as I'm aware the SMMUs in CN88xx SoCs are usable in general), but if the firmware description is wrong to the point that DMA ops translation doesn't work, then no other translation (e.g. VFIO) is likely to work either. In that case it's simply not safe to enable the SMMU at all, and fudging the default domain type merely hides one symptom of the problem. Robin. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 2020-02-17 13:08 ` Robin Murphy @ 2020-02-17 14:58 ` Jerry Snitselaar 0 siblings, 0 replies; 5+ messages in thread From: Jerry Snitselaar @ 2020-02-17 14:58 UTC (permalink / raw) To: Robin Murphy; +Cc: iommu, Will Deacon On Mon Feb 17 20, Robin Murphy wrote: >On 16/02/2020 10:11 pm, Jerry Snitselaar wrote: >>On Fri Feb 14 20, Robin Murphy wrote: >>>Hi Jerry, >>> >>>On 2020-02-14 8:13 pm, Jerry Snitselaar wrote: >>>>Hi Will, >>>> >>>>On a gigabyte system with Cavium CN8xx, when doing a fio test against >>>>an nvme drive we are seeing the following: >>>> >>>>[ 637.161194] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x8010003f6000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.174329] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x801000036000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.186887] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x8010002ee000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.199275] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x8010003c7000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.211885] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x801000392000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.224580] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x801000018000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.237241] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x801000360000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.249657] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x8010000ba000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.262120] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x80100003e000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>>>[ 637.274468] arm-smmu arm-smmu.1.auto: Unhandled context >>>>fault: fsr=0x80000402, iova=0x801000304000, fsynr=0x70091, >>>>cbfrsynra=0x9000, cb=7 >>> >>>Those "IOVAs" don't look much like IOVAs from the DMA allocator - >>>if they were physical addresses, would they correspond to an >>>expected region of the physical memory map? >>> >>>I would suspect that this is most likely misbehaviour in the NVMe >>>driver (issuing a write to a non-DMA-mapped address), and the SMMU >>>is just doing its job in blocking and reporting it. >>> >>>>I also reproduced with 5.5-rc7, and will check 5.6-rc1 later >>>>today. I couldn't narrow it down further into 5.4-rc1. >>>>I don't know smmu or the code well, any thoughts on where to >>>>start digging into this? >>>> >>>>fio test that is being run is: >>>> >>>>#fio -filename=/dev/nvme0n1 -iodepth=64 -thread -rw=randwrite >>>>-ioengine=libaio -bs=4k -runtime=43200 -size=-group_reporting >>>>-name=mytest -numjobs=32 >>> >>>Just to clarify, do other tests work OK on the same device? >>> >>>Thanks, >>>Robin. >>> >> >>I was able to get back on the system today. I think I know what the >>problem is: >> >>[ 0.036189] iommu: Gigabyte R120-T34-00 detected, force iommu >>passthrough mode >>[ 6.324282] iommu: Default domain type: Translated >> >>So the new default domain code in 5.4 overrides the iommu quirk code >>setting default >>passthrough. Testing a quick patch that tracks whether the default >>domain was set >>in the quirk code, and leaves it alone if it was. So far it seems to >>be working. > >Ah, OK. Could you point me at that quirk code? I can't seem to track >it down in mainline, and seeing this much leaves me dubious that it's >even correct - matching a particular board implies that it's a >firmware issue (as far as I'm aware the SMMUs in CN88xx SoCs are >usable in general), but if the firmware description is wrong to the >point that DMA ops translation doesn't work, then no other translation >(e.g. VFIO) is likely to work either. In that case it's simply not >safe to enable the SMMU at all, and fudging the default domain type >merely hides one symptom of the problem. > >Robin. > Ugh. It is a RHEL only patch, but for some reason it is applied to the ark kernel builds as well. Sorry for the noise. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-02-17 14:58 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-02-14 20:13 arm-smmu.1.auto: Unhandled context fault starting with 5.4-rc1 Jerry Snitselaar 2020-02-14 20:58 ` Robin Murphy 2020-02-16 22:11 ` Jerry Snitselaar 2020-02-17 13:08 ` Robin Murphy 2020-02-17 14:58 ` Jerry Snitselaar
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).