Re: SMMU performance

[Robin Murphy <robin.murphy@arm.com>]

On 30/09/2019 12:54, Will Deacon wrote:
> On Mon, Sep 30, 2019 at 12:45:28PM +0100, Robin Murphy wrote:
>> On 30/09/2019 12:00, Russell King - ARM Linux admin wrote:
>>> While using iperf on a platform using the ARM SMMU (v2), I notice the
>>> following behaviour on v5.3 with Will's iommu patch set merged, kernel
>>> lock debugging disabled.
>>>
>>> With iommu.passthrough=1, three consecutive runs:
>>> [  3]  0.0-10.0 sec  4.51 GBytes  3.87 Gbits/sec
>>> [  3]  0.0-10.0 sec  4.53 GBytes  3.89 Gbits/sec
>>> [  3]  0.0-10.0 sec  4.49 GBytes  3.86 Gbits/sec
>>>
>>> With iommu.passthrough=0:
>>> [  3]  0.0-10.0 sec  1.77 GBytes  1.52 Gbits/sec
>>> [  3]  0.0-10.0 sec  1.82 GBytes  1.56 Gbits/sec
>>> [  3]  0.0-10.0 sec  1.69 GBytes  1.45 Gbits/sec
>>>
>>> Running perf record -a -g ... followed by perf report --no-children
>>> shows:
>>>
>>> -   15.72%  iperf            [kernel.vmlinux]    [k] _raw_spin_unlock_irqrestor
>>>      - _raw_spin_unlock_irqrestore
>>>         - 8.95% arm_smmu_tlb_sync_context
>>>              arm_smmu_iotlb_sync
>>>            - __iommu_dma_unmap
>>>               + 4.54% iommu_dma_unmap_sg
>>>               + 4.41% iommu_dma_unmap_page
>>>         - 2.92% alloc_iova_fast
>>>            - iommu_dma_alloc_iova.isra.26
>>>               + 1.54% iommu_dma_map_sg
>>>               + 1.38% __iommu_dma_map
>>>         - 2.64% free_iova_fast
>>>              iommu_dma_free_iova
>>>            - __iommu_dma_unmap
>>>               + 1.35% iommu_dma_unmap_sg
>>>               + 1.29% iommu_dma_unmap_page
>>>
>>> which seems to be pointing to the SMMU code as a bottleneck.
>>>
>>> Will suggests that his iommu changes (in his for-joerg/arm-smmu/updates
>>> branch), allows IOMMU driver modifications that may have a beneficial
>>> effect.  Any thoughts?
>>
>> We default to synchronous invalidation on unmaps, since it gives the
>> greatest degree of security against misbehaving devices (and proves quite
>> useful for smoking out dodgy drivers too). If you're happy with deferred
>> invalidation as x86 defaults to, try "iommu.strict=0" - that should avoid
>> the main serialising bottleneck. As for the IOVA allocation overhead, that's
>> probably about as low as it's likely to get now - what remains is the
>> inevitable "doing anything vs. doing nothing" tradeoff.
>>
>> The major changes in 5.4 are for SMMUv3, so won't impact your platform.
> 
> I was wondering whether rigging up the gather stuff would help here but,
> looking at the backtrace, the time is spent on the sync itself so I suspect
> it won't help. Hmm... I wonder if we can do better using a sequence number
> so that we can ride off the back of somebody else's sync?

The trouble with v2 is that then we'd have to introduce locking around 
the invalidates as well, in order to keep track of what the last 
'command' issued in each context was - that's almost certainly going to 
have far more overhead than eliding syncs could possibly save.

Robin.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel