Re: SMMU performance

[Will Deacon <will@kernel.org>]

On Mon, Sep 30, 2019 at 12:45:28PM +0100, Robin Murphy wrote:
> On 30/09/2019 12:00, Russell King - ARM Linux admin wrote:
> > While using iperf on a platform using the ARM SMMU (v2), I notice the
> > following behaviour on v5.3 with Will's iommu patch set merged, kernel
> > lock debugging disabled.
> > 
> > With iommu.passthrough=1, three consecutive runs:
> > [  3]  0.0-10.0 sec  4.51 GBytes  3.87 Gbits/sec
> > [  3]  0.0-10.0 sec  4.53 GBytes  3.89 Gbits/sec
> > [  3]  0.0-10.0 sec  4.49 GBytes  3.86 Gbits/sec
> > 
> > With iommu.passthrough=0:
> > [  3]  0.0-10.0 sec  1.77 GBytes  1.52 Gbits/sec
> > [  3]  0.0-10.0 sec  1.82 GBytes  1.56 Gbits/sec
> > [  3]  0.0-10.0 sec  1.69 GBytes  1.45 Gbits/sec
> > 
> > Running perf record -a -g ... followed by perf report --no-children
> > shows:
> > 
> > -   15.72%  iperf            [kernel.vmlinux]    [k] _raw_spin_unlock_irqrestor
> >     - _raw_spin_unlock_irqrestore
> >        - 8.95% arm_smmu_tlb_sync_context
> >             arm_smmu_iotlb_sync
> >           - __iommu_dma_unmap
> >              + 4.54% iommu_dma_unmap_sg
> >              + 4.41% iommu_dma_unmap_page
> >        - 2.92% alloc_iova_fast
> >           - iommu_dma_alloc_iova.isra.26
> >              + 1.54% iommu_dma_map_sg
> >              + 1.38% __iommu_dma_map
> >        - 2.64% free_iova_fast
> >             iommu_dma_free_iova
> >           - __iommu_dma_unmap
> >              + 1.35% iommu_dma_unmap_sg
> >              + 1.29% iommu_dma_unmap_page
> > 
> > which seems to be pointing to the SMMU code as a bottleneck.
> > 
> > Will suggests that his iommu changes (in his for-joerg/arm-smmu/updates
> > branch), allows IOMMU driver modifications that may have a beneficial
> > effect.  Any thoughts?
> 
> We default to synchronous invalidation on unmaps, since it gives the
> greatest degree of security against misbehaving devices (and proves quite
> useful for smoking out dodgy drivers too). If you're happy with deferred
> invalidation as x86 defaults to, try "iommu.strict=0" - that should avoid
> the main serialising bottleneck. As for the IOVA allocation overhead, that's
> probably about as low as it's likely to get now - what remains is the
> inevitable "doing anything vs. doing nothing" tradeoff.
> 
> The major changes in 5.4 are for SMMUv3, so won't impact your platform.

I was wondering whether rigging up the gather stuff would help here but,
looking at the backtrace, the time is spent on the sync itself so I suspect
it won't help. Hmm... I wonder if we can do better using a sequence number
so that we can ride off the back of somebody else's sync?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel