High lock contention on async_umap_flush_lock when intel_iommu=on

High lock contention on async_umap_flush_lock when intel_iommu=on

[Charles Suresh]

When bridging traffic across two Intel X540-AT2 NICs using openvswitch(OVS),
I can get ~10 Mpps of small packet (64 byte) traffic on a Dell R620 system
using 2x Intel E5-2650 v2 @ 2.6 GHz using a Spirent generator.

Using the identical system and after enabling intel_iommu=on during boot,
the throughput drops to ~500 Kpps.

Looking at the output from perf, I see that ~80% of the CPUs are consumed
spinning on locks. The highest contention appears to be on async_umap_flush_lock
acquired in add_unmap. This area of code appears unchanged since it was
introduced back in 2008. The code review back then called out the lack of
scalability of the code as seen in this snippet from 
http://lkml.iu.edu/hypermail/linux/kernel/0802.3/2944.html :

---begin quote---
...
>> +static void add_unmap(struct dmar_domain *dom, struct iova *iova)
>> +{
>> + unsigned long flags;
>> +
>> + spin_lock_irqsave(&async_umap_flush_lock, flags);
> 
> How scalable is this?

Its not very, but I'm doing an insert walk of a list and need the locks.
...
---end quote---

>From looking at bugzilla no one else appears to have reported this issue
so I'm wondering if there is an easy way to side step this - is there a
workaround so that I don't need to use "intel_iommu=on"?

I'm testing with the "intel_iommu=on" setting so that I can get SR-IOV NICs
passed through to VMs running over KVM. Is this possible any other way?

Let me know if you want me to open a bug on bugzilla and I can upload the
perf.data and perf report output there instead of attaching large files to this
email. For now, if it helps, here's the snippet from the top of perf report:

---begin extract from perf report---
...
    5.43%    ksoftirqd/0  [kernel.kallsyms]   [k] _raw_spin_lock_irqsave                      
             |
             --- _raw_spin_lock_irqsave
                |          
                |--62.66%-- add_unmap
                |          |          
                |          |--99.93%-- intel_unmap_page.part.40
                |          |          intel_unmap_page
                |          |          ixgbe_poll
                |          |          net_rx_action
                |          |          __do_softirq
...
---end extract from perf report---

I’m not on the iommu alias - so I’d appreciate being cc’ed on any replies.

Thanks,
Charles