Re: [PATCH 00/13] Rework IOMMU API to allow for batching of invalidation

From: John Garry <john.garry@huawei.com>
To: Will Deacon <will@kernel.org>
Cc: Jean-Philippe Brucker <jean-philippe@linaro.org>,
	Vijay Kilary <vkilari@codeaurora.org>,
	Jon Masters <jcm@redhat.com>, Jan Glauber <jglauber@marvell.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	iommu@lists.linux-foundation.org,
	Jayachandran Chandrasekharan Nair <jnair@marvell.com>,
	David Woodhouse <dwmw2@infradead.org>,
	Robin Murphy <robin.murphy@arm.com>
Subject: Re: [PATCH 00/13] Rework IOMMU API to allow for batching of invalidation
Date: Fri, 16 Aug 2019 11:11:02 +0100	[thread overview]
Message-ID: <4834c243-465c-f4ef-30df-aab230456c93@huawei.com> (raw)
In-Reply-To: <20190815135528.d7ip6dkqdwwvjy64@willie-the-truck>

On 15/08/2019 14:55, Will Deacon wrote:
> On Thu, Aug 15, 2019 at 12:19:58PM +0100, John Garry wrote:
>> On 14/08/2019 18:56, Will Deacon wrote:
>>> If you'd like to play with the patches, then I've also pushed them here:
>>>
>>>   https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/unmap
>>>
>>> but they should behave as a no-op on their own.
>>
>> As anticipated, my storage testing scenarios roughly give parity throughput
>> and CPU loading before and after this series.
>>
>> Patches to convert the
>>> Arm SMMUv3 driver to the new API are here:
>>>
>>>   https://git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/log/?h=iommu/cmdq
>>
>> I quickly tested this again and now I see a performance lift:
>>
>> 			before (5.3-rc1)		after
>> D05 8x SAS disks	907K IOPS			970K IOPS
>> D05 1x NVMe		450K IOPS			466K IOPS
>> D06 1x NVMe		467K IOPS			466K IOPS
>>
>> The CPU loading seems to track throughput, so nothing much to say there.
>>
>> Note: From 5.2 testing, I was seeing >900K IOPS from that NVMe disk for
>> !IOMMU.
>
> Cheers, John. For interest, how do things look if you pass iommu.strict=0?
> That might give some indication about how much the invalidation is still
> hurting us.

So I tested for iommu/cmdq for NVMe only, and I see:

	 !SMMU		5.3-rc4 strict/!strict	cmdq strict/!strict
D05 NVMe 750K IOPS	456K/540K IOPS		466K/537K
D06 NVMe 750K IOPS	456K/740K IOPS		466K/745K

I don't know why the D06 iommu.strict performance is ~ same as D05, 
while !strict is so much better. D06 SMMU implementation is supposed to 
be generally much better than that of D05, so I would have thought that 
the strict performance would be better (than that of D05).

>
>> BTW, what were your thoughts on changing
>> arm_smmu_atc_inv_domain()->arm_smmu_atc_inv_master() to batching? It seems
>> suitable, but looks untouched. Were you waiting for a resolution to the
>> performance issue which Leizhen reported?
>
> In principle, I'm supportive of such a change, but I'm not currently able
> to test any ATS stuff so somebody else would need to write the patch.
> Jean-Philippe is on holiday at the moment, but I'd be happy to review
> something from you if you send it out.

Unfortunately I don't have anything ATS-enabled either. Not many do, it 
seems.

Cheers,
John

>
> Will
>
> .
>

_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu