From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhang, Yang Z" Subject: Re: VT-d spin loops Date: Tue, 15 Jul 2014 08:00:21 +0000 Message-ID: References: <53A96B80020000780001CB8F@mail.emea.novell.com> <53AC124E020000780001D8FF@mail.emea.novell.com> <53AC124E020000780001D8FF@mail.emea.novell.com> <53BE62FB.2090105@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1X6xf3-0001re-RS for xen-devel@lists.xenproject.org; Tue, 15 Jul 2014 08:00:30 +0000 In-Reply-To: <53BE62FB.2090105@citrix.com> Content-Language: en-US List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Andrew Cooper , "Tian, Kevin" , Jan Beulich , xen-devel Cc: Malcolm Crossley , "Dugger, Donald D" List-Id: xen-devel@lists.xenproject.org Andrew Cooper wrote on 2014-07-10: > On 10/07/14 00:22, Tian, Kevin wrote: >>> From: Jan Beulich [mailto:JBeulich@suse.com] >>> Sent: Thursday, June 26, 2014 3:30 AM >>> >>> All, >>> >>> VT-d code currently has a number of cases where completion of >>> certain operations is being waited for by way of spinning. The >>> various instances can be identified relatively easily by grep-ing >>> for all uses of DMAR_OPERATION_TIMEOUT (the majority of instances >>> use that variable indirectly through IOMMU_WAIT_OP()), allowing for >>> loops of up to 1 second. While in many of the cases this _may_ be >>> acceptable (which would need to be proven for each individual case, >>> also taking into consideration how many of these spinning loops may >>> be executed in a row with no preemption/scheduling in between), the >>> invalidation case seems particularly problematic: Using >>> DMAR_OPERATION_TIMEOUT is a mistake here in the first place, as the >>> timeout here isn't related to response times by the IOMMU engine. >>> Instead - with ATS in use - the specification mandates a timeout of >>> 1 _minute_ (with a 50% slack, the meaning of which none of us >>> [Andrew and Malcolm brought this issue to my attention in private >>> talks on the hackathon] was able to really interpret in a sensible way). >> yes, that's not a good design. Most waits happen in IOMMU >> initialization, where 1s timeout is less a big issue. At runtime >> cache/tlb flush and root entry manipulation are definitely not good >> with long spinning . >> >>> So there are two things that need doing rather sooner than later: >>> >>> First and foremost the ATS case needs to be taken into >>> consideration when doing invalidations. Obviously we can't spin for >>> a minute, so invalidation absolutely needs to be converted to a non-spinning model. >>> We realize this isn't going to be trivial, which is why we bring >>> this up here rather than coming forward with a patch right away. >> ATS should be fine. Device TLB can ONLY be validated through qinval >> interface, which is asynchronous so no need to consider 1 minute >> timeout even in current spinning model. > > There are currently no asynchronous invalidations in Xen. ATS > certainly is a problem. How Linux upstream handle ATS? Does it have any asynchronous invalidations mechanism? > >> >>> Second, looking at Linux (which interestingly enough also spins, >>> and that even without any timeout) there are flags in the fault >>> status register that can be used to detect at least some loop abort conditions. >>> We should definitely make use of anything that can shorten these >>> spinning loops (as was already done in commit dd6d87a4 ["VT-d: drop >>> redundant calls to invalidate_sync()"] as a very tiny first step). >>> The main problem with trying to clone at least some of the >>> functionality from Linux is that I'm not convinced the replaying >>> they do can actually do anything good. Plus it's clear that - >>> spinning or not - the consequences of an invalidation request not >>> completing successfully need to be taken care of (and it's of no >>> help that in all cases I looked at so far errors passed up from the >>> leaf functions sooner or later get dropped on the floor or mis-interpreted). >>> >>> And finally, all other spinning instances need to be audited to >>> make sure they can't add up to multiple-second spins (iirc we can't >>> tolerate more than about 4s without running into time problems on >>> certain hardware). >>> >> In general yes a non-spinning model is better, but it requires >> non-trivial change to make all IOMMU operations asynchronous. If ATS >> is not a concern, is it still worthy of change besides auditing existing usages? > > Even if the invalidation is only at the IOMMU, waiting milliseconds > for the completion is still time better spent elsewhere, such as running VMs. > > Do you have any numbers for typical completion times for invalidate requests? > The invalidations are completed fairly quickly by hardware. So the cost for spin can be ignored? > ~Andrew Best regards, Yang