From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jan Beulich" Subject: Re: S3 crash with VTD Queue Invalidation enabled Date: Wed, 05 Jun 2013 16:14:17 +0100 Message-ID: <51AF71E902000078000DB8B7@nat28.tlf.novell.com> References: <51ACECEB.9030904@citrix.com> <51ADC77402000078000DAF95@nat28.tlf.novell.com> <51AE0F6602000078000DB1F4@nat28.tlf.novell.com> <51AF11F102000078000DB589@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Content-Disposition: inline List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ben Guthro Cc: Andrew Cooper , xiantao.zhang@intel.com, xen-devel List-Id: xen-devel@lists.xenproject.org >>> On 05.06.13 at 15:54, Ben Guthro wrote: > On Wed, Jun 5, 2013 at 4:24 AM, Jan Beulich wrote: >>>>> On 04.06.13 at 23:09, Ben Guthro wrote: >>> On Tue, Jun 4, 2013 at 3:49 PM, Ben Guthro wrote: >>>> On Tue, Jun 4, 2013 at 3:20 PM, Ben Guthro wrote: >>>>> On Tue, Jun 4, 2013 at 10:01 AM, Jan Beulich wrote: >>>>>>>>> On 04.06.13 at 14:25, Ben Guthro wrote: >>>>>>> On Tue, Jun 4, 2013 at 4:54 AM, Jan Beulich wrote: >>>>>>>> Is this perhaps having some similarity with >>>>>>>> http://lists.xen.org/archives/html/xen-devel/2013-04/msg00343.html? >>>>>>>> We're clearly running single-CPU only here and there... >>>>>>> >>>>>>> We certainly should be, as we have gone through the >>>>>>> disable_nonboot_cpus() by this point - and I can verify that from the >>>>>>> logs. >>>>>> >>>>>> I'm much more tending towards the connection here, noting that >>>>>> Andrew's original thread didn't really lead anywhere (i.e. we still >>>>>> don't know what the panic he saw was actually caused by). >>>>>> >>>>> >>>>> I'm starting to think you're on to something here. >>>> >>>> hmm - maybe not. >>>> I get the same crash with "maxcpus=1" >>>> >>>> >>>> >>>>> I've put a bunch of trace throughout the functions in qinval.c >>>>> >>>>> It seems that everything is functioning properly, up until we go >>>>> through the disable_nonboot_cpus() path. >>>>> Prior to this, I see the qinval.c functions being executed on all >>>>> cpus, and both drhd units >>>>> Afterward, it gets stuck in queue_invalidate_wait on the first drhd >>>>> unit.. and eventually panics. >>>>> >>>>> I'm not exactly sure what to make of this yet. >>> >>> querying status of the hardware all seems to be working correctly... >>> it just doesn't work with querying the QINVAL_STAT_DONE state, as far >>> as I can tell. >>> >>> Other register state is: >>> >>> (XEN) VER = 10 >>> (XEN) CAP = c0000020e60262 >>> (XEN) n_fault_reg = 1 >>> (XEN) fault_recording_offset = 200 >>> (XEN) fault_recording_reg_l = 0 >>> (XEN) fault_recording_reg_h = 0 >>> (XEN) ECAP = f0101a >>> (XEN) GCMD = 0 >>> (XEN) GSTS = c7000000 With #define DMA_GCMD_QIE (((u64)1) << 26) and #define DMA_GSTS_QIES (((u64)1) <<26) this means qinval is still enabled at this point. >>> (XEN) RTADDR = 137a31000 >>> (XEN) CCMD = 800000000000000 >>> (XEN) FSTS = 0 >>> (XEN) FECTL = 0 >>> (XEN) FEDATA = 4128 >>> (XEN) FEADDR = fee0000c >>> (XEN) FEUADDR = 0 >>> >>> (with code lifted from print_iommu_regs() ) >>> >>> >>> None of this looks suspicious to my untrained eye - but I'm including >>> it here in case someone else sees something I don't. >> >> Xiantao, you certainly will want to give some advice here. I won't >> be able to look into this more deeply right away. > > Thanks Jan. Xiantao - I'd appreciate any insight you may have. > > One curious thing I have found, that seems buggy to me, is that > {dis,en}able_qinval() is being called prior to the platform quirks > being executed. > It appears they are being called through iommu_{en,dis}able_x2apic_IR() That's because this setup needs to happen when interrupt (i.e. APIC) initialization is happening, not when the IOMMUs get set up (which is a process that assumes interrupts can already be requested). In effect we have lapic_suspend() -> iommu_disable_x2apic_IR() -> disable_intremap()/disable_qinval() after iommu_suspend() -> vtd_suspend() -> disable_qinval() but the latter tail call only when !iommu_intremap, and you have been running with interrupt remapping enabled, so only the former code path would result in qinval getting disabled, which is after the point of the hang. Depending on whether ATS is in use, more than one invalidation can be done in the processing here - could you therefore check whether there's any sign of ATS use ("iommu=verbose" should make you see respective messages), and if so see whether disabling it ("ats=off") makes a difference? Jan