Re: [xen-unstable test] 65141: regressions - FAIL

From: "Tian, Kevin" <kevin.tian@intel.com>
To: Jan Beulich <JBeulich@suse.com>,
	Ian Campbell <ian.campbell@citrix.com>,
	Ian Jackson <Ian.Jackson@eu.citrix.com>,
	"Nakajima, Jun" <jun.nakajima@intel.com>,
	"Hu, Robert" <robert.hu@intel.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>,
	"xen-devel@lists.xensource.com" <xen-devel@lists.xensource.com>,
	osstestservice owner <osstest-admin@xenproject.org>
Subject: Re: [xen-unstable test] 65141: regressions - FAIL
Date: Tue, 8 Dec 2015 02:46:58 +0000	[thread overview]
Message-ID: <AADFC41AFE54684AB9EE6CBC0274A5D15F732AA9@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <5665BF5202000078000BCC0E@prv-mh.provo.novell.com>

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, December 08, 2015 12:18 AM
> 
> >>> On 05.12.15 at 09:09, <ian.campbell@citrix.com> wrote:
> > On Wed, 2015-12-02 at 13:51 +0000, Ian Campbell wrote:
> >
> >> http://osstest.test-lab.xenproject.org/~osstest/pub/logs/65301/
> >>
> >> I think that ought to give a baseline for the bisector to work with. I'll
> >> prod it to do so.
> >
> > Results are below. TL;DR: d02e84b9d9d "vVMX: use latched VMCS machine
> > address" is somehow at fault.
> >
> > It appears to be somewhat machine specific, the one this has been
> > failing on is godello* which says "CPU0: Intel(R) Xeon(R) CPU E3-1220
> > v3 @ 3.10GHz stepping 03" in its serial log.
> >
> > Andy suggested this might be related to cpu_has_vmx_vmcs_shadowing
> > so Haswell and newer vs IvyBridge and older.
> 
> Yeah, but on irc it was also made clear that the regression is on a
> system without that capability.
> 
> At this point we certainly need to seriously consider reverting the
> whole change. The reason I continue to be hesitant is that I'm
> afraid this may result in no-one trying to find out what the problem
> here is. While I could certainly try to, I'm sure I won't find time to
> do so within the foreseeable future. And since we didn't get any
> real feedback from Intel so far, I thought I'd ping them to at least
> share some status before we decide. That pinging has happened
> a few minutes ago. I'd therefore like to give it, say, another day,
> and if by then we don't have an estimate for when a fix might
> become available, I'd do the revert. Unless of course somebody
> feels strongly about doing the revert immediately.
> 

I didn't see an obvious error from the commit, so some debug
would be required to identify the problematic code. However this 
issue was not reproduced immediately in our internal environment, 
and the guy familiar with this area (Yang) just left Intel. It takes 
some time to identify a new developer and get him ramped up to
fix issues in this area. Given that fact, I'd suggest to revert related
code now (as you discussed not the whole commit). In parallel
we'll find someone to look at original commit as a rampup task.

Thanks
Kevin