Re: [for-4.9] Re: HVM guest performance regression

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
To: Dario Faggioli <dario.faggioli@citrix.com>
Cc: Juergen Gross <jgross@suse.com>,
	xen-devel <xen-devel@lists.xenproject.org>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Ian Jackson <ian.jackson@eu.citrix.com>,
	Wei Liu <wei.liu2@citrix.com>
Subject: Re: [for-4.9] Re: HVM guest performance regression
Date: Sat, 10 Jun 2017 22:27:30 -0400	[thread overview]
Message-ID: <20170611022730.GD30966@char.us.oracle.com> (raw)
In-Reply-To: <1496955634.26212.6.camel@citrix.com>

On Thu, Jun 08, 2017 at 11:00:34PM +0200, Dario Faggioli wrote:
> Bringing in Konrad because...
> 
> On Thu, 2017-06-08 at 11:37 +0200, Juergen Gross wrote:
> > On 07/06/17 20:19, Stefano Stabellini wrote:
> > > On Wed, 7 Jun 2017, Juergen Gross wrote:
> > > > On 06/06/17 21:08, Stefano Stabellini wrote:
> > > > > 
> > > > > 2) PV suspend/resume
> > > > > 3) vector callback
> > > > > 4) interrupt remapping
> > > > > 
> > > > > 2) is not on the hot path.
> > > > > I did individual measurements of 3) at some points and it was a
> > > > > clear win.
> > > > 
> > > > That might depend on the hardware. Could it be newer processors
> > > > are
> > > > faster here?
> > > 
> > > I don't think so: the alternative it's an emulated interrupt. It's
> > > slower under all points of view.
> > 
> > What about APIC virtualization of modern processors? Are you sure
> > e.g.
> > timer interrupts aren't handled completely by the processor? I guess
> > this might be faster than letting it be handled by the hypervisor and
> > then use the callback into the guest.
> > 
> ... I kind of remember an email exchange we had, not here on the list,
> but in private, about some apparently weird scheduling behavior you
> were seeing, there at Oracle, on a particular benchmark/customer's
> workload.
> 
> Not that this is directly related, but I seem to also recall that you
> managed to find out that some of the perf difference (between baremetal
> and guest) was due to vAPIC being faster than the PV path we were
> taking? What I don't recall, though, is whether your guest was PV or
> (PV)HVM... Do you remember anything more precisely than this?

It was HVM and it was a 2.6.39 kernel. And it was due to the Linux kernel
scheduler not scheduling applications back to back on the same CPU
but instead having them on seperate CPUs. The end result being that when
an application blocked, the kernel would call 'schedule()' and instead
of the other application resuming, the kernel would go to sleep (until
a VIRQ TIMER).

The solution was quite simple - force the kernel to think that every
CPU was a sibling. And we flushed this out more and provided an
'smt=1' option so that the CPU topology was exposed (along with
pinning).

> 
> It was like one or two years ago... (I'll dig in the archives for the
> emails.)
> 
> Regards,
> Dario
> -- 
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel