Xen 4.5 development update

Xen 4.5 development update

[konrad.wilk]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 10278 bytes --]

Below is a summary of the projects / features being worked on for the 4.5
time frame.  The tentative feature freeze is scheduled for September 10th,
which is months away.  With that in mind, I think it's time to take
stock of the development, so we know whether to ask for more help or divert
resources.

The "prognosis" is now the likelihood of completion in the 4.5 timeframe.
Instead of values (0->100) it is now five states:

none - nothing yet
fair - still working on it, patches are prototypes or RFC
ok   - patches posted, acting on review
good - some last minute pieces
done - all done, might have bugs


For items involving code hosted on the Xen.org site (including qemu-xen),
that means a likelihood of having the feature code-complete and mostly
working by the feature freeze.  (It's OK if there are still bugs to be
worked out.)  For items in Linux, it would mean having items on track
to make it into the kernel released just after the scheduled 4.5 time frame.

Not exactly sure what that is in terms of libvirt.

= Timeline =

We are planning on a 9-month release cycle.  Based on that, below are
our estimated dates:

* Feature Freeze: 10th September 2014
* First RC: 10th October
* Release: 10th December 2014

The RCs and release will of course depend on stability and bugs, and
will therefore be fairly unpredictable.  The feature freeze may be
slipped for especially important features which are near completion.

= Prognosis =

The states are: none -> fair -> ok -> good -> done

none - nothing yet
fair - still working on it, patches are prototypes or RFC
ok   - patches posted, acting on review
good - some last minute pieces
done - all done, might have bugs


= Open =

== ARM ==

*  ARM - Device assigment on ARM (good)
   RFC for the hypervisor side sent.
   Linux parts at risk.
  -  Julien Grall

*  ARM XEN_DOMCTL_memory_mapping hypercall for ARM (good)
  -  Arianna Avanzini

*  ARM Xen UEFI booting on ARM (ok)
   RFC posted
  -  Roy Franz

*  ARM PSCI v0.2 (good)
   v2 posted
  -  Parth Dixit

*  ARM GICv3 support (ok)
   Split in two patchsets: v6a and v6
   v7a posted which does GIC and VGIC code refactoring
  -  Vijay Kilari

*  ARM GICv2m support (none)
  -  Linaro (unknown)

*  ARM VM save/restore/live migration (ok)
  -  Junghyun Yoo

*  ARM remote processor iommu module (GPUs + IPUs) (ok)
  -  Andrii Tseglytskyi

*  ARM: Use super pages in p2m  (ok)
   v4 posted
  -  Ian Campbell

*  ARM DRA7 support (ok)
  -  Andrii Tseglytskyi

*  ARM - MiniOS (ok)
   v5 posted
  -  Thomas Leonard

== x86 ==

*  New migration. (ok)
   tmem&remus need work
   libxc is RFC
  -  Andrew Cooper & David Vrabel

*  PVH - AMD hardware support. (fair)
   Issues with FSBASE MSRs
  -  Mukesh Rathor

*  Xen multiboot2-EFI support (fair)
   Needed for SecureBoot
  -  Daniel Kiper

*  Xen HPET interrupt fixes (fair)
   behind migration v2
  -  Andrew Cooper

*  HT enabled, virtualization overhead is high (Xen 4.4) (none)
   kernbench demonstrated it
   looking and tracing it
  -  Dario Faggioli

*  IOMMU ABI for guests to map their DMA regions (fair)
  -  Malcolm Crossley

*  dirty vram / IOMMU bug (fair)
  -  Zhang, Yang Z

*  Convert tasklet to per-cpu tasklets (fair)
  -  Konrad Rzeszutek Wilk

*  VMware backdoor (hypercall) (ok)
   Needs to be split up and redone
  -  Don Slutz

*  VPMU - 'perf' support in Xen (good)
   v5 posted
  -  Boris Ostrovsky

*  Soft affinity for vcpus (was NUMA affinity for vcpus) (good)
   v11 posted
  -  Dario Faggioli

*  Cache QoS Monitoring - hypercalls (fair)
   Just hypercalls - no toolstack changes.
   v11 posted
  -  Dongxiao Xu and Shantong Kang

*  HVM guest NUMA (none)
  -  Matt Wilson

*  1TB slow destruction (ok)
  -  Bob Liu

*  extending mem_access support to PV domain (fair)
  -  Aravindh Puthiyaparambil (aravindp)

*  Stability fix (good)
  -  Jan Beulich

*  Support controlling the max C-state sub-state (ok)
   v3 posted
  -  Ross Lagerwall

*  Repurpose SEDF Scheduler for Real-time (ok)
   RFC patch posted (v1)
  -  Joshua Whitehead, Robert VanVossen

== QEMU ==

*  Rebase of QEMU 2.0 and SeaBIOS (fair)
  -  Ian Jackson

*  Using qemu-upstream in a stubdomain (fair)
  -  Ian Jackson

*  Intel IGD PCI GPU passthrough (ok)
   v5 posted
  -  Chen, Tiejun

*  AMD Radeon PCI GPU passthrough (none)
   Focusing on Xen 4.2 and qemu-traditional
  -  Kelly Zytaruk

*  vNUMA in Xen (ok)
   git://gitorious.org/xenvnuma_v5/xenvnuma_v5.git
  -  Elena Ufimtseva

== lib{xc,xl} and toolstack ==

*  libxl/xl - xm compatibility mode for mem-max and mem-set; (ok)
  -  Daniel Kiper

*  Rearrange and cleanup installation destination directories (/var -> var/lib/xen) (fair)
  -  Daniel Kiper

*  libxl work - JSON to keep track of guest configs (ok)
  -  Wei Liu

*  pvscsi should be targeted for 4.5, a prototype exists (fair)
  -  Olaf Hering

*  xl list --long (and some related xl commands) have some bugs (none)
  -  Zhigang Wang

*  PVH - Migration of PVH DomUs. (none)
   Depends on migration2 code
  -  Roger Pau Monné

*  PVH - Migration of guests from a PVH dom0  (none)
   Depends on migration2 code
  -  Roger Pau Monné

*  xl does not handle migrate interruption gracefully (none)
   If you start a localhost migrate, and press "Ctrl-C" in the middle, you get two hung domains
  -  Ian Jackson

*  Remus in Xen (ok)
   url: https://github.com/laijs/xen remus-v13
  -  Gui Jianfeng
  -  Yang Hongyang
  -  Dong, Eddie

*  COLO in Xen (fair)
  -  Gui Jianfeng
  -  Yang Hongyang
  -  Dong, Eddie

*  systemd support (ok)
   v6
  -  Luis R. Rodriguez

*  snapshot API extension (checkpointing disk) (ok)
  -  Bamvor Jian Zhang

*  extend the xenstore ring with a 'closing' signal
   RFC patch posted
  -  David Scott

== Linux ==

*  Linux block multiqueue (fair)
  -  Arianna Avanzini

*  Netback grant table manipulations (ok)
  -  Zoltan Kiss

*  VPMU - 'perf' support in Linux (ok)
   Depends on Xen patches
  -  Boris Ostrovsky

*  vNUMA in Linux (ok)
   git://gitorious.org/xenvnuma_v5/xenvnuma_v5.git
  -  Elena Ufimtseva

*  vsyscall in Linux (fair)
  -  Konrad Rzeszutek Wilk

*  Linux pvops of Xen EFI hypercall support (good)
  -  Daniel Kiper

*  COLO Agent in Linux (fair)
  -  Gui Jianfeng
  -  Yang Hongyang
  -  Dong, Eddie

*  vAPIC in PVHVM guests (Linux side) (none)
  -  Boris Ostrovsky

*  pvSCSI (ok)
   Initial patch posted.
  -  Juergen Gross

== FreeBSD ==

*  PVH FreeBSD dom0 (ok)
   FreeBSD 11 goal. Toolstack side done in Xen 4.5
  -  Roger Pau Monné

== Other OSes (MiniOS, QNX) ==

*  PV drivers for automotive kernels (fair)
  -  Artem Mygaiev

*  mini-os: xenbus changes for rump kernels (ok)
   git://xenbits.xen.org/people/iwj/rumpuser-xen.git
   branch: base.dev-xen-xenbus.v1..dev-xen-xenbus.v1
  -  Ian Jackson

== GRUB2 ==

*  GRUB2 multiboot2 (fair)
  -  Daniel Kiper

== OSSTEST ==

*  OSSTest: libvirt (good)
  -  Ian Campbell

== Deferred to Xen 4.6 ==

*  Default to credit2 (none)
   cpu pinning, numa affinity and cpu reservation
  -  George Dunlap

*  "Short" grant copy (just header) of packets. (none)
  -  Zoltan Kiss

*  cpuid leveling (none)
   http://xenbits.xen.org/people/andrewcoop/feature-levelling/feature-levelling-D.pdf
  -  Andrew Cooper

*  live migration knobs, there is no suitable code yet, just ideas (none)
    http://lists.xenproject.org/archives/html/xen-devel/2014-03/msg00785.html
  -  Olaf Hering

*  Further tmem cleanups/fixes (16TB etc) (fair)
  -  Bob Liu

== Up for grabs ==

*  PoD fixes
   if you boot with memory <= maxmem we have a size estimation bug

*  TLB flushing without locks in Xen

*  xl does not support specifying virtual function for passthrough device
   http://bugs.xenproject.org/xen/bug/22

*  PCI hole resize support hvmloader/qemu-traditional/qemu-upstream with PCI/GPU passthrough
   http://bugs.xenproject.org/xen/bug/28

*  libx{c,l} error handling cleanup

*  Adding missing 'xend' features in libxl

*  xl list -l on a dom0-only system

*  xl list -l doesn't contain tty console port

*  xl: passing more defaults in configuration in xl.conf
   There are a number of options for which it might be useful to pass a default in xl.conf.  For example, if we could have a default "backend" parameter for vifs, then it would be easy to switch back and forth between a backend in a driver domain and a backend in dom0.

*  PVH - PVH working with shadow.
   Based on Tim's work

*  PVH - PCI passthrough for DomU.

*  AMD performance regressions

*  Performance due to hypercall preemption. More preemptions - slower. (none)

== Completed ==

*  alternative_asm in Xen (done)
  -  Feng Wu

*  SMAP (done)
  -  Feng Wu

*  Re-write of vHPET (done)
   aka hvm/hpet: Detect comparator values in the past
  -  Don Slutz

*  vAPIC in PVHVM guests (Xen side) (done)
  -  Boris Ostrovsky

*  libvirt and xl discard support, so that libvirt can start using it (done)
  -  Olaf Hering

*  Xen PVH dom0 (done)
  -  Mukesh Rathor

*  Linux PVH dom0 (done)
  -  Mukesh Rathor

*  OSSTest: upstream QEMU (done)
  -  Ian Campbell

*  amd_ucode cleanups, verify patch size(enhancement) (mostly in master except one patch)

*  Data breakpoint Extension support (new-feat) (in master)

*  Feature masking MSR support (enhancement) (in master)

*  Support BRCM TruManage chip (Serial over LAN support) (new-feat) (in master)

*  fix vmce_amd* functions, unify mce_amd mcheck initialization (fixes/cleanups)
  -  Aravind and Suravee

*  NUMA memory scrubbing (done)
  -  Konrad Rzeszutek Wilk

*  ARM  - IOMMU support (done)
  -  Julien Grall

*  ioreq-server, aka secondary emulators (done)
  -  Paul Durrant

*  Netback multiqueue (good)
  -  Wei Liu

*  ARM Interrupt latency reduction (no maintenance interrupts) (good)
  -  Stefano Stabellini

*  Bigger PCI hole in QEMU (done)
   Needs to be rebased
  -  Don Slutz

*  rework VM Generation ID (ok)
   v7 posted
  -  David Vrabel



[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Xen 4.5 development update

[George Dunlap]

On 07/01/2014 05:43 PM, konrad.wilk@oracle.com wrote:
> *  PoD fixes
>     if you boot with memory <= maxmem we have a size estimation bug

Speaking of which, did anyone do a write-up of what we discussed at the 
hackathon?

ISTR coming to some kind of conclusion, but I'm a bit hazy as to exactly 
what it was...

  -George

Re: Xen 4.5 development update

[Jan Beulich]

>>> On 02.07.14 at 13:33, <george.dunlap@eu.citrix.com> wrote:
> On 07/01/2014 05:43 PM, konrad.wilk@oracle.com wrote:
>> *  PoD fixes
>>     if you boot with memory <= maxmem we have a size estimation bug
> 
> Speaking of which, did anyone do a write-up of what we discussed at the 
> hackathon?
> 
> ISTR coming to some kind of conclusion, but I'm a bit hazy as to exactly 
> what it was...

IIRC we didn't really manage to finish up, but were at the point where
we thought XENMEM_current_reservation returning an unadjusted
value was wrong, fixing of which might be the way to go. Later I
started thinking that this is probably not correct - not just because
of the change in behavior of an existing hypercall, but also since the
current reservation of a domain _is_ d->tot_pages without any
adjustment.

Sadly I don't recall what exact adjustment we had been thinking of.

Jan

Re: Xen 4.5 development update

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1360 bytes --]

On mar, 2014-07-01 at 12:43 -0400, konrad.wilk@oracle.com wrote:

> none - nothing yet
> fair - still working on it, patches are prototypes or RFC
> ok   - patches posted, acting on review
> good - some last minute pieces
> done - all done, might have bugs

> = Open =

> == x86 == 
> 

> *  HT enabled, virtualization overhead is high (Xen 4.4) (none)
>    kernbench demonstrated it
>    looking and tracing it
>   -  Dario Faggioli
>
Right. Anyway, I shall say more about this in a new thread.

> *  Soft affinity for vcpus (was NUMA affinity for vcpus) (good)
>    v11 posted
>   -  Dario Faggioli
> 
Indeed. v12 coming today.

> *  Repurpose SEDF Scheduler for Real-time (ok)
>    RFC patch posted (v1)
>   -  Joshua Whitehead, Robert VanVossen
>
TBF, Patches are still RFC.

Also, either somehow related to this, or, if you want as a new separate
item (something like 'New Real-Time Scheduling'):

 Upstreaming bits of RT-Xen by Meng Xu, Sisu Xi
 RFC posted (v1)

Given the RFC status, I'd say this is fair, according to legend above.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Virt overehead with HT [was: Re: Xen 4.5 development update]

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2796 bytes --]

[Sorry to the ones that will receive this mail twice, but I managed to
drop xen-devel when replying the first time :-(]

On mar, 2014-07-01 at 12:43 -0400, konrad.wilk@oracle.com wrote:
> == x86 == 

> *  HT enabled, virtualization overhead is high (Xen 4.4) (none)
>    kernbench demonstrated it
>    looking and tracing it
>   -  Dario Faggioli
> 
I spent a few time running kernbench on different boxes and with
different configurations. After all this, here's what I found.

So, on a non-NUMA, HT and EPT capable box, both BAREMETAL and HVM case
were using 8G RAM and 8 CPUs/VCPUs. HT was enabled in BIOS:

Elapsed(stddev)   BAREMETAL             HVM
kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
kernbench -j      27.358 (0.440307)     27.49 (0.364897)

With HT disabled in BIOS (which means only 4 CPUs for both):
Elapsed(stddev)   BAREMETAL             HVM
kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
kernbench -j      32.316 (0.0270185)    33.084 (0.600442)

So, first of all, no much difference, in terms of performance
degradation when going from baremetal to guest, between the HT and no-HT
cases.

In the HT enabled case, there is a slight degradation in perf on the
least loaded case, but certainly not of the nature Stefano saw, and all
goes pretty well when load increases.

I guess I can investigate a bit more about what happens with '-j4'. What
I suspect is that the scheduler may make a few non-optimal decisions wrt
HT, when there are more PCPUs than busy guest VCPUs. This may be due to
the fact that Dom0 (or another guest VCPU doing other stuff than
kernbench) may be already running on PCPUs that are on different cores
than the guest's one (i.e., the guest VCPUs that wants to run
kernbench), and that may force two guest's vCPUs to execute on two HTs
some of the time (which of course is something that does not happen on
baremetal!).

However, that is, I think, a separate issue, and it looks to me that
the original HT perf regression we were chasing, the one this item is
about, may actually have been disappear, or it was caused to something
different than HT. :-P

Thoughts? Do we think this is enough to kill the "disable hyperthreading
hint" from the performance tuning page on the Wiki?
http://wiki.xenproject.org/wiki/Tuning_Xen_for_Performance#Hyperthreading_in_Xen_4.3_and_4.4

Regards,
Dario


-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Gordan Bobic]

On 07/14/2014 05:12 PM, Dario Faggioli wrote:
> [Sorry to the ones that will receive this mail twice, but I managed to
> drop xen-devel when replying the first time :-(]
>
> On mar, 2014-07-01 at 12:43 -0400, konrad.wilk@oracle.com wrote:
>> == x86 ==
>
>> *  HT enabled, virtualization overhead is high (Xen 4.4) (none)
>>     kernbench demonstrated it
>>     looking and tracing it
>>    -  Dario Faggioli
>>
> I spent a few time running kernbench on different boxes and with
> different configurations. After all this, here's what I found.
>
> So, on a non-NUMA, HT and EPT capable box, both BAREMETAL and HVM case
> were using 8G RAM and 8 CPUs/VCPUs. HT was enabled in BIOS:
>
> Elapsed(stddev)   BAREMETAL             HVM
> kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
> kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
> kernbench -j      27.358 (0.440307)     27.49 (0.364897)
>
> With HT disabled in BIOS (which means only 4 CPUs for both):
> Elapsed(stddev)   BAREMETAL             HVM
> kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
> kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
> kernbench -j      32.316 (0.0270185)    33.084 (0.600442)

Just to make sure I'm reading this right - _disabling_ HT causes a near 
50% performance drop?

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1753 bytes --]

On Mon, 2014-07-14 at 17:32 +0100, Gordan Bobic wrote:
> On 07/14/2014 05:12 PM, Dario Faggioli wrote:
> > Elapsed(stddev)   BAREMETAL             HVM
> > kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
> > kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
> > kernbench -j      27.358 (0.440307)     27.49 (0.364897)
> >
> > With HT disabled in BIOS (which means only 4 CPUs for both):
> > Elapsed(stddev)   BAREMETAL             HVM
> > kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
> > kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
> > kernbench -j      32.316 (0.0270185)    33.084 (0.600442)
> 
BTW, there's a mistake here. The three runs, in the no-HT case are as
follows:
 kernbench -j2
 kernbench -j4
 kernbench -j

I.e., half the number of VCPUs, as much as there are VCPUs and
unlimited, exactly as for the HT case.

The numbers are the right one.

> Just to make sure I'm reading this right - _disabling_ HT causes a near 
> 50% performance drop?
> 
For kernbench, and if you consider the "-j <half_of_nr_cpus>" run, yes,
nearly. And that is both for baremetal and HVM guest. And with
baremetal, I mean just bare Linux, no Xen at all involved.

Doesn't this make sense? Well, perhaps the wrong indication I gave about
the actual number of jobs used was misleading... better now?

BTW, the idea here was to compare perf between baremetal and HVM, and
they appear to be consistent.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[George Dunlap]

On 07/14/2014 05:44 PM, Dario Faggioli wrote:
> On Mon, 2014-07-14 at 17:32 +0100, Gordan Bobic wrote:
>> On 07/14/2014 05:12 PM, Dario Faggioli wrote:
>>> Elapsed(stddev)   BAREMETAL             HVM
>>> kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
>>> kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
>>> kernbench -j      27.358 (0.440307)     27.49 (0.364897)
>>>
>>> With HT disabled in BIOS (which means only 4 CPUs for both):
>>> Elapsed(stddev)   BAREMETAL             HVM
>>> kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
>>> kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
>>> kernbench -j      32.316 (0.0270185)    33.084 (0.600442)
> BTW, there's a mistake here. The three runs, in the no-HT case are as
> follows:
>   kernbench -j2
>   kernbench -j4
>   kernbench -j
>
> I.e., half the number of VCPUs, as much as there are VCPUs and
> unlimited, exactly as for the HT case.

Ah -- that's a pretty critical piece of information.

So actually, on native, HT enabled and disabled effectively produce the 
same exact thing if HT is not actually being used:  31 seconds in both 
cases.  But on Xen, enabling HT when it's not being used (i.e., when in 
theory each core should have exactly one process running), performance 
goes from 31 seconds to 34 seconds -- roughly a 10% degradation.

  -George

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2921 bytes --]

On Mon, 2014-07-14 at 17:55 +0100, George Dunlap wrote:
> On 07/14/2014 05:44 PM, Dario Faggioli wrote:
> > On Mon, 2014-07-14 at 17:32 +0100, Gordan Bobic wrote:
> >> On 07/14/2014 05:12 PM, Dario Faggioli wrote:
> >>> Elapsed(stddev)   BAREMETAL             HVM
> >>> kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
> >>> kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
> >>> kernbench -j      27.358 (0.440307)     27.49 (0.364897)
> >>>
> >>> With HT disabled in BIOS (which means only 4 CPUs for both):
> >>> Elapsed(stddev)   BAREMETAL             HVM
> >>> kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
> >>> kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
> >>> kernbench -j      32.316 (0.0270185)    33.084 (0.600442)
> > BTW, there's a mistake here. The three runs, in the no-HT case are as
> > follows:
> >   kernbench -j2
> >   kernbench -j4
> >   kernbench -j
> >
> > I.e., half the number of VCPUs, as much as there are VCPUs and
> > unlimited, exactly as for the HT case.
> 
> Ah -- that's a pretty critical piece of information.
> 
> So actually, on native, HT enabled and disabled effectively produce the 
> same exact thing if HT is not actually being used:  31 seconds in both 
> cases.  But on Xen, enabling HT when it's not being used (i.e., when in 
> theory each core should have exactly one process running), performance 
> goes from 31 seconds to 34 seconds -- roughly a 10% degradation.
> 
Yes. 7.96% degradation, to be precise.

I attempted an analysis in my first e-mail. Cutting and pasting it
here... What do you think?

"I guess I can investigate a bit more about what happens with '-j4'.
 What I suspect is that the scheduler may make a few non-optimal
 decisions wrt HT, when there are more PCPUs than busy guest VCPUs. This
 may be due to the fact that Dom0 (or another guest VCPU doing other
 stuff than kernbench) may be already running on PCPUs that are on
 different cores than the guest's one (i.e., the guest VCPUs that wants
 to run kernbench), and that may force two guest's vCPUs to execute on
 two HTs some of the time (which of course is something that does not
 happen on baremetal!)."

I just re-run the benchmark with credit2, which has no SMT knowledge,
and the first run (the one that does not use HT)  ended up to be 37.54,
while the other two were pretty much the same of above (26.81 and
27.92).

This confirms, for me, that it's an SMT balancing issue that we're seen.

I'll try more runs, e.g. with number of VCPUs equal less than
nr_corse/2 and see what happens.

Again, thoughts?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Gordan Bobic]

On 07/14/2014 06:22 PM, Dario Faggioli wrote:
> On Mon, 2014-07-14 at 17:55 +0100, George Dunlap wrote:
>> On 07/14/2014 05:44 PM, Dario Faggioli wrote:
>>> On Mon, 2014-07-14 at 17:32 +0100, Gordan Bobic wrote:
>>>> On 07/14/2014 05:12 PM, Dario Faggioli wrote:
>>>>> Elapsed(stddev)   BAREMETAL             HVM
>>>>> kernbench -j4     31.604 (0.0963328)    34.078 (0.168582)
>>>>> kernbench -j8     26.586 (0.145705)     26.672 (0.0432435)
>>>>> kernbench -j      27.358 (0.440307)     27.49 (0.364897)
>>>>>
>>>>> With HT disabled in BIOS (which means only 4 CPUs for both):
>>>>> Elapsed(stddev)   BAREMETAL             HVM
>>>>> kernbench -j4     57.754 (0.0642651)    56.46 (0.0578792)
>>>>> kernbench -j8     31.228 (0.0775887)    31.362 (0.210998)
>>>>> kernbench -j      32.316 (0.0270185)    33.084 (0.600442)
>>> BTW, there's a mistake here. The three runs, in the no-HT case are as
>>> follows:
>>>    kernbench -j2
>>>    kernbench -j4
>>>    kernbench -j
>>>
>>> I.e., half the number of VCPUs, as much as there are VCPUs and
>>> unlimited, exactly as for the HT case.
>>
>> Ah -- that's a pretty critical piece of information.
>>
>> So actually, on native, HT enabled and disabled effectively produce the
>> same exact thing if HT is not actually being used:  31 seconds in both
>> cases.  But on Xen, enabling HT when it's not being used (i.e., when in
>> theory each core should have exactly one process running), performance
>> goes from 31 seconds to 34 seconds -- roughly a 10% degradation.
>>
> Yes. 7.96% degradation, to be precise.
>
> I attempted an analysis in my first e-mail. Cutting and pasting it
> here... What do you think?
>
> "I guess I can investigate a bit more about what happens with '-j4'.
>   What I suspect is that the scheduler may make a few non-optimal
>   decisions wrt HT, when there are more PCPUs than busy guest VCPUs. This
>   may be due to the fact that Dom0 (or another guest VCPU doing other
>   stuff than kernbench) may be already running on PCPUs that are on
>   different cores than the guest's one (i.e., the guest VCPUs that wants
>   to run kernbench), and that may force two guest's vCPUs to execute on
>   two HTs some of the time (which of course is something that does not
>   happen on baremetal!)."
>
> I just re-run the benchmark with credit2, which has no SMT knowledge,
> and the first run (the one that does not use HT)  ended up to be 37.54,
> while the other two were pretty much the same of above (26.81 and
> 27.92).
>
> This confirms, for me, that it's an SMT balancing issue that we're seen.
>
> I'll try more runs, e.g. with number of VCPUs equal less than
> nr_corse/2 and see what happens.
>
> Again, thoughts?

Have you tried it with VCPUs pinned to appropriate PCPUs?

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1996 bytes --]

On lun, 2014-07-14 at 19:31 +0100, Gordan Bobic wrote:
> On 07/14/2014 06:22 PM, Dario Faggioli wrote:

> > I'll try more runs, e.g. with number of VCPUs equal less than
> > nr_corse/2 and see what happens.
> >
> > Again, thoughts?
> 
> Have you tried it with VCPUs pinned to appropriate PCPUs?
> 
Define "appropriate".

I have a run for which I pinned VCPU#1-->PCPU#1, VCPU#2-->PCPU#2, and so
on, and the result is even worse:

Average Half load -j 4 Run (std deviation):
 Elapsed Time 37.808 (0.538999)
Average Optimal load -j 8 Run (std deviation):
 Elapsed Time 26.594 (0.235223)
Average Maximal load -j Run (std deviation):
 Elapsed Time 27.9 (0.131149)

This is actually something I expected, since you do not allow the VCPUs
to move away from an HT with a busy sibling, even when it could have.

In fact, you may expect better result from pinning only if you were to
pin not only the VCPUs to the PCPUs, but also the kernbench's build jobs
on the appropriate (V)CPUs in the guest.. but that's something not only
really unpractical, but also very few representative as a benchmark, I
think.

If you pin VCPU#1 to PCPU#1 and VCPU#2 to PCPU#2, with PCPU#1 and PCPU#2
being HT siblings, what prevents Linux (in the guest) to run two of the
four build jobs on VCPU#1 and VCPU#2 (i.e., on siblings PCPUs!!) for all
the length of the benchmark? Nothing, I think.

And in fact, pinning would also result in good (near to native,
perhaps?) performance, if we were exposing the SMT topology details to
guests as, in that case, Linux would do the balancing properly. However,
that's not the case either. :-(

But, perhaps, you were referring to a different pinning strategy?

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Gordan Bobic]

On 07/14/2014 11:44 PM, Dario Faggioli wrote:
> On lun, 2014-07-14 at 19:31 +0100, Gordan Bobic wrote:
>> On 07/14/2014 06:22 PM, Dario Faggioli wrote:
>
>>> I'll try more runs, e.g. with number of VCPUs equal less than
>>> nr_corse/2 and see what happens.
>>>
>>> Again, thoughts?
>>
>> Have you tried it with VCPUs pinned to appropriate PCPUs?
>>
> Define "appropriate".
>
> I have a run for which I pinned VCPU#1-->PCPU#1, VCPU#2-->PCPU#2, and so
> on, and the result is even worse:
>
> Average Half load -j 4 Run (std deviation):
>   Elapsed Time 37.808 (0.538999)
> Average Optimal load -j 8 Run (std deviation):
>   Elapsed Time 26.594 (0.235223)
> Average Maximal load -j Run (std deviation):
>   Elapsed Time 27.9 (0.131149)
>
> This is actually something I expected, since you do not allow the VCPUs
> to move away from an HT with a busy sibling, even when it could have.
>
> In fact, you may expect better result from pinning only if you were to
> pin not only the VCPUs to the PCPUs, but also the kernbench's build jobs
> on the appropriate (V)CPUs in the guest.. but that's something not only
> really unpractical, but also very few representative as a benchmark, I
> think.
>
> If you pin VCPU#1 to PCPU#1 and VCPU#2 to PCPU#2, with PCPU#1 and PCPU#2
> being HT siblings, what prevents Linux (in the guest) to run two of the
> four build jobs on VCPU#1 and VCPU#2 (i.e., on siblings PCPUs!!) for all
> the length of the benchmark? Nothing, I think.

That would imply that Xen can somehow make a better decision that the 
domU's kernel scheduler, something that doesn't seem that likely. I 
would expect not pinning CPUs to increase process migration because Xen 
might migrate the CPU even though the kernel in domU decided which 
presented CPU was most lightly loaded.

> And in fact, pinning would also result in good (near to native,
> perhaps?) performance, if we were exposing the SMT topology details to
> guests as, in that case, Linux would do the balancing properly. However,
> that's not the case either. :-(

I see, so you are referring specifically to the HT case. I can see how 
that could cause a problem. Does pinning improve the performance with HT 
disabled?

Gordan

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2067 bytes --]

On mar, 2014-07-15 at 01:10 +0100, Gordan Bobic wrote:
> On 07/14/2014 11:44 PM, Dario Faggioli wrote:

> > If you pin VCPU#1 to PCPU#1 and VCPU#2 to PCPU#2, with PCPU#1 and PCPU#2
> > being HT siblings, what prevents Linux (in the guest) to run two of the
> > four build jobs on VCPU#1 and VCPU#2 (i.e., on siblings PCPUs!!) for all
> > the length of the benchmark? Nothing, I think.
> 
> That would imply that Xen can somehow make a better decision that the 
> domU's kernel scheduler, something that doesn't seem that likely.
>
Well, as far as SMT load balancing is concerned, that is _exactly_ the
case. The reason is simple: Xen knows the hw topology, and hence knows
whether the sibling of an idle core is idle or busy. The guest kernel
sees nothing about this, it just treat all its (V)CPUs as full cores, so
it most likely will do a bad job in this case.

> > And in fact, pinning would also result in good (near to native,
> > perhaps?) performance, if we were exposing the SMT topology details to
> > guests as, in that case, Linux would do the balancing properly. However,
> > that's not the case either. :-(
> 
> I see, so you are referring specifically to the HT case. 
>
Yeah, well, that's what this benchmarks where all about  :-)

> I can see how 
> that could cause a problem. Does pinning improve the performance with HT 
> disabled?
> 
HT disabled had pretty goo perf. already. Anyhow, I tried:

Average Half load -j 2 Run (std deviation):
 Elapsed Time 56.462 (0.109179)
Average Optimal load -j 4 Run (std deviation):
 Elapsed Time 31.526 (0.224789)
Average Maximal load -j Run (std deviation):
 Elapsed Time 33.04 (0.439147)

So a lot similar to the no-HT unpinned case, which on it's turn was a
lot similar to baremetal without HT.

Daio

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Virt overehead with HT [was: Re: Xen 4.5 development update]

[Gordan Bobic]

On 07/15/2014 03:30 AM, Dario Faggioli wrote:
> On mar, 2014-07-15 at 01:10 +0100, Gordan Bobic wrote:
>> On 07/14/2014 11:44 PM, Dario Faggioli wrote:
>
>>> If you pin VCPU#1 to PCPU#1 and VCPU#2 to PCPU#2, with PCPU#1 and PCPU#2
>>> being HT siblings, what prevents Linux (in the guest) to run two of the
>>> four build jobs on VCPU#1 and VCPU#2 (i.e., on siblings PCPUs!!) for all
>>> the length of the benchmark? Nothing, I think.
>>
>> That would imply that Xen can somehow make a better decision that the
>> domU's kernel scheduler, something that doesn't seem that likely.
>>
> Well, as far as SMT load balancing is concerned, that is _exactly_ the
> case. The reason is simple: Xen knows the hw topology, and hence knows
> whether the sibling of an idle core is idle or busy. The guest kernel
> sees nothing about this, it just treat all its (V)CPUs as full cores, so
> it most likely will do a bad job in this case.
>
>>> And in fact, pinning would also result in good (near to native,
>>> perhaps?) performance, if we were exposing the SMT topology details to
>>> guests as, in that case, Linux would do the balancing properly. However,
>>> that's not the case either. :-(
>>
>> I see, so you are referring specifically to the HT case.
>>
> Yeah, well, that's what this benchmarks where all about  :-)
>
>> I can see how
>> that could cause a problem. Does pinning improve the performance with HT
>> disabled?
>>
> HT disabled had pretty goo perf. already. Anyhow, I tried:
>
> Average Half load -j 2 Run (std deviation):
>   Elapsed Time 56.462 (0.109179)
> Average Optimal load -j 4 Run (std deviation):
>   Elapsed Time 31.526 (0.224789)
> Average Maximal load -j Run (std deviation):
>   Elapsed Time 33.04 (0.439147)
>
> So a lot similar to the no-HT unpinned case, which on it's turn was a
> lot similar to baremetal without HT.

Just out of interest - in cases where there is a non-negligible 
performance discrepancy with HT enabled (bare metal or Xen), does 
disabling the C6 CPU state support in the BIOS help? C6 state 
selectively disables CPU threads when the CPU is idle for power saving 
purposes, but the disabling threshold can be too sensitive. Does Xen 
handle this?

Gordan