PV performance degraded after live migration

PV performance degraded after live migration

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 1552 bytes --]

After reports about degraded performance after a PV domU was migrated
from one dom0 to another it turned out that this issue happens with
every version of Xen and every version of domU kernel.

The used benchmark is 'sysbench memory'. I hacked it up to show how long
the actual work takes, and that loop takes longer to execute after the
domU is migrated. In my testing the loop (memory_execute_event) takes
about 1200ns, after migration it takes about 1500ns. It just writes 0 to
an array of memory. In total sysbench reports 6500 MiB/sec, after
migration its just 3350 MiB/sec.
The source of the modified test can be found here:
https://github.com/olafhering/sysbench/compare/master...pv

This happens on several hosts. NUMA or not makes no difference. CPU
pinning or not makes no difference. Pinning of the test pthreads makes
no difference. It was initially reported with xen-4.4, I see it with
staging too. The guest kernel makes no difference, several variants of
xenlinux or pvops based show the slowdown. Also live migration to
localhost is affected.

The domU.cfg looks like that:
name='pv'
memory=1024
vcpus=4
cpus=[ "4", "5", "6", "7" ]
disk=[ 'file:/disk0.raw,xvda,w', ]
vif=[ 'bridge=br0,type=netfront', ]
vfb=[ 'type=vnc,vncunused=1,keymap=de', ]
serial="pty"
kernel="/usr/lib/grub2/x86_64-xen/grub.xen"

Xen is booted with "console=com1 com1=115200 loglvl=all guest_loglvl=all
dom0_max_vcpus=2 dom0_vcpus_pin=on".

I wonder what the cause might be, and how to check where the time is
spent.


Olaf

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Andrew Cooper]

On 15/03/17 11:20, Olaf Hering wrote:
> After reports about degraded performance after a PV domU was migrated
> from one dom0 to another it turned out that this issue happens with
> every version of Xen and every version of domU kernel.
>
> The used benchmark is 'sysbench memory'. I hacked it up to show how long
> the actual work takes, and that loop takes longer to execute after the
> domU is migrated. In my testing the loop (memory_execute_event) takes
> about 1200ns, after migration it takes about 1500ns. It just writes 0 to
> an array of memory. In total sysbench reports 6500 MiB/sec, after
> migration its just 3350 MiB/sec.
> The source of the modified test can be found here:
> https://github.com/olafhering/sysbench/compare/master...pv
>
> This happens on several hosts. NUMA or not makes no difference. CPU
> pinning or not makes no difference. Pinning of the test pthreads makes
> no difference. It was initially reported with xen-4.4, I see it with
> staging too. The guest kernel makes no difference, several variants of
> xenlinux or pvops based show the slowdown. Also live migration to
> localhost is affected.
>
> The domU.cfg looks like that:
> name='pv'
> memory=1024
> vcpus=4
> cpus=[ "4", "5", "6", "7" ]
> disk=[ 'file:/disk0.raw,xvda,w', ]
> vif=[ 'bridge=br0,type=netfront', ]
> vfb=[ 'type=vnc,vncunused=1,keymap=de', ]
> serial="pty"
> kernel="/usr/lib/grub2/x86_64-xen/grub.xen"
>
> Xen is booted with "console=com1 com1=115200 loglvl=all guest_loglvl=all
> dom0_max_vcpus=2 dom0_vcpus_pin=on".
>
> I wonder what the cause might be, and how to check where the time is
> spent.

As a crazy idea, doest this help?

diff --git a/tools/libxc/xc_sr_common_x86.c b/tools/libxc/xc_sr_common_x86.c
index 98f1cef..ee8089d 100644
--- a/tools/libxc/xc_sr_common_x86.c
+++ b/tools/libxc/xc_sr_common_x86.c
@@ -34,7 +34,7 @@ int handle_tsc_info(struct xc_sr_context *ctx, struct
xc_sr_record *rec)
     }
 
     if ( xc_domain_set_tsc_info(xch, ctx->domid, tsc->mode,
-                                tsc->nsec, tsc->khz, tsc->incarnation) )
+                                tsc->nsec, tsc->khz, 0 /*
tsc->incarnation */) )
     {
         PERROR("Unable to set TSC information");
         return -1;


~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 242 bytes --]

On Wed, Mar 15, Andrew Cooper wrote:

> As a crazy idea, doest this help?

> tsc->incarnation = 0

Had to move to another testhost and this seems to help. Will do more testing
once the original testsystems are accessible again. Thanks!

Olaf

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 5272 bytes --]

On Wed, Mar 15, Olaf Hering wrote:

> On Wed, Mar 15, Andrew Cooper wrote:
> > As a crazy idea, doest this help?
> > tsc->incarnation = 0

This does indeed help. One system shows now the results below, which
means the performance goes down during migration (to localhost) and goes
back to normal after migration.

What impact has such change to ->incarnation?

Olaf

./sysbench --memory-block-size=8k --memory-total-size=1024T --verbosity=5 --threads=4 --report-interval=3 --time=1234 memory run
...
[ 3s ]  6322.61 MiB/sec 2421800: 96212(006fb 0088a 00c7e6) 96387(0074f 0083d 00506f) 926b7(006d4 0088f 00e5c9) 92706(00488 008c4 0458a6) 
[ 6s ]  6319.72 MiB/sec 2418340: 95dab(004a9 00864 01e42f) 961cc(006dc 0087a 00e824) 921b3(0049b 00898 00cbd5) 924e5(00705 007ea 00f51e) 
[ 9s ]  6319.37 MiB/sec 2418534: 95f1d(00748 009ab 00e5e4) 9607f(006aa 008ef 00b358) 92237(00497 008a6 00b332) 9253a(0071a 008a8 00e605) 
[ 12s ]  6320.20 MiB/sec 2419101: 95c70(006fb 008cb 08cbb8) 9611b(00722 00855 00dfc1) 92383(006ce 008d6 00fbc6) 92745(006de 0073c 00fbde) 
[ 15s ]  6320.12 MiB/sec 2419118: 95fc3(00779 00867 00e36b) 96199(00703 0082f 00e356) 9229e(004ab 0089b 1b5c72) 9242f(00714 0076b 04807b) 
[ 18s ]  6320.38 MiB/sec 2419150: 95ede(00717 0086b 008fbf) 9618b(006de 00855 00c119) 922e8(0049c 00846 1b1f1a) 92548(00725 007bd 0052f5) 
[ 21s ]  6321.28 MiB/sec 2419539: 95f23(006fc 008bc 00cce3) 9624c(00701 008ae 00bec0) 921ee(00495 008b7 1b6358) 92686(0071e 008c6 00bf2e) 
[ 24s ]  6319.37 MiB/sec 2418534: 96061(006e2 00904 00c7c8) 96131(006de 00812 00de5d) 9213e(006d9 00872 1b29a0) 9243e(00712 007cd 00df76) 
[ 27s ]  6320.15 MiB/sec 2418970: 95e0c(006e1 00869 00df81) 9621a(0056f 0088e 00dfc1) 921f4(004ad 008ae 1b309c) 9261e(006e6 00895 00ee5d) 
[ 30s ]  6322.09 MiB/sec 2419663: 96014(0045f 0084a 00ea91) 96142(006fb 00868 00eb32) 922f1(006c8 008cc 1b2ce4) 926db(006f8 007b9 00e6f7) 
[ 33s ]  6321.72 MiB/sec 2419470: 95f3a(00506 008ed 00c75c) 96223(006b2 008b5 00b685) 92412(00497 00909 1b1e69) 92527(00727 0084e 00c72c) 
[ 36s ]  6321.09 MiB/sec 2419262: 95ffa(006f5 00878 00e7fc) 960e5(006de 00868 00e87d) 921ea(00763 008eb 1b28e2) 926da(00461 007ca 00ed17) 
[ 39s ]  6321.06 MiB/sec 2419519: 95df8(006e8 0095f 00fee4) 96175(006da 00847 00ea77) 92334(004a5 00915 00fecb) 926f5(006ac 008c8 0050e0) 
[ 42s ]  6319.72 MiB/sec 2418867: 95f05(004c7 00885 00b39d) 9620a(006fd 0086b 00b70c) 9208b(0048f 0089f 1b24c4) 925fa(006f2 007eb 00b6d6) 
[ 45s ]  6323.08 MiB/sec 2420218: 95fbe(0075a 00864 00eede) 96298(00749 0086b 004397) 923c9(0049d 008f7 1b1c50) 9267f(0070f 00753 04003e) 
[ 48s ]  6317.78 MiB/sec 2417897: 95dde(0076c 00862 00e858) 9612c(006bd 008a7 00e859) 92031(0048d 00836 00e916) 92570(006cc 0078b 00e8d7) 
[ 51s ]  6321.18 MiB/sec 2419442: 95f84(0070d 00829 00e37a) 9616a(006bd 0081f 00ea08) 9224b(006e4 008ac 00e9d6) 9268d(00724 007bc 00e461) 
[ 54s ]  6321.25 MiB/sec 2419366: 95ed3(004b1 0089e 004103) 961c7(0055e 008aa 00b832) 922c2(0048f 00857 1b1880) 92682(00706 0081f 00deeb) 
[ 57s ]  6322.12 MiB/sec 2419798: 95f73(00766 00825 00e8e5) 9620a(006d5 008a1 00e841) 92359(00497 00890 1b16b2) 9265a(006dc 0085d 00e8d7) 
[ 60s ]  6320.37 MiB/sec 2419005: 95fc0(006d4 0082b 002927) 96180(006df 0088a 005721) 92224(0052b 00960 00be51) 92528(00706 008da 00bd32) 
[ 63s ]  6318.10 MiB/sec 2418347: 95e00(006ec 008a0 00c124) 96307(006c7 00801 0043ac) 91feb(00709 00851 00c226) 92433(006e6 00802 00bbbf) 
[ 66s ]  6320.55 MiB/sec 2419144: 95ed1(006f8 00825 00c152) 9636e(00774 00832 00bb27) 92107(004a3 008d7 00c21b) 9258d(00714 007e1 006673) 
[ 69s ]  6319.12 MiB/sec 2418691: 95f9a(006ea 00864 00c31c) 962ba(006f8 00891 00b9d4) 92022(006a5 008ed 00c3a0) 9244b(00680 00789 00bc68) 
[ 72s ]  1522.05 MiB/sec 584258: 23c48(0077e 0172d 0c9a40) 23c92(0068d 04250 0c9b64) 2391e(004a4 01561 11369a) 2398f(00733 01412 0c6867) 
[ 75s ]  1284.93 MiB/sec 493314: 1e1db(00d70 0105a 00c227) 1e1e6(00db9 0130f 00db37) 1e18e(00df0 01677 00f62f) 1e1b3(00da8 00fd6 1a1b4f) 
[ 78s ]  1301.39 MiB/sec 499734: 1e79f(00cd3 01034 420524) 1e867(00cbc 01071 00871e) 1e7ec(0097f 01732 00f5fc) 1e825(00d10 012bf 00cf0f) 
[ 81s ]  1300.86 MiB/sec 499525: 1e7ea(00d6a 0121b 01a401) 1e7e7(00c65 0108d 00f67b) 1e799(009a2 0186d 00adcc) 1e7dc(00e20 01157 015b74) 
[ 84s ]  1290.08 MiB/sec 495391: 1e3d8(00e13 01133 007bb1) 1e3e8(00d0a 01395 00d77c) 1e399(00828 0187c 00ee9a) 1e3c7(00d15 0115c 00ae06) 
[ 87s ]  1297.33 MiB/sec 498175: 1e6ae(00d13 010e4 00f34e) 1e686(00be0 00ebe 00c62f) 1e64b(00cbe 015ee 01118b) 1e681(00d96 0118d 01ea53) 
[ 90s ]  4939.91 MiB/sec 1890347: 750c7(00635 00844 3898ce) 751ee(00657 008b1 300e9fe) 724c2(0066d 009a8 2f3c63) 72a6a(0065c 00785 2dc2723) 
[ 93s ]  6232.37 MiB/sec 2385348: 936ff(0045f 0087b 2d56e3) 946ff(0067d 00889 21db77) 90e65(00486 008ce 1a6b04) 8f82a(006cb 008af 298f81) 
[ 96s ]  6320.31 MiB/sec 2419131: 95fda(006e3 0086c 00c323) 96229(006c8 00874 01132a) 920b0(00771 00890 1b88ae) 925b6(00723 0078f 00c2af) 
[ 99s ]  6320.29 MiB/sec 2419279: 95f57(006e1 0087a 00b85e) 96152(00721 0084b 003ada) 9214e(00496 0084f 003d47) 92679(006d4 007d2 00c492) 
[ 102s ]  6318.06 MiB/sec 2418173: 95f2f(006f7 0084e 00c617) 96233(0070c 0084f 00c621) 92056(004a6 0088e 1b2abd) 9235b(006c5 0081c 3406f9) 


[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Andrew Cooper]

On 15/03/17 15:23, Olaf Hering wrote:
> On Wed, Mar 15, Olaf Hering wrote:
>
>> On Wed, Mar 15, Andrew Cooper wrote:
>>> As a crazy idea, doest this help?
>>> tsc->incarnation = 0
> This does indeed help. One system shows now the results below, which
> means the performance goes down during migration (to localhost) and goes
> back to normal after migration.
>
> What impact has such change to ->incarnation?

So what this means is that, after migrate, Xen sees that the advertised
TSC value doesn't match the current hardwares TSC, so enables rdtsc
interception so the values reported back can be emulated at the old
frequency.

There is no easy solution to this problem.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Juergen Gross]

On 15/03/17 16:26, Andrew Cooper wrote:
> On 15/03/17 15:23, Olaf Hering wrote:
>> On Wed, Mar 15, Olaf Hering wrote:
>>
>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>> As a crazy idea, doest this help?
>>>> tsc->incarnation = 0
>> This does indeed help. One system shows now the results below, which
>> means the performance goes down during migration (to localhost) and goes
>> back to normal after migration.
>>
>> What impact has such change to ->incarnation?
> 
> So what this means is that, after migrate, Xen sees that the advertised
> TSC value doesn't match the current hardwares TSC, so enables rdtsc
> interception so the values reported back can be emulated at the old
> frequency.

Why does Xen detect a different frequency after a localhost migration?


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Alan Robinson]

Hi Olaf,

On Wed, Mar 15, 2017 at 12:20:44PM +0100, Olaf Hering wrote:
> After reports about degraded performance after a PV domU was migrated
> from one dom0 to another it turned out that this issue happens with
> every version of Xen and every version of domU kernel.
> 
> The used benchmark is 'sysbench memory'. I hacked it up to show how long
> the actual work takes, and that loop takes longer to execute after the
> domU is migrated. In my testing the loop (memory_execute_event) takes
> about 1200ns, after migration it takes about 1500ns. It just writes 0 to
> an array of memory. In total sysbench reports 6500 MiB/sec, after
> migration its just 3350 MiB/sec.

We have seen something similar being caused by the superpage
attribute getting dropped when the domU was migrated. The new
copy of the domU only has 4K pages instead of 2MB pages eventually
this would seem to fit your memory access sysbench results. 

Alan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Boris Ostrovsky]

On 03/15/2017 11:26 AM, Andrew Cooper wrote:
> On 15/03/17 15:23, Olaf Hering wrote:
>> On Wed, Mar 15, Olaf Hering wrote:
>>
>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>> As a crazy idea, doest this help?
>>>> tsc->incarnation = 0
>> This does indeed help. One system shows now the results below, which
>> means the performance goes down during migration (to localhost) and goes
>> back to normal after migration.
>>
>> What impact has such change to ->incarnation?
> So what this means is that, after migrate, Xen sees that the advertised
> TSC value doesn't match the current hardwares TSC, so enables rdtsc
> interception so the values reported back can be emulated at the old
> frequency.
>
> There is no easy solution to this problem.

Would

    tsc_mode="never"

help?

-boris

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Andrew Cooper]

On 15/03/17 15:43, Alan Robinson wrote:
> Hi Olaf,
>
> On Wed, Mar 15, 2017 at 12:20:44PM +0100, Olaf Hering wrote:
>> After reports about degraded performance after a PV domU was migrated
>> from one dom0 to another it turned out that this issue happens with
>> every version of Xen and every version of domU kernel.
>>
>> The used benchmark is 'sysbench memory'. I hacked it up to show how long
>> the actual work takes, and that loop takes longer to execute after the
>> domU is migrated. In my testing the loop (memory_execute_event) takes
>> about 1200ns, after migration it takes about 1500ns. It just writes 0 to
>> an array of memory. In total sysbench reports 6500 MiB/sec, after
>> migration its just 3350 MiB/sec.
> We have seen something similar being caused by the superpage
> attribute getting dropped when the domU was migrated. The new
> copy of the domU only has 4K pages instead of 2MB pages eventually
> this would seem to fit your memory access sysbench results.

PV guests with superpages can't be migrated at all.  There is no way to
guarantee that there is an available superpage on the destination, and
the absence of one can't be handled on behalf of the guest.

Before my migration v2 work, this was pot luck and you might resume with
the 2M mapping simply missing.  After my migration v2 work, this
situation is detected and causes an abort on the send side.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Jan Beulich]

>>> On 15.03.17 at 16:43, <Alan.Robinson@ts.fujitsu.com> wrote:
> On Wed, Mar 15, 2017 at 12:20:44PM +0100, Olaf Hering wrote:
>> After reports about degraded performance after a PV domU was migrated
>> from one dom0 to another it turned out that this issue happens with
>> every version of Xen and every version of domU kernel.
>> 
>> The used benchmark is 'sysbench memory'. I hacked it up to show how long
>> the actual work takes, and that loop takes longer to execute after the
>> domU is migrated. In my testing the loop (memory_execute_event) takes
>> about 1200ns, after migration it takes about 1500ns. It just writes 0 to
>> an array of memory. In total sysbench reports 6500 MiB/sec, after
>> migration its just 3350 MiB/sec.
> 
> We have seen something similar being caused by the superpage
> attribute getting dropped when the domU was migrated. The new
> copy of the domU only has 4K pages instead of 2MB pages eventually
> this would seem to fit your memory access sysbench results. 

That would be an explanation for HVM, but here we consider PV only.
And whether a HVM guest can be re-created using large pages
mainly depends on how fragmented the destination host memory is.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Juergen Gross]

On 15/03/17 16:43, Boris Ostrovsky wrote:
> On 03/15/2017 11:26 AM, Andrew Cooper wrote:
>> On 15/03/17 15:23, Olaf Hering wrote:
>>> On Wed, Mar 15, Olaf Hering wrote:
>>>
>>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>>> As a crazy idea, doest this help?
>>>>> tsc->incarnation = 0
>>> This does indeed help. One system shows now the results below, which
>>> means the performance goes down during migration (to localhost) and goes
>>> back to normal after migration.
>>>
>>> What impact has such change to ->incarnation?
>> So what this means is that, after migrate, Xen sees that the advertised
>> TSC value doesn't match the current hardwares TSC, so enables rdtsc
>> interception so the values reported back can be emulated at the old
>> frequency.
>>
>> There is no easy solution to this problem.
> 
> Would
> 
>     tsc_mode="never"
> 
> help?

Only for frequency matched hosts.

Hmm, especially for pv guests this should be solvable: after a migration
the guest kernel could resync the tsc frequency and there wouldn't be
further tsc emulation needed. This would just require a new hypercall
for obtaining the current tsc frequency. This hypercall would:

- switch off tsc emulation for the calling vcpu (if allowed by tsc_mode)
- return the real tsc frequency (and offset?)

As the guest kernel is aware of the migration it could issue the new
hypercall on each vcpu and everyone is happy again.

Thoughts?


Juergen


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Alan Robinson]

On Wed, Mar 15, 2017 at 09:53:02AM -0600, Jan Beulich wrote:
> >>> On 15.03.17 at 16:43, <Alan.Robinson@ts.fujitsu.com> wrote:
...
> > We have seen something similar being caused by the superpage
> > attribute getting dropped when the domU was migrated. The new
> > copy of the domU only has 4K pages instead of 2MB pages eventually
> > this would seem to fit your memory access sysbench results. 
> 
> That would be an explanation for HVM, but here we consider PV only.
> And whether a HVM guest can be re-created using large pages
> mainly depends on how fragmented the destination host memory is.
>

Yes indeed - sorry for the HVM noise..

Alan

-- 
Alan Robinson
Principal Developer, Enterprise Platform Services, Germany

Fujitsu
Mies-van-der-Rohe-Str. 8, 80807 Muenchen, Deutschland
Tel.: +49 (89) 62060 3927
Mob.: +49 (172) 8512843
E-Mail: Alan.Robinson@ts.fujitsu.com
Web: http://www.fujitsu.com/de/

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Jan Beulich]

>>> On 15.03.17 at 16:26, <andrew.cooper3@citrix.com> wrote:
> On 15/03/17 15:23, Olaf Hering wrote:
>> On Wed, Mar 15, Olaf Hering wrote:
>>
>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>> As a crazy idea, doest this help?
>>>> tsc->incarnation = 0
>> This does indeed help. One system shows now the results below, which
>> means the performance goes down during migration (to localhost) and goes
>> back to normal after migration.
>>
>> What impact has such change to ->incarnation?
> 
> So what this means is that, after migrate, Xen sees that the advertised
> TSC value doesn't match the current hardwares TSC, so enables rdtsc
> interception so the values reported back can be emulated at the old
> frequency.
> 
> There is no easy solution to this problem.

Especially for localhost migration it ought to be possible to deal
with this. What I'm wondering is why the frequency comparison
in default mode handling in tsc_set_info() is HVM-only:

        if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() &&
             (has_hvm_container_domain(d) ?
              (d->arch.tsc_khz == cpu_khz ||
               hvm_get_tsc_scaling_ratio(d->arch.tsc_khz)) :
              incarnation == 0) )
        {

Boris, you've made it be this way in 82713ec8d2 ("x86: use native
RDTSC(P) execution when guest and host frequencies are the
same").

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Jan Beulich]

>>> On 15.03.17 at 16:53, <JGross@suse.com> wrote:
> On 15/03/17 16:43, Boris Ostrovsky wrote:
>> On 03/15/2017 11:26 AM, Andrew Cooper wrote:
>>> On 15/03/17 15:23, Olaf Hering wrote:
>>>> On Wed, Mar 15, Olaf Hering wrote:
>>>>
>>>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>>>> As a crazy idea, doest this help?
>>>>>> tsc->incarnation = 0
>>>> This does indeed help. One system shows now the results below, which
>>>> means the performance goes down during migration (to localhost) and goes
>>>> back to normal after migration.
>>>>
>>>> What impact has such change to ->incarnation?
>>> So what this means is that, after migrate, Xen sees that the advertised
>>> TSC value doesn't match the current hardwares TSC, so enables rdtsc
>>> interception so the values reported back can be emulated at the old
>>> frequency.
>>>
>>> There is no easy solution to this problem.
>> 
>> Would
>> 
>>     tsc_mode="never"
>> 
>> help?
> 
> Only for frequency matched hosts.
> 
> Hmm, especially for pv guests this should be solvable: after a migration
> the guest kernel could resync the tsc frequency and there wouldn't be
> further tsc emulation needed. This would just require a new hypercall
> for obtaining the current tsc frequency. This hypercall would:
> 
> - switch off tsc emulation for the calling vcpu (if allowed by tsc_mode)
> - return the real tsc frequency (and offset?)
> 
> As the guest kernel is aware of the migration it could issue the new
> hypercall on each vcpu and everyone is happy again.

Why a new hypercall? The guest has all the information available.
Not sure where in pv-ops code this is, but in our XenoLinux forward
port init_cpu_khz() takes care of this also after migration. In fact I
don't really understand why PV is being penalized here at all by
default, and iirc PV migration didn't have any issue with TSC freq
changing during migration prior to the introduction of vTSC.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Boris Ostrovsky]

On 03/15/2017 12:02 PM, Jan Beulich wrote:
>>>> On 15.03.17 at 16:26, <andrew.cooper3@citrix.com> wrote:
>> On 15/03/17 15:23, Olaf Hering wrote:
>>> On Wed, Mar 15, Olaf Hering wrote:
>>>
>>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>>> As a crazy idea, doest this help?
>>>>> tsc->incarnation = 0
>>> This does indeed help. One system shows now the results below, which
>>> means the performance goes down during migration (to localhost) and goes
>>> back to normal after migration.
>>>
>>> What impact has such change to ->incarnation?
>> So what this means is that, after migrate, Xen sees that the advertised
>> TSC value doesn't match the current hardwares TSC, so enables rdtsc
>> interception so the values reported back can be emulated at the old
>> frequency.
>>
>> There is no easy solution to this problem.
> Especially for localhost migration it ought to be possible to deal
> with this. What I'm wondering is why the frequency comparison
> in default mode handling in tsc_set_info() is HVM-only:
>
>         if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() &&
>              (has_hvm_container_domain(d) ?
>               (d->arch.tsc_khz == cpu_khz ||
>                hvm_get_tsc_scaling_ratio(d->arch.tsc_khz)) :
>               incarnation == 0) )
>         {
>
> Boris, you've made it be this way in 82713ec8d2 ("x86: use native
> RDTSC(P) execution when guest and host frequencies are the
> same").

Don't know why.

In fact, I looked at the history of this patch and earlier versions had

@@ -1889,10 +1890,14 @@ void tsc_set_info(struct domain *d,
         d->arch.vtsc_offset = get_s_time() - elapsed_nsec;
         d->arch.tsc_khz = gtsc_khz ? gtsc_khz : cpu_khz;
         set_time_scale(&d->arch.vtsc_to_ns, d->arch.tsc_khz * 1000 );
-        /* use native TSC if initial host has safe TSC, has not migrated
-         * yet and tsc_khz == cpu_khz */
-        if ( host_tsc_is_safe() && incarnation == 0 &&
-                d->arch.tsc_khz == cpu_khz )
+        /*
+         * Use native TSC if initial host has safe TSC and either has not
+         * migrated yet or tsc_khz == cpu_khz (either "naturally" or via
+         * TSC scaling)
+         */
+        if ( host_tsc_is_safe() &&
+             (incarnation == 0 || d->arch.tsc_khz == cpu_khz ||
+              cpu_has_tsc_ratio) )
             d->arch.vtsc = 0;
         else 
             d->arch.ns_to_vtsc = scale_reciprocal(d->arch.vtsc_to_ns);


However, the original code was even stricter.


-boris




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Jan Beulich]

>>> On 15.03.17 at 17:41, <boris.ostrovsky@oracle.com> wrote:
> On 03/15/2017 12:02 PM, Jan Beulich wrote:
>>>>> On 15.03.17 at 16:26, <andrew.cooper3@citrix.com> wrote:
>>> On 15/03/17 15:23, Olaf Hering wrote:
>>>> On Wed, Mar 15, Olaf Hering wrote:
>>>>
>>>>> On Wed, Mar 15, Andrew Cooper wrote:
>>>>>> As a crazy idea, doest this help?
>>>>>> tsc->incarnation = 0
>>>> This does indeed help. One system shows now the results below, which
>>>> means the performance goes down during migration (to localhost) and goes
>>>> back to normal after migration.
>>>>
>>>> What impact has such change to ->incarnation?
>>> So what this means is that, after migrate, Xen sees that the advertised
>>> TSC value doesn't match the current hardwares TSC, so enables rdtsc
>>> interception so the values reported back can be emulated at the old
>>> frequency.
>>>
>>> There is no easy solution to this problem.
>> Especially for localhost migration it ought to be possible to deal
>> with this. What I'm wondering is why the frequency comparison
>> in default mode handling in tsc_set_info() is HVM-only:
>>
>>         if ( tsc_mode == TSC_MODE_DEFAULT && host_tsc_is_safe() &&
>>              (has_hvm_container_domain(d) ?
>>               (d->arch.tsc_khz == cpu_khz ||
>>                hvm_get_tsc_scaling_ratio(d->arch.tsc_khz)) :
>>               incarnation == 0) )
>>         {
>>
>> Boris, you've made it be this way in 82713ec8d2 ("x86: use native
>> RDTSC(P) execution when guest and host frequencies are the
>> same").
> 
> Don't know why.
> 
> In fact, I looked at the history of this patch and earlier versions had
> 
> @@ -1889,10 +1890,14 @@ void tsc_set_info(struct domain *d,
>          d->arch.vtsc_offset = get_s_time() - elapsed_nsec;
>          d->arch.tsc_khz = gtsc_khz ? gtsc_khz : cpu_khz;
>          set_time_scale(&d->arch.vtsc_to_ns, d->arch.tsc_khz * 1000 );
> -        /* use native TSC if initial host has safe TSC, has not migrated
> -         * yet and tsc_khz == cpu_khz */
> -        if ( host_tsc_is_safe() && incarnation == 0 &&
> -                d->arch.tsc_khz == cpu_khz )
> +        /*
> +         * Use native TSC if initial host has safe TSC and either has not
> +         * migrated yet or tsc_khz == cpu_khz (either "naturally" or via
> +         * TSC scaling)
> +         */
> +        if ( host_tsc_is_safe() &&
> +             (incarnation == 0 || d->arch.tsc_khz == cpu_khz ||
> +              cpu_has_tsc_ratio) )
>              d->arch.vtsc = 0;

Which, as pointed out back then, was not PV-correct.

> However, the original code was even stricter.

True.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Olaf Hering]

[-- Attachment #1.1: Type: text/plain, Size: 250 bytes --]

On Wed, Mar 15, Jan Beulich wrote:

> ..., and iirc PV migration didn't have any issue with TSC freq
> changing during migration prior to the introduction of vTSC.

When was this introduced? There is a claim that this slowdown is a
regression.

Olaf

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: PV performance degraded after live migration

[Jan Beulich]

>>> On 16.03.17 at 09:13, <olaf@aepfle.de> wrote:
> On Wed, Mar 15, Jan Beulich wrote:
> 
>> ..., and iirc PV migration didn't have any issue with TSC freq
>> changing during migration prior to the introduction of vTSC.
> 
> When was this introduced? There is a claim that this slowdown is a
> regression.

Many, many years ago (perhaps in the 4.0 time frame).

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel