performance trouble

performance trouble

[David Cure]

		Hello,

	I use several kvm box, and no problem at all except for 1
application that have bad response time.

	The VM runs Windows 2008R2 and the application is an
client-server app develop with progress software and talk to an Oracle
databasei (on another server) and we access this app with RDS/TSE.
	The physical server runs Debian testing to have qemu-kvm 1.0 and
linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network
and use the last driver for Windows (from RH).

	We have 2 references servers : one physical and one running
Vmware.

	Response time :
	o physical = 7s
	o VM under vmware = 8s
	o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and
kernel 2.6.32 we have 22s ...).

	I attach the libvirt xml of my vm.

	How can I see what's append ? Do you have idea to increase
performance ?

	David.
	

Re: performance trouble

[David Cure]

[-- Attachment #1.1: Type: text/plain, Size: 145 bytes --]

Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait :
> 
> 	I attach the libvirt xml of my vm.

	I forget to attach ;)

	David.

[-- Attachment #1.2: rds.xml --]
[-- Type: application/xml, Size: 2598 bytes --]

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: performance trouble

[David Cure]

Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait :
> 
> 	I use several kvm box, and no problem at all except for 1
> application that have bad response time.
> 
> 	The VM runs Windows 2008R2 and the application is an
> client-server app develop with progress software and talk to an Oracle
> databasei (on another server) and we access this app with RDS/TSE.
> 	The physical server runs Debian testing to have qemu-kvm 1.0 and
> linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network
> and use the last driver for Windows (from RH).
> 
> 	We have 2 references servers : one physical and one running
> Vmware.
> 
> 	Response time :
> 	o physical = 7s
> 	o VM under vmware = 8s
> 	o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and
> kernel 2.6.32 we have 22s ...).
> 
> 	I attach the libvirt xml of my vm.
> 
> 	How can I see what's append ? Do you have idea to increase
> performance ?

	no one interesting to try to track this issue where kvm is so
slow ?

	David.

Re: performance trouble

[Brian Jackson]

On Monday, January 30, 2012 09:36:55 AM David Cure wrote:
> Le Mon, Jan 23, 2012 at 09:28:37AM +0100, David Cure ecrivait :
> > 	I use several kvm box, and no problem at all except for 1
> > 
> > application that have bad response time.
> > 
> > 	The VM runs Windows 2008R2 and the application is an
> > 
> > client-server app develop with progress software and talk to an Oracle
> > databasei (on another server) and we access this app with RDS/TSE.
> > 
> > 	The physical server runs Debian testing to have qemu-kvm 1.0 and
> > 
> > linux kernel 3.1 and libvirt 0.9.8. We use virtio for disk and network
> > and use the last driver for Windows (from RH).
> > 
> > 	We have 2 references servers : one physical and one running
> > 
> > Vmware.
> > 
> > 	Response time :
> > 	o physical = 7s
> > 	o VM under vmware = 8s
> > 	o VM under KVM = 12s (to complete with qemu-kvm 0.12.5 and
> > 
> > kernel 2.6.32 we have 22s ...).
> > 
> > 	I attach the libvirt xml of my vm.
> > 	
> > 	How can I see what's append ? Do you have idea to increase
> > 
> > performance ?
> 
> 	no one interesting to try to track this issue where kvm is so
> slow ?
> 
> 	David.


Without more info or some way to reproduce the problem, it would be pointless 
for one of the devs to spend much time on it.

Re: performance trouble

[David Cure]

Le Mon, Jan 30, 2012 at 10:20:34AM -0600, Brian Jackson ecrivait :
> 
> Without more info or some way to reproduce the problem, it would be pointless 
> for one of the devs to spend much time on it.

	yes for sure.

	But I mean there is some way to trace at the KVM level what happens when the
program runs in the VM ? And perhaps with these infos some one see the
problem ;) (because with the upgrade of KVM/qemu we already cut the
response time by 2).

	It's difficult to have more infos because the software use
Progress (an L4G development tool that is not free/open-source) to
connect to Oracle database, and I'm not the developper, only the person
that install it under KVM.

	And in the same VM, we have another software (developed only in
Java) and it works fine. So we suspect some strange behaviour with
progress under KVM.

	To remember, the user connect with RDS/TSE to the VM to runs the
program.

	David.

Re: performance trouble

[Avi Kivity]

On 01/30/2012 06:51 PM, David Cure wrote:
> Le Mon, Jan 30, 2012 at 10:20:34AM -0600, Brian Jackson ecrivait :
> > 
> > Without more info or some way to reproduce the problem, it would be pointless 
> > for one of the devs to spend much time on it.
>
> 	yes for sure.
>
> 	But I mean there is some way to trace at the KVM level what happens when the
> program runs in the VM ? And perhaps with these infos some one see the
> problem ;) (because with the upgrade of KVM/qemu we already cut the
> response time by 2).
>

Start with top and vmstat to see if the cpu or I/O are the bottleneck
(also helpful to run the Windows performance monitor in the guest).  If
it's the cpu, run kvm_stat and report its output.

-- 
error compiling committee.c: too many arguments to function

Re: performance trouble

[David Cure]

Le Mon, Jan 30, 2012 at 07:21:20PM +0200, Avi Kivity ecrivait :
> 
> Start with top and vmstat to see if the cpu or I/O are the bottleneck

	with top at the kvm layer, the VM take 50% of CPU (the physical
box have 16 real cores-not HT- : AMD Opteron 6134).
	one result of vmstat :

procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 5  0      0 9904544  19612 195440    0    0     0    11    1    0  6  5
88  0

	and I run it 1H and the result is always the same (just free
result change time to time).

> (also helpful to run the Windows performance monitor in the guest).  If

	between the start of the function and the response, the are no
performance pic or something like that in Windows.

	What we can see : 
	o we start the function, 
	o we see network communication with the database server during 
~3/4s 
	o after nothing during 10s before display the result

	David.	

Re: performance trouble

[Avi Kivity]

What's up with the cc list?


On 01/31/2012 07:15 PM, David Cure wrote:
> Le Mon, Jan 30, 2012 at 07:21:20PM +0200, Avi Kivity ecrivait :
> > 
> > Start with top and vmstat to see if the cpu or I/O are the bottleneck
>
> 	with top at the kvm layer, the VM take 50% of CPU (the physical
> box have 16 real cores-not HT- : AMD Opteron 6134).
> 	one result of vmstat :
>
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
> id wa
>  5  0      0 9904544  19612 195440    0    0     0    11    1    0  6  5
> 88  0
>
> 	and I run it 1H and the result is always the same (just free
> result change time to time).

Running vmstat with no arguments gives the average for all uptime.  It's
meaningless.  Run 'vmstat 1'.

How many vcpus are there in total (over all guests)?

What does top show?

>
> > (also helpful to run the Windows performance monitor in the guest).  If
>
> 	between the start of the function and the response, the are no
> performance pic or something like that in Windows.
>
> 	What we can see : 
> 	o we start the function, 
> 	o we see network communication with the database server during 
> ~3/4s 
> 	o after nothing during 10s before display the result
>
>

That's really strange.  No network traffic/cpu/block I/O?

What do 'vmstat 1' and 'top -d 1' say during this?

-- 
error compiling committee.c: too many arguments to function

Re: performance trouble

[David Cure]

		Hello,

Le Tue, Jan 31, 2012 at 07:21:25PM +0200, Avi Kivity ecrivait :
> 
> How many vcpus are there in total (over all guests)?

	For test we run only 2 VM (RDS/TSE) with 2vcpus for each

> That's really strange.  No network traffic/cpu/block I/O?
> 
> What do 'vmstat 1' and 'top -d 1' say during this?

	vmstat 1 during the slowly function :

procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 0  0      0 16104568  20524 195588    0    0     0     0 9918 19301  1
1 98  0
 1  0      0 16104348  20524 195588    0    0     0     0 9868 19312  1
1 98  0
 0  0      0 16104224  20524 195588    0    0     0    12 10598 19956  0
1 99  0
 3  0      0 16104224  20524 195588    0    0    16     4 15044 25466  6
3 91  0
 2  0      0 16104352  20524 195588    0    0     0     0 16126 26861  6
4 90  0
 0  0      0 16104476  20524 195588    0    0     0     0 14598 24664  6
3 91  0
 0  0      0 16104600  20524 195588    0    0     0     0 15025 24970  7
3 90  0
 3  0      0 16104740  20524 195588    0    0     0     0 14859 24942  5
2 93  0
 3  0      0 16104748  20524 195588    0    0     0    12 14898 24633  9
3 87  0
 2  0      0 16104872  20524 195588    0    0     0     0 14625 24840  9
5 86  0
 3  0      0 16104872  20524 195588    0    0     0     0 14765 24801  4
2 95  0
 1  0      0 16104748  20524 195588    0    0     0   146 12356 20437  7
2 91  0
 1  0      0 16104616  20524 195588    0    0     0     0 10768 17531 11
1 88  0
 2  0      0 16104724  20524 195588    0    0     0    12 11729 18996 10
2 89  0


	top : during the function, top for the KVM where the function
runs increase from ~50% to 130% and decrease to ~50% at the end of the
function.

	In Windows, we see that one CPU is used.

	The CPU in this KVM server runs at 2.3GHz (in the vmware
server, cpu runs at 2GHz).

	For kvm_stats, I mean it's better to have only one VM with one
user for the test so I send this evening or tomorrow morning.

	David. 

Re: performance trouble

[David Cure]

[-- Attachment #1: Type: text/plain, Size: 1641 bytes --]

		Hello,

Le Thu, Feb 02, 2012 at 11:41:48AM +0100, David Cure ecrivait :
> 
> 	For kvm_stats, I mean it's better to have only one VM with one
> user for the test so I send this evening or tomorrow morning.

	I attach snapshot of the kvm_stat : the first one just before to
run the function and the second one at the end.

	Here are the vmstat 1 take when the function is running :


 4  0      0 32601320  21360 198776    0    0    49  1123 14577 21601  8
4 88  0
procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 4  0      0 32601196  21360 198776    0    0     0   141 16559 21315 12
3 84  0
 2  0      0 32601212  21360 198776    0    0     0   128 17377 24949 10
4 86  0
 2  0      0 32601128  21360 198776    0    0     0   120 13064 18986 13
3 85  0
 4  0      0 32601216  21360 198776    0    0     0   284 16052 21569 11
4 85  0
 3  0      0 32601132  21360 198776    0    0     0   192 15846 23323 10
3 87  0
 3  0      0 32601492  21360 198776    0    0     0   564 19116 27478 10
4 85  0
 4  0      0 32601492  21360 198776    0    0     0     0 16996 24286 14
5 81  0
 2  0      0 32601492  21360 198776    0    0     0   661 15754 23371  9
4 87  0
 3  0      0 32601268  21360 198776    0    0    36     0 9808 13849 11
1 88  0
 2  0      0 32601304  21360 198776    0    0     0    36 11846 13826 14
1 85  0
 3  0      0 32601304  21360 198776    0    0    64    37 9445 12263 18
1 81  0
 4  0      0 32601604  21360 198776    0    0   290    20 9959 14786 10
2 88  0

	I hope you can see something strange ;)

	David.

[-- Attachment #2: kvm_stat1.png --]
[-- Type: image/png, Size: 15279 bytes --]

[-- Attachment #3: kvm_stat2.png --]
[-- Type: image/png, Size: 15775 bytes --]

Re: performance trouble

[Avi Kivity]

On 02/03/2012 10:59 AM, David Cure wrote:
> 		Hello,
>
> Le Thu, Feb 02, 2012 at 11:41:48AM +0100, David Cure ecrivait :
> > 
> > 	For kvm_stats, I mean it's better to have only one VM with one
> > user for the test so I send this evening or tomorrow morning.
>
> 	I attach snapshot of the kvm_stat : the first one just before to
> run the function and the second one at the end.
>
> 	Here are the vmstat 1 take when the function is running :
>

Please post a trace as documented in http://www.linux-kvm.org/page/Tracing.

-- 
error compiling committee.c: too many arguments to function

Re: performance trouble

[David Cure]

		hello,

Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait :
> 
> Please post a trace as documented in http://www.linux-kvm.org/page/Tracing.

	I made the trace : started just before the slow function launch
and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one
user connected to the VM to launch the test.

	The trace file is too big to post here, I gzip it and the file
is available here : http://www.roullier.net/report.txt.gz

	I hope you can find something strange.

	David.

Re: performance trouble

[Avi Kivity]

On 02/10/2012 12:09 PM, David Cure wrote:
> 		hello,
>
> Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait :
> > 
> > Please post a trace as documented in http://www.linux-kvm.org/page/Tracing.
>
> 	I made the trace : started just before the slow function launch
> and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one
> user connected to the VM to launch the test.
>
> 	The trace file is too big to post here, I gzip it and the file
> is available here : http://www.roullier.net/report.txt.gz
>
> 	I hope you can find something strange.
>

It's reading the HPET like crazy.  There are also tons of interrupts. 
Please use the windows performance tools to see which devices trigger
these interrupts.

The HPET issue will be fixed by the hyper-V enlightenments, but these
will take some time to cook.

You can also try vhost-net to improve networking latency.

-- 
error compiling committee.c: too many arguments to function

Re: performance trouble

[Gleb Natapov]

On Tue, Feb 14, 2012 at 03:32:16PM +0200, Avi Kivity wrote:
> On 02/10/2012 12:09 PM, David Cure wrote:
> > 		hello,
> >
> > Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait :
> > > 
> > > Please post a trace as documented in http://www.linux-kvm.org/page/Tracing.
> >
> > 	I made the trace : started just before the slow function launch
> > and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one
> > user connected to the VM to launch the test.
> >
> > 	The trace file is too big to post here, I gzip it and the file
> > is available here : http://www.roullier.net/report.txt.gz
> >
> > 	I hope you can find something strange.
> >
> 
> It's reading the HPET like crazy.  There are also tons of interrupts. 
> Please use the windows performance tools to see which devices trigger
> these interrupts.
> 
> The HPET issue will be fixed by the hyper-V enlightenments, but these
> will take some time to cook.
> 
Try to add -no-hpet to qemu command line and see if it helps.

> You can also try vhost-net to improve networking latency.
> 
> -- 
> error compiling committee.c: too many arguments to function
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

----- Original Message -----
From: "Avi Kivity" <avi@redhat.com>
To: "David Cure" <kvm@cure.nom.fr>
Cc: kvm@vger.kernel.org, "Vadim Rozenfeld" <vrozenfe@redhat.com>
Sent: Tuesday, February 14, 2012 3:32:16 PM
Subject: Re: performance trouble

On 02/10/2012 12:09 PM, David Cure wrote:
> 		hello,
>
> Le Sun, Feb 05, 2012 at 11:38:34AM +0200, Avi Kivity ecrivait :
> > 
> > Please post a trace as documented in http://www.linux-kvm.org/page/Tracing.
>
> 	I made the trace : started just before the slow function launch
> and stoped just after. I start only one VM with 2 vcpus/16G RAM and only one
> user connected to the VM to launch the test.
>
> 	The trace file is too big to post here, I gzip it and the file
> is available here : http://www.roullier.net/report.txt.gz
>
> 	I hope you can find something strange.
>

It's reading the HPET like crazy.  There are also tons of interrupts. 
Please use the windows performance tools to see which devices trigger
these interrupts.

[VR]
+1 
Try Microsoft Windows Performance Toolkit from Windows SDK 
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3138
It's really good.


The HPET issue will be fixed by the hyper-V enlightenments, but these
will take some time to cook.

You can also try vhost-net to improve networking latency.

-- 
error compiling committee.c: too many arguments to function

Re: performance trouble

[David Cure]

		hello,

Le Tue, Feb 14, 2012 at 03:40:30PM +0200, Gleb Natapov ecrivait :
> 
> Try to add -no-hpet to qemu command line and see if it helps.

	I add this line in my xml definition for libvirt :

	<timer name='hpet' present='no'/> in the clock block. And I see
the -no-hpet in the command line :

/usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 16384 -smp
2,sockets=2,cores=1,threads=1 -name rds1 -uuid
4f72a64e-de1a-b3da-adac-5dffdb9eb6aa -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/rds1.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
-no-hpet -no-shutdown -drive
file=/dev/drbd1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-drive
file=/Iso/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008R2_64-bit_French_X15-59758.ISO,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
-netdev tap,fd=18,id=hostnet0 -device
virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:36:0c:a1:c4,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -usb -device
usb-tablet,id=input0 -vnc 127.0.0.1:0 -k fr -vga std -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

	But unfortunatly, the result is the same : always 12s to runs
this function (to remember 8s on ESXi 4).

	David.

Re: performance trouble

[Gleb Natapov]

On Thu, Feb 16, 2012 at 09:55:53AM +0100, David Cure wrote:
> 		hello,
> 
> Le Tue, Feb 14, 2012 at 03:40:30PM +0200, Gleb Natapov ecrivait :
> > 
> > Try to add -no-hpet to qemu command line and see if it helps.
> 
> 	I add this line in my xml definition for libvirt :
> 
> 	<timer name='hpet' present='no'/> in the clock block. And I see
> the -no-hpet in the command line :
> 
> /usr/bin/kvm -S -M pc-1.0 -enable-kvm -m 16384 -smp
> 2,sockets=2,cores=1,threads=1 -name rds1 -uuid
> 4f72a64e-de1a-b3da-adac-5dffdb9eb6aa -nodefconfig -nodefaults -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/rds1.monitor,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime
> -no-hpet -no-shutdown -drive
> file=/dev/drbd1,if=none,id=drive-virtio-disk0,format=raw,cache=none,aio=native
> -device
> virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
> -drive
> file=/Iso/SW_DVD5_Windows_Svr_DC_EE_SE_Web_2008R2_64-bit_French_X15-59758.ISO,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0
> -netdev tap,fd=18,id=hostnet0 -device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=00:16:36:0c:a1:c4,bus=pci.0,addr=0x3
> -chardev pty,id=charserial0 -device
> isa-serial,chardev=charserial0,id=serial0 -usb -device
> usb-tablet,id=input0 -vnc 127.0.0.1:0 -k fr -vga std -device
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> 
> 	But unfortunatly, the result is the same : always 12s to runs
> this function (to remember 8s on ESXi 4).
> 
Can you do the trace again with -no-hpet on the command line?

--
			Gleb.

Re: performance trouble

[David Cure]

Le Tue, Feb 14, 2012 at 03:32:16PM +0200, Avi Kivity ecrivait :
> 
> It's reading the HPET like crazy.  There are also tons of interrupts. 
> Please use the windows performance tools to see which devices trigger
> these interrupts.
> 
> The HPET issue will be fixed by the hyper-V enlightenments, but these
> will take some time to cook.

	as you can see, I try -no-hpet and doesn't solve the issue, so
perhaps not related to it.


Le Tue, Feb 14, 2012 at 08:48:56AM -0500, Vadim Rozenfeld ecrivait :
> 
> [VR]
> +1 
> Try Microsoft Windows Performance Toolkit from Windows SDK 
> http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3138
> It's really good.

	I download and install it. Let you know what we can see.

	Thanks,

	David.

Re: performance trouble

[David Cure]

Le Thu, Feb 16, 2012 at 11:01:26AM +0200, Gleb Natapov ecrivait :
>
> Can you do the trace again with -no-hpet on the command line?

	Is there a way to only trace events on specific VM ?

	David.

Re: performance trouble

[Gleb Natapov]

On Fri, Feb 17, 2012 at 09:59:39AM +0100, David Cure wrote:
> Le Thu, Feb 16, 2012 at 11:01:26AM +0200, Gleb Natapov ecrivait :
> >
> > Can you do the trace again with -no-hpet on the command line?
> 
> 	Is there a way to only trace events on specific VM ?
> 
-P pid

--
			Gleb.

Re: performance trouble

[David Cure]

Le Fri, Feb 17, 2012 at 12:07:14PM +0200, Gleb Natapov ecrivait :
> 
> > > Can you do the trace again with -no-hpet on the command line?
> > 
> > 	Is there a way to only trace events on specific VM ?
> > 
> -P pid

	ok. I take a trace of the VM with 1 user : start trace just
before to launch slowly function and stop trace just after the function
finish.
	the report is available here
http://www.roullier.net/report-no-hpet.txt.gz

	I hope you can see something strange,

	David.

Re: performance trouble

[David Cure]

Le Tue, Feb 14, 2012 at 08:48:56AM -0500, Vadim Rozenfeld ecrivait :
> 
> +1 
> Try Microsoft Windows Performance Toolkit from Windows SDK 
> http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=3138
> It's really good.

	I run xperf -on DiagEasy, and as I'm not an windows-er the trace
file is available here http://www.roullier.net/srv-rds1.zip

	I hope someone can see something strange ;)

	David.

Re: performance trouble

[Gleb Natapov]

On Fri, Feb 17, 2012 at 03:01:07PM +0100, David Cure wrote:
> Le Fri, Feb 17, 2012 at 12:07:14PM +0200, Gleb Natapov ecrivait :
> > 
> > > > Can you do the trace again with -no-hpet on the command line?
> > > 
> > > 	Is there a way to only trace events on specific VM ?
> > > 
> > -P pid
> 
> 	ok. I take a trace of the VM with 1 user : start trace just
> before to launch slowly function and stop trace just after the function
> finish.
> 	the report is available here
> http://www.roullier.net/report-no-hpet.txt.gz
> 
How have you acquired this trace? It does not trace all kvm events only
those that have "set_irq" in them.

> 	I hope you can see something strange,
> 

Nothing particularly strange here. ~1745 IRQs are injected into the
guest each second.  RTC clock is configured to 1kH and other devices
contribute ~721 irqs (MSI mostly) per second more. Hardly unusual.

--
			Gleb.

Re: performance trouble

[David Cure]

Le Sun, Feb 19, 2012 at 11:13:15AM +0200, Gleb Natapov ecrivait :
> 
> > http://www.roullier.net/report-no-hpet.txt.gz
> > 
> How have you acquired this trace? It does not trace all kvm events only
> those that have "set_irq" in them.

	I run : trace-cmd record -b 20000 -e kvm -P <process_pid>

> Nothing particularly strange here. ~1745 IRQs are injected into the
> guest each second.  RTC clock is configured to 1kH and other devices
> contribute ~721 irqs (MSI mostly) per second more. Hardly unusual.

	ok, so no ideas to decrease the response time ?

	David.

Re: performance trouble

[David Cure]

	To complete and because I see another thread on performance
trouble with w2008R2

Le Wed, Feb 22, 2012 at 05:33:56PM +0100, David Cure ecrivait :
> 
> > Nothing particularly strange here. ~1745 IRQs are injected into the
> > guest each second.  RTC clock is configured to 1kH and other devices
> > contribute ~721 irqs (MSI mostly) per second more. Hardly unusual.
> 
> 	ok, so no ideas to decrease the response time ?

	Before to see the response time problem, we see long network
response and download : our app was on one samba server and we see
slowly download when the user start the application. To solve, we copy
the app directly on the RDS server (with synchro mechanism from samba
for new version). And after we see the response time problem.

	David.

Re: performance trouble

[Gleb Natapov]

On Wed, Feb 22, 2012 at 05:33:56PM +0100, David Cure wrote:
> Le Sun, Feb 19, 2012 at 11:13:15AM +0200, Gleb Natapov ecrivait :
> > 
> > > http://www.roullier.net/report-no-hpet.txt.gz
> > > 
> > How have you acquired this trace? It does not trace all kvm events only
> > those that have "set_irq" in them.
> 
> 	I run : trace-cmd record -b 20000 -e kvm -P <process_pid>
> 
Ah, I guess the reason is that it records events only of IO thread. You
need to trace all vcpu threads too. Not sure trace-cmd allows more then
one -P option though.

> > Nothing particularly strange here. ~1745 IRQs are injected into the
> > guest each second.  RTC clock is configured to 1kH and other devices
> > contribute ~721 irqs (MSI mostly) per second more. Hardly unusual.
> 
> 	ok, so no ideas to decrease the response time ?
> 
Without knowing the reason of the slowdown no. Probably you have the
same problem as this other performance thread (frequent access to pm
timer), but we need better trace to confirm that.

--
			Gleb.

Re: performance trouble

[David Cure]

		hello,

	sorry for the delay,

Le Thu, Feb 23, 2012 at 10:38:07AM +0200, Gleb Natapov ecrivait :
> 
> Ah, I guess the reason is that it records events only of IO thread. You
> need to trace all vcpu threads too. Not sure trace-cmd allows more then
> one -P option though.

	I manage to have the physical server with only one VM with the
slowly function and take trace during the slowly function.
	I upload trace in http://www.roullier.net/Report/ with :
	o report.txt.3.1.gz : with kernel 3.1
	o report.txt.3.2.gz : with kernel 3.2
	o report.txt.vhost-net-3.1.gz : with kernel 3.1 and vhost-net
	o report.txt.vhost-net.3.2.gz : with kernel 3.2 and vhost-net

	With 3.2 + vhost-net we have 10.5s (to remember 8s with
vmware esxi 4).

	David.

Re: performance trouble

[Gleb Natapov]

On Fri, Mar 16, 2012 at 11:13:31AM +0100, David Cure wrote:
> 		hello,
> 
> 	sorry for the delay,
> 
> Le Thu, Feb 23, 2012 at 10:38:07AM +0200, Gleb Natapov ecrivait :
> > 
> > Ah, I guess the reason is that it records events only of IO thread. You
> > need to trace all vcpu threads too. Not sure trace-cmd allows more then
> > one -P option though.
> 
> 	I manage to have the physical server with only one VM with the
> slowly function and take trace during the slowly function.
> 	I upload trace in http://www.roullier.net/Report/ with :
> 	o report.txt.3.1.gz : with kernel 3.1
> 	o report.txt.3.2.gz : with kernel 3.2
> 	o report.txt.vhost-net-3.1.gz : with kernel 3.1 and vhost-net
> 	o report.txt.vhost-net.3.2.gz : with kernel 3.2 and vhost-net
> 
> 	With 3.2 + vhost-net we have 10.5s (to remember 8s with
> vmware esxi 4).
> 
Can you run the same test on Linux guest and on Windows vm with 1 cpu?
I see a lot of IPIs between vcpus.

--
			Gleb.

Re: performance trouble

[David Cure]

		hello,

Le Mon, Mar 19, 2012 at 12:51:34PM +0200, Gleb Natapov ecrivait :
>
> Can you run the same test on Linux guest and on Windows vm with 1 cpu?
> I see a lot of IPIs between vcpus.

	I run the same slowly fonction (to remember the guest is
Windows2008R2 64bits). The report is here :
	http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu.txt.gz

	and the response time is ~13s.
  
	thanks for tracking this trouble,

	David.

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 20, 2012 at 10:32:20AM +0100, David Cure wrote:
> 		hello,
> 
> Le Mon, Mar 19, 2012 at 12:51:34PM +0200, Gleb Natapov ecrivait :
> >
> > Can you run the same test on Linux guest and on Windows vm with 1 cpu?
> > I see a lot of IPIs between vcpus.
> 
> 	I run the same slowly fonction (to remember the guest is
> Windows2008R2 64bits). The report is here :
> 	http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu.txt.gz
> 
> 	and the response time is ~13s.
>   

This is without -no-hpet :) Also I hope you are not measuring time while
tracing since it supposed to be slower.  Can you please try to run with
-cpu host,-hypervisor and -no-hpet and see if times are better.

--
			Gleb.

Re: performance trouble

[David Cure]

Le Tue, Mar 20, 2012 at 11:45:41AM +0200, Gleb Natapov ecrivait :
> 
> This is without -no-hpet :) Also I hope you are not measuring time while

	haaaaa yes, sorry for the mistake.

> tracing since it supposed to be slower.  Can you please try to run with
> -cpu host,-hypervisor and -no-hpet and see if times are better.

	I use libvirt to start the VM, do you know how to modify the xml
file to have this option generated (I know for the -no-hpet, but not for
other) ?
	If not, I capture the command and modify it to run directly.

	David.

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 20, 2012 at 12:18:39PM +0100, David Cure wrote:
> Le Tue, Mar 20, 2012 at 11:45:41AM +0200, Gleb Natapov ecrivait :
> > 
> > This is without -no-hpet :) Also I hope you are not measuring time while
> 
> 	haaaaa yes, sorry for the mistake.
> 
> > tracing since it supposed to be slower.  Can you please try to run with
> > -cpu host,-hypervisor and -no-hpet and see if times are better.
> 
> 	I use libvirt to start the VM, do you know how to modify the xml
> file to have this option generated (I know for the -no-hpet, but not for
> other) ?
> 	If not, I capture the command and modify it to run directly.
> 
I was pointed to here:
http://berrange.com/posts/2010/02/15/guest-cpu-model-configuration-in-libvirt-with-qemukvm/

Try to add <feature policy='disable' name='hypervisor'/> to cpu
definition in XML and check command line.

--
			Gleb.

Re: performance trouble

[David Cure]

		hello,

Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> 
> Try to add <feature policy='disable' name='hypervisor'/> to cpu
> definition in XML and check command line.

	ok I try this but I can't use <cpu model> to map the host cpu
(my libvirt is 0.9.8) so I use :

  <cpu match='exact'>
    <model>Opteron_G3</model>
    <feature policy='disable' name='hypervisor'/>
  </cpu>

	(the physical server use Opteron CPU).

	The log is here :
http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz

	And now with only 1 vcpu, the response time is 8.5s, great
improvment. We keep this configuration for production : we check the
response time when some other users are connected.

	David.

Re: performance trouble

[Peter Lieven]

On 21.03.2012 12:10, David Cure wrote:
> 		hello,
>
> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
>> Try to add<feature policy='disable' name='hypervisor'/>  to cpu
>> definition in XML and check command line.
> 	ok I try this but I can't use<cpu model>  to map the host cpu
> (my libvirt is 0.9.8) so I use :
>
>    <cpu match='exact'>
>      <model>Opteron_G3</model>
>      <feature policy='disable' name='hypervisor'/>
>    </cpu>
>
> 	(the physical server use Opteron CPU).
>
> 	The log is here :
> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
>
> 	And now with only 1 vcpu, the response time is 8.5s, great
> improvment. We keep this configuration for production : we check the
> response time when some other users are connected.
please keep in mind, that setting -hypervisor, disabling hpet and only 
one vcpu
makes windows use tsc as clocksource. you have to make sure, that your vm
is not switching between physical sockets on your system and that you have
constant_tsc feature to have a stable tsc between the cores in the
same socket. its also likely that the vm will crash when live migrated.

@gleb: do you know whats the state of in-kernel hyper-v timers?

peter

> 	David.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: performance trouble

[Gleb Natapov]

On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> On 21.03.2012 12:10, David Cure wrote:
> >		hello,
> >
> >Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> >>Try to add<feature policy='disable' name='hypervisor'/>  to cpu
> >>definition in XML and check command line.
> >	ok I try this but I can't use<cpu model>  to map the host cpu
> >(my libvirt is 0.9.8) so I use :
> >
> >   <cpu match='exact'>
> >     <model>Opteron_G3</model>
> >     <feature policy='disable' name='hypervisor'/>
> >   </cpu>
> >
> >	(the physical server use Opteron CPU).
> >
> >	The log is here :
> >http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
> >
> >	And now with only 1 vcpu, the response time is 8.5s, great
> >improvment. We keep this configuration for production : we check the
> >response time when some other users are connected.
> please keep in mind, that setting -hypervisor, disabling hpet and
> only one vcpu
> makes windows use tsc as clocksource. you have to make sure, that your vm
> is not switching between physical sockets on your system and that you have
> constant_tsc feature to have a stable tsc between the cores in the
> same socket. its also likely that the vm will crash when live migrated.
> 
All true. I asked to try -hypervisor only to verify where we loose
performance. Since you get good result with it frequent access to PM
timer is probably the reason. I do not recommend using -hypervisor for
production!

> @gleb: do you know whats the state of in-kernel hyper-v timers?
> 
Vadim is working on it. I'll let him answer.

> peter
> 
> >	David.
> >--
> >To unsubscribe from this list: send the line "unsubscribe kvm" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
			Gleb.

Re: performance trouble

[Peter Lieven]

On 22.03.2012 08:53, Gleb Natapov wrote:
> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>> On 21.03.2012 12:10, David Cure wrote:
>>> 		hello,
>>>
>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
>>>> Try to add<feature policy='disable' name='hypervisor'/>   to cpu
>>>> definition in XML and check command line.
>>> 	ok I try this but I can't use<cpu model>   to map the host cpu
>>> (my libvirt is 0.9.8) so I use :
>>>
>>>    <cpu match='exact'>
>>>      <model>Opteron_G3</model>
>>>      <feature policy='disable' name='hypervisor'/>
>>>    </cpu>
>>>
>>> 	(the physical server use Opteron CPU).
>>>
>>> 	The log is here :
>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
>>>
>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>> improvment. We keep this configuration for production : we check the
>>> response time when some other users are connected.
>> please keep in mind, that setting -hypervisor, disabling hpet and
>> only one vcpu
>> makes windows use tsc as clocksource. you have to make sure, that your vm
>> is not switching between physical sockets on your system and that you have
>> constant_tsc feature to have a stable tsc between the cores in the
>> same socket. its also likely that the vm will crash when live migrated.
>>
> All true. I asked to try -hypervisor only to verify where we loose
> performance. Since you get good result with it frequent access to PM
> timer is probably the reason. I do not recommend using -hypervisor for
> production!
>
>> @gleb: do you know whats the state of in-kernel hyper-v timers?
>>
> Vadim is working on it. I'll let him answer.
@avi, gleb: another option would be to revisit the old in-kernel 
pm-timer implementation
and check if its feasible to use this as an alternative. it would also 
help non hyper-v aware
systems (i think bsd and old windows like xp). i rebased this old 
implementation and can
confirm that it does also solve the performance slow down.

peter


>> peter
>>
>>> 	David.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> 			Gleb.

Re: performance trouble

[David Cure]

Le Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven ecrivait :
>
> please keep in mind, that setting -hypervisor, disabling hpet and only  
> one vcpu
> makes windows use tsc as clocksource. you have to make sure, that your vm
> is not switching between physical sockets on your system and that you have
> constant_tsc feature to have a stable tsc between the cores in the
> same socket. its also likely that the vm will crash when live migrated.

	ok, yet I "only" have disable hpet and use 1vcpu.
	for the switching I need to pin the vcpu on 1 physical proc,
right ?
	for constant_tsc, how can I check if I use it ?
	for live migration : what is the "feature" that cause trouble :
-hypervisor, hpet, vcpu or all ?

	David.

Re: performance trouble

[David Cure]

Le Thu, Mar 22, 2012 at 09:53:45AM +0200, Gleb Natapov ecrivait :
> 
> All true. I asked to try -hypervisor only to verify where we loose
> performance. Since you get good result with it frequent access to PM
> timer is probably the reason. I do not recommend using -hypervisor for
> production!

	so if I leave cpu as previous (not defined) and only disable
hpet and use 1 vcpu, it's ok for production ?

	Is there a workaround for this PM access ?

	David.

Re: performance trouble

[David Cure]

		hello,

Le Thu, Mar 22, 2012 at 08:57:08AM +0100, Peter Lieven ecrivait :
>
> @avi, gleb: another option would be to revisit the old in-kernel  
> pm-timer implementation
> and check if its feasible to use this as an alternative. it would also  
> help non hyper-v aware
> systems (i think bsd and old windows like xp). i rebased this old  
> implementation and can
> confirm that it does also solve the performance slow down.

	Are some patches available for testing ?

	David.

Re: performance trouble

[Peter Lieven]

On 22.03.2012 09:31, David Cure wrote:
> Le Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven ecrivait :
>> please keep in mind, that setting -hypervisor, disabling hpet and only
>> one vcpu
>> makes windows use tsc as clocksource. you have to make sure, that your vm
>> is not switching between physical sockets on your system and that you have
>> constant_tsc feature to have a stable tsc between the cores in the
>> same socket. its also likely that the vm will crash when live migrated.
> 	ok, yet I "only" have disable hpet and use 1vcpu.
you have to use 1 vcpu on 32-bit windows. 64-bit seems to
work with more than 1 vcpu. why all those limitations:
windows avoids using tsc in a hypervisor which is a good decision.
problem is it falls back to pm_timer or hpet. both of them are very
expensive in emulation currently because kvm exits kernel mode
and the userspace qemu-kvm handles this. i have done experiments
where i saw ~20.000 userpace exits just for pmtimer reads. this
made up ~30-40% of the whole processing power. every call to a
QPC timer in windows causes a pm_timer/hpet read. especially
each i/o request seems to cause a QPC timer read and which
is odd as well a lazy fpu call (but this is a differnt issue) which also
is very expensive to emulate (currently).

> 	for the switching I need to pin the vcpu on 1 physical proc,
> right ?
you need 1 vcpu for 32-bit windows and disabling hypervisor to
cheat windows and make it think it runs on real hardware. it then
uses tsc.
> 	for constant_tsc, how can I check if I use it ?
cat /proc/cpuinfo on the host. there should be a flag 'constant_tsc'.
it might be that also rdtscp is necessary.
> 	for live migration : what is the "feature" that cause trouble :
> -hypervisor, hpet, vcpu or all ?
using tsc as clocksource is the problem not the features themselves.

peter
> 	David.
>

Re: performance trouble

[Vadim Rozenfeld]

On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > On 21.03.2012 12:10, David Cure wrote:
> > >		hello,
> > >
> > >Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> > >>Try to add<feature policy='disable' name='hypervisor'/>  to cpu
> > >>definition in XML and check command line.
> > >>
> > >	ok I try this but I can't use<cpu model>  to map the host cpu
> > >
> > >(my libvirt is 0.9.8) so I use :
> > >   <cpu match='exact'>
> > >   
> > >     <model>Opteron_G3</model>
> > >     <feature policy='disable' name='hypervisor'/>
> > >   
> > >   </cpu>
> > >	
> > >	(the physical server use Opteron CPU).
> > >
> > >	The log is here :
> > >http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
> > >
> > >	And now with only 1 vcpu, the response time is 8.5s, great
> > >
> > >improvment. We keep this configuration for production : we check the
> > >response time when some other users are connected.
> > 
> > please keep in mind, that setting -hypervisor, disabling hpet and
> > only one vcpu
> > makes windows use tsc as clocksource. you have to make sure, that your vm
> > is not switching between physical sockets on your system and that you
> > have constant_tsc feature to have a stable tsc between the cores in the
> > same socket. its also likely that the vm will crash when live migrated.
> 
> All true. I asked to try -hypervisor only to verify where we loose
> performance. Since you get good result with it frequent access to PM
> timer is probably the reason. I do not recommend using -hypervisor for
> production!
> 
> > @gleb: do you know whats the state of in-kernel hyper-v timers?
> 
> Vadim is working on it. I'll let him answer.
It would be nice to have synthetic timers supported. But,  at the moment,
I'm only researching  this feature.
> 
> > peter
> > 
> > >	David.
> > >
> > >--
> > >To unsubscribe from this list: send the line "unsubscribe kvm" in
> > >the body of a message to majordomo@vger.kernel.org
> > >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 22.03.2012 09:33, David Cure wrote:
> Le Thu, Mar 22, 2012 at 09:53:45AM +0200, Gleb Natapov ecrivait :
>> All true. I asked to try -hypervisor only to verify where we loose
>> performance. Since you get good result with it frequent access to PM
>> timer is probably the reason. I do not recommend using -hypervisor for
>> production!
> 	so if I leave cpu as previous (not defined) and only disable
> hpet and use 1 vcpu, it's ok for production ?
this is ok, but windows will use pm timer so you will have bad performance.
> 	Is there a workaround for this PM access ?
there exists old patches from 2010 for in-kernel pmtimer. they work, but 
only
partly. problem here is windows enables the pmtimer overflow interrupt 
which this
patch did not address (amongst other things). i simply ignored it and 
windows ran
nevertheless. but i would not do this in production because i do not 
know which side
effects it might have.

there are to possible solutions:

a) a real in-kernel pmtimer implementation (which does also help other 
systems not only windows)
b) hyper-v support in-kernel at least partly (for the timing stuff). 
this is being worked on by Vadim.

Peter
> 	David.
>

Re: performance trouble

[Peter Lieven]

On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>> On 21.03.2012 12:10, David Cure wrote:
>>>> 		hello,
>>>>
>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
>>>>> Try to add<feature policy='disable' name='hypervisor'/>   to cpu
>>>>> definition in XML and check command line.
>>>>>
>>>> 	ok I try this but I can't use<cpu model>   to map the host cpu
>>>>
>>>> (my libvirt is 0.9.8) so I use :
>>>>    <cpu match='exact'>
>>>>
>>>>      <model>Opteron_G3</model>
>>>>      <feature policy='disable' name='hypervisor'/>
>>>>
>>>>    </cpu>
>>>> 	
>>>> 	(the physical server use Opteron CPU).
>>>>
>>>> 	The log is here :
>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
>>>>
>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>
>>>> improvment. We keep this configuration for production : we check the
>>>> response time when some other users are connected.
>>> please keep in mind, that setting -hypervisor, disabling hpet and
>>> only one vcpu
>>> makes windows use tsc as clocksource. you have to make sure, that your vm
>>> is not switching between physical sockets on your system and that you
>>> have constant_tsc feature to have a stable tsc between the cores in the
>>> same socket. its also likely that the vm will crash when live migrated.
>> All true. I asked to try -hypervisor only to verify where we loose
>> performance. Since you get good result with it frequent access to PM
>> timer is probably the reason. I do not recommend using -hypervisor for
>> production!
>>
>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
>> Vadim is working on it. I'll let him answer.
> It would be nice to have synthetic timers supported. But,  at the moment,
> I'm only researching  this feature.

So it will take months at least?

What do the others think, would it be feasible to make a proper in-kernel
pmtimer solution in the meantime.

I think Windows guest performance is very important for the success of KVM.

Peter

>>> peter
>>>
>>>> 	David.
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>> On 21.03.2012 12:10, David Cure wrote:
> >>>> 		hello,
> >>>> 
> >>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> >>>>> Try to add<feature policy='disable' name='hypervisor'/>   to cpu
> >>>>> definition in XML and check command line.
> >>>>> 
> >>>> 	ok I try this but I can't use<cpu model>   to map the host cpu
> >>>> 
> >>>> (my libvirt is 0.9.8) so I use :
> >>>>    <cpu match='exact'>
> >>>>    
> >>>>      <model>Opteron_G3</model>
> >>>>      <feature policy='disable' name='hypervisor'/>
> >>>>    
> >>>>    </cpu>
> >>>> 	
> >>>> 	(the physical server use Opteron CPU).
> >>>> 
> >>>> 	The log is here :
> >>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
> >>>> 
> >>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> >>>> 
> >>>> improvment. We keep this configuration for production : we check the
> >>>> response time when some other users are connected.
> >>> 
> >>> please keep in mind, that setting -hypervisor, disabling hpet and
> >>> only one vcpu
> >>> makes windows use tsc as clocksource. you have to make sure, that your
> >>> vm is not switching between physical sockets on your system and that
> >>> you have constant_tsc feature to have a stable tsc between the cores
> >>> in the same socket. its also likely that the vm will crash when live
> >>> migrated.
> >> 
> >> All true. I asked to try -hypervisor only to verify where we loose
> >> performance. Since you get good result with it frequent access to PM
> >> timer is probably the reason. I do not recommend using -hypervisor for
> >> production!
> >> 
> >>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> >> 
> >> Vadim is working on it. I'll let him answer.
> > 
> > It would be nice to have synthetic timers supported. But,  at the moment,
> > I'm only researching  this feature.
> 
> So it will take months at least?

I would say weeks.

> 
> What do the others think, would it be feasible to make a proper in-kernel
> pmtimer solution in the meantime.
> 
> I think Windows guest performance is very important for the success of KVM.
> 
> Peter
> 
> >>> peter
> >>> 
> >>>> 	David.
> >>>> 
> >>>> --
> >>>> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >>>> the body of a message to majordomo@vger.kernel.org
> >>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> 
> >> --
> >> 
> >> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>> 		hello,
>>>>>>
>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    to cpu
>>>>>>> definition in XML and check command line.
>>>>>>>
>>>>>> 	ok I try this but I can't use<cpu model>    to map the host cpu
>>>>>>
>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>     <cpu match='exact'>
>>>>>>
>>>>>>       <model>Opteron_G3</model>
>>>>>>       <feature policy='disable' name='hypervisor'/>
>>>>>>
>>>>>>     </cpu>
>>>>>> 	
>>>>>> 	(the physical server use Opteron CPU).
>>>>>>
>>>>>> 	The log is here :
>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
>>>>>>
>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>
>>>>>> improvment. We keep this configuration for production : we check the
>>>>>> response time when some other users are connected.
>>>>> please keep in mind, that setting -hypervisor, disabling hpet and
>>>>> only one vcpu
>>>>> makes windows use tsc as clocksource. you have to make sure, that your
>>>>> vm is not switching between physical sockets on your system and that
>>>>> you have constant_tsc feature to have a stable tsc between the cores
>>>>> in the same socket. its also likely that the vm will crash when live
>>>>> migrated.
>>>> All true. I asked to try -hypervisor only to verify where we loose
>>>> performance. Since you get good result with it frequent access to PM
>>>> timer is probably the reason. I do not recommend using -hypervisor for
>>>> production!
>>>>
>>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
>>>> Vadim is working on it. I'll let him answer.
>>> It would be nice to have synthetic timers supported. But,  at the moment,
>>> I'm only researching  this feature.
>> So it will take months at least?
> I would say weeks.
Is there a way, we could contribute and help you with this?

Peter

Re: performance trouble

[Vadim Rozenfeld]

On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>> 		hello,
> >>>>>> 
> >>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> >>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    to cpu
> >>>>>>> definition in XML and check command line.
> >>>>>>> 
> >>>>>> 	ok I try this but I can't use<cpu model>    to map the host cpu
> >>>>>> 
> >>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>     <cpu match='exact'>
> >>>>>>     
> >>>>>>       <model>Opteron_G3</model>
> >>>>>>       <feature policy='disable' name='hypervisor'/>
> >>>>>>     
> >>>>>>     </cpu>
> >>>>>> 	
> >>>>>> 	(the physical server use Opteron CPU).
> >>>>>> 
> >>>>>> 	The log is here :
> >>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
> >>>>>> 
> >>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>> 
> >>>>>> improvment. We keep this configuration for production : we check the
> >>>>>> response time when some other users are connected.
> >>>>> 
> >>>>> please keep in mind, that setting -hypervisor, disabling hpet and
> >>>>> only one vcpu
> >>>>> makes windows use tsc as clocksource. you have to make sure, that
> >>>>> your vm is not switching between physical sockets on your system and
> >>>>> that you have constant_tsc feature to have a stable tsc between the
> >>>>> cores in the same socket. its also likely that the vm will crash
> >>>>> when live migrated.
> >>>> 
> >>>> All true. I asked to try -hypervisor only to verify where we loose
> >>>> performance. Since you get good result with it frequent access to PM
> >>>> timer is probably the reason. I do not recommend using -hypervisor for
> >>>> production!
> >>>> 
> >>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> >>>> 
> >>>> Vadim is working on it. I'll let him answer.
> >>> 
> >>> It would be nice to have synthetic timers supported. But,  at the
> >>> moment, I'm only researching  this feature.
> >> 
> >> So it will take months at least?
> > 
> > I would say weeks.
> 
> Is there a way, we could contribute and help you with this?
Hi Peter,
You are welcome to add  an appropriate handler.
Best regards,
Vadim.
> 
> Peter

Re: performance trouble

[Gleb Natapov]

On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> > On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > > On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> > >> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > >>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> > >>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > >>>>> On 21.03.2012 12:10, David Cure wrote:
> > >>>>>> 		hello,
> > >>>>>> 
> > >>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> > >>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    to cpu
> > >>>>>>> definition in XML and check command line.
> > >>>>>>> 
> > >>>>>> 	ok I try this but I can't use<cpu model>    to map the host cpu
> > >>>>>> 
> > >>>>>> (my libvirt is 0.9.8) so I use :
> > >>>>>>     <cpu match='exact'>
> > >>>>>>     
> > >>>>>>       <model>Opteron_G3</model>
> > >>>>>>       <feature policy='disable' name='hypervisor'/>
> > >>>>>>     
> > >>>>>>     </cpu>
> > >>>>>> 	
> > >>>>>> 	(the physical server use Opteron CPU).
> > >>>>>> 
> > >>>>>> 	The log is here :
> > >>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.txt.gz
> > >>>>>> 
> > >>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> > >>>>>> 
> > >>>>>> improvment. We keep this configuration for production : we check the
> > >>>>>> response time when some other users are connected.
> > >>>>> 
> > >>>>> please keep in mind, that setting -hypervisor, disabling hpet and
> > >>>>> only one vcpu
> > >>>>> makes windows use tsc as clocksource. you have to make sure, that
> > >>>>> your vm is not switching between physical sockets on your system and
> > >>>>> that you have constant_tsc feature to have a stable tsc between the
> > >>>>> cores in the same socket. its also likely that the vm will crash
> > >>>>> when live migrated.
> > >>>> 
> > >>>> All true. I asked to try -hypervisor only to verify where we loose
> > >>>> performance. Since you get good result with it frequent access to PM
> > >>>> timer is probably the reason. I do not recommend using -hypervisor for
> > >>>> production!
> > >>>> 
> > >>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> > >>>> 
> > >>>> Vadim is working on it. I'll let him answer.
> > >>> 
> > >>> It would be nice to have synthetic timers supported. But,  at the
> > >>> moment, I'm only researching  this feature.
> > >> 
> > >> So it will take months at least?
> > > 
> > > I would say weeks.
> > 
> > Is there a way, we could contribute and help you with this?
> Hi Peter,
> You are welcome to add  an appropriate handler.
I think Vadim refers to this HV MSR
http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85%29.aspx

--
			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

[-- Attachment #1: Type: Text/Plain, Size: 3238 bytes --]

On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> > On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> > > On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > > > On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> > > >> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > > >>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> > > >>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > > >>>>> On 21.03.2012 12:10, David Cure wrote:
> > > >>>>>> 		hello,
> > > >>>>>> 
> > > >>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> > > >>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    to
> > > >>>>>>> cpu definition in XML and check command line.
> > > >>>>>>> 
> > > >>>>>> 	ok I try this but I can't use<cpu model>    to map the host cpu
> > > >>>>>> 
> > > >>>>>> (my libvirt is 0.9.8) so I use :
> > > >>>>>>     <cpu match='exact'>
> > > >>>>>>     
> > > >>>>>>       <model>Opteron_G3</model>
> > > >>>>>>       <feature policy='disable' name='hypervisor'/>
> > > >>>>>>     
> > > >>>>>>     </cpu>
> > > >>>>>> 	
> > > >>>>>> 	(the physical server use Opteron CPU).
> > > >>>>>> 
> > > >>>>>> 	The log is here :
> > > >>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx
> > > >>>>>> t.gz
> > > >>>>>> 
> > > >>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> > > >>>>>> 
> > > >>>>>> improvment. We keep this configuration for production : we check
> > > >>>>>> the response time when some other users are connected.
> > > >>>>> 
> > > >>>>> please keep in mind, that setting -hypervisor, disabling hpet and
> > > >>>>> only one vcpu
> > > >>>>> makes windows use tsc as clocksource. you have to make sure, that
> > > >>>>> your vm is not switching between physical sockets on your system
> > > >>>>> and that you have constant_tsc feature to have a stable tsc
> > > >>>>> between the cores in the same socket. its also likely that the
> > > >>>>> vm will crash when live migrated.
> > > >>>> 
> > > >>>> All true. I asked to try -hypervisor only to verify where we loose
> > > >>>> performance. Since you get good result with it frequent access to
> > > >>>> PM timer is probably the reason. I do not recommend using
> > > >>>> -hypervisor for production!
> > > >>>> 
> > > >>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> > > >>>> 
> > > >>>> Vadim is working on it. I'll let him answer.
> > > >>> 
> > > >>> It would be nice to have synthetic timers supported. But,  at the
> > > >>> moment, I'm only researching  this feature.
> > > >> 
> > > >> So it will take months at least?
> > > > 
> > > > I would say weeks.
> > > 
> > > Is there a way, we could contribute and help you with this?
> > 
> > Hi Peter,
> > You are welcome to add  an appropriate handler.
> 
> I think Vadim refers to this HV MSR
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85
> %29.aspx

This one is pretty simple to support. Please see attachments for more details.
I was thinking about synthetic  timers http://msdn.microsoft.com/en-
us/library/windows/hardware/ff542758(v=vs.85).aspx
> 
> --
> 			Gleb.

[-- Attachment #2: kvm.diff --]
[-- Type: text/x-patch, Size: 2016 bytes --]

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 74c9edf..fafc8ff 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -535,6 +535,7 @@ struct kvm_arch {
 	/* fields used by HYPER-V emulation */
 	u64 hv_guest_os_id;
 	u64 hv_hypercall;
+        u64 hv_ref_count;
 
 	atomic_t reader_counter;
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index c9d99e5..4562581 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1393,6 +1393,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr)
 	switch (msr) {
 	case HV_X64_MSR_GUEST_OS_ID:
 	case HV_X64_MSR_HYPERCALL:
+        case HV_X64_MSR_TIME_REF_COUNT:
 		r = true;
 		break;
 	}
@@ -1432,6 +1433,7 @@ static int set_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 		if (__copy_to_user((void __user *)addr, instructions, 4))
 			return 1;
 		kvm->arch.hv_hypercall = data;
+                kvm->arch.hv_ref_count = get_kernel_ns();
 		break;
 	}
 	default:
@@ -1467,6 +1469,9 @@ static int set_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 data)
 		return kvm_hv_vapic_msr_write(vcpu, APIC_ICR, data);
 	case HV_X64_MSR_TPR:
 		return kvm_hv_vapic_msr_write(vcpu, APIC_TASKPRI, data);
+        case 0x40000021:
+                break;
 	default:
 		pr_unimpl(vcpu, "HYPER-V unimplemented wrmsr: 0x%x "
 			  "data 0x%llx\n", msr, data);
@@ -1842,6 +1847,10 @@ static int get_msr_hyperv_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 	case HV_X64_MSR_HYPERCALL:
 		data = kvm->arch.hv_hypercall;
 		break;
+        case HV_X64_MSR_TIME_REF_COUNT:
+                data = get_kernel_ns() - kvm->arch.hv_ref_count; 
+                break;
 	default:
 		pr_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
 		return 1;
@@ -1873,6 +1882,9 @@ static int get_msr_hyperv(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata)
 	case HV_X64_MSR_APIC_ASSIST_PAGE:
 		data = vcpu->arch.hv_vapic;
 		break;
 	default:
 		pr_unimpl(vcpu, "Hyper-V unhandled rdmsr: 0x%x\n", msr);
 		return 1;

[-- Attachment #3: qemu.diff --]
[-- Type: text/x-patch, Size: 3111 bytes --]

diff --git a/target-i386/cpuid.c b/target-i386/cpuid.c
index c2edb64..5c85492 100644
--- a/target-i386/cpuid.c
+++ b/target-i386/cpuid.c
@@ -778,6 +778,9 @@ static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *cpu_model)
             hyperv_enable_relaxed_timing(true);
         } else if (!strcmp(featurestr, "hv_vapic")) {
             hyperv_enable_vapic_recommended(true);
+        } else if (!strcmp(featurestr, "hv_refcnt")) {
+            hyperv_enable_ref_count(true);
         } else {
             fprintf(stderr, "feature string `%s' not in format (+feature|-feature|feature=xyz)\n", featurestr);
             goto error;
diff --git a/target-i386/hyperv.c b/target-i386/hyperv.c
index f284e99..13a1cb7 100644
--- a/target-i386/hyperv.c
+++ b/target-i386/hyperv.c
@@ -15,6 +15,7 @@
 static bool hyperv_vapic;
 static bool hyperv_relaxed_timing;
 static int hyperv_spinlock_attempts = HYPERV_SPINLOCK_NEVER_RETRY;
+static bool hyperv_ref_count;
 
 void hyperv_enable_vapic_recommended(bool val)
 {
@@ -34,9 +35,17 @@ void hyperv_set_spinlock_retries(int val)
     }
 }
 
+void hyperv_enable_ref_count(bool val)
+{
+    hyperv_ref_count = val;
+}
+
 bool hyperv_enabled(void)
 {
-    return hyperv_hypercall_available() || hyperv_relaxed_timing_enabled();
+    return hyperv_hypercall_available() || 
+           hyperv_relaxed_timing_enabled() ||
+           hyperv_ref_counter_enabled();
 }
 
 bool hyperv_hypercall_available(void)
@@ -62,3 +71,9 @@ int hyperv_get_spinlock_retries(void)
 {
     return hyperv_spinlock_attempts;
 }
+
+bool hyperv_ref_counter_enabled(void)
+{
+    return hyperv_ref_count;
+}
diff --git a/target-i386/hyperv.h b/target-i386/hyperv.h
index bacb1d4..a65aa5f 100644
--- a/target-i386/hyperv.h
+++ b/target-i386/hyperv.h
@@ -30,10 +30,12 @@
 void hyperv_enable_vapic_recommended(bool val);
 void hyperv_enable_relaxed_timing(bool val);
 void hyperv_set_spinlock_retries(int val);
+void hyperv_enable_ref_count(bool val);
 #else
 static inline void hyperv_enable_vapic_recommended(bool val) { }
 static inline void hyperv_enable_relaxed_timing(bool val) { }
 static inline void hyperv_set_spinlock_retries(int val) { }
+static inline void hyperv_enable_ref_count(bool val) {}
 #endif
 
 bool hyperv_enabled(void);
@@ -41,5 +43,5 @@ bool hyperv_hypercall_available(void);
 bool hyperv_vapic_recommended(void);
 bool hyperv_relaxed_timing_enabled(void);
 int hyperv_get_spinlock_retries(void);
-
+bool hyperv_ref_counter_enabled(void);
 #endif /* QEMU_HW_HYPERV_H */
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 9a73207..31ca04a 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -414,7 +414,12 @@ int kvm_arch_init_vcpu(CPUState *env)
             c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
             c->eax |= HV_X64_MSR_APIC_ACCESS_AVAILABLE;
         }
-
+        if (hyperv_ref_counter_enabled()) {
+            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+            c->eax |= 0x200;
+        }
         c = &cpuid_data.entries[cpuid_i++];
         memset(c, 0, sizeof(*c));
         c->function = HYPERV_CPUID_ENLIGHTMENT_INFO;

Re: performance trouble

[Peter Lieven]

On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>> 		hello,
>>>>>>>>>>
>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>     to
>>>>>>>>>>> cpu definition in XML and check command line.
>>>>>>>>>>>
>>>>>>>>>> 	ok I try this but I can't use<cpu model>     to map the host cpu
>>>>>>>>>>
>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>      <cpu match='exact'>
>>>>>>>>>>
>>>>>>>>>>        <model>Opteron_G3</model>
>>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>
>>>>>>>>>>      </cpu>
>>>>>>>>>> 	
>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>
>>>>>>>>>> 	The log is here :
>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx
>>>>>>>>>> t.gz
>>>>>>>>>>
>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>
>>>>>>>>>> improvment. We keep this configuration for production : we check
>>>>>>>>>> the response time when some other users are connected.
>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet and
>>>>>>>>> only one vcpu
>>>>>>>>> makes windows use tsc as clocksource. you have to make sure, that
>>>>>>>>> your vm is not switching between physical sockets on your system
>>>>>>>>> and that you have constant_tsc feature to have a stable tsc
>>>>>>>>> between the cores in the same socket. its also likely that the
>>>>>>>>> vm will crash when live migrated.
>>>>>>>> All true. I asked to try -hypervisor only to verify where we loose
>>>>>>>> performance. Since you get good result with it frequent access to
>>>>>>>> PM timer is probably the reason. I do not recommend using
>>>>>>>> -hypervisor for production!
>>>>>>>>
>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>> It would be nice to have synthetic timers supported. But,  at the
>>>>>>> moment, I'm only researching  this feature.
>>>>>> So it will take months at least?
>>>>> I would say weeks.
>>>> Is there a way, we could contribute and help you with this?
>>> Hi Peter,
>>> You are welcome to add  an appropriate handler.
>> I think Vadim refers to this HV MSR
>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs.85
>> %29.aspx
> This one is pretty simple to support. Please see attachments for more details.
> I was thinking about synthetic  timers http://msdn.microsoft.com/en-
> us/library/windows/hardware/ff542758(v=vs.85).aspx
is this what microsoft qpc uses as clocksource in hyper-v?
i will check tomorrow.

thanks
vadim

>> --
>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> > On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>> 		hello,
> >>>>>>>>>> 
> >>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> >>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>     to
> >>>>>>>>>>> cpu definition in XML and check command line.
> >>>>>>>>>>> 
> >>>>>>>>>> 	ok I try this but I can't use<cpu model>     to map the host
> >>>>>>>>>> 	cpu
> >>>>>>>>>> 
> >>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>      <cpu match='exact'>
> >>>>>>>>>>      
> >>>>>>>>>>        <model>Opteron_G3</model>
> >>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>      
> >>>>>>>>>>      </cpu>
> >>>>>>>>>> 	
> >>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>> 
> >>>>>>>>>> 	The log is here :
> >>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx
> >>>>>>>>>> t.gz
> >>>>>>>>>> 
> >>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>>>>>> 
> >>>>>>>>>> improvment. We keep this configuration for production : we check
> >>>>>>>>>> the response time when some other users are connected.
> >>>>>>>>> 
> >>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet and
> >>>>>>>>> only one vcpu
> >>>>>>>>> makes windows use tsc as clocksource. you have to make sure, that
> >>>>>>>>> your vm is not switching between physical sockets on your system
> >>>>>>>>> and that you have constant_tsc feature to have a stable tsc
> >>>>>>>>> between the cores in the same socket. its also likely that the
> >>>>>>>>> vm will crash when live migrated.
> >>>>>>>> 
> >>>>>>>> All true. I asked to try -hypervisor only to verify where we loose
> >>>>>>>> performance. Since you get good result with it frequent access to
> >>>>>>>> PM timer is probably the reason. I do not recommend using
> >>>>>>>> -hypervisor for production!
> >>>>>>>> 
> >>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> >>>>>>>> 
> >>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>> 
> >>>>>>> It would be nice to have synthetic timers supported. But,  at the
> >>>>>>> moment, I'm only researching  this feature.
> >>>>>> 
> >>>>>> So it will take months at least?
> >>>>> 
> >>>>> I would say weeks.
> >>>> 
> >>>> Is there a way, we could contribute and help you with this?
> >>> 
> >>> Hi Peter,
> >>> You are welcome to add  an appropriate handler.
> >> 
> >> I think Vadim refers to this HV MSR
> >> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs
> >> .85 %29.aspx
> > 
> > This one is pretty simple to support. Please see attachments for more
> > details. I was thinking about synthetic  timers
> > http://msdn.microsoft.com/en-
> > us/library/windows/hardware/ff542758(v=vs.85).aspx
> 
> is this what microsoft qpc uses as clocksource in hyper-v?
Yes, it should be enough for Win7 / W2K8R2.  
Cheers,
Vadim.
> i will check tomorrow.
> 
> thanks
> vadim
> 
> >> --
> >> 
> >> 			Gleb.

Re: performance trouble

[Gleb Natapov]

On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> > On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> > > On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> > >> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> > >>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> > >>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > >>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> > >>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > >>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> > >>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > >>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> > >>>>>>>>>> 		hello,
> > >>>>>>>>>> 
> > >>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov ecrivait :
> > >>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>     to
> > >>>>>>>>>>> cpu definition in XML and check command line.
> > >>>>>>>>>>> 
> > >>>>>>>>>> 	ok I try this but I can't use<cpu model>     to map the host
> > >>>>>>>>>> 	cpu
> > >>>>>>>>>> 
> > >>>>>>>>>> (my libvirt is 0.9.8) so I use :
> > >>>>>>>>>>      <cpu match='exact'>
> > >>>>>>>>>>      
> > >>>>>>>>>>        <model>Opteron_G3</model>
> > >>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> > >>>>>>>>>>      
> > >>>>>>>>>>      </cpu>
> > >>>>>>>>>> 	
> > >>>>>>>>>> 	(the physical server use Opteron CPU).
> > >>>>>>>>>> 
> > >>>>>>>>>> 	The log is here :
> > >>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cpu.tx
> > >>>>>>>>>> t.gz
> > >>>>>>>>>> 
> > >>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> > >>>>>>>>>> 
> > >>>>>>>>>> improvment. We keep this configuration for production : we check
> > >>>>>>>>>> the response time when some other users are connected.
> > >>>>>>>>> 
> > >>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet and
> > >>>>>>>>> only one vcpu
> > >>>>>>>>> makes windows use tsc as clocksource. you have to make sure, that
> > >>>>>>>>> your vm is not switching between physical sockets on your system
> > >>>>>>>>> and that you have constant_tsc feature to have a stable tsc
> > >>>>>>>>> between the cores in the same socket. its also likely that the
> > >>>>>>>>> vm will crash when live migrated.
> > >>>>>>>> 
> > >>>>>>>> All true. I asked to try -hypervisor only to verify where we loose
> > >>>>>>>> performance. Since you get good result with it frequent access to
> > >>>>>>>> PM timer is probably the reason. I do not recommend using
> > >>>>>>>> -hypervisor for production!
> > >>>>>>>> 
> > >>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v timers?
> > >>>>>>>> 
> > >>>>>>>> Vadim is working on it. I'll let him answer.
> > >>>>>>> 
> > >>>>>>> It would be nice to have synthetic timers supported. But,  at the
> > >>>>>>> moment, I'm only researching  this feature.
> > >>>>>> 
> > >>>>>> So it will take months at least?
> > >>>>> 
> > >>>>> I would say weeks.
> > >>>> 
> > >>>> Is there a way, we could contribute and help you with this?
> > >>> 
> > >>> Hi Peter,
> > >>> You are welcome to add  an appropriate handler.
> > >> 
> > >> I think Vadim refers to this HV MSR
> > >> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28v=vs
> > >> .85 %29.aspx
> > > 
> > > This one is pretty simple to support. Please see attachments for more
> > > details. I was thinking about synthetic  timers
> > > http://msdn.microsoft.com/en-
> > > us/library/windows/hardware/ff542758(v=vs.85).aspx
> > 
> > is this what microsoft qpc uses as clocksource in hyper-v?
> Yes, it should be enough for Win7 / W2K8R2.  
To clarify the thing that microsoft qpc uses is what is implemented by
the patch Vadim attached to his previous email. But I believe that
additional qemu patch is needed for Windows to actually use it.

--
			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> > On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> > > On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> > > > On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> > > >> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> > > >>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> > > >>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > > >>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> > > >>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > > >>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> > > >>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > > >>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> > > >>>>>>>>>> 		hello,
> > > >>>>>>>>>> 
> > > >>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov 
ecrivait :
> > > >>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    
> > > >>>>>>>>>>> to cpu definition in XML and check command line.
> > > >>>>>>>>>>> 
> > > >>>>>>>>>> 	ok I try this but I can't use<cpu model>     to map the
> > > >>>>>>>>>> 	host cpu
> > > >>>>>>>>>> 
> > > >>>>>>>>>> (my libvirt is 0.9.8) so I use :
> > > >>>>>>>>>>      <cpu match='exact'>
> > > >>>>>>>>>>      
> > > >>>>>>>>>>        <model>Opteron_G3</model>
> > > >>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> > > >>>>>>>>>>      
> > > >>>>>>>>>>      </cpu>
> > > >>>>>>>>>> 	
> > > >>>>>>>>>> 	(the physical server use Opteron CPU).
> > > >>>>>>>>>> 
> > > >>>>>>>>>> 	The log is here :
> > > >>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> > > >>>>>>>>>> u.tx t.gz
> > > >>>>>>>>>> 
> > > >>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> > > >>>>>>>>>> 
> > > >>>>>>>>>> improvment. We keep this configuration for production : we
> > > >>>>>>>>>> check the response time when some other users are
> > > >>>>>>>>>> connected.
> > > >>>>>>>>> 
> > > >>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
> > > >>>>>>>>> and only one vcpu
> > > >>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
> > > >>>>>>>>> that your vm is not switching between physical sockets on
> > > >>>>>>>>> your system and that you have constant_tsc feature to have a
> > > >>>>>>>>> stable tsc between the cores in the same socket. its also
> > > >>>>>>>>> likely that the vm will crash when live migrated.
> > > >>>>>>>> 
> > > >>>>>>>> All true. I asked to try -hypervisor only to verify where we
> > > >>>>>>>> loose performance. Since you get good result with it frequent
> > > >>>>>>>> access to PM timer is probably the reason. I do not recommend
> > > >>>>>>>> using -hypervisor for production!
> > > >>>>>>>> 
> > > >>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> > > >>>>>>>>> timers?
> > > >>>>>>>> 
> > > >>>>>>>> Vadim is working on it. I'll let him answer.
> > > >>>>>>> 
> > > >>>>>>> It would be nice to have synthetic timers supported. But,  at
> > > >>>>>>> the moment, I'm only researching  this feature.
> > > >>>>>> 
> > > >>>>>> So it will take months at least?
> > > >>>>> 
> > > >>>>> I would say weeks.
> > > >>>> 
> > > >>>> Is there a way, we could contribute and help you with this?
> > > >>> 
> > > >>> Hi Peter,
> > > >>> You are welcome to add  an appropriate handler.
> > > >> 
> > > >> I think Vadim refers to this HV MSR
> > > >> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> > > >> v=vs .85 %29.aspx
> > > > 
> > > > This one is pretty simple to support. Please see attachments for more
> > > > details. I was thinking about synthetic  timers
> > > > http://msdn.microsoft.com/en-
> > > > us/library/windows/hardware/ff542758(v=vs.85).aspx
> > > 
> > > is this what microsoft qpc uses as clocksource in hyper-v?
> > 
> > Yes, it should be enough for Win7 / W2K8R2.
> 
> To clarify the thing that microsoft qpc uses is what is implemented by
> the patch Vadim attached to his previous email. But I believe that
> additional qemu patch is needed for Windows to actually use it.

You are right.
bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
should be completely removed from ACPI. 
> 
> --
> 			Gleb.

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 27, 2012 at 11:23:33AM +0200, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> > On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> > > On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> > > > On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> > > > > On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> > > > >> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> > > > >>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> > > > >>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> > > > >>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> > > > >>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> > > > >>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> > > > >>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> > > > >>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> > > > >>>>>>>>>> 		hello,
> > > > >>>>>>>>>> 
> > > > >>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov 
> ecrivait :
> > > > >>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>    
> > > > >>>>>>>>>>> to cpu definition in XML and check command line.
> > > > >>>>>>>>>>> 
> > > > >>>>>>>>>> 	ok I try this but I can't use<cpu model>     to map the
> > > > >>>>>>>>>> 	host cpu
> > > > >>>>>>>>>> 
> > > > >>>>>>>>>> (my libvirt is 0.9.8) so I use :
> > > > >>>>>>>>>>      <cpu match='exact'>
> > > > >>>>>>>>>>      
> > > > >>>>>>>>>>        <model>Opteron_G3</model>
> > > > >>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> > > > >>>>>>>>>>      
> > > > >>>>>>>>>>      </cpu>
> > > > >>>>>>>>>> 	
> > > > >>>>>>>>>> 	(the physical server use Opteron CPU).
> > > > >>>>>>>>>> 
> > > > >>>>>>>>>> 	The log is here :
> > > > >>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> > > > >>>>>>>>>> u.tx t.gz
> > > > >>>>>>>>>> 
> > > > >>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> > > > >>>>>>>>>> 
> > > > >>>>>>>>>> improvment. We keep this configuration for production : we
> > > > >>>>>>>>>> check the response time when some other users are
> > > > >>>>>>>>>> connected.
> > > > >>>>>>>>> 
> > > > >>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
> > > > >>>>>>>>> and only one vcpu
> > > > >>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
> > > > >>>>>>>>> that your vm is not switching between physical sockets on
> > > > >>>>>>>>> your system and that you have constant_tsc feature to have a
> > > > >>>>>>>>> stable tsc between the cores in the same socket. its also
> > > > >>>>>>>>> likely that the vm will crash when live migrated.
> > > > >>>>>>>> 
> > > > >>>>>>>> All true. I asked to try -hypervisor only to verify where we
> > > > >>>>>>>> loose performance. Since you get good result with it frequent
> > > > >>>>>>>> access to PM timer is probably the reason. I do not recommend
> > > > >>>>>>>> using -hypervisor for production!
> > > > >>>>>>>> 
> > > > >>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> > > > >>>>>>>>> timers?
> > > > >>>>>>>> 
> > > > >>>>>>>> Vadim is working on it. I'll let him answer.
> > > > >>>>>>> 
> > > > >>>>>>> It would be nice to have synthetic timers supported. But,  at
> > > > >>>>>>> the moment, I'm only researching  this feature.
> > > > >>>>>> 
> > > > >>>>>> So it will take months at least?
> > > > >>>>> 
> > > > >>>>> I would say weeks.
> > > > >>>> 
> > > > >>>> Is there a way, we could contribute and help you with this?
> > > > >>> 
> > > > >>> Hi Peter,
> > > > >>> You are welcome to add  an appropriate handler.
> > > > >> 
> > > > >> I think Vadim refers to this HV MSR
> > > > >> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> > > > >> v=vs .85 %29.aspx
> > > > > 
> > > > > This one is pretty simple to support. Please see attachments for more
> > > > > details. I was thinking about synthetic  timers
> > > > > http://msdn.microsoft.com/en-
> > > > > us/library/windows/hardware/ff542758(v=vs.85).aspx
> > > > 
> > > > is this what microsoft qpc uses as clocksource in hyper-v?
> > > 
> > > Yes, it should be enough for Win7 / W2K8R2.
> > 
> > To clarify the thing that microsoft qpc uses is what is implemented by
> > the patch Vadim attached to his previous email. But I believe that
> > additional qemu patch is needed for Windows to actually use it.
> 
> You are right.
> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> should be completely removed from ACPI. 
Upstream SeaBIOS handles HPET properly now.

--
			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> ecrivait :
>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>      to map the
>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>       <cpu match='exact'>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>         <model>Opteron_G3</model>
>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>       </cpu>
>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>
>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>> timers?
>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>> So it will take months at least?
>>>>>>>>> I would say weeks.
>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>> Hi Peter,
>>>>>>> You are welcome to add  an appropriate handler.
>>>>>> I think Vadim refers to this HV MSR
>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>> v=vs .85 %29.aspx
>>>>> This one is pretty simple to support. Please see attachments for more
>>>>> details. I was thinking about synthetic  timers
>>>>> http://msdn.microsoft.com/en-
>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>> Yes, it should be enough for Win7 / W2K8R2.
>> To clarify the thing that microsoft qpc uses is what is implemented by
>> the patch Vadim attached to his previous email. But I believe that
>> additional qemu patch is needed for Windows to actually use it.
> You are right.
> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> should be completely removed from ACPI.

could you advise how to do this and/or make a patch?

the stuff you send yesterday is for qemu, right? would
it be possible to use it in qemu-kvm also?

peter

>> --
>> 			Gleb.

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>>On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>>>>>>On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>		hello,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >ecrivait :
> >>>>>>>>>>>>>>>Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>	ok I try this but I can't use<cpu model>      to map the
> >>>>>>>>>>>>>>	host cpu
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>(my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>      <cpu match='exact'>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>        <model>Opteron_G3</model>
> >>>>>>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>      </cpu>
> >>>>>>>>>>>>>>	
> >>>>>>>>>>>>>>	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>	The log is here :
> >>>>>>>>>>>>>>http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> >>>>>>>>>>>>>>u.tx t.gz
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>improvment. We keep this configuration for production : we
> >>>>>>>>>>>>>>check the response time when some other users are
> >>>>>>>>>>>>>>connected.
> >>>>>>>>>>>>>please keep in mind, that setting -hypervisor, disabling hpet
> >>>>>>>>>>>>>and only one vcpu
> >>>>>>>>>>>>>makes windows use tsc as clocksource. you have to make sure,
> >>>>>>>>>>>>>that your vm is not switching between physical sockets on
> >>>>>>>>>>>>>your system and that you have constant_tsc feature to have a
> >>>>>>>>>>>>>stable tsc between the cores in the same socket. its also
> >>>>>>>>>>>>>likely that the vm will crash when live migrated.
> >>>>>>>>>>>>All true. I asked to try -hypervisor only to verify where we
> >>>>>>>>>>>>loose performance. Since you get good result with it frequent
> >>>>>>>>>>>>access to PM timer is probably the reason. I do not recommend
> >>>>>>>>>>>>using -hypervisor for production!
> >>>>>>>>>>>>
> >>>>>>>>>>>>>@gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>timers?
> >>>>>>>>>>>>Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>It would be nice to have synthetic timers supported. But,  at
> >>>>>>>>>>>the moment, I'm only researching  this feature.
> >>>>>>>>>>So it will take months at least?
> >>>>>>>>>I would say weeks.
> >>>>>>>>Is there a way, we could contribute and help you with this?
> >>>>>>>Hi Peter,
> >>>>>>>You are welcome to add  an appropriate handler.
> >>>>>>I think Vadim refers to this HV MSR
> >>>>>>http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> >>>>>>v=vs .85 %29.aspx
> >>>>>This one is pretty simple to support. Please see attachments for more
> >>>>>details. I was thinking about synthetic  timers
> >>>>>http://msdn.microsoft.com/en-
> >>>>>us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>is this what microsoft qpc uses as clocksource in hyper-v?
> >>>Yes, it should be enough for Win7 / W2K8R2.
> >>To clarify the thing that microsoft qpc uses is what is implemented by
> >>the patch Vadim attached to his previous email. But I believe that
> >>additional qemu patch is needed for Windows to actually use it.
> >You are right.
> >bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >should be completely removed from ACPI.
> 
> could you advise how to do this and/or make a patch?
> 
> the stuff you send yesterday is for qemu, right? would
> it be possible to use it in qemu-kvm also?
> 
No, they are for kernel.

--
			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> > 
> > ecrivait :
> >>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>> to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>      to map the
> >>>>>>>>>>>>>> 	host cpu
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>       <cpu match='exact'>
> >>>>>>>>>>>>>>       
> >>>>>>>>>>>>>>         <model>Opteron_G3</model>
> >>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>       
> >>>>>>>>>>>>>>       </cpu>
> >>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> >>>>>>>>>>>>>> u.tx t.gz
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> improvment. We keep this configuration for production : we
> >>>>>>>>>>>>>> check the response time when some other users are
> >>>>>>>>>>>>>> connected.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
> >>>>>>>>>>>>> and only one vcpu
> >>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
> >>>>>>>>>>>>> that your vm is not switching between physical sockets on
> >>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
> >>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
> >>>>>>>>>>>>> likely that the vm will crash when live migrated.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
> >>>>>>>>>>>> loose performance. Since you get good result with it frequent
> >>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
> >>>>>>>>>>>> using -hypervisor for production!
> >>>>>>>>>>>> 
> >>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>> timers?
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>> 
> >>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
> >>>>>>>>>>> the moment, I'm only researching  this feature.
> >>>>>>>>>> 
> >>>>>>>>>> So it will take months at least?
> >>>>>>>>> 
> >>>>>>>>> I would say weeks.
> >>>>>>>> 
> >>>>>>>> Is there a way, we could contribute and help you with this?
> >>>>>>> 
> >>>>>>> Hi Peter,
> >>>>>>> You are welcome to add  an appropriate handler.
> >>>>>> 
> >>>>>> I think Vadim refers to this HV MSR
> >>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> >>>>>> v=vs .85 %29.aspx
> >>>>> 
> >>>>> This one is pretty simple to support. Please see attachments for more
> >>>>> details. I was thinking about synthetic  timers
> >>>>> http://msdn.microsoft.com/en-
> >>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>> 
> >>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>> 
> >>> Yes, it should be enough for Win7 / W2K8R2.
> >> 
> >> To clarify the thing that microsoft qpc uses is what is implemented by
> >> the patch Vadim attached to his previous email. But I believe that
> >> additional qemu patch is needed for Windows to actually use it.
> > 
> > You are right.
> > bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> > should be completely removed from ACPI.
> 
> could you advise how to do this and/or make a patch?
Gleb mentioned that it properly handled in upstream,
otherwise just comment the entire HPET section in 
acpi-dsdt.dsl file.
> 
> the stuff you send yesterday is for qemu, right? would
> it be possible to use it in qemu-kvm also?
Yes, but don't forget about kvm patch as well.
> 
> peter
> 
> >> --
> >> 
> >> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 12:40, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>> ecrivait :
>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>       to map the
>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>        <cpu match='exact'>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>          <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>          <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>        </cpu>
>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>> I would say weeks.
>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>> Hi Peter,
>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>>>> v=vs .85 %29.aspx
>>>>>>> This one is pretty simple to support. Please see attachments for more
>>>>>>> details. I was thinking about synthetic  timers
>>>>>>> http://msdn.microsoft.com/en-
>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>> the patch Vadim attached to his previous email. But I believe that
>>>> additional qemu patch is needed for Windows to actually use it.
>>> You are right.
>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>> should be completely removed from ACPI.
>> could you advise how to do this and/or make a patch?
> Gleb mentioned that it properly handled in upstream,
> otherwise just comment the entire HPET section in
> acpi-dsdt.dsl file.
i have upstream bios installed. so -no-hpet should disable hpet completely.
can you give a hint, what
"bits 1 and 9 must be set to on in leaf 0x40000003" means?

>> the stuff you send yesterday is for qemu, right? would
>> it be possible to use it in qemu-kvm also?
> Yes, but don't forget about kvm patch as well.
ok, i will try my best. would you consider your patch a quick hack
or do you think it would be worth to be uploaded to the upstream repository?

peter
>> peter
>>
>>>> --
>>>>
>>>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

[-- Attachment #1: Type: Text/Plain, Size: 6119 bytes --]

On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
> On 27.03.2012 12:40, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
> >> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>> 
> >>> ecrivait :
> >>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>       to map 
the
> >>>>>>>>>>>>>>>> 	host cpu
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>        <cpu match='exact'>
> >>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>          <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>          <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>        </cpu>
> >>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-
> >>>>>>>>>>>>>>>> cp u.tx t.gz
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
> >>>>>>>>>>>>>>>> 	great
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
> >>>>>>>>>>>>>>>> check the response time when some other users are
> >>>>>>>>>>>>>>>> connected.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling
> >>>>>>>>>>>>>>> hpet and only one vcpu
> >>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
> >>>>>>>>>>>>>>> sure, that your vm is not switching between physical
> >>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
> >>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the same
> >>>>>>>>>>>>>>> socket. its also likely that the vm will crash when live
> >>>>>>>>>>>>>>> migrated.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
> >>>>>>>>>>>>>> loose performance. Since you get good result with it
> >>>>>>>>>>>>>> frequent access to PM timer is probably the reason. I do
> >>>>>>>>>>>>>> not recommend using -hypervisor for production!
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>> timers?
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
> >>>>>>>>>>>>> the moment, I'm only researching  this feature.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>> 
> >>>>>>>>>>> I would say weeks.
> >>>>>>>>>> 
> >>>>>>>>>> Is there a way, we could contribute and help you with this?
> >>>>>>>>> 
> >>>>>>>>> Hi Peter,
> >>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>> 
> >>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%
> >>>>>>>> 28 v=vs .85 %29.aspx
> >>>>>>> 
> >>>>>>> This one is pretty simple to support. Please see attachments for
> >>>>>>> more details. I was thinking about synthetic  timers
> >>>>>>> http://msdn.microsoft.com/en-
> >>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>> 
> >>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>> 
> >>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>> 
> >>>> To clarify the thing that microsoft qpc uses is what is implemented by
> >>>> the patch Vadim attached to his previous email. But I believe that
> >>>> additional qemu patch is needed for Windows to actually use it.
> >>> 
> >>> You are right.
> >>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>> should be completely removed from ACPI.
> >> 
> >> could you advise how to do this and/or make a patch?
> > 
> > Gleb mentioned that it properly handled in upstream,
> > otherwise just comment the entire HPET section in
> > acpi-dsdt.dsl file.
> 
> i have upstream bios installed. so -no-hpet should disable hpet completely.
> can you give a hint, what
> "bits 1 and 9 must be set to on in leaf 0x40000003" means?
I mean the following code:
+        if (hyperv_ref_counter_enabled()) {
+            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+            c->eax |= 0x200;
+        }

Please see attached file for more information.

> 
> >> the stuff you send yesterday is for qemu, right? would
> >> it be possible to use it in qemu-kvm also?
> > 
> > Yes, but don't forget about kvm patch as well.
> 
> ok, i will try my best. would you consider your patch a quick hack
> or do you think it would be worth to be uploaded to the upstream
> repository?
It was just a brief attempt from my side, mostly inspirited by our with Gleb
conversation,  to see what it worth to turn this option on.
It is not fully tested. It will crash Win8 (as well as the rest of the 
currently introduced hyper-v features). 
I wouldn't commit this code without comprehensive testing.
Vadim. 
> 
> peter
> 
> >> peter
> >> 
> >>>> --
> >>>> 
> >>>> 			Gleb.

[-- Attachment #2: qemu.diff --]
[-- Type: text/x-patch, Size: 3111 bytes --]

diff --git a/target-i386/cpuid.c b/target-i386/cpuid.c
index c2edb64..5c85492 100644
--- a/target-i386/cpuid.c
+++ b/target-i386/cpuid.c
@@ -778,6 +778,9 @@ static int cpu_x86_find_by_name(x86_def_t *x86_cpu_def, const char *cpu_model)
             hyperv_enable_relaxed_timing(true);
         } else if (!strcmp(featurestr, "hv_vapic")) {
             hyperv_enable_vapic_recommended(true);
+        } else if (!strcmp(featurestr, "hv_refcnt")) {
+            hyperv_enable_ref_count(true);
         } else {
             fprintf(stderr, "feature string `%s' not in format (+feature|-feature|feature=xyz)\n", featurestr);
             goto error;
diff --git a/target-i386/hyperv.c b/target-i386/hyperv.c
index f284e99..13a1cb7 100644
--- a/target-i386/hyperv.c
+++ b/target-i386/hyperv.c
@@ -15,6 +15,7 @@
 static bool hyperv_vapic;
 static bool hyperv_relaxed_timing;
 static int hyperv_spinlock_attempts = HYPERV_SPINLOCK_NEVER_RETRY;
+static bool hyperv_ref_count;
 
 void hyperv_enable_vapic_recommended(bool val)
 {
@@ -34,9 +35,17 @@ void hyperv_set_spinlock_retries(int val)
     }
 }
 
+void hyperv_enable_ref_count(bool val)
+{
+    hyperv_ref_count = val;
+}
+
 bool hyperv_enabled(void)
 {
-    return hyperv_hypercall_available() || hyperv_relaxed_timing_enabled();
+    return hyperv_hypercall_available() || 
+           hyperv_relaxed_timing_enabled() ||
+           hyperv_ref_counter_enabled();
 }
 
 bool hyperv_hypercall_available(void)
@@ -62,3 +71,9 @@ int hyperv_get_spinlock_retries(void)
 {
     return hyperv_spinlock_attempts;
 }
+
+bool hyperv_ref_counter_enabled(void)
+{
+    return hyperv_ref_count;
+}
diff --git a/target-i386/hyperv.h b/target-i386/hyperv.h
index bacb1d4..a65aa5f 100644
--- a/target-i386/hyperv.h
+++ b/target-i386/hyperv.h
@@ -30,10 +30,12 @@
 void hyperv_enable_vapic_recommended(bool val);
 void hyperv_enable_relaxed_timing(bool val);
 void hyperv_set_spinlock_retries(int val);
+void hyperv_enable_ref_count(bool val);
 #else
 static inline void hyperv_enable_vapic_recommended(bool val) { }
 static inline void hyperv_enable_relaxed_timing(bool val) { }
 static inline void hyperv_set_spinlock_retries(int val) { }
+static inline void hyperv_enable_ref_count(bool val) {}
 #endif
 
 bool hyperv_enabled(void);
@@ -41,5 +43,5 @@ bool hyperv_hypercall_available(void);
 bool hyperv_vapic_recommended(void);
 bool hyperv_relaxed_timing_enabled(void);
 int hyperv_get_spinlock_retries(void);
-
+bool hyperv_ref_counter_enabled(void);
 #endif /* QEMU_HW_HYPERV_H */
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index 9a73207..31ca04a 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -414,7 +414,12 @@ int kvm_arch_init_vcpu(CPUState *env)
             c->eax |= HV_X64_MSR_HYPERCALL_AVAILABLE;
             c->eax |= HV_X64_MSR_APIC_ACCESS_AVAILABLE;
         }
-
+        if (hyperv_ref_counter_enabled()) {
+            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
+            c->eax |= 0x200;
+        }
         c = &cpuid_data.entries[cpuid_i++];
         memset(c, 0, sizeof(*c));
         c->function = HYPERV_CPUID_ENLIGHTMENT_INFO;

Re: performance trouble

[Peter Lieven]

On 27.03.2012 12:00, Gleb Natapov wrote:
> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>> ecrivait :
>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>       to map the
>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>       <cpu match='exact'>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>         <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>       </cpu>
>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>> I would say weeks.
>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>> Hi Peter,
>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>>>> v=vs .85 %29.aspx
>>>>>>> This one is pretty simple to support. Please see attachments for more
>>>>>>> details. I was thinking about synthetic  timers
>>>>>>> http://msdn.microsoft.com/en-
>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>> the patch Vadim attached to his previous email. But I believe that
>>>> additional qemu patch is needed for Windows to actually use it.
>>> You are right.
>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>> should be completely removed from ACPI.
>> could you advise how to do this and/or make a patch?
>>
>> the stuff you send yesterday is for qemu, right? would
>> it be possible to use it in qemu-kvm also?
>>
> No, they are for kernel.
i meant the qemu.diff file.

if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?

peter



> --
> 			Gleb.

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
> On 27.03.2012 12:00, Gleb Natapov wrote:
> >On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> >>On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>>On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>>On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>>>>On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>>>>>>>>On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>		hello,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>>ecrivait :
> >>>>>>>>>>>>>>>>>Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>	ok I try this but I can't use<cpu model>       to map the
> >>>>>>>>>>>>>>>>	host cpu
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>(my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>      <cpu match='exact'>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>        <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>      </cpu>
> >>>>>>>>>>>>>>>>	
> >>>>>>>>>>>>>>>>	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>	The log is here :
> >>>>>>>>>>>>>>>>http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> >>>>>>>>>>>>>>>>u.tx t.gz
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>improvment. We keep this configuration for production : we
> >>>>>>>>>>>>>>>>check the response time when some other users are
> >>>>>>>>>>>>>>>>connected.
> >>>>>>>>>>>>>>>please keep in mind, that setting -hypervisor, disabling hpet
> >>>>>>>>>>>>>>>and only one vcpu
> >>>>>>>>>>>>>>>makes windows use tsc as clocksource. you have to make sure,
> >>>>>>>>>>>>>>>that your vm is not switching between physical sockets on
> >>>>>>>>>>>>>>>your system and that you have constant_tsc feature to have a
> >>>>>>>>>>>>>>>stable tsc between the cores in the same socket. its also
> >>>>>>>>>>>>>>>likely that the vm will crash when live migrated.
> >>>>>>>>>>>>>>All true. I asked to try -hypervisor only to verify where we
> >>>>>>>>>>>>>>loose performance. Since you get good result with it frequent
> >>>>>>>>>>>>>>access to PM timer is probably the reason. I do not recommend
> >>>>>>>>>>>>>>using -hypervisor for production!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>@gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>>timers?
> >>>>>>>>>>>>>>Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>It would be nice to have synthetic timers supported. But,  at
> >>>>>>>>>>>>>the moment, I'm only researching  this feature.
> >>>>>>>>>>>>So it will take months at least?
> >>>>>>>>>>>I would say weeks.
> >>>>>>>>>>Is there a way, we could contribute and help you with this?
> >>>>>>>>>Hi Peter,
> >>>>>>>>>You are welcome to add  an appropriate handler.
> >>>>>>>>I think Vadim refers to this HV MSR
> >>>>>>>>http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> >>>>>>>>v=vs .85 %29.aspx
> >>>>>>>This one is pretty simple to support. Please see attachments for more
> >>>>>>>details. I was thinking about synthetic  timers
> >>>>>>>http://msdn.microsoft.com/en-
> >>>>>>>us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>Yes, it should be enough for Win7 / W2K8R2.
> >>>>To clarify the thing that microsoft qpc uses is what is implemented by
> >>>>the patch Vadim attached to his previous email. But I believe that
> >>>>additional qemu patch is needed for Windows to actually use it.
> >>>You are right.
> >>>bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>should be completely removed from ACPI.
> >>could you advise how to do this and/or make a patch?
> >>
> >>the stuff you send yesterday is for qemu, right? would
> >>it be possible to use it in qemu-kvm also?
> >>
> >No, they are for kernel.
> i meant the qemu.diff file.
> 
Yes, I missed the second attachment.

> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
> 
Looks like it.

--
			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 14:26, Gleb Natapov wrote:
> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>        to map the
>>>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>       <cpu match='exact'>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>         <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>       </cpu>
>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>> Hi Peter,
>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>>>>>> v=vs .85 %29.aspx
>>>>>>>>> This one is pretty simple to support. Please see attachments for more
>>>>>>>>> details. I was thinking about synthetic  timers
>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>>>> the patch Vadim attached to his previous email. But I believe that
>>>>>> additional qemu patch is needed for Windows to actually use it.
>>>>> You are right.
>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>> should be completely removed from ACPI.
>>>> could you advise how to do this and/or make a patch?
>>>>
>>>> the stuff you send yesterday is for qemu, right? would
>>>> it be possible to use it in qemu-kvm also?
>>>>
>>> No, they are for kernel.
>> i meant the qemu.diff file.
>>
> Yes, I missed the second attachment.
>
>> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
>>
> Looks like it.
ok, so it would be interesting if it helps to avoid the pmtimer reads
we observed earlier. right?

peter

> --
> 			Gleb.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: performance trouble

[Gleb Natapov]

On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
> On 27.03.2012 14:26, Gleb Natapov wrote:
> >On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
> >>On 27.03.2012 12:00, Gleb Natapov wrote:
> >>>On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> >>>>On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>>On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>>On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>>On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>>On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>>On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>>>>On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>>On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>>>>On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>>>>>>On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
> >>>>>>>>>>>>>>>>>On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>>		hello,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>>>>ecrivait :
> >>>>>>>>>>>>>>>>>>>Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>>to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>	ok I try this but I can't use<cpu model>        to map the
> >>>>>>>>>>>>>>>>>>	host cpu
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>(my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>      <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>        <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>        <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>      </cpu>
> >>>>>>>>>>>>>>>>>>	
> >>>>>>>>>>>>>>>>>>	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>	The log is here :
> >>>>>>>>>>>>>>>>>>http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
> >>>>>>>>>>>>>>>>>>u.tx t.gz
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>	And now with only 1 vcpu, the response time is 8.5s, great
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>improvment. We keep this configuration for production : we
> >>>>>>>>>>>>>>>>>>check the response time when some other users are
> >>>>>>>>>>>>>>>>>>connected.
> >>>>>>>>>>>>>>>>>please keep in mind, that setting -hypervisor, disabling hpet
> >>>>>>>>>>>>>>>>>and only one vcpu
> >>>>>>>>>>>>>>>>>makes windows use tsc as clocksource. you have to make sure,
> >>>>>>>>>>>>>>>>>that your vm is not switching between physical sockets on
> >>>>>>>>>>>>>>>>>your system and that you have constant_tsc feature to have a
> >>>>>>>>>>>>>>>>>stable tsc between the cores in the same socket. its also
> >>>>>>>>>>>>>>>>>likely that the vm will crash when live migrated.
> >>>>>>>>>>>>>>>>All true. I asked to try -hypervisor only to verify where we
> >>>>>>>>>>>>>>>>loose performance. Since you get good result with it frequent
> >>>>>>>>>>>>>>>>access to PM timer is probably the reason. I do not recommend
> >>>>>>>>>>>>>>>>using -hypervisor for production!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>@gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>>>>timers?
> >>>>>>>>>>>>>>>>Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>>It would be nice to have synthetic timers supported. But,  at
> >>>>>>>>>>>>>>>the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>>So it will take months at least?
> >>>>>>>>>>>>>I would say weeks.
> >>>>>>>>>>>>Is there a way, we could contribute and help you with this?
> >>>>>>>>>>>Hi Peter,
> >>>>>>>>>>>You are welcome to add  an appropriate handler.
> >>>>>>>>>>I think Vadim refers to this HV MSR
> >>>>>>>>>>http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
> >>>>>>>>>>v=vs .85 %29.aspx
> >>>>>>>>>This one is pretty simple to support. Please see attachments for more
> >>>>>>>>>details. I was thinking about synthetic  timers
> >>>>>>>>>http://msdn.microsoft.com/en-
> >>>>>>>>>us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>>is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>>Yes, it should be enough for Win7 / W2K8R2.
> >>>>>>To clarify the thing that microsoft qpc uses is what is implemented by
> >>>>>>the patch Vadim attached to his previous email. But I believe that
> >>>>>>additional qemu patch is needed for Windows to actually use it.
> >>>>>You are right.
> >>>>>bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>>should be completely removed from ACPI.
> >>>>could you advise how to do this and/or make a patch?
> >>>>
> >>>>the stuff you send yesterday is for qemu, right? would
> >>>>it be possible to use it in qemu-kvm also?
> >>>>
> >>>No, they are for kernel.
> >>i meant the qemu.diff file.
> >>
> >Yes, I missed the second attachment.
> >
> >>if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
> >>
> >Looks like it.
> ok, so it would be interesting if it helps to avoid the pmtimer reads
> we observed earlier. right?
> 
Yes.

--
			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 14:29, Gleb Natapov wrote:
> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
>> On 27.03.2012 14:26, Gleb Natapov wrote:
>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         to map the
>>>>>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>       <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>       </cpu>
>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>>>>>>>> v=vs .85 %29.aspx
>>>>>>>>>>> This one is pretty simple to support. Please see attachments for more
>>>>>>>>>>> details. I was thinking about synthetic  timers
>>>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>>>>>> the patch Vadim attached to his previous email. But I believe that
>>>>>>>> additional qemu patch is needed for Windows to actually use it.
>>>>>>> You are right.
>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>> should be completely removed from ACPI.
>>>>>> could you advise how to do this and/or make a patch?
>>>>>>
>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>> it be possible to use it in qemu-kvm also?
>>>>>>
>>>>> No, they are for kernel.
>>>> i meant the qemu.diff file.
>>>>
>>> Yes, I missed the second attachment.
>>>
>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
>>>>
>>> Looks like it.
>> ok, so it would be interesting if it helps to avoid the pmtimer reads
>> we observed earlier. right?
>>
> Yes.
ok, will try it.

can you give me a short advice if the patch has to applied to qemu-kvm 
or qemu latest
from git?

peter

> --
> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 14:29, Gleb Natapov wrote:
> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
>> On 27.03.2012 14:26, Gleb Natapov wrote:
>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         to map the
>>>>>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>       <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>       </cpu>
>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-cp
>>>>>>>>>>>>>>>>>>>> u.tx t.gz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s, great
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling hpet
>>>>>>>>>>>>>>>>>>> and only one vcpu
>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make sure,
>>>>>>>>>>>>>>>>>>> that your vm is not switching between physical sockets on
>>>>>>>>>>>>>>>>>>> your system and that you have constant_tsc feature to have a
>>>>>>>>>>>>>>>>>>> stable tsc between the cores in the same socket. its also
>>>>>>>>>>>>>>>>>>> likely that the vm will crash when live migrated.
>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>>>>>> loose performance. Since you get good result with it frequent
>>>>>>>>>>>>>>>>>> access to PM timer is probably the reason. I do not recommend
>>>>>>>>>>>>>>>>>> using -hypervisor for production!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%28
>>>>>>>>>>>> v=vs .85 %29.aspx
>>>>>>>>>>> This one is pretty simple to support. Please see attachments for more
>>>>>>>>>>> details. I was thinking about synthetic  timers
>>>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>>>>>> the patch Vadim attached to his previous email. But I believe that
>>>>>>>> additional qemu patch is needed for Windows to actually use it.
>>>>>>> You are right.
>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>> should be completely removed from ACPI.
>>>>>> could you advise how to do this and/or make a patch?
>>>>>>
>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>> it be possible to use it in qemu-kvm also?
>>>>>>
>>>>> No, they are for kernel.
>>>> i meant the qemu.diff file.
>>>>
>>> Yes, I missed the second attachment.
>>>
>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
>>>>
>>> Looks like it.
>> ok, so it would be interesting if it helps to avoid the pmtimer reads
>> we observed earlier. right?
>>
> Yes.
first feedback: performance seems to be amazing. i cannot confirm that 
it breaks hv_spinlocks, hv_vapic and hv_relaxed.
why did you assume this?

no more pmtimer reads. i can now almost fully utililizy a 1GBit 
interface with a file transfer while there was not one
cpu core fully utilized as observed with pmtimer. some live migration 
tests revealed that it did not crash even under load.

@vadim: i think we need a proper patch for the others to test this ;-)

what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing 
in msrs_to_save[] in x86/x86.c of the kernel module?

thanks for you help,
peter

> --
> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 13:43, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
>> On 27.03.2012 12:40, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven wrote:
>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>        to map
> the
>>>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>         <cpu match='exact'>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>           <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>           <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>         </cpu>
>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcpu-
>>>>>>>>>>>>>>>>>> cp u.tx t.gz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
>>>>>>>>>>>>>>>>>> 	great
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production : we
>>>>>>>>>>>>>>>>>> check the response time when some other users are
>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling
>>>>>>>>>>>>>>>>> hpet and only one vcpu
>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
>>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
>>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
>>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the same
>>>>>>>>>>>>>>>>> socket. its also likely that the vm will crash when live
>>>>>>>>>>>>>>>>> migrated.
>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where we
>>>>>>>>>>>>>>>> loose performance. Since you get good result with it
>>>>>>>>>>>>>>>> frequent access to PM timer is probably the reason. I do
>>>>>>>>>>>>>>>> not recommend using -hypervisor for production!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,  at
>>>>>>>>>>>>>>> the moment, I'm only researching  this feature.
>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>> Hi Peter,
>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542633%
>>>>>>>>>> 28 v=vs .85 %29.aspx
>>>>>>>>> This one is pretty simple to support. Please see attachments for
>>>>>>>>> more details. I was thinking about synthetic  timers
>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>> To clarify the thing that microsoft qpc uses is what is implemented by
>>>>>> the patch Vadim attached to his previous email. But I believe that
>>>>>> additional qemu patch is needed for Windows to actually use it.
>>>>> You are right.
>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>> should be completely removed from ACPI.
>>>> could you advise how to do this and/or make a patch?
>>> Gleb mentioned that it properly handled in upstream,
>>> otherwise just comment the entire HPET section in
>>> acpi-dsdt.dsl file.
>> i have upstream bios installed. so -no-hpet should disable hpet completely.
>> can you give a hint, what
>> "bits 1 and 9 must be set to on in leaf 0x40000003" means?
> I mean the following code:
> +        if (hyperv_ref_counter_enabled()) {
> +            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
> +            c->eax |= 0x200;
> +        }
>
> Please see attached file for more information.
>
>>>> the stuff you send yesterday is for qemu, right? would
>>>> it be possible to use it in qemu-kvm also?
>>> Yes, but don't forget about kvm patch as well.
>> ok, i will try my best. would you consider your patch a quick hack
>> or do you think it would be worth to be uploaded to the upstream
>> repository?
> It was just a brief attempt from my side, mostly inspirited by our with Gleb
> conversation,  to see what it worth to turn this option on.
> It is not fully tested. It will crash Win8 (as well as the rest of the
> currently introduced hyper-v features).
i can confirm that windows 8 installer does not start and resets the
vm continously. it tries to access hv msr 0x40000021

http://msdn.microsoft.com/en-us/library/windows/hardware/ff542648%28v=vs.85%29.aspx

it is possible to tell the guest that the host is not iTSC (how they 
call it) capable. i will try to hack
a patch for this.

peter

> I wouldn't commit this code without comprehensive testing.
> Vadim.
>> peter
>>
>>>> peter
>>>>
>>>>>> --
>>>>>>
>>>>>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
> On 27.03.2012 13:43, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
> >> On 27.03.2012 12:40, Vadim Rozenfeld wrote:
> >>> On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
> >>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
> >>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven 
wrote:
> >>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>>>> 
> >>>>> ecrivait :
> >>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>        to map
> > 
> > the
> > 
> >>>>>>>>>>>>>>>>>> 	host cpu
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>         <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>         
> >>>>>>>>>>>>>>>>>>           <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>           <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>         
> >>>>>>>>>>>>>>>>>>         </cpu>
> >>>>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcp
> >>>>>>>>>>>>>>>>>> u- cp u.tx t.gz
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
> >>>>>>>>>>>>>>>>>> 	great
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production :
> >>>>>>>>>>>>>>>>>> we check the response time when some other users are
> >>>>>>>>>>>>>>>>>> connected.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling
> >>>>>>>>>>>>>>>>> hpet and only one vcpu
> >>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
> >>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
> >>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
> >>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the
> >>>>>>>>>>>>>>>>> same socket. its also likely that the vm will crash when
> >>>>>>>>>>>>>>>>> live migrated.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where
> >>>>>>>>>>>>>>>> we loose performance. Since you get good result with it
> >>>>>>>>>>>>>>>> frequent access to PM timer is probably the reason. I do
> >>>>>>>>>>>>>>>> not recommend using -hypervisor for production!
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>>>> timers?
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But, 
> >>>>>>>>>>>>>>> at the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> I would say weeks.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> Is there a way, we could contribute and help you with this?
> >>>>>>>>>>> 
> >>>>>>>>>>> Hi Peter,
> >>>>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>>>> 
> >>>>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff54263
> >>>>>>>>>> 3% 28 v=vs .85 %29.aspx
> >>>>>>>>> 
> >>>>>>>>> This one is pretty simple to support. Please see attachments for
> >>>>>>>>> more details. I was thinking about synthetic  timers
> >>>>>>>>> http://msdn.microsoft.com/en-
> >>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>> 
> >>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>> 
> >>>>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>>>> 
> >>>>>> To clarify the thing that microsoft qpc uses is what is implemented
> >>>>>> by the patch Vadim attached to his previous email. But I believe
> >>>>>> that additional qemu patch is needed for Windows to actually use
> >>>>>> it.
> >>>>> 
> >>>>> You are right.
> >>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>> should be completely removed from ACPI.
> >>>> 
> >>>> could you advise how to do this and/or make a patch?
> >>> 
> >>> Gleb mentioned that it properly handled in upstream,
> >>> otherwise just comment the entire HPET section in
> >>> acpi-dsdt.dsl file.
> >> 
> >> i have upstream bios installed. so -no-hpet should disable hpet
> >> completely. can you give a hint, what
> >> "bits 1 and 9 must be set to on in leaf 0x40000003" means?
> > 
> > I mean the following code:
> > +        if (hyperv_ref_counter_enabled()) {
> > +            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
> > +            c->eax |= 0x200;
> > +        }
> > 
> > Please see attached file for more information.
> > 
> >>>> the stuff you send yesterday is for qemu, right? would
> >>>> it be possible to use it in qemu-kvm also?
> >>> 
> >>> Yes, but don't forget about kvm patch as well.
> >> 
> >> ok, i will try my best. would you consider your patch a quick hack
> >> or do you think it would be worth to be uploaded to the upstream
> >> repository?
> > 
> > It was just a brief attempt from my side, mostly inspirited by our with
> > Gleb conversation,  to see what it worth to turn this option on.
> > It is not fully tested. It will crash Win8 (as well as the rest of the
> > currently introduced hyper-v features).
> 
> i can confirm that windows 8 installer does not start and resets the
> vm continously. it tries to access hv msr 0x40000021
> 
Win8 needs more comprehensive Hyper-V support.  
> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542648%28v=vs.85
> %29.aspx
> 
> it is possible to tell the guest that the host is not iTSC (how they
> call it) capable. i will try to hack
> a patch for this.
> 
> peter
> 
> > I wouldn't commit this code without comprehensive testing.
> > Vadim.
> > 
> >> peter
> >> 
> >>>> peter
> >>>> 
> >>>>>> --
> >>>>>> 
> >>>>>> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 17:37, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
>> On 27.03.2012 13:43, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
>>>> On 27.03.2012 12:40, Vadim Rozenfeld wrote:
>>>>> On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov wrote:
>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
> wrote:
>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>> to cpu definition in XML and check command line.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         to map
>>> the
>>>
>>>>>>>>>>>>>>>>>>>> 	host cpu
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>          <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>            <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>            <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>          </cpu>
>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1vcp
>>>>>>>>>>>>>>>>>>>> u- cp u.tx t.gz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
>>>>>>>>>>>>>>>>>>>> 	great
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production :
>>>>>>>>>>>>>>>>>>>> we check the response time when some other users are
>>>>>>>>>>>>>>>>>>>> connected.
>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor, disabling
>>>>>>>>>>>>>>>>>>> hpet and only one vcpu
>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
>>>>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
>>>>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
>>>>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the
>>>>>>>>>>>>>>>>>>> same socket. its also likely that the vm will crash when
>>>>>>>>>>>>>>>>>>> live migrated.
>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify where
>>>>>>>>>>>>>>>>>> we loose performance. Since you get good result with it
>>>>>>>>>>>>>>>>>> frequent access to PM timer is probably the reason. I do
>>>>>>>>>>>>>>>>>> not recommend using -hypervisor for production!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,
>>>>>>>>>>>>>>>>> at the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff54263
>>>>>>>>>>>> 3% 28 v=vs .85 %29.aspx
>>>>>>>>>>> This one is pretty simple to support. Please see attachments for
>>>>>>>>>>> more details. I was thinking about synthetic  timers
>>>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>> To clarify the thing that microsoft qpc uses is what is implemented
>>>>>>>> by the patch Vadim attached to his previous email. But I believe
>>>>>>>> that additional qemu patch is needed for Windows to actually use
>>>>>>>> it.
>>>>>>> You are right.
>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>> should be completely removed from ACPI.
>>>>>> could you advise how to do this and/or make a patch?
>>>>> Gleb mentioned that it properly handled in upstream,
>>>>> otherwise just comment the entire HPET section in
>>>>> acpi-dsdt.dsl file.
>>>> i have upstream bios installed. so -no-hpet should disable hpet
>>>> completely. can you give a hint, what
>>>> "bits 1 and 9 must be set to on in leaf 0x40000003" means?
>>> I mean the following code:
>>> +        if (hyperv_ref_counter_enabled()) {
>>> +            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
>>> +            c->eax |= 0x200;
>>> +        }
>>>
>>> Please see attached file for more information.
>>>
>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>> it be possible to use it in qemu-kvm also?
>>>>> Yes, but don't forget about kvm patch as well.
>>>> ok, i will try my best. would you consider your patch a quick hack
>>>> or do you think it would be worth to be uploaded to the upstream
>>>> repository?
>>> It was just a brief attempt from my side, mostly inspirited by our with
>>> Gleb conversation,  to see what it worth to turn this option on.
>>> It is not fully tested. It will crash Win8 (as well as the rest of the
>>> currently introduced hyper-v features).
>> i can confirm that windows 8 installer does not start and resets the
>> vm continously. it tries to access hv msr 0x40000021
>>
> Win8 needs more comprehensive Hyper-V support.
yes it seems. i read your comment wrong. i was believing the
hv_refcnt breaks the other hv_features and windows 8, but
i guess you said any of the hv_features will break win 8?!

peter

>
>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542648%28v=vs.85
>> %29.aspx
>>
>> it is possible to tell the guest that the host is not iTSC (how they
>> call it) capable. i will try to hack
>> a patch for this.
>>
>> peter
>>
>>> I wouldn't commit this code without comprehensive testing.
>>> Vadim.
>>>
>>>> peter
>>>>
>>>>>> peter
>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
> On 27.03.2012 14:29, Gleb Natapov wrote:
> > On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
> >> On 27.03.2012 14:26, Gleb Natapov wrote:
> >>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
> >>>> On 27.03.2012 12:00, Gleb Natapov wrote:
> >>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> >>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld 
wrote:
> >>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov 
wrote:
> >>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven 
wrote:
> >>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>>>>>> 
> >>>>>>> ecrivait :
> >>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>> name='hypervisor'/> to cpu definition in XML and
> >>>>>>>>>>>>>>>>>>>>> check command line.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         to
> >>>>>>>>>>>>>>>>>>>> 	map the host cpu
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>>>       <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>>>       
> >>>>>>>>>>>>>>>>>>>>         <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>>>         <feature policy='disable' name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>>>       
> >>>>>>>>>>>>>>>>>>>>       </cpu>
> >>>>>>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1v
> >>>>>>>>>>>>>>>>>>>> cpu-cp u.tx t.gz
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
> >>>>>>>>>>>>>>>>>>>> 	great
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production
> >>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
> >>>>>>>>>>>>>>>>>>>> are connected.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
> >>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
> >>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
> >>>>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
> >>>>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
> >>>>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the
> >>>>>>>>>>>>>>>>>>> same socket. its also likely that the vm will crash
> >>>>>>>>>>>>>>>>>>> when live migrated.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
> >>>>>>>>>>>>>>>>>> where we loose performance. Since you get good result
> >>>>>>>>>>>>>>>>>> with it frequent access to PM timer is probably the
> >>>>>>>>>>>>>>>>>> reason. I do not recommend using -hypervisor for
> >>>>>>>>>>>>>>>>>> production!
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>>>>>> timers?
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,
> >>>>>>>>>>>>>>>>>  at the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> I would say weeks.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi Peter,
> >>>>>>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542
> >>>>>>>>>>>> 633%28 v=vs .85 %29.aspx
> >>>>>>>>>>> 
> >>>>>>>>>>> This one is pretty simple to support. Please see attachments
> >>>>>>>>>>> for more details. I was thinking about synthetic  timers
> >>>>>>>>>>> http://msdn.microsoft.com/en-
> >>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>>>> 
> >>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>>>> 
> >>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>>>>>> 
> >>>>>>>> To clarify the thing that microsoft qpc uses is what is
> >>>>>>>> implemented by the patch Vadim attached to his previous email.
> >>>>>>>> But I believe that additional qemu patch is needed for Windows to
> >>>>>>>> actually use it.
> >>>>>>> 
> >>>>>>> You are right.
> >>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>>>> should be completely removed from ACPI.
> >>>>>> 
> >>>>>> could you advise how to do this and/or make a patch?
> >>>>>> 
> >>>>>> the stuff you send yesterday is for qemu, right? would
> >>>>>> it be possible to use it in qemu-kvm also?
> >>>>> 
> >>>>> No, they are for kernel.
> >>>> 
> >>>> i meant the qemu.diff file.
> >>> 
> >>> Yes, I missed the second attachment.
> >>> 
> >>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
> >>> 
> >>> Looks like it.
> >> 
> >> ok, so it would be interesting if it helps to avoid the pmtimer reads
> >> we observed earlier. right?
> > 
> > Yes.
> 
> first feedback: performance seems to be amazing. i cannot confirm that
> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
> why did you assume this?
I didn't mean that hv_refcnt will break any other hyper-v features.
I just want to say that turning hv_refcnt on (as any other hv_ option) will
crash Win8 on boot-up.

Cheers,
Vadim.
 
> 
> no more pmtimer reads. i can now almost fully utililizy a 1GBit
> interface with a file transfer while there was not one
> cpu core fully utilized as observed with pmtimer. some live migration
> tests revealed that it did not crash even under load.
> 
> @vadim: i think we need a proper patch for the others to test this ;-)
> 
> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing
> in msrs_to_save[] in x86/x86.c of the kernel module?
> 
> thanks for you help,
> peter
> 
> > --
> > 
> > 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 05:39:10 PM Peter Lieven wrote:
> On 27.03.2012 17:37, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 04:44:51 PM Peter Lieven wrote:
> >> On 27.03.2012 13:43, Vadim Rozenfeld wrote:
> >>> On Tuesday, March 27, 2012 12:49:58 PM Peter Lieven wrote:
> >>>> On 27.03.2012 12:40, Vadim Rozenfeld wrote:
> >>>>> On Tuesday, March 27, 2012 11:26:29 AM Peter Lieven wrote:
> >>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld 
wrote:
> >>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
> >>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov 
wrote:
> >>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
> > 
> > wrote:
> >>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
> >>>>>>> 
> >>>>>>> ecrivait :
> >>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>> name='hypervisor'/> to cpu definition in XML and
> >>>>>>>>>>>>>>>>>>>>> check command line.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         to
> >>>>>>>>>>>>>>>>>>>> 	map
> >>> 
> >>> the
> >>> 
> >>>>>>>>>>>>>>>>>>>> 	host cpu
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>>>          <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>>>          
> >>>>>>>>>>>>>>>>>>>>            <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>>>            <feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>            name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>>>          
> >>>>>>>>>>>>>>>>>>>>          </cpu>
> >>>>>>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1v
> >>>>>>>>>>>>>>>>>>>> cp u- cp u.tx t.gz
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
> >>>>>>>>>>>>>>>>>>>> 	great
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production
> >>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
> >>>>>>>>>>>>>>>>>>>> are connected.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
> >>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
> >>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
> >>>>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
> >>>>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
> >>>>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the
> >>>>>>>>>>>>>>>>>>> same socket. its also likely that the vm will crash
> >>>>>>>>>>>>>>>>>>> when live migrated.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
> >>>>>>>>>>>>>>>>>> where we loose performance. Since you get good result
> >>>>>>>>>>>>>>>>>> with it frequent access to PM timer is probably the
> >>>>>>>>>>>>>>>>>> reason. I do not recommend using -hypervisor for
> >>>>>>>>>>>>>>>>>> production!
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
> >>>>>>>>>>>>>>>>>>> timers?
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,
> >>>>>>>>>>>>>>>>> at the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> I would say weeks.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Hi Peter,
> >>>>>>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542
> >>>>>>>>>>>> 63 3% 28 v=vs .85 %29.aspx
> >>>>>>>>>>> 
> >>>>>>>>>>> This one is pretty simple to support. Please see attachments
> >>>>>>>>>>> for more details. I was thinking about synthetic  timers
> >>>>>>>>>>> http://msdn.microsoft.com/en-
> >>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>>>> 
> >>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>>>> 
> >>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>>>>>> 
> >>>>>>>> To clarify the thing that microsoft qpc uses is what is
> >>>>>>>> implemented by the patch Vadim attached to his previous email.
> >>>>>>>> But I believe that additional qemu patch is needed for Windows to
> >>>>>>>> actually use it.
> >>>>>>> 
> >>>>>>> You are right.
> >>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>>>> should be completely removed from ACPI.
> >>>>>> 
> >>>>>> could you advise how to do this and/or make a patch?
> >>>>> 
> >>>>> Gleb mentioned that it properly handled in upstream,
> >>>>> otherwise just comment the entire HPET section in
> >>>>> acpi-dsdt.dsl file.
> >>>> 
> >>>> i have upstream bios installed. so -no-hpet should disable hpet
> >>>> completely. can you give a hint, what
> >>>> "bits 1 and 9 must be set to on in leaf 0x40000003" means?
> >>> 
> >>> I mean the following code:
> >>> +        if (hyperv_ref_counter_enabled()) {
> >>> +            c->eax |= HV_X64_MSR_TIME_REF_COUNT_AVAILABLE;
> >>> +            c->eax |= 0x200;
> >>> +        }
> >>> 
> >>> Please see attached file for more information.
> >>> 
> >>>>>> the stuff you send yesterday is for qemu, right? would
> >>>>>> it be possible to use it in qemu-kvm also?
> >>>>> 
> >>>>> Yes, but don't forget about kvm patch as well.
> >>>> 
> >>>> ok, i will try my best. would you consider your patch a quick hack
> >>>> or do you think it would be worth to be uploaded to the upstream
> >>>> repository?
> >>> 
> >>> It was just a brief attempt from my side, mostly inspirited by our with
> >>> Gleb conversation,  to see what it worth to turn this option on.
> >>> It is not fully tested. It will crash Win8 (as well as the rest of the
> >>> currently introduced hyper-v features).
> >> 
> >> i can confirm that windows 8 installer does not start and resets the
> >> vm continously. it tries to access hv msr 0x40000021
> > 
> > Win8 needs more comprehensive Hyper-V support.
> 
> yes it seems. i read your comment wrong. i was believing the
> hv_refcnt breaks the other hv_features and windows 8, but
> i guess you said any of the hv_features will break win 8?!
> 

Right. 
Seems like Win8 needs synthetic interrupts supported.


> peter
> 
> >> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542648%28v=vs
> >> .85 %29.aspx
> >> 
> >> it is possible to tell the guest that the host is not iTSC (how they
> >> call it) capable. i will try to hack
> >> a patch for this.
> >> 
> >> peter
> >> 
> >>> I wouldn't commit this code without comprehensive testing.
> >>> Vadim.
> >>> 
> >>>> peter
> >>>> 
> >>>>>> peter
> >>>>>> 
> >>>>>>>> --
> >>>>>>>> 
> >>>>>>>> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 17:44, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
>> On 27.03.2012 14:29, Gleb Natapov wrote:
>>> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
>>>> On 27.03.2012 14:26, Gleb Natapov wrote:
>>>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>>>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
> wrote:
>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven wrote:
>>>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
> wrote:
>>>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
> wrote:
>>>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb Natapov
>>>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
>>>>>>>>>>>>>>>>>>>>>>> name='hypervisor'/>  to cpu definition in XML and
>>>>>>>>>>>>>>>>>>>>>>> check command line.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>          to
>>>>>>>>>>>>>>>>>>>>>> 	map the host cpu
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>>>        <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>          <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>>>          <feature policy='disable' name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>        </cpu>
>>>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-1v
>>>>>>>>>>>>>>>>>>>>>> cpu-cp u.tx t.gz
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is 8.5s,
>>>>>>>>>>>>>>>>>>>>>> 	great
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for production
>>>>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
>>>>>>>>>>>>>>>>>>>>>> are connected.
>>>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
>>>>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
>>>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to make
>>>>>>>>>>>>>>>>>>>>> sure, that your vm is not switching between physical
>>>>>>>>>>>>>>>>>>>>> sockets on your system and that you have constant_tsc
>>>>>>>>>>>>>>>>>>>>> feature to have a stable tsc between the cores in the
>>>>>>>>>>>>>>>>>>>>> same socket. its also likely that the vm will crash
>>>>>>>>>>>>>>>>>>>>> when live migrated.
>>>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
>>>>>>>>>>>>>>>>>>>> where we loose performance. Since you get good result
>>>>>>>>>>>>>>>>>>>> with it frequent access to PM timer is probably the
>>>>>>>>>>>>>>>>>>>> reason. I do not recommend using -hypervisor for
>>>>>>>>>>>>>>>>>>>> production!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel hyper-v
>>>>>>>>>>>>>>>>>>>>> timers?
>>>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported. But,
>>>>>>>>>>>>>>>>>>>   at the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>>>> Is there a way, we could contribute and help you with this?
>>>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff542
>>>>>>>>>>>>>> 633%28 v=vs .85 %29.aspx
>>>>>>>>>>>>> This one is pretty simple to support. Please see attachments
>>>>>>>>>>>>> for more details. I was thinking about synthetic  timers
>>>>>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>>>> To clarify the thing that microsoft qpc uses is what is
>>>>>>>>>> implemented by the patch Vadim attached to his previous email.
>>>>>>>>>> But I believe that additional qemu patch is needed for Windows to
>>>>>>>>>> actually use it.
>>>>>>>>> You are right.
>>>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>>>> should be completely removed from ACPI.
>>>>>>>> could you advise how to do this and/or make a patch?
>>>>>>>>
>>>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>>>> it be possible to use it in qemu-kvm also?
>>>>>>> No, they are for kernel.
>>>>>> i meant the qemu.diff file.
>>>>> Yes, I missed the second attachment.
>>>>>
>>>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to qemu?
>>>>> Looks like it.
>>>> ok, so it would be interesting if it helps to avoid the pmtimer reads
>>>> we observed earlier. right?
>>> Yes.
>> first feedback: performance seems to be amazing. i cannot confirm that
>> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
>> why did you assume this?
> I didn't mean that hv_refcnt will break any other hyper-v features.
> I just want to say that turning hv_refcnt on (as any other hv_ option) will
> crash Win8 on boot-up.
yes, i got it meanwhile ;-)

let me know what you think should be done to further test
the refcnt implementation.

i would suggest to return at least 0xFFFFFFFF if msr 0x40000021
is read.

peter
> Cheers,
> Vadim.
>
>> no more pmtimer reads. i can now almost fully utililizy a 1GBit
>> interface with a file transfer while there was not one
>> cpu core fully utilized as observed with pmtimer. some live migration
>> tests revealed that it did not crash even under load.
>>
>> @vadim: i think we need a proper patch for the others to test this ;-)
>>
>> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing
>> in msrs_to_save[] in x86/x86.c of the kernel module?
>>
>> thanks for you help,
>> peter
>>
>>> --
>>>
>>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote:
> On 27.03.2012 17:44, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
> >> On 27.03.2012 14:29, Gleb Natapov wrote:
> >>> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
> >>>> On 27.03.2012 14:26, Gleb Natapov wrote:
> >>>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
> >>>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
> >>>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> >>>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
> > 
> > wrote:
> >>>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven 
wrote:
> >>>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
> > 
> > wrote:
> >>>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
> > 
> > wrote:
> >>>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb
> >>>>>>>>>>>>>>>>>>>>>> Natapov
> >>>>>>>>> 
> >>>>>>>>> ecrivait :
> >>>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>>>> name='hypervisor'/>  to cpu definition in XML and
> >>>>>>>>>>>>>>>>>>>>>>> check command line.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>         
> >>>>>>>>>>>>>>>>>>>>>> 	to map the host cpu
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>>>>>        <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>>>>          <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>>>>>          <feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>>>          name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>>>>>        
> >>>>>>>>>>>>>>>>>>>>>>        </cpu>
> >>>>>>>>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-
> >>>>>>>>>>>>>>>>>>>>>> 1v cpu-cp u.tx t.gz
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is
> >>>>>>>>>>>>>>>>>>>>>> 	8.5s, great
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for
> >>>>>>>>>>>>>>>>>>>>>> production
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> are connected.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
> >>>>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
> >>>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to
> >>>>>>>>>>>>>>>>>>>>> make sure, that your vm is not switching between
> >>>>>>>>>>>>>>>>>>>>> physical sockets on your system and that you have
> >>>>>>>>>>>>>>>>>>>>> constant_tsc feature to have a stable tsc between
> >>>>>>>>>>>>>>>>>>>>> the cores in the same socket. its also likely that
> >>>>>>>>>>>>>>>>>>>>> the vm will crash when live migrated.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
> >>>>>>>>>>>>>>>>>>>> where we loose performance. Since you get good result
> >>>>>>>>>>>>>>>>>>>> with it frequent access to PM timer is probably the
> >>>>>>>>>>>>>>>>>>>> reason. I do not recommend using -hypervisor for
> >>>>>>>>>>>>>>>>>>>> production!
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel
> >>>>>>>>>>>>>>>>>>>>> hyper-v timers?
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported.
> >>>>>>>>>>>>>>>>>>> But,
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>   at the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> I would say weeks.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> Is there a way, we could contribute and help you with
> >>>>>>>>>>>>>>>> this?
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> Hi Peter,
> >>>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff5
> >>>>>>>>>>>>>> 42 633%28 v=vs .85 %29.aspx
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> This one is pretty simple to support. Please see attachments
> >>>>>>>>>>>>> for more details. I was thinking about synthetic  timers
> >>>>>>>>>>>>> http://msdn.microsoft.com/en-
> >>>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>>>>>> 
> >>>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>>>>>> 
> >>>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>>>>>>>> 
> >>>>>>>>>> To clarify the thing that microsoft qpc uses is what is
> >>>>>>>>>> implemented by the patch Vadim attached to his previous email.
> >>>>>>>>>> But I believe that additional qemu patch is needed for Windows
> >>>>>>>>>> to actually use it.
> >>>>>>>>> 
> >>>>>>>>> You are right.
> >>>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>>>>>> should be completely removed from ACPI.
> >>>>>>>> 
> >>>>>>>> could you advise how to do this and/or make a patch?
> >>>>>>>> 
> >>>>>>>> the stuff you send yesterday is for qemu, right? would
> >>>>>>>> it be possible to use it in qemu-kvm also?
> >>>>>>> 
> >>>>>>> No, they are for kernel.
> >>>>>> 
> >>>>>> i meant the qemu.diff file.
> >>>>> 
> >>>>> Yes, I missed the second attachment.
> >>>>> 
> >>>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to
> >>>>>> qemu?
> >>>>> 
> >>>>> Looks like it.
> >>>> 
> >>>> ok, so it would be interesting if it helps to avoid the pmtimer reads
> >>>> we observed earlier. right?
> >>> 
> >>> Yes.
> >> 
> >> first feedback: performance seems to be amazing. i cannot confirm that
> >> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
> >> why did you assume this?
> > 
> > I didn't mean that hv_refcnt will break any other hyper-v features.
> > I just want to say that turning hv_refcnt on (as any other hv_ option)
> > will crash Win8 on boot-up.
> 
> yes, i got it meanwhile ;-)
> 
> let me know what you think should be done to further test
> the refcnt implementation.
> 
> i would suggest to return at least 0xFFFFFFFF if msr 0x40000021
> is read.
IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes.
> 
> peter
> 
> > Cheers,
> > Vadim.
> > 
> >> no more pmtimer reads. i can now almost fully utililizy a 1GBit
> >> interface with a file transfer while there was not one
> >> cpu core fully utilized as observed with pmtimer. some live migration
> >> tests revealed that it did not crash even under load.
> >> 
> >> @vadim: i think we need a proper patch for the others to test this ;-)
> >> 
> >> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing
> >> in msrs_to_save[] in x86/x86.c of the kernel module?
> >> 
> >> thanks for you help,
> >> peter
> >> 
> >>> --
> >>> 
> >>> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 18:12, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote:
>> On 27.03.2012 17:44, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
>>>> On 27.03.2012 14:29, Gleb Natapov wrote:
>>>>> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
>>>>>> On 27.03.2012 14:26, Gleb Natapov wrote:
>>>>>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>>>>>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>>>>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>>>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
>>> wrote:
>>>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven
> wrote:
>>>>>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter Lieven
>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb
>>>>>>>>>>>>>>>>>>>>>>>> Natapov
>>>>>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
>>>>>>>>>>>>>>>>>>>>>>>>> name='hypervisor'/>   to cpu definition in XML and
>>>>>>>>>>>>>>>>>>>>>>>>> check command line.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>
>>>>>>>>>>>>>>>>>>>>>>>> 	to map the host cpu
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>>>>>         <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>           <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>>>>>           <feature policy='disable'
>>>>>>>>>>>>>>>>>>>>>>>>           name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>         </cpu>
>>>>>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-net-
>>>>>>>>>>>>>>>>>>>>>>>> 1v cpu-cp u.tx t.gz
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is
>>>>>>>>>>>>>>>>>>>>>>>> 	8.5s, great
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for
>>>>>>>>>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> are connected.
>>>>>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
>>>>>>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
>>>>>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to
>>>>>>>>>>>>>>>>>>>>>>> make sure, that your vm is not switching between
>>>>>>>>>>>>>>>>>>>>>>> physical sockets on your system and that you have
>>>>>>>>>>>>>>>>>>>>>>> constant_tsc feature to have a stable tsc between
>>>>>>>>>>>>>>>>>>>>>>> the cores in the same socket. its also likely that
>>>>>>>>>>>>>>>>>>>>>>> the vm will crash when live migrated.
>>>>>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
>>>>>>>>>>>>>>>>>>>>>> where we loose performance. Since you get good result
>>>>>>>>>>>>>>>>>>>>>> with it frequent access to PM timer is probably the
>>>>>>>>>>>>>>>>>>>>>> reason. I do not recommend using -hypervisor for
>>>>>>>>>>>>>>>>>>>>>> production!
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel
>>>>>>>>>>>>>>>>>>>>>>> hyper-v timers?
>>>>>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported.
>>>>>>>>>>>>>>>>>>>>> But,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>    at the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>>>>>> Is there a way, we could contribute and help you with
>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/ff5
>>>>>>>>>>>>>>>> 42 633%28 v=vs .85 %29.aspx
>>>>>>>>>>>>>>> This one is pretty simple to support. Please see attachments
>>>>>>>>>>>>>>> for more details. I was thinking about synthetic  timers
>>>>>>>>>>>>>>> http://msdn.microsoft.com/en-
>>>>>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>>>>>> To clarify the thing that microsoft qpc uses is what is
>>>>>>>>>>>> implemented by the patch Vadim attached to his previous email.
>>>>>>>>>>>> But I believe that additional qemu patch is needed for Windows
>>>>>>>>>>>> to actually use it.
>>>>>>>>>>> You are right.
>>>>>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>>>>>> should be completely removed from ACPI.
>>>>>>>>>> could you advise how to do this and/or make a patch?
>>>>>>>>>>
>>>>>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>>>>>> it be possible to use it in qemu-kvm also?
>>>>>>>>> No, they are for kernel.
>>>>>>>> i meant the qemu.diff file.
>>>>>>> Yes, I missed the second attachment.
>>>>>>>
>>>>>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to
>>>>>>>> qemu?
>>>>>>> Looks like it.
>>>>>> ok, so it would be interesting if it helps to avoid the pmtimer reads
>>>>>> we observed earlier. right?
>>>>> Yes.
>>>> first feedback: performance seems to be amazing. i cannot confirm that
>>>> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
>>>> why did you assume this?
>>> I didn't mean that hv_refcnt will break any other hyper-v features.
>>> I just want to say that turning hv_refcnt on (as any other hv_ option)
>>> will crash Win8 on boot-up.
>> yes, i got it meanwhile ;-)
>>
>> let me know what you think should be done to further test
>> the refcnt implementation.
>>
>> i would suggest to return at least 0xFFFFFFFF if msr 0x40000021
>> is read.
> IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes.
you mean win7 only writes, don't you?
at least you put a break in set_msr_hyperv for this msr.

i just thought that it would be ok to return the value that
is defined for iTSC is not supported?

peter


>> peter
>>
>>> Cheers,
>>> Vadim.
>>>
>>>> no more pmtimer reads. i can now almost fully utililizy a 1GBit
>>>> interface with a file transfer while there was not one
>>>> cpu core fully utilized as observed with pmtimer. some live migration
>>>> tests revealed that it did not crash even under load.
>>>>
>>>> @vadim: i think we need a proper patch for the others to test this ;-)
>>>>
>>>> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is missing
>>>> in msrs_to_save[] in x86/x86.c of the kernel module?
>>>>
>>>> thanks for you help,
>>>> peter
>>>>
>>>>> --
>>>>>
>>>>> 			Gleb.

Re: performance trouble

[Vadim Rozenfeld]

On Tuesday, March 27, 2012 06:16:11 PM Peter Lieven wrote:
> On 27.03.2012 18:12, Vadim Rozenfeld wrote:
> > On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote:
> >> On 27.03.2012 17:44, Vadim Rozenfeld wrote:
> >>> On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
> >>>> On 27.03.2012 14:29, Gleb Natapov wrote:
> >>>>> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
> >>>>>> On 27.03.2012 14:26, Gleb Natapov wrote:
> >>>>>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
> >>>>>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
> >>>>>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
> >>>>>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
> >>>>>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
> >>>>>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld 
wrote:
> >>>>>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
> >>>>>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
> >>>>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
> >>> 
> >>> wrote:
> >>>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
> >>>>>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven
> > 
> > wrote:
> >>>>>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
> >>>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
> >>> 
> >>> wrote:
> >>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter
> >>>>>>>>>>>>>>>>>>>>>> Lieven
> >>> 
> >>> wrote:
> >>>>>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
> >>>>>>>>>>>>>>>>>>>>>>>> 		hello,
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb
> >>>>>>>>>>>>>>>>>>>>>>>> Natapov
> >>>>>>>>>>> 
> >>>>>>>>>>> ecrivait :
> >>>>>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>>>>>> name='hypervisor'/>   to cpu definition in XML
> >>>>>>>>>>>>>>>>>>>>>>>>> and check command line.
> >>>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>
> >>>>>>>>>>>>>>>>>>>>>>>> 	to map the host cpu
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
> >>>>>>>>>>>>>>>>>>>>>>>>         <cpu match='exact'>
> >>>>>>>>>>>>>>>>>>>>>>>>         
> >>>>>>>>>>>>>>>>>>>>>>>>           <model>Opteron_G3</model>
> >>>>>>>>>>>>>>>>>>>>>>>>           <feature policy='disable'
> >>>>>>>>>>>>>>>>>>>>>>>>           name='hypervisor'/>
> >>>>>>>>>>>>>>>>>>>>>>>>         
> >>>>>>>>>>>>>>>>>>>>>>>>         </cpu>
> >>>>>>>>>>>>>>>>>>>>>>>> 	
> >>>>>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> 	The log is here :
> >>>>>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-ne
> >>>>>>>>>>>>>>>>>>>>>>>> t- 1v cpu-cp u.tx t.gz
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is
> >>>>>>>>>>>>>>>>>>>>>>>> 	8.5s, great
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for
> >>>>>>>>>>>>>>>>>>>>>>>> production
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
> >>>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>>> are connected.
> >>>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
> >>>>>>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
> >>>>>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to
> >>>>>>>>>>>>>>>>>>>>>>> make sure, that your vm is not switching between
> >>>>>>>>>>>>>>>>>>>>>>> physical sockets on your system and that you have
> >>>>>>>>>>>>>>>>>>>>>>> constant_tsc feature to have a stable tsc between
> >>>>>>>>>>>>>>>>>>>>>>> the cores in the same socket. its also likely that
> >>>>>>>>>>>>>>>>>>>>>>> the vm will crash when live migrated.
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
> >>>>>>>>>>>>>>>>>>>>>> where we loose performance. Since you get good
> >>>>>>>>>>>>>>>>>>>>>> result with it frequent access to PM timer is
> >>>>>>>>>>>>>>>>>>>>>> probably the reason. I do not recommend using
> >>>>>>>>>>>>>>>>>>>>>> -hypervisor for production!
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel
> >>>>>>>>>>>>>>>>>>>>>>> hyper-v timers?
> >>>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported.
> >>>>>>>>>>>>>>>>>>>>> But,
> >>>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>>>    at the moment, I'm only researching  this feature.
> >>>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>>> So it will take months at least?
> >>>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>>> I would say weeks.
> >>>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>>> Is there a way, we could contribute and help you with
> >>>>>>>>>>>>>>>>>> this?
> >>>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>>> Hi Peter,
> >>>>>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
> >>>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>>> I think Vadim refers to this HV MSR
> >>>>>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/f
> >>>>>>>>>>>>>>>> f5 42 633%28 v=vs .85 %29.aspx
> >>>>>>>>>>>>>>> 
> >>>>>>>>>>>>>>> This one is pretty simple to support. Please see
> >>>>>>>>>>>>>>> attachments for more details. I was thinking about
> >>>>>>>>>>>>>>> synthetic  timers http://msdn.microsoft.com/en-
> >>>>>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
> >>>>>>>>>>>>>> 
> >>>>>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
> >>>>>>>>>>>>> 
> >>>>>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> To clarify the thing that microsoft qpc uses is what is
> >>>>>>>>>>>> implemented by the patch Vadim attached to his previous email.
> >>>>>>>>>>>> But I believe that additional qemu patch is needed for Windows
> >>>>>>>>>>>> to actually use it.
> >>>>>>>>>>> 
> >>>>>>>>>>> You are right.
> >>>>>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
> >>>>>>>>>>> should be completely removed from ACPI.
> >>>>>>>>>> 
> >>>>>>>>>> could you advise how to do this and/or make a patch?
> >>>>>>>>>> 
> >>>>>>>>>> the stuff you send yesterday is for qemu, right? would
> >>>>>>>>>> it be possible to use it in qemu-kvm also?
> >>>>>>>>> 
> >>>>>>>>> No, they are for kernel.
> >>>>>>>> 
> >>>>>>>> i meant the qemu.diff file.
> >>>>>>> 
> >>>>>>> Yes, I missed the second attachment.
> >>>>>>> 
> >>>>>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to
> >>>>>>>> qemu?
> >>>>>>> 
> >>>>>>> Looks like it.
> >>>>>> 
> >>>>>> ok, so it would be interesting if it helps to avoid the pmtimer
> >>>>>> reads we observed earlier. right?
> >>>>> 
> >>>>> Yes.
> >>>> 
> >>>> first feedback: performance seems to be amazing. i cannot confirm that
> >>>> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
> >>>> why did you assume this?
> >>> 
> >>> I didn't mean that hv_refcnt will break any other hyper-v features.
> >>> I just want to say that turning hv_refcnt on (as any other hv_ option)
> >>> will crash Win8 on boot-up.
> >> 
> >> yes, i got it meanwhile ;-)
> >> 
> >> let me know what you think should be done to further test
> >> the refcnt implementation.
> >> 
> >> i would suggest to return at least 0xFFFFFFFF if msr 0x40000021
> >> is read.
> > 
> > IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes.
> 
> you mean win7 only writes, don't you?
Oh, yes. It only writes. 
Actually it works this way: kernel allocates one page, maps it into the system 
space and writes the its address to  
0x40000021 MSR. But to make guest accessing this page, the partition 
reference tsc facility must be enabled, otherwise you can keep any garbage
there without breaking your guest.
  
> at least you put a break in set_msr_hyperv for this msr.
> 
> i just thought that it would be ok to return the value that
> is defined for iTSC is not supported?
> 
> peter
> 
> >> peter
> >> 
> >>> Cheers,
> >>> Vadim.
> >>> 
> >>>> no more pmtimer reads. i can now almost fully utililizy a 1GBit
> >>>> interface with a file transfer while there was not one
> >>>> cpu core fully utilized as observed with pmtimer. some live migration
> >>>> tests revealed that it did not crash even under load.
> >>>> 
> >>>> @vadim: i think we need a proper patch for the others to test this ;-)
> >>>> 
> >>>> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is
> >>>> missing in msrs_to_save[] in x86/x86.c of the kernel module?
> >>>> 
> >>>> thanks for you help,
> >>>> peter
> >>>> 
> >>>>> --
> >>>>> 
> >>>>> 			Gleb.

Re: performance trouble

[Peter Lieven]

On 27.03.2012 19:06, Vadim Rozenfeld wrote:
> On Tuesday, March 27, 2012 06:16:11 PM Peter Lieven wrote:
>> On 27.03.2012 18:12, Vadim Rozenfeld wrote:
>>> On Tuesday, March 27, 2012 05:58:01 PM Peter Lieven wrote:
>>>> On 27.03.2012 17:44, Vadim Rozenfeld wrote:
>>>>> On Tuesday, March 27, 2012 04:06:13 PM Peter Lieven wrote:
>>>>>> On 27.03.2012 14:29, Gleb Natapov wrote:
>>>>>>> On Tue, Mar 27, 2012 at 02:28:04PM +0200, Peter Lieven wrote:
>>>>>>>> On 27.03.2012 14:26, Gleb Natapov wrote:
>>>>>>>>> On Tue, Mar 27, 2012 at 02:20:23PM +0200, Peter Lieven wrote:
>>>>>>>>>> On 27.03.2012 12:00, Gleb Natapov wrote:
>>>>>>>>>>> On Tue, Mar 27, 2012 at 11:26:29AM +0200, Peter Lieven wrote:
>>>>>>>>>>>> On 27.03.2012 11:23, Vadim Rozenfeld wrote:
>>>>>>>>>>>>> On Tuesday, March 27, 2012 10:56:05 AM Gleb Natapov wrote:
>>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 10:11:43PM +0200, Vadim Rozenfeld
> wrote:
>>>>>>>>>>>>>>> On Monday, March 26, 2012 08:54:50 PM Peter Lieven wrote:
>>>>>>>>>>>>>>>> On 26.03.2012 20:36, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:52:49 PM Gleb Natapov wrote:
>>>>>>>>>>>>>>>>>> On Mon, Mar 26, 2012 at 07:46:03PM +0200, Vadim Rozenfeld
>>>>> wrote:
>>>>>>>>>>>>>>>>>>> On Monday, March 26, 2012 07:00:32 PM Peter Lieven wrote:
>>>>>>>>>>>>>>>>>>>> On 22.03.2012 10:38, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 10:52:42 AM Peter Lieven
>>> wrote:
>>>>>>>>>>>>>>>>>>>>>> On 22.03.2012 09:48, Vadim Rozenfeld wrote:
>>>>>>>>>>>>>>>>>>>>>>> On Thursday, March 22, 2012 09:53:45 AM Gleb Natapov
>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>> On Wed, Mar 21, 2012 at 06:31:02PM +0100, Peter
>>>>>>>>>>>>>>>>>>>>>>>> Lieven
>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>> On 21.03.2012 12:10, David Cure wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>> 		hello,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Le Tue, Mar 20, 2012 at 02:38:22PM +0200, Gleb
>>>>>>>>>>>>>>>>>>>>>>>>>> Natapov
>>>>>>>>>>>>> ecrivait :
>>>>>>>>>>>>>>>>>>>>>>>>>>> Try to add<feature policy='disable'
>>>>>>>>>>>>>>>>>>>>>>>>>>> name='hypervisor'/>    to cpu definition in XML
>>>>>>>>>>>>>>>>>>>>>>>>>>> and check command line.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 	ok I try this but I can't use<cpu model>
>>>>>>>>>>>>>>>>>>>>>>>>>> 	to map the host cpu
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> (my libvirt is 0.9.8) so I use :
>>>>>>>>>>>>>>>>>>>>>>>>>>          <cpu match='exact'>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>            <model>Opteron_G3</model>
>>>>>>>>>>>>>>>>>>>>>>>>>>            <feature policy='disable'
>>>>>>>>>>>>>>>>>>>>>>>>>>            name='hypervisor'/>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>          </cpu>
>>>>>>>>>>>>>>>>>>>>>>>>>> 	
>>>>>>>>>>>>>>>>>>>>>>>>>> 	(the physical server use Opteron CPU).
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 	The log is here :
>>>>>>>>>>>>>>>>>>>>>>>>>> http://www.roullier.net/Report/report-3.2-vhost-ne
>>>>>>>>>>>>>>>>>>>>>>>>>> t- 1v cpu-cp u.tx t.gz
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 	And now with only 1 vcpu, the response time is
>>>>>>>>>>>>>>>>>>>>>>>>>> 	8.5s, great
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> improvment. We keep this configuration for
>>>>>>>>>>>>>>>>>>>>>>>>>> production
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> : we check the response time when some other users
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> are connected.
>>>>>>>>>>>>>>>>>>>>>>>>> please keep in mind, that setting -hypervisor,
>>>>>>>>>>>>>>>>>>>>>>>>> disabling hpet and only one vcpu
>>>>>>>>>>>>>>>>>>>>>>>>> makes windows use tsc as clocksource. you have to
>>>>>>>>>>>>>>>>>>>>>>>>> make sure, that your vm is not switching between
>>>>>>>>>>>>>>>>>>>>>>>>> physical sockets on your system and that you have
>>>>>>>>>>>>>>>>>>>>>>>>> constant_tsc feature to have a stable tsc between
>>>>>>>>>>>>>>>>>>>>>>>>> the cores in the same socket. its also likely that
>>>>>>>>>>>>>>>>>>>>>>>>> the vm will crash when live migrated.
>>>>>>>>>>>>>>>>>>>>>>>> All true. I asked to try -hypervisor only to verify
>>>>>>>>>>>>>>>>>>>>>>>> where we loose performance. Since you get good
>>>>>>>>>>>>>>>>>>>>>>>> result with it frequent access to PM timer is
>>>>>>>>>>>>>>>>>>>>>>>> probably the reason. I do not recommend using
>>>>>>>>>>>>>>>>>>>>>>>> -hypervisor for production!
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> @gleb: do you know whats the state of in-kernel
>>>>>>>>>>>>>>>>>>>>>>>>> hyper-v timers?
>>>>>>>>>>>>>>>>>>>>>>>> Vadim is working on it. I'll let him answer.
>>>>>>>>>>>>>>>>>>>>>>> It would be nice to have synthetic timers supported.
>>>>>>>>>>>>>>>>>>>>>>> But,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>     at the moment, I'm only researching  this feature.
>>>>>>>>>>>>>>>>>>>>>> So it will take months at least?
>>>>>>>>>>>>>>>>>>>>> I would say weeks.
>>>>>>>>>>>>>>>>>>>> Is there a way, we could contribute and help you with
>>>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>>> Hi Peter,
>>>>>>>>>>>>>>>>>>> You are welcome to add  an appropriate handler.
>>>>>>>>>>>>>>>>>> I think Vadim refers to this HV MSR
>>>>>>>>>>>>>>>>>> http://msdn.microsoft.com/en-us/library/windows/hardware/f
>>>>>>>>>>>>>>>>>> f5 42 633%28 v=vs .85 %29.aspx
>>>>>>>>>>>>>>>>> This one is pretty simple to support. Please see
>>>>>>>>>>>>>>>>> attachments for more details. I was thinking about
>>>>>>>>>>>>>>>>> synthetic  timers http://msdn.microsoft.com/en-
>>>>>>>>>>>>>>>>> us/library/windows/hardware/ff542758(v=vs.85).aspx
>>>>>>>>>>>>>>>> is this what microsoft qpc uses as clocksource in hyper-v?
>>>>>>>>>>>>>>> Yes, it should be enough for Win7 / W2K8R2.
>>>>>>>>>>>>>> To clarify the thing that microsoft qpc uses is what is
>>>>>>>>>>>>>> implemented by the patch Vadim attached to his previous email.
>>>>>>>>>>>>>> But I believe that additional qemu patch is needed for Windows
>>>>>>>>>>>>>> to actually use it.
>>>>>>>>>>>>> You are right.
>>>>>>>>>>>>> bits 1 and 9 must be set to on in leaf 0x40000003 and HPET
>>>>>>>>>>>>> should be completely removed from ACPI.
>>>>>>>>>>>> could you advise how to do this and/or make a patch?
>>>>>>>>>>>>
>>>>>>>>>>>> the stuff you send yesterday is for qemu, right? would
>>>>>>>>>>>> it be possible to use it in qemu-kvm also?
>>>>>>>>>>> No, they are for kernel.
>>>>>>>>>> i meant the qemu.diff file.
>>>>>>>>> Yes, I missed the second attachment.
>>>>>>>>>
>>>>>>>>>> if i understand correctly i have to pass -cpu host,+hv_refcnt to
>>>>>>>>>> qemu?
>>>>>>>>> Looks like it.
>>>>>>>> ok, so it would be interesting if it helps to avoid the pmtimer
>>>>>>>> reads we observed earlier. right?
>>>>>>> Yes.
>>>>>> first feedback: performance seems to be amazing. i cannot confirm that
>>>>>> it breaks hv_spinlocks, hv_vapic and hv_relaxed.
>>>>>> why did you assume this?
>>>>> I didn't mean that hv_refcnt will break any other hyper-v features.
>>>>> I just want to say that turning hv_refcnt on (as any other hv_ option)
>>>>> will crash Win8 on boot-up.
>>>> yes, i got it meanwhile ;-)
>>>>
>>>> let me know what you think should be done to further test
>>>> the refcnt implementation.
>>>>
>>>> i would suggest to return at least 0xFFFFFFFF if msr 0x40000021
>>>> is read.
>>> IIRC Win7(W2k8R2) only reads this MSR. Win8 reads and writes.
>> you mean win7 only writes, don't you?
> Oh, yes. It only writes.
> Actually it works this way: kernel allocates one page, maps it into the system
> space and writes the its address to
> 0x40000021 MSR. But to make guest accessing this page, the partition
> reference tsc facility must be enabled, otherwise you can keep any garbage
> there without breaking your guest.
yes, but garbage is different from failing the msr read, isn't it?

another thing i came across with the refcnt. the spec says it runs
at units of 100ns. from what is see inside windows (e.g. when
pinging a local device in the network). it seems that it is 100times
to slow. i will fix this and test further.

peter
>
>> at least you put a break in set_msr_hyperv for this msr.
>>
>> i just thought that it would be ok to return the value that
>> is defined for iTSC is not supported?
>>
>> peter
>>
>>>> peter
>>>>
>>>>> Cheers,
>>>>> Vadim.
>>>>>
>>>>>> no more pmtimer reads. i can now almost fully utililizy a 1GBit
>>>>>> interface with a file transfer while there was not one
>>>>>> cpu core fully utilized as observed with pmtimer. some live migration
>>>>>> tests revealed that it did not crash even under load.
>>>>>>
>>>>>> @vadim: i think we need a proper patch for the others to test this ;-)
>>>>>>
>>>>>> what i observed: is it right, that HV_X64_MSR_TIME_REF_COUNT is
>>>>>> missing in msrs_to_save[] in x86/x86.c of the kernel module?
>>>>>>
>>>>>> thanks for you help,
>>>>>> peter
>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> 			Gleb.