paravirtualization & cr3-cache feature

paravirtualization & cr3-cache feature

[Omar Khan]

hi,
    Can someone briefly explain what the cr3-cache feature on intel VT
processors is and how it is used to improve performance in paravirtualization. 

Thanks

Omar 


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>hi,
>    Can someone briefly explain what the cr3-cache feature on intel VT
>processors is and how it is used to improve performance in
>paravirtualization.

In short, this is a unique Intel VT feature that supposed to save vmexit
on cr3 switch. The vmexit operation is quite expensive at the moment
(something like 2-5 ms!). PV kernels can use this feature by sharing
memory between the host & the guest for the cr3 cache. When the guest
wants to change cr3 it looks at the cache for a hit. If there is a hit
there is no need to go out to the hypervisor and the guest continue to
execute without the time consuming vmexit.

You can check out Ingo's patch that does just it at
http://people.redhat.com/mingo/kvm-paravirt-patches/kvm-paravirt-cr3-cac
he.patch

The down side is that the currently complex mmu implementation get's
more complicated, while Intel & AMD will reduce the vmexit cost in the
future.

HTH,
Dor Laor.

>
>Thanks
>
>Omar
>
>
>-----------------------------------------------------------------------
--
>Using Tomcat but need to do more? Need to support web services,
security?
>Get stuff done quickly with pre-integrated technology to make your job
>easier.
>Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
>http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=12164
2
>_______________________________________________
>kvm-devel mailing list
>kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
>https://lists.sourceforge.net/lists/listinfo/kvm-devel

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Omar Khan]

Thanks Dor.

Omar



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Joerg Roedel]

On Sat, Feb 10, 2007 at 02:22:25PM -0800, Dor Laor wrote:
> 
> >hi,
> >    Can someone briefly explain what the cr3-cache feature on intel VT
> >processors is and how it is used to improve performance in
> >paravirtualization.
> 
> In short, this is a unique Intel VT feature that supposed to save vmexit
> on cr3 switch. The vmexit operation is quite expensive at the moment
> (something like 2-5 ms!). PV kernels can use this feature by sharing
  ^^^^^^^^^^^^^^^^^^^^^^^^

Where do you measured this unbelievable large number? On AMD-V I
measured around 2400 processor cycles for a VMEXIT/VMENTRY roundtrip. I
would really really wonder if Intel-VT is that slow...

Joerg

-- 
Joerg Roedel
Operating System Research Center
AMD Saxony LLC & Co. KG



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>>
>> >hi,
>> >    Can someone briefly explain what the cr3-cache feature on intel
VT
>> >processors is and how it is used to improve performance in
>> >paravirtualization.
>>
>> In short, this is a unique Intel VT feature that supposed to save
vmexit
>> on cr3 switch. The vmexit operation is quite expensive at the moment
>> (something like 2-5 ms!). PV kernels can use this feature by sharing
>  ^^^^^^^^^^^^^^^^^^^^^^^^
>
>Where do you measured this unbelievable large number? On AMD-V I
>measured around 2400 processor cycles for a VMEXIT/VMENTRY roundtrip. I
>would really really wonder if Intel-VT is that slow...

Intel is indeed slower than AMD with this one.
I didn't measure such roundtrip specifically. We usually see that the
VMETNRY/VMEXIT cycle limits our performance (for io/mmio,..)
On Intel core duo we cannot get more than 150k-200k VMEXITS per second
for regular guest doing extensive IO. 

Anyway 2ms for a 2.4G processor equals 4800 cycles so it's not that far.

>
>Joerg
>
>--
>Joerg Roedel
>Operating System Research Center
>AMD Saxony LLC & Co. KG
>


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Daniel Veillard]

On Mon, Feb 12, 2007 at 05:21:26AM -0800, Dor Laor wrote:
> >>
> >> >hi,
> >> >    Can someone briefly explain what the cr3-cache feature on intel
> VT
> >> >processors is and how it is used to improve performance in
> >> >paravirtualization.
> >>
> >> In short, this is a unique Intel VT feature that supposed to save
> vmexit
> >> on cr3 switch. The vmexit operation is quite expensive at the moment
> >> (something like 2-5 ms!). PV kernels can use this feature by sharing
> >  ^^^^^^^^^^^^^^^^^^^^^^^^
> >
> >Where do you measured this unbelievable large number? On AMD-V I
> >measured around 2400 processor cycles for a VMEXIT/VMENTRY roundtrip. I
> >would really really wonder if Intel-VT is that slow...
> 
> Intel is indeed slower than AMD with this one.
> I didn't measure such roundtrip specifically. We usually see that the
> VMETNRY/VMEXIT cycle limits our performance (for io/mmio,..)
> On Intel core duo we cannot get more than 150k-200k VMEXITS per second
> for regular guest doing extensive IO. 
> 
> Anyway 2ms for a 2.4G processor equals 4800 cycles so it's not that far.

  Hum, ms means millisecond in the SI system, so on a 2.4 billion cycle/s
2ms would be 4.8 million cycles, hence IMHO the unbelievably large perception.
I assume you mean VMEXITS would be around  2-5 microsecond, which would fit
quite better in the overall picture :-)

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>On Mon, Feb 12, 2007 at 05:21:26AM -0800, Dor Laor wrote:
>> >>
>> >> >hi,
>> >> >    Can someone briefly explain what the cr3-cache feature on
intel
>> VT
>> >> >processors is and how it is used to improve performance in
>> >> >paravirtualization.
>> >>
>> >> In short, this is a unique Intel VT feature that supposed to save
>> vmexit
>> >> on cr3 switch. The vmexit operation is quite expensive at the
moment
>> >> (something like 2-5 ms!). PV kernels can use this feature by
sharing
>> >  ^^^^^^^^^^^^^^^^^^^^^^^^
>> >
>> >Where do you measured this unbelievable large number? On AMD-V I
>> >measured around 2400 processor cycles for a VMEXIT/VMENTRY
roundtrip. I
>> >would really really wonder if Intel-VT is that slow...
>>
>> Intel is indeed slower than AMD with this one.
>> I didn't measure such roundtrip specifically. We usually see that the
>> VMETNRY/VMEXIT cycle limits our performance (for io/mmio,..)
>> On Intel core duo we cannot get more than 150k-200k VMEXITS per
second
>> for regular guest doing extensive IO.
>>
>> Anyway 2ms for a 2.4G processor equals 4800 cycles so it's not that
far.
>
>  Hum, ms means millisecond in the SI system, so on a 2.4 billion
cycle/s
>2ms would be 4.8 million cycles, hence IMHO the unbelievably large
>perception.
>I assume you mean VMEXITS would be around  2-5 microsecond, which would
fit
>quite better in the overall picture :-)


Thanks man, my mistake, I meant microsecond when wrote the 'ms'.

>Daniel
>
>--
>Red Hat Virtualization group http://redhat.com/virtualization/
>Daniel Veillard      | virtualization library  http://libvirt.org/
>veillard-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org  | libxml GNOME XML XSLT toolkit
http://xmlsoft.org/
>http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Gregory Haskins]

>>> On Mon, Feb 12, 2007 at  9:23 AM, in message
<64F9B87B6B770947A9F8391472E032160A66C0CF-yEcIvxbTEBqsx+V+t5oei8rau4O3wl8o3fe8/T/H7NteoWH0uzbU5w@public.gmane.org>,
"Dor Laor" <dor.laor-atKUWr5tajBWk0Htik3J/w@public.gmane.org> wrote: 

> 
> Thanks man, my mistake, I meant microsecond when wrote the 'ms'.

FYI: The SI symbol for micro is the greek lower-case letter Mu.  However, since that would be a pain in ASCII, the letter "u" is used instead (e.g. "us" is microseconds....ms is milli, ns is nano, etc)  

-Greg

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[richardvoigt-Re5JQEeQqe8AvxtiuMwx3w]

> > Intel is indeed slower than AMD with this one.
> > I didn't measure such roundtrip specifically. We usually see that the
> > VMETNRY/VMEXIT cycle limits our performance (for io/mmio,..)
> > On Intel core duo we cannot get more than 150k-200k VMEXITS per second
> > for regular guest doing extensive IO.
> >

/me raises hand

Why so many vm switches?  First up, a typical I/O system maxes at
about 1Gb/s, right?  That would be a gigabit NIC, or striped RAID, or
something like that.  This suggests an average of only about 300
bytes/transfer, to get >150k individual transfers per second?  I
thought block I/O usually dealt with 1kbyte or more.  Next, we're
living in a world where most CPUs supporting the required extended
instruction set are multi-core, and you specifically said core duo.
Shouldn't an extensive I/O workload  tend toward one CPU in each VM,
with contraposed producer-consumer queues, and almost zero context
switches?

-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>> > Intel is indeed slower than AMD with this one.
>> > I didn't measure such roundtrip specifically. We usually see that
the
>> > VMETNRY/VMEXIT cycle limits our performance (for io/mmio,..)
>> > On Intel core duo we cannot get more than 150k-200k VMEXITS per
second
>> > for regular guest doing extensive IO.
>> >
>
>/me raises hand


Don't give up so fast ;)
>
>Why so many vm switches?  First up, a typical I/O system maxes at
>about 1Gb/s, right?  That would be a gigabit NIC, or striped RAID, or
>something like that.  This suggests an average of only about 300
>bytes/transfer, to get >150k individual transfers per second?  I
>thought block I/O usually dealt with 1kbyte or more.  Next, we're
>living in a world where most CPUs supporting the required extended
>instruction set are multi-core, and you specifically said core duo.
>Shouldn't an extensive I/O workload  tend toward one CPU in each VM,
>with contraposed producer-consumer queues, and almost zero context
>switches?

First 150k-200k vm switches is the maximum number we can reach for a
single core. The exits are not necessarily related to IO, for instance
before the new MMU code page fault exits were performance bottle neck
which major part of it was the vm switches.

Second, we currently use Qemu's device emulation were the ne2k device
does dozens of IO accesses per packet! The rtl8139 is better and does
about 3 IO(MMIO) per packet. The current maximum throughput using the
rtl8139 is ~30Mbps. Very soon we'll have PV drivers that will boost
performance and use the P/C queues you were talking about. PV drivers
will boost performance beyond 1G/bps.

Also note that at the moment KVM supports only single cpu guests.



-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Arjan van de Ven]

> Second, we currently use Qemu's device emulation were the ne2k device
> does dozens of IO accesses per packet! The rtl8139 is better and does
> about 3 IO(MMIO) per packet. The current maximum throughput using the

and this is where paravirtualized device drivers can help, like Ingo's
network driver or the block driver I'm working on (which currently
happily writes 215Mbyte/sec from guest userspace using normal write() to
the virtual disk)...

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>
>> Second, we currently use Qemu's device emulation were the ne2k device
>> does dozens of IO accesses per packet! The rtl8139 is better and does
>> about 3 IO(MMIO) per packet. The current maximum throughput using the
>
>and this is where paravirtualized device drivers can help, like Ingo's
>network driver or the block driver I'm working on (which currently
>happily writes 215Mbyte/sec from guest userspace using normal write()
to
>the virtual disk)...

Great news.
Expecting to see the patch soon.

Do you base your code over the kernel's paravirt_ops and Ingo's KVM
hypercall?
Jusy yesterday Yaniv Kamay & I have discovered some bugs in the
'kvm_hypercall' function. The cache_regs/decahce_regs were mixed up.
Also I added a x86_64 hypercall.h file.
Hope to commit it today or tomorrow.

>
>--
>if you want to mail me at work (you don't), use arjan (at)
linux.intel.com
>Test the interaction between Linux and your BIOS via
>http://www.linuxfirmwarekit.org


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[Arjan van de Ven]

> Great news.
> Expecting to see the patch soon.

there's still a wart or two on the host side but it's getting ready
quickly.. give me a few days ;)


> Do you base your code over the kernel's paravirt_ops and Ingo's KVM
> hypercall?

I'm using Ingo's KVM hypercall yes; I don't need paravirt_ops at all.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org


-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier.
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642

Re: paravirtualization & cr3-cache feature

[richardvoigt-Re5JQEeQqe8AvxtiuMwx3w]

> >Why so many vm switches?  First up, a typical I/O system maxes at
> >about 1Gb/s, right?  That would be a gigabit NIC, or striped RAID, or
> >something like that.  This suggests an average of only about 300
> >bytes/transfer, to get >150k individual transfers per second?  I
> >thought block I/O usually dealt with 1kbyte or more.  Next, we're
> >living in a world where most CPUs supporting the required extended
> >instruction set are multi-core, and you specifically said core duo.
> >Shouldn't an extensive I/O workload  tend toward one CPU in each VM,
> >with contraposed producer-consumer queues, and almost zero context
> >switches?
>
> First 150k-200k vm switches is the maximum number we can reach for a
> single core. The exits are not necessarily related to IO, for instance
> before the new MMU code page fault exits were performance bottle neck
> which major part of it was the vm switches.
>
> Second, we currently use Qemu's device emulation were the ne2k device
> does dozens of IO accesses per packet! The rtl8139 is better and does
> about 3 IO(MMIO) per packet. The current maximum throughput using the
> rtl8139 is ~30Mbps. Very soon we'll have PV drivers that will boost
> performance and use the P/C queues you were talking about. PV drivers
> will boost performance beyond 1G/bps.

Thanks for the explanation.  Two things had thrown me off: first the
phrase "extensive I/O" made me think of disk I/O with rather large
blocks, not a bunch of smallish network packets.  And I'd forgotten
that we're dealing with full virt, so you can't pipeline requests,
since the driver expects synchronous memory access.  Is it possible
for any of qemu's hardware emulation code to run without a VMEXIT,
more or less like a software interrupt within the VM?  Or is VMEXIT
already the equivalent of a software interrupt or interrupt from
request for privileged instruction from ring 3, as applied to virtual
machine instead of ring 3 user mode?

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: paravirtualization & cr3-cache feature

[Dor Laor]

>> >Why so many vm switches?  First up, a typical I/O system maxes at
>> >about 1Gb/s, right?  That would be a gigabit NIC, or striped RAID,
or
>> >something like that.  This suggests an average of only about 300
>> >bytes/transfer, to get >150k individual transfers per second?  I
>> >thought block I/O usually dealt with 1kbyte or more.  Next, we're
>> >living in a world where most CPUs supporting the required extended
>> >instruction set are multi-core, and you specifically said core duo.
>> >Shouldn't an extensive I/O workload  tend toward one CPU in each VM,
>> >with contraposed producer-consumer queues, and almost zero context
>> >switches?
>>
>> First 150k-200k vm switches is the maximum number we can reach for a
>> single core. The exits are not necessarily related to IO, for
instance
>> before the new MMU code page fault exits were performance bottle neck
>> which major part of it was the vm switches.
>>
>> Second, we currently use Qemu's device emulation were the ne2k device
>> does dozens of IO accesses per packet! The rtl8139 is better and does
>> about 3 IO(MMIO) per packet. The current maximum throughput using the
>> rtl8139 is ~30Mbps. Very soon we'll have PV drivers that will boost
>> performance and use the P/C queues you were talking about. PV drivers
>> will boost performance beyond 1G/bps.
>
>Thanks for the explanation.  Two things had thrown me off: first the
>phrase "extensive I/O" made me think of disk I/O with rather large
>blocks, not a bunch of smallish network packets.  And I'd forgotten
>that we're dealing with full virt, so you can't pipeline requests,
>since the driver expects synchronous memory access.  Is it possible
>for any of qemu's hardware emulation code to run without a VMEXIT,
>more or less like a software interrupt within the VM?  Or is VMEXIT
>already the equivalent of a software interrupt or interrupt from
>request for privileged instruction from ring 3, as applied to virtual
>machine instead of ring 3 user mode?

I didn't completely understand, do you mean that the guest would send
the software interrupts for cheaper VM exit?
If that's the case, I find it impossible for fully virtualized devices
since these devices use I/O or MMIO for connectivity with software.

We currently add PV drivers (even to fully virtualized guests) and we
will queue/coalesce packet together.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV