* Reducing I/O introduced domain scheduling
@ 2010-10-12 1:12 Dong, Eddie
2010-10-12 6:38 ` Keir Fraser
2010-10-12 9:15 ` Paul Durrant
0 siblings, 2 replies; 10+ messages in thread
From: Dong, Eddie @ 2010-10-12 1:12 UTC (permalink / raw)
To: Keir Fraser; +Cc: xen-devel, Dong, Eddie, Zhang, Xiantao
Keir:
When running vConsolidation on top of Xen in a 4-core platform, we noticed the I/O introduced scheduling per CPU is ~3K Hz, which seems to be too frequent and cause frequent involve of domain 0 / Qemu, which may polute cache of the guest and thus increase CPI (cycle per instruction).
We are thinking if we can reduce the domin switch here, and think the output of I/O can be buffered and return immediately. The buffered I/O can be flushed out at next IN emulation (or any Hypervisor emulated I/O) or timeout such as 10 or 100 us to guarantee minimal response.
Ideally it can cover both PIO & MMIO, but we can start from PIO.
How do you think of that?
Thx, Eddie
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Reducing I/O introduced domain scheduling
2010-10-12 1:12 Reducing I/O introduced domain scheduling Dong, Eddie
@ 2010-10-12 6:38 ` Keir Fraser
2010-10-12 7:19 ` Dong, Eddie
2010-10-12 9:15 ` Paul Durrant
1 sibling, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2010-10-12 6:38 UTC (permalink / raw)
To: Dong, Eddie; +Cc: xen-devel, Zhang, Xiantao
On 12/10/2010 02:12, "Dong, Eddie" <eddie.dong@intel.com> wrote:
> Keir:
> When running vConsolidation on top of Xen in a 4-core platform, we noticed
> the I/O introduced scheduling per CPU is ~3K Hz, which seems to be too
> frequent and cause frequent involve of domain 0 / Qemu, which may polute cache
> of the guest and thus increase CPI (cycle per instruction).
>
> We are thinking if we can reduce the domin switch here, and think the output
> of I/O can be buffered and return immediately. The buffered I/O can be flushed
> out at next IN emulation (or any Hypervisor emulated I/O) or timeout such as
> 10 or 100 us to guarantee minimal response.
>
> Ideally it can cover both PIO & MMIO, but we can start from PIO.
>
> How do you think of that?
First of all have you tested with PV drivers? Performance tests with no PV
drivers are not that interesting.
Apart from that this sounds like a generalisation of the buffered-i/o
mechanism we already have for svga emulation. I suppose it might be
plausible.
-- Keir
> Thx, Eddie
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Reducing I/O introduced domain scheduling
2010-10-12 6:38 ` Keir Fraser
@ 2010-10-12 7:19 ` Dong, Eddie
2010-10-12 8:47 ` Keir Fraser
0 siblings, 1 reply; 10+ messages in thread
From: Dong, Eddie @ 2010-10-12 7:19 UTC (permalink / raw)
To: Keir Fraser; +Cc: xen-devel, Dong, Eddie, Zhang, Xiantao
Keir Fraser wrote:
> On 12/10/2010 02:12, "Dong, Eddie" <eddie.dong@intel.com> wrote:
>
>> Keir:
>> When running vConsolidation on top of Xen in a 4-core platform, we
>> noticed the I/O introduced scheduling per CPU is ~3K Hz, which seems
>> to be too frequent and cause frequent involve of domain 0 / Qemu,
>> which may polute cache of the guest and thus increase CPI (cycle per
>> instruction).
>>
>> We are thinking if we can reduce the domin switch here, and think
>> the output of I/O can be buffered and return immediately. The
>> buffered I/O can be flushed out at next IN emulation (or any
>> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
>> guarantee minimal response.
>>
>> Ideally it can cover both PIO & MMIO, but we can start from PIO.
>>
>> How do you think of that?
>
> First of all have you tested with PV drivers? Performance tests with
> no PV drivers are not that interesting.
We used PV driver as more as possible, however the C disk in Windows guest doesn't support PV yet. (A task in BIOS side or Win HAL side?)
>
> Apart from that this sounds like a generalisation of the buffered-i/o
> mechanism we already have for svga emulation. I suppose it might be
> plausible.
It is generic, similar to SVGA, but the timeout in generic I/O may be much smaller than display stuff, if a guest is waiting for an interrupt after several OUT instruction.
>
> -- Keir
>
Thx, Eddie
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Reducing I/O introduced domain scheduling
2010-10-12 7:19 ` Dong, Eddie
@ 2010-10-12 8:47 ` Keir Fraser
2010-10-12 8:55 ` James Harper
0 siblings, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2010-10-12 8:47 UTC (permalink / raw)
To: Dong, Eddie; +Cc: Paul Durrant, xen-devel, Zhang, Xiantao, James Harper
On 12/10/2010 08:19, "Dong, Eddie" <eddie.dong@intel.com> wrote:
>> First of all have you tested with PV drivers? Performance tests with
>> no PV drivers are not that interesting.
>
> We used PV driver as more as possible, however the C disk in Windows guest
> doesn't support PV yet. (A task in BIOS side or Win HAL side?)
Running the system disk in PV mode certainly works okay with the Citrix PV
drivers. Might this simply be a current limitation of the GPLPV drivers? If
so, I definitely think you are better off putting effort into removing that
limitation, rather than implementing your proposed optimisation. You will
definitely get a bigger perf boost that way, and no suspicious hypervisor
hacks needed.
Cc'ing James Harper, who will best know what the current GPLPV drivers
can/can't do. Also Paul Durrant, who can likely help with advice on how to
PV up the Windows system disk.
-- Keir
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Reducing I/O introduced domain scheduling
2010-10-12 8:47 ` Keir Fraser
@ 2010-10-12 8:55 ` James Harper
0 siblings, 0 replies; 10+ messages in thread
From: James Harper @ 2010-10-12 8:55 UTC (permalink / raw)
To: Keir Fraser; +Cc: Paul Durrant, xen-devel, Dong, Eddie, Zhang, Xiantao
Gplpv has supported all disks pv since the beginning. Recent versions turn off pv for the CDROM as qemu is more flexible (real CDROM works properly etc) but I think only under xen 3.4.1 or newer
Sent from my iPhone
On 12/10/2010, at 19:48, "Keir Fraser" <keir@xen.org> wrote:
> On 12/10/2010 08:19, "Dong, Eddie" <eddie.dong@intel.com> wrote:
>
>>> First of all have you tested with PV drivers? Performance tests with
>>> no PV drivers are not that interesting.
>>
>> We used PV driver as more as possible, however the C disk in Windows guest
>> doesn't support PV yet. (A task in BIOS side or Win HAL side?)
>
> Running the system disk in PV mode certainly works okay with the Citrix PV
> drivers. Might this simply be a current limitation of the GPLPV drivers? If
> so, I definitely think you are better off putting effort into removing that
> limitation, rather than implementing your proposed optimisation. You will
> definitely get a bigger perf boost that way, and no suspicious hypervisor
> hacks needed.
>
> Cc'ing James Harper, who will best know what the current GPLPV drivers
> can/can't do. Also Paul Durrant, who can likely help with advice on how to
> PV up the Windows system disk.
>
> -- Keir
>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: Reducing I/O introduced domain scheduling
2010-10-12 1:12 Reducing I/O introduced domain scheduling Dong, Eddie
2010-10-12 6:38 ` Keir Fraser
@ 2010-10-12 9:15 ` Paul Durrant
2010-10-12 9:18 ` Keir Fraser
1 sibling, 1 reply; 10+ messages in thread
From: Paul Durrant @ 2010-10-12 9:15 UTC (permalink / raw)
To: Dong, Eddie, Keir Fraser; +Cc: xen-devel, Zhang, Xiantao
Just wondering... does Xen/can Xen take VM exits on fences? If not then I don't see you could safely buffer MMIO writes.
Paul
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Dong, Eddie
> Sent: 12 October 2010 02:12
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao
> Subject: [Xen-devel] Reducing I/O introduced domain scheduling
>
> Keir:
> When running vConsolidation on top of Xen in a 4-core
> platform, we noticed the I/O introduced scheduling per CPU is ~3K
> Hz, which seems to be too frequent and cause frequent involve of
> domain 0 / Qemu, which may polute cache of the guest and thus
> increase CPI (cycle per instruction).
>
> We are thinking if we can reduce the domin switch here, and
> think the output of I/O can be buffered and return immediately. The
> buffered I/O can be flushed out at next IN emulation (or any
> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
> guarantee minimal response.
>
> Ideally it can cover both PIO & MMIO, but we can start from
> PIO.
>
> How do you think of that?
>
> Thx, Eddie
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RE: Reducing I/O introduced domain scheduling
2010-10-12 9:15 ` Paul Durrant
@ 2010-10-12 9:18 ` Keir Fraser
2010-10-12 9:30 ` Paul Durrant
0 siblings, 1 reply; 10+ messages in thread
From: Keir Fraser @ 2010-10-12 9:18 UTC (permalink / raw)
To: Paul Durrant, Dong, Eddie; +Cc: xen-devel, Zhang, Xiantao
No, you can't vmexit on a fence. I don't know whether that matters, so long
as buffered writes get flushed before the guest can observe their effects
(presumably via some kind of I/O read). Agree that generalising the buffered
I/O concept feels a bit dodgy however.
-- Keir
On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:
> Just wondering... does Xen/can Xen take VM exits on fences? If not then I
> don't see you could safely buffer MMIO writes.
>
> Paul
>
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Dong, Eddie
>> Sent: 12 October 2010 02:12
>> To: Keir Fraser
>> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao
>> Subject: [Xen-devel] Reducing I/O introduced domain scheduling
>>
>> Keir:
>> When running vConsolidation on top of Xen in a 4-core
>> platform, we noticed the I/O introduced scheduling per CPU is ~3K
>> Hz, which seems to be too frequent and cause frequent involve of
>> domain 0 / Qemu, which may polute cache of the guest and thus
>> increase CPI (cycle per instruction).
>>
>> We are thinking if we can reduce the domin switch here, and
>> think the output of I/O can be buffered and return immediately. The
>> buffered I/O can be flushed out at next IN emulation (or any
>> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
>> guarantee minimal response.
>>
>> Ideally it can cover both PIO & MMIO, but we can start from
>> PIO.
>>
>> How do you think of that?
>>
>> Thx, Eddie
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: RE: Reducing I/O introduced domain scheduling
2010-10-12 9:18 ` Keir Fraser
@ 2010-10-12 9:30 ` Paul Durrant
2010-10-12 9:44 ` Keir Fraser
2010-10-12 9:46 ` Tim Deegan
0 siblings, 2 replies; 10+ messages in thread
From: Paul Durrant @ 2010-10-12 9:30 UTC (permalink / raw)
To: Keir Fraser, Dong, Eddie; +Cc: xen-devel, Zhang, Xiantao
My concern is a read from a non-MMIO page following a write to an MMIO page.
Paul
> -----Original Message-----
> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir
> Fraser
> Sent: 12 October 2010 10:19
> To: Paul Durrant; Dong, Eddie
> Cc: xen-devel@lists.xensource.com; Zhang, Xiantao
> Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain
> scheduling
>
> No, you can't vmexit on a fence. I don't know whether that matters,
> so long
> as buffered writes get flushed before the guest can observe their
> effects
> (presumably via some kind of I/O read). Agree that generalising the
> buffered
> I/O concept feels a bit dodgy however.
>
> -- Keir
>
> On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:
>
> > Just wondering... does Xen/can Xen take VM exits on fences? If not
> then I
> > don't see you could safely buffer MMIO writes.
> >
> > Paul
> >
> >> -----Original Message-----
> >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> >> bounces@lists.xensource.com] On Behalf Of Dong, Eddie
> >> Sent: 12 October 2010 02:12
> >> To: Keir Fraser
> >> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao
> >> Subject: [Xen-devel] Reducing I/O introduced domain scheduling
> >>
> >> Keir:
> >> When running vConsolidation on top of Xen in a 4-core
> >> platform, we noticed the I/O introduced scheduling per CPU is ~3K
> >> Hz, which seems to be too frequent and cause frequent involve of
> >> domain 0 / Qemu, which may polute cache of the guest and thus
> >> increase CPI (cycle per instruction).
> >>
> >> We are thinking if we can reduce the domin switch here, and
> >> think the output of I/O can be buffered and return immediately.
> The
> >> buffered I/O can be flushed out at next IN emulation (or any
> >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
> >> guarantee minimal response.
> >>
> >> Ideally it can cover both PIO & MMIO, but we can start from
> >> PIO.
> >>
> >> How do you think of that?
> >>
> >> Thx, Eddie
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RE: Reducing I/O introduced domain scheduling
2010-10-12 9:30 ` Paul Durrant
@ 2010-10-12 9:44 ` Keir Fraser
2010-10-12 9:46 ` Tim Deegan
1 sibling, 0 replies; 10+ messages in thread
From: Keir Fraser @ 2010-10-12 9:44 UTC (permalink / raw)
To: Paul Durrant, Dong, Eddie; +Cc: xen-devel, Zhang, Xiantao
Are there any such scenarios that could reveal not-allowed-by-hardware-spec
buffering? If so then that would make the optimisation invalid. But anyway,
it seems that the Intel guys should just fix their test setup to get their
system disk using PV mode.
-- Keir
On 12/10/2010 10:30, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:
> My concern is a read from a non-MMIO page following a write to an MMIO page.
>
> Paul
>
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir
>> Fraser
>> Sent: 12 October 2010 10:19
>> To: Paul Durrant; Dong, Eddie
>> Cc: xen-devel@lists.xensource.com; Zhang, Xiantao
>> Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain
>> scheduling
>>
>> No, you can't vmexit on a fence. I don't know whether that matters,
>> so long
>> as buffered writes get flushed before the guest can observe their
>> effects
>> (presumably via some kind of I/O read). Agree that generalising the
>> buffered
>> I/O concept feels a bit dodgy however.
>>
>> -- Keir
>>
>> On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:
>>
>>> Just wondering... does Xen/can Xen take VM exits on fences? If not
>> then I
>>> don't see you could safely buffer MMIO writes.
>>>
>>> Paul
>>>
>>>> -----Original Message-----
>>>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>>>> bounces@lists.xensource.com] On Behalf Of Dong, Eddie
>>>> Sent: 12 October 2010 02:12
>>>> To: Keir Fraser
>>>> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao
>>>> Subject: [Xen-devel] Reducing I/O introduced domain scheduling
>>>>
>>>> Keir:
>>>> When running vConsolidation on top of Xen in a 4-core
>>>> platform, we noticed the I/O introduced scheduling per CPU is ~3K
>>>> Hz, which seems to be too frequent and cause frequent involve of
>>>> domain 0 / Qemu, which may polute cache of the guest and thus
>>>> increase CPI (cycle per instruction).
>>>>
>>>> We are thinking if we can reduce the domin switch here, and
>>>> think the output of I/O can be buffered and return immediately.
>> The
>>>> buffered I/O can be flushed out at next IN emulation (or any
>>>> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
>>>> guarantee minimal response.
>>>>
>>>> Ideally it can cover both PIO & MMIO, but we can start from
>>>> PIO.
>>>>
>>>> How do you think of that?
>>>>
>>>> Thx, Eddie
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: RE: Reducing I/O introduced domain scheduling
2010-10-12 9:30 ` Paul Durrant
2010-10-12 9:44 ` Keir Fraser
@ 2010-10-12 9:46 ` Tim Deegan
1 sibling, 0 replies; 10+ messages in thread
From: Tim Deegan @ 2010-10-12 9:46 UTC (permalink / raw)
To: Paul Durrant; +Cc: Dong, Eddie, xen-devel, Keir Fraser, Zhang, Xiantao
At 10:30 +0100 on 12 Oct (1286879457), Paul Durrant wrote:
> My concern is a read from a non-MMIO page following a write to an MMIO page.
Yes, buffering MMIO in the general case is totally unsafe. That's why
the existing buffered-MMIO ring is only used for VGA.
Tim.
> Paul
>
> > -----Original Message-----
> > From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir
> > Fraser
> > Sent: 12 October 2010 10:19
> > To: Paul Durrant; Dong, Eddie
> > Cc: xen-devel@lists.xensource.com; Zhang, Xiantao
> > Subject: Re: [Xen-devel] RE: Reducing I/O introduced domain
> > scheduling
> >
> > No, you can't vmexit on a fence. I don't know whether that matters,
> > so long
> > as buffered writes get flushed before the guest can observe their
> > effects
> > (presumably via some kind of I/O read). Agree that generalising the
> > buffered
> > I/O concept feels a bit dodgy however.
> >
> > -- Keir
> >
> > On 12/10/2010 10:15, "Paul Durrant" <Paul.Durrant@citrix.com> wrote:
> >
> > > Just wondering... does Xen/can Xen take VM exits on fences? If not
> > then I
> > > don't see you could safely buffer MMIO writes.
> > >
> > > Paul
> > >
> > >> -----Original Message-----
> > >> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> > >> bounces@lists.xensource.com] On Behalf Of Dong, Eddie
> > >> Sent: 12 October 2010 02:12
> > >> To: Keir Fraser
> > >> Cc: xen-devel@lists.xensource.com; Dong, Eddie; Zhang, Xiantao
> > >> Subject: [Xen-devel] Reducing I/O introduced domain scheduling
> > >>
> > >> Keir:
> > >> When running vConsolidation on top of Xen in a 4-core
> > >> platform, we noticed the I/O introduced scheduling per CPU is ~3K
> > >> Hz, which seems to be too frequent and cause frequent involve of
> > >> domain 0 / Qemu, which may polute cache of the guest and thus
> > >> increase CPI (cycle per instruction).
> > >>
> > >> We are thinking if we can reduce the domin switch here, and
> > >> think the output of I/O can be buffered and return immediately.
> > The
> > >> buffered I/O can be flushed out at next IN emulation (or any
> > >> Hypervisor emulated I/O) or timeout such as 10 or 100 us to
> > >> guarantee minimal response.
> > >>
> > >> Ideally it can cover both PIO & MMIO, but we can start from
> > >> PIO.
> > >>
> > >> How do you think of that?
> > >>
> > >> Thx, Eddie
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@lists.xensource.com
> > >> http://lists.xensource.com/xen-devel
> > >
> > > _______________________________________________
> > > Xen-devel mailing list
> > > Xen-devel@lists.xensource.com
> > > http://lists.xensource.com/xen-devel
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
--
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, XenServer Engineering
Citrix Systems UK Ltd. (Company #02937203, SL9 0BG)
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-10-12 9:46 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-10-12 1:12 Reducing I/O introduced domain scheduling Dong, Eddie
2010-10-12 6:38 ` Keir Fraser
2010-10-12 7:19 ` Dong, Eddie
2010-10-12 8:47 ` Keir Fraser
2010-10-12 8:55 ` James Harper
2010-10-12 9:15 ` Paul Durrant
2010-10-12 9:18 ` Keir Fraser
2010-10-12 9:30 ` Paul Durrant
2010-10-12 9:44 ` Keir Fraser
2010-10-12 9:46 ` Tim Deegan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.