VCPUOP_set_periodic_timer

VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 659 bytes --]

Hi all,

I need a periodic timer running at ideally at 125 microseconds and at 
least 500 microseconds. I've just found the VCPUOP_set_periodic_timer, 
however there is a comment saying "periods less than one millisecond may 
not be supported".

I will be running on an x64 machine. Is this supported? If not, is there 
any alternate means of generating a fast interrupt?

Regards.


--
  _     _  Debian GNU User   Simon Martin
| |   (_)_ __  _   ___  __  Project Manager
| |   | | '_ \| | | \ \/ /  Milliways
| |___| | | | | |_| |>  <   mailto: smartin@milliways.cl
|_____|_|_| |_|\__,_/_/\_\
Si Hoc Legere Scis Nimium Eruditionis Habes

[-- Attachment #1.2: Type: text/html, Size: 1795 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 853 bytes --]

On 14/11/2013 21:18, Simon Martin wrote:
> Hi all,
>  
> I need a periodic timer running at ideally at 125 microseconds and at
> least 500 microseconds. I've just found the VCPUOP_set_periodic_timer,
> however there is a comment saying "periods less than one millisecond
> may not be supported".
>  
> I will be running on an x64 machine. Is this supported? If not, is
> there any alternate means of generating a fast interrupt?
>  
> Regards.

What is the usecase here?  125us is very short indeed.  Xen certainly
cant guarantee anything more accurate than 50us.  Unless the affected
vcpu is running uncontested on the hardware, there is very little chance
that the vcpu will indeed be woken up again in 125us.

It sounds as if you are looking for some pseudo realtime system, at
which point you might want to consider a different scheduler.

~Andrew


[-- Attachment #1.2: Type: text/html, Size: 2081 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 1973 bytes --]

Hi Andrew,

The operating system I am trying to port to Xen is an industrial servo 
controller. We are currently running at 125 microsecond servo update 
rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 
125 microsecond ideal interrupt period). We will be driving EtherCAT 
servo drives that need to be updated at 500 microseconds (hence at least 
500 microsecond interrupt period). If we can achieve this on a single 
core ARM11 clocked at 532 MHz, it must be achievable on PC hardware. As 
I said before the idea is to dedicate a CPU core to this with all other 
functions running on the other cores.

Luckily we can accept a reasonable amount of jitter on the EtherCAT 
network as we can hand over clock synchronisation to a slave. This gives 
us a little leeway.

Regards.


------ Original Message ------
From: "Andrew Cooper" <andrew.cooper3@citrix.com>
To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" 
<xen-devel@lists.xen.org>
Sent: 14/11/2013 18:39:21
Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer

>On 14/11/2013 21:18, Simon Martin wrote:
>>Hi all,
>>
>>I need a periodic timer running at ideally at 125 microseconds and at 
>>least 500 microseconds. I've just found the VCPUOP_set_periodic_timer, 
>>however there is a comment saying "periods less than one millisecond 
>>may not be supported".
>>
>>I will be running on an x64 machine. Is this supported? If not, is 
>>there any alternate means of generating a fast interrupt?
>>
>>Regards.
>
>What is the usecase here?  125us is very short indeed.  Xen certainly 
>cant guarantee anything more accurate than 50us.  Unless the affected 
>vcpu is running uncontested on the hardware, there is very little 
>chance that the vcpu will indeed be woken up again in 125us.
>
>It sounds as if you are looking for some pseudo realtime system, at 
>which point you might want to consider a different scheduler.
>
>~Andrew
>

[-- Attachment #1.2: Type: text/html, Size: 3880 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 2755 bytes --]

The periodic_timer is really a coarse-grained timer for driving a legacy
ticker. It is considered low accuracy and low priority ­ for example, it is
stopped entirely while a vcpu is descheduled!

You should use VCPUOP_set_singleshot_timer, and call it again at the end of
your timer handler to get the desired periodic behaviour. If you have
multiple timers in your OS you would manage them in your OS and pick the
nearest deadline to program down to Xen.

 -- Keir

On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote:

> Hi Andrew,
>  
> The operating system I am trying to port to Xen is an industrial servo
> controller. We are currently running at 125 microsecond servo update rate on
> our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125 microsecond
> ideal interrupt period). We will be driving EtherCAT servo drives that need to
> be updated at 500 microseconds (hence at least 500 microsecond interrupt
> period). If we can achieve this on a single core ARM11 clocked at 532 MHz, it
> must be achievable on PC hardware. As I said before the idea is to dedicate a
> CPU core to this with all other functions running on the other cores.
>  
> Luckily we can accept a reasonable amount of jitter on the EtherCAT network as
> we can hand over clock synchronisation to a slave. This gives us a little
> leeway.
>  
> Regards.
>  
>  
> ------ Original Message ------
> From: "Andrew Cooper" <andrew.cooper3@citrix.com>
> To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org"
> <xen-devel@lists.xen.org>
> Sent: 14/11/2013 18:39:21
> Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>  
>> On 14/11/2013 21:18, Simon Martin wrote:
>>> Hi all,
>>>  
>>> I need a periodic timer running at ideally at 125 microseconds and at least
>>> 500 microseconds. I've just found the VCPUOP_set_periodic_timer, however
>>> there is a comment saying "periods less than one millisecond may not be
>>> supported".
>>>  
>>> I will be running on an x64 machine. Is this supported? If not, is there any
>>> alternate means of generating a fast interrupt?
>>>  
>>> Regards.
>> 
>> What is the usecase here?  125us is very short indeed.  Xen certainly cant
>> guarantee anything more accurate than 50us.  Unless the affected vcpu is
>> running uncontested on the hardware, there is very little chance that the
>> vcpu will indeed be woken up again in 125us.
>> 
>> It sounds as if you are looking for some pseudo realtime system, at which
>> point you might want to consider a different scheduler.
>> 
>> ~Andrew
>> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel


[-- Attachment #1.2: Type: text/html, Size: 4163 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 2694 bytes --]

On 15/11/13 11:24, Simon Martin wrote:
> Hi Andrew,
>  
> The operating system I am trying to port to Xen is an industrial servo
> controller. We are currently running at 125 microsecond servo update
> rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the
> 125 microsecond ideal interrupt period). We will be driving EtherCAT
> servo drives that need to be updated at 500 microseconds (hence at
> least 500 microsecond interrupt period). If we can achieve this on a
> single core ARM11 clocked at 532 MHz, it must be achievable on PC
> hardware. As I said before the idea is to dedicate a CPU core to this
> with all other functions running on the other cores.
>  
> Luckily we can accept a reasonable amount of jitter on the EtherCAT
> network as we can hand over clock synchronisation to a slave. This
> gives us a little leeway.
>  
> Regards.
>  
>  

It certainly is achievable on PC hardware, but not under the standard
assumptions in virtualisation.  You want to use
VCPUOP_set_singleshot_timer, but be aware than there are still no normal
guarantees that your domain vcpu will be run when the timer expires; the
credit scheduler will pick the highest priority vcpu to run, which might
not be the one which is expecting a timer interrupt.

You should investigate using the arinc653 scheduler in Xen, which is a
realtime scheduler, and might be more appropriate for your usecase.

~Andrew


> ------ Original Message ------
> From: "Andrew Cooper" <andrew.cooper3@citrix.com
> <mailto:andrew.cooper3@citrix.com>>
> To: "Simon Martin" <smartin@milliways.cl
> <mailto:smartin@milliways.cl>>; "xen-devel@lists.xen.org"
> <xen-devel@lists.xen.org <mailto:xen-devel@lists.xen.org>>
> Sent: 14/11/2013 18:39:21
> Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>  
>> On 14/11/2013 21:18, Simon Martin wrote:
>>> Hi all,
>>>  
>>> I need a periodic timer running at ideally at 125 microseconds and
>>> at least 500 microseconds. I've just found the
>>> VCPUOP_set_periodic_timer, however there is a comment saying
>>> "periods less than one millisecond may not be supported".
>>>  
>>> I will be running on an x64 machine. Is this supported? If not, is
>>> there any alternate means of generating a fast interrupt?
>>>  
>>> Regards.
>>
>> What is the usecase here?  125us is very short indeed.  Xen certainly
>> cant guarantee anything more accurate than 50us.  Unless the affected
>> vcpu is running uncontested on the hardware, there is very little
>> chance that the vcpu will indeed be woken up again in 125us.
>>
>> It sounds as if you are looking for some pseudo realtime system, at
>> which point you might want to consider a different scheduler.
>>
>> ~Andrew
>>


[-- Attachment #1.2: Type: text/html, Size: 5732 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 3298 bytes --]

Thanks Keir,

That sounds quite reasonable. What granularity can I achieve with this? 
Any idea on the jitter?

Regards.


------ Original Message ------
From: "Keir Fraser" <keir.xen@gmail.com>
To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" 
<andrew.cooper3@citrix.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Sent: 15/11/2013 08:36:45
Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer

>The periodic_timer is really a coarse-grained timer for driving a 
>legacy ticker. It is considered low accuracy and low priority – for 
>example, it is stopped entirely while a vcpu is descheduled!
>
>You should use VCPUOP_set_singleshot_timer, and call it again at the 
>end of your timer handler to get the desired periodic behaviour. If you 
>have multiple timers in your OS you would manage them in your OS and 
>pick the nearest deadline to program down to Xen.
>
>  -- Keir
>
>On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote:
>
>>Hi Andrew,
>>
>>The operating system I am trying to port to Xen is an industrial servo 
>>controller. We are currently running at 125 microsecond servo update 
>>rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 
>>125 microsecond ideal interrupt period). We will be driving EtherCAT 
>>servo drives that need to be updated at 500 microseconds (hence at 
>>least 500 microsecond interrupt period). If we can achieve this on a 
>>single core ARM11 clocked at 532 MHz, it must be achievable on PC 
>>hardware. As I said before the idea is to dedicate a CPU core to this 
>>with all other functions running on the other cores.
>>
>>Luckily we can accept a reasonable amount of jitter on the EtherCAT 
>>network as we can hand over clock synchronisation to a slave. This 
>>gives us a little leeway.
>>
>>Regards.
>>
>>
>>------ Original Message ------
>>From: "Andrew Cooper" <andrew.cooper3@citrix.com>
>>To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org" 
>><xen-devel@lists.xen.org>
>>Sent: 14/11/2013 18:39:21
>>Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>>
>>>On 14/11/2013 21:18, Simon Martin wrote:
>>>>Hi all,
>>>>
>>>>I need a periodic timer running at ideally at 125 microseconds and 
>>>>at least 500 microseconds. I've just found the 
>>>>VCPUOP_set_periodic_timer, however there is a comment saying 
>>>>"periods less than one millisecond may not be supported".
>>>>
>>>>I will be running on an x64 machine. Is this supported? If not, is 
>>>>there any alternate means of generating a fast interrupt?
>>>>
>>>>Regards.
>>>
>>>What is the usecase here?  125us is very short indeed.  Xen certainly 
>>>cant guarantee anything more accurate than 50us.  Unless the affected 
>>>vcpu is running uncontested on the hardware, there is very little 
>>>chance that the vcpu will indeed be woken up again in 125us.
>>>
>>>It sounds as if you are looking for some pseudo realtime system, at 
>>>which point you might want to consider a different scheduler.
>>>
>>>~Andrew
>>>
>>
>>--------------------------------------------------------------------------------
>>_______________________________________________
>>Xen-devel mailing list
>>Xen-devel@lists.xen.org
>>http://lists.xen.org/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 5562 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Keir Fraser]

[-- Attachment #1.1: Type: text/plain, Size: 4014 bytes --]

If you have dedicated a core to the vcpu, so you can discount scheduling
crosstalk, you shouldn¹t miss deadlines by more than a microsecond or two.
Just enough time to wake up the CPU, handle the timer interrupt, and wake up
the vcpu through the scheduler. There are infrequent operations that can
take out all CPUs for a period: taking a CPU offline for example. You could
avoid doing such things in your setup perhaps. :) Ultimately you will need
to measure and confirm deadline-to-notification latency distribution if it
is critical to you.

 -- Keir

On 15/11/2013 11:45, "Simon Martin" <smartin@milliways.cl> wrote:

> Thanks Keir,
>  
> That sounds quite reasonable. What granularity can I achieve with this? Any
> idea on the jitter?
>  
> Regards.
>  
>  
> ------ Original Message ------
> From: "Keir Fraser" <keir.xen@gmail.com>
> To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper"
> <andrew.cooper3@citrix.com>
> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
> Sent: 15/11/2013 08:36:45
> Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>  
>> The periodic_timer is really a coarse-grained timer for driving a legacy
>> ticker. It is considered low accuracy and low priority ­ for example, it is
>> stopped entirely while a vcpu is descheduled!
>> 
>> You should use VCPUOP_set_singleshot_timer, and call it again at the end of
>> your timer handler to get the desired periodic behaviour. If you have
>> multiple timers in your OS you would manage them in your OS and pick the
>> nearest deadline to program down to Xen.
>> 
>>  -- Keir
>> 
>> On 15/11/2013 11:24, "Simon Martin" <smartin@milliways.cl> wrote:
>> 
>>> Hi Andrew,
>>>  
>>> The operating system I am trying to port to Xen is an industrial servo
>>> controller. We are currently running at 125 microsecond servo update rate on
>>> our bespoke hardware (at the moment MIPS64 and ARM) (hence the 125
>>> microsecond ideal interrupt period). We will be driving EtherCAT servo
>>> drives that need to be updated at 500 microseconds (hence at least 500
>>> microsecond interrupt period). If we can achieve this on a single core ARM11
>>> clocked at 532 MHz, it must be achievable on PC hardware. As I said before
>>> the idea is to dedicate a CPU core to this with all other functions running
>>> on the other cores.
>>>  
>>> Luckily we can accept a reasonable amount of jitter on the EtherCAT network
>>> as we can hand over clock synchronisation to a slave. This gives us a little
>>> leeway.
>>>  
>>> Regards.
>>>  
>>>  
>>> ------ Original Message ------
>>> From: "Andrew Cooper" <andrew.cooper3@citrix.com>
>>> To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org"
>>> <xen-devel@lists.xen.org>
>>> Sent: 14/11/2013 18:39:21
>>> Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>>>  
>>>> On 14/11/2013 21:18, Simon Martin wrote:
>>>>> Hi all,
>>>>>  
>>>>> I need a periodic timer running at ideally at 125 microseconds and at
>>>>> least 500 microseconds. I've just found the VCPUOP_set_periodic_timer,
>>>>> however there is a comment saying "periods less than one millisecond may
>>>>> not be supported".
>>>>>  
>>>>> I will be running on an x64 machine. Is this supported? If not, is there
>>>>> any alternate means of generating a fast interrupt?
>>>>>  
>>>>> Regards.
>>>> 
>>>> What is the usecase here?  125us is very short indeed.  Xen certainly cant
>>>> guarantee anything more accurate than 50us.  Unless the affected vcpu is
>>>> running uncontested on the hardware, there is very little chance that the
>>>> vcpu will indeed be woken up again in 125us.
>>>> 
>>>> It sounds as if you are looking for some pseudo realtime system, at which
>>>> point you might want to consider a different scheduler.
>>>> 
>>>> ~Andrew
>>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
> 


[-- Attachment #1.2: Type: text/html, Size: 6020 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 3430 bytes --]

Thanks Andrew,

I've tried to load the arinc653 scheduler, however Xen crashes whenever 
I create a CPU pool with this scheduler. I reported this and it was 
confirmed by Seeing as I've been fighting to get my little port up, 
running and communicating via shared pages with a Windows domU I've kind 
of ignored this. However before I go and invest more time and effort 
with this I just wanted to confirm that I'm not barking up the wrong 
tree.

Regards.

------ Original Message ------
From: "Andrew Cooper" <andrew.cooper3@citrix.com>
To: "Simon Martin" <smartin@milliways.cl>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Sent: 15/11/2013 08:41:44
Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer

>On 15/11/13 11:24, Simon Martin wrote:
>>Hi Andrew,
>>
>>The operating system I am trying to port to Xen is an industrial servo 
>>controller. We are currently running at 125 microsecond servo update 
>>rate on our bespoke hardware (at the moment MIPS64 and ARM) (hence the 
>>125 microsecond ideal interrupt period). We will be driving EtherCAT 
>>servo drives that need to be updated at 500 microseconds (hence at 
>>least 500 microsecond interrupt period). If we can achieve this on a 
>>single core ARM11 clocked at 532 MHz, it must be achievable on PC 
>>hardware. As I said before the idea is to dedicate a CPU core to this 
>>with all other functions running on the other cores.
>>
>>Luckily we can accept a reasonable amount of jitter on the EtherCAT 
>>network as we can hand over clock synchronisation to a slave. This 
>>gives us a little leeway.
>>
>>Regards.
>>
>>
>
>It certainly is achievable on PC hardware, but not under the standard 
>assumptions in virtualisation.  You want to use 
>VCPUOP_set_singleshot_timer, but be aware than there are still no 
>normal guarantees that your domain vcpu will be run when the timer 
>expires; the credit scheduler will pick the highest priority vcpu to 
>run, which might not be the one which is expecting a timer interrupt.
>
>You should investigate using the arinc653 scheduler in Xen, which is a 
>realtime scheduler, and might be more appropriate for your usecase.
>
>~Andrew
>
>
>>------ Original Message ------
>>From: "Andrew Cooper" <andrew.cooper3@citrix.com>
>>To: "Simon Martin" <smartin@milliways.cl>; 
>>mailto:xen-devel@lists.xen.org <xen-devel@lists.xen.org>
>>Sent: 14/11/2013 18:39:21
>>Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
>>
>>>On 14/11/2013 21:18, Simon Martin wrote:
>>>>Hi all,
>>>>
>>>>I need a periodic timer running at ideally at 125 microseconds and 
>>>>at least 500 microseconds. I've just found the 
>>>>VCPUOP_set_periodic_timer, however there is a comment saying 
>>>>"periods less than one millisecond may not be supported".
>>>>
>>>>I will be running on an x64 machine. Is this supported? If not, is 
>>>>there any alternate means of generating a fast interrupt?
>>>>
>>>>Regards.
>>>
>>>What is the usecase here?  125us is very short indeed.  Xen certainly 
>>>cant guarantee anything more accurate than 50us.  Unless the affected 
>>>vcpu is running uncontested on the hardware, there is very little 
>>>chance that the vcpu will indeed be woken up again in 125us.
>>>
>>>It sounds as if you are looking for some pseudo realtime system, at 
>>>which point you might want to consider a different scheduler.
>>>
>>>~Andrew
>>>
>

[-- Attachment #1.2: Type: text/html, Size: 6616 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 630 bytes --]

On 15/11/13 12:02, Simon Martin wrote:
> Thanks Andrew,
>  
> I've tried to load the arinc653 scheduler, however Xen crashes
> whenever I create a CPU pool with this scheduler. I reported this and
> it was confirmed by Seeing as I've been fighting to get my little port
> up, running and communicating via shared pages with a Windows domU
> I've kind of ignored this. However before I go and invest more time
> and effort with this I just wanted to confirm that I'm not barking up
> the wrong tree.
>  
> Regards.

The cpupool issue with the arinc653 scheduler is now fixed (or at least
believed to be) in xen-unstable.

~Andrew


[-- Attachment #1.2: Type: text/html, Size: 2814 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 1524 bytes --]

Latency isn't a big problem. Jitter could be. CPU offline shouldn't be a 
problem as the system is simple and static, at power up I create 2 CPU 
pools, one for the realtime machine domU running on a single core, and 
one for dom0 and a Windows machine running on the remaining cores.

Any idea on the timings I can expect? For example, assuming there are no 
CPU contention issues, if I request a 125 microsecond single shot timer 
interrupt, will I get 125 microseconds +/- 5 microseconds (perfectly 
acceptable) or 500 microseconds (because that is the minimum 
granularity) +/- 50 microseconds (totally unacceptable)?


------ Original Message ------
From: "Keir Fraser" <keir.xen@gmail.com>
To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper" 
<andrew.cooper3@citrix.com>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
Sent: 15/11/2013 08:56:18
Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer

>If you have dedicated a core to the vcpu, so you can discount 
>scheduling crosstalk, you shouldn’t miss deadlines by more than a 
>microsecond or two. Just enough time to wake up the CPU, handle the 
>timer interrupt, and wake up the vcpu through the scheduler. There are 
>infrequent operations that can take out all CPUs for a period: taking a 
>CPU offline for example. You could avoid doing such things in your 
>setup perhaps. Ultimately you will need to measure and confirm 
>deadline-to-notification latency distribution if it is critical to you.
>
>  -- Keir

[-- Attachment #1.1.2: Type: text/html, Size: 2799 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Nate Studer]

On 11/15/2013 7:17 AM, Andrew Cooper wrote:
> 
> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed
> to be) in xen-unstable.

The crashes are fixed in unstable, but pools still do not work in xen-unstable
unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus
parameter is set to 1, and cpu0 is never removed from Pool-0.

I have pools working locally without these restrictions, but I was unable to get
a patchset around before the feature freeze was declared, so I believe it is too
late to try and get it into unstable.  (George, correct me if I am wrong.)

However, I am more than happy to provide Simon with my patches to get arinc653
cpu-pools completely working with the understanding that they may change before
they make it into xen-unstable.

Also, the arinc653 scheduler currently only supports a scheduling resolution of
1ms, but it is a trivial task to change it some other resolution.

> 
> ~Andrew
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
> 

Re: VCPUOP_set_periodic_timer

[Andrew Cooper]

On 15/11/13 12:46, Nate Studer wrote:
> On 11/15/2013 7:17 AM, Andrew Cooper wrote:
>> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed
>> to be) in xen-unstable.
> The crashes are fixed in unstable, but pools still do not work in xen-unstable
> unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus
> parameter is set to 1, and cpu0 is never removed from Pool-0.
>
> I have pools working locally without these restrictions, but I was unable to get
> a patchset around before the feature freeze was declared, so I believe it is too
> late to try and get it into unstable.  (George, correct me if I am wrong.)
>
> However, I am more than happy to provide Simon with my patches to get arinc653
> cpu-pools completely working with the understanding that they may change before
> they make it into xen-unstable.
>
> Also, the arinc653 scheduler currently only supports a scheduling resolution of
> 1ms, but it is a trivial task to change it some other resolution.

Absolutely post the patches.  The worst that can happen is that they are
labelled as differed, but at least they are available for people to use.

It is George's call, but if they are small and fairly contained then
they might be fine to go in.

~Andrew

Re: VCPUOP_set_periodic_timer

[George Dunlap]

On 15/11/13 12:46, Nate Studer wrote:
> On 11/15/2013 7:17 AM, Andrew Cooper wrote:
>> The cpupool issue with the arinc653 scheduler is now fixed (or at least believed
>> to be) in xen-unstable.
> The crashes are fixed in unstable, but pools still do not work in xen-unstable
> unless the system is booted with the arinc653 scheduler, the dom0_max_vcpus
> parameter is set to 1, and cpu0 is never removed from Pool-0.
>
> I have pools working locally without these restrictions, but I was unable to get
> a patchset around before the feature freeze was declared, so I believe it is too
> late to try and get it into unstable.  (George, correct me if I am wrong.)

Bug fixes are not features. :-)

Bug fixes are accepted very late in the release process.  The only 
reason they might ever be rejected is if we think there may be a risk 
that the fix will break other working functionality -- particularly if 
we think that breakage may not be discovered until after the release.

If the fixes involve generic scheduling code, I imagine they would be 
accepted well into RC3 if they were not terribly complicated; if they 
were limited to the arinc653 scheduler itself, they could probably be 
accepted until just before the release.

  -George

Re: VCPUOP_set_periodic_timer

[Keir Fraser]

[-- Attachment #1.1.1: Type: text/plain, Size: 1983 bytes --]

You¹ll have to measure to confirm, but if the core is dedicated to your vcpu
then there is no reason there should be 50us jitter. Also there is not much
point playing with different schedulers ‹ if there are no other vcpus
schedulable on the cpu, the scheduler can¹t have multiple runnable vcpus to
choose between.

 -- Keir

On 15/11/2013 12:37, "Simon Martin" <smartin@milliways.cl> wrote:

> Latency isn't a big problem. Jitter could be. CPU offline shouldn't be a
> problem as the system is simple and static, at power up I create 2 CPU pools,
> one for the realtime machine domU running on a single core, and one for dom0
> and a Windows machine running on the remaining cores.
>  
> Any idea on the timings I can expect? For example, assuming there are no CPU
> contention issues, if I request a 125 microsecond single shot timer interrupt,
> will I get 125 microseconds +/- 5 microseconds (perfectly acceptable) or 500
> microseconds (because that is the minimum granularity) +/- 50 microseconds
> (totally unacceptable)?
>  
>  
> ------ Original Message ------
> From: "Keir Fraser" <keir.xen@gmail.com>
> To: "Simon Martin" <smartin@milliways.cl>; "Andrew Cooper"
> <andrew.cooper3@citrix.com>
> Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>
> Sent: 15/11/2013 08:56:18
> Subject: Re: Re[2]: [Xen-devel] VCPUOP_set_periodic_timer
>  
>> If you have dedicated a core to the vcpu, so you can discount scheduling
>> crosstalk, you shouldn¹t miss deadlines by more than a microsecond or two.
>> Just enough time to wake up the CPU, handle the timer interrupt, and wake up
>> the vcpu through the scheduler. There are infrequent operations that can take
>> out all CPUs for a period: taking a CPU offline for example. You could avoid
>> doing such things in your setup perhaps.  Ultimately you will need to measure
>> and confirm deadline-to-notification latency distribution if it is critical
>> to you. 
>> 
>>  -- Keir
> 


[-- Attachment #1.1.2: Type: text/html, Size: 2946 bytes --]

[-- Attachment #1.2: image.png --]
[-- Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Andrew Cooper]

[-- Attachment #1.1: Type: text/plain, Size: 613 bytes --]

On 15/11/13 13:10, Keir Fraser wrote:
> Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You'll have to
> measure to confirm, but if the core is dedicated to your vcpu then
> there is no reason there should be 50us jitter. Also there is not much
> point playing with different schedulers --- if there are no other
> vcpus schedulable on the cpu, the scheduler can't have multiple
> runnable vcpus to choose between.
>
>  -- Keir

The deadline timer code as up to 50us of slop when calculating when to
wake up and when timers expire.  It is somewhat poor, and is certainly
an area in need of improvement.

~Andrew

[-- Attachment #1.2: Type: text/html, Size: 1282 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Keir Fraser]

On 15/11/2013 13:13, "Andrew Cooper" <andrew.cooper3@citrix.com> wrote:

>    
> On 15/11/13 13:10, Keir Fraser wrote:
>  
>  
>>   Re: Re[4]: [Xen-devel] VCPUOP_set_periodic_timer You¹ll have to measure to
>> confirm, but if the core is dedicated to your vcpu then there is no reason
>> there should be 50us jitter. Also there is not much point playing with
>> different schedulers ‹ if there are no other vcpus schedulable on the cpu,
>> the scheduler can¹t have multiple runnable vcpus to choose between.
>>  
>>   -- Keir
>>  
>  
>  The deadline timer code as up to 50us of slop when calculating when to wake
> up and when timers expire.  It is somewhat poor, and is certainly an area in
> need of improvement.

Forgot about that. For real-time applications, timer_slop=0 on the Xen
command line will turn it off. But there could certainly be other gotchas
lurking; actual performance needs to be measured.

 -- Keir

>  ~Andrew
>  

Re: VCPUOP_set_periodic_timer

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 4838 bytes --]

On ven, 2013-11-15 at 11:41 +0000, Andrew Cooper wrote:
> On 15/11/13 11:24, Simon Martin wrote: 
> > The operating system I am trying to port to Xen is an industrial
> > servo controller. We are currently running at 125 microsecond servo
> > update rate on our bespoke hardware (at the moment MIPS64 and ARM)
> > (hence the 125 microsecond ideal interrupt period). We will be
> > driving EtherCAT servo drives that need to be updated at 500
> > microseconds (hence at least 500 microsecond interrupt period). If
> > we can achieve this on a single core ARM11 clocked at 532 MHz, it
> > must be achievable on PC hardware. As I said before the idea is to
> > dedicate a CPU core to this with all other functions running on the
> > other cores.
> >  
> > Luckily we can accept a reasonable amount of jitter on the EtherCAT
> > network as we can hand over clock synchronisation to a slave. This
> > gives us a little leeway.
> >  
> > Regards.
> >  
> 
> You should investigate using the arinc653 scheduler in Xen, which is a
> realtime scheduler, and might be more appropriate for your usecase.
> 
Right, or even SEDF, if only it were functional! :-/

That's another area where we really need to do better. Luckily enough,
we've got Roland, Joshua and Drek (from GVSU and HsH) working on trying
to 'resurrect' the SEDF scheduler.

Coming to Simon's situation, I think that, as it has already been said,
if you can partition the system in such a way that the real-time VM/OS
gets one full core, without any other interference (either via cpupool
or pinning), there is few point in changing scheduler or scheduling
parameters, and that's probably the setup that would guarantee the most
reliable performances.

If that is not the case, you sure should check arinc, which is probably
a rather "static" algorithm, but I'm quite ignorante (sigh) about how it
actually works, so I can't say anything about it (and I see Nathan is on
it already, so you're in good hands :-)).

If you need something more advanced, while waiting for SEDF to be
'restored', you perhaps can check at RT-Xen (Sisu, one of the main
developers, is also in Cc). Basically, with that, you can enable some
SEDF-alike schedulers, with even more complex and advanced
functionalities.

https://sites.google.com/site/realtimexen/

These schedulers allow you to specify, for each vcpu, a period and a
certain amount of cpu time it will get every period (right Sisu?). This
may turn out to be handy in your case, because you then will have the
guarantee that the vcpu will get a chance to run at least once every
<period> time units. At this point coming up with a period that
guarantees the time granularity that you need shouldn't be too hard,
especially is the system is after all pretty simple, as it looks like.

Of course, it remains to be verified whether the scheduling parameters
can tolerate being set at the small values that your workload
requires... But you cannot get along with a RT-ish system without doing
some measurements, right? :-)

Regards,
Dario

> ~Andrew
> 
> 
> > ------ Original Message ------
> > From: "Andrew Cooper" <andrew.cooper3@citrix.com>
> > To: "Simon Martin" <smartin@milliways.cl>; "xen-devel@lists.xen.org"
> > <xen-devel@lists.xen.org>
> > Sent: 14/11/2013 18:39:21
> > Subject: Re: [Xen-devel] VCPUOP_set_periodic_timer
> >  
> > > On 14/11/2013 21:18, Simon Martin wrote:
> > > 
> > > > Hi all,
> > > >  
> > > > I need a periodic timer running at ideally at 125 microseconds
> > > > and at least 500 microseconds. I've just found the
> > > > VCPUOP_set_periodic_timer, however there is a comment saying
> > > > "periods less than one millisecond may not be supported".
> > > >  
> > > > I will be running on an x64 machine. Is this supported? If not,
> > > > is there any alternate means of generating a fast interrupt?
> > > >  
> > > > Regards.
> > > 
> > > What is the usecase here?  125us is very short indeed.  Xen
> > > certainly cant guarantee anything more accurate than 50us.  Unless
> > > the affected vcpu is running uncontested on the hardware, there is
> > > very little chance that the vcpu will indeed be woken up again in
> > > 125us.
> > > 
> > > It sounds as if you are looking for some pseudo realtime system,
> > > at which point you might want to consider a different scheduler.
> > > 
> > > ~Andrew
> > > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 2653 bytes --]


>Coming to Simon's situation, I think that, as it has already been said,
>if you can partition the system in such a way that the real-time VM/OS
>gets one full core, without any other interference (either via cpupool
>or pinning), there is few point in changing scheduler or scheduling
>parameters, and that's probably the setup that would guarantee the most
>reliable performances.
This is the configuration I am working on. The system is reasonably 
simple, a single flat memory space with it's own simple scheduler, so I 
can't see why it shouldn't work, as long as I can get the granularity 
out of the underlying system.


>
>If that is not the case, you sure should check arinc, which is probably
>a rather "static" algorithm, but I'm quite ignorante (sigh) about how 
>it
>actually works, so I can't say anything about it (and I see Nathan is 
>on
>it already, so you're in good hands ).
In this kind of environment, a static algorithm is good. It is much 
better to be predictable than flexible. In an industrial setting where 
this is controlling machinery that can potentially kill the operators 
the principle of least surprise is what you need.


>
>If you need something more advanced, while waiting for SEDF to be
>'restored', you perhaps can check at RT-Xen (Sisu, one of the main
>developers, is also in Cc). Basically, with that, you can enable some
>SEDF-alike schedulers, with even more complex and advanced
>functionalities.
Before I embarked on this I looked at RealTime Xen. Very interesting 
project, however from my understanding of the Xen architecture I thought 
that the vanilla Xen would be sufficient. This is still my fallback 
position.

>
>Of course, it remains to be verified whether the scheduling parameters
>can tolerate being set at the small values that your workload
>requires... But you cannot get along with a RT-ish system without doing
>some measurements, right?
This is what is missing. There is a steep learning curve on how to write 
a PV guest. The mini-os is a good starting point, and so is David 
Chisnall's book, however in many ways the mini-os is overkill for what I 
want as a proof of concept where I want to do the bare minimum to 
interface with Xen and just work on the fixed flat memory space that it 
initialises with.

Once I have this working smoothly I'll look at timings, however I really 
needed to know whether what I am asking is feasible or not. From all the 
answers I think that it is more than feasible, not only due to the Xen 
infrastructure, but also because of the good will of the people 
involved.

Thanks very much.


[-- Attachment #1.1.2: Type: text/html, Size: 4273 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 685 bytes --]

[-- Attachment #1.3: Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: VCPUOP_set_periodic_timer

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1803 bytes --]

On sab, 2013-11-16 at 20:37 +0000, Simon Martin wrote:

> This is the configuration I am working on. The system is reasonably
> simple, a single flat memory space with it's own simple scheduler, so
> I can't see why it shouldn't work, as long as I can get the
> granularity out of the underlying system.
>
It indeed should work. It's the fact that the "underlying system" is an
hypervisor that makes things fun! :-)

There is quite some interest in enabling at least a certain level of
real-time behavior in/on top of Xen these days (mostly thanks to the ARM
port), which means a lot of people will be looking forward of what
you'll be able to achieve and how.

> Before I embarked on this I looked at RealTime Xen. Very interesting
> project, however from my understanding of the Xen architecture I
> thought that the vanilla Xen would be sufficient. This is still my
> fallback position.
>
Yep, it's good to have alternatives. Long term, I think we should aim at
both fixing SEDF and having something like RT-Xen merged upstream
(people are working on both these projects already).
 
> Once I have this working smoothly I'll look at timings, however I
> really needed to know whether what I am asking is feasible or not.
> From all the answers I think that it is more than feasible, not only
> due to the Xen infrastructure, but also because of the good will of
> the people involved.
>
HeHe... Nice to hear you feel comfortable with us. :-)

Keep us posted on how it goes.

Thanks and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

PV guest timings

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 1389 bytes --]

Hi all,

I have finally got my little PV guest running. My critical error was not 
setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and 
put it back together a few times I decided to debug my Makefile script 
comparing it to Mini-OS.

I am now starting to do some simple timings and the numbers are so bad 
I'm wondering what could be wrong with my test.

As mentioned previously I have created a CPU pool with one CPU and one 
domU. I am using the standard credit scheduler. I set timer_slop=0 on 
the Xen command line.

I initialise console, traps, events, and TSC clock in my PV and then 
start a periodic operation running. I then calculate latency as (clock 
time - deadline) and period as (clock time - previous deadline).At the 
moment I'm just displaying min/max values. I also increment a tick count 
on every cycle.

Looking at the statistics I see that I get the expected number of ticks 
per second, however I get latencies in the range [-1ms, +25us] and the 
periods in the range of [3us, 1.02ms]. The upper bounds look OK, but the 
lower bounds are all over the place.

This makes me think that I'm not the only thing using VIRQ_TIMER. I seem 
to remember that this is called nominally every 10ms by the Hypervisor.

Is there any way to recognize the origin of the timer?

Can anyone suggest other things to try?

Regards.

[-- Attachment #1.2: Type: text/html, Size: 2725 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Keir Fraser]

On 26/11/2013 14:50, "Simon Martin" <smartin@milliways.cl> wrote:

> Hi all,
>  
> I have finally got my little PV guest running. My critical error was not
> setting __XEN_INTERFACE_VERSION__. After tearing to pieces the code and put it
> back together a few times I decided to debug my Makefile script comparing it
> to Mini-OS.
>  
> I am now starting to do some simple timings and the numbers are so bad I'm
> wondering what could be wrong with my test.
>  
> As mentioned previously I have created a CPU pool with one CPU and one domU. I
> am using the standard credit scheduler. I set timer_slop=0 on the Xen command
> line.
>  
> I initialise console, traps, events, and TSC clock in my PV and then start a
> periodic operation running. I then calculate latency as (clock time -
> deadline) and period as (clock time - previous deadline).At the moment I'm
> just displaying min/max values. I also increment a tick count on every cycle.
>  
> Looking at the statistics I see that I get the expected number of ticks per
> second, however I get latencies in the range [-1ms, +25us] and the periods in
> the range of [3us, 1.02ms]. The upper bounds look OK, but the lower bounds are
> all over the place.
>  
> This makes me think that I'm not the only thing using VIRQ_TIMER. I seem to
> remember that this is called nominally every 10ms by the Hypervisor.
>  
> Is there any way to recognize the origin of the timer?
>  
> Can anyone suggest other things to try?

Are you using the single-shot timer or the periodic timer?

How are you calculating clock time?

 -- Keir

> Regards.
>  
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 708 bytes --]

Hi Keir,

>Are you using the single-shot timer or the periodic timer?
I'm using the single-shot timer.

>
>How are you calculating clock time?
Copied code in from Mini-OS. To make sure I don't get drift between 
deadlines as this is a single shot timer I read monotonic_clock at 
startup and then calculate every deadline by incrementing this register 
by the period on each timer call.

As an additional comment. I decided to filter out the samples with 
negative latency. This seems to happen 100 times a second, i.e. the 10 
ms Hypervisor timer event. This improves the timings considerably. So it 
looks like I'm on the right track looking for ways to distinguish 
between timer events.

[-- Attachment #1.2: Type: text/html, Size: 1730 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2858 bytes --]

On mar, 2013-11-26 at 14:50 +0000, Simon Martin wrote:
> Hi all,
>  
Hi Simon!

> I have finally got my little PV guest running. My critical error was
> not setting __XEN_INTERFACE_VERSION__. After tearing to pieces the
> code and put it back together a few times I decided to debug my
> Makefile script comparing it to Mini-OS.
>  
Cool! :-)

> I am now starting to do some simple timings and the numbers are so bad
> I'm wondering what could be wrong with my test. 
>
Not so cool! :-(
 
> As mentioned previously I have created a CPU pool with one CPU and one
> domU. I am using the standard credit scheduler. I set timer_slop=0 on
> the Xen command line.
>  
> I initialise console, traps, events, and TSC clock in my PV and then
> start a periodic operation running. I then calculate latency as (clock
> time - deadline) and period as (clock time - previous deadline).At the
> moment I'm just displaying min/max values. I also increment a tick
> count on every cycle.
>  
> Looking at the statistics I see that I get the expected number of
> ticks per second, however I get latencies in the range [-1ms, +25us]
> and the periods in the range of [3us, 1.02ms]. The upper bounds look
> OK, but the lower bounds are all over the place.
>
Ok. Well, I'm not super expert in that area (yet), but even for the ones
that are, I think it's pretty hard to guess what could be going on
without seeing the actual code.

Is there any chance that you can share/show it? At least the relevant
chunks... Some questions:
 - "looking at the statistics", what statistics? How do you collect 
   them?
 - you say you get latencies in some range and periods in some other
   range. It may be my fault, but I'm not sure I understand what you
   mean with these two terms in this context, could you clarify?
 
> This makes me think that I'm not the only thing using VIRQ_TIMER. I
> seem to remember that this is called nominally every 10ms by the
> Hypervisor.
>  
> Is there any way to recognize the origin of the timer?
>
I'm not sure if fits your exact needs, and you probably know it already,
but, just in case, have you looked at xentrace and xenalyze? There's a
blog post that could be a nice introduction to them:
http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/

If the exact information you're after is not among the already existing
tracepoints, it shouldn't be to hard to add one or more of those, in the
spots where you need them.

But again, it's very hard to judge without seeing the code...

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 546 bytes --]

On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote:


> As an additional comment. I decided to filter out the samples with
> negative latency. This seems to happen 100 times a second, i.e. the 10
> ms Hypervisor timer event.
>
Again, what's "negative latency" ?

Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 447 bytes --]


>On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote:
>
>
>>  As an additional comment. I decided to filter out the samples with
>>  negative latency. This seems to happen 100 times a second, i.e. the 
>>10
>>  ms Hypervisor timer event.
>>
>Again, what's "negative latency" ?
Latency is the time between the deadline and the handler being called. 
Negative latency means the handler was called before the next deadline.

Regards

[-- Attachment #1.2: Type: text/html, Size: 1458 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 2511 bytes --]


>>
>>  Looking at the statistics I see that I get the expected number of
>>  ticks per second, however I get latencies in the range [-1ms, +25us]
>>  and the periods in the range of [3us, 1.02ms]. The upper bounds look
>>  OK, but the lower bounds are all over the place.
>>
>Ok. Well, I'm not super expert in that area (yet), but even for the 
>ones
>that are, I think it's pretty hard to guess what could be going on
>without seeing the actual code.
>
>Is there any chance that you can share/show it? At least the relevant
>chunks...
No problems, I can share the complete code it you want, there is nothing 
proprietary in there yet. What's the best way to do it? Put it on 
github? Send a tarball?

>Some questions:
>  - "looking at the statistics", what statistics? How do you collect
>    them?
At every cycle I compare the current time with the expected time and 
calculate latency (time from when I expect the event to when it actually 
happens) and period.

>  - you say you get latencies in some range and periods in some other
>    range. It may be my fault, but I'm not sure I understand what you
>    mean with these two terms in this context, could you clarify?
>
Latency = difference between expected time and actual time of the event.

Period = time between two consecutive events.
>>  This makes me think that I'm not the only thing using VIRQ_TIMER. I
>>  seem to remember that this is called nominally every 10ms by the
>>  Hypervisor.
>>
>>  Is there any way to recognize the origin of the timer?
>>
>I'm not sure if fits your exact needs, and you probably know it 
>already,
>but, just in case, have you looked at xentrace and xenalyze? There's a
>blog post that could be a nice introduction to them:
>http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
>
I hadn't read that. I'll look at it tomorrow (it's almost midnight 
here).

>If the exact information you're after is not among the already existing
>tracepoints, it shouldn't be to hard to add one or more of those, in 
>the
>spots where you need them.
>
>But again, it's very hard to judge without seeing the code...
I totally understand. Thanks for the effort.

>
>Regards,
>Dario
>
>--
><<This happens because I choose it to happen!>> (Raistlin Majere)
>-----------------------------------------------------------------
>Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>

[-- Attachment #1.2: Type: text/html, Size: 4852 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1354 bytes --]

On mer, 2013-11-27 at 02:32 +0000, Simon Martin wrote:
>  
> > On mar, 2013-11-26 at 15:38 +0000, Simon Martin wrote: 
> >  
> >  
> > >  As an additional comment. I decided to filter out the samples
> > > with 
> > >  negative latency. This seems to happen 100 times a second, i.e.
> > > the 10 
> > >  ms Hypervisor timer event. 
> > >  
> > Again, what's "negative latency" ? 
>
> Latency is the time between the deadline and the handler being called.
>
Ok, that's a reasonable definition of latency. However, if you don't
mind another question, 'deadline' here is basically the deadline of the
periodic instance x-1, and you expect a timer firing at that time in
order to activate instance x? Or is that a proper deadline (and the
above is just called period) and you want to have some handler running
when it's missed?

>  Negative latency means the handler was called before the next
> deadline.
>  
Oh, wow... I see. So you're saying that you have timers firing early
than their expiry time? Is that expected? Why is that happening? :-O

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2714 bytes --]

On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote:

> >  
> > Is there any chance that you can share/show it? At least the
> > relevant 
> > chunks... 
> No problems, I can share the complete code it you want, there is
> nothing proprietary in there yet. What's the best way to do it? Put it
> on github? Send a tarball?
>
Well, I think a repo somewhere would be preferrable over a tarball,
espacially if you have it in a (although private) repo already, so that
we gt to see the history, the commit changelogs, etc.

Also, if you go for github, or whatever similar service, people would be
able to both checkout your code, or browse it on-line, depending on what
they like most.

If you don't, I still personally prefer repositories, but, really,
whatever you're more comfortable with would do.

> > Some questions: 
> >  - "looking at the statistics", what statistics? How do you collect 
> >    them? 
> At every cycle I compare the current time with the expected time and
> calculate latency (time from when I expect the event to when it
> actually happens) and period.
>  
Sure, that part I got. I was more asking how you get these numbers out,
i.e., you print them online, store somewhere and print later/on request,
etc. (I guess I could have specified this more clearly, sorry about
that :-) )

> >  - you say you get latencies in some range and periods in some
> > other 
> >    range. It may be my fault, but I'm not sure I understand what
> > you 
> >    mean with these two terms in this context, could you clarify? 
> >   
> Latency = difference between expected time and actual time of the
> event.
>  
> Period = time between two consecutive events. 
>
Ok, I think I see now. Just to confirm that I do, this means that the
values/ranges you get by measuring both are related, i.e., not
completely independent, right? Again, I'm not saying it's not useful to
have both, just trying to double check I understood what you're doing.

> > I'm not sure if fits your exact needs, and you probably know it
> > already, 
> > but, just in case, have you looked at xentrace and xenalyze? There's
> > a 
> > blog post that could be a nice introduction to them: 
> > http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/ 
> >  
> I hadn't read that. I'll look at it tomorrow (it's almost midnight
> here).
>  
Ask again if you need anything.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[David Vrabel]

On 26/11/2013 15:38, Simon Martin wrote:
> Hi Keir,
>  
>> Are you using the single-shot timer or the periodic timer? 
> I'm using the single-shot timer.
>  
>>  
>> How are you calculating clock time? 
> Copied code in from Mini-OS. To make sure I don't get drift between
> deadlines as this is a single shot timer I read monotonic_clock at
> startup and then calculate every deadline by incrementing this register
> by the period on each timer call.
>  
> As an additional comment. I decided to filter out the samples with
> negative latency. This seems to happen 100 times a second, i.e. the 10
> ms Hypervisor timer event. This improves the timings considerably. So it
> looks like I'm on the right track looking for ways to distinguish
> between timer events.

You should disable the periodic timer if you don't need it.

VCPUOP_stop_periodic_timer

David

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 1267 bytes --]


>>  Latency is the time between the deadline and the handler being 
>>called.
>>
>Ok, that's a reasonable definition of latency. However, if you don't
>mind another question, 'deadline' here is basically the deadline of the
>periodic instance x-1, and you expect a timer firing at that time in
>order to activate instance x? Or is that a proper deadline (and the
>above is just called period) and you want to have some handler running
>when it's missed?
Each time the timer handler is called it calculates the deadline for the 
next handler and calls VCPUOP_set_singleshot_timer specifying the 
deadline. The deadline is calculated as the "previous deadline + the 
period". So handler instance 'n' will set the deadline for handler 
instance 'n+1' as "deadline n + period".

I'm not sure if that answers the question, but that is what is 
happening.


>
>>   Negative latency means the handler was called before the next
>>  deadline.
>>
>Oh, wow... I see. So you're saying that you have timers firing early
>than their expiry time? Is that expected? Why is that happening?
Yes. As I said I think this is the 100 Hz Hypervisor periodic timer. I 
need to be able to distinguish between the different timer interrupt 
sources.

Regards.

[-- Attachment #1.1.2: Type: text/html, Size: 2605 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 692 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 3839 bytes --]



------ Original Message ------
From: "Dario Faggioli" <dario.faggioli@citrix.com>
To: "Simon Martin" <smartin@milliways.cl>
Cc: "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>; "Andrew Cooper" 
<andrew.cooper3@citrix.com>; "Roland Heusser" <heusserr@mail.gvsu.edu>; 
"Sisu Xi" <xisisu@gmail.com>; "Joshua Whitehead" 
<whitehej@mail.gvsu.edu>; "Drek Darkover" <wackerei@gmail.com>; "Nate 
Studer" <nate.studer@dornerworks.com>
Sent: 27/11/2013 05:56:26
Subject: Re: Re[2]: [Xen-devel] PV guest timings

>On mer, 2013-11-27 at 02:36 +0000, Simon Martin wrote:
>
>>  >
>>  > Is there any chance that you can share/show it? At least the
>>  > relevant
>>  > chunks...
>>  No problems, I can share the complete code it you want, there is
>>  nothing proprietary in there yet. What's the best way to do it? Put 
>>it
>>  on github? Send a tarball?
>>
>Well, I think a repo somewhere would be preferrable over a tarball,
>espacially if you have it in a (although private) repo already, so that
>we gt to see the history, the commit changelogs, etc.
Here's the URL. https://github.com/FurryFuttock/micro-pv. There is no 
commit history as I mostly use SVN. I just created this and will keep it 
up to date as I carry on playing with it.

>>  > Some questions:
>>  > - "looking at the statistics", what statistics? How do you collect
>>  > them?
>>  At every cycle I compare the current time with the expected time and
>>  calculate latency (time from when I expect the event to when it
>>  actually happens) and period.
>>
>Sure, that part I got. I was more asking how you get these numbers out,
>i.e., you print them online, store somewhere and print later/on 
>request,
>etc. (I guess I could have specified this more clearly, sorry about
>that )
Here's an example. There are 2 lines per sample. Line 0 has format 
<time> <ticks per second>, line 1 has <latency min/max>,<period min/max>

timeofday=12:56:41 - 1000
1043,47287,955121,2001800
timeofday=12:56:42 - 1000
1024,30989,970082,2000885
timeofday=12:56:43 - 1000
1042,30888,970167,2000947
timeofday=12:56:44 - 1000
1042,30910,970661,2001063
timeofday=12:56:45 - 1000
1043,30909,970162,2000057
timeofday=12:56:46 - 1000
1043,30883,970183,2000049
timeofday=12:56:47 - 1000
1066,30923,970160,2001268
timeofday=12:56:48 - 1000
1048,30922,970140,2000031
timeofday=12:56:49 - 1000
1041,30864,970211,2001090
timeofday=12:56:50 - 1000
1043,30898,970162,2000880
timeofday=12:56:51 - 1000
1042,30948,970106,2000947
timeofday=12:56:52 - 1000
1044,30777,970752,2000058
>
>>  > - you say you get latencies in some range and periods in some
>>  > other
>>  > range. It may be my fault, but I'm not sure I understand what
>>  > you
>>  > mean with these two terms in this context, could you clarify?
>>  >
>>  Latency = difference between expected time and actual time of the
>>  event.
>>
>>  Period = time between two consecutive events.
>>
>Ok, I think I see now. Just to confirm that I do, this means that the
>values/ranges you get by measuring both are related, i.e., not
>completely independent, right? Again, I'm not saying it's not useful to
>have both, just trying to double check I understood what you're doing.
They're not really related. Period is time between events, Latency is 
time from event to handler.

>
>>  > I'm not sure if fits your exact needs, and you probably know it
>>  > already,
>>  > but, just in case, have you looked at xentrace and xenalyze? 
>>There's
>>  > a
>>  > blog post that could be a nice introduction to them:
>>  > 
>>http://blog.xen.org/index.php/2012/09/27/tracing-with-xentrace-and-xenalyze/
>>  >
>>  I hadn't read that. I'll look at it tomorrow (it's almost midnight
>>  here).
>>
>Ask again if you need anything.
I will ! Thanks.

Regards.

[-- Attachment #1.1.2: Type: text/html, Size: 6936 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 685 bytes --]

[-- Attachment #1.3: Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1: Type: text/plain, Size: 217 bytes --]


>You should disable the periodic timer if you don't need it.
>
>VCPUOP_stop_periodic_timer
>
>David
Perfect. Thanks David, that got rid of my strange counts. I now only 
have one VIRQ_TIMER source.

Regards.

[-- Attachment #1.2: Type: text/html, Size: 1126 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 3684 bytes --]

On mer, 2013-11-27 at 13:00 +0000, Simon Martin wrote:
 ------ Original Message ------
> From: "Dario Faggioli" <dario.faggioli@citrix.com>

> > > 
> > Well, I think a repo somewhere would be preferrable over a tarball, 
> > espacially if you have it in a (although private) repo already, so
> > that 
> > we gt to see the history, the commit changelogs, etc. 
> Here's the URL. https://github.com/FurryFuttock/micro-pv. There is no
> commit history as I mostly use SVN. I just created this and will keep
> it up to date as I carry on playing with it.
>
Once more, thanks for this. It looks really interesting and, if you ask
me, I'll sure it will be useful!

I took a quick look at it, and added to my TODO list to properly look
into it. Hopefully, it won't take that long. :-)

> > Sure, that part I got. I was more asking how you get these numbers
> > out, 
> > i.e., you print them online, store somewhere and print later/on
> > request, 
> > etc. (I guess I could have specified this more clearly, sorry about 
> > that :-) ) 
> Here's an example. There are 2 lines per sample. Line 0 has format
> <time> <ticks per second>, line 1 has <latency min/max>,<period
> min/max>
>  
> timeofday=12:56:41 - 1000
> 1043,47287,955121,2001800
> timeofday=12:56:42 - 1000
> 1024,30989,970082,2000885
> timeofday=12:56:43 - 1000
> 1042,30888,970167,2000947
> ...
>
Ok, I was wondering whether, e.g., the printing could interfere with the
measurements, and stuff like that, but I checked this out in your code,
and it doesn't look like it is happening.

> > Ok, I think I see now. Just to confirm that I do, this means that
> > the 
> > values/ranges you get by measuring both are related, i.e., not 
> > completely independent, right? Again, I'm not saying it's not useful
> > to 
> > have both, just trying to double check I understood what you're
> > doing. 
> They're not really related. Period is time between events, Latency is
> time from event to handler.
>
"Period is time between events", what events? I was thinking the events
to be two subsequent invocation of the handler. In an ideal world, that
would be exactly equal to the PERIOD that you choose, which also mean
latency would be 0. OTOH, if you get some latency in invoking the
handler, you'll see different values for the actual time difference
between different invocations... Otherwise, how could period vary over
time?

I mean, if at t0 you set the next event to occur at

 t1=t0+T,

still in the ideal world, you'll get the event both triggered and
handled exactly at t1, i.e.,

 period: p1=t0+T-t0=T,
 latency: l1=t1-t1=0.

If, OTOH, you get some latency, i.e., you are processing the event in
the handler at

 t1'=t0+T+l1,

that means

 latency: t1'-t1=t0+T+l1-t0-T=l1,

and that also makes the distance between the two handling event

 p1'=t1'-t0=t0+T+l1-t0=T+l1.

Going further on, you get

 t2=t1+T=t0+T+T=t0+2T
 t2'=t2+l2

and thus

 p2'=t2'-t1'=t0+2T+l2-t0-T-l1=T+l2-l1

That's what I meant with "related".

Checking the code (but very quickly, so bear with me if I'm getting
something wrong), _BUT_, I totally understand that this relationship is
only useful if you perform the calculation a bit differently, and the
way you're doing them allows you to monitor both latency and jitter good
enough anyway, so I'm cool with it. :-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)


[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: PV guest timings

[Simon Martin]

[-- Attachment #1.1.1: Type: text/plain, Size: 3247 bytes --]


>Once more, thanks for this. It looks really interesting and, if you ask
>me, I'll sure it will be useful!
>
>I took a quick look at it, and added to my TODO list to properly look
>into it. Hopefully, it won't take that long.
Thanks. All feedback is welcome.

>
>>
>>  timeofday=12:56:41 - 1000
>>  1043,47287,955121,2001800
>>  timeofday=12:56:42 - 1000
>>  1024,30989,970082,2000885
>>  timeofday=12:56:43 - 1000
>>  1042,30888,970167,2000947
>>  ...
>>
>Ok, I was wondering whether, e.g., the printing could interfere with 
>the
>measurements, and stuff like that, but I checked this out in your code,
>and it doesn't look like it is happening.
It shouldn't. I calculate numbers in the periodic interrupt and then 
print them from the normal run loop.

>
>>  > Ok, I think I see now. Just to confirm that I do, this means that
>>  > the
>>  > values/ranges you get by measuring both are related, i.e., not
>>  > completely independent, right? Again, I'm not saying it's not 
>>useful
>>  > to
>>  > have both, just trying to double check I understood what you're
>>  > doing.
>>  They're not really related. Period is time between events, Latency is
>>  time from event to handler.
>>
>"Period is time between events", what events?
As you say, events in this context are subsequent invocations of the 
handler.

>In an ideal world, that
>would be exactly equal to the PERIOD that you choose, which also mean
>latency would be 0. OTOH, if you get some latency in invoking the
>handler, you'll see different values for the actual time difference
>between different invocations... Otherwise, how could period vary over
>time?
Not quite. Let us say that latency is always 5 µs, no jitter. That means 
that the measured period will always be exactly the same, just slightly 
delayed in time. This is fine.

However there IS jitter in the latency (+/- 30 µs) so the period varies 
likewise. One event comes in late, the next comes in on time, period is 
less than expected. One comes in on time the next comes in late, period 
is longer than expected.

>
>I mean, if at t0 you set the next event to occur at
>
>  t1=t0+T,
>
>still in the ideal world, you'll get the event both triggered and
>handled exactly at t1, i.e.,
>
>  period: p1=t0+T-t0=T,
>  latency: l1=t1-t1=0.
>
>If, OTOH, you get some latency, i.e., you are processing the event in
>the handler at
>
>  t1'=t0+T+l1,
>
>that means
>
>  latency: t1'-t1=t0+T+l1-t0-T=l1,
>
>and that also makes the distance between the two handling event
>
>  p1'=t1'-t0=t0+T+l1-t0=T+l1.
>
>Going further on, you get
>
>  t2=t1+T=t0+T+T=t0+2T
>  t2'=t2+l2
>
>and thus
>
>  p2'=t2'-t1'=t0+2T+l2-t0-T-l1=T+l2-l1
>
>That's what I meant with "related".
I agree, it is not a direct relationship, but yes they are related.

>
>Checking the code (but very quickly, so bear with me if I'm getting
>something wrong), _BUT_, I totally understand that this relationship is
>only useful if you perform the calculation a bit differently, and the
>way you're doing them allows you to monitor both latency and jitter 
>good
>enough anyway, so I'm cool with it.
I'm glad I haven't got it too wrong at the first attempt.

[-- Attachment #1.1.2: Type: text/html, Size: 6338 bytes --]

[-- Attachment #1.2: Type: image/png, Size: 685 bytes --]

[-- Attachment #1.3: Type: image/png, Size: 685 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel