About profiling xen

About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 527 bytes --]

Hi there,
I am getting very low throughput (around 0.29Mbps) while running netperf benchmark for guest to guest communication on a single physical host. To analyse where the time is spent in hypervisor I wanna use profiling. Please help me choosing good profiler according to my requirments. Should it be better in my case, Xentrace, gprof, xenoprof  or Oprofile. Is it possible to use some vtune version on FC11?
 
Regards,
Fasiha Ashraf


      Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 812 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: [Xen-users] About profiling xen

[Marco Tizzoni]

[-- Attachment #1.1: Type: text/plain, Size: 1395 bytes --]

I experienced the same problem with netperf and with hping also. In general
every tool that sends packet at fixed rates seems to suffer this problem. If
you use the flood option ( sends packets as fast as possible - for example
'ping -f' or 'hping3 -f') instead of a fixed rate the rate become much
higher. What I discovered is this issue should be related to Xen's signal
handling/delivery.
Netperf use setitimer() to send packets at fixed rate. In my experience on
dom0/U, with no load, Xen cannot go over 250 signals per second. Because
netperf use signal wakeup the function that send packets I think your
problem can be related to this issue. If you want to be sure, count the
average number of packet per seconds. It should be about 250 packets/s (not
more).
For more info you can see this:
http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html

Regards.
Marco

On Wed, Sep 30, 2009 at 8:07 AM, Fasiha Ashraf <feehapk@yahoo.co.in> wrote:

>  Hi there,
> I am getting very low throughput (around 0.29Mbps) while running netperf
> benchmark for guest to guest communication on a single physical host. To
> analyse where the time is spent in hypervisor I wanna use profiling. Please
> help me choosing good profiler according to my requirments. Should it be
> better in my case, Xentrace, gprof, xenoprof  or Oprofile. Is it possible to
> use some vtune version on FC11?
>

[-- Attachment #1.2: Type: text/html, Size: 2096 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] About profiling xen

[Pasi Kärkkäinen]

On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote:
>    I experienced the same problem with netperf and with hping also. In
>    general every tool that sends packet at fixed rates seems to suffer this
>    problem. If you use the flood option ( sends packets as fast as possible -
>    for example 'ping -f' or 'hping3 -f') instead of a fixed rate the rate
>    become much higher. What I discovered is this issue should be related to
>    Xen's signal handling/delivery.
>    Netperf use setitimer() to send packets at fixed rate. In my experience on
>    dom0/U, with no load, Xen cannot go over 250 signals per second. Because
>    netperf use signal wakeup the function that send packets I think your
>    problem can be related to this issue. If you want to be sure, count the
>    average number of packet per seconds. It should be about 250 packets/s
>    (not more).


250 per second.. sounds like kernel HZ value? 

-- Pasi

>    For more info you can see
>    this:[1]http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html
> 
>    Regards.
>    Marco
> 
>    On Wed, Sep 30, 2009 at 8:07 AM, Fasiha Ashraf <[2]feehapk@yahoo.co.in>
>    wrote:
> 
>      Hi there,
>      I am getting very low throughput (around 0.29Mbps) while running netperf
>      benchmark for guest to guest communication on a single physical host. To
>      analyse where the time is spent in hypervisor I wanna use profiling.
>      Please help me choosing good profiler according to my requirments.
>      Should it be better in my case, Xentrace, gprof, xenoprof  or Oprofile.
>      Is it possible to use some vtune version on FC11?
> 
> References
> 
>    Visible links
>    1. http://lists.xensource.com/archives/html/xen-devel/2009-09/msg01159.html
>    2. mailto:feehapk@yahoo.co.in

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote:
>>    I experienced the same problem with netperf and with hping also. In
>>    general every tool that sends packet at fixed rates seems to suffer this
>>    problem. If you use the flood option ( sends packets as fast as possible -
>>    for example 'ping -f' or 'hping3 -f') instead of a fixed rate the rate
>>    become much higher. What I discovered is this issue should be related to
>>    Xen's signal handling/delivery.
>>    Netperf use setitimer() to send packets at fixed rate. In my experience on
>>    dom0/U, with no load, Xen cannot go over 250 signals per second. Because
>>    netperf use signal wakeup the function that send packets I think your
>>    problem can be related to this issue. If you want to be sure, count the
>>    average number of packet per seconds. It should be about 250 packets/s
>>    (not more).
>
>
> 250 per second.. sounds like kernel HZ value?

Yes, infact I'm going to test it with 1000Khz. Stay tuned.

:)

m-

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> 250 per second.. sounds like kernel HZ value?
>
> Yes, infact I'm going to test it with 1000Khz. Stay tuned.

Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
Why this behaviour occurs only on xenolinux?

m-

Re: [Xen-devel] Re: About profiling xen

[Marco Tizzoni]

Enabling debug on kernel I realized the problem is related to timer accuracy.
------------------------------------------------------------------------
hal9k-dom0 ~ # cat /proc/timer_list
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 784689453083 nsecs

cpu: 0
 clock 0:
 .index:      0
 .resolution: 999848 nsecs
[........]
hal9k-dom0 ~ # cat
/sys/devices/system/clocksource/clocksource0/available_clocksource
xen jiffies
hal9k-dom0 ~ # cat
/sys/devices/system/clocksource/clocksource0/current_clocksource
xen
------------------------------------------------------------------------

Is there a way to make available a better timer (hpet,acpi....)?

Also, on another xen box (rhel 5.2), I've a 'pit' clocksource in
available_clocksource.
After:
# sysctl -w xen.independent_wallclock=1

and
------------------------------------------------------------------------------------------------------------------------------------
# echo pit > /sys/devices/system/clocksource/clocksource0/current_clocksource
[root@station1 init.d]# cat
/sys/devices/system/clocksource/clocksource0/current_clocksource
jiffies
------------------------------------------------------------------------------------------------------------------------------------

the clocksource is unmodified.

m-

Re: Re: [Xen-users] About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 1366 bytes --]

Thanks alot for the useful information. I will give it a try what is suggested.
Kindly tell me one more thing. I dont found "xenbr0" interface while running "ifconfig -a" command in "dom0". It doesn't come with xen3.4.2 and FC11 pv os kernel 2.6.30-rc6-tip or there's some problem here on my side?

Regards,
Fasiha Ashraf

--- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:

From: Marco Tizzoni <marco.tizzoni@gmail.com>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Pasi Kärkkäinen" <pasik@iki.fi>
Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Wednesday, 30 September, 2009, 12:45 PM

On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>> 250 per second.. sounds like kernel HZ value?
>
> Yes, infact I'm going to test it with 1000Khz. Stay tuned.

Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
Why this behaviour occurs only on xenolinux?

m-

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      Try the new Yahoo! India Homepage. Click here. http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 2228 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-devel] Re: About profiling xen

[Pasi Kärkkäinen]

On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf wrote:
>    Thanks alot for the useful information. I will give it a try what is
>    suggested.
>    Kindly tell me one more thing. I dont found "xenbr0" interface while
>    running "ifconfig -a" command in "dom0". It doesn't come with xen3.4.2 and
>    FC11 pv os kernel 2.6.30-rc6-tip or there's some problem here on my side?
> 

Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch,
since that's the only "supported" and maintained branch now.

-- Pasi

>    Regards,
>    Fasiha Ashraf
> 
>    --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
> 
>      From: Marco Tizzoni <marco.tizzoni@gmail.com>
>      Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
>      To: "Pasi Kärkkäinen" <pasik@iki.fi>
>      Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>,
>      "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
>      Date: Wednesday, 30 September, 2009, 12:45 PM
> 
>      On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni
>      <[1]marco.tizzoni@gmail.com> wrote:
>      > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <[2]pasik@iki.fi>
>      wrote:
>      >> 250 per second.. sounds like kernel HZ value?
>      >
>      > Yes, infact I'm going to test it with 1000Khz. Stay tuned.
> 
>      Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
>      Why this behaviour occurs only on xenolinux?
> 
>      m-
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [3]Xen-devel@lists.xensource.com
>      [4]http://lists.xensource.com/xen-devel
> 
>    --------------------------------------------------------------------------
> 
>    Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how.
> 
> References
> 
>    Visible links
>    1. file:///mc/compose?to=marco.tizzoni@gmail.com
>    2. file:///mc/compose?to=pasik@iki.fi
>    3. file:///mc/compose?to=Xen-devel@lists.xensource.com
>    4. http://lists.xensource.com/xen-devel
>    5. http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photos

RE: [Xen-devel] Re: About profiling xen

[Dan Magenheimer]

> Is there a way to make available a better timer (hpet,acpi....)?
> 
> Also, on another xen box (rhel 5.2), I've a 'pit' clocksource in
> available_clocksource.
> After:
> # sysctl -w xen.independent_wallclock=1
> 
> and
> --------------------------------------------------------------
> ----------------------------------------------------------------------
> # echo pit > 
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> [root@station1 init.d]# cat
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> jiffies
> --------------------------------------------------------------
> ----------------------------------------------------------------------
> 
> the clocksource is unmodified.
> 

Because of save/restore and migrate, no hardware timer is
suitable for a pv domain.  The code that is handling the
clock is the paravirtualized clock code (aka pvclock).
Nobody ever changed the sysfs reporting mechanism in
pv versions of released Red Hat kernels, so clocksource
gets reported as "jiffies", and even though "pit"
appears to be available, it is not (since it wouldn't
work).  I suspect this is also the case for other
distros' pv kernels.

Dan

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Wed, Sep 30, 2009 at 4:21 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:

> Because of save/restore and migrate, no hardware timer is
> suitable for a pv domain.  The code that is handling the
> clock is the paravirtualized clock code (aka pvclock).
> Nobody ever changed the sysfs reporting mechanism in
> pv versions of released Red Hat kernels, so clocksource
> gets reported as "jiffies", and even though "pit"
> appears to be available, it is not (since it wouldn't
> work).  I suspect this is also the case for other
> distros' pv kernels.

How can I get a different, and I hope better, clocksource? In time.c I
found acpi and hpet are available.

m-

RE: Re: [Xen-users] About profiling xen

[Dan Magenheimer]

> On Wed, Sep 30, 2009 at 4:21 PM, Dan Magenheimer
> <dan.magenheimer@oracle.com> wrote:
> 
> > Because of save/restore and migrate, no hardware timer is
> > suitable for a pv domain.  The code that is handling the
> > clock is the paravirtualized clock code (aka pvclock).
> > Nobody ever changed the sysfs reporting mechanism in
> > pv versions of released Red Hat kernels, so clocksource
> > gets reported as "jiffies", and even though "pit"
> > appears to be available, it is not (since it wouldn't
> > work).  I suspect this is also the case for other
> > distros' pv kernels.
> 
> How can I get a different, and I hope better, clocksource? In time.c I
> found acpi and hpet are available.

You can't.  Paravirtualization disables all other
clocksources.  You can modify the source to change
HZ and rebuild (which you apparently already did),
or you can use an HVM instead (but that will change
many things, so you will be comparing apples and
oranges).

I'm no networking expert, but if your packets are
only being processed once per HZ, that seems like
a completely different problem.  Even if you are
able to get a hardware high resolution timer, the
resolution has nothing to do with the clock
interrupt frequency (HZ).

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Wed, Sep 30, 2009 at 6:04 PM, Dan Magenheimer
<dan.magenheimer@oracle.com> wrote:
>> How can I get a different, and I hope better, clocksource? In time.c I
>> found acpi and hpet are available.
>
>
> I'm no networking expert, but if your packets are
> only being processed once per HZ, that seems like
> a completely different problem.  Even if you are
> able to get a hardware high resolution timer, the
> resolution has nothing to do with the clock
> interrupt frequency (HZ).

Netperf use setitimer() to send packets at regular intervals to test
network speed.
If the timer is not finer enough the signal is not delivered on time
and becomes impossibile to reach the requested rate, i.e. if timer is
250Hz you can get at most 250 alarm/s.

>From my point of view this issue should be documented and as well as
the clocksource option. I cactched some information googling but it's
not clearly explained.

m-

Re: [Xen-devel] Re: About profiling xen

[Jeremy Fitzhardinge]

On 09/30/09 00:45, Marco Tizzoni wrote:
> On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
>   
>> On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>>     
>>> 250 per second.. sounds like kernel HZ value?
>>>       
>> Yes, infact I'm going to test it with 1000Khz. Stay tuned.
>>     
> Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
> Why this behaviour occurs only on xenolinux?
>   

Which kernel are you using?  Do you have:

CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

configured?

    J

Re: [Xen-devel] Re: About profiling xen

[Jeremy Fitzhardinge]

On 09/30/09 02:06, Marco Tizzoni wrote:
> Enabling debug on kernel I realized the problem is related to timer accuracy.
> ------------------------------------------------------------------------
> hal9k-dom0 ~ # cat /proc/timer_list
> Timer List Version: v0.3
> HRTIMER_MAX_CLOCK_BASES: 2
> now at 784689453083 nsecs
>
> cpu: 0
>  clock 0:
>  .index:      0
>  .resolution: 999848 nsecs
> [........]
> hal9k-dom0 ~ # cat
> /sys/devices/system/clocksource/clocksource0/available_clocksource
> xen jiffies
> hal9k-dom0 ~ # cat
> /sys/devices/system/clocksource/clocksource0/current_clocksource
> xen
> ------------------------------------------------------------------------
>
> Is there a way to make available a better timer (hpet,acpi....)?
>   

Your clocksource is "xen", which is the best possible for a PV xen guest.

However, a clocksource is for measuring elapsed time, not triggering
timers.  Unfortunately there doesn't seem to be a /sys file to show the
current clockevent source in use, but if you have "xen" clocksource it's
almost certainly the xen clockevent.

However, this is only relevent if you have CONFIG_NO_HZ and
CONFIG_HIGHRES_TIMERS enabled.  If you don't then all timers will be
quantized to HZ (and your guests will burn lots CPU being needlessly
interrupted at HZ).

    J

Re: Re: [Xen-users] About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 2902 bytes --]

Thanks, Ok! I'll give it a try. But let me tell you that the compile n build was quite ok. Just the problem occurs when it comes to Guest communication on single physical host.

Regards,
Fasiha Ashraf

--- On Wed, 30/9/09, Pasi Kärkkäinen <pasik@iki.fi> wrote:

From: Pasi Kärkkäinen <pasik@iki.fi>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Fasiha Ashraf" <feehapk@yahoo.co.in>
Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Wednesday, 30 September, 2009, 2:23 PM

On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf wrote:
>    Thanks alot for the useful information. I will give it a try what is
>    suggested.
>    Kindly tell me one more thing. I dont found "xenbr0" interface while
>    running "ifconfig -a" command in "dom0". It doesn't come with xen3.4.2 and
>    FC11 pv os kernel 2.6.30-rc6-tip or there's some problem here on my side?
> 

Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch,
since that's the only "supported" and maintained branch now.

-- Pasi

>    Regards,
>    Fasiha Ashraf
> 
>    --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
> 
>      From: Marco Tizzoni <marco.tizzoni@gmail.com>
>      Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
>      To: "Pasi Kärkkäinen" <pasik@iki.fi>
>      Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>,
>      "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
>      Date: Wednesday, 30 September, 2009, 12:45 PM
> 
>      On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni
>      <[1]marco.tizzoni@gmail.com> wrote:
>      > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <[2]pasik@iki.fi>
>      wrote:
>      >> 250 per second.. sounds like kernel HZ value?
>      >
>      > Yes, infact I'm going to test it with 1000Khz. Stay tuned.
> 
>      Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
>      Why this behaviour occurs only on xenolinux?
> 
>      m-
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [3]Xen-devel@lists.xensource.com
>      [4]http://lists.xensource.com/xen-devel
> 
>    --------------------------------------------------------------------------
> 
>    Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how.
> 
> References
> 
>    Visible links
>    1. file:///mc/compose?to=marco.tizzoni@gmail.com
>    2. file:///mc/compose?to=pasik@iki.fi
>    3. file:///mc/compose?to=Xen-devel@lists.xensource.com
>    4. http://lists.xensource.com/xen-devel
>    5. http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photos



      Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 5581 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-devel] Re: About profiling xen

[Marco Tizzoni]

On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> Your clocksource is "xen", which is the best possible for a PV xen guest.
>
> However, a clocksource is for measuring elapsed time, not triggering
> timers.

Why? How would you solve the problem of raising events at a fixed rate?

> Unfortunately there doesn't seem to be a /sys file to show the
> current clockevent source in use, but if you have "xen" clocksource it's
> almost certainly the xen clockevent.
>
> However, this is only relevent if you have CONFIG_NO_HZ and
> CONFIG_HIGHRES_TIMERS enabled.

I've tried both set/unset but nothing change.

m-

Re: [Xen-devel] Re: About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 4004 bytes --]

While compiling Xen3.4 with linux kernel 2.6.31.1 At the end of build it
gives me a warning "WARNING: Package 'fig2dev' is required=     to
build Xen documentation" What does that mean? Is that of no harm or it
will effect the performance and working of hypervisor in future?
I follow these steps for compilation



	
	
	
	
	
	

hg clone
http://xenbits.xensource.com/xen-unstable.hg

           hg clone
http://xenbits.xensource.com/linux-2.6.18-xen.hg

           cd xen-unstable.hg

           make world

           make install

this is how  my /boot contains




	
	
	
	
	
	

-rw-r--r-- 1 root root   68783 2009-10-01 13:45 config-2.6.31.1

-rw-r--r-- 1 root root 1882027 2009-10-01 13:45
System.map-2.6.31.1

-rw-r--r-- 1 root root 4437728 2009-10-01 13:45 vmlinuz-2.6.31.1

lrwxrwxrwx 1 root root      19 2009-10-01 13:44 xen-3.5.gz ->
xen-3.5-unstable.gz

-rw-r--r-- 1 root root  517162 2009-10-01 13:44
xen-3.5-unstable.gz

lrwxrwxrwx 1 root root      19 2009-10-01 13:44 xen-3.gz ->
xen-3.5-unstable.gz

lrwxrwxrwx 1 root root      19 2009-10-01 13:44 xen.gz ->
xen-3.5-unstable.gz

-rw-r--r-- 1 root root 7370192 2009-10-01 13:44
xen-syms-3.5-unstable

there is no entry in grub.conf should I add it myself?

Regards
Fasiha Ashraf

--- On Wed, 30/9/09, Pasi Kärkkäinen <pasik@iki.fi> wrote:

From: Pasi Kärkkäinen <pasik@iki.fi>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Fasiha Ashraf" <feehapk@yahoo.co.in>
Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Wednesday, 30 September, 2009, 2:23 PM

On Wed, Sep 30, 2009 at 02:46:46PM +0530, Fasiha Ashraf wrote:
>    Thanks alot for the useful information. I will give it a try what is
>    suggested.
>    Kindly tell me one more thing. I dont found "xenbr0" interface while
>    running "ifconfig -a" command in "dom0". It doesn't come with xen3.4.2 and
>    FC11 pv os kernel 2.6.30-rc6-tip or there's some problem here on my side?
> 

Btw you should use 2.6.31 pv_ops dom0 kernel (from xen/master) branch,
since that's the only "supported" and maintained branch now.

-- Pasi

>    Regards,
>    Fasiha Ashraf
> 
>    --- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:
> 
>      From: Marco Tizzoni <marco.tizzoni@gmail.com>
>      Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
>      To: "Pasi Kärkkäinen" <pasik@iki.fi>
>      Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>,
>      "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
>      Date: Wednesday, 30 September, 2009, 12:45 PM
> 
>      On Wed, Sep 30, 2009 at 8:40 AM, Marco Tizzoni
>      <[1]marco.tizzoni@gmail.com> wrote:
>      > On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <[2]pasik@iki.fi>
>      wrote:
>      >> 250 per second.. sounds like kernel HZ value?
>      >
>      > Yes, infact I'm going to test it with 1000Khz. Stay tuned.
> 
>      Building dom0 kernel with 1000hz the rate reach 1k. Ok, one point. Then?
>      Why this behaviour occurs only on xenolinux?
> 
>      m-
> 
>      _______________________________________________
>      Xen-devel mailing list
>      [3]Xen-devel@lists.xensource.com
>      [4]http://lists.xensource.com/xen-devel
> 
>    --------------------------------------------------------------------------
> 
>    Now, send attachments up to 25MB with Yahoo! India Mail. [5]Learn how.
> 
> References
> 
>    Visible links
>    1. file:///mc/compose?to=marco.tizzoni@gmail.com
>    2. file:///mc/compose?to=pasik@iki.fi
>    3. file:///mc/compose?to=Xen-devel@lists.xensource.com
>    4. http://lists.xensource.com/xen-devel
>    5. http://in.rd.yahoo.com/tagline_galaxy_2/*http:/in.overview.mail.yahoo.com/photos



      Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 7633 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

[-- Attachment #1: Type: text/plain, Size: 1671 bytes --]

On 10/01/09 00:35, Marco Tizzoni wrote:
> On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>   
>> Your clocksource is "xen", which is the best possible for a PV xen guest.
>>
>> However, a clocksource is for measuring elapsed time, not triggering
>> timers.
>>     
> Why? How would you solve the problem of raising events at a fixed rate?
>   

I'm not sure I follow you.  In the kernel, the clock*source* subsystem
is for measuring time: its used to implement gettimeofday, as well as
internal time accounting.  It isn't directly related to time events.

The clock*event* mechanism is all about setting up timers to raise events.

When running paravirtualized, we use Xen-specific versions of both.

>> Unfortunately there doesn't seem to be a /sys file to show the
>> current clockevent source in use, but if you have "xen" clocksource it's
>> almost certainly the xen clockevent.
>>
>> However, this is only relevent if you have CONFIG_NO_HZ and
>> CONFIG_HIGHRES_TIMERS enabled.
>>     
> I've tried both set/unset but nothing change.
>   

Hm.  Its best to leave both enabled either way.

Try the attached program to get some info about how well timers are
working.  Compile with:
$ gcc -o testtimer testtimer.c -O -lrt

And test various results:
$ ./testtimer .1 > xen-0.1.out
$ ./testtimer .01 > xen-0.01.out
$ ./testtimer .001 > xen-0.001.out

You can plot the results with:
$ gnuplot
gnuplot> plot "xen-0.001.out" with lines

The plot is delta from the expected time, so the ideal is that they all
be 0.

On my system in dom0 I see about 10% overhead at .001 and .0005 sec, so
the timers aren't being quantized to 100/250Hz.

    J

[-- Attachment #2: testtimer.c --]
[-- Type: text/x-csrc, Size: 1033 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
#include <sys/time.h>

#define SAMPLES	1000

static double samples[SAMPLES];
static const char prog[] = "/-\\|";

#define NS 1000000000

int main(int argc, char **argv)
{
	int i, iter;
	struct timespec start, end;
	struct timespec sleep;
	double base;

	if (argc < 2) 
		base = .1;
	else
		base = strtod(argv[1], NULL);

	iter = 20 / base;
	if (iter > SAMPLES)
		iter = SAMPLES;

	fprintf(stderr, "%d iterations at %f sec\n", iter, base);

	sleep.tv_sec = 0;
	sleep.tv_nsec = base * NS;

	while (sleep.tv_nsec >= NS) {
		sleep.tv_sec++;
		sleep.tv_nsec -= NS;
	}

	for(i = 0; i < iter; i++) {
		double delta;

		fprintf(stderr, "%c\b", prog[i % (sizeof(prog)-1)]);
		clock_gettime(CLOCK_MONOTONIC, &start);
		nanosleep(&sleep, NULL);
		clock_gettime(CLOCK_MONOTONIC, &end);

		delta = (end.tv_sec * NS + end.tv_nsec) -
			(start.tv_sec * NS + start.tv_nsec);

		samples[i] = delta - (base * NS);
	}

	for(i = 0; i < iter; i++)
		printf("%f\n", samples[i] / NS);

	return 0;
}

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [Xen-devel] Re: About profiling xen

[Marco Tizzoni]

On Thu, Oct 1, 2009 at 10:54 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 10/01/09 00:35, Marco Tizzoni wrote:
>> On Thu, Oct 1, 2009 at 1:23 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
>>
>>> Your clocksource is "xen", which is the best possible for a PV xen guest.
>>>
>>> However, a clocksource is for measuring elapsed time, not triggering
>>> timers.
>>>
>> Why? How would you solve the problem of raising events at a fixed rate?
>>
>
> I'm not sure I follow you.  In the kernel, the clock*source* subsystem
> is for measuring time: its used to implement gettimeofday, as well as
> internal time accounting.  It isn't directly related to time events.

Ok.

> The clock*event* mechanism is all about setting up timers to raise events.
>
> When running paravirtualized, we use Xen-specific versions of both.

Ok.  My problem is granularity. In xen I can't raise events more
frequently than 1000/s (with timer set to 1000hz).
Net test softwares such as netperf or hping use setitimer to send
packet at a fixed rate. On a non-xen box they works fine even  with
100k packets/sec (i.e. 100k alarms/sec). This can be problematic for
streaming software and more in general for soft real-time
applications.

>>> Unfortunately there doesn't seem to be a /sys file to show the
>>> current clockevent source in use, but if you have "xen" clocksource it's
>>> almost certainly the xen clockevent.
>>>
>>> However, this is only relevent if you have CONFIG_NO_HZ and
>>> CONFIG_HIGHRES_TIMERS enabled.
>>>
>> I've tried both set/unset but nothing change.
>>
>
> Hm.  Its best to leave both enabled either way.
>
> Try the attached program to get some info about how well timers are
> working.  Compile with:

Thanks for your test, I tried and timers work as expected.

m-

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

On 10/01/09 14:05, Marco Tizzoni wrote:
>> The clock*event* mechanism is all about setting up timers to raise events.
>>
>> When running paravirtualized, we use Xen-specific versions of both.
>>     
> Ok.  My problem is granularity. In xen I can't raise events more
> frequently than 1000/s (with timer set to 1000hz).
> Net test softwares such as netperf or hping use setitimer to send
> packet at a fixed rate. On a non-xen box they works fine even  with
> 100k packets/sec (i.e. 100k alarms/sec). This can be problematic for
> streaming software and more in general for soft real-time
> applications.
> [...]
> Thanks for your test, I tried and timers work as expected.

OK, good, that takes timers out of the equation.

I guess the problem is the rate at which Xen will context switch between
two domains.  Hm.

Does anything change if you pin your domains to different cpus?

    J

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Thu, Oct 1, 2009 at 11:16 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 10/01/09 14:05, Marco Tizzoni wrote:
>>> The clock*event* mechanism is all about setting up timers to raise events.
>>>
>>> When running paravirtualized, we use Xen-specific versions of both.
>>>
>> Ok.  My problem is granularity. In xen I can't raise events more
>> frequently than 1000/s (with timer set to 1000hz).
>> Net test softwares such as netperf or hping use setitimer to send
>> packet at a fixed rate. On a non-xen box they works fine even  with
>> 100k packets/sec (i.e. 100k alarms/sec). This can be problematic for
>> streaming software and more in general for soft real-time
>> applications.
>> [...]
>> Thanks for your test, I tried and timers work as expected.
>
> OK, good, that takes timers out of the equation.
>
> I guess the problem is the rate at which Xen will context switch between
> two domains.  Hm.
>
> Does anything change if you pin your domains to different cpus?

mmmh, no. I think without a finer timer the problem could not be solved.
For my test application, no problem, I'll use a while() loop with
counters and sched_yeld(),
but, for real application, it could be a serious issue to deal with.

m-

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

On 10/01/09 14:26, Marco Tizzoni wrote:
>> OK, good, that takes timers out of the equation.
>>
>> I guess the problem is the rate at which Xen will context switch between
>> two domains.  Hm.
>>
>> Does anything change if you pin your domains to different cpus?
>>     
> mmmh, no. I think without a finer timer the problem could not be solved.
>   

Why's that?  You said that the timers "work as expected", which I read
to mean that they will generate sub-millisecond events.  Did I
misunderstand you?

> For my test application, no problem, I'll use a while() loop with
> counters and sched_yeld(),
> but, for real application, it could be a serious issue to deal with.
>   

I don't see why that would be necessary if timers are working properly.

    J

Re: [Xen-devel] Re: About profiling xen

[Marco Tizzoni]

On Thu, Oct 1, 2009 at 11:40 PM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 10/01/09 14:26, Marco Tizzoni wrote:
>>> OK, good, that takes timers out of the equation.
>>>
>>> I guess the problem is the rate at which Xen will context switch between
>>> two domains.  Hm.
>>>
>>> Does anything change if you pin your domains to different cpus?
>>>
>> mmmh, no. I think without a finer timer the problem could not be solved.
>>
>
> Why's that?  You said that the timers "work as expected", which I read
> to mean that they will generate sub-millisecond events.  Did I
> misunderstand you?

Sorry, I was not clear. It works as I expect, i.e going not over 1000
hz (my dom0 kernel has been compiled with 1000hz).
Following a couple of run of your testtimer:

hal9k-dom0 ~ # ./testtimer .000001 |head -n10
1000 iterations at  0.000001 sec
0.000662
0.000946
0.000956
0.000960
0.000962
0.000956
0.000964
0.000963
0.000964
0.000964

hal9k-dom0 ~ # ./testtimer .00001 |head -n10
1000 iterations at  0.000010 sec
0.000407
0.000942
0.000941
0.000954
0.000956
0.000954
0.000955
0.000932
0.000955
0.000955

Hoping this clarify.

m-

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

On 10/01/09 14:55, Marco Tizzoni wrote:
> Sorry, I was not clear. It works as I expect, i.e going not over 1000
> hz (my dom0 kernel has been compiled with 1000hz).
>   

OK, that's definitely *not* expected.  If you're running with
CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have
nothing to do with your HZ configuration.

> Following a couple of run of your testtimer:
>
> hal9k-dom0 ~ # ./testtimer .000001 |head -n10
>   

I think expecting a timer to work at 1MHz is unreasonable.

> 1000 iterations at  0.000001 sec
> 0.000662
> 0.000946
> 0.000956
> 0.000960
> 0.000962
> 0.000956
> 0.000964
> 0.000963
> 0.000964
> 0.000964
>
> hal9k-dom0 ~ # ./testtimer .00001 |head -n10
>   
100kHz is also very demanding.

How does it work with 2kHz? 10kHz?

    J

Re: Re: [Xen-users] About profiling xen

[Marco Tizzoni]

On Fri, Oct 2, 2009 at 12:01 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 10/01/09 14:55, Marco Tizzoni wrote:
>> Sorry, I was not clear. It works as I expect, i.e going not over 1000
>> hz (my dom0 kernel has been compiled with 1000hz).
>>
>
> OK, that's definitely *not* expected.  If you're running with
> CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have
> nothing to do with your HZ configuration.

> 100kHz is also very demanding.
> How does it work with 2kHz? 10kHz?

hal9k-dom0 ~ # ./testtimer .0001 |head -n10
1000 iterations at  0.000100 sec
0.000051
0.000885
0.000890
0.000892
0.000891
0.000892
0.000893
0.000893
0.000879
0.000893
hal9k-dom0 ~ # ./testtimer .002 |head -n10
1000 iterations at  0.002000 sec
0.000744
0.000978
0.000990
0.000990
0.000990
0.000991
0.000992
0.000991
0.000992
0.000990


m-

Re: Re: [Xen-users] About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 1743 bytes --]

Is this the issue that is affecting guest to guest throughput? Since I have timer frequency 1000Hz and high resolution timer both enabled in my dom0. Also i am using cup cores pinned for each VM an Dom0. Please guide me some more about getting increased throughput between guests. 
Could it be an ebtable filter issue?
Regards,
Fasiha Ashraf

--- On Fri, 2/10/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:

From: Marco Tizzoni <marco.tizzoni@gmail.com>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Jeremy Fitzhardinge" <jeremy@goop.org>
Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Friday, 2 October, 2009, 3:20 AM

On Fri, Oct 2, 2009 at 12:01 AM, Jeremy Fitzhardinge <jeremy@goop.org> wrote:
> On 10/01/09 14:55, Marco Tizzoni wrote:
>> Sorry, I was not clear. It works as I expect, i.e going not over 1000
>> hz (my dom0 kernel has been compiled with 1000hz).
>>
>
> OK, that's definitely *not* expected.  If you're running with
> CONFIG_NO_HZ and HIGH_RES_TIMERS then timer resolution should have
> nothing to do with your HZ configuration.

> 100kHz is also very demanding.
> How does it work with 2kHz? 10kHz?

hal9k-dom0 ~ # ./testtimer .0001 |head -n10
1000 iterations at  0.000100 sec
0.000051
0.000885
0.000890
0.000892
0.000891
0.000892
0.000893
0.000893
0.000879
0.000893
hal9k-dom0 ~ # ./testtimer .002 |head -n10
1000 iterations at  0.002000 sec
0.000744
0.000978
0.000990
0.000990
0.000990
0.000991
0.000992
0.000991
0.000992
0.000990


m-



      From cricket scores to your friends. Try the Yahoo! India Homepage! http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 2370 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

On 10/05/09 04:38, Fasiha Ashraf wrote:
> Is this the issue that is affecting guest to guest throughput? Since I
> have timer frequency 1000Hz and high resolution timer both enabled in
> my dom0. Also i am using cup cores pinned for each VM an Dom0. Please
> guide me some more about getting increased throughput between guests.
>

I haven't specifically looked at the problem of domain<->domain
performance.  The quantization of time at 1kHz is a concern, but its not
clear to me whether it comes from Linux, Xen or some interaction between
them.

    J

Re: [Xen-devel] Re: About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 1211 bytes --]

Which IDE is best suitable for inspecting xen code? Netbeans or eclipse can be used?

Regards,
Fasiha Ashraf

--- On Tue, 6/10/09, Jeremy Fitzhardinge <jeremy@goop.org> wrote:

From: Jeremy Fitzhardinge <jeremy@goop.org>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Fasiha Ashraf" <feehapk@yahoo.co.in>
Cc: "Marco Tizzoni" <marco.tizzoni@gmail.com>, "Fajar A." <fajar@fajar.net>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Tuesday, 6 October, 2009, 1:53 AM

On 10/05/09 04:38, Fasiha Ashraf wrote:
> Is this the issue that is affecting guest to guest throughput? Since I
> have timer frequency 1000Hz and high resolution timer both enabled in
> my dom0. Also i am using cup cores pinned for each VM an Dom0. Please
> guide me some more about getting increased throughput between guests.
>

I haven't specifically looked at the problem of domain<->domain
performance.  The quantization of time at 1kHz is a concern, but its not
clear to me whether it comes from Linux, Xen or some interaction between
them.

    J



      Connect more, do more and share more with Yahoo! India Mail. Learn more. http://in.overview.mail.yahoo.com/

[-- Attachment #1.2: Type: text/html, Size: 1686 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Re: Re: [Xen-users] About profiling xen

[Jeremy Fitzhardinge]

On 10/05/09 22:01, Fasiha Ashraf wrote:
> Which IDE is best suitable for inspecting xen code? Netbeans or
> eclipse can be used?
>

I use emacs with cscope for navigating around.  I don't use IDEs much.

    J

Re: [Xen-devel] Re: About profiling xen

[Fasiha Ashraf]

[-- Attachment #1.1: Type: text/plain, Size: 2104 bytes --]

Is this guest to guest throuput issue resolved? 
What  is affecting guest to guest throughput? Since I have timer frequency 1000Hz and high resolution timer both enabled in my dom0. Also i am using cpu cores pinned for each VM an Dom0. Please guide me some more about getting increased throughput between guests. 

Regards,
Fasiha Ashraf

--- On Wed, 30/9/09, Marco Tizzoni <marco.tizzoni@gmail.com> wrote:


From: Marco Tizzoni <marco.tizzoni@gmail.com>
Subject: Re: [Xen-devel] Re: [Xen-users] About profiling xen
To: "Pasi Kärkkäinen" <pasik@iki.fi>
Cc: "Fajar A." <fajar@fajar.net>, "Fasiha Ashraf" <feehapk@yahoo.co.in>, "xen" <xen-devel@lists.xensource.com>, xen-users@lists.xensource.com
Date: Wednesday, 30 September, 2009, 11:40 AM


On Wed, Sep 30, 2009 at 8:37 AM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Wed, Sep 30, 2009 at 08:33:19AM +0200, Marco Tizzoni wrote:
>>    I experienced the same problem with netperf and with hping also. In
>>    general every tool that sends packet at fixed rates seems to suffer this
>>    problem. If you use the flood option ( sends packets as fast as possible -
>>    for example 'ping -f' or 'hping3 -f') instead of a fixed rate the rate
>>    become much higher. What I discovered is this issue should be related to
>>    Xen's signal handling/delivery.
>>    Netperf use setitimer() to send packets at fixed rate. In my experience on
>>    dom0/U, with no load, Xen cannot go over 250 signals per second. Because
>>    netperf use signal wakeup the function that send packets I think your
>>    problem can be related to this issue. If you want to be sure, count the
>>    average number of packet per seconds. It should be about 250 packets/s
>>    (not more).
>
>
> 250 per second.. sounds like kernel HZ value?

Yes, infact I'm going to test it with 1000Khz. Stay tuned.

:)

m-

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      Yahoo! India has a new look. Take a sneak peek http://in.yahoo.com/trynew

[-- Attachment #1.2: Type: text/html, Size: 3184 bytes --]

[-- Attachment #2: Type: text/plain, Size: 137 bytes --]

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users