linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
@ 2003-05-08 22:12 Chris Friesen
  2003-05-09  0:13 ` William Lee Irwin III
  2003-05-09  8:17 ` how to measure scheduler latency on powerpc? realfeel doesn Giuliano Pochini
  0 siblings, 2 replies; 26+ messages in thread
From: Chris Friesen @ 2003-05-08 22:12 UTC (permalink / raw)
  To: linux-kernel


I'm trying to test the scheduler latency on a powerpc platform.  It appears that 
a realfeel type of program won't work since you can't program /dev/rtc to 
generated interrupts on powerpc.  Is there anything similar which could be done?

Thanks,

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com



^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-08 22:12 how to measure scheduler latency on powerpc? realfeel doesn't work due to /dev/rtc issues Chris Friesen
@ 2003-05-09  0:13 ` William Lee Irwin III
  2003-05-09  0:38   ` Davide Libenzi
  2003-05-09  8:17 ` how to measure scheduler latency on powerpc? realfeel doesn Giuliano Pochini
  1 sibling, 1 reply; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09  0:13 UTC (permalink / raw)
  To: Chris Friesen; +Cc: linux-kernel

On Thu, May 08, 2003 at 06:12:12PM -0400, Chris Friesen wrote:
> I'm trying to test the scheduler latency on a powerpc platform.  It appears 
> that a realfeel type of program won't work since you can't program /dev/rtc 
> to generated interrupts on powerpc.  Is there anything similar which could 
> be done?

Why would you want to use an interrupt? Just count jiffies in sched.c


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  0:13 ` William Lee Irwin III
@ 2003-05-09  0:38   ` Davide Libenzi
  2003-05-09  0:38     ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Davide Libenzi @ 2003-05-09  0:38 UTC (permalink / raw)
  To: William Lee Irwin III; +Cc: Chris Friesen, Linux Kernel Mailing List

On Thu, 8 May 2003, William Lee Irwin III wrote:

> On Thu, May 08, 2003 at 06:12:12PM -0400, Chris Friesen wrote:
> > I'm trying to test the scheduler latency on a powerpc platform.  It appears
> > that a realfeel type of program won't work since you can't program /dev/rtc
> > to generated interrupts on powerpc.  Is there anything similar which could
> > be done?
>
> Why would you want to use an interrupt? Just count jiffies in sched.c

I don't know what he does mean for scheduler latency, but if it is the ctx
switch one something like get_cycles() will be better instead of jiffies.



- Davide


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  0:38   ` Davide Libenzi
@ 2003-05-09  0:38     ` William Lee Irwin III
  2003-05-09  0:56       ` Richard B. Johnson
  0 siblings, 1 reply; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09  0:38 UTC (permalink / raw)
  To: Davide Libenzi; +Cc: Chris Friesen, Linux Kernel Mailing List

On Thu, May 08, 2003 at 06:12:12PM -0400, Chris Friesen wrote:
>>> I'm trying to test the scheduler latency on a powerpc platform.  It appears
>>> that a realfeel type of program won't work since you can't program /dev/rtc
>>> to generated interrupts on powerpc.  Is there anything similar which could
>>> be done?

On Thu, 8 May 2003, William Lee Irwin III wrote:
> > Why would you want to use an interrupt? Just count jiffies in sched.c

On Thu, May 08, 2003 at 05:38:23PM -0700, Davide Libenzi wrote:
> I don't know what he does mean for scheduler latency, but if it is the ctx
> switch one something like get_cycles() will be better instead of jiffies.

True, if you're looking for performance tweaks and not pathologies (which
I was) you'll need something that accurate.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  0:38     ` William Lee Irwin III
@ 2003-05-09  0:56       ` Richard B. Johnson
  2003-05-09  3:52         ` Chris Friesen
  0 siblings, 1 reply; 26+ messages in thread
From: Richard B. Johnson @ 2003-05-09  0:56 UTC (permalink / raw)
  To: William Lee Irwin III
  Cc: Davide Libenzi, Chris Friesen, Linux Kernel Mailing List

On Thu, 8 May 2003, William Lee Irwin III wrote:

> On Thu, May 08, 2003 at 06:12:12PM -0400, Chris Friesen wrote:
> >>> I'm trying to test the scheduler latency on a powerpc platform.  It appears
> >>> that a realfeel type of program won't work since you can't program /dev/rtc
> >>> to generated interrupts on powerpc.  Is there anything similar which could
> >>> be done?
>
> On Thu, 8 May 2003, William Lee Irwin III wrote:
> > > Why would you want to use an interrupt? Just count jiffies in sched.c
>
> On Thu, May 08, 2003 at 05:38:23PM -0700, Davide Libenzi wrote:
> > I don't know what he does mean for scheduler latency, but if it is the ctx
> > switch one something like get_cycles() will be better instead of jiffies.
>
> True, if you're looking for performance tweaks and not pathologies (which
> I was) you'll need something that accurate.
>
>
> -- wli
Does it have a printer port like the Intel machines?
If so, set it up to generate interrupts on the 'event' pin
(paper-out, etc.) and have the ISR parrot the status bits
out the printer-port bits.

Start it  up and put function generator on the event bit.
measure the delay beteen that bin and the data data bit(s)
with a 'scope. This tells you the whole story, the total
time necessary for an ISR to actually do something.



Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips).
Why is the government concerned about the lunatic fringe? Think about it.


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  0:56       ` Richard B. Johnson
@ 2003-05-09  3:52         ` Chris Friesen
  2003-05-09  4:13           ` Roland Dreier
  2003-05-09  4:26           ` William Lee Irwin III
  0 siblings, 2 replies; 26+ messages in thread
From: Chris Friesen @ 2003-05-09  3:52 UTC (permalink / raw)
  To: root; +Cc: William Lee Irwin III, Davide Libenzi, Linux Kernel Mailing List

Richard B. Johnson wrote:
> On Thu, 8 May 2003, William Lee Irwin III wrote:
>>>>Why would you want to use an interrupt? Just count jiffies in sched.c

I'm trying to get a feel for the maximum time from an interrupt coming in until 
the userspace handler gets notified.  On intel you can program the hardware to 
generate interupts through /dev/rtc.  The powerpc doesn't seem to support this.

Jiffies are not accurate enough, I am expecting max latencies in the 1-2 ms range.

> Does it have a printer port like the Intel machines?

Unfortunately no.  USB/Firewire/Ethernet on the desktop, ethernet/serial for 
compactPCI.

I want to find an additional programmable interrupt source.  It bites that cheap 
PCs have this, and the powerpc doesn't.

Chris



-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  3:52         ` Chris Friesen
@ 2003-05-09  4:13           ` Roland Dreier
  2003-05-09  6:07             ` Chris Friesen
  2003-05-09  4:26           ` William Lee Irwin III
  1 sibling, 1 reply; 26+ messages in thread
From: Roland Dreier @ 2003-05-09  4:13 UTC (permalink / raw)
  To: Chris Friesen; +Cc: Linux Kernel Mailing List

    Chris> I want to find an additional programmable interrupt source.
    Chris> It bites that cheap PCs have this, and the powerpc doesn't.

I don't know which PowerPC CPU you are using, but for example the IBM
4xx series (and all Book E processors) have a "fixed interval timer"
interrupt that is currently not used at all by Linux.  (The
"programmable interval timer" is used to increment jiffies)

On something like the Motorola 74xx, you might be able to use the
something like the performance monitor to generate an exception.

Best,
  Roland

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  3:52         ` Chris Friesen
  2003-05-09  4:13           ` Roland Dreier
@ 2003-05-09  4:26           ` William Lee Irwin III
  2003-05-09  6:14             ` Chris Friesen
  2003-05-09  8:23             ` mikpe
  1 sibling, 2 replies; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09  4:26 UTC (permalink / raw)
  To: Chris Friesen; +Cc: Davide Libenzi, Linux Kernel Mailing List

On Thu, 8 May 2003, William Lee Irwin III wrote:
>>>>> Why would you want to use an interrupt? Just count jiffies in sched.c

On Thu, May 08, 2003 at 11:52:29PM -0400, Chris Friesen wrote:
> I'm trying to get a feel for the maximum time from an interrupt coming in 
> until the userspace handler gets notified.  On intel you can program the 
> hardware to generate interupts through /dev/rtc.  The powerpc doesn't seem 
> to support this.
> Jiffies are not accurate enough, I am expecting max latencies in the 1-2 ms 
> range.
> Unfortunately no.  USB/Firewire/Ethernet on the desktop, ethernet/serial 
> for compactPCI.
> I want to find an additional programmable interrupt source.  It bites that 
> cheap PCs have this, and the powerpc doesn't.

Try the timebase instead.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  4:13           ` Roland Dreier
@ 2003-05-09  6:07             ` Chris Friesen
  0 siblings, 0 replies; 26+ messages in thread
From: Chris Friesen @ 2003-05-09  6:07 UTC (permalink / raw)
  To: Roland Dreier; +Cc: Linux Kernel Mailing List

Roland Dreier wrote:
>     Chris> I want to find an additional programmable interrupt source.
>     Chris> It bites that cheap PCs have this, and the powerpc doesn't.

> On something like the Motorola 74xx, you might be able to use the
> something like the performance monitor to generate an exception.

That's the chip.  I'll look into that further--thanks for the suggestion.

Chris




-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  4:26           ` William Lee Irwin III
@ 2003-05-09  6:14             ` Chris Friesen
  2003-05-09  6:20               ` William Lee Irwin III
  2003-05-09 11:37               ` paubert
  2003-05-09  8:23             ` mikpe
  1 sibling, 2 replies; 26+ messages in thread
From: Chris Friesen @ 2003-05-09  6:14 UTC (permalink / raw)
  To: William Lee Irwin III; +Cc: Davide Libenzi, Linux Kernel Mailing List

William Lee Irwin III wrote:

> Try the timebase instead.

The timestamp is not hard to get.  The problem is getting a medium-frequency 
(2KHz or so) hardware interrupt to drive the test.

On intel, you can do this by programming /dev/rtc.  This does not work in powerpc.

Chris



-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  6:14             ` Chris Friesen
@ 2003-05-09  6:20               ` William Lee Irwin III
  2003-05-09  6:53                 ` Chris Friesen
  2003-05-09 11:37               ` paubert
  1 sibling, 1 reply; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09  6:20 UTC (permalink / raw)
  To: Chris Friesen; +Cc: Davide Libenzi, Linux Kernel Mailing List

William Lee Irwin III wrote:
>> Try the timebase instead.

On Fri, May 09, 2003 at 02:14:13AM -0400, Chris Friesen wrote:
> The timestamp is not hard to get.  The problem is getting a 
> medium-frequency (2KHz or so) hardware interrupt to drive the test.
> On intel, you can do this by programming /dev/rtc.  This does not work in 
> powerpc.

I don't understand why you're obsessed with interrupts. Just run your
load and spray the scheduler latency stats out /proc/


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  6:20               ` William Lee Irwin III
@ 2003-05-09  6:53                 ` Chris Friesen
  2003-05-09  7:01                   ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Chris Friesen @ 2003-05-09  6:53 UTC (permalink / raw)
  To: William Lee Irwin III; +Cc: Linux Kernel Mailing List

William Lee Irwin III wrote:

> I don't understand why you're obsessed with interrupts. Just run your
> load and spray the scheduler latency stats out /proc/

I'm obsessed with interrupts because it gives me a higher sampling rate.

I could set up and itimer for a recurring 10ms timeout and see how much extra I 
waited, but then I can only get 100 samples/sec.

With /dev/rtc (on intel) you can get 20x more samples in the same amount of time.

Chris



-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  6:53                 ` Chris Friesen
@ 2003-05-09  7:01                   ` William Lee Irwin III
  2003-05-09 16:47                     ` Robert Love
  0 siblings, 1 reply; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09  7:01 UTC (permalink / raw)
  To: Chris Friesen; +Cc: Linux Kernel Mailing List

On Fri, May 09, 2003 at 02:53:00AM -0400, Chris Friesen wrote:
> I'm obsessed with interrupts because it gives me a higher sampling rate.
> I could set up and itimer for a recurring 10ms timeout and see how much 
> extra I waited, but then I can only get 100 samples/sec.
> With /dev/rtc (on intel) you can get 20x more samples in the same amount of 
> time.

Why not just keep track of it in the scheduler? The statistic is well-
defined in terms of things measurable at context switch and wakeup.
Just stamping processes with when they became runnable and when they
were first run with the timebase and/or TSC and/or whatever would seem
to provide the answer you want for T(x). W(x) is slightly more involved
but can be measured properly in the same way.

I think your stats will be more accurate and serve your own (not mine;
I have _zero_ cause to fish this stuff out myself besides curiostiy)
purposes better if measured in the way I'm suggesting. That said, only
your own purposes really matter for this in the end so if I
misunderstand you by all means generate all the interrupts you want.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* RE: how to measure scheduler latency on powerpc?  realfeel doesn
  2003-05-08 22:12 how to measure scheduler latency on powerpc? realfeel doesn't work due to /dev/rtc issues Chris Friesen
  2003-05-09  0:13 ` William Lee Irwin III
@ 2003-05-09  8:17 ` Giuliano Pochini
  1 sibling, 0 replies; 26+ messages in thread
From: Giuliano Pochini @ 2003-05-09  8:17 UTC (permalink / raw)
  To: Chris Friesen; +Cc: linux-kernel


On 08-May-2003 Chris Friesen wrote:
> 
> I'm trying to test the scheduler latency on a powerpc platform.  It appears that 
> a realfeel type of program won't work since you can't program /dev/rtc to 
> generated interrupts on powerpc.  Is there anything similar which could be done?

If you are using a legacy Mac, you can use the serial port. If you have
a sound chip, you can use that one. All archs have a programmable timer, even
if /dev/rtc doesn't suport it. Try to ask in the linuxppc-dev mailing list.


Bye.


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  4:26           ` William Lee Irwin III
  2003-05-09  6:14             ` Chris Friesen
@ 2003-05-09  8:23             ` mikpe
  1 sibling, 0 replies; 26+ messages in thread
From: mikpe @ 2003-05-09  8:23 UTC (permalink / raw)
  To: William Lee Irwin III
  Cc: Chris Friesen, Davide Libenzi, Linux Kernel Mailing List

William Lee Irwin III writes:
 > On Thu, 8 May 2003, William Lee Irwin III wrote:
 > >>>>> Why would you want to use an interrupt? Just count jiffies in sched.c
 > 
 > On Thu, May 08, 2003 at 11:52:29PM -0400, Chris Friesen wrote:
 > > I'm trying to get a feel for the maximum time from an interrupt coming in 
 > > until the userspace handler gets notified.  On intel you can program the 
 > > hardware to generate interupts through /dev/rtc.  The powerpc doesn't seem 
 > > to support this.
 > > Jiffies are not accurate enough, I am expecting max latencies in the 1-2 ms 
 > > range.
 > > Unfortunately no.  USB/Firewire/Ethernet on the desktop, ethernet/serial 
 > > for compactPCI.
 > > I want to find an additional programmable interrupt source.  It bites that 
 > > cheap PCs have this, and the powerpc doesn't.
 > 
 > Try the timebase instead.

On all Moto PPCs I've checked, the time-base runs at 1/4 the bus clock
rather than at the core clock like we're used to on x86.

I believe many PPCs allow you to program one of the performance counters
to count core clocks. Be advised though that many Moto PPCs in the 750-7410
range have an errata that prevents using performance counter interrupts,
so you'd have to run the counter in non-interrupting mode.

This isn't available to user-space (yet), so you'd also have to hack
the kernel to use this facility.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  6:14             ` Chris Friesen
  2003-05-09  6:20               ` William Lee Irwin III
@ 2003-05-09 11:37               ` paubert
  1 sibling, 0 replies; 26+ messages in thread
From: paubert @ 2003-05-09 11:37 UTC (permalink / raw)
  To: Chris Friesen
  Cc: William Lee Irwin III, Davide Libenzi, Linux Kernel Mailing List

On Fri, May 09, 2003 at 02:14:13AM -0400, Chris Friesen wrote:
> William Lee Irwin III wrote:
> 
> >Try the timebase instead.
> 
> The timestamp is not hard to get.  The problem is getting a medium-frequency 
> 
> (2KHz or so) hardware interrupt to drive the test.

Reload the decrementer with a smaller value, and read the decrementer value
at the beginning of every timer_interrupt. Timekeeping only depends on the
timebase on PPC (I wrote most of the code), taking additional decrementer
interrupts does not harm. Actually the simplest and most efficient way
to perform a self IPI on PPC is to write 0 to the decrementer (2 machine
instructions). 

Actually, a way to measure the longest time during which interrupts are masked
would be:

	- when disabling interrupts, set decrementer to zero if they
	were previously enabled.
	- read the decrementer on every timer/external interrupt,
	if the value is negative and too large print the value
	and the point at which the interrupts were reenabled.

That's not perfect (it will generate not too many false positives but
quite some overhead, but it should work).

	Gabriel 

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09  7:01                   ` William Lee Irwin III
@ 2003-05-09 16:47                     ` Robert Love
  2003-05-09 16:53                       ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Robert Love @ 2003-05-09 16:47 UTC (permalink / raw)
  To: William Lee Irwin III; +Cc: Chris Friesen, Linux Kernel Mailing List

On Fri, 2003-05-09 at 03:01, William Lee Irwin III wrote:

> Why not just keep track of it in the scheduler? The statistic is well-
> defined in terms of things measurable at context switch and wakeup.

This would measure context switch latency.  Or something.

By definition, scheduling latency is the time from an interrupt which
wakes the task up until the task is actually running.

Historically, it has been measured by things like realfeel or amlat or
whatever which generate interrupts and wake a waiting task up. You then
measure the latency between the interrupt and when the task actually
runs in user-space.

So Chris can then go run this test under varying loads and see how bad
the latency gets.  I understand his question, but (sorry Chris) I have
no idea of the solution on PPC.

	Robert Love


^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09 16:47                     ` Robert Love
@ 2003-05-09 16:53                       ` William Lee Irwin III
  2003-05-09 17:38                         ` Chris Friesen
  0 siblings, 1 reply; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-09 16:53 UTC (permalink / raw)
  To: Robert Love; +Cc: Chris Friesen, Linux Kernel Mailing List

On Fri, 2003-05-09 at 03:01, William Lee Irwin III wrote:
>> Why not just keep track of it in the scheduler? The statistic is well-
>> defined in terms of things measurable at context switch and wakeup.

On Fri, May 09, 2003 at 12:47:05PM -0400, Robert Love wrote:
> This would measure context switch latency.  Or something.
> By definition, scheduling latency is the time from an interrupt which
> wakes the task up until the task is actually running.
> Historically, it has been measured by things like realfeel or amlat or
> whatever which generate interrupts and wake a waiting task up. You then
> measure the latency between the interrupt and when the task actually
> runs in user-space.
> So Chris can then go run this test under varying loads and see how bad
> the latency gets.  I understand his question, but (sorry Chris) I have
> no idea of the solution on PPC.

Not at all. Just stamp at wakeup and difference when it runs.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-09 16:53                       ` William Lee Irwin III
@ 2003-05-09 17:38                         ` Chris Friesen
  0 siblings, 0 replies; 26+ messages in thread
From: Chris Friesen @ 2003-05-09 17:38 UTC (permalink / raw)
  To: Linux Kernel Mailing List


Well, since time was short I didn't want to hack the kernel. What I ended up
doing was setting up an itimer for 10ms intervals and then measuring how much
longer than 10ms I was asleep for.  While this only collects samples at 100Hz,
it will work with the default kernel.

Thanks for the suggestions,

Chris


-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com


^ permalink raw reply	[flat|nested] 26+ messages in thread

* RE: how to measure scheduler latency on powerpc?  realfeel doesn' t work due to /dev/rtc issues
@ 2003-05-13  2:08 Perez-Gonzalez, Inaky
  0 siblings, 0 replies; 26+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-13  2:08 UTC (permalink / raw)
  To: 'William Lee Irwin III'
  Cc: 'Chris Friesen', 'Linux Kernel Mailing List'

> From: William Lee Irwin III [mailto:wli@holomorphy.com]
> 
> From: William Lee Irwin III [mailto:wli@holomorphy.com]
> >> This is ridiculous. Just make sure you're not sharing interrupts and
> >> count cycles starting at the ISR instead of wakeup and tag events
> >> properly if you truly believe that to be your metric. You, as the
> >> kernel, are notified whenever the interrupts occur and can just look
> >> at the time of day and cycle counts.
> 
> On Mon, May 12, 2003 at 05:20:39PM -0700, Perez-Gonzalez, Inaky wrote:
> > Well, I am only suggesting a way to _FORCE_ interrupts to happen
> > at a certain rate controllable by _SOMEBODY_, not as the system
> > gets them. Chris was concerned about not having a way to
> > _GENERATE_ interrupts at a certain rate.
> > What you are suggesting is the other part of the picture, how to
> > measure the latency and AFAICS, it is not part of the problem of
> > generating the interrupts.
> 
> It also seems somewhat pointless to measure it under artificial
> conditions. Interrupts happen often anyway and you probably want to

Your artificial conditions are your control measurements. Then you
add the loads in the background; by being able to selectively add
and remove loads (the real live loads), then you can more easily
identify who is causing delays and under what conditions. It is
not as thorough as a full code analysis ... but if your coverage
is well done can help a lot.

But I am sure you know all this already.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn' t work due to /dev/rtc issues
  2003-05-13  0:20 how to measure scheduler latency on powerpc? realfeel doesn' t " Perez-Gonzalez, Inaky
@ 2003-05-13  1:02 ` William Lee Irwin III
  0 siblings, 0 replies; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-13  1:02 UTC (permalink / raw)
  To: Perez-Gonzalez, Inaky
  Cc: 'Chris Friesen', 'Linux Kernel Mailing List'

From: William Lee Irwin III [mailto:wli@holomorphy.com]
>> This is ridiculous. Just make sure you're not sharing interrupts and
>> count cycles starting at the ISR instead of wakeup and tag events
>> properly if you truly believe that to be your metric. You, as the
>> kernel, are notified whenever the interrupts occur and can just look
>> at the time of day and cycle counts.

On Mon, May 12, 2003 at 05:20:39PM -0700, Perez-Gonzalez, Inaky wrote:
> Well, I am only suggesting a way to _FORCE_ interrupts to happen
> at a certain rate controllable by _SOMEBODY_, not as the system
> gets them. Chris was concerned about not having a way to 
> _GENERATE_ interrupts at a certain rate.
> What you are suggesting is the other part of the picture, how to
> measure the latency and AFAICS, it is not part of the problem of
> generating the interrupts.

It also seems somewhat pointless to measure it under artificial
conditions. Interrupts happen often anyway and you probably want to
measure the effects of real drivers and kernel subsystems on the time
it takes for the blocked task to resume in userspace for real loads.
By the time you've done up custom drivers and interrupt load generators
you've completely divorced whatever you're measuring from whatever will
affect runtime in the field.

-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* RE: how to measure scheduler latency on powerpc?  realfeel doesn' t work due to /dev/rtc issues
@ 2003-05-13  0:20 Perez-Gonzalez, Inaky
  2003-05-13  1:02 ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-13  0:20 UTC (permalink / raw)
  To: 'William Lee Irwin III'
  Cc: 'Chris Friesen', 'Linux Kernel Mailing List'

> From: William Lee Irwin III [mailto:wli@holomorphy.com]
> 
> William Lee Irwin III wrote:
> >>> I don't understand why you're obsessed with interrupts. Just run your
> >>> load and spray the scheduler latency stats out /proc/
> 
> From: Chris Friesen [mailto:cfriesen@nortelnetworks.com]
> >> I'm obsessed with interrupts because it gives me a higher sampling
rate.
> >> I could set up and itimer for a recurring 10ms timeout and see how much
> >> extra I waited, but then I can only get 100 samples/sec. With
> >> /dev/rtc (on intel) you can get 20x more samples in the same amount
> >> of time.
> 
> On Fri, May 09, 2003 at 05:39:03PM -0700, Perez-Gonzalez, Inaky wrote:
> > Okay, crazy idea here ...
> > You are talking about a bladed system, right? So probably you
> > have two network interfaces in there [it should work only with
> > one too].
> > What if you rip off the driver for the network interface and
> > create a new breed. Set an special link with a null Ethernet
> > cable and have one machine sending really short Ethernet frames
> 
> This is ridiculous. Just make sure you're not sharing interrupts and
> count cycles starting at the ISR instead of wakeup and tag events
> properly if you truly believe that to be your metric. You, as the
> kernel, are notified whenever the interrupts occur and can just look
> at the time of day and cycle counts.

Well, I am only suggesting a way to _FORCE_ interrupts to happen
at a certain rate controllable by _SOMEBODY_, not as the system
gets them. Chris was concerned about not having a way to 
_GENERATE_ interrupts at a certain rate.

What you are suggesting is the other part of the picture, how to
measure the latency and AFAICS, it is not part of the problem of
generating the interrupts.

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn' t work due to /dev/rtc issues
  2003-05-10  0:39 how to measure scheduler latency on powerpc? realfeel doesn' t work due to /dev/rtc issues Perez-Gonzalez, Inaky
@ 2003-05-12 23:55 ` William Lee Irwin III
  0 siblings, 0 replies; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-12 23:55 UTC (permalink / raw)
  To: Perez-Gonzalez, Inaky
  Cc: 'Chris Friesen', 'Linux Kernel Mailing List'

William Lee Irwin III wrote:
>>> I don't understand why you're obsessed with interrupts. Just run your
>>> load and spray the scheduler latency stats out /proc/

From: Chris Friesen [mailto:cfriesen@nortelnetworks.com]
>> I'm obsessed with interrupts because it gives me a higher sampling rate.
>> I could set up and itimer for a recurring 10ms timeout and see how much
>> extra I waited, but then I can only get 100 samples/sec. With
>> /dev/rtc (on intel) you can get 20x more samples in the same amount
>> of time.

On Fri, May 09, 2003 at 05:39:03PM -0700, Perez-Gonzalez, Inaky wrote:
> Okay, crazy idea here ...
> You are talking about a bladed system, right? So probably you
> have two network interfaces in there [it should work only with
> one too].
> What if you rip off the driver for the network interface and 
> create a new breed. Set an special link with a null Ethernet
> cable and have one machine sending really short Ethernet frames

This is ridiculous. Just make sure you're not sharing interrupts and
count cycles starting at the ISR instead of wakeup and tag events
properly if you truly believe that to be your metric. You, as the
kernel, are notified whenever the interrupts occur and can just look
at the time of day and cycle counts.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
  2003-05-12  5:04 ` how to measure scheduler latency on powerpc? realfeel doesn't " Peter Chubb
@ 2003-05-12  5:08   ` William Lee Irwin III
  0 siblings, 0 replies; 26+ messages in thread
From: William Lee Irwin III @ 2003-05-12  5:08 UTC (permalink / raw)
  To: Peter Chubb; +Cc: Robert Love, Chris Friesen, Linux Kernel Mailing List

"William" == William Lee Irwin, <William> writes:
William> Not at all. Just stamp at wakeup and difference when it runs.


On Mon, May 12, 2003 at 03:04:09PM +1000, Peter Chubb wrote:
> That then doesn't include interrupt latency.  The nice thing about the
> amlat tests is that the test predicts when the next interrupt should
> occur, then measures the time between that prediction and the process
> running in userspace.  If you just timestamp at wakeup, you miss all
> the time between interrupt generation and noticing that the process is
> to wake up.

Of course. But that is not the scheduler's problem.


-- wli

^ permalink raw reply	[flat|nested] 26+ messages in thread

* Re: how to measure scheduler latency on powerpc?  realfeel doesn't work due to /dev/rtc issues
       [not found] <493798056@toto.iv>
@ 2003-05-12  5:04 ` Peter Chubb
  2003-05-12  5:08   ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Peter Chubb @ 2003-05-12  5:04 UTC (permalink / raw)
  To: William Lee Irwin III
  Cc: Robert Love, Chris Friesen, Linux Kernel Mailing List

>>>>> "William" == William Lee Irwin, <William> writes:
William> Not at all. Just stamp at wakeup and difference when it runs.


That then doesn't include interrupt latency.  The nice thing about the
amlat tests is that the test predicts when the next interrupt should
occur, then measures the time between that prediction and the process
running in userspace.  If you just timestamp at wakeup, you miss all
the time between interrupt generation and noticing that the process is
to wake up.

--
Dr Peter Chubb  http://www.gelato.unsw.edu.au  peterc AT gelato.unsw.edu.au
You are lost in a maze of BitKeeper repositories,   all slightly different.

^ permalink raw reply	[flat|nested] 26+ messages in thread

* RE: how to measure scheduler latency on powerpc?  realfeel doesn' t work due to /dev/rtc issues
@ 2003-05-10  0:39 Perez-Gonzalez, Inaky
  2003-05-12 23:55 ` William Lee Irwin III
  0 siblings, 1 reply; 26+ messages in thread
From: Perez-Gonzalez, Inaky @ 2003-05-10  0:39 UTC (permalink / raw)
  To: 'Chris Friesen', 'William Lee Irwin III'
  Cc: 'Linux Kernel Mailing List'

[-- Attachment #1: Type: text/plain, Size: 2382 bytes --]


> From: Chris Friesen [mailto:cfriesen@nortelnetworks.com]
> 
> William Lee Irwin III wrote:
> 
> > I don't understand why you're obsessed with interrupts. Just run your
> > load and spray the scheduler latency stats out /proc/
> 
> I'm obsessed with interrupts because it gives me a higher sampling rate.
> 
> I could set up and itimer for a recurring 10ms timeout and see how much
extra I
> waited, but then I can only get 100 samples/sec.
> 
> With /dev/rtc (on intel) you can get 20x more samples in the same amount
of time.

Okay, crazy idea here ...

You are talking about a bladed system, right? So probably you
have two network interfaces in there [it should work only with
one too].

What if you rip off the driver for the network interface and 
create a new breed. Set an special link with a null Ethernet
cable and have one machine sending really short Ethernet frames
to the sampling machine.

Maybe if you can manage to get the Ethernet chip to interrupt
every time a new frame arrives, you can use that as a sampling
measure. I'd say the key would be to have the sending machine
be really precise about the sending ... I guess it can be worked
out.

I don't know how fast an interrupt rate you could get, OTOH 
rough numbers ... let's say 100 MBit/s is 10 MByte/s, use
a really small frame [let's say a few bytes only, 32], add
the MACS {I don't remember the frame format, assuming 12 bytes
for source and destination MACs, plus 8 in overhead [again, I
made it up], 52 bytes ... let's round up to 64 bytes per frame.

So

10 MB/s / 64 B/frame = 163840 frames/s

I don't know how really possible is this or my calculations
are screwed up, but it might be worth a try ...

I did a quick test; from one of my computers, m1, I did:

m1:~ $ while true; do cat BIGFILE; done | ssh m2 cat > /dev/null

while on m2, I did:

m2:~ $ grep eth0 /proc/interrupts; sleep 2m; grep eth0 /proc/interrupts
 18:      77457      68483   IO-APIC-level  eth0
 18:     397390     412559   IO-APIC-level  eth0
m2:~ $ 

total    319933  +  344076   = 664009
in 120 seconds ... 664009 / 120 = 5533 Hz ~ 2500 Hz per CPU.

not bad, wouldn't this work?

[this is with a 1500 MTU through a hub ... or a switch, I
don't really know ...]

Iñaky Pérez-González -- Not speaking for Intel -- all opinions are my own
(and my fault)


[-- Attachment #2: t.txt --]
[-- Type: text/plain, Size: 205 bytes --]

 18:      77457      68483   IO-APIC-level  eth0
 18:     397390     412559   IO-APIC-level  eth0

total    319933  +  344076   = 664009
in 120 seconds ... 664009 / 120 = 5533 Hz ~ 2500 Hz per CPU.


^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, other threads:[~2003-05-13  1:55 UTC | newest]

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-05-08 22:12 how to measure scheduler latency on powerpc? realfeel doesn't work due to /dev/rtc issues Chris Friesen
2003-05-09  0:13 ` William Lee Irwin III
2003-05-09  0:38   ` Davide Libenzi
2003-05-09  0:38     ` William Lee Irwin III
2003-05-09  0:56       ` Richard B. Johnson
2003-05-09  3:52         ` Chris Friesen
2003-05-09  4:13           ` Roland Dreier
2003-05-09  6:07             ` Chris Friesen
2003-05-09  4:26           ` William Lee Irwin III
2003-05-09  6:14             ` Chris Friesen
2003-05-09  6:20               ` William Lee Irwin III
2003-05-09  6:53                 ` Chris Friesen
2003-05-09  7:01                   ` William Lee Irwin III
2003-05-09 16:47                     ` Robert Love
2003-05-09 16:53                       ` William Lee Irwin III
2003-05-09 17:38                         ` Chris Friesen
2003-05-09 11:37               ` paubert
2003-05-09  8:23             ` mikpe
2003-05-09  8:17 ` how to measure scheduler latency on powerpc? realfeel doesn Giuliano Pochini
2003-05-10  0:39 how to measure scheduler latency on powerpc? realfeel doesn' t work due to /dev/rtc issues Perez-Gonzalez, Inaky
2003-05-12 23:55 ` William Lee Irwin III
     [not found] <493798056@toto.iv>
2003-05-12  5:04 ` how to measure scheduler latency on powerpc? realfeel doesn't " Peter Chubb
2003-05-12  5:08   ` William Lee Irwin III
2003-05-13  0:20 how to measure scheduler latency on powerpc? realfeel doesn' t " Perez-Gonzalez, Inaky
2003-05-13  1:02 ` William Lee Irwin III
2003-05-13  2:08 Perez-Gonzalez, Inaky

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).