Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sat, 2004-01-03 at 20:40, Mark Hahn wrote:
> > yeah, I think so... but as generating output in a shell is a very common
> > thing to do there should either be an option to turn that unwanted
> > behaviour off or to fix this issue...
> 
> has anyone said it's desired behavior?  you probably need to describe 
> your setup more.  for instance, is your X niced to negative?  are there
> some background processes which would be consuming cycles?

freshly booted system with X running at niceness 0 no other processes consume cpu cycles.

it is reproducable by creating any kind of output which reads from disk... so
i.e. a find ./ in my home directory takes sometimes like 30 minutes on
2.6 (100%cpu load) and sometimes 5 minutes (on 2.4 always 5 minutes
~40%load).

dmesg is another candidate... just doing cat <file> seems not to trigger
that problem.

As Willy Tarreau also oberves this very same weirdness - I now know the
problem is there and it is not specific to my setup.

Soeren.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 07:19, Soeren Sonnenburg wrote:
> On Sat, 2004-01-03 at 20:40, Mark Hahn wrote:
> > > yeah, I think so... but as generating output in a shell is a very
> > > common thing to do there should either be an option to turn that
> > > unwanted behaviour off or to fix this issue...
> >
> > has anyone said it's desired behavior?  you probably need to describe
> > your setup more.  for instance, is your X niced to negative?  are there
> > some background processes which would be consuming cycles?
>
> freshly booted system with X running at niceness 0 no other processes
> consume cpu cycles.
>
> it is reproducable by creating any kind of output which reads from disk...
> so i.e. a find ./ in my home directory takes sometimes like 30 minutes on
> 2.6 (100%cpu load) and sometimes 5 minutes (on 2.4 always 5 minutes
> ~40%load).
>
> dmesg is another candidate... just doing cat <file> seems not to trigger
> that problem.
>
> As Willy Tarreau also oberves this very same weirdness - I now know the
> problem is there and it is not specific to my setup.

There is a BASH bug that Linus noticed brought out by the more sensitive 
timing in 2.6. The BASH developer has been informed it is there and it is 
fixed in the latest version. Perhaps you're both seeing that. Check the lkml 
archives.

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sat, 2004-01-03 at 22:00, Con Kolivas wrote:
> On Sun, 4 Jan 2004 07:19, Soeren Sonnenburg wrote:
> > On Sat, 2004-01-03 at 20:40, Mark Hahn wrote:

[it is a bash bug]
huuhh ? How can this be a bash bug if the output is done by other
programs like dmesg or find ? however I just tested - this issue happens
with csh/tcsh too...

Soeren.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 08:10, Soeren Sonnenburg wrote:
> On Sat, 2004-01-03 at 22:00, Con Kolivas wrote:
> > On Sun, 4 Jan 2004 07:19, Soeren Sonnenburg wrote:
> > > On Sat, 2004-01-03 at 20:40, Mark Hahn wrote:
>
> [it is a bash bug]
> huuhh ? How can this be a bash bug if the output is done by other
> programs like dmesg or find ? however I just tested - this issue happens
> with csh/tcsh too...

The bash bug was waiting on a pipe problem; not just bash alone. I was just 
offering a suggestion. I have no idea what exactly to blame without evidence 
of what's at fault.

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

On Sun, Jan 04, 2004 at 08:15:54AM +1100, Con Kolivas wrote:
> The bash bug was waiting on a pipe problem; not just bash alone. I was just 
> offering a suggestion. I have no idea what exactly to blame without evidence 
> of what's at fault.

Well, I did a few tests here with both 2.4.23 and 2.6.1rc1. In fact, it is
also partially xterm which is to blame, because it is not really that fast
on 2.4. Look below :

Context
=======

- X not reniced (0)
- cd to a directory with about 2500 files
- ls -l

1) Simple standard 'ls -l' in an xterm
======================================

2.4.23$ time ls -l

real    0m0.444s
user    0m0.090s
sys     0m0.020s

2.6.1rc1$ time ls -l

real    0m3.995s
user    0m0.081s
sys     0m0.035s

2) introduction of a pipe : ls -l | cat
=======================================

2.4.23$ time ls -l |cat

real    0m0.444s
user    0m0.090s
sys     0m0.020s

2.6.1rc1$ time ls -l|cat

real    0m0.486s
user    0m0.048s
sys     0m0.029s

3) it is not xterm which gets all the CPU either :
==================================================

2.4.23$ time xterm -e sh -c "ls -l"

real    0m0.462s
user    0m0.140s
sys     0m0.040s

2.6.1rc1$ time xterm -e sh -c "ls -l"

real    0m4.010s
user    0m0.236s
sys     0m0.079s

4) CPU is eaten by the X server itself !
========================================

It appears that on 2.4.23, xterm quickly switches over to jump scrolling
which prevents X from eating all CPU. On 2.6.1rc1, it is not the case.
If I disable jump scrolling and list 2 directories totalizing 4000 files :

2.4.23$ time xterm +j -e sh -c "ls -l incoming tmp"

real    0m7.137s
user    0m0.390s
sys     0m0.080s

2.6.1rc1$ time xterm +j -e sh -c "ls -l incoming tmp"

real    0m7.113s
user    0m0.314s
sys     0m0.121s

And here is what 'top' says during last command :

top - 23:37:54 up 17 min,  5 users,  load average: 0.72, 0.62, 0.50
Tasks:  58 total,   4 running,  54 sleeping,   0 stopped,   0 zombie
Cpu(s):  94.5% user,   5.5% system,   0.0% nice,   0.0% idle,   0.0% IO-wait
Mem:    515292k total,   510688k used,     4604k free,   175784k buffers
Swap:   265064k total,        0k used,   265064k free,    19400k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  234 root      16   0 22400  12m  11m S 90.0  2.4   2:20.01 X                 
  417 root      19   0  4764 2660 3844 R  7.3  0.5   0:00.24 xterm             
  418 willy     20   0  1900 1100 1304 R  2.3  0.2   0:00.10 ls                
  394 willy     16   0  1692  940 1556 R  0.5  0.2   0:00.94 top               
    1 root      16   0   348  192  316 S  0.0  0.0   0:05.05 init              
    2 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0       
    3 root       5 -10     0    0    0 S  0.0  0.0   0:00.23 events/0          
    4 root       5 -10     0    0    0 S  0.0  0.0   0:00.24 kblockd/0         
    5 root      25   0     0    0    0 S  0.0  0.0   0:00.00 pdflush           

5) Tried again with X reniced to +10
====================================

2.6.1rc1$ xterm +j -e sh -c "ls -l incoming tmp"

top - 23:48:33 up 27 min,  6 users,  load average: 0.53, 0.46, 0.44
Tasks:  60 total,   2 running,  58 sleeping,   0 stopped,   0 zombie
Cpu(s):   4.9% user,   4.9% system,  90.2% nice,   0.0% idle,   0.0% IO-wait
Mem:    515292k total,   510652k used,     4640k free,   175712k buffers
Swap:   265064k total,        0k used,   265064k free,    19624k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  234 root      35  10 22400  12m  11m R 92.0  2.4   3:58.50 X                 
  499 root      15   0  4764 2656 3844 S  4.0  0.5   0:00.24 xterm             
  500 willy     15   0  2024 1160 1416 S  3.0  0.2   0:00.10 ls                
    1 root      16   0   348  192  316 S  0.0  0.0   0:05.05 init              
    2 root      34  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0       
    3 root       5 -10     0    0    0 S  0.0  0.0   0:00.28 events/0          
    4 root       5 -10     0    0    0 S  0.0  0.0   0:00.24 kblockd/0         
    5 root      25   0     0    0    0 S  0.0  0.0   0:00.00 pdflush           
    6 root      15   0     0    0    0 S  0.0  0.0   0:00.00 pdflush           

And now with jump scrolling enabled again with X still reniced to +10 :

2.4.23$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.648s
user    0m0.230s
sys     0m0.040s

2.6.1rc1$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.691s
user    0m0.230s
sys     0m0.063s

6) Conclusion
=============

Under 2.4, xterm uses jump scrolling which it does not use by default under
2.6 if X responds fast enough. The first dirty solution which comes to mind
is to renice X to >+10 to slow it a bit so that xterm hits the high water
level and jumps.

But it's not an effect of the scheduler alone, but a side effect of the
scheduler and xterm both trying to automatically adjust their behaviour in
a different manner. If either the scheduler or xterm was a bit smarter or
used different thresholds, the problem would go away. It would also explain
why there are people who cannot reproduce it. Perhaps a somewhat faster or
slower system makes the problem go away. Honnestly, it's the first time that
I notice that my xterms are jump-scrolling, it was so much fluid anyway.

BTW, I found an even simplest way to reproduce the problem while ensuring
that it is not I/O related. For this, I use the 'seq' utility :

2.4.23$ $ time xterm -e seq 1 4000

real    0m0.142s
user    0m0.060s
sys     0m0.030s

2.4.23$ time xterm +j -e seq 1 4000

real    0m4.079s
user    0m0.100s
sys     0m0.080s

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 00:35, Willy Tarreau wrote:
> On Sun, Jan 04, 2004 at 08:15:54AM +1100, Con Kolivas wrote:
[...]
> 6) Conclusion
> =============
> 
> Under 2.4, xterm uses jump scrolling which it does not use by default under
> 2.6 if X responds fast enough. The first dirty solution which comes to mind
> is to renice X to >+10 to slow it a bit so that xterm hits the high water
> level and jumps.
> 
> But it's not an effect of the scheduler alone, but a side effect of the
> scheduler and xterm both trying to automatically adjust their behaviour in
> a different manner. If either the scheduler or xterm was a bit smarter or
> used different thresholds, the problem would go away. It would also explain
> why there are people who cannot reproduce it. Perhaps a somewhat faster or
> slower system makes the problem go away. Honnestly, it's the first time that
> I notice that my xterms are jump-scrolling, it was so much fluid anyway.

Hmmhh, indeed I could reproduce that here.
However I am using multi-gnome-terminal and I've heard of others
observing this issue using gnome-terminal. 
As they are probably all derived from xterm I now do wonder how to fix
them in such a way that it works smoothly with 2.4 and 2.6.

Soeren.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 10:35, Willy Tarreau wrote:
> 6) Conclusion
> =============
>
> Under 2.4, xterm uses jump scrolling which it does not use by default under
> 2.6 if X responds fast enough. The first dirty solution which comes to mind
> is to renice X to >+10 to slow it a bit so that xterm hits the high water
> level and jumps.
>
> But it's not an effect of the scheduler alone, but a side effect of the
> scheduler and xterm both trying to automatically adjust their behaviour in
> a different manner. 

Not quite. The scheduler retains high priority for X for longer so it's no new 
dynamic adjustment of any sort, just better cpu usage by X (which is why it's 
smoother now at nice 0 than previously). 

> If either the scheduler or xterm was a bit smarter or 
> used different thresholds, the problem would go away. It would also explain
> why there are people who cannot reproduce it. Perhaps a somewhat faster or
> slower system makes the problem go away. Honnestly, it's the first time
> that I notice that my xterms are jump-scrolling, it was so much fluid
> anyway.

Very thorough but not a scheduler problem as far as I'm concerned. Can you not 
disable smooth scrolling and force jump scrolling?

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Tim Connors]

Con Kolivas <kernel@kolivas.org> said on Sun, 4 Jan 2004 12:42:47 +1100:
> On Sun, 4 Jan 2004 10:35, Willy Tarreau wrote:
> > 6) Conclusion
> > =============
> >
> > Under 2.4, xterm uses jump scrolling which it does not use by default under
> > 2.6 if X responds fast enough. The first dirty solution which comes to mind
> > is to renice X to >+10 to slow it a bit so that xterm hits the high water
> > level and jumps.
> >
> > But it's not an effect of the scheduler alone, but a side effect of the
> > scheduler and xterm both trying to automatically adjust their behaviour in
> > a different manner. 
> 
> Not quite. The scheduler retains high priority for X for longer so it's no new 
> dynamic adjustment of any sort, just better cpu usage by X (which is why it's 
> smoother now at nice 0 than previously). 
> 
> > If either the scheduler or xterm was a bit smarter or 
> > used different thresholds, the problem would go away. It would also explain
> > why there are people who cannot reproduce it. Perhaps a somewhat faster or
> > slower system makes the problem go away. Honnestly, it's the first time
> > that I notice that my xterms are jump-scrolling, it was so much fluid
> > anyway.
> 
> Very thorough but not a scheduler problem as far as I'm concerned. Can you not 
> disable smooth scrolling and force jump scrolling?

AFAIK the definition of jump scrolling is that if xterm is falling
behind, it jumps. Jump scrolling is enabled by default.

What this slowness means is that xterm is getting CPU at just the
right moments that it isn't falling behind, so it doesn't jump - which
means X gets all the CPU to redraw, which means your ls/dmesg anything
else that reads from disk[1] doesn't get any CPU. 

Xterm is already functioning as designed - you can't force jump
scrolling to jump more - it is at the mercy of how it gets
scheduled. If there is nothing more in the pipe to draw, it has to
draw.

These bloody interactive changes to make X more responsive are at the
expense of anything that does *real* work.


[1] Others say that it only affects them first time through - after
something is cached, it goes back to normal speed. For me - it is slow
*all* the time. If I pipe it to cat or tail or something, it is a
*lot* quicker.



-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Beware of Programmers who carry screwdrivers.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 14:32, Tim Connors wrote:
> > Not quite. The scheduler retains high priority for X for longer so it's
> > no new dynamic adjustment of any sort, just better cpu usage by X (which
> > is why it's smoother now at nice 0 than previously).
> >
> > > If either the scheduler or xterm was a bit smarter or
> > > used different thresholds, the problem would go away. It would also
> > > explain why there are people who cannot reproduce it. Perhaps a
> > > somewhat faster or slower system makes the problem go away. Honnestly,
> > > it's the first time that I notice that my xterms are jump-scrolling, it
> > > was so much fluid anyway.
> >
> > Very thorough but not a scheduler problem as far as I'm concerned. Can
> > you not disable smooth scrolling and force jump scrolling?
>
> AFAIK the definition of jump scrolling is that if xterm is falling
> behind, it jumps. Jump scrolling is enabled by default.
>
> What this slowness means is that xterm is getting CPU at just the
> right moments that it isn't falling behind, so it doesn't jump - which
> means X gets all the CPU to redraw, which means your ls/dmesg anything
> else that reads from disk[1] doesn't get any CPU.
>
> Xterm is already functioning as designed - you can't force jump
> scrolling to jump more - it is at the mercy of how it gets
> scheduled. If there is nothing more in the pipe to draw, it has to
> draw.
>
> These bloody interactive changes to make X more responsive are at the
> expense of anything that does *real* work.

Harsh words considering it is the timing sensitive nature of xterm that relies 
on X running out of steam in the old scheduler design to appear smooth.

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Peter Osterlund]

Con Kolivas <kernel@kolivas.org> writes:

> On Sun, 4 Jan 2004 14:32, Tim Connors wrote:
> > > Not quite. The scheduler retains high priority for X for longer so it's
> > > no new dynamic adjustment of any sort, just better cpu usage by X (which
> > > is why it's smoother now at nice 0 than previously).
> > >
> > > > If either the scheduler or xterm was a bit smarter or
> > > > used different thresholds, the problem would go away. It would also
> > > > explain why there are people who cannot reproduce it. Perhaps a
> > > > somewhat faster or slower system makes the problem go away. Honnestly,
> > > > it's the first time that I notice that my xterms are jump-scrolling, it
> > > > was so much fluid anyway.
> > >
> > > Very thorough but not a scheduler problem as far as I'm concerned. Can
> > > you not disable smooth scrolling and force jump scrolling?
> >
> > AFAIK the definition of jump scrolling is that if xterm is falling
> > behind, it jumps. Jump scrolling is enabled by default.
> >
> > What this slowness means is that xterm is getting CPU at just the
> > right moments that it isn't falling behind, so it doesn't jump - which
> > means X gets all the CPU to redraw, which means your ls/dmesg anything
> > else that reads from disk[1] doesn't get any CPU.
> >
> > Xterm is already functioning as designed - you can't force jump
> > scrolling to jump more - it is at the mercy of how it gets
> > scheduled. If there is nothing more in the pipe to draw, it has to
> > draw.
> >
> > These bloody interactive changes to make X more responsive are at the
> > expense of anything that does *real* work.
> 
> Harsh words considering it is the timing sensitive nature of xterm that relies 
> on X running out of steam in the old scheduler design to appear smooth.

But the scheduler is also far from fair in this situation. If I run

        perl -e 'for(;;){printf("ddddddddddddddddddddddddddd\n");}'

in an xterm, the system enters a steady state where top shows:

Cpu(s): 78.3% us, 20.9% sy,  0.0% ni,  0.0% id,  0.0% wa,  0.0% hi,  0.9% si
Mem:     61660k total,    60544k used,     1116k free,     4568k buffers
Swap:   104824k total,    22064k used,    82760k free,    17596k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 2562 root      15   0 17284 7088 7664 S 39.2 11.5   3:21.20 X
 2755 petero    15   0  5652 3260 4528 S 31.7  5.3   1:53.89 xterm
 2808 petero    25   0  3424 1248 2744 R  3.9  2.0   0:01.71 perl
 2807 petero    16   0  1764  940 1636 R  1.6  1.5   0:00.87 top

What happens is that X and xterm start with highest possible
interactivity credit and CURRENT_BONUS, because they were mostly idle
before the test started. The perl program starts at some PR>15 and
slowly climbs to 25. Its interactivity credit remains at 0. As soon as
the perl process delivers one line of output to xterm, xterm and later
X are scheduled, because they have a smaller priority value than
perl. When they have finished scrolling one line, perl is scheduled
again and produces another line of output.

However, since X and xterm both have HIGH_CREDIT and CURRENT_BONUS ==
MAX_BONUS, they only get charged 1/10th of their runtime, because of
this code in schedule():

	if (HIGH_CREDIT(prev))
		run_time /= (CURRENT_BONUS(prev) ? : 1);

Since both processes spend approximately 50% of their time sleeping,
the sleep_avg increase in recalc_task_prio() is more than enough to
keep the processes at maximum interactivity level forever.

The situation for the perl process is that it always loses the cpu a
short time before it would voluntarily go to sleep. The perl process
gets approximately 4% cpu, but it wants 4+epsilon percent cpu, so it
is considered a cpu hog, and eventually gets maximum punishment
(PR=25).

The end result is that the perl process, which requires very little
cpu time, is considered a cpu hog, and the two real cpu hogs (X and
xterm) are considered interactive. This is not a transient state. The
situation does not go away until I kill the perl process or start some
other cpu hog to disturb the scheduler.

-- 
Peter Osterlund - petero2@telia.com
http://w1.894.telia.com/~u89404340

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Peter Osterlund wrote:

>Con Kolivas <kernel@kolivas.org> writes:
>
>
>>On Sun, 4 Jan 2004 14:32, Tim Connors wrote:
>>
>>>>Not quite. The scheduler retains high priority for X for longer so it's
>>>>no new dynamic adjustment of any sort, just better cpu usage by X (which
>>>>is why it's smoother now at nice 0 than previously).
>>>>
>>>>
>>>>>If either the scheduler or xterm was a bit smarter or
>>>>>used different thresholds, the problem would go away. It would also
>>>>>explain why there are people who cannot reproduce it. Perhaps a
>>>>>somewhat faster or slower system makes the problem go away. Honnestly,
>>>>>it's the first time that I notice that my xterms are jump-scrolling, it
>>>>>was so much fluid anyway.
>>>>>
>>>>Very thorough but not a scheduler problem as far as I'm concerned. Can
>>>>you not disable smooth scrolling and force jump scrolling?
>>>>
>>>AFAIK the definition of jump scrolling is that if xterm is falling
>>>behind, it jumps. Jump scrolling is enabled by default.
>>>
>>>What this slowness means is that xterm is getting CPU at just the
>>>right moments that it isn't falling behind, so it doesn't jump - which
>>>means X gets all the CPU to redraw, which means your ls/dmesg anything
>>>else that reads from disk[1] doesn't get any CPU.
>>>
>>>Xterm is already functioning as designed - you can't force jump
>>>scrolling to jump more - it is at the mercy of how it gets
>>>scheduled. If there is nothing more in the pipe to draw, it has to
>>>draw.
>>>
>>>These bloody interactive changes to make X more responsive are at the
>>>expense of anything that does *real* work.
>>>
>>Harsh words considering it is the timing sensitive nature of xterm that relies 
>>on X running out of steam in the old scheduler design to appear smooth.
>>
>
>But the scheduler is also far from fair in this situation. If I run
>

snip a good analysis...

... but fairness is not about a set of numbers the scheduler gives to
each process, its about the amount of CPU time processes are given.

In this case I don't know if I find it objectionable that X and xterm
are considered interactive and perl considered a CPU hog. What is the
actual problem?

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Peter Osterlund]

Nick Piggin <piggin@cyberone.com.au> writes:

> Peter Osterlund wrote:
> 
> >But the scheduler is also far from fair in this situation. If I run
> 
> snip a good analysis...
> 
> ... but fairness is not about a set of numbers the scheduler gives to
> each process, its about the amount of CPU time processes are given.
> 
> In this case I don't know if I find it objectionable that X and xterm
> are considered interactive and perl considered a CPU hog. What is the
> actual problem?

The problem is that if perl would get only slightly more cpu time, it
would get ahead of xterm, which would make this test case run
something like 10 times faster than it currently does. (Because xterm
switches to jump scrolling when it can't keep up.)

I guess it would be possible to fix this by introducing a
usleep(10000) at some strategic place in the xterm source code, but I
still find it strange that two tasks eating 40% cpu time each are
considered interactive, while a task eating 4% is considered a cpu
hog, especially since the 4% task never got a chance to prove that it
didn't want to steal all cpu time. All that was proven was that it
wanted more than 4% of the cpu.

Also, while my test case runs, other tasks (such as running "ps" from
a network login) are very slow, at least until the extra load makes
the scheduler realize that the two tasks eating most of the cpu time
should not have maximum priority bonus.

-- 
Peter Osterlund - petero2@telia.com
http://w1.894.telia.com/~u89404340

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Peter Osterlund wrote:

>Nick Piggin <piggin@cyberone.com.au> writes:
>
>
>>Peter Osterlund wrote:
>>
>>
>>>But the scheduler is also far from fair in this situation. If I run
>>>
>>snip a good analysis...
>>
>>... but fairness is not about a set of numbers the scheduler gives to
>>each process, its about the amount of CPU time processes are given.
>>
>>In this case I don't know if I find it objectionable that X and xterm
>>are considered interactive and perl considered a CPU hog. What is the
>>actual problem?
>>
>
>The problem is that if perl would get only slightly more cpu time, it
>would get ahead of xterm, which would make this test case run
>something like 10 times faster than it currently does. (Because xterm
>switches to jump scrolling when it can't keep up.)
>
>I guess it would be possible to fix this by introducing a
>usleep(10000) at some strategic place in the xterm source code, but I
>still find it strange that two tasks eating 40% cpu time each are
>considered interactive, while a task eating 4% is considered a cpu
>hog, especially since the 4% task never got a chance to prove that it
>didn't want to steal all cpu time. All that was proven was that it
>wanted more than 4% of the cpu.
>
>Also, while my test case runs, other tasks (such as running "ps" from
>a network login) are very slow, at least until the extra load makes
>the scheduler realize that the two tasks eating most of the cpu time
>should not have maximum priority bonus.
>

OK yeah you are right. I perl should get more CPU time if it wants it.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Peter Osterlund wrote:

>Nick Piggin <piggin@cyberone.com.au> writes:
>
>
>>Peter Osterlund wrote:
>>
>>
>>>But the scheduler is also far from fair in this situation. If I run
>>>
>>snip a good analysis...
>>
>>... but fairness is not about a set of numbers the scheduler gives to
>>each process, its about the amount of CPU time processes are given.
>>
>>In this case I don't know if I find it objectionable that X and xterm
>>are considered interactive and perl considered a CPU hog. What is the
>>actual problem?
>>
>
>The problem is that if perl would get only slightly more cpu time, it
>would get ahead of xterm, which would make this test case run
>something like 10 times faster than it currently does. (Because xterm
>switches to jump scrolling when it can't keep up.)
>
>I guess it would be possible to fix this by introducing a
>usleep(10000) at some strategic place in the xterm source code, but I
>still find it strange that two tasks eating 40% cpu time each are
>considered interactive, while a task eating 4% is considered a cpu
>hog, especially since the 4% task never got a chance to prove that it
>didn't want to steal all cpu time. All that was proven was that it
>wanted more than 4% of the cpu.
>
>Also, while my test case runs, other tasks (such as running "ps" from
>a network login) are very slow, at least until the extra load makes
>the scheduler realize that the two tasks eating most of the cpu time
>should not have maximum priority bonus.
>

This is what top looks like with my (now fixed) scheduler
(priorities go from 0 to 59). In theory I guess it looks like what
you want.

  920 root      21 -10 88156  17m  79m S 50.8  7.0  11:08.47 XFree86
16762 npiggin   40   0  4804 2396 4356 R 37.9  0.9   0:07.76 xterm
16784 npiggin   23   0  3128 1212 2664 S  9.3  0.5   0:01.93 perl

xterm is a CPU hog, XFree86 is half and half, perl is at close to highest
priority (which is 20 for a nice 0 process). When using xterm without jump
scrolling, it seems to mess up X's scheduler by flooding it with so many
small requests.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Bryan Whitehead]

> [1] Others say that it only affects them first time through - after
> something is cached, it goes back to normal speed. For me - it is slow
> *all* the time. If I pipe it to cat or tail or something, it is a
> *lot* quicker.

I get slow all the time at home with a single CPU system.

At work I have a Dual p4 1.7Ghz XEON... Only when I keep both CPU's busy 
(example: make -j 4) does it happen on the dual machine. So basically if 
my dual machine is busy (one CPU has to be pegged or both are over 50% 
usage) I start getting very crappy xterm/gnome-terminal response.

I've been running each xterm/gnome-terminal with screen and when it 
starts grinding I detach/reattach.

-- 
Bryan Whitehead
SysAdmin - JPL - Interferometry and Large Optical Systems
Phone: 818 354 2903
driver@jpl.nasa.gov

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 02:42, Con Kolivas wrote:
> On Sun, 4 Jan 2004 10:35, Willy Tarreau wrote:
> > 6) Conclusion
> > =============
> >
> > Under 2.4, xterm uses jump scrolling which it does not use by default under
> > 2.6 if X responds fast enough. The first dirty solution which comes to mind
> > is to renice X to >+10 to slow it a bit so that xterm hits the high water
> > level and jumps.
> >
> > But it's not an effect of the scheduler alone, but a side effect of the
> > scheduler and xterm both trying to automatically adjust their behaviour in
> > a different manner. 
> 
> Not quite. The scheduler retains high priority for X for longer so it's no new 
> dynamic adjustment of any sort, just better cpu usage by X (which is why it's 
> smoother now at nice 0 than previously). 

But why is i.e. running the command a second or third time much faster
compared with the first run ? Does X get less priority then ?

> > If either the scheduler or xterm was a bit smarter or 
> > used different thresholds, the problem would go away. It would also explain
> > why there are people who cannot reproduce it. Perhaps a somewhat faster or
> > slower system makes the problem go away. Honnestly, it's the first time
> > that I notice that my xterms are jump-scrolling, it was so much fluid
> > anyway.
> 
> Very thorough but not a scheduler problem as far as I'm concerned. Can you not 
> disable smooth scrolling and force jump scrolling?

I think it is a schedulers problem as it works if you run the program in
question often enough (dmesg/find/whatever). Suddenly it comes back to
full scrolling speed. 

Judging from the xfree4.3 xterm sources this is the function that gets
called when there is something to scroll. And it looks perfectly ok to
*me* as it scrolls amount lines at once. So the output of (dmesg/...)
seems to be received slower or xterm updates more often than in 2.4.
leading to fewer lines to scroll (== amount). It cannot be xterm
updating more often as it would a) be faster than and b) amount would be
>>> 1 on later on which would lead to high scrolling speed again.

So IMHO it must be the output of the program that is coming in slowly.

I added a fprintf(stderr, "%d\n", amount); to that code and indeed
amount was *always* 1 no matter what I did (it even was 1 when the
(dmesg/...) output came in fast). And jump scrolling would take place if
amount > 59 in my case... can this still be not a schedulers issue ?

from the xfree86-4.3.0 build-tree ./xc/programs/xterm/util.c

/*
 * scrolls the screen by amount lines, erases bottom, doesn't alter
 * cursor position (i.e. cursor moves down amount relative to text).
 * All done within the scrolling region, of course.
 * requires: amount > 0
 */
void
xtermScroll(TScreen * screen, int amount)
{
    int i = screen->bot_marg - screen->top_marg + 1;
    int shift;
    int bot;
    int refreshtop = 0;
    int refreshheight;
    int scrolltop;
    int scrollheight;

    if (screen->cursor_state)
    HideCursor();
    if (amount > i)
    amount = i;
    if (screen->jumpscroll) {
    if (screen->scroll_amt > 0) {
        if (screen->refresh_amt + amount > i)
        FlushScroll(screen);
        screen->scroll_amt += amount;
        screen->refresh_amt += amount;
    } else {
        if (screen->scroll_amt < 0)
        FlushScroll(screen);
        screen->scroll_amt = amount;
        screen->refresh_amt = amount;
    }
    refreshheight = 0;
    } else {
    ScrollSelection(screen, -(amount));
    if (amount == i) {
        ClearScreen(screen);
        return;
    }
    shift = -screen->topline;
    bot = screen->max_row - shift;
    scrollheight = i - amount;
    refreshheight = amount;
[...]

Looking at that how can it not be a scheduling problem ....

Soeren

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 19:09, Soeren Sonnenburg wrote:
> On Sun, 2004-01-04 at 02:42, Con Kolivas wrote:
> > Not quite. The scheduler retains high priority for X for longer so it's
> > no new dynamic adjustment of any sort, just better cpu usage by X (which
> > is why it's smoother now at nice 0 than previously).
>
> But why is i.e. running the command a second or third time much faster
> compared with the first run ? Does X get less priority then ?

Yes. X retains high priority for longer than previous scheduler designs.

> > > If either the scheduler or xterm was a bit smarter or
> > > used different thresholds, the problem would go away. It would also
> > > explain why there are people who cannot reproduce it. Perhaps a
> > > somewhat faster or slower system makes the problem go away. Honnestly,
> > > it's the first time that I notice that my xterms are jump-scrolling, it
> > > was so much fluid anyway.
> >
> > Very thorough but not a scheduler problem as far as I'm concerned. Can
> > you not disable smooth scrolling and force jump scrolling?
>
> I think it is a schedulers problem as it works if you run the program in
> question often enough (dmesg/find/whatever). Suddenly it comes back to
> full scrolling speed.
>
> Judging from the xfree4.3 xterm sources this is the function that gets
> called when there is something to scroll. And it looks perfectly ok to
> *me* as it scrolls amount lines at once. So the output of (dmesg/...)
> seems to be received slower or xterm updates more often than in 2.4.
> leading to fewer lines to scroll (== amount). It cannot be xterm
> updating more often as it would a) be faster than and b) amount would be

It's a timing issue. X gets more priority for longer than previously so X gets 
to do more work.

> >>> 1 on later on which would lead to high scrolling speed again.
>
> So IMHO it must be the output of the program that is coming in slowly.
>
> I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> amount was *always* 1 no matter what I did (it even was 1 when the
> (dmesg/...) output came in fast). And jump scrolling would take place if
> amount > 59 in my case... can this still be not a schedulers issue ?
>

> Looking at that how can it not be a scheduling problem ....

Scheduling problem, yes; of a sort.

Solution by altering the scheduler, no. 

My guess is that turning the xterm graphic candy up or down will change the 
balance. Trying to be both gui intensive and a console is where it's 
happening. On some hardware you are falling on both sides of the fence with 
2.6 where previously you would be on one side.

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:

> > I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> > amount was *always* 1 no matter what I did (it even was 1 when the
> > (dmesg/...) output came in fast). And jump scrolling would take place if
> > amount > 59 in my case... can this still be not a schedulers issue ?
> >
> 
> > Looking at that how can it not be a scheduling problem ....
> 
> Scheduling problem, yes; of a sort.
> 
> Solution by altering the scheduler, no. 
> 
> My guess is that turning the xterm graphic candy up or down will change the 
> balance. Trying to be both gui intensive and a console is where it's 
> happening. On some hardware you are falling on both sides of the fence with 
> 2.6 where previously you would be on one side.
> 

So its Ok for 'eye candy' to 'lag', but xmms should not skip?  Anyhow,
its xterm that he have issues with, not gnome-terminal or such with
transparency.  I smell something ...


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 12:13, Martin Schlemmer wrote:
> On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> 
> > > I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> > > amount was *always* 1 no matter what I did (it even was 1 when the
> > > (dmesg/...) output came in fast). And jump scrolling would take place if
> > > amount > 59 in my case... can this still be not a schedulers issue ?
> > >
> > 
> > > Looking at that how can it not be a scheduling problem ....
> > 
> > Scheduling problem, yes; of a sort.
> > 
> > Solution by altering the scheduler, no. 
> > 
> > My guess is that turning the xterm graphic candy up or down will change the 
> > balance. Trying to be both gui intensive and a console is where it's 
> > happening. On some hardware you are falling on both sides of the fence with 
> > 2.6 where previously you would be on one side.
> > 
> 
> So its Ok for 'eye candy' to 'lag', but xmms should not skip?  Anyhow,
> its xterm that he have issues with, not gnome-terminal or such with
> transparency.  I smell something ...

I would not call it 'lag' if an ls of /usr/bin takes 20 secs vs 1 sec
before... I mean the scroll speed limits e.g. the compile speed...

Well I now tried xterm, gnome-terminal (gtk2), multi-gnome-terminal
(gtk1), xvt, aterm, wterm, Eterm, and yes they are all (except for
Eterm) plain text on solid background - so no eye candy. Interestingly
Eterm, wterm and aterm seem to not be affected by that issue.

Soeren

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Con Kolivas]

On Sun, 4 Jan 2004 22:13, Martin Schlemmer wrote:
> On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> > > I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> > > amount was *always* 1 no matter what I did (it even was 1 when the
> > > (dmesg/...) output came in fast). And jump scrolling would take place
> > > if amount > 59 in my case... can this still be not a schedulers issue ?
> > >
> > >
> > > Looking at that how can it not be a scheduling problem ....
> >
> > Scheduling problem, yes; of a sort.
> >
> > Solution by altering the scheduler, no.
> >
> > My guess is that turning the xterm graphic candy up or down will change
> > the balance. Trying to be both gui intensive and a console is where it's
> > happening. On some hardware you are falling on both sides of the fence
> > with 2.6 where previously you would be on one side.
>
> So its Ok for 'eye candy' to 'lag', but xmms should not skip?  Anyhow,
> its xterm that he have issues with, not gnome-terminal or such with
> transparency.  I smell something ...

Sigh... 

Xmms was a simple test case long forgotten but most still think all I did was 
make an xmms scheduler. Deleting one character from sched.c before all of my 
patches would make the scheduler ideal for xmms. Any braindead idiot can tune 
a scheduler for just one application. An application that changes it's 
behaviour dynamically well in the setting of a particular scheduler, though? 
Should a scheduler be tuned to suit a coding style or quirk? 

I should go back to lurking before people start calling me names. This thread 
has gone long enough for that. If I hadn't said anything it would have died 
out by now. Instead I'm drawing attention to my fundamentally flawed code.

Con

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 3611 bytes --]

On Sun, 2004-01-04 at 14:45, Con Kolivas wrote:
> On Sun, 4 Jan 2004 22:13, Martin Schlemmer wrote:
> > On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> > > > I added a fprintf(stderr, "%d\n", amount); to that code and indeed
> > > > amount was *always* 1 no matter what I did (it even was 1 when the
> > > > (dmesg/...) output came in fast). And jump scrolling would take place
> > > > if amount > 59 in my case... can this still be not a schedulers issue ?
> > > >
> > > >
> > > > Looking at that how can it not be a scheduling problem ....
> > >
> > > Scheduling problem, yes; of a sort.
> > >
> > > Solution by altering the scheduler, no.
> > >
> > > My guess is that turning the xterm graphic candy up or down will change
> > > the balance. Trying to be both gui intensive and a console is where it's
> > > happening. On some hardware you are falling on both sides of the fence
> > > with 2.6 where previously you would be on one side.
> >
> > So its Ok for 'eye candy' to 'lag', but xmms should not skip?  Anyhow,
> > its xterm that he have issues with, not gnome-terminal or such with
> > transparency.  I smell something ...
> 
> Sigh... 
> 
> Xmms was a simple test case long forgotten but most still think all I did was 
> make an xmms scheduler. Deleting one character from sched.c before all of my 
> patches would make the scheduler ideal for xmms. Any braindead idiot can tune 
> a scheduler for just one application.

Well, its the favorite example 8)

>  An application that changes it's 
> behaviour dynamically well in the setting of a particular scheduler, though? 
> Should a scheduler be tuned to suit a coding style or quirk? 
> 

But the scheduler changes to a particular application?  I still am of
opinion that the current scheduler in mainline 'breaks' priorities ...
call it dynamic tuning or whatever you like.  Now something gets
priority while something else starves.

> I should go back to lurking before people start calling me names. This thread 
> has gone long enough for that. If I hadn't said anything it would have died 
> out by now.

Well, I have stayed out of this for months now, as its always 'they' at
fault - that app, or piece of code.  Sure, I am one of those whining
users, and I have no particular interest in the scheduler code - that
is if it behaves like it should.  But whatever is in now, just do not
behave as expected, and call it a feature or whatever you want, if it
deviates the definition, then what should we call it?  Or if its a
feature, can we have the weirdness in priorities disabled by default
with a sysctl or sysfs switch?

> Instead I'm drawing attention to my fundamentally flawed code.
> 

The scrolling is but one part.  Just starting an app, or running
'vim /etc/fstab' for example takes ages some times, even with
minimal load.  If xterm, gnome-term, aterm, multi-gnome-term,
etc is broken, how do we fix it then?  What about some of the
other issues?  If its a problem with those apps, why is it I still
wonder what they are doing wrong, and it not fixed?

Do not worry, _I_ will go back to lurking about this issue _again_,
but after _once_again_ seeing a issue about this being blown off
as being something wrong with 'it', and some facts (you did see
that the skipping code for the other user _never_ kicked in)
were just ignored, I just could not help myself - sorry.

At least I will not experience those issues of the others, and
hopefully Nick will not stop his work, or things change too much
to adapt his patch.


Thanks,

-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[mikeg]

At 04:42 PM 1/4/2004 +0200, Martin Schlemmer wrote:

>Do not worry, _I_ will go back to lurking about this issue _again_,

Wrong answer.  Jump in with both feet instead (much fun, highly 
educational).  The code's not complicated (the problem space is though).

>but after _once_again_ seeing a issue about this being blown off

By you personally.

>as being something wrong with 'it', and some facts (you did see
>that the skipping code for the other user _never_ kicked in)
>were just ignored, I just could not help myself - sorry.

Don't apologize... scratch that itch :)

         -Mike 

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[szonyi calin]

 --- Martin Schlemmer <azarah@nosferatu.za.org> a écrit : > On
Sun, 2004-01-04 at 14:45, Con Kolivas wrote:
> > On Sun, 4 Jan 2004 22:13, Martin Schlemmer wrote:
> > > On Sun, 2004-01-04 at 10:49, Con Kolivas wrote:
> > > > > I added a fprintf(stderr, "%d\n", amount); to that
> code and indeed
> > > > > amount was *always* 1 no matter what I did (it even
> was 1 when the
> > > > > (dmesg/...) output came in fast). And jump scrolling
> would take place
> > > > > if amount > 59 in my case... can this still be not a
> schedulers issue ?
> > > > >
> > > > >
> > > > > Looking at that how can it not be a scheduling problem
> ....
> > > >
> > > > Scheduling problem, yes; of a sort.
> > > >
> > > > Solution by altering the scheduler, no.
> > > >
> > > > My guess is that turning the xterm graphic candy up or
> down will change
> > > > the balance. Trying to be both gui intensive and a
> console is where it's
> > > > happening. On some hardware you are falling on both
> sides of the fence
> > > > with 2.6 where previously you would be on one side.
> > >
> > > So its Ok for 'eye candy' to 'lag', but xmms should not
> skip?  Anyhow,
> > > its xterm that he have issues with, not gnome-terminal or
> such with
> > > transparency.  I smell something ...
> > 
> > Sigh... 
> > 
> > Xmms was a simple test case long forgotten but most still
> think all I did was 
> > make an xmms scheduler. Deleting one character from sched.c
> before all of my 
> > patches would make the scheduler ideal for xmms. Any
> braindead idiot can tune 
> > a scheduler for just one application.
> 
> Well, its the favorite example 8)
> 
> >  An application that changes it's 
> > behaviour dynamically well in the setting of a particular
> scheduler, though? 
> > Should a scheduler be tuned to suit a coding style or quirk?
> 
> > 
> 
> But the scheduler changes to a particular application?  I
> still am of
> opinion that the current scheduler in mainline 'breaks'
> priorities ...
> call it dynamic tuning or whatever you like.  Now something
> gets
> priority while something else starves.
> 
> > I should go back to lurking before people start calling me
> names. This thread 
> > has gone long enough for that. If I hadn't said anything it
> would have died 
> > out by now.
> 
> Well, I have stayed out of this for months now, as its always
> 'they' at
> fault - that app, or piece of code.  Sure, I am one of those
> whining
> users, and I have no particular interest in the scheduler code
> - that
> is if it behaves like it should.  But whatever is in now, just
> do not
> behave as expected, and call it a feature or whatever you
> want, if it
> deviates the definition, then what should we call it?  Or if
> its a
> feature, can we have the weirdness in priorities disabled by
> default
> with a sysctl or sysfs switch?
> 
> > Instead I'm drawing attention to my fundamentally flawed
> code.
> > 
> 
> The scrolling is but one part.  Just starting an app, or
> running
> 'vim /etc/fstab' for example takes ages some times, even with
> minimal load.  If xterm, gnome-term, aterm, multi-gnome-term,
> etc is broken, how do we fix it then?  What about some of the
> other issues?  If its a problem with those apps, why is it I
> still
> wonder what they are doing wrong, and it not fixed?
> 
> Do not worry, _I_ will go back to lurking about this issue
> _again_,
> but after _once_again_ seeing a issue about this being blown
> off
> as being something wrong with 'it', and some facts (you did
> see
> that the skipping code for the other user _never_ kicked in)
> were just ignored, I just could not help myself - sorry.
> 
> At least I will not experience those issues of the others, and
> hopefully Nick will not stop his work, or things change too
> much
> to adapt his patch.
> 

how much free memory do you have when this happens ?
I had 
a similar problem. It was easily reproducive doing 
a du -sh / and then trying to do other things.
It didn't happend all the time but most of the time

Doing a 
echo 16384 >/proc/sys/vm/min_free_kbytes
seems to help the kernel remember that it has some swap and he
*has* to use it in some cases

> 
> Thanks,
> 
> -- 
> Martin Schlemmer
> 

> ATTACHMENT part 2 application/pgp-signature name=signature.asc


=====
--
A mouse is a device used to point at 
the xterm you want to type in.
Kim Alm on a.s.r.

_________________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français !
Yahoo! Mail : http://fr.mail.yahoo.com

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

On Sun, Jan 04, 2004 at 11:58:27PM +0100, szonyi calin wrote:
 
> how much free memory do you have when this happens ?
> I had 
> a similar problem. It was easily reproducive doing 
> a du -sh / and then trying to do other things.
> It didn't happend all the time but most of the time

It's not the problem here. always between 200 and 400 MB free.
BTW, the system is not swapping when this happens. The scrolling
is very smooth and relatively fast (about 100 lines/s) which is
enough to understand that X eats all the CPU scrolling one line
at a time. I have yet to understand why 'ls|cat' behaves
differently, but fortunately it works and it has already saved
me some useful time.

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

On Mon, 05 Jan 2004 00:33:12 +0100, Willy Tarreau said:

> at a time. I have yet to understand why 'ls|cat' behaves
> differently, but fortunately it works and it has already saved

I suspect that ls and cat do different buffering to their outputs, and/or the
fact there's now 4 processes (ls, cat, xterm, Xserver) rather than just 3.  End
result is that things wake up in a different order and happen to schedule
better. (For instance, ls may be able to make progress while cat is blocked
waiting for the xterm to read the next block, etc).  I remember at least some
versions of 'dd' would fork off a sub-process so there would be a reader side
and a writer side with a shared memory buffer, for just that reason.


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Mike Fedyk]

On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> at a time. I have yet to understand why 'ls|cat' behaves
> differently, but fortunately it works and it has already saved
> me some useful time.

cat probably does some buffering for you, and sends the output to xterm in
larger blocks.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > at a time. I have yet to understand why 'ls|cat' behaves
> > differently, but fortunately it works and it has already saved
> > me some useful time.
> 
> cat probably does some buffering for you, and sends the output to xterm in
> larger blocks.

interestingly running ls on a remote machine in a directory with a
similiar amount of files (local xterm with ssh connection to that
machine) is also as fast as this ls | cat workaround...

Soeren

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 1297 bytes --]

On Mon, 2004-01-05 at 10:39, Soeren Sonnenburg wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> interestingly running ls on a remote machine in a directory with a
> similiar amount of files (local xterm with ssh connection to that
> machine) is also as fast as this ls | cat workaround...
> 

Maybe it is because the process generating the output, and the
xterm is not on the same box?  The X server gets too much time,
so the two childs (xterm and ls/whatever) do not get enough time?
And that might be why renice +10 `pidof X` helps? Although that
do not seem right, as the process level seems the same:

  xterm->bash->ssh vs xterm->bash->ls

Might be because the startup time is ruled out (maybe that is
the big issue - startup of child processes?).  Could it be that
the 'ls | cat' situation now again influence startup times (now
it is xterm->bash->ls->cat) if above could be taken as an reason?


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > at a time. I have yet to understand why 'ls|cat' behaves
> > differently, but fortunately it works and it has already saved
> > me some useful time.
> 
> cat probably does some buffering for you, and sends the output to xterm in
> larger blocks.

yes indeed, judging from the cat source it does chose optimal buffer
size, here 1024 byte... so it reads/writes larger chunks... and jump
scrolling takes place...

Hmmhh,
Soeren

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Martin Schlemmer]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

On Mon, 2004-01-05 at 11:18, Soeren Sonnenburg wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> yes indeed, judging from the cat source it does chose optimal buffer
> size, here 1024 byte... so it reads/writes larger chunks... and jump
> scrolling takes place...
> 

I cannot reboot right now, so have wrong kernel for testing, but could
anyone see what happens if you start X reniced to +10 or such?  Maybe
some other numbers?


-- 
Martin Schlemmer

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

On Mon, Jan 05, 2004 at 07:20:08PM +0200, Martin Schlemmer wrote:
> On Mon, 2004-01-05 at 11:18, Soeren Sonnenburg wrote:
> > On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > > at a time. I have yet to understand why 'ls|cat' behaves
> > > > differently, but fortunately it works and it has already saved
> > > > me some useful time.
> > > 
> > > cat probably does some buffering for you, and sends the output to xterm in
> > > larger blocks.
> > 
> > yes indeed, judging from the cat source it does chose optimal buffer
> > size, here 1024 byte... so it reads/writes larger chunks... and jump
> > scrolling takes place...
> > 
> 
> I cannot reboot right now, so have wrong kernel for testing, but could
> anyone see what happens if you start X reniced to +10 or such?  Maybe
> some other numbers?

I posted such tests with numbers in a previous mail in this thread. IIRC,
renicing X to +10 was indeed a work-around for most cases.

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Kenneth Johansson]

On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > at a time. I have yet to understand why 'ls|cat' behaves
> > differently, but fortunately it works and it has already saved
> > me some useful time.
> 
> cat probably does some buffering for you, and sends the output to xterm in
> larger blocks.

you can try with "ls |dd bs=1"

I also see this problem but it is not constant. I noticed that "ps ax"
sometimes takes like 10 times longer than usual. But I can only get this
in a gnome-terminal not in xterm. The problem is that it should really
not be that big difference when the load of the system is the same. 

One thing to run is to start 4 xterm and run this little thing in
everyone.

while true ;do ls ;done

In a perfect world they should all appear to scroll at the same time.
This is not what is happening on my system. 

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> you can try with "ls |dd bs=1"

yes it is slow as #$%$##@

when incresing to say bs=100 it is as fast as it used to be.

> I also see this problem but it is not constant. I noticed that "ps ax"
> sometimes takes like 10 times longer than usual. But I can only get this
> in a gnome-terminal not in xterm. The problem is that it should really
> not be that big difference when the load of the system is the same. 

well I can get it in both here. just any program that produces output...

Soeren.

solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Soeren Sonnenburg]

On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > at a time. I have yet to understand why 'ls|cat' behaves
> > > differently, but fortunately it works and it has already saved
> > > me some useful time.
> > 
> > cat probably does some buffering for you, and sends the output to xterm in
> > larger blocks.
> 
> you can try with "ls |dd bs=1"
> 
> I also see this problem but it is not constant. I noticed that "ps ax"
> sometimes takes like 10 times longer than usual. But I can only get this
> in a gnome-terminal not in xterm. The problem is that it should really
> not be that big difference when the load of the system is the same. 

Ok, there is indeed an issue in the *terminals. As above pointed out
buffering the programs output helps. Also a usleep of 5ms in the read
loop of the *terms would help.

I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
It is filled as long as there is input comming in within 10ms.
If the buffer is full or 10ms passed the buffer is written out to the
screen. This makes it also 2-3 times faster on kernel 2.4.

So in my eyes it is definitely not a scheduler bug, but in every single
terminal out there. However it seems as if kernel 2.6 s scheduling is so
fast that data can be continously outputted. And the busy loop in the
terminal makes it eat up all cpu time instead of jump scrolling take
place.

 static void
 zvt_term_readdata (gpointer data, gint fd, GdkInputCondition condition) {
[...]  
+  while ( (count>0) && (select_retval==1) && (total_count<32768) )
+  {
+         count=0;
+         int maxread=32768-total_count;
+         if (maxread>4096)
+                 maxread=4096;
+
+         count = read (fd, &buffer[total_count], maxread);
+         saveerrno=errno;
[...]
+         if (count>0)
+                 total_count+=count;
+
+         FD_ZERO(&rfds);
+         FD_SET(fd, &rfds);
+         tv.tv_sec = 0;
+         tv.tv_usec = 10000;
+         select_retval = select(fd+1, &rfds, NULL, NULL, &tv);
[...]
+  }

Soeren.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> On Mon, 2004-01-05 at 10:50, Kenneth Johansson wrote:
> > On Mon, 2004-01-05 at 00:47, Mike Fedyk wrote:
> > > On Mon, Jan 05, 2004 at 12:33:12AM +0100, Willy Tarreau wrote:
> > > > at a time. I have yet to understand why 'ls|cat' behaves
> > > > differently, but fortunately it works and it has already saved
> > > > me some useful time.
> > > 
> > > cat probably does some buffering for you, and sends the output to xterm in
> > > larger blocks.
> > 
> > you can try with "ls |dd bs=1"
> > 
> > I also see this problem but it is not constant. I noticed that "ps ax"
> > sometimes takes like 10 times longer than usual. But I can only get this
> > in a gnome-terminal not in xterm. The problem is that it should really
> > not be that big difference when the load of the system is the same. 
> 
> Ok, there is indeed an issue in the *terminals. As above pointed out
> buffering the programs output helps. Also a usleep of 5ms in the read
> loop of the *terms would help.
> 
> I fixed this issue in multi-gnome-terminal by using a buffer of 32kb.
> It is filled as long as there is input comming in within 10ms.
> If the buffer is full or 10ms passed the buffer is written out to the
> screen. This makes it also 2-3 times faster on kernel 2.4.

A factor of 2 or 3 though?

In 2.4, to ls -lA my home directory with its 510 files, took less than
0.5 sec. Currently, buffering via cat in 2.6 takes 0.5 sec. Just
straight ls -lA takes 6 seconds or so.

Does your factor of 3 bring you up to what you were seeing in 2.4, or
do you still have a regression?

-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Some witty text here,
can be any number of lines
long

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Tim Connors]

Tim Connors <tconnors+linuxkernel1080970205@astro.swin.edu.au> said on Sat, 3 Apr 2004 15:35:29 +1000:
> Soeren Sonnenburg <kernel@nn7.de> said on Fri, 02 Apr 2004 20:22:12 +0200:
> > Ok, there is indeed an issue in the *terminals. As above pointed out
> > buffering the programs output helps. Also a usleep of 5ms in the read
> > loop of the *terms would help.

I forgot to mention that I see other 2.6 scheduler regressions too.

I am on ADSL, and run wwwoffle to cache my www feed. In mozilla,
running through the proxy will quite often crawl - you can see it
displaying tags and words one by one. I have been unable to work out
under what circumstances this is triggered, but it is easily
understood given that mozilla on this box chews CPU (especially when
rendering whilst downloading), as does X. So this is much like the
xterm situation - 2 CPU chewing things, interacting with a third short
lived low CPU job (ls or some wwwoffle fork) that the other two are
relying on (mozilla and xterm directly, X indirectly)

This is my reason for not thinking this is an xterm bug. The scheduler
looks inherently fragile.


To help you work out my datapoint in relation to someone elses, my box
is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
accelarated (crappy old nvidia card), so X likes to chew CPU easily.





-- 
TimC -- http://astronomy.swin.edu.au/staff/tconnors/
Cult: (n) a small, unpopular religion.
Religion: (n) a large, popular cult.

Re: solved (was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!)

[Jamie Lokier]

Tim Connors wrote:
> So this is much like the xterm situation - 2 CPU chewing things,
> interacting with a third short lived low CPU job (ls or some
> wwwoffle fork) that the other two are relying on (mozilla and xterm
> directly, X indirectly)
> 
> This is my reason for not thinking this is an xterm bug. The scheduler
> looks inherently fragile.

It looks like scheduler fragility to me too.

I see the same with Gnome Terminal.  Most of the time, ls or cat are
fast, with jump scrolling.  But occasionally they are much slower.

There are two stable scheduling modes here - rapid switching with a
few characters moved, or slow switching with lots of characters moved.

We'd like the former to decay spontaneously to the latter, quickly.

I like the observation that the terminal program can solve this
problem by introducing a short delay after receiving a few bytes.
However, that's not feasible for X itself, which must draw things very
soon after receiving the commands so that animations are smooth when
that's intended, which happens when a client deliberately sleeps
rather than being preempted by having just sent a command to X.

I think the problem is that the scheduler is aggressively preempting a
task which has just written through a pipe/terminal/socket, when the
task at the other end becomes ready to run as a result.

If the writing task is still runnable, usually you want the writing
task to continue for a little while more.  The scheduler is getting
that right most times on my box with ls+Gnome Terminal+X, but
occasionally gets stuck in a mode where it consistently gets it wrong
until all the programs become idle again.  Then it recovers.
This mode is quite rare, once every few days.

> To help you work out my datapoint in relation to someone elses, my box
> is a 500MHz AMD KII, now running 2.6.4. The video card is in no way
> accelarated (crappy old nvidia card), so X likes to chew CPU easily.

Dual Athlon 1500 MHz, Matrox Millenium video card.

-- Jamie

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 23:58, szonyi calin wrote:
[...]
> how much free memory do you have when this happens ?
> I had 
> a similar problem. It was easily reproducive doing 
> a du -sh / and then trying to do other things.
> It didn't happend all the time but most of the time
> 
> Doing a 
> echo 16384 >/proc/sys/vm/min_free_kbytes
> seems to help the kernel remember that it has some swap and he
> *has* to use it in some cases

Hmmm... this machine has 1G memory and it happens after fresh
reboots.... so memory is not the issue.

Soeren.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Lincoln Dale]

At 07:09 PM 4/01/2004, Soeren Sonnenburg wrote:
[..]
>Looking at that how can it not be a scheduling problem ....

out of interest, have you tried to see how 2.4.xx compares when compiled 
with HZ set to 1000?
(or conversely, 2.6 compiled with HZ set to 100)


cheers,

lincoln.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Lincoln Dale wrote:

> At 07:09 PM 4/01/2004, Soeren Sonnenburg wrote:
> [..]
>
>> Looking at that how can it not be a scheduling problem ....
>
>
> out of interest, have you tried to see how 2.4.xx compares when 
> compiled with HZ set to 1000?
> (or conversely, 2.6 compiled with HZ set to 100)
>

Or, out of interest, an alternate scheduler?

http://www.kerneltrap.org/~npiggin/w29p2.gz
(applies 2.6.1-rc1-mm1, please renice X to -10 or so)

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

[...]
> Or, out of interest, an alternate scheduler?
> 
> http://www.kerneltrap.org/~npiggin/w29p2.gz
> (applies 2.6.1-rc1-mm1, please renice X to -10 or so)

Thats nothing *I* can try out as I am on the powerpc benh tree.

Soeren.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Mike Fedyk]

On Sun, Jan 04, 2004 at 11:24:39AM +0100, Soeren Sonnenburg wrote:
> [...]
> > Or, out of interest, an alternate scheduler?
> > 
> > http://www.kerneltrap.org/~npiggin/w29p2.gz
> > (applies 2.6.1-rc1-mm1, please renice X to -10 or so)
> 
> Thats nothing *I* can try out as I am on the powerpc benh tree.
> 

Says who?  The scheduler isn't platform specific.  Nick, do you have any per
arch defines in your patch?

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 12:12, Mike Fedyk wrote:
> On Sun, Jan 04, 2004 at 11:24:39AM +0100, Soeren Sonnenburg wrote:
> > [...]
> > > Or, out of interest, an alternate scheduler?
> > > 
> > > http://www.kerneltrap.org/~npiggin/w29p2.gz
> > > (applies 2.6.1-rc1-mm1, please renice X to -10 or so)
> > 
> > Thats nothing *I* can try out as I am on the powerpc benh tree.
> > 
> 
> Says who?  The scheduler isn't platform specific.  Nick, do you have any per
> arch defines in your patch?

Says me, as quite a lot of stuff does not apply cleanly... and probably
same with when applying -mm1.

S

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Mike Fedyk]

On Sun, Jan 04, 2004 at 12:17:10PM +0100, Soeren Sonnenburg wrote:
> Says me, as quite a lot of stuff does not apply cleanly... and probably
> same with when applying -mm1.

Does -mm1 compile on ppc for you?

If not, then post some compile errors to the list...

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

On Sun, Jan 04, 2004 at 03:12:57AM -0800, Mike Fedyk wrote:
> On Sun, Jan 04, 2004 at 11:24:39AM +0100, Soeren Sonnenburg wrote:
> > [...]
> > > Or, out of interest, an alternate scheduler?
> > > 
> > > http://www.kerneltrap.org/~npiggin/w29p2.gz
> > > (applies 2.6.1-rc1-mm1, please renice X to -10 or so)
> > 
> > Thats nothing *I* can try out as I am on the powerpc benh tree.
> > 
> 
> Says who?  The scheduler isn't platform specific.  Nick, do you have any per
> arch defines in your patch?

I found slight changes to arch specific files, but IMHO this should not be
the problem. But AFAIR, benh's ppc patches are somewhat complete and may
introduce conflicts.

BTW, for Nick, the patch didn't compile, I had to change sched_clock()
definition from unsigned long long to unsigned long in
arch/i386/kernel/timers/timer_tsc.c. Don't know if it was the right thing to
do, but it compiles and boots.

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Willy Tarreau wrote:

>On Sun, Jan 04, 2004 at 03:12:57AM -0800, Mike Fedyk wrote:
>
>>On Sun, Jan 04, 2004 at 11:24:39AM +0100, Soeren Sonnenburg wrote:
>>
>>>[...]
>>>
>>>>Or, out of interest, an alternate scheduler?
>>>>
>>>>http://www.kerneltrap.org/~npiggin/w29p2.gz
>>>>(applies 2.6.1-rc1-mm1, please renice X to -10 or so)
>>>>
>>>Thats nothing *I* can try out as I am on the powerpc benh tree.
>>>
>>>
>>Says who?  The scheduler isn't platform specific.  Nick, do you have any per
>>arch defines in your patch?
>>
>
>I found slight changes to arch specific files, but IMHO this should not be
>the problem. But AFAIR, benh's ppc patches are somewhat complete and may
>introduce conflicts.
>

The problem is just that the mm tree is currently carrying some
intrusive (patch wise, not functionality wise) scheduler stuff
which will go to linus which is why I am working from there. So
the patch will only apply cleanly on the mm tree, but only because
of the cleanups + minor fixes. Nothing fundamental.

>
>BTW, for Nick, the patch didn't compile, I had to change sched_clock()
>definition from unsigned long long to unsigned long in
>arch/i386/kernel/timers/timer_tsc.c. Don't know if it was the right thing to
>do, but it compiles and boots.
>

Yeah thats right

Nicks's scheduler's OK [was Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!]

[Willy Tarreau]

On Sun, Jan 04, 2004 at 08:17:24PM +1100, Nick Piggin wrote:
> 
> Or, out of interest, an alternate scheduler?
> 
> http://www.kerneltrap.org/~npiggin/w29p2.gz
> (applies 2.6.1-rc1-mm1, please renice X to -10 or so)

Nick's scheduler seems rather interesting. X is nearly insensible to renice
as it was in 2.4. I cannot get a slow scrolling anymore except with xterm +j.
And the differences in time between renice -15 and renice +10 are about 5%,
which is perfectly acceptable to me. I got back the 2.4 behaviour (= a usable
desktop). Now recompiling Con's "noint" patch on 2.6.0 for reference.

Cheers,
Willy

1) X not reniced.
=================

w29p2$ time xterm -e seq 1 5000

real    0m0.487s
user    0m0.244s
sys     0m0.075s

w29p2$ time xterm -e sh -c "ls -l incoming tmp"

real    0m1.000s
user    0m0.491s
sys     0m0.106s

2) Now renicing X to -10 as suggested by Nick
=============================================

w29p2$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.998s
user    0m0.476s
sys     0m0.126s

w29p2$ time xterm -e seq 1 5000

real    0m0.420s
user    0m0.276s
sys     0m0.082s

3) Now renicing X to +10 to compare with my previous tests
==========================================================

w29p2$ time xterm -e seq 1 5000

real    0m0.528s
user    0m0.282s
sys     0m0.056s

w29p2$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.967s
user    0m0.498s
sys     0m0.111s

w29p2$ time find incoming tmp |wc -l
  204276

real    0m0.937s
user    0m0.299s
sys     0m0.593s

w29p2$ time xterm -e sh -c "find incoming tmp"

real    0m21.368s
user    0m5.838s
sys     0m1.341s

top - 12:31:23 up 11 min,  6 users,  load average: 0.59, 0.68, 0.39
Tasks:  61 total,   2 running,  59 sleeping,   0 stopped,   0 zombie
Cpu(s):  29.1% user,   6.8% system,  64.1% nice,   0.0% idle,   0.0% IO-wait
Mem:    515248k total,   231160k used,   284088k free,    99172k buffers
Swap:   265064k total,        0k used,   265064k free,    45328k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  302 root      50  10 21500  11m  11m R 63.9  2.3   1:41.67 X                 
  475 root      37   0  4764 2660 3844 S 31.0  0.5   0:01.68 xterm             
  476 willy     39   0  1420  540 1252 S  3.9  0.1   0:00.23 find              
    1 root      25   0   348  192  316 S  0.0  0.0   0:05.06 init              
    2 root      41  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0       
    3 root      12 -10     0    0    0 S  0.0  0.0   0:00.20 events/0          
    4 root      16 -10     0    0    0 S  0.0  0.0   0:00.03 kblockd/0         

4) Same with renice -15 :
=========================

w29p2$ time xterm -e sh -c "find incoming tmp"

real    0m19.147s
user    0m5.085s
sys     0m1.238s

top - 12:33:33 up 13 min,  6 users,  load average: 0.65, 0.68, 0.42
Tasks:  61 total,   4 running,  57 sleeping,   0 stopped,   0 zombie
Cpu(s):  93.1% user,   6.9% system,   0.0% nice,   0.0% idle,   0.0% IO-wait
Mem:    515248k total,   233464k used,   281784k free,    99540k buffers
Swap:   265064k total,        0k used,   265064k free,    45992k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  302 root      24 -15 21500  11m  11m R 66.2  2.3   2:08.25 X                 
  482 root      40   0  4764 2660 3844 R 29.2  0.5   0:01.50 xterm             
  483 willy     37   0  1420  540 1252 R  4.9  0.1   0:00.25 find              
    1 root      26   0   348  192  316 S  0.0  0.0   0:05.06 init              
    2 root      41  19     0    0    0 S  0.0  0.0   0:00.00 ksoftirqd/0       
    3 root      12 -10     0    0    0 S  0.0  0.0   0:00.21 events/0          
    4 root      15 -10     0    0    0 S  0.0  0.0   0:00.03 kblockd/0         
    5 root      39   0     0    0    0 S  0.0  0.0   0:00.00 pdflush           
    6 root      23   0     0    0    0 S  0.0  0.0   0:00.00 pdflush           

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

Now testing Con's noint patch against 2.6.0. It returns somewhat simmilar
results to Nick's w29p2, and behaves normally. The only noticeable difference
is that a simple task like "while :; do :; done&" eats about 100ms each second,
so if you start 10 of these, you're able to type only once a second (tested).
But I understand that this 'dumbness' was exactly the goal of this patch.
I think that I'll try to use 2.6 + Nick's scheduler for some time on my
notebook to get an overall idea on how it behaves.

BTW, Nick, does your patch rely on -mm1 exclusive features, or would it be
possible to back-port it to plain 2.6 ?

Cheers,
Willy

1) X not reniced.
=================

noint$ time xterm -e seq 1 5000

real    0m0.548s
user    0m0.267s
sys     0m0.052s

noint$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.974s
user    0m0.481s
sys     0m0.091s

2) Now renicing X to -10
========================

noint$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.997s
user    0m0.471s
sys     0m0.101s

noint$ time xterm -e seq 1 5000

real    0m0.390s
user    0m0.261s
sys     0m0.058s

3) Now renicing X to +10 to compare with my previous tests
==========================================================

noint$ time xterm -e seq 1 5000

real    0m0.452s
user    0m0.257s
sys     0m0.070s

noint$ time xterm -e sh -c "ls -l incoming tmp"

real    0m0.956s
user    0m0.441s
sys     0m0.091s

noint$ time find incoming tmp |wc -l
  204276

real    0m0.914s
user    0m0.257s
sys     0m0.653s

noint$ time xterm -e sh -c "find incoming tmp"

real    0m23.107s
user    0m5.652s
sys     0m1.631s

top - 12:56:42 up 9 min,  5 users,  load average: 0.61, 0.54, 0.26
Tasks:  59 total,   3 running,  56 sleeping,   0 stopped,   0 zombie
Cpu(s):  24.8% user,   8.9% system,  66.3% nice,   0.0% idle,   0.0% IO-wait
Mem:    515292k total,   217816k used,   297476k free,    94344k buffers
Swap:   265064k total,        0k used,   265064k free,    45236k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  239 root      30  10 23632  13m  11m R 66.2  2.7   0:11.11 X                 
  374 root      20   0  4764 2660 3844 S 24.7  0.5   0:01.22 xterm             
  375 willy     20   0  1420  540 1252 S  7.9  0.1   0:00.35 find              
  373 root      20   0  1692  940 1556 R  1.0  0.2   0:00.08 top               

4) Same with renice -15 :
=========================

noint$ time xterm -e sh -c "find incoming tmp"

real    0m22.622s
user    0m5.681s
sys     0m1.807s

top - 12:57:45 up 10 min,  5 users,  load average: 0.33, 0.48, 0.26
Tasks:  59 total,   3 running,  56 sleeping,   0 stopped,   0 zombie
Cpu(s):  88.8% user,  11.2% system,   0.0% nice,   0.0% idle,   0.0% IO-wait
Mem:    515292k total,   217880k used,   297412k free,    94404k buffers
Swap:   265064k total,        0k used,   265064k free,    45236k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  Command           
  239 root       5 -15 23632  13m  11m S 64.3  2.7   0:25.55 X                 
  380 root      20   0  4764 2660 3844 R 26.1  0.5   0:00.82 xterm             
  381 willy     20   0  1420  540 1252 S  7.5  0.1   0:00.23 find              

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Willy Tarreau wrote:

>Now testing Con's noint patch against 2.6.0. It returns somewhat simmilar
>results to Nick's w29p2, and behaves normally. The only noticeable difference
>is that a simple task like "while :; do :; done&" eats about 100ms each second,
>so if you start 10 of these, you're able to type only once a second (tested).
>But I understand that this 'dumbness' was exactly the goal of this patch.
>I think that I'll try to use 2.6 + Nick's scheduler for some time on my
>notebook to get an overall idea on how it behaves.
>
>BTW, Nick, does your patch rely on -mm1 exclusive features, or would it be
>possible to back-port it to plain 2.6 ?
>

No, you should be able to port it to 2.6 quite easily if you just apply
the scheduler patches from -mm1 to it first. I'll do a proper release
in the next day or so and include patches for both trees.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

Hi Nick,

during a kernel compilation today while browsing the net on another
virtual screen, I noticed extremely long delays to redraw the screen.
My setup is the following :
 - kernel 2.6.1-rc1-mm1 + w29p2
 - X either reniced to -10 or 0 (doesnt make a difference)
 - window maker
 - opera open on a fairly simple page (marc.theaimsgroup.com) in VS 1
 - 2 xterms on VS2
 - 4 xterms in VS3, one of which runs make -j 3 bzImage, another one 'top'.

-> going from VS3 to VS2 then VS1 takes up to 5 or 6 seconds redrawing
   parts of windows on each screen. I also noticed that make -j 8 dep
   blocked the xterm it was run in for about a third of a second about
   twice a second. It does not seem to slow the process however since
   it takes the same amount of time as make -j 8 dep >/dev/null.


Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Nick Piggin]

Willy Tarreau wrote:

>Hi Nick,
>
>during a kernel compilation today while browsing the net on another
>virtual screen, I noticed extremely long delays to redraw the screen.
>My setup is the following :
> - kernel 2.6.1-rc1-mm1 + w29p2
> - X either reniced to -10 or 0 (doesnt make a difference)
> - window maker
> - opera open on a fairly simple page (marc.theaimsgroup.com) in VS 1
> - 2 xterms on VS2
> - 4 xterms in VS3, one of which runs make -j 3 bzImage, another one 'top'.
>
>-> going from VS3 to VS2 then VS1 takes up to 5 or 6 seconds redrawing
>   parts of windows on each screen. I also noticed that make -j 8 dep
>   blocked the xterm it was run in for about a third of a second about
>   twice a second. It does not seem to slow the process however since
>   it takes the same amount of time as make -j 8 dep >/dev/null.
>

Hi Willy,
Yeah the w29p2 patch you got is broken. I'll fix it.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sun, 2004-01-04 at 09:54, Lincoln Dale wrote:
> At 07:09 PM 4/01/2004, Soeren Sonnenburg wrote:
> [..]
> >Looking at that how can it not be a scheduling problem ....
> 
> out of interest, have you tried to see how 2.4.xx compares when compiled 
> with HZ set to 1000?
> (or conversely, 2.6 compiled with HZ set to 100)

assuming you mean changing the HZ value in include/param.h to 1000/100
yes 2.4 with HZ=1000 is fine and 2.6 with HZ=100 still #%$@$^&!!

Soeren

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[venom]

On Sun, 4 Jan 2004, Soeren Sonnenburg wrote:

> > out of interest, have you tried to see how 2.4.xx compares when compiled
> > with HZ set to 1000?
> > (or conversely, 2.6 compiled with HZ set to 100)
>
> assuming you mean changing the HZ value in include/param.h to 1000/100
> yes 2.4 with HZ=1000 is fine and 2.6 with HZ=100 still #%$@$^&!!
>
mmhhh! depends...

On a DB that has to write often big, long data streams HZ=100 on a 2.6.0 kernel
allows a better performance instead of HZ=1000 (no kernel preemption).

on a DB that has to read often small, short data streams HZ=1000 is better than
HZ=100 with a 2.6.0 kernel.

Luigi

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

Hi Con,

On Sun, Jan 04, 2004 at 08:00:06AM +1100, Con Kolivas wrote:
> There is a BASH bug that Linus noticed brought out by the more sensitive 
> timing in 2.6. The BASH developer has been informed it is there and it is 
> fixed in the latest version. Perhaps you're both seeing that. Check the lkml 
> archives.

I don't think it has anything to do with the bash bug, because it only
involved pipes creation. Bash creates no pipe when you simply launch
'ls -l' in an xterm.

It seems that I'll start some new compilations this evening just to refresh
my mind on this problem...

Cheers,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Bob Gill]

Hi.  I've seen this too.  It happens when I run ls -al or ls -alR on an
uncached directory (usually after boot).  If the directory is big, I
wait 3-4 seconds for a listing.  I've updated bash with yum (currently
I'm using bash 2.05b on Fedora with 2.6.1-pre1).  It probably doesn't
help that my drives are slow (5400 rpm).  Doing regression performance
testing on the drives using hdparm helps, but the problem still exists.

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

On Sat, 2004-01-03 at 20:03, Mark Hahn wrote:
> > Anyone else with that problem - ideas of the cause ?
> 
> the 2.6 scheduler tries to be much more adaptive.  doesn't it look like
> it's decided to make some wrong decisions wrt scheduling in your first run?

yeah, I think so... but as generating output in a shell is a very common
thing to do there should either be an option to turn that unwanted
behaviour off or to fix this issue...

Soeren

xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Soeren Sonnenburg]

[-- Attachment #1: Type: text/plain, Size: 905 bytes --]

Hi...

I notice a severe speed regression with 2.6 and working in e.g. xterm on
X11 (no framebuffer device in use). It is all about scrolling speed. I
used to type e.g. just dmesg in a fullscreen xterm (or MGT) in 2.4 and
the whole output was there lightning fast, i.e. 

in 2.4:
=======
time dmesg
real    0m0.096s
user    0m0.007s
sys     0m0.005s

in 2.6 (first run)
======
time dmesg
[...]
real    0m3.054s
user    0m0.008s
sys     0m0.008s

in 2.6 (second run)
===================
dmesg >d
time cat d
real    0m0.096s
user    0m0.002s
sys 0m0.005s

in 2.6 (third run)
==================
time dmesg
real    0m0.095s
user    0m0.007s
sys     0m0.004s

So it is like 30 times slower on 2.6. when running for the first time...
this also happens if I do e.g. find ./ and watch the output pass by... 

This is without preemption on powerpc...

Anyone else with that problem - ideas of the cause ?

Soeren.

[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 7048 bytes --]

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy Tarreau]

Hi !

On Sat, Jan 03, 2004 at 07:52:36PM +0100, Soeren Sonnenburg wrote:
 
> So it is like 30 times slower on 2.6. when running for the first time...
> this also happens if I do e.g. find ./ and watch the output pass by... 
> 
> This is without preemption on powerpc...
> 
> Anyone else with that problem - ideas of the cause ?

Yes, already reported a while ago. I was having this problem with ls in
large directories where it sometimes took more than 10 seconds to scroll.
It is clearly a scheduler problem, but most people don't seem concerned
by this one since the most important to their ears is that xmms should not
skip during make -j $UINT_MAX :-/ It's a bit of a shame since other than
that, it seems stable enough for daily use on a desktop, but this problem
is too much annoying for me to definitely switch to 2.6. I wanted to give
a try to Nick's scheduler, but didn't have time yet.

Regards,
Willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Peter Chubb]

>>>>> "Willy" == Willy Tarreau <willy@w.ods.org> writes:

Willy> Hi !  On Sat, Jan 03, 2004 at 07:52:36PM +0100, Soeren
Willy> Sonnenburg wrote:
 
>> So it is like 30 times slower on 2.6. when running for the first
>> time...  this also happens if I do e.g. find ./ and watch the
>> output pass by...
>> 
>> This is without preemption on powerpc...
>> 
>> Anyone else with that problem - ideas of the cause ?

I see a very similar problem ... seems like a process doing disc I.O
isn't being woken up fast enough after the I/O completes.

For some processes, allowing interrupts back on (hdparm -u1) helps;
for others, switching to the deadline elevator helps; neither are
complete solutions.

PeterC

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Willy TARREAU]

Hi Peter,

On Mon, Jan 05, 2004 at 07:47:43AM +1100, Peter Chubb wrote:

> >> So it is like 30 times slower on 2.6. when running for the first
> >> time...  this also happens if I do e.g. find ./ and watch the
> >> output pass by...
> >> 
> >> This is without preemption on powerpc...
> >> 
> >> Anyone else with that problem - ideas of the cause ?
> 
> I see a very similar problem ... seems like a process doing disc I.O
> isn't being woken up fast enough after the I/O completes.
> 
> For some processes, allowing interrupts back on (hdparm -u1) helps;
> for others, switching to the deadline elevator helps; neither are
> complete solutions.

This is not I/O related since the problem happens even with simple
programs such as dmesg and seq.

Cheers,
willy

Re: xterm scrolling speed - scheduling weirdness in 2.6 ?!

[Peter Chubb]

>>>>> "Willy" == Willy TARREAU <willy@w.ods.org> writes:
> >> So it is like 30 times slower on 2.6. when running for the first
> >> time...  this also happens if I do e.g. find ./ and watch the
> >> output pass by...
>> 
>> For some processes, allowing interrupts back on (hdparm -u1) helps;
>> for others, switching to the deadline elevator helps; neither are
>> complete solutions.

Willy> This is not I/O related since the problem happens even with
Willy> simple programs such as dmesg and seq.

Yes ... the first time you run a program it's fetched from disc and
shows a significant slowdown.  Second and subsequent times, it runs at
normal speed.  At least, that's what *I'm* seeing.

Peter C