BFS vs. mainline scheduler benchmarks and measurements

BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

hi Con,

I've read your BFS announcement/FAQ with great interest:

    http://ck.kolivas.org/patches/bfs/bfs-faq.txt

First and foremost, let me say that i'm happy that you are hacking 
the Linux scheduler again. It's perhaps proof that hacking the 
scheduler is one of the most addictive things on the planet ;-)

I understand that BFS is still early code and that you are not 
targeting BFS for mainline inclusion - but BFS is an interesting 
and bold new approach, cutting a _lot_ of code out of 
kernel/sched*.c, so it raised my curiosity and interest :-)

In the announcement and on your webpage you have compared BFS to 
the mainline scheduler in various workloads - showing various 
improvements over it. I have tried and tested BFS and ran a set of 
benchmarks - this mail contains the results and my (quick) 
findings.

So ... to get to the numbers - i've tested both BFS and the tip of 
the latest upstream scheduler tree on a testbox of mine. I 
intentionally didnt test BFS on any really large box - because you 
described its upper limit like this in the announcement:

-----------------------
|
| How scalable is it?
|
| I don't own the sort of hardware that is likely to suffer from 
| using it, so I can't find the upper limit. Based on first 
| principles about the overhead of locking, and the way lookups 
| occur, I'd guess that a machine with more than 16 CPUS would 
| start to have less performance. BIG NUMA machines will probably 
| suck a lot with this because it pays no deference to locality of 
| the NUMA nodes when deciding what cpu to use. It just keeps them 
| all busy. The so-called "light NUMA" that constitutes commodity 
| hardware these days seems to really like BFS.
|
-----------------------

I generally agree with you that "light NUMA" is what a Linux 
scheduler needs to concentrate on (at most) in terms of 
scalability. Big NUMA, 4096 CPUs is not very common and we tune the 
Linux scheduler for desktop and small-server workloads mostly.

So the testbox i picked fits into the upper portion of what i 
consider a sane range of systems to tune for - and should still fit 
into BFS's design bracket as well according to your description: 
it's a dual quad core system with hyperthreading. It has twice as 
many cores as the quad you tested on but it's not excessive and 
certainly does not have 4096 CPUs ;-)

Here are the benchmark results:

  kernel build performance:
     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild.jpg     

  pipe performance:
     http://redhat.com/~mingo/misc/bfs-vs-tip-pipe.jpg

  messaging performance (hackbench):
     http://redhat.com/~mingo/misc/bfs-vs-tip-messaging.jpg  

  OLTP performance (postgresql + sysbench)
     http://redhat.com/~mingo/misc/bfs-vs-tip-oltp.jpg

Alas, as it can be seen in the graphs, i can not see any BFS 
performance improvements, on this box.

Here's a more detailed description of the results:

| Kernel build performance
---------------------------

  http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild.jpg     

In the kbuild test BFS is showing significant weaknesses up to 16 
CPUs. On 8 CPUs utilized (half load) it's 27.6% slower. All results 
(-j1, -j2... -j15 are slower. The peak at 100% utilization at -j16 
is slightly stronger under BFS, by 1.5%. The 'absolute best' result 
is sched-devel at -j64 with 46.65 seconds - the best BFS result is 
47.38 seconds (at -j64) - 1.5% better.

| Pipe performance
-------------------

  http://redhat.com/~mingo/misc/bfs-vs-tip-pipe.jpg

Pipe performance is a very simple test, two tasks message to each 
other via pipes. I measured 1 million such messages:

   http://redhat.com/~mingo/cfs-scheduler/tools/pipe-test-1m.c

The pipe test ran a number of them in parallel:

   for ((i=0;i<$NR;i++)); do ~/sched-tests/pipe-test-1m & done; wait

and measured elapsed time. This tests two things: basic scheduler 
performance and also scheduler fairness. (if one of these parallel 
jobs is delayed unfairly then the test will finish later.)

[ see further below for a simpler pipe latency benchmark as well. ]

As can be seen in the graph BFS performed very poorly in this test: 
at 8 pairs of tasks it had a runtime of 45.42 seconds - while 
sched-devel finished them in 3.8 seconds.

I saw really bad interactivity in the BFS test here - the system 
was starved for as long as the test ran. I stopped the tests at 8 
loops - the system was unusable and i was getting IO timeouts due 
to the scheduling lag:

 sd 0:0:0:0: [sda] Unhandled error code
 sd 0:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
 end_request: I/O error, dev sda, sector 81949243
 Aborting journal on device sda2.
 ext3_abort called.
 EXT3-fs error (device sda2): ext3_journal_start_sb: Detected aborted journal
 Remounting filesystem read-only

I measured interactivity during this test:

   $ time ssh aldebaran /bin/true
   real  2m17.968s
   user  0m0.009s
   sys   0m0.003s

A single command took more than 2 minutes.

| Messaging performance
------------------------

  http://redhat.com/~mingo/misc/bfs-vs-tip-messaging.jpg  

Hackbench ran better - but mainline sched-devel is significantly 
faster for smaller and larger loads as well. With 20 groups 
mainline ran 61.5% faster.

| OLTP performance
--------------------

http://redhat.com/~mingo/misc/bfs-vs-tip-oltp.jpg

As can be seen in the graph for sysbench OLTP performance 
sched-devel outperforms BFS on each of the main stages:

   single client load   (   1 client  -   6.3% faster )
   half load            (   8 clients -  57.6% faster )
   peak performance     (  16 clients - 117.6% faster )
   overload             ( 512 clients - 288.3% faster )

| Other tests
--------------

I also tested a couple of other things, such as lat_tcp:

  BFS:          TCP latency using localhost: 16.5608 microseconds
  sched-devel:  TCP latency using localhost: 13.5528 microseconds [22.1% faster]

lat_pipe:

  BFS:          Pipe latency: 4.9703 microseconds
  sched-devel:  Pipe latency: 2.6137 microseconds [90.1% faster]

General interactivity of BFS seemed good to me - except for the 
pipe test when there was significant lag over a minute. I think 
it's some starvation bug, not an inherent design property of BFS, 
so i'm looking forward to re-test it with the fix.

Test environment: i used latest BFS (205 and then i re-ran under 
208 and the numbers are all from 208), and the latest mainline 
scheduler development tree from:

  http://people.redhat.com/mingo/tip.git/README

Commit 840a065 in particular. It's on a .31-rc8 base while BFS is 
on a .30 base - will be able to test BFS on a .31 base as well once 
you release it. (but it doesnt matter much to the results - there 
werent any heavy core kernel changes impacting these workloads.)

The system had enough RAM to have the workloads cached, and i 
repeated all tests to make sure it's all representative. 
Nevertheless i'd like to encourage others to repeat these (or 
other) tests - the more testing the better.

I also tried to configure the kernel in a BFS friendly way, i used 
HZ=1000 as recommended, turned off all debug options, etc. The 
kernel config i used can be found here:

  http://redhat.com/~mingo/misc/config

( Let me know if you need any more info about any of the tests i
  conducted. )

Also, i'd like to outline that i agree with the general goals 
described by you in the BFS announcement - small desktop systems 
matter more than large systems. We find it critically important 
that the mainline Linux scheduler performs well on those systems 
too - and if you (or anyone else) can reproduce suboptimal behavior 
please let the scheduler folks know so that we can fix/improve it.

I hope to be able to work with you on this, please dont hesitate 
sending patches if you wish - and we'll also be following BFS for 
good ideas and code to adopt to mainline.

Thanks,

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

Ingo Molnar wrote:
> So the testbox i picked fits into the upper portion of what i
> consider a sane range of systems to tune for - and should still fit
> into BFS's design bracket as well according to your description:
> it's a dual quad core system with hyperthreading.

Ingo,

Nice that you've looked into this.

Would it be possible for you to run the same tests on e.g. a dual core 
and/or a UP system (or maybe just offline some CPUs?)? It would be very 
interesting to see whether BFS does better in the lower portion of the 
range, or if the differences you show between the two schedulers are 
consistent across the range.

Cheers,
FJP

[quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Frans Pop <elendil@planet.nl> wrote:

> Ingo Molnar wrote:
> > So the testbox i picked fits into the upper portion of what i
> > consider a sane range of systems to tune for - and should still fit
> > into BFS's design bracket as well according to your description:
> > it's a dual quad core system with hyperthreading.
> 
> Ingo,
> 
> Nice that you've looked into this.
> 
> Would it be possible for you to run the same tests on e.g. a dual 
> core and/or a UP system (or maybe just offline some CPUs?)? It 
> would be very interesting to see whether BFS does better in the 
> lower portion of the range, or if the differences you show between 
> the two schedulers are consistent across the range.

Sure!

Note that usually we can extrapolate ballpark-figure quad and dual 
socket results from 8 core results. Trends as drastic as the ones 
i reported do not get reversed as one shrinks the number of cores. 

[ This technique is not universal - for example borderline graphs
  on cannot be extrapolated down reliably - but the graphs i 
  posted were far from borderline. ]

Con posted single-socket quad comparisons/graphs so to make it 100% 
apples to apples i re-tested with a single-socket (non-NUMA) quad as 
well, and have uploaded the new graphs/results to:

  kernel build performance on quad:
     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg

  pipe performance on quad:
     http://redhat.com/~mingo/misc/bfs-vs-tip-pipe-quad.jpg

  messaging performance (hackbench) on quad:
     http://redhat.com/~mingo/misc/bfs-vs-tip-messaging-quad.jpg

  OLTP performance (postgresql + sysbench) on quad:
     http://redhat.com/~mingo/misc/bfs-vs-tip-oltp-quad.jpg

It shows similar curves and behavior to the 8-core results i posted 
- BFS is slower than mainline in virtually every measurement. The 
ratios are different for different parts of the graphs - but the 
trend is similar.

I also re-ran a few standalone kernel latency tests with a single 
quad:

lat_tcp:

  BFS:          TCP latency using localhost: 16.9926 microseconds
  sched-devel:  TCP latency using localhost: 12.4141 microseconds [36.8% faster]

  as a comparison, the 8 core lat_tcp result was:

  BFS:          TCP latency using localhost: 16.5608 microseconds
  sched-devel:  TCP latency using localhost: 13.5528 microseconds [22.1% faster]

lat_pipe quad result:

  BFS:          Pipe latency: 4.6978 microseconds
  sched-devel:  Pipe latency: 2.6860 microseconds [74.8% faster]

  as a comparison, the 8 core lat_pipe result was:

  BFS:          Pipe latency: 4.9703 microseconds
  sched-devel:  Pipe latency: 2.6137 microseconds [90.1% faster]

On the desktop interactivity front, i also still saw that bad 
starvation artifact with BFS with multiple copies of CPU-bound 
pipe-test-1m.c running in parallel:

   http://redhat.com/~mingo/cfs-scheduler/tools/pipe-test-1m.c

Start up a few copies of them like this:

  for ((i=0;i<32;i++)); do ./pipe-test-1m & done

and the quad eventually came to a halt here - until the tasks 
finished running.

I also tested a few key data points on dual core and it shows 
similar trends as well (as expected from the 8 and 4 core results).

But ... i'd really encourage everyone to test these things yourself 
as well and not take anyone's word on this as granted. The more 
people provide numbers, the better. The latest BFS patch can be 
found at:

  http://ck.kolivas.org/patches/bfs/

The mainline sched-devel tree can be found at:

  http://people.redhat.com/mingo/tip.git/README

Thanks,

	Ingo

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Stefan Richter]

Ingo Molnar wrote:
> i'd really encourage everyone to test these things yourself 
> as well and not take anyone's word on this as granted. The more 
> people provide numbers, the better.

Besides mean values from bandwidth and latency focused tests, standard 
deviations or variance, or e.g. 90th percentiles and perhaps maxima of 
latency focused tests might be of interest.  Or graphs with error bars.
-- 
Stefan Richter
-=====-==--= =--= --===
http://arcgraph.de/sr/

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Markus Tornqvist]

Please Cc me as I'm not a subscriber.

(LKML bounced this message once already for 8-bit headers, I'm retrying
now - sorry if someone gets it twice)

On Mon, Sep 07, 2009 at 02:16:13PM +0200, Ingo Molnar wrote:
>
>Con posted single-socket quad comparisons/graphs so to make it 100% 
>apples to apples i re-tested with a single-socket (non-NUMA) quad as 
>well, and have uploaded the new graphs/results to:
>
>  kernel build performance on quad:
>     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
[...]
>
>It shows similar curves and behavior to the 8-core results i posted 
>- BFS is slower than mainline in virtually every measurement. The 
>ratios are different for different parts of the graphs - but the 
>trend is similar.

Dude, not cool.

1. Quad HT is not the same as a 4-core desktop, you're doing it with 8 cores
2. You just proved BFS is better on the job_count == core_count case, as BFS
   says it is, if you look at the graph
3. You're comparing an old version of BFS against an unreleased dev kernel

Also, you said on http://article.gmane.org/gmane.linux.kernel/886319
"I also tried to configure the kernel in a BFS friendly way, i used 
HZ=1000 as recommended, turned off all debug options, etc. The 
kernel config i used can be found here:
http://redhat.com/~mingo/misc/config
"

Quickly looking at the conf you have
CONFIG_HZ_250=y
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set

CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y

And other DEBUG.

-- 
mjt

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Markus T?rnqvist <mjt@nysv.org> wrote:

> Please Cc me as I'm not a subscriber.
> 
> On Mon, Sep 07, 2009 at 02:16:13PM +0200, Ingo Molnar wrote:
> >
> >Con posted single-socket quad comparisons/graphs so to make it 100% 
> >apples to apples i re-tested with a single-socket (non-NUMA) quad as 
> >well, and have uploaded the new graphs/results to:
> >
> >  kernel build performance on quad:
> >     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
> [...]
> >
> >It shows similar curves and behavior to the 8-core results i posted 
> >- BFS is slower than mainline in virtually every measurement. The 
> >ratios are different for different parts of the graphs - but the 
> >trend is similar.
> 
> Dude, not cool.
> 
> 1. Quad HT is not the same as a 4-core desktop, you're doing it with 8 cores

No, it's 4 cores. HyperThreading adds two 'siblings' per core, which 
are not 'cores'.

> 2. You just proved BFS is better on the job_count == core_count case, as BFS
>    says it is, if you look at the graph

I pointed that out too. I think the graphs speak for themselves:

     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild.jpg

> 3. You're comparing an old version of BFS against an unreleased dev kernel

bfs-208 was 1 day old (and it is a 500K+ kernel patch) when i tested 
it against the 2 days old sched-devel tree. Btw., i initially 
measured 205 as well and spent one more day on acquiring and 
analyzing the 208 results.

There's bfs-209 out there today. These tests take 8+ hours to 
complete and validate. I'll re-test BFS in the future too, and as i 
said it in the first mail i'll test it on a .31 base as well once 
BFS has been ported to it:

> > It's on a .31-rc8 base while BFS is on a .30 base - will be able 
> > to test BFS on a .31 base as well once you release it. (but it 
> > doesnt matter much to the results - there werent any heavy core 
> > kernel changes impacting these workloads.)

> Also, you said on http://article.gmane.org/gmane.linux.kernel/886319
> "I also tried to configure the kernel in a BFS friendly way, i used 
> HZ=1000 as recommended, turned off all debug options, etc. The 
> kernel config i used can be found here:
> http://redhat.com/~mingo/misc/config
> "
> 
> Quickly looking at the conf you have
> CONFIG_HZ_250=y
> CONFIG_PREEMPT_NONE=y
> # CONFIG_PREEMPT_VOLUNTARY is not set
> # CONFIG_PREEMPT is not set

Indeed. HZ does not seem to matter according to what i see in my 
measurements. Can you measure such sensitivity?

> CONFIG_ARCH_WANT_FRAME_POINTERS=y
> CONFIG_FRAME_POINTER=y
> 
> And other DEBUG.

These are the defaults and they dont make a measurable difference to 
these results. What other debug options do you mean and do they make 
a difference?

	Ingo

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Markus Tornqvist]

Let's test gmane's followup feature ;)

Ingo Molnar <mingo <at> elte.hu> writes:

> > Please Cc me as I'm not a subscriber.
> > >  kernel build performance on quad:
> > >     http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
> > [...]
> > >
> > >It shows similar curves and behavior to the 8-core results i posted 
> > >- BFS is slower than mainline in virtually every measurement. The 
> > >ratios are different for different parts of the graphs - but the 
> > >trend is similar.
> > 
> > Dude, not cool.
> > 
> > 1. Quad HT is not the same as a 4-core desktop, you're doing it with 8 cores
> 
> No, it's 4 cores. HyperThreading adds two 'siblings' per core, which 
> are not 'cores'.

Like Serge Belyshev says in
http://article.gmane.org/gmane.linux.kernel/886881
and Con thanks you inthe FAQ for confiming it:
"h/w threads" - My Sempron II X4 lists four of those, and it seems
to be a common setup.
 
> > 2. You just proved BFS is better on the job_count == core_count case, as BFS
> >    says it is, if you look at the graph
> 
> I pointed that out too. I think the graphs speak for themselves:
> 
>      http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
>      http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild.jpg

Those are in alignment with the FAQ, for the hardware threads.

Mr Belyshev's benchmarks are closer to a common desktop and they rock
over CFS.

That's also something that IMO "we" forgot here: it doesn't really matter!

BFS is not up for merging, it feels way better than CFS on the desktop
and it does not scale.

This thread can be about improving CFS, I do not care personally, and
will stay out of that discussion.

> There's bfs-209 out there today. These tests take 8+ hours to 
> complete and validate. I'll re-test BFS in the future too, and as i 
> said it in the first mail i'll test it on a .31 base as well once 
> BFS has been ported to it:

Apropos your tests, under which circumstances would I have a million
piped messages on my desktop?

Would you care to comment on the relevance of your other tests from
a desktop point of view?

Fortunately you got help from the community as posted on the list.

> > Also, you said on http://article.gmane.org/gmane.linux.kernel/886319
> > "I also tried to configure the kernel in a BFS friendly way, i used 
> > HZ=1000 as recommended, turned off all debug options, etc. The 
> > kernel config i used can be found here:
> > http://redhat.com/~mingo/misc/config
> > "
> > 
> > Quickly looking at the conf you have
> > CONFIG_HZ_250=y
> > CONFIG_PREEMPT_NONE=y
> > # CONFIG_PREEMPT_VOLUNTARY is not set
> > # CONFIG_PREEMPT is not set
> 
> Indeed. HZ does not seem to matter according to what i see in my 
> measurements. Can you measure such sensitivity?

Hardly the point - You said one thing and got caught with something else,
which doesn't give a credible image.

Can I measure it? IANAKH, and I think there are people more passionate
here to run benchmark scripts and endless analyses.

All I can "measure" is that my desktop experience isn't stuttery and jittery
with basic stuff like scrolling over Firefox tabs with my mouse wheel
while watching pr0n.
 
> > CONFIG_ARCH_WANT_FRAME_POINTERS=y
> > CONFIG_FRAME_POINTER=y
> > 
> > And other DEBUG.
> 
> These are the defaults and they dont make a measurable difference to 
> these results. What other debug options do you mean and do they make 
> a difference?

Don't care as long as your kernel comparisons truly were with equivalent
settings to each other.

Köszönöm.

-- 
mjt

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Mon, 7 Sep 2009 16:41:51 +0300
> >It shows similar curves and behavior to the 8-core results i posted 
> >- BFS is slower than mainline in virtually every measurement. The 
> >ratios are different for different parts of the graphs - but the 
> >trend is similar.
> 
> Dude, not cool.
> 
> 1. Quad HT is not the same as a 4-core desktop, you're doing it with
> 8 cores 

4 cores, 8 threads. Which is basically the standard desktop cpu going
forward... (4 cores already is today, 8 threads is that any day now)



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

On Monday 07 September 2009, Arjan van de Ven wrote:
> 4 cores, 8 threads. Which is basically the standard desktop cpu going
> forward... (4 cores already is today, 8 threads is that any day now)

Despite that I'm personally more interested in what I have available here 
*now*. And that's various UP Pentium systems, one dual core Pentium D and 
Core Duo.

I've been running BFS on my laptop today while doing CPU intensive jobs 
(not disk intensive), and I must say that BFS does seem very responsive.
OTOH, I've also noticed some surprising things, such as processors staying 
on lower frequencies while doing CPU-intensive work.

I feels like I have less of the mouse cursor and typing freezes I'm used 
to with CFS, even when I'm *not* doing anything special. I've been 
blaming those on still running with ordered mode ext3, but now I'm 
starting to wonder.

I'll try to do more structured testing, comparisons and measurements 
later. At the very least it's nice to have something to compare _with_.

Cheers,
FJP

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Mon, 7 Sep 2009 17:20:33 +0200
Frans Pop <elendil@planet.nl> wrote:

> On Monday 07 September 2009, Arjan van de Ven wrote:
> > 4 cores, 8 threads. Which is basically the standard desktop cpu
> > going forward... (4 cores already is today, 8 threads is that any
> > day now)
> 
> Despite that I'm personally more interested in what I have available
> here *now*. And that's various UP Pentium systems, one dual core
> Pentium D and Core Duo.
> 
> I've been running BFS on my laptop today while doing CPU intensive
> jobs (not disk intensive), and I must say that BFS does seem very
> responsive. OTOH, I've also noticed some surprising things, such as
> processors staying on lower frequencies while doing CPU-intensive
> work.
> 
> I feels like I have less of the mouse cursor and typing freezes I'm
> used to with CFS, even when I'm *not* doing anything special. I've
> been blaming those on still running with ordered mode ext3, but now
> I'm starting to wonder.
> 
> I'll try to do more structured testing, comparisons and measurements 
> later. At the very least it's nice to have something to compare
> _with_.
> 

it's a shameless plug since I wrote it, but latencytop will be able to
tell you what your bottleneck is...
and that is very interesting to know, regardless of the "what scheduler
code" discussion;

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

On Monday 07 September 2009, Arjan van de Ven wrote:
> it's a shameless plug since I wrote it, but latencytop will be able to
> tell you what your bottleneck is...
> and that is very interesting to know, regardless of the "what scheduler
> code" discussion;

I'm very much aware of that and I've tried pinning it down a few times, 
but failed to come up with anything conclusive. I plan to make a new 
effort in this context as the freezes have increasingly been annoying me.

Unfortunately latencytop only shows a blank screen when used with BFS, but 
I guess that's not totally unexpected.

Cheers,
FJP

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Xavier Bestel]

On Mon, 2009-09-07 at 07:45 -0700, Arjan van de Ven wrote:
> On Mon, 7 Sep 2009 16:41:51 +0300
> > >It shows similar curves and behavior to the 8-core results i posted 
> > >- BFS is slower than mainline in virtually every measurement. The 
> > >ratios are different for different parts of the graphs - but the 
> > >trend is similar.
> > 
> > Dude, not cool.
> > 
> > 1. Quad HT is not the same as a 4-core desktop, you're doing it with
> > 8 cores 
> 
> 4 cores, 8 threads. Which is basically the standard desktop cpu going
> forward... (4 cores already is today, 8 threads is that any day now)

Except on your typical smartphone, which will run linux and probably
vastly outnumber the number of "traditional" linux desktops.

	Xav

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Mon, 07 Sep 2009 17:24:29 +0200
Xavier Bestel <xavier.bestel@free.fr> wrote:

> 
> On Mon, 2009-09-07 at 07:45 -0700, Arjan van de Ven wrote:
> > On Mon, 7 Sep 2009 16:41:51 +0300
> > > >It shows similar curves and behavior to the 8-core results i
> > > >posted 
> > > >- BFS is slower than mainline in virtually every measurement.
> > > >The ratios are different for different parts of the graphs - but
> > > >the trend is similar.
> > > 
> > > Dude, not cool.
> > > 
> > > 1. Quad HT is not the same as a 4-core desktop, you're doing it
> > > with 8 cores 
> > 
> > 4 cores, 8 threads. Which is basically the standard desktop cpu
> > going forward... (4 cores already is today, 8 threads is that any
> > day now)
> 
> Except on your typical smartphone, which will run linux and probably
> vastly outnumber the number of "traditional" linux desktops.

yeah the trend in cellphones is only quad core without HT, not quad
core WITH ht ;-)



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Diego Calleja]

On Lunes 07 Septiembre 2009 17:24:29 Xavier Bestel escribió:
> Except on your typical smartphone, which will run linux and probably
> vastly outnumber the number of "traditional" linux desktops.

Smartphones will probably start using ARM dualcore cpus the next year,
the embedded land is no SMP-free.

Re: [quad core results] BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/07/2009 03:16 PM, Ingo Molnar wrote:
> [...]
> Note that usually we can extrapolate ballpark-figure quad and dual
> socket results from 8 core results. Trends as drastic as the ones
> i reported do not get reversed as one shrinks the number of cores.
>
> Con posted single-socket quad comparisons/graphs so to make it 100%
> apples to apples i re-tested with a single-socket (non-NUMA) quad as
> well, and have uploaded the new graphs/results to:
>
>    kernel build performance on quad:
>       http://redhat.com/~mingo/misc/bfs-vs-tip-kbuild-quad.jpg
>
>    pipe performance on quad:
>       http://redhat.com/~mingo/misc/bfs-vs-tip-pipe-quad.jpg
>
>    messaging performance (hackbench) on quad:
>       http://redhat.com/~mingo/misc/bfs-vs-tip-messaging-quad.jpg
>
>    OLTP performance (postgresql + sysbench) on quad:
>       http://redhat.com/~mingo/misc/bfs-vs-tip-oltp-quad.jpg
>
> It shows similar curves and behavior to the 8-core results i posted
> - BFS is slower than mainline in virtually every measurement.

Except for numbers, what's your *experience* with BFS when it comes to 
composited desktops + games + multimedia apps?  (Watching high 
definition videos, playing some latest high-tech 3D game, etc.)  I 
described the exact problems experienced with mainline in a previous reply.

Are you even using that stuff actually?  Because it would be hard to 
tell if your desktop consists mainly of Emacs and an xterm; you even 
seem to be using Mutt so I suspect your desktop probably doesn't look 
very Windows Vista/OS X/Compiz-like.  Usually, with "multimedia desktop 
PC" one doesn't mean:

   http://foss.math.aegean.gr/~realnc/pics/desktop2.png

but rather:

   http://foss.math.aegean.gr/~realnc/pics/desktop1.png

BFS probably wouldn't offer the former anything, while on the latter it 
does make a difference.  If your usage of the "desktop" bears a 
resemblance to the first example, I'd say you might be not the most 
qualified person to judge on the "Linux desktop experience."  That is 
not meant be offensive or patronizing, just an observation and I even 
might be totally wrong about it.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/06/2009 11:59 PM, Ingo Molnar wrote:
>[...]
> Also, i'd like to outline that i agree with the general goals
> described by you in the BFS announcement - small desktop systems
> matter more than large systems. We find it critically important
> that the mainline Linux scheduler performs well on those systems
> too - and if you (or anyone else) can reproduce suboptimal behavior
> please let the scheduler folks know so that we can fix/improve it.

BFS improved behavior of many applications on my Intel Core 2 box in a 
way that can't be benchmarked.  Examples:

mplayer using OpenGL renderer doesn't drop frames anymore when dragging 
and dropping the video window around in an OpenGL composited desktop 
(KDE 4.3.1).  (Start moving the mplayer window around; then drop it. At 
the moment the move starts and at the moment you drop the window back to 
the desktop, there's a big frame skip as if mplayer was frozen for a 
bit; around 200 or 300ms.)

Composite desktop effects like zoom and fade out don't stall for 
sub-second periods of time while there's CPU load in the background.  In 
other words, the desktop is more fluid and less skippy even during heavy 
CPU load.  Moving windows around with CPU load in the background doesn't 
result in short skips.

LMMS (a tool utilizing real-time sound synthesis) does not produce 
"pops", "crackles" and drops in the sound during real-time playback due 
to buffer under-runs.  Those problems amplify when there's heavy CPU 
load in the background, while with BFS heavy load doesn't produce those 
artifacts (though LMMS makes itself run SCHED_ISO with BFS)  Also, 
hitting a key on the keyboard needs less time for the note to become 
audible when using BFS.  Same should hold true for other tools who 
traditionally benefit from the "-rt" kernel sources.

Games like Doom 3 and such don't "freeze" periodically for small amounts 
of time (again for sub-second amounts) when something in the background 
grabs CPU time (be it my mailer checking for new mail or a cron job, or 
whatever.)

And, the most drastic improvement here, with BFS I can do a "make -j2" 
in the kernel tree and the GUI stays fluid.  Without BFS, things start 
to lag, even with in-RAM builds (like having the whole kernel tree 
inside a tmpfs) and gcc running with nice 19 and ionice -c 3.

Unfortunately, I can't come up with any way to somehow benchmark all of 
this.  There's no benchmark for "fluidity" and "responsiveness". 
Running the Doom 3 benchmark, or any other benchmark, doesn't say 
anything about responsiveness, it only measures how many frames were 
calculated in a specific period of time.  How "stable" (with no stalls) 
those frames were making it to the screen is not measurable.

If BFS would imply small drops in pure performance counted in 
instructions per seconds, that would be a totally acceptable regression 
for desktop/multimedia/gaming PCs.  Not for server machines, of course. 
  However, on my machine, BFS is faster in classic workloads.  When I 
run "make -j2" with BFS and the standard scheduler, BFS always finishes 
a bit faster.  Not by much, but still.  One thing I'm noticing here is 
that BFS produces 100% CPU load on each core with "make -j2" while the 
normal scheduler stays at about 90-95% with -j2 or higher in at least 
one of the cores.  There seems to be under-utilization of CPU time.

Also, by searching around the net but also through discussions on 
various mailing lists, there seems to be a trend: the problems for some 
reason seem to occur more often with Intel CPUs (Core 2 chips and lower; 
I can't say anything about Core I7) while people on AMD CPUs mostly not 
being affected by most or even all of the above.  (And due to this flame 
wars often break out, with one party accusing the other of imagining 
things).  Can the integrated memory controller on AMD chips have 
something to do with this?  Do AMD chips generally offer better 
"multithrading" behavior?  Unfortunately, you didn't mention on what CPU 
you ran your tests.  If it was AMD, it might be a good idea to run tests 
on Pentium and Core 2 CPUs.

For reference, my system is:

CPU: Intel Core 2 Duo E6600 (2.4GHz)
Mainboard: Asus P5E (Intel X38 chipset)
RAM: 6GB (2+2+1+1) dual channel DDR2 800
GPU: RV770 (Radeon HD4870).

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frederic Weisbecker]

On Mon, Sep 07, 2009 at 06:38:36AM +0300, Nikos Chantziaras wrote:
> Unfortunately, I can't come up with any way to somehow benchmark all of  
> this.  There's no benchmark for "fluidity" and "responsiveness". Running 
> the Doom 3 benchmark, or any other benchmark, doesn't say anything about 
> responsiveness, it only measures how many frames were calculated in a 
> specific period of time.  How "stable" (with no stalls) those frames were 
> making it to the screen is not measurable.



That looks eventually benchmarkable. This is about latency.
For example, you could try to run high load tasks in the
background and then launch a task that wakes up in middle/large
periods to do something. You could measure the time it takes to wake
it up to perform what it wants.

We have some events tracing infrastructure in the kernel that can
snapshot the wake up and sched switch events.

Having CONFIG_EVENT_TRACING=y should be sufficient for that.

You just need to mount a debugfs point, say in /debug.

Then you can activate these sched events by doing:

echo 0 > /debug/tracing/tracing_on
echo 1 > /debug/tracing/events/sched/sched_switch/enable
echo 1 > /debug/tracing/events/sched/sched_wake_up/enable

#Launch your tasks

echo 1 > /debug/tracing/tracing_on

#Wait for some time

echo 0 > /debug/tracing/tracing_off

That will require some parsing of the result in /debug/tracing/trace
to get the delays between wake_up events and switch in events
for the task that periodically wakes up and then produce some
statistics such as the average or the maximum latency.

That's a bit of a rough approach to measure such latencies but that
should work.


> If BFS would imply small drops in pure performance counted in  
> instructions per seconds, that would be a totally acceptable regression  
> for desktop/multimedia/gaming PCs.  Not for server machines, of course.  
> However, on my machine, BFS is faster in classic workloads.  When I run 
> "make -j2" with BFS and the standard scheduler, BFS always finishes a bit 
> faster.  Not by much, but still.  One thing I'm noticing here is that BFS 
> produces 100% CPU load on each core with "make -j2" while the normal 
> scheduler stays at about 90-95% with -j2 or higher in at least one of the 
> cores.  There seems to be under-utilization of CPU time.



That also could be benchmarkable by using the above sched events and
look at the average time spent in a cpu to run the idle tasks.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/07/2009 02:01 PM, Frederic Weisbecker wrote:
> On Mon, Sep 07, 2009 at 06:38:36AM +0300, Nikos Chantziaras wrote:
>> Unfortunately, I can't come up with any way to somehow benchmark all of
>> this.  There's no benchmark for "fluidity" and "responsiveness". Running
>> the Doom 3 benchmark, or any other benchmark, doesn't say anything about
>> responsiveness, it only measures how many frames were calculated in a
>> specific period of time.  How "stable" (with no stalls) those frames were
>> making it to the screen is not measurable.
>
>
> That looks eventually benchmarkable. This is about latency.
> For example, you could try to run high load tasks in the
> background and then launch a task that wakes up in middle/large
> periods to do something. You could measure the time it takes to wake
> it up to perform what it wants.
>
> We have some events tracing infrastructure in the kernel that can
> snapshot the wake up and sched switch events.
>
> Having CONFIG_EVENT_TRACING=y should be sufficient for that.
>
> You just need to mount a debugfs point, say in /debug.
>
> Then you can activate these sched events by doing:
>
> echo 0>  /debug/tracing/tracing_on
> echo 1>  /debug/tracing/events/sched/sched_switch/enable
> echo 1>  /debug/tracing/events/sched/sched_wake_up/enable
>
> #Launch your tasks
>
> echo 1>  /debug/tracing/tracing_on
>
> #Wait for some time
>
> echo 0>  /debug/tracing/tracing_off
>
> That will require some parsing of the result in /debug/tracing/trace
> to get the delays between wake_up events and switch in events
> for the task that periodically wakes up and then produce some
> statistics such as the average or the maximum latency.
>
> That's a bit of a rough approach to measure such latencies but that
> should work.

I've tried this with 2.6.31-rc9 while running mplayer and alt+tabbing 
repeatedly to the point where mplayer starts to stall and drop frames. 
This produced a 4.1MB trace file (132k bzip2'ed):

     http://foss.math.aegean.gr/~realnc/kernel/trace1.bz2

Uncompressed for online viewing:

     http://foss.math.aegean.gr/~realnc/kernel/trace1

I must admit that I don't know what it is I'm looking at :P

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frederic Weisbecker]

On Tue, Sep 08, 2009 at 09:15:22PM +0300, Nikos Chantziaras wrote:
> On 09/07/2009 02:01 PM, Frederic Weisbecker wrote:
>> That looks eventually benchmarkable. This is about latency.
>> For example, you could try to run high load tasks in the
>> background and then launch a task that wakes up in middle/large
>> periods to do something. You could measure the time it takes to wake
>> it up to perform what it wants.
>>
>> We have some events tracing infrastructure in the kernel that can
>> snapshot the wake up and sched switch events.
>>
>> Having CONFIG_EVENT_TRACING=y should be sufficient for that.
>>
>> You just need to mount a debugfs point, say in /debug.
>>
>> Then you can activate these sched events by doing:
>>
>> echo 0>  /debug/tracing/tracing_on
>> echo 1>  /debug/tracing/events/sched/sched_switch/enable
>> echo 1>  /debug/tracing/events/sched/sched_wake_up/enable
>>
>> #Launch your tasks
>>
>> echo 1>  /debug/tracing/tracing_on
>>
>> #Wait for some time
>>
>> echo 0>  /debug/tracing/tracing_off
>>
>> That will require some parsing of the result in /debug/tracing/trace
>> to get the delays between wake_up events and switch in events
>> for the task that periodically wakes up and then produce some
>> statistics such as the average or the maximum latency.
>>
>> That's a bit of a rough approach to measure such latencies but that
>> should work.
>
> I've tried this with 2.6.31-rc9 while running mplayer and alt+tabbing  
> repeatedly to the point where mplayer starts to stall and drop frames.  
> This produced a 4.1MB trace file (132k bzip2'ed):
>
>     http://foss.math.aegean.gr/~realnc/kernel/trace1.bz2
>
> Uncompressed for online viewing:
>
>     http://foss.math.aegean.gr/~realnc/kernel/trace1
>
> I must admit that I don't know what it is I'm looking at :P


Hehe :-)

Basically you have samples of two kind of events:

- wake up (when thread A wakes up B)

The format is as follows:


	task-pid
	(the waker A)
	   |
	   |     cpu     timestamp   event-name        wakee(B)    prio    status
	   |      |          |           |               |          |        |
	X-11482 [001]  1023.219246: sched_wakeup: task kwin:11571 [120] success=1

Here X is awakening kwin.


- sched switch (when the scheduler stops A and launches B)

	                                            A, task                  B, task
	                                            that gets                that gets
	                                            sched                    sched
                                                     out                      in
	  A      cpu    timestamp     event-name       |       A prio          |        B prio
	  |       |         |             |            |         |             |          |
	X-11482 [001]  1023.219247: sched_switch: task X:11482 [120] (R) ==> kwin:11571 [120]
	                                                              |
	                                                              |
                                                                    State of A
For A state we can have either:

R: TASK_RUNNING, the task is not sleeping but it is rescheduled for later
   to let another task run

S: TASK_INTERRUPTIBLE, the task is sleeping, waiting for an event that may
   wake it up. The task can be waked by a signal

D: TASK_UNINTERRUPTIBLE, same as above but can't be waked by a signal.


Now what could be interesting interesting is to measure the time between
such pair of events:

	- t0: A wakes up B
	- t1: B is sched in

t1 - t0 would then be the scheduler latency, or at least part of it:

The scheduler latency may be an addition of several factors:

	- the time it takes for the actual wake up to perform (re-insert
	  the task into a runqueue, which can be subject to the runqueue(s)
	  design, the rebalancing if needed, etc..

	- the time between a task is waked up and the scheduler eventually
	  decide to schedule it in.

	- the time it takes to perform the task switch, which is not only
	  in the scheduler scope. But the time it takes may depend of a
	  rebalancing decision (cache cold, etc..)

Unfortunately we can only measure the second part with the above ftrace
events. But that's still an interesting scheduler abstract that is a
large part of the scheduler latency.

We could write a tiny parser that could walk through such ftrace traces
and produce some average, maximum, standard deviation numbers.

But we have userspace tools that can parse ftrace events (through perf
counter), so I'm trying to write something there, hopefully I could get
a relevant end result.

Thanks.
                                                                    

Re: BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Mon, 07 Sep 2009 06:38:36 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> On 09/06/2009 11:59 PM, Ingo Molnar wrote:
> >[...]
> > Also, i'd like to outline that i agree with the general goals
> > described by you in the BFS announcement - small desktop systems
> > matter more than large systems. We find it critically important
> > that the mainline Linux scheduler performs well on those systems
> > too - and if you (or anyone else) can reproduce suboptimal behavior
> > please let the scheduler folks know so that we can fix/improve it.
> 
> BFS improved behavior of many applications on my Intel Core 2 box in
> a way that can't be benchmarked.  Examples:

Have you tried to see if latencytop catches such latencies ?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/07/2009 05:40 PM, Arjan van de Ven wrote:
> On Mon, 07 Sep 2009 06:38:36 +0300
> Nikos Chantziaras<realnc@arcor.de>  wrote:
>
>> On 09/06/2009 11:59 PM, Ingo Molnar wrote:
>>> [...]
>>> Also, i'd like to outline that i agree with the general goals
>>> described by you in the BFS announcement - small desktop systems
>>> matter more than large systems. We find it critically important
>>> that the mainline Linux scheduler performs well on those systems
>>> too - and if you (or anyone else) can reproduce suboptimal behavior
>>> please let the scheduler folks know so that we can fix/improve it.
>>
>> BFS improved behavior of many applications on my Intel Core 2 box in
>> a way that can't be benchmarked.  Examples:
>
> Have you tried to see if latencytop catches such latencies ?

I've just tried it.

I start latencytop and then mplayer on a video that doesn't max out the 
CPU (needs about 20-30% of a single core (out of 2 available)).  Then, 
while the video is playing, I press Alt+Tab repeatedly which makes the 
desktop compositor kick-in and stay active (it lays out all windows as a 
"flip-switch", similar to the Microsoft Vista Aero alt+tab effect). 
Repeatedly pressing alt+tab results in the compositor (in this case KDE 
4.3.1) keep doing processing.  With the mainline scheduler, mplayer 
starts dropping frames and skip sound like crazy for the whole duration 
of this exercise.

latencytop has this to say:

   http://foss.math.aegean.gr/~realnc/pics/latop1.png

Though I don't really understand what this tool is trying to tell me, I 
hope someone does.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Tue, 08 Sep 2009 10:19:06 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> latencytop has this to say:
> 
>    http://foss.math.aegean.gr/~realnc/pics/latop1.png
> 
> Though I don't really understand what this tool is trying to tell me,
> I hope someone does.

unfortunately this is both an older version of latencytop, and it's
incorrectly installed ;-(
Latencytop is supposed to translate those cryptic strings to english,
but due to not being correctly installed, it does not do this ;(

the latest version of latencytop also has a GUI (thanks to Ben) 

-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

Arjan van de Ven wrote:
> the latest version of latencytop also has a GUI (thanks to Ben)

That looks nice, but...

I kind of miss the split screen feature where latencytop would show both 
the overall figures + the ones for the currently most affected task. 
Downside of that last was that I never managed to keep the display on a 
specific task.

The graphical display also makes it impossible to simply copy and paste 
the results.

Having the freeze button is nice though.

Would it be possible to have a command line switch that allows to start 
the old textual mode?

Looks like the man page needs updating too :-)

Cheers,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[Michal Schmidt]

Dne Tue, 8 Sep 2009 22:22:43 +0200
Frans Pop <elendil@planet.nl> napsal(a):
> Would it be possible to have a command line switch that allows to
> start the old textual mode?

I use:
DISPLAY= latencytop

:-)
Michal

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

On Tuesday 08 September 2009, Frans Pop wrote:
> Arjan van de Ven wrote:
> > the latest version of latencytop also has a GUI (thanks to Ben)
>
> That looks nice, but...
>
> I kind of miss the split screen feature where latencytop would show
> both the overall figures + the ones for the currently most affected
> task. Downside of that last was that I never managed to keep the
> display on a specific task.
[...]
> Would it be possible to have a command line switch that allows to start
> the old textual mode?

I got a private reply suggesting that --nogui might work, and it does.
Thanks a lot Nikos!

> Looks like the man page needs updating too :-)

So this definitely needs attention :-P
Support of the standard -h and --help options would be great too.

Cheers,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[GeunSik Lim]

On Wed, Sep 9, 2009 at 6:11 AM, Frans Pop<elendil@planet.nl> wrote:
>> Would it be possible to have a command line switch that allows to start
>> the old textual mode?
> I got a private reply suggesting that --nogui might work, and it does.
Um. You means that you tested with runlevel 3(multiuser mode). Is it right?
Frans. Can you share me your linux distribution for this test?
I want to check with same conditions(e.g:linux distribution like
fedora 11,ubuntu9.04 , runlevel, and so on.).
> Thanks a lot Nikos!
>> Looks like the man page needs updating too :-)
> So this definitely needs attention :-P
> Support of the standard -h and --help options would be great too.
> Cheers,
> FJP
> --

Thanks,
GeunSik Lim.



-- 
Regards,
GeunSik Lim ( Samsung Electronics )
Blog : http://blog.naver.com/invain/
e-Mail: geunsik.lim@samsung.com
           leemgs@gmail.com , leemgs1@gmail.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

On Tuesday 08 September 2009, you wrote:
> On Wed, Sep 9, 2009 at 6:11 AM, Frans Pop<elendil@planet.nl> wrote:
> >> Would it be possible to have a command line switch that allows to
> >> start the old textual mode?
> >
> > I got a private reply suggesting that --nogui might work, and it
> > does.
>
> Um. You means that you tested with runlevel 3(multiuser mode). Is it
> right? Frans. Can you share me your linux distribution for this test? I
> want to check with same conditions(e.g:linux distribution like fedora
> 11,ubuntu9.04 , runlevel, and so on.).

I ran it from KDE's konsole by just entering 'sudo latencytop --nogui' at 
the command prompt.

Distro is Debian stable ("Lenny"), which does not have differences between 
runlevels: by default they all start a desktop environment (if a display 
manager like xdm/kdm/gdm is installed). But if you really want to know, 
the runlevel was 2 ;-)

Cheers,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

On Tue, 2009-09-08 at 22:22 +0200, Frans Pop wrote:
> Arjan van de Ven wrote:
> > the latest version of latencytop also has a GUI (thanks to Ben)
> 
> That looks nice, but...
> 
> I kind of miss the split screen feature where latencytop would show both 
> the overall figures + the ones for the currently most affected task. 
> Downside of that last was that I never managed to keep the display on a 
> specific task.

Any idea of how to present it ? I'm happy to spend 5mn improving the
GUI :-) 

> The graphical display also makes it impossible to simply copy and paste 
> the results.

Ah that's right. I'm not 100% sure how to do that (first experiments
with gtk). I suppose I could try to do some kind of "snapshot" feature
which saves the results in textual form.

> Having the freeze button is nice though.
> 
> Would it be possible to have a command line switch that allows to start 
> the old textual mode?

It's there iirc. --nogui :-)

Cheers,
Ben.

> Looks like the man page needs updating too :-)
> 
> Cheers,
> FJP
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: BFS vs. mainline scheduler benchmarks and measurements

[David Newall]

Benjamin Herrenschmidt wrote:
> On Tue, 2009-09-08 at 22:22 +0200, Frans Pop wrote:
>   
>> Arjan van de Ven wrote:
>>     
>>> the latest version of latencytop also has a GUI (thanks to Ben)
>>>       
>> That looks nice, but...
>>
>> I kind of miss the split screen feature where latencytop would show both 
>> the overall figures + the ones for the currently most affected task. 
>> Downside of that last was that I never managed to keep the display on a 
>> specific task.
>>     
>
> Any idea of how to present it ? I'm happy to spend 5mn improving the
> GUI :-) 

Use a second window.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

On Wed, 2009-09-09 at 20:44 +0930, David Newall wrote:
> Benjamin Herrenschmidt wrote:
> > On Tue, 2009-09-08 at 22:22 +0200, Frans Pop wrote:
> >   
> >> Arjan van de Ven wrote:
> >>     
> >>> the latest version of latencytop also has a GUI (thanks to Ben)
> >>>       
> >> That looks nice, but...
> >>
> >> I kind of miss the split screen feature where latencytop would show both 
> >> the overall figures + the ones for the currently most affected task. 
> >> Downside of that last was that I never managed to keep the display on a 
> >> specific task.
> >>     
> >
> > Any idea of how to present it ? I'm happy to spend 5mn improving the
> > GUI :-) 
> 
> Use a second window.

I'm not too fan of cluttering the screen with windows... I suppose I
could have a separate pane for the "global" view but I haven't found a
way to lay it out in a way that doesn't suck :-) I could have done a 3rd
colums on the right with the overall view but it felt like using too
much screen real estate.

I'll experiment a bit, maybe 2 windows is indeed the solution. But you
get into the problem of what to do if only one of them is closed ? Do I
add a menu bar on each of them to re-open the "other" one if closed ?
etc...

Don't get me wrong, I have a shitload of experience doing GUIs (back in
the old days when I was hacking on MacOS), though I'm relatively new to
GTK. But GUI design is rather hard in general :-)

Ben.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

On Wednesday 09 September 2009, Benjamin Herrenschmidt wrote:
> On Tue, 2009-09-08 at 22:22 +0200, Frans Pop wrote:
> > Arjan van de Ven wrote:
> > > the latest version of latencytop also has a GUI (thanks to Ben)
> >
> > That looks nice, but...
> >
> > I kind of miss the split screen feature where latencytop would show
> > both the overall figures + the ones for the currently most affected
> > task. Downside of that last was that I never managed to keep the
> > display on a specific task.
>
> Any idea of how to present it ? I'm happy to spend 5mn improving the
> GUI :-)

I'd say add an extra horizontal split in the second column, so you'd get 
three areas in the right column:
- top for the global target (permanently)
- middle for current, either:
  - "current most lagging" if "Global" is selected in left column
  - selected process if a specific target is selected in left column
- bottom for backtrace

Maybe with that setup "Global" in the left column should be renamed to 
something like "Dynamic".

The backtrace area would show selection from either top or middle areas 
(so selecting a cause in top or middle area should unselect causes in the 
other).

Cheers,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

> I'd say add an extra horizontal split in the second column, so you'd get 
> three areas in the right column:
> - top for the global target (permanently)
> - middle for current, either:
>   - "current most lagging" if "Global" is selected in left column
>   - selected process if a specific target is selected in left column
> - bottom for backtrace
> 
> Maybe with that setup "Global" in the left column should be renamed to 
> something like "Dynamic".
> 
> The backtrace area would show selection from either top or middle areas 
> (so selecting a cause in top or middle area should unselect causes in the 
> other).

I'll have a look after the merge window madness. Multiple windows is
also still an option I suppose even if i don't like it that much: we
could support double-click on an app or "global" in the left list,
making that pop a new window with the same content as the right pane for
that app (or global) that updates at the same time as the rest.

Somebody ping me if I seem to have forgotten about it in 2 weeks :-)

Ben.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

Benjamin Herrenschmidt wrote:
> I'll have a look after the merge window madness. Multiple windows is
> also still an option I suppose even if i don't like it that much: we
> could support double-click on an app or "global" in the left list,
> making that pop a new window with the same content as the right pane for
> that app (or global) that updates at the same time as the rest.
 
I have another request. If I select a specific application to watch (say a 
mail client) but it is idle for a while and thus has no latencies, it will 
get dropped from the list and thus my selection of it will be lost.

It would be nice if in that case a selected application would stay visible 
and selected, or maybe get reselected automatically when it appears again.

Thanks,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

On Wed, 2009-09-16 at 20:27 +0200, Frans Pop wrote:
> Benjamin Herrenschmidt wrote:
> > I'll have a look after the merge window madness. Multiple windows is
> > also still an option I suppose even if i don't like it that much: we
> > could support double-click on an app or "global" in the left list,
> > making that pop a new window with the same content as the right pane for
> > that app (or global) that updates at the same time as the rest.
>  
> I have another request. If I select a specific application to watch (say a 
> mail client) but it is idle for a while and thus has no latencies, it will 
> get dropped from the list and thus my selection of it will be lost.
> 
> It would be nice if in that case a selected application would stay visible 
> and selected, or maybe get reselected automatically when it appears again.

Hrm... I though I forced the selected app to remain ... or maybe I
wanted to do that and failed :-) Ok. On the list. Please ping me next
week if nothing happens.

Ben.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Frans Pop]

Benjamin Herrenschmidt wrote:
> On Wed, 2009-09-16 at 20:27 +0200, Frans Pop wrote:
>> Benjamin Herrenschmidt wrote:
>> > I'll have a look after the merge window madness. Multiple windows is
>> > also still an option I suppose even if i don't like it that much: we
>> > could support double-click on an app or "global" in the left list,
>> > making that pop a new window with the same content as the right pane
>> > for that app (or global) that updates at the same time as the rest.
>>  
>> I have another request. If I select a specific application to watch (say
>> a mail client) but it is idle for a while and thus has no latencies, it
>> will get dropped from the list and thus my selection of it will be lost.
>> 
>> It would be nice if in that case a selected application would stay
>> visible and selected, or maybe get reselected automatically when it
>> appears again.
> 
> Hrm... I though I forced the selected app to remain ... or maybe I
> wanted to do that and failed :-) Ok. On the list. Please ping me next
> week if nothing happens.

As requested: ping?

And while I'm writing anyway, one more suggestion.
I find the fact that the buttons jump twice every 30 seconds (because of a 
change in the timer between <10 and >=10 seconds) slightly annoying.
Any chance of making the position of the buttons fixed? One option could be 
moving the timer to the left side of the bottom bar.

Cheers,
FJP

Re: BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Tue, 08 Sep 2009 10:19:06 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> latencytop has this to say:
> 
>    http://foss.math.aegean.gr/~realnc/pics/latop1.png
> 
> Though I don't really understand what this tool is trying to tell me,
> I hope someone does.

despite the untranslated content, it is clear that you have scheduler
delays (either due to scheduler bugs or cpu contention) of upto 68
msecs... Second in line is your binary AMD graphics driver that is
chewing up 14% of your total latency...


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 11:38 AM, Arjan van de Ven wrote:
> On Tue, 08 Sep 2009 10:19:06 +0300
> Nikos Chantziaras<realnc@arcor.de>  wrote:
>
>> latencytop has this to say:
>>
>>     http://foss.math.aegean.gr/~realnc/pics/latop1.png
>>
>> Though I don't really understand what this tool is trying to tell me,
>> I hope someone does.
>
> despite the untranslated content, it is clear that you have scheduler
> delays (either due to scheduler bugs or cpu contention) of upto 68
> msecs... Second in line is your binary AMD graphics driver that is
> chewing up 14% of your total latency...

I've now used a correctly installed and up-to-date version of latencytop 
and repeated the test.  Also, I got rid of AMD's binary blob and used 
kernel DRM drivers for my graphics card to throw fglrx out of the 
equation (which btw didn't help; the exact same problems occur).

Here the result:

     http://foss.math.aegean.gr/~realnc/pics/latop2.png

Again: this is on an Intel Core 2 Duo CPU.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Juergen Beisert]

On Dienstag, 8. September 2009, Nikos Chantziaras wrote:
> On 09/08/2009 11:38 AM, Arjan van de Ven wrote:
> > On Tue, 08 Sep 2009 10:19:06 +0300
> >
> > Nikos Chantziaras<realnc@arcor.de>  wrote:
> >> latencytop has this to say:
> >>
> >>     http://foss.math.aegean.gr/~realnc/pics/latop1.png
> >>
> >> Though I don't really understand what this tool is trying to tell me,
> >> I hope someone does.
> >
> > despite the untranslated content, it is clear that you have scheduler
> > delays (either due to scheduler bugs or cpu contention) of upto 68
> > msecs... Second in line is your binary AMD graphics driver that is
> > chewing up 14% of your total latency...
>
> I've now used a correctly installed and up-to-date version of latencytop
> and repeated the test.  Also, I got rid of AMD's binary blob and used
> kernel DRM drivers for my graphics card to throw fglrx out of the
> equation (which btw didn't help; the exact same problems occur).
>
> Here the result:
>
>      http://foss.math.aegean.gr/~realnc/pics/latop2.png
>
> Again: this is on an Intel Core 2 Duo CPU.

Just an idea: Maybe some system management code hits you?

jbe

-- 
Pengutronix e.K.                              | Juergen Beisert             |
Linux Solutions for Science and Industry      | Phone: +49-8766-939 228     |
Vertretung Sued/Muenchen, Germany             | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686              | http://www.pengutronix.de/  |

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 02:32 PM, Juergen Beisert wrote:
> On Dienstag, 8. September 2009, Nikos Chantziaras wrote:
>> On 09/08/2009 11:38 AM, Arjan van de Ven wrote:
>>> On Tue, 08 Sep 2009 10:19:06 +0300
>>>
>>> Nikos Chantziaras<realnc@arcor.de>   wrote:
>>>> latencytop has this to say:
>>>>
>>>>      http://foss.math.aegean.gr/~realnc/pics/latop1.png
>>>>
>>>> Though I don't really understand what this tool is trying to tell me,
>>>> I hope someone does.
>>>
>>> despite the untranslated content, it is clear that you have scheduler
>>> delays (either due to scheduler bugs or cpu contention) of upto 68
>>> msecs... Second in line is your binary AMD graphics driver that is
>>> chewing up 14% of your total latency...
>>
>> I've now used a correctly installed and up-to-date version of latencytop
>> and repeated the test.  Also, I got rid of AMD's binary blob and used
>> kernel DRM drivers for my graphics card to throw fglrx out of the
>> equation (which btw didn't help; the exact same problems occur).
>>
>> Here the result:
>>
>>       http://foss.math.aegean.gr/~realnc/pics/latop2.png
>>
>> Again: this is on an Intel Core 2 Duo CPU.
>
> Just an idea: Maybe some system management code hits you?

I'm not sure what is meant with "system management code."

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jiri Kosina]

On Wed, 9 Sep 2009, Nikos Chantziaras wrote:

> > > Here the result:
> > > 
> > >       http://foss.math.aegean.gr/~realnc/pics/latop2.png
> > > 
> > > Again: this is on an Intel Core 2 Duo CPU.
> > 
> > Just an idea: Maybe some system management code hits you?
> 
> I'm not sure what is meant with "system management code."

System management interrupt happens when firmware/BIOS/HW-debugger is 
executed in privilege mode so high, that even OS can't do anything about 
that.

It is used in many situations, such as

- memory errors
- ACPI (mostly fan control)
- TPM

OS has small to none possibility to influence SMI/SMM. But if this would 
be the cause, you should probably obtain completely different results on 
different hardware configuration (as it is likely to have completely 
different SMM behavior).

-- 
Jiri Kosina
SUSE Labs, Novell Inc.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 02:20 AM, Jiri Kosina wrote:
> On Wed, 9 Sep 2009, Nikos Chantziaras wrote:
>
>>>> Here the result:
>>>>
>>>>        http://foss.math.aegean.gr/~realnc/pics/latop2.png
>>>>
>>>> Again: this is on an Intel Core 2 Duo CPU.
>>>
>>> Just an idea: Maybe some system management code hits you?
>>
>> I'm not sure what is meant with "system management code."
>
> System management interrupt happens when firmware/BIOS/HW-debugger is
> executed in privilege mode so high, that even OS can't do anything about
> that.
>
> It is used in many situations, such as
>
> - memory errors
> - ACPI (mostly fan control)
> - TPM
>
> OS has small to none possibility to influence SMI/SMM. But if this would
> be the cause, you should probably obtain completely different results on
> different hardware configuration (as it is likely to have completely
> different SMM behavior).

Wouldn't that mean that a BFS-patched kernel would suffer from this too?

In any case, of the above, only fan control is active, and I've run with 
it disabled on occasion (hot summer days, I wanted to just keep it max 
with no fan control) with no change.  As far as I can tell, the Asus P5E 
doesn't have a TPM (the "Deluxe" and "VM" models seem to have one.)  As 
for memory errors, I use unbuffered non-ECC RAM which passes a 
memtest86+ cycle cleanly (well, at least the last time I ran it through 
one, a few months ago.)

Re: BFS vs. mainline scheduler benchmarks and measurements

[Theodore Tso]

On Tue, Sep 08, 2009 at 01:13:34PM +0300, Nikos Chantziaras wrote:
>> despite the untranslated content, it is clear that you have scheduler
>> delays (either due to scheduler bugs or cpu contention) of upto 68
>> msecs... Second in line is your binary AMD graphics driver that is
>> chewing up 14% of your total latency...
>
> I've now used a correctly installed and up-to-date version of latencytop  
> and repeated the test.  Also, I got rid of AMD's binary blob and used  
> kernel DRM drivers for my graphics card to throw fglrx out of the  
> equation (which btw didn't help; the exact same problems occur).
>
> Here the result:
>
>     http://foss.math.aegean.gr/~realnc/pics/latop2.png

This was with an unmodified 2.6.31-rcX kernel?  Does Latencytop do
anything useful on a BFS-patched kernel?

							- Ted

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 03:03 PM, Theodore Tso wrote:
> On Tue, Sep 08, 2009 at 01:13:34PM +0300, Nikos Chantziaras wrote:
>>> despite the untranslated content, it is clear that you have scheduler
>>> delays (either due to scheduler bugs or cpu contention) of upto 68
>>> msecs... Second in line is your binary AMD graphics driver that is
>>> chewing up 14% of your total latency...
>>
>> I've now used a correctly installed and up-to-date version of latencytop
>> and repeated the test.  Also, I got rid of AMD's binary blob and used
>> kernel DRM drivers for my graphics card to throw fglrx out of the
>> equation (which btw didn't help; the exact same problems occur).
>>
>> Here the result:
>>
>>      http://foss.math.aegean.gr/~realnc/pics/latop2.png
>
> This was with an unmodified 2.6.31-rcX kernel?

Yes (-rc9).  I also tested with 2.6.30.5 and getting the same results.


> Does Latencytop do anything useful on a BFS-patched kernel?

Nope.  BFS does not support any form of tracing yet.  latencytop runs 
but only shows a blank list.  All I can say is that a BFS patched kernel 
with the same .config fixes all visible latency issues.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Tue, 08 Sep 2009 13:13:34 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> On 09/08/2009 11:38 AM, Arjan van de Ven wrote:
> > On Tue, 08 Sep 2009 10:19:06 +0300
> > Nikos Chantziaras<realnc@arcor.de>  wrote:
> >
> >> latencytop has this to say:
> >>
> >>     http://foss.math.aegean.gr/~realnc/pics/latop1.png
> >>
> >> Though I don't really understand what this tool is trying to tell
> >> me, I hope someone does.
> >
> > despite the untranslated content, it is clear that you have
> > scheduler delays (either due to scheduler bugs or cpu contention)
> > of upto 68 msecs... Second in line is your binary AMD graphics
> > driver that is chewing up 14% of your total latency...
> 
> I've now used a correctly installed and up-to-date version of
> latencytop and repeated the test.  Also, I got rid of AMD's binary
> blob and used kernel DRM drivers for my graphics card to throw fglrx
> out of the equation (which btw didn't help; the exact same problems
> occur).
> 
> Here the result:
> 
>      http://foss.math.aegean.gr/~realnc/pics/latop2.png
> 
> Again: this is on an Intel Core 2 Duo CPU.


so we finally have objective numbers!

now the interesting part is also WHERE the latency hits. Because
fundamentally, if you oversubscribe the CPU, you WILL get scheduling
latency.. simply you have more to run than there is CPU.

Now the scheduler impacts this latency in two ways
* Deciding how long apps run before someone else gets to take over
  ("time slicing")
* Deciding who gets to run first/more; eg priority between apps

the first one more or less controls the maximum, while the second one
controls which apps get to enjoy this maximum.

latencytop shows you both, but it is interesting to see how much the
apps get that you care about latency for....



-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 05:20 PM, Arjan van de Ven wrote:
> On Tue, 08 Sep 2009 13:13:34 +0300
> Nikos Chantziaras<realnc@arcor.de>  wrote:
>
>> On 09/08/2009 11:38 AM, Arjan van de Ven wrote:
>>> On Tue, 08 Sep 2009 10:19:06 +0300
>>> Nikos Chantziaras<realnc@arcor.de>   wrote:
>>>
>>>> latencytop has this to say:
>>>>
>>>>      http://foss.math.aegean.gr/~realnc/pics/latop1.png
>>>>
>>>> Though I don't really understand what this tool is trying to tell
>>>> me, I hope someone does.
>>>
>>> despite the untranslated content, it is clear that you have
>>> scheduler delays (either due to scheduler bugs or cpu contention)
>>> of upto 68 msecs... Second in line is your binary AMD graphics
>>> driver that is chewing up 14% of your total latency...
>>
>> I've now used a correctly installed and up-to-date version of
>> latencytop and repeated the test.  Also, I got rid of AMD's binary
>> blob and used kernel DRM drivers for my graphics card to throw fglrx
>> out of the equation (which btw didn't help; the exact same problems
>> occur).
>>
>> Here the result:
>>
>>       http://foss.math.aegean.gr/~realnc/pics/latop2.png
>>
>> Again: this is on an Intel Core 2 Duo CPU.
>
>
> so we finally have objective numbers!
>
> now the interesting part is also WHERE the latency hits. Because
> fundamentally, if you oversubscribe the CPU, you WILL get scheduling
> latency.. simply you have more to run than there is CPU.

Sounds plausible.  However, with mainline this latency is very, very 
noticeable.  With BFS I need to look really hard to detect it or do 
outright silly things, like a "make -j50".  (At first I wrote "-j20" 
here but then went ahead an tested it just for kicks, and BFS would 
still let me use the GUI smoothly, LOL.  So then I corrected it to 
"-j50"...)

Re: BFS vs. mainline scheduler benchmarks and measurements

[Thomas Fjellstrom]

On Sun September 6 2009, Nikos Chantziaras wrote:
> On 09/06/2009 11:59 PM, Ingo Molnar wrote:
> >[...]
> > Also, i'd like to outline that i agree with the general goals
> > described by you in the BFS announcement - small desktop systems
> > matter more than large systems. We find it critically important
> > that the mainline Linux scheduler performs well on those systems
> > too - and if you (or anyone else) can reproduce suboptimal behavior
> > please let the scheduler folks know so that we can fix/improve it.
> 
> BFS improved behavior of many applications on my Intel Core 2 box in a
> way that can't be benchmarked.  Examples:
> 
> mplayer using OpenGL renderer doesn't drop frames anymore when dragging
> and dropping the video window around in an OpenGL composited desktop
> (KDE 4.3.1).  (Start moving the mplayer window around; then drop it. At
> the moment the move starts and at the moment you drop the window back to
> the desktop, there's a big frame skip as if mplayer was frozen for a
> bit; around 200 or 300ms.)
> 
> Composite desktop effects like zoom and fade out don't stall for
> sub-second periods of time while there's CPU load in the background.  In
> other words, the desktop is more fluid and less skippy even during heavy
> CPU load.  Moving windows around with CPU load in the background doesn't
> result in short skips.
> 
> LMMS (a tool utilizing real-time sound synthesis) does not produce
> "pops", "crackles" and drops in the sound during real-time playback due
> to buffer under-runs.  Those problems amplify when there's heavy CPU
> load in the background, while with BFS heavy load doesn't produce those
> artifacts (though LMMS makes itself run SCHED_ISO with BFS)  Also,
> hitting a key on the keyboard needs less time for the note to become
> audible when using BFS.  Same should hold true for other tools who
> traditionally benefit from the "-rt" kernel sources.
> 
> Games like Doom 3 and such don't "freeze" periodically for small amounts
> of time (again for sub-second amounts) when something in the background
> grabs CPU time (be it my mailer checking for new mail or a cron job, or
> whatever.)
> 
> And, the most drastic improvement here, with BFS I can do a "make -j2"
> in the kernel tree and the GUI stays fluid.  Without BFS, things start
> to lag, even with in-RAM builds (like having the whole kernel tree
> inside a tmpfs) and gcc running with nice 19 and ionice -c 3.
> 
> Unfortunately, I can't come up with any way to somehow benchmark all of
> this.  There's no benchmark for "fluidity" and "responsiveness".
> Running the Doom 3 benchmark, or any other benchmark, doesn't say
> anything about responsiveness, it only measures how many frames were
> calculated in a specific period of time.  How "stable" (with no stalls)
> those frames were making it to the screen is not measurable.
> 
> If BFS would imply small drops in pure performance counted in
> instructions per seconds, that would be a totally acceptable regression
> for desktop/multimedia/gaming PCs.  Not for server machines, of course.
>   However, on my machine, BFS is faster in classic workloads.  When I
> run "make -j2" with BFS and the standard scheduler, BFS always finishes
> a bit faster.  Not by much, but still.  One thing I'm noticing here is
> that BFS produces 100% CPU load on each core with "make -j2" while the
> normal scheduler stays at about 90-95% with -j2 or higher in at least
> one of the cores.  There seems to be under-utilization of CPU time.
> 
> Also, by searching around the net but also through discussions on
> various mailing lists, there seems to be a trend: the problems for some
> reason seem to occur more often with Intel CPUs (Core 2 chips and lower;
> I can't say anything about Core I7) while people on AMD CPUs mostly not
> being affected by most or even all of the above.  (And due to this flame
> wars often break out, with one party accusing the other of imagining
> things).  Can the integrated memory controller on AMD chips have
> something to do with this?  Do AMD chips generally offer better
> "multithrading" behavior?  Unfortunately, you didn't mention on what CPU
> you ran your tests.  If it was AMD, it might be a good idea to run tests
> on Pentium and Core 2 CPUs.
> 
> For reference, my system is:
> 
> CPU: Intel Core 2 Duo E6600 (2.4GHz)
> Mainboard: Asus P5E (Intel X38 chipset)
> RAM: 6GB (2+2+1+1) dual channel DDR2 800
> GPU: RV770 (Radeon HD4870).
> 

My Phenom 9550 (2.2Ghz) whips the pants off my Intel Q6600 (2.6Ghz). I and a 
friend of mine both get large amounts of stalling when doing a lot of IO. I 
haven't seen such horrible desktop interactivity since before the new 
schedulers and the -ck patchset came out for 2.4.x. Its a heck of a lot better 
on my AMD Phenom's, but some lag is noticeable these days, even when it wasn't 
a few kernel releases ago.

Intel Specs:
CPU: Intel Core 2 Quad Q6600 (2.6Ghz)
Mainboard: ASUS P5K-SE (Intel p35 iirc)
RAM: 4G 800Mhz DDR2 dual channel (4x1G)
GPU: NVidia 8800GTS 320M

AMD Specs:
CPU: AMD Phenom I 9550 (2.2Ghz)
Mainboard: Gigabyte MA78GM-S2H
RAM: 4G 800Mhz DDR2 dual channel (2x2G)
GPU: Onboard Radeon 3200HD

AMD Specs x2:
CPU: AMD Phenom II 810 (2.6Ghz)
Mainboard: Gigabyte MA790FXT-UD5P
RAM: 4G 1066Mhz DDR3 dual channel (2x2G)
GPU: NVidia 8800GTS 320M (or currently a 8400GS)

Of course I get better performance out of the Phenom II vs either other box, 
but it surprises me that I'd get more out of the budget AMD box over the not 
so budget Intel box.

-- 
Thomas Fjellstrom
tfjellstrom@shaw.ca

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 02:54 AM, Thomas Fjellstrom wrote:
> On Sun September 6 2009, Nikos Chantziaras wrote:
>>  [...]
>> For reference, my system is:
>>
>> CPU: Intel Core 2 Duo E6600 (2.4GHz)
>> Mainboard: Asus P5E (Intel X38 chipset)
>> RAM: 6GB (2+2+1+1) dual channel DDR2 800
>> GPU: RV770 (Radeon HD4870).
>>
>
> My Phenom 9550 (2.2Ghz) whips the pants off my Intel Q6600 (2.6Ghz). I and a
> friend of mine both get large amounts of stalling when doing a lot of IO. I
> haven't seen such horrible desktop interactivity since before the new
> schedulers and the -ck patchset came out for 2.4.x. Its a heck of a lot better
> on my AMD Phenom's, but some lag is noticeable these days, even when it wasn't
> a few kernel releases ago.

It seems someone tried BFS on quite slower hardware: Android.  According 
to the feedback, the device is much more responsive with BFS: 
http://twitter.com/cyanogen

Re: BFS vs. mainline scheduler benchmarks and measurements

[Con Kolivas]

2009/9/7 Ingo Molnar <mingo@elte.hu>:
> hi Con,

Sigh..

Well hello there.

>
> I've read your BFS announcement/FAQ with great interest:
>
>    http://ck.kolivas.org/patches/bfs/bfs-faq.txt

> I understand that BFS is still early code and that you are not
> targeting BFS for mainline inclusion - but BFS is an interesting
> and bold new approach, cutting a _lot_ of code out of
> kernel/sched*.c, so it raised my curiosity and interest :-)

Hard to keep a project under wraps and get an audience at the same
time, it is. I do realise it was inevitable LKML would invade my
personal space no matter how much I didn't want it to, but it would be
rude of me to not respond.

> In the announcement and on your webpage you have compared BFS to
> the mainline scheduler in various workloads - showing various
> improvements over it. I have tried and tested BFS and ran a set of
> benchmarks - this mail contains the results and my (quick)
> findings.

/me sees Ingo run off to find the right combination of hardware and
benchmark to prove his point.

[snip lots of bullshit meaningless benchmarks showing how great cfs is
and/or how bad bfs is, along with telling people they should use these
artificial benchmarks to determine how good it is, demonstrating yet
again why benchmarks fail the desktop]

I'm not interested in a long protracted discussion about this since
I'm too busy to live linux the way full time developers do, so I'll
keep it short, and perhaps you'll understand my intent better if the
FAQ wasn't clear enough.


Do you know what a normal desktop PC looks like? No, a more realistic
question based on what you chose to benchmark to prove your point
would be: Do you know what normal people actually do on them?


Feel free to treat the question as rhetorical.

Regards,
-ck

/me checks on his distributed computing client's progress, fires up
his next H264 encode, changes music tracks and prepares to have his
arse whooped on quakelive.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jerome Glisse]

On Mon, 2009-09-07 at 13:50 +1000, Con Kolivas wrote:

> /me checks on his distributed computing client's progress, fires up
> his next H264 encode, changes music tracks and prepares to have his
> arse whooped on quakelive.
> --

For such computer usage i would strongly suggest that you look into
GPU driver development there is a lot of performances to win in this
area and my feeling is that you can improve what you are doing
(games -> opengl (so GPU), H264 (encoding is harder to accelerate
with a GPU but for decoding and displaying it you definitely want
to involve the GPU), and tons of others things you are doing on your
linux desktop would go faster if GPU was put to more use. A wild guess
is that you can get a 2 or even 3 figures percentage improvement
with better GPU driver. My point is that i don't think a linux
scheduler improvement (compared to what we have now) will give a
significant boost for the linux desktop, on the contrary any even
slight improvement to the GPU driver stack can give you a boost.
Another way of saying that, there is no point into prioritizing  X or
desktop app if CPU has to do all the drawing by itself (CPU is
several magnitude slower than GPU at doing such kind of task).

Regards,
Jerome Glisse

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Sun, Sep 06 2009, Ingo Molnar wrote:
> So ... to get to the numbers - i've tested both BFS and the tip of 
> the latest upstream scheduler tree on a testbox of mine. I 
> intentionally didnt test BFS on any really large box - because you 
> described its upper limit like this in the announcement:

I ran a simple test as well, since I was curious to see how it performed
wrt interactiveness. One of my pet peeves with the current scheduler is
that I have to nice compile jobs, or my X experience is just awful while
the compile is running.

Now, this test case is something that attempts to see what
interactiveness would be like. It'll run a given command line while at
the same time logging delays. The delays are measured as follows:

- The app creates a pipe, and forks a child that blocks on reading from
  that pipe.
- The app sleeps for a random period of time, anywhere between 100ms
  and 2s. When it wakes up, it gets the current time and writes that to
  the pipe.
- The child then gets woken, checks the time on its own, and logs the
  difference between the two.

The idea here being that the delay between writing to the pipe and the
child reading the data and comparing should (in some way) be indicative
of how responsive the system would seem to a user.

The test app was quickly hacked up, so don't put too much into it. The
test run is a simple kernel compile, using -jX where X is the number of
threads in the system. The files are cache hot, so little IO is done.
The -x2 run is using the double number of processes as we have threads,
eg -j128 on a 64 thread box.

And I have to apologize for using a large system to test this on, I
realize it's out of the scope of BFS, but it's just easier to fire one
of these beasts up than it is to sacrifice my notebook or desktop
machine... So it's a 64 thread box. CFS -jX runtime is the baseline at
100, lower number means faster and vice versa. The latency numbers are
in msecs.


Scheduler       Runtime         Max lat     Avg lat     Std dev
----------------------------------------------------------------
CFS             100             951         462         267
CFS-x2          100             983         484         308
BFS
BFS-x2

And unfortunately this is where it ends for now, since BFS doesn't boot
on the two boxes I tried. It hard hangs right after disk detection. But
the latency numbers look pretty appalling for CFQ, so it's a bit of a
shame that I did not get to compare. I'll try again later with a newer
revision, when available.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/07/2009 12:49 PM, Jens Axboe wrote:
> [...]
> And I have to apologize for using a large system to test this on, I
> realize it's out of the scope of BFS, but it's just easier to fire one
> of these beasts up than it is to sacrifice my notebook or desktop
> machine...

How does a kernel rebuild constitute "sacrifice"?


> So it's a 64 thread box. CFS -jX runtime is the baseline at
> 100, lower number means faster and vice versa. The latency numbers are
> in msecs.
>
>
> Scheduler       Runtime         Max lat     Avg lat     Std dev
> ----------------------------------------------------------------
> CFS             100             951         462         267
> CFS-x2          100             983         484         308
> BFS
> BFS-x2
>
> And unfortunately this is where it ends for now, since BFS doesn't boot
> on the two boxes I tried.

Then who post this in the first place?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Nikos Chantziaras wrote:
> On 09/07/2009 12:49 PM, Jens Axboe wrote:
>> [...]
>> And I have to apologize for using a large system to test this on, I
>> realize it's out of the scope of BFS, but it's just easier to fire one
>> of these beasts up than it is to sacrifice my notebook or desktop
>> machine...
>
> How does a kernel rebuild constitute "sacrifice"?

It's more of a bother since I have to physically be at the notebook,
where as the server type boxes usually have remote management. The
workstation I use currently, so it'd be very disruptive to do it there.
And as things are apparently very alpha on the bfs side currently, it's
easier to 'sacrifice' an idle test box. That's the keyword, 'test'
boxes. You know, machines used for testing. Not production machines.

Plus the notebook is using btrfs which isn't format compatible with
2.6.30 on disk format.

Is there a point to this question?

>> So it's a 64 thread box. CFS -jX runtime is the baseline at
>> 100, lower number means faster and vice versa. The latency numbers are
>> in msecs.
>>
>>
>> Scheduler       Runtime         Max lat     Avg lat     Std dev
>> ----------------------------------------------------------------
>> CFS             100             951         462         267
>> CFS-x2          100             983         484         308
>> BFS
>> BFS-x2
>>
>> And unfortunately this is where it ends for now, since BFS doesn't boot
>> on the two boxes I tried.
>
> Then who post this in the first place?

You snipped the relevant part of the conclusion, the part where I make a
comment on the cfs latencies.

Don't bother replying to any of my emails if YOU continue writing emails
in this fashion. I have MUCH better things to do than entertain kiddies.
If you do get your act together and want to reply, follow lkml etiquette
and group reply.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Jens Axboe wrote:
> Scheduler       Runtime         Max lat     Avg lat     Std dev
> ----------------------------------------------------------------
> CFS             100             951         462         267
> CFS-x2          100             983         484         308
> BFS
> BFS-x2

Those numbers are buggy, btw, it's not nearly as bad. But responsiveness
under compile load IS bad though, the test app just didn't quantify it
correctly. I'll see if I can get it working properly.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Mon, Sep 07 2009, Jens Axboe wrote:
> > Scheduler       Runtime         Max lat     Avg lat     Std dev
> > ----------------------------------------------------------------
> > CFS             100             951         462         267
> > CFS-x2          100             983         484         308
> > BFS
> > BFS-x2
> 
> Those numbers are buggy, btw, it's not nearly as bad. But 
> responsiveness under compile load IS bad though, the test app just 
> didn't quantify it correctly. I'll see if I can get it working 
> properly.

What's the default latency target on your box:

  cat /proc/sys/kernel/sched_latency_ns

?

And yes, it would be wonderful to get a test-app from you that would 
express the kind of pain you are seeing during compile jobs.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Mon, Sep 07 2009, Jens Axboe wrote:
> > > Scheduler       Runtime         Max lat     Avg lat     Std dev
> > > ----------------------------------------------------------------
> > > CFS             100             951         462         267
> > > CFS-x2          100             983         484         308
> > > BFS
> > > BFS-x2
> > 
> > Those numbers are buggy, btw, it's not nearly as bad. But 
> > responsiveness under compile load IS bad though, the test app just 
> > didn't quantify it correctly. I'll see if I can get it working 
> > properly.
> 
> What's the default latency target on your box:
> 
>   cat /proc/sys/kernel/sched_latency_ns
> 
> ?

It's off right now, but it is set to whatever is the default. I don't
touch it.

> And yes, it would be wonderful to get a test-app from you that would 
> express the kind of pain you are seeing during compile jobs.

I was hoping this one would, but it's not showing anything. I even added
support for doing the ping and wakeup over a socket, to see if the pipe
test was doing well because of the sync wakeup we do there. The net
latency is a little worse, but still good. So no luck in making that app
so far.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1454 bytes --]

On Mon, Sep 07 2009, Jens Axboe wrote:
> > And yes, it would be wonderful to get a test-app from you that would 
> > express the kind of pain you are seeing during compile jobs.
> 
> I was hoping this one would, but it's not showing anything. I even added
> support for doing the ping and wakeup over a socket, to see if the pipe
> test was doing well because of the sync wakeup we do there. The net
> latency is a little worse, but still good. So no luck in making that app
> so far.

Here's a version that bounces timestamps between a producer and a number
of consumers (clients). Not really tested much, but perhaps someone can
compare this on a box that boots BFS and see what happens.

To run it, use -cX where X is the number of children that you wait for a
response from. The max delay between this children is logged for each
wakeup. You can invoke it ala:

$ ./latt -c4 'make -j4'

and it'll dump the max/avg/stddev bounce time after make has completed,
or if you just want to play around, start the compile in one xterm and
do:

$ ./latt -c4 'sleep 5'

to just log for a small period of time. Vary the number of clients to
see how that changes the aggregated latency. 1 should be fast, adding
more clients quickly adds up.

Additionally, it has a -f and -t option that controls the window of
sleep time for the parent between each message. The numbers are in
msecs, and it defaults to a minimum of 100msecs and up to 500msecs.

-- 
Jens Axboe


[-- Attachment #2: latt.c --]
[-- Type: text/x-csrc, Size: 5561 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <getopt.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <time.h>
#include <math.h>


/*
 * In msecs
 */
static unsigned int min_delay = 100;
static unsigned int max_delay = 500;
static unsigned int clients = 1;

#define MAX_CLIENTS		512

struct delays {
	unsigned long nr_delays;
	unsigned long mmap_entries;
	unsigned long max_delay;
	unsigned long delays[0];
};

static struct delays *delays;
static int pipes[MAX_CLIENTS][2];

static unsigned long avg;
static double stddev;

static pid_t app_pid;

#define CLOCKSOURCE		CLOCK_MONOTONIC

#define DEF_ENTRIES		1024

static int parse_options(int argc, char *argv[])
{
	struct option l_opts[] = {
		{ "min-delay", 	1, 	NULL,	'f' },
		{ "max-delay",	1,	NULL,	't' },
		{ "clients",	1,	NULL,	'c' }
	};
	int c, res, index = 0;

	while ((c = getopt_long(argc, argv, "f:t:c:", l_opts, &res)) != -1) {
		index++;
		switch (c) {
			case 'f':
				min_delay = atoi(optarg);
				break;
			case 't':
				max_delay = atoi(optarg);
				break;
			case 'c':
				clients = atoi(optarg);
				if (clients > MAX_CLIENTS)
					clients = MAX_CLIENTS;
				break;
		}
	}

	return index + 1;
}

static pid_t fork_off(const char *app)
{
	pid_t pid;

	pid = fork();
	if (pid)
		return pid;

	exit(system(app));
}

#define entries_to_size(n)	((n) * sizeof(unsigned long) + sizeof(struct delays))

static unsigned long usec_since(struct timespec *start, struct timespec *end)
{
	long secs, nsecs, delay;

	secs = end->tv_sec - start->tv_sec;
	nsecs = end->tv_nsec - start->tv_nsec;

	delay = secs * 1000000L;
	delay += (nsecs / 1000L);

	return delay;
}

static unsigned long usec_since_now(struct timespec *start)
{
	struct timespec e;

	clock_gettime(CLOCKSOURCE, &e);
	return usec_since(start, &e);
}

static void log_delay(unsigned long delay)
{
	if (delays->nr_delays == delays->mmap_entries) {
		unsigned long new_size;

		delays->mmap_entries <<= 1;
		new_size = entries_to_size(delays->mmap_entries);
		delays = realloc(delays, new_size);
	}

	delays->delays[delays->nr_delays++] = delay;

	if (delay > delays->max_delay)
		delays->max_delay = delay;
}

static void run_child(int *pipe)
{
	struct timespec ts;

	do {
		int ret;

		ret = read(pipe[0], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		clock_gettime(CLOCKSOURCE, &ts);

		ret = write(pipe[1], &ts, sizeof(ts));
		if (ret <= 0)
			break;
	} while (1);
}

static void do_rand_sleep(void)
{
	unsigned int msecs;

	msecs = min_delay + ((float) max_delay * (rand() / (RAND_MAX + 1.0)));
	usleep(msecs * 1000);
}

static void kill_connection(void)
{
	int i;

	for (i = 0; i < clients; i++) {
		if (pipes[i][0] != -1) {
			close(pipes[i][0]);
			pipes[i][0] = -1;
		}
		if (pipes[i][1] != -1) {
			close(pipes[i][1]);
			pipes[i][1] = -1;
		}
	}
}

static void run_parent(void)
{
	struct timespec *t1, t2;
	int status, ret, do_exit = 0, i;

	t1 = malloc(sizeof(struct timespec) * clients);

	srand(1234);

	do {
		unsigned long delay, max_delay = 0;

		do_rand_sleep();

		ret = waitpid(app_pid, &status, WNOHANG);
		if (ret < 0) {
			perror("waitpid");
			break;
		} else if (ret == app_pid &&
			   (WIFSIGNALED(status) || WIFEXITED(status))) {
			do_exit = 1;
			kill_connection();
		}
			
		for (i = 0; i < clients; i++) {
			clock_gettime(CLOCKSOURCE, &t1[i]);
			if (write(pipes[i][1], &t1[i], sizeof(t2)) != sizeof(t2)) {
				do_exit = 1;
				break;
			}
		}

		for (i = 0; i < clients; i++) {
			if (read(pipes[i][0], &t2, sizeof(t2)) != sizeof(t2)) {
				do_exit = 1;
				break;
			}
			delay = usec_since(&t1[i], &t2);
			if (delay > max_delay)
				max_delay = delay;
		}

		log_delay(max_delay);
	} while (!do_exit);

	kill_connection();
}

static void parent_setup_connection(void)
{
	int i;

	for (i = 0; i < clients; i++) {
		if (pipe(pipes[i])) {
			perror("pipe");
			return;
		}
	}
}

static void run_test(void)
{
	pid_t cpids[MAX_CLIENTS];
	int i, status;

	parent_setup_connection();

	for (i = 0; i < clients; i++) {
		cpids[i] = fork();
		if (cpids[i])
			continue;

		run_child(pipes[i]);
		exit(0);
	}

	run_parent();

	for (i = 0; i < clients; i++)
		kill(cpids[i], SIGQUIT);
	for (i = 0; i < clients; i++)
		waitpid(cpids[i], &status, 0);
}

static void setup_shared_area(void)
{
	delays = malloc(entries_to_size(DEF_ENTRIES));
	delays->nr_delays = 0;
	delays->mmap_entries = DEF_ENTRIES;
}

static void calc_latencies(void)
{
	unsigned long long sum = 0;
	int i;

	if (!delays->nr_delays)
		return;

	for (i = 0; i < delays->nr_delays; i++)
		sum += delays->delays[i];

	avg = sum / delays->nr_delays;

	if (delays->nr_delays < 2)
		return;

	sum = 0;
	for (i = 0; i < delays->nr_delays; i++) {
		long diff;

		diff = delays->delays[i] - avg;
		sum += (diff * diff);
	}

	stddev = sqrt(sum / (delays->nr_delays - 1));
}

static void handle_sigint(int sig)
{
	kill(app_pid, SIGINT);
}

int main(int argc, char *argv[])
{
	int app_offset, off;
	char app[256];

	setup_shared_area();

	off = 0;
	app_offset = parse_options(argc, argv);
	while (app_offset < argc) {
		if (off) {
			app[off] = ' ';
			off++;
		}
		off += sprintf(app + off, "%s", argv[app_offset]);
		app_offset++;
	}

	signal(SIGINT, handle_sigint);
	app_pid = fork_off(app);
	run_test();

	calc_latencies();

	printf("Entries: %lu (clients=%d)\n", delays->nr_delays, clients);
	printf("\nAverages (in usecs)\n");
	printf("-------------------\n");
	printf("\tMax\t %lu\n", delays->max_delay);
	printf("\tAvg\t %lu\n", avg);
	printf("\tStdev\t %.0f\n", stddev);

	free(delays);
	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1068 bytes --]

On Mon, Sep 07 2009, Jens Axboe wrote:
> On Mon, Sep 07 2009, Jens Axboe wrote:
> > > And yes, it would be wonderful to get a test-app from you that would 
> > > express the kind of pain you are seeing during compile jobs.
> > 
> > I was hoping this one would, but it's not showing anything. I even added
> > support for doing the ping and wakeup over a socket, to see if the pipe
> > test was doing well because of the sync wakeup we do there. The net
> > latency is a little worse, but still good. So no luck in making that app
> > so far.
> 
> Here's a version that bounces timestamps between a producer and a number
> of consumers (clients). Not really tested much, but perhaps someone can
> compare this on a box that boots BFS and see what happens.

And here's a newer version. It ensures that clients are running before
sending a timestamp, and it drops the first and last log entry to
eliminate any weird effects there. Accuracy should also be improved.

On an idle box, it'll usually log all zeroes. Sometimes I see 3-4msec
latencies, weird.

-- 
Jens Axboe


[-- Attachment #2: latt.c --]
[-- Type: text/x-csrc, Size: 9792 bytes --]

/*
 * Simple latency tester that combines multiple processes.
 *
 * Compile: gcc -Wall -O2 -D_GNU_SOURCE -lrt -lm -o latt latt.c
 *
 * Run with: latt -c8 'program --args'
 *
 * Options:
 *
 *	-cX	Use X number of clients
 *	-fX	Use X msec as the minimum sleep time for the parent
 *	-tX	Use X msec as the maximum sleep time for the parent
 *	-v	Print all delays as they are logged
 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <getopt.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <time.h>
#include <math.h>
#include <poll.h>
#include <pthread.h>


/*
 * In msecs
 */
static unsigned int min_delay = 100;
static unsigned int max_delay = 500;
static unsigned int clients = 1;
static unsigned int verbose;

#define MAX_CLIENTS		512

struct delays {
	unsigned long nr_delays;
	unsigned long total_entries;
	unsigned long max_delay;
	unsigned long delays[0];
};

#define entries_to_size(n)	\
	((n) * sizeof(unsigned long) + sizeof(struct delays))

static struct delays *delays;
static int pipes[MAX_CLIENTS][2];

static unsigned long avg;
static double stddev;

static pid_t app_pid;

#define CLOCKSOURCE		CLOCK_MONOTONIC

#define DEF_ENTRIES		1024

struct mutex {
	pthread_mutex_t lock;
	pthread_cond_t cond;
	int value;
	int waiters;
};

static void init_mutex(struct mutex *mutex)
{
	pthread_mutexattr_t attr;
	pthread_condattr_t cond;

	pthread_mutexattr_init(&attr);
	pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
	pthread_condattr_init(&cond);
	pthread_condattr_setpshared(&cond, PTHREAD_PROCESS_SHARED);
	pthread_cond_init(&mutex->cond, &cond);
	pthread_mutex_init(&mutex->lock, &attr);

	mutex->value = 0;
	mutex->waiters = 0;
}

static void mutex_down(struct mutex *mutex)
{
	pthread_mutex_lock(&mutex->lock);

	while (!mutex->value) {
		mutex->waiters++;
		pthread_cond_wait(&mutex->cond, &mutex->lock);
		mutex->waiters--;
	}

	mutex->value--;
	pthread_mutex_unlock(&mutex->lock);
}

static void mutex_up(struct mutex *mutex)
{
	pthread_mutex_lock(&mutex->lock);
	if (!mutex->value && mutex->waiters)
		pthread_cond_signal(&mutex->cond);
	mutex->value++;
	pthread_mutex_unlock(&mutex->lock);
}

static int parse_options(int argc, char *argv[])
{
	struct option l_opts[] = {
		{ "min-delay", 	1, 	NULL,	'f' },
		{ "max-delay",	1,	NULL,	't' },
		{ "clients",	1,	NULL,	'c' },
		{ "verbose",	1,	NULL,	'v' }
	};
	int c, res, index = 0;

	while ((c = getopt_long(argc, argv, "f:t:c:v", l_opts, &res)) != -1) {
		index++;
		switch (c) {
			case 'f':
				min_delay = atoi(optarg);
				break;
			case 't':
				max_delay = atoi(optarg);
				break;
			case 'c':
				clients = atoi(optarg);
				if (clients > MAX_CLIENTS)
					clients = MAX_CLIENTS;
				break;
			case 'v':
				verbose = 1;
				break;
		}
	}

	return index + 1;
}

static pid_t fork_off(const char *app)
{
	pid_t pid;

	pid = fork();
	if (pid)
		return pid;

	exit(system(app));
}

static unsigned long usec_since(struct timespec *start, struct timespec *end)
{
	long secs, nsecs;

	secs = end->tv_sec - start->tv_sec;
	nsecs = end->tv_nsec - start->tv_nsec;

	return secs * 1000000L + nsecs / 1000;
}

static void log_delay(unsigned long delay)
{
	if (verbose)
		printf("log delay %8lu usec\n", delay);

	if (delays->nr_delays == delays->total_entries) {
		unsigned long new_size;

		delays->total_entries <<= 1;
		new_size = entries_to_size(delays->total_entries);
		delays = realloc(delays, new_size);
	}

	delays->delays[delays->nr_delays++] = delay;
}

/*
 * Reads a timestamp (which is ignored, it's just a wakeup call), and replies
 * with the timestamp of when we saw it
 */
static void run_child(int *pipe, struct mutex *mutex)
{
	struct timespec ts;

	mutex_up(mutex);

	do {
		int ret;

		ret = read(pipe[0], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		clock_gettime(CLOCKSOURCE, &ts);

		ret = write(pipe[1], &ts, sizeof(ts));
		if (ret <= 0)
			break;
	} while (1);
}

/*
 * Do a random sleep between min and max delay
 */
static void do_rand_sleep(void)
{
	unsigned int msecs;

	msecs = min_delay + ((float) max_delay * (rand() / (RAND_MAX + 1.0)));
	usleep(msecs * 1000);
}

static void kill_connection(void)
{
	int i;

	for (i = 0; i < clients; i++) {
		if (pipes[i][0] != -1) {
			close(pipes[i][0]);
			pipes[i][0] = -1;
		}
		if (pipes[i][1] != -1) {
			close(pipes[i][1]);
			pipes[i][1] = -1;
		}
	}
}

static int __write_ts(int fd, struct timespec *ts)
{
	clock_gettime(CLOCKSOURCE, ts);

	return write(fd, ts, sizeof(*ts)) != sizeof(*ts);
}

static int write_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts)
{
	unsigned int i;

	for (i = 0; i < clients; i++) {
		if (pfd[i].revents & (POLLERR | POLLHUP | POLLNVAL))
			return 1;
		if (pfd[i].revents & POLLOUT) {
			pfd[i].events = 0;
			if (__write_ts(pfd[i].fd, &ts[i]))
				return 1;
			nr--;
		}
		if (!nr)
			break;
	}

	return 0;
}

static long __read_ts(int fd, struct timespec *ts)
{
	struct timespec t;

	if (read(fd, &t, sizeof(t)) != sizeof(t))
		return -1;

	return usec_since(ts, &t);
}

static long read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts)
{
	long delay, max_delay = 0;
	unsigned int i;

	for (i = 0; i < clients; i++) {
		if (pfd[i].revents & (POLLERR | POLLHUP | POLLNVAL))
			return -1L;
		if (pfd[i].revents & POLLIN) {
			pfd[i].events = 0;
			delay = __read_ts(pfd[i].fd, &ts[i]);
			if (delay < 0)
				return -1L;
			else if (delay > max_delay)
				max_delay = delay;
			nr--;
		}
		if (!nr)
			break;
	}

	return max_delay;
}

static int app_has_exited(void)
{
	int ret, status;

	/*
	 * If our app has exited, stop
	 */
	ret = waitpid(app_pid, &status, WNOHANG);
	if (ret < 0) {
		perror("waitpid");
		return 1;
	} else if (ret == app_pid &&
		   (WIFSIGNALED(status) || WIFEXITED(status))) {
		return 1;
	}

	return 0;
}

/*
 * While our given app is running, send a timestamp to each client and
 * log the maximum latency for each of them to wakeup and reply
 */
static void run_parent(void)
{
	struct pollfd *ipfd, *opfd;
	int do_exit = 0, i;
	struct timespec *t1;

	t1 = malloc(sizeof(struct timespec) * clients);
	opfd = malloc(sizeof(struct pollfd) * clients);
	ipfd = malloc(sizeof(struct pollfd) * clients);

	srand(1234);

	do {
		unsigned long max_delay = 0;
		unsigned pending_events;

		do_rand_sleep();

		for (i = 0; i < clients; i++) {
			ipfd[i].fd = pipes[i][0];
			ipfd[i].events = POLLIN;
			opfd[i].fd = pipes[i][1];
			opfd[i].events = POLLOUT;
		}

		/*
		 * Write wakeup calls
		 */
		pending_events = clients;
		while (pending_events) {
			int evts = poll(opfd, clients, 0);

			if (app_has_exited()) {
				do_exit = 1;
				break;
			}

			if (evts < 0) {
				do_exit = 1;
				break;
			} else if (!evts)
				continue;

			if (write_ts(opfd, evts, t1)) {
				do_exit = 1;
				break;
			}

			pending_events -= evts;
		}

		if (do_exit)
			break;

		/*
		 * Poll and read replies
		 */
		pending_events = clients;
		while (pending_events) {
			int evts = poll(ipfd, clients, 0);

			if (app_has_exited()) {
				do_exit = 1;
				break;
			}

			if (evts < 0) {
				do_exit = 1;
				break;
			} else if (!evts)
				continue;

			max_delay = read_ts(ipfd, evts, t1);
			if (max_delay < 0) {
				do_exit = 1;
				break;
			}

			pending_events -= evts;
		}
		log_delay(max_delay);
	} while (!do_exit);

	free(t1);
	free(ipfd);
	free(opfd);
	kill_connection();
}

static void run_test(void)
{
	struct mutex *mutex;
	pid_t *cpids;
	int i, status;

	mutex = mmap(NULL, sizeof(*mutex), PROT_READ|PROT_WRITE,
			MAP_SHARED | MAP_ANONYMOUS, 0, 0);
	if (mutex == MAP_FAILED) {
		perror("mmap");
		return;
	}

	init_mutex(mutex);

	for (i = 0; i < clients; i++) {
		if (pipe(pipes[i])) {
			perror("pipe");
			return;
		}
	}

	cpids = malloc(sizeof(pid_t) * clients);

	for (i = 0; i < clients; i++) {
		cpids[i] = fork();
		if (cpids[i]) {
			mutex_down(mutex);
			continue;
		}

		run_child(pipes[i], mutex);
		exit(0);
	}

	run_parent();

	for (i = 0; i < clients; i++)
		kill(cpids[i], SIGQUIT);
	for (i = 0; i < clients; i++)
		waitpid(cpids[i], &status, 0);

	free(cpids);
	munmap(mutex, sizeof(*mutex));
}

static void setup_log(void)
{
	delays = malloc(entries_to_size(DEF_ENTRIES));
	delays->nr_delays = 0;
	delays->total_entries = DEF_ENTRIES;
}

/*
 * Calculate average and stddev for the entries in the log. Drop the
 * first and last entry.
 */
static int calc_latencies(void)
{
	unsigned long long sum = 0;
	int i;

	if (delays->nr_delays <= 2)
		return 1;

	for (i = 1; i < delays->nr_delays - 1; i++) {
		unsigned long delay = delays->delays[i];

		if (delay > delays->max_delay)
			delays->max_delay = delay;

		sum += delay;
	}

	avg = sum / (delays->nr_delays - 2);

	if (delays->nr_delays <= 3)
		return 0;

	sum = 0;
	for (i = 1; i < delays->nr_delays - 1; i++) {
		long diff;

		diff = delays->delays[i] - avg;
		sum += (diff * diff);
	}

	stddev = sqrt(sum / (delays->nr_delays - 3));
	return 0;
}

static void handle_sigint(int sig)
{
	kill(app_pid, SIGINT);
}

int main(int argc, char *argv[])
{
	int app_offset, off;
	char app[256];

	setup_log();

	off = 0;
	app_offset = parse_options(argc, argv);
	while (app_offset < argc) {
		if (off) {
			app[off] = ' ';
			off++;
		}
		off += sprintf(app + off, "%s", argv[app_offset]);
		app_offset++;
	}

	signal(SIGINT, handle_sigint);

	/*
	 * Start app and start logging latencies
	 */
	app_pid = fork_off(app);
	run_test();

	if (calc_latencies()) {
		printf("Runtime too short to render result\n");
		return 1;
	}

	printf("Entries: %lu (clients=%d)\n", delays->nr_delays, clients);
	printf("\nAverages:\n");
	printf("------------------------------\n");
	printf("\tMax\t %8lu usec\n", delays->max_delay);
	printf("\tAvg\t %8lu usec\n", avg);
	printf("\tStdev\t %8.0f usec\n", stddev);

	free(delays);
	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

[-- Attachment #1: Type: text/plain, Size: 1272 bytes --]

On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> And here's a newer version.

I tinkered a bit with your proglet and finally found the problem.

You used a single pipe per child, this means the loop in run_child()
would consume what it just wrote out until it got force preempted by the
parent which would also get woken.

This results in the child spinning a while (its full quota) and only
reporting the last timestamp to the parent.

Since consumer (parent) is a single thread the program basically
measures the worst delay in a thundering herd wakeup of N children.

The below version yields:

idle

[root@opteron sched]# ./latt -c8 sleep 30
Entries: 664 (clients=8)

Averages:
------------------------------
        Max           128 usec
        Avg            26 usec
        Stdev          16 usec


make -j4

[root@opteron sched]# ./latt -c8 sleep 30
Entries: 648 (clients=8)                 

Averages:
------------------------------
        Max         20861 usec
        Avg          3763 usec
        Stdev        4637 usec


Mike's patch, make -j4

[root@opteron sched]# ./latt -c8 sleep 30
Entries: 648 (clients=8)

Averages:
------------------------------
        Max         17854 usec
        Avg          6298 usec
        Stdev        4735 usec


[-- Attachment #2: latt.c --]
[-- Type: text/x-csrc, Size: 9214 bytes --]

/*
 * Simple latency tester that combines multiple processes.
 *
 * Compile: gcc -Wall -O2 -D_GNU_SOURCE -lrt -lm -o latt latt.c
 *
 * Run with: latt -c8 'program --args'
 *
 * Options:
 *
 *	-cX	Use X number of clients
 *	-fX	Use X msec as the minimum sleep time for the parent
 *	-tX	Use X msec as the maximum sleep time for the parent
 *	-v	Print all delays as they are logged
 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <getopt.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <time.h>
#include <math.h>
#include <poll.h>
#include <pthread.h>


/*
 * In msecs
 */
static unsigned int min_delay = 100;
static unsigned int max_delay = 500;
static unsigned int clients = 1;
static unsigned int verbose;

#define MAX_CLIENTS		512

struct stats
{
	double n, mean, M2, max;
};

static void update_stats(struct stats *stats, unsigned long long val)
{
	double delta, x = val;

	stats->n++;
	delta = x - stats->mean;
	stats->mean += delta / stats->n;
	stats->M2 += delta*(x - stats->mean);

	if (stats->max < x)
		stats->max = x;
}

static unsigned long nr_stats(struct stats *stats)
{
	return stats->n;
}

static double max_stats(struct stats *stats)
{
	return stats->max;
}

static double avg_stats(struct stats *stats)
{
	return stats->mean;
}

/*
 * http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
 *
 *       (\Sum n_i^2) - ((\Sum n_i)^2)/n
 * s^2 = -------------------------------
 *                  n - 1
 *
 * http://en.wikipedia.org/wiki/Stddev
 */
static double stddev_stats(struct stats *stats)
{
	double variance = stats->M2 / (stats->n - 1);

	return sqrt(variance);
}

/*
 * The std dev of the mean is related to the std dev by:
 *
 *             s
 * s_mean = -------
 *          sqrt(n)
 *
 */
static double stddev_mean_stats(struct stats *stats)
{
	double variance = stats->M2 / (stats->n - 1);
	double variance_mean = variance / stats->n;

	return sqrt(variance_mean);
}

struct stats delay_stats;

static int pipes[MAX_CLIENTS*2][2];

static pid_t app_pid;

#define CLOCKSOURCE		CLOCK_MONOTONIC

struct sem {
	pthread_mutex_t lock;
	pthread_cond_t cond;
	int value;
	int waiters;
};

static void init_sem(struct sem *sem)
{
	pthread_mutexattr_t attr;
	pthread_condattr_t cond;

	pthread_mutexattr_init(&attr);
	pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
	pthread_condattr_init(&cond);
	pthread_condattr_setpshared(&cond, PTHREAD_PROCESS_SHARED);
	pthread_cond_init(&sem->cond, &cond);
	pthread_mutex_init(&sem->lock, &attr);

	sem->value = 0;
	sem->waiters = 0;
}

static void sem_down(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);

	while (!sem->value) {
		sem->waiters++;
		pthread_cond_wait(&sem->cond, &sem->lock);
		sem->waiters--;
	}

	sem->value--;
	pthread_mutex_unlock(&sem->lock);
}

static void sem_up(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);
	if (!sem->value && sem->waiters)
		pthread_cond_signal(&sem->cond);
	sem->value++;
	pthread_mutex_unlock(&sem->lock);
}

static int parse_options(int argc, char *argv[])
{
	struct option l_opts[] = {
		{ "min-delay", 	1, 	NULL,	'f' },
		{ "max-delay",	1,	NULL,	't' },
		{ "clients",	1,	NULL,	'c' },
		{ "verbose",	1,	NULL,	'v' }
	};
	int c, res, index = 0;

	while ((c = getopt_long(argc, argv, "f:t:c:v", l_opts, &res)) != -1) {
		index++;
		switch (c) {
			case 'f':
				min_delay = atoi(optarg);
				break;
			case 't':
				max_delay = atoi(optarg);
				break;
			case 'c':
				clients = atoi(optarg);
				if (clients > MAX_CLIENTS)
					clients = MAX_CLIENTS;
				break;
			case 'v':
				verbose = 1;
				break;
		}
	}

	return index + 1;
}

static pid_t fork_off(const char *app)
{
	pid_t pid;

	pid = fork();
	if (pid)
		return pid;

	exit(system(app));
}

static unsigned long usec_since(struct timespec *start, struct timespec *end)
{
	unsigned long long s, e;

	s = start->tv_sec * 1000000000ULL + start->tv_nsec;
	e =   end->tv_sec * 1000000000ULL +   end->tv_nsec;

	return (e - s) / 1000;
}

static void log_delay(unsigned long delay)
{
	if (verbose) {
		fprintf(stderr, "log delay %8lu usec\n", delay);
		fflush(stderr);
	}

	update_stats(&delay_stats, delay);
}

/*
 * Reads a timestamp (which is ignored, it's just a wakeup call), and replies
 * with the timestamp of when we saw it
 */
static void run_child(int *in, int *out, struct sem *sem)
{
	struct timespec ts;

	if (verbose) {
		fprintf(stderr, "present: %d\n", getpid());
		fflush(stderr);
	}

	sem_up(sem);

	do {
		int ret;

		ret = read(in[0], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		if (ret != sizeof(ts))
			printf("bugger3\n");

		clock_gettime(CLOCKSOURCE, &ts);

		ret = write(out[1], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		if (ret != sizeof(ts))
			printf("bugger4\n");

		if (verbose) {
			fprintf(stderr, "alife: %d\n", getpid());
			fflush(stderr);
		}
	} while (1);
}

/*
 * Do a random sleep between min and max delay
 */
static void do_rand_sleep(void)
{
	unsigned int msecs;

	msecs = min_delay + ((float) max_delay * (rand() / (RAND_MAX + 1.0)));
	if (verbose) {
		fprintf(stderr, "sleeping for: %u msec\n", msecs);
		fflush(stderr);
	}
	usleep(msecs * 1000);
}

static void kill_connection(void)
{
	int i;

	for (i = 0; i < 2*clients; i++) {
		if (pipes[i][0] != -1) {
			close(pipes[i][0]);
			pipes[i][0] = -1;
		}
		if (pipes[i][1] != -1) {
			close(pipes[i][1]);
			pipes[i][1] = -1;
		}
	}
}

static int __write_ts(int i, struct timespec *ts)
{
	int fd = pipes[2*i][1];

	clock_gettime(CLOCKSOURCE, ts);

	return write(fd, ts, sizeof(*ts)) != sizeof(*ts);
}

static long __read_ts(int i, struct timespec *ts)
{
	int fd = pipes[2*i+1][0];
	struct timespec t;

	if (read(fd, &t, sizeof(t)) != sizeof(t))
		return -1;

	log_delay(usec_since(ts, &t));

	return 0;
}

static int read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts)
{
	unsigned int i;

	for (i = 0; i < clients; i++) {
		if (pfd[i].revents & (POLLERR | POLLHUP | POLLNVAL))
			return -1L;
		if (pfd[i].revents & POLLIN) {
			pfd[i].events = 0;
			if (__read_ts(i, &ts[i]))
				return -1L;
			nr--;
		}
		if (!nr)
			break;
	}

	return 0;
}

static int app_has_exited(void)
{
	int ret, status;

	/*
	 * If our app has exited, stop
	 */
	ret = waitpid(app_pid, &status, WNOHANG);
	if (ret < 0) {
		perror("waitpid");
		return 1;
	} else if (ret == app_pid &&
		   (WIFSIGNALED(status) || WIFEXITED(status))) {
		return 1;
	}

	return 0;
}

/*
 * While our given app is running, send a timestamp to each client and
 * log the maximum latency for each of them to wakeup and reply
 */
static void run_parent(pid_t *cpids)
{
	struct pollfd *ipfd;
	int do_exit = 0, i;
	struct timespec *t1;

	t1 = malloc(sizeof(struct timespec) * clients);
	ipfd = malloc(sizeof(struct pollfd) * clients);

	srand(1234);

	do {
		unsigned long delay;
		unsigned pending_events;

		do_rand_sleep();

		if (app_has_exited())
			break;

		for (i = 0; i < clients; i++) {
			ipfd[i].fd = pipes[2*i+1][0];
			ipfd[i].events = POLLIN;
		}

		/*
		 * Write wakeup calls
		 */
		for (i = 0; i < clients; i++) {
			if (verbose) {
				fprintf(stderr, "waking: %d\n", cpids[i]);
				fflush(stderr);
			}

			if (__write_ts(i, t1+i)) {
				do_exit = 1;
				break;
			}
		}

		if (do_exit)
			break;

		/*
		 * Poll and read replies
		 */
		pending_events = clients;
		while (pending_events) {
			int evts = poll(ipfd, clients, 0);

			if (evts < 0) {
				do_exit = 1;
				break;
			} else if (!evts) {
				/* printf("bugger2\n"); */
				continue;
			}

			if (read_ts(ipfd, evts, t1)) {
				do_exit = 1;
				break;
			}

			pending_events -= evts;
		}
	} while (!do_exit);

	free(t1);
	free(ipfd);
	kill_connection();
}

static void run_test(void)
{
	struct sem *sem;
	pid_t *cpids;
	int i, status;

	sem = mmap(NULL, sizeof(*sem), PROT_READ|PROT_WRITE,
			MAP_SHARED | MAP_ANONYMOUS, 0, 0);
	if (sem == MAP_FAILED) {
		perror("mmap");
		return;
	}

	init_sem(sem);

	for (i = 0; i < 2*clients; i++) {
		if (pipe(pipes[i])) {
			perror("pipe");
			return;
		}
	}

	cpids = malloc(sizeof(pid_t) * clients);

	for (i = 0; i < clients; i++) {
		cpids[i] = fork();
		if (cpids[i]) {
			sem_down(sem);
			continue;
		}

		run_child(pipes[2*i], pipes[2*i+1], sem);
		exit(0);
	}

	run_parent(cpids);

	for (i = 0; i < clients; i++)
		kill(cpids[i], SIGQUIT);
	for (i = 0; i < clients; i++)
		waitpid(cpids[i], &status, 0);

	free(cpids);
	munmap(sem, sizeof(*sem));
}

static void handle_sigint(int sig)
{
	kill(app_pid, SIGINT);
}

int main(int argc, char *argv[])
{
	int app_offset, off;
	char app[256];

	off = 0;
	app_offset = parse_options(argc, argv);
	while (app_offset < argc) {
		if (off) {
			app[off] = ' ';
			off++;
		}
		off += sprintf(app + off, "%s", argv[app_offset]);
		app_offset++;
	}

	signal(SIGINT, handle_sigint);

	/*
	 * Start app and start logging latencies
	 */
	app_pid = fork_off(app);
	run_test();

	printf("Entries: %lu (clients=%d)\n", nr_stats(&delay_stats), clients);
	printf("\nAverages:\n");
	printf("------------------------------\n");
	printf("\tMax\t %8.0f usec\n", max_stats(&delay_stats));
	printf("\tAvg\t %8.0f usec\n", avg_stats(&delay_stats));
	printf("\tStdev\t %8.0f usec\n", stddev_stats(&delay_stats));

	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Tue, Sep 08 2009, Peter Zijlstra wrote:
> On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > And here's a newer version.
> 
> I tinkered a bit with your proglet and finally found the problem.
> 
> You used a single pipe per child, this means the loop in run_child()
> would consume what it just wrote out until it got force preempted by the
> parent which would also get woken.
> 
> This results in the child spinning a while (its full quota) and only
> reporting the last timestamp to the parent.

Oh doh, that's not well thought out. Well it was a quick hack :-)
Thanks for the fixup, now it's at least usable to some degree.

> Since consumer (parent) is a single thread the program basically
> measures the worst delay in a thundering herd wakeup of N children.

Yes, it's really meant to measure how long it takes to wake a group of
processes, assuming that this is where things fall down on the 'box
loaded, switch desktop' case. Now whether that's useful or not or
whether this test app is worth the bits it takes up on the hard drive,
is another question.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > And here's a newer version.
> > 
> > I tinkered a bit with your proglet and finally found the 
> > problem.
> > 
> > You used a single pipe per child, this means the loop in 
> > run_child() would consume what it just wrote out until it got 
> > force preempted by the parent which would also get woken.
> > 
> > This results in the child spinning a while (its full quota) and 
> > only reporting the last timestamp to the parent.
> 
> Oh doh, that's not well thought out. Well it was a quick hack :-) 
> Thanks for the fixup, now it's at least usable to some degree.

What kind of latencies does it report on your box?

Our vanilla scheduler default latency targets are:

  single-core: 20 msecs
    dual-core: 40 msecs
    quad-core: 60 msecs
    opto-core: 80 msecs

You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
/proc/sys/kernel/sched_latency_ns:

   echo 10000000 > /proc/sys/kernel/sched_latency_ns

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 09:13 AM, Ingo Molnar wrote:
>
> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>
>> On Tue, Sep 08 2009, Peter Zijlstra wrote:
>>> On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
>>>> And here's a newer version.
>>>
>>> I tinkered a bit with your proglet and finally found the
>>> problem.
>>>
>>> You used a single pipe per child, this means the loop in
>>> run_child() would consume what it just wrote out until it got
>>> force preempted by the parent which would also get woken.
>>>
>>> This results in the child spinning a while (its full quota) and
>>> only reporting the last timestamp to the parent.
>>
>> Oh doh, that's not well thought out. Well it was a quick hack :-)
>> Thanks for the fixup, now it's at least usable to some degree.
>
> What kind of latencies does it report on your box?
>
> Our vanilla scheduler default latency targets are:
>
>    single-core: 20 msecs
>      dual-core: 40 msecs
>      quad-core: 60 msecs
>      opto-core: 80 msecs
>
> You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via
> /proc/sys/kernel/sched_latency_ns:
>
>     echo 10000000>  /proc/sys/kernel/sched_latency_ns

I've tried values ranging from 10000000 down to 100000.  This results in 
the stalls/freezes being a bit shorter, but clearly still there.  It 
does not eliminate them.

If there's anything else I can try/test, I would be happy to do so.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > > And here's a newer version.
> > > 
> > > I tinkered a bit with your proglet and finally found the 
> > > problem.
> > > 
> > > You used a single pipe per child, this means the loop in 
> > > run_child() would consume what it just wrote out until it got 
> > > force preempted by the parent which would also get woken.
> > > 
> > > This results in the child spinning a while (its full quota) and 
> > > only reporting the last timestamp to the parent.
> > 
> > Oh doh, that's not well thought out. Well it was a quick hack :-) 
> > Thanks for the fixup, now it's at least usable to some degree.
> 
> What kind of latencies does it report on your box?
> 
> Our vanilla scheduler default latency targets are:
> 
>   single-core: 20 msecs
>     dual-core: 40 msecs
>     quad-core: 60 msecs
>     opto-core: 80 msecs
> 
> You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
> /proc/sys/kernel/sched_latency_ns:
> 
>    echo 10000000 > /proc/sys/kernel/sched_latency_ns

He would also need to lower min_granularity, otherwise, it'd be larger
than the whole latency target.

I'm testing right now, and one thing that is definitely a problem is the
amount of sleeper fairness we're giving.  A full latency is just too
much short term fairness in my testing.  While sleepers are catching up,
hogs languish.  That's the biggest issue going on.

I've also been doing some timings of make -j4 (looking at idle time),
and find that child_runs_first is mildly detrimental to fork/exec load,
as are buddies.

I'm running with the below at the moment.  (the kthread/workqueue thing
is just because I don't see any reason for it to exist, so consider it
to be a waste of perfectly good math;)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index 6ec4643..a44210e 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -16,8 +16,6 @@
 #include <linux/mutex.h>
 #include <trace/events/sched.h>
 
-#define KTHREAD_NICE_LEVEL (-5)
-
 static DEFINE_SPINLOCK(kthread_create_lock);
 static LIST_HEAD(kthread_create_list);
 
@@ -150,7 +148,6 @@ struct task_struct *kthread_create(int (*threadfn)(void *data),
 		 * The kernel thread should not inherit these properties.
 		 */
 		sched_setscheduler_nocheck(create.result, SCHED_NORMAL, &param);
-		set_user_nice(create.result, KTHREAD_NICE_LEVEL);
 		set_cpus_allowed_ptr(create.result, cpu_all_mask);
 	}
 	return create.result;
@@ -226,7 +223,6 @@ int kthreadd(void *unused)
 	/* Setup a clean context for our children to inherit. */
 	set_task_comm(tsk, "kthreadd");
 	ignore_signals(tsk);
-	set_user_nice(tsk, KTHREAD_NICE_LEVEL);
 	set_cpus_allowed_ptr(tsk, cpu_all_mask);
 	set_mems_allowed(node_possible_map);
 
diff --git a/kernel/sched.c b/kernel/sched.c
index c512a02..e68c341 100644
--- a/kernel/sched.c
+++ b/kernel/sched.c
@@ -7124,33 +7124,6 @@ void __cpuinit init_idle(struct task_struct *idle, int cpu)
  */
 cpumask_var_t nohz_cpu_mask;
 
-/*
- * Increase the granularity value when there are more CPUs,
- * because with more CPUs the 'effective latency' as visible
- * to users decreases. But the relationship is not linear,
- * so pick a second-best guess by going with the log2 of the
- * number of CPUs.
- *
- * This idea comes from the SD scheduler of Con Kolivas:
- */
-static inline void sched_init_granularity(void)
-{
-	unsigned int factor = 1 + ilog2(num_online_cpus());
-	const unsigned long limit = 200000000;
-
-	sysctl_sched_min_granularity *= factor;
-	if (sysctl_sched_min_granularity > limit)
-		sysctl_sched_min_granularity = limit;
-
-	sysctl_sched_latency *= factor;
-	if (sysctl_sched_latency > limit)
-		sysctl_sched_latency = limit;
-
-	sysctl_sched_wakeup_granularity *= factor;
-
-	sysctl_sched_shares_ratelimit *= factor;
-}
-
 #ifdef CONFIG_SMP
 /*
  * This is how migration works:
@@ -9356,7 +9329,6 @@ void __init sched_init_smp(void)
 	/* Move init over to a non-isolated CPU */
 	if (set_cpus_allowed_ptr(current, non_isolated_cpus) < 0)
 		BUG();
-	sched_init_granularity();
 	free_cpumask_var(non_isolated_cpus);
 
 	alloc_cpumask_var(&fallback_doms, GFP_KERNEL);
@@ -9365,7 +9337,6 @@ void __init sched_init_smp(void)
 #else
 void __init sched_init_smp(void)
 {
-	sched_init_granularity();
 }
 #endif /* CONFIG_SMP */
 
diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index e386e5d..ff7fec9 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -51,7 +51,7 @@ static unsigned int sched_nr_latency = 5;
  * After fork, child runs first. (default) If set to 0 then
  * parent will (try to) run first.
  */
-const_debug unsigned int sysctl_sched_child_runs_first = 1;
+const_debug unsigned int sysctl_sched_child_runs_first = 0;
 
 /*
  * sys_sched_yield() compat mode
@@ -713,7 +713,7 @@ place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int initial)
 	if (!initial) {
 		/* sleeps upto a single latency don't count. */
 		if (sched_feat(NEW_FAIR_SLEEPERS)) {
-			unsigned long thresh = sysctl_sched_latency;
+			unsigned long thresh = sysctl_sched_min_granularity;
 
 			/*
 			 * Convert the sleeper threshold into virtual time.
@@ -1502,7 +1502,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
 	 */
 	if (sched_feat(LAST_BUDDY) && likely(se->on_rq && curr != rq->idle))
 		set_last_buddy(se);
-	set_next_buddy(pse);
+	if (sched_feat(NEXT_BUDDY))
+		set_next_buddy(pse);
 
 	/*
 	 * We can come here with TIF_NEED_RESCHED already set from new task
diff --git a/kernel/sched_features.h b/kernel/sched_features.h
index 4569bfa..85d30d1 100644
--- a/kernel/sched_features.h
+++ b/kernel/sched_features.h
@@ -13,5 +13,6 @@ SCHED_FEAT(LB_BIAS, 1)
 SCHED_FEAT(LB_WAKEUP_UPDATE, 1)
 SCHED_FEAT(ASYM_EFF_LOAD, 1)
 SCHED_FEAT(WAKEUP_OVERLAP, 0)
-SCHED_FEAT(LAST_BUDDY, 1)
+SCHED_FEAT(LAST_BUDDY, 0)
+SCHED_FEAT(NEXT_BUDDY, 0)
 SCHED_FEAT(OWNER_SPIN, 1)
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 3c44b56..addfe2d 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -317,8 +317,6 @@ static int worker_thread(void *__cwq)
 	if (cwq->wq->freezeable)
 		set_freezable();
 
-	set_user_nice(current, -5);
-
 	for (;;) {
 		prepare_to_wait(&cwq->more_work, &wait, TASK_INTERRUPTIBLE);
 		if (!freezing(current) &&

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

On Wed, 2009-09-09 at 10:52 +0200, Mike Galbraith wrote:
> @@ -1502,7 +1502,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
>          */
>         if (sched_feat(LAST_BUDDY) && likely(se->on_rq && curr != rq->idle))
>                 set_last_buddy(se);
> -       set_next_buddy(pse);
> +       if (sched_feat(NEXT_BUDDY))
> +               set_next_buddy(pse);
>  
>         /*
>          * We can come here with TIF_NEED_RESCHED already set from new task

You might want to test stuff like sysbench again, iirc we went on a
cache-trashing rampage without buddies.

Our goal is not to excel at any one load but to not suck at any one
load.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Wed, 2009-09-09 at 11:02 +0200, Peter Zijlstra wrote:
> On Wed, 2009-09-09 at 10:52 +0200, Mike Galbraith wrote:
> > @@ -1502,7 +1502,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
> >          */
> >         if (sched_feat(LAST_BUDDY) && likely(se->on_rq && curr != rq->idle))
> >                 set_last_buddy(se);
> > -       set_next_buddy(pse);
> > +       if (sched_feat(NEXT_BUDDY))
> > +               set_next_buddy(pse);
> >  
> >         /*
> >          * We can come here with TIF_NEED_RESCHED already set from new task
> 
> You might want to test stuff like sysbench again, iirc we went on a
> cache-trashing rampage without buddies.
> 
> Our goal is not to excel at any one load but to not suck at any one
> load.

Oh absolutely.  I wouldn't want buddies disabled by default, I only
added the buddy knob to test effects on fork/exec.

I only posted to patch to give Jens something canned to try out.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 11:52 AM, Mike Galbraith wrote:
> On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
>> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>>
>>> On Tue, Sep 08 2009, Peter Zijlstra wrote:
>>>> On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
>>>>> And here's a newer version.
>>>>
>>>> I tinkered a bit with your proglet and finally found the
>>>> problem.
>>>>
>>>> You used a single pipe per child, this means the loop in
>>>> run_child() would consume what it just wrote out until it got
>>>> force preempted by the parent which would also get woken.
>>>>
>>>> This results in the child spinning a while (its full quota) and
>>>> only reporting the last timestamp to the parent.
>>>
>>> Oh doh, that's not well thought out. Well it was a quick hack :-)
>>> Thanks for the fixup, now it's at least usable to some degree.
>>
>> What kind of latencies does it report on your box?
>>
>> Our vanilla scheduler default latency targets are:
>>
>>    single-core: 20 msecs
>>      dual-core: 40 msecs
>>      quad-core: 60 msecs
>>      opto-core: 80 msecs
>>
>> You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via
>> /proc/sys/kernel/sched_latency_ns:
>>
>>     echo 10000000>  /proc/sys/kernel/sched_latency_ns
>
> He would also need to lower min_granularity, otherwise, it'd be larger
> than the whole latency target.

Thank you for mentioning min_granularity.  After:

   echo 10000000 > /proc/sys/kernel/sched_latency_ns
   echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns

I can clearly see an improvement: animations that are supposed to be 
fluid "skip" much less now, and in one occasion (simply moving the video 
window around) have been eliminated completely.  However, there seems to 
be a side effect from having CONFIG_SCHED_DEBUG enabled; things seem to 
be generally a tad more "jerky" with that option enabled, even when not 
even touching the latency and granularity defaults.

I'll try the patch you posted and see if this further improves things.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:

> Thank you for mentioning min_granularity.  After:
> 
>    echo 10000000 > /proc/sys/kernel/sched_latency_ns
>    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns

You might also want to do:

     echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns

That affects when a newly woken task will preempt an already running
task.

> I can clearly see an improvement: animations that are supposed to be 
> fluid "skip" much less now, and in one occasion (simply moving the video 
> window around) have been eliminated completely.  However, there seems to 
> be a side effect from having CONFIG_SCHED_DEBUG enabled; things seem to 
> be generally a tad more "jerky" with that option enabled, even when not 
> even touching the latency and granularity defaults.

There's more code in the scheduler with that enabled but unless you've
got a terrible high ctx rate that really shouldn't affect things.

Anyway, you can always poke at these numbers in the code, and like Mike
did, kill sched_init_granularity().

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 12:17 PM, Peter Zijlstra wrote:
> On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
>
>> Thank you for mentioning min_granularity.  After:
>>
>>     echo 10000000>  /proc/sys/kernel/sched_latency_ns
>>     echo 2000000>  /proc/sys/kernel/sched_min_granularity_ns
>
> You might also want to do:
>
>       echo 2000000>  /proc/sys/kernel/sched_wakeup_granularity_ns
>
> That affects when a newly woken task will preempt an already running
> task.

Lowering wakeup_granularity seems to make things worse in an interesting 
way:

With low wakeup_granularity, the video itself will start skipping if I 
move the window around.  However, the window manager's effect of moving 
a window around is smooth.

With high wakeup_granularity, the video itself will not skip while 
moving the window around.  But this time, the window manager's effect of 
the window move is skippy.

(I should point out that only with the BFS-patched kernel can I have a 
smooth video *and* a smooth window-moving effect at the same time.) 
Mainline seems to prioritize one of the two according to whether 
wakeup_granularity is raised or lowered.  However, I have not tested 
Mike's patch yet (but will do so ASAP.)

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 12:40 PM, Nikos Chantziaras wrote:
> On 09/09/2009 12:17 PM, Peter Zijlstra wrote:
>> On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
>>
>>> Thank you for mentioning min_granularity. After:
>>>
>>> echo 10000000> /proc/sys/kernel/sched_latency_ns
>>> echo 2000000> /proc/sys/kernel/sched_min_granularity_ns
>>
>> You might also want to do:
>>
>> echo 2000000> /proc/sys/kernel/sched_wakeup_granularity_ns
>>
>> That affects when a newly woken task will preempt an already running
>> task.
>
> Lowering wakeup_granularity seems to make things worse in an interesting
> way:
>
> With low wakeup_granularity, the video itself will start skipping if I
> move the window around. However, the window manager's effect of moving a
> window around is smooth.
>
> With high wakeup_granularity, the video itself will not skip while
> moving the window around. But this time, the window manager's effect of
> the window move is skippy.
>
> (I should point out that only with the BFS-patched kernel can I have a
> smooth video *and* a smooth window-moving effect at the same time.)
> Mainline seems to prioritize one of the two according to whether
> wakeup_granularity is raised or lowered. However, I have not tested
> Mike's patch yet (but will do so ASAP.)

I've tested Mike's patch and it achieves the same effect as raising 
sched_min_granularity.

To round it up:

By testing various values for sched_latency_ns, sched_min_granularity_ns 
and sched_wakeup_granularity_ns, I can achieve three results:

   1. Fluid animations for the foreground app, skippy ones for
      the rest (video plays nicely, rest of the desktop lags.)

   2. Fluid animations for the background apps, a skippy one for
      the one in the foreground (dekstop behaves nicely, video lags.)

   3. Equally skippy/jerky behavior for all of them.

Unfortunately, a "4. Equally fluid behavior for all of them" cannot be 
achieved with mainline, unless I missed some other tweak.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 1239 bytes --]

Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > Thank you for mentioning min_granularity.  After:
> >
> >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> 
> You might also want to do:
> 
>      echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
> 
> That affects when a newly woken task will preempt an already running
> task.

Heh that scheduler thing again... and unfortunately Col appearing to feel 
hurt while I am think that Ingo is honest on his offer on collaboration...

While it makes fun playing with that numbers and indeed experiencing 
subjectively a more fluid deskopt how about just a

echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload

Or to say it in other words: The Linux kernel should not require me to 
fine-tune three or more values to have the scheduler act in a way that 
matches my workload.

I am willing to test stuff on my work thinkpad and my Amarok thinkpad in 
order to help improving with that.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Martin Steigerwald <Martin@lichtvoll.de> wrote:

> Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> > On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > > Thank you for mentioning min_granularity.  After:
> > >
> > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> > 
> > You might also want to do:
> > 
> >      echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
> > 
> > That affects when a newly woken task will preempt an already running
> > task.
> 
> Heh that scheduler thing again... and unfortunately Col appearing 
> to feel hurt while I am think that Ingo is honest on his offer on 
> collaboration...
> 
> While it makes fun playing with that numbers and indeed 
> experiencing subjectively a more fluid deskopt how about just a
> 
> echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload

No need to do that, that's supposed to be the default :-) The knobs 
are really just there to help us make it even more so - i.e. you 
dont need to tune them. But it really relies on people helping us 
out and tell us which combinations work best ...

> Or to say it in other words: The Linux kernel should not require 
> me to fine-tune three or more values to have the scheduler act in 
> a way that matches my workload.
> 
> I am willing to test stuff on my work thinkpad and my Amarok 
> thinkpad in order to help improving with that.

It would be great if you could check latest -tip:

  http://people.redhat.com/mingo/tip.git/README

and compare it to vanilla .31?

Also, could you outline the interactivity problems/complaints you 
have?

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Martin Steigerwald]

Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> * Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> > > On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > > > Thank you for mentioning min_granularity.  After:
> > > >
> > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> > >
> > > You might also want to do:
> > >
> > >      echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
> > >
> > > That affects when a newly woken task will preempt an already
> > > running task.
> >
> > Heh that scheduler thing again... and unfortunately Col appearing
> > to feel hurt while I am think that Ingo is honest on his offer on
> > collaboration...
> >
> > While it makes fun playing with that numbers and indeed
> > experiencing subjectively a more fluid deskopt how about just a
> >
> > echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload
> 
> No need to do that, that's supposed to be the default :-) The knobs
> are really just there to help us make it even more so - i.e. you
> dont need to tune them. But it really relies on people helping us
> out and tell us which combinations work best ...

Well currently I have:

shambhala:/proc/sys/kernel> grep "" sched_latency_ns 
sched_min_granularity_ns sched_wakeup_granularity_ns
sched_latency_ns:100000
sched_min_granularity_ns:200000
sched_wakeup_granularity_ns:0

And this give me *a completely different* desktop experience.

I am using KDE 4.3.1 on a mixture of Debian Squeeze/Sid/Experimental, with 
compositing. And now when I flip desktops or open a window I can *actually 
see* the animation. Before I jusooooooooooooooooooooot saw two to five 
steps of the animation, 
now its really a lot more fluid. 

perceived _latency--! Well its like 
oooooooooooooooooooooooooooooooooooooooooooooooooooooooopening the eyes 
again cause I tended 
to take the jerky behavior as normal and possibly related to having KDE 
4.3.1 with compositing enabled on a ThinkPad T42 with 
ooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo
01:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV350 
[Mobility Radeon 9600 M10] [1002:4e50]

which I consider to be low end for that workload. But then why actually? 
Next to me is a Sam440ep with PPC 440 667 MHz and and even older Radeon M9 
with AmigaOS 4.1 and some simple transparency effects with compositing. And 
well this combo does feel like it wheel spins cause the hardware is 
actually to fast 
foooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo

> 
> > Or to say it in other words: The Linux kernel should not require
> > me to fine-tune three or more values to have the scheduler act in
> > a way that matches my workload.
> >
> > I am willing to test stuff on my work thinkpad and my Amarok
> > thinkpad in order to help improving with that.
> 
> It would be great if you could check latest -tip:
> 
>   http://people.redhat.com/mingo/tip.git/README
> 
> and compare it to vanilla .31?
> 
> Also, could you outline the interactivity problems/complaints you
> have?
> 
> 	Ingo
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel"
>  in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Martin Steigerwald <Martin@lichtvoll.de> wrote:

> Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> > * Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > > Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> > > > On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > > > > Thank you for mentioning min_granularity.  After:
> > > > >
> > > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > > >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> > > >
> > > > You might also want to do:
> > > >
> > > >      echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
> > > >
> > > > That affects when a newly woken task will preempt an already
> > > > running task.
> > >
> > > Heh that scheduler thing again... and unfortunately Col appearing
> > > to feel hurt while I am think that Ingo is honest on his offer on
> > > collaboration...
> > >
> > > While it makes fun playing with that numbers and indeed
> > > experiencing subjectively a more fluid deskopt how about just a
> > >
> > > echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload
> > 
> > No need to do that, that's supposed to be the default :-) The knobs
> > are really just there to help us make it even more so - i.e. you
> > dont need to tune them. But it really relies on people helping us
> > out and tell us which combinations work best ...
> 
> Well currently I have:
> 
> shambhala:/proc/sys/kernel> grep "" sched_latency_ns 
> sched_min_granularity_ns sched_wakeup_granularity_ns
> sched_latency_ns:100000
> sched_min_granularity_ns:200000
> sched_wakeup_granularity_ns:0
> 
> And this give me *a completely different* desktop experience.

what is /debug/sched_features - is NO_NEW_FAIR_SLEEPERS set? If not 
set yet then try it:

  echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features

that too might make things more fluid.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 2332 bytes --]

Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> * Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> > > * Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > > > Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> > > > > On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > > > > > Thank you for mentioning min_granularity.  After:
> > > > > >
> > > > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > > > >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> > > > >
> > > > > You might also want to do:
> > > > >
> > > > >      echo 2000000 >
> > > > > /proc/sys/kernel/sched_wakeup_granularity_ns
> > > > >
> > > > > That affects when a newly woken task will preempt an already
> > > > > running task.
> > > >
> > > > Heh that scheduler thing again... and unfortunately Col appearing
> > > > to feel hurt while I am think that Ingo is honest on his offer on
> > > > collaboration...
> > > >
> > > > While it makes fun playing with that numbers and indeed
> > > > experiencing subjectively a more fluid deskopt how about just a
> > > >
> > > > echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload
> > >
> > > No need to do that, that's supposed to be the default :-) The knobs
> > > are really just there to help us make it even more so - i.e. you
> > > dont need to tune them. But it really relies on people helping us
> > > out and tell us which combinations work best ...
> >
> > Well currently I have:
> >
> > shambhala:/proc/sys/kernel> grep "" sched_latency_ns
> > sched_min_granularity_ns sched_wakeup_granularity_ns
> > sched_latency_ns:100000
> > sched_min_granularity_ns:200000
> > sched_wakeup_granularity_ns:0
> >
> > And this give me *a completely different* desktop experience.
> 
> what is /debug/sched_features - is NO_NEW_FAIR_SLEEPERS set? If not
> set yet then try it:
> 
>   echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features
> 
> that too might make things more fluid.

Hmmm, need to mount that first. But not today, cause I have to dig out on 
how to do it. Have to pack some things for tomorrow. And then sleep time.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mat]

Martin Steigerwald <Martin <at> lichtvoll.de> writes:

> 
> Am Donnerstag 10 September 2009 schrieb Ingo Molnar:

[snip]

> > what is /debug/sched_features - is NO_NEW_FAIR_SLEEPERS set? If not
> > set yet then try it:
> > 
> >   echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features
> > 
> > that too might make things more fluid.


Hi Martin,

it made an tremendous difference which still has to be tested out :)

Hi Ingo,

which adverse effect could 

cat /proc/sys/kernel/sched_wakeup_granularity_ns 
0

have on the throughput ?


Concerning that "NO_NEW_FAIR_SLEEPERS" switch - isn't it as easy as to 

do the following ? (I'm not sure if there's supposed to be another debug)

echo NO_NEW_FAIR_SLEEPERS > /sys/kernel/debug/sched_features

which after the change says:

cat /sys/kernel/debug/sched_features 
NO_NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT
START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK NO_DOUBLE_TICK
ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD NO_WAKEUP_OVERLAP LAST_BUDDY
OWNER_SPIN

I hope that's the correct switch ^^

Greetings and please keep on improving the scheduler (especially with regards to
the desktop crowd)

Regards

Mat

Re: BFS vs. mainline scheduler benchmarks and measurements

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 1706 bytes --]

Am Freitag 11 September 2009 schrieb Mat:
> Martin Steigerwald <Martin <at> lichtvoll.de> writes:
> > Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> 
> [snip]
> 
> > > what is /debug/sched_features - is NO_NEW_FAIR_SLEEPERS set? If not
> > > set yet then try it:
> > >
> > >   echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features
> > >
> > > that too might make things more fluid.
> 
> Hi Martin,

Hi Mat,

> it made an tremendous difference which still has to be tested out :)

[...]

> Concerning that "NO_NEW_FAIR_SLEEPERS" switch - isn't it as easy as to
> 
> do the following ? (I'm not sure if there's supposed to be another
>  debug)
> 
> echo NO_NEW_FAIR_SLEEPERS > /sys/kernel/debug/sched_features
> 
> which after the change says:
> 
> cat /sys/kernel/debug/sched_features
> NO_NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT
> START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK
>  NO_DOUBLE_TICK ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD
>  NO_WAKEUP_OVERLAP LAST_BUDDY OWNER_SPIN
> 
> I hope that's the correct switch ^^

Thanks. Appears to work here nicely ;-). I thought this might be a debug 
fs that I need to mount separately, but its already there here. I will see 
how it works out.

I wondered whethere it might be a good idea to have a

echo default >  /sys/kernel/kernel-tuning-knob

that will reset it to the compiled in factory defaults. Would be a nice 
way to go back to safe settings again once you got carried away to far 
with trying those tuning knobs.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Wed, Sep 09 2009, Mike Galbraith wrote:
> On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
> > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > 
> > > On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > > > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > > > And here's a newer version.
> > > > 
> > > > I tinkered a bit with your proglet and finally found the 
> > > > problem.
> > > > 
> > > > You used a single pipe per child, this means the loop in 
> > > > run_child() would consume what it just wrote out until it got 
> > > > force preempted by the parent which would also get woken.
> > > > 
> > > > This results in the child spinning a while (its full quota) and 
> > > > only reporting the last timestamp to the parent.
> > > 
> > > Oh doh, that's not well thought out. Well it was a quick hack :-) 
> > > Thanks for the fixup, now it's at least usable to some degree.
> > 
> > What kind of latencies does it report on your box?
> > 
> > Our vanilla scheduler default latency targets are:
> > 
> >   single-core: 20 msecs
> >     dual-core: 40 msecs
> >     quad-core: 60 msecs
> >     opto-core: 80 msecs
> > 
> > You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
> > /proc/sys/kernel/sched_latency_ns:
> > 
> >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> 
> He would also need to lower min_granularity, otherwise, it'd be larger
> than the whole latency target.
> 
> I'm testing right now, and one thing that is definitely a problem is the
> amount of sleeper fairness we're giving.  A full latency is just too
> much short term fairness in my testing.  While sleepers are catching up,
> hogs languish.  That's the biggest issue going on.
> 
> I've also been doing some timings of make -j4 (looking at idle time),
> and find that child_runs_first is mildly detrimental to fork/exec load,
> as are buddies.
> 
> I'm running with the below at the moment.  (the kthread/workqueue thing
> is just because I don't see any reason for it to exist, so consider it
> to be a waste of perfectly good math;)

Using latt, it seems better than -rc9. The below are entries logged
while running make -j128 on a 64 thread box. I did two runs on each, and
latt is using 8 clients.

-rc9
        Max                23772 usec
        Avg                 1129 usec
        Stdev               4328 usec
        Stdev mean           117 usec

        Max                32709 usec
        Avg                 1467 usec
        Stdev               5095 usec
        Stdev mean           136 usec

-rc9 + patch

        Max                11561 usec
        Avg                 1532 usec
        Stdev               1994 usec
        Stdev mean            48 usec

        Max                 9590 usec
        Avg                 1550 usec
        Stdev               2051 usec
        Stdev mean            50 usec

max latency is way down, and much smaller variation as well.


-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Wed, Sep 09 2009, Jens Axboe wrote:
> On Wed, Sep 09 2009, Mike Galbraith wrote:
> > On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
> > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > > > > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > > > > And here's a newer version.
> > > > > 
> > > > > I tinkered a bit with your proglet and finally found the 
> > > > > problem.
> > > > > 
> > > > > You used a single pipe per child, this means the loop in 
> > > > > run_child() would consume what it just wrote out until it got 
> > > > > force preempted by the parent which would also get woken.
> > > > > 
> > > > > This results in the child spinning a while (its full quota) and 
> > > > > only reporting the last timestamp to the parent.
> > > > 
> > > > Oh doh, that's not well thought out. Well it was a quick hack :-) 
> > > > Thanks for the fixup, now it's at least usable to some degree.
> > > 
> > > What kind of latencies does it report on your box?
> > > 
> > > Our vanilla scheduler default latency targets are:
> > > 
> > >   single-core: 20 msecs
> > >     dual-core: 40 msecs
> > >     quad-core: 60 msecs
> > >     opto-core: 80 msecs
> > > 
> > > You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
> > > /proc/sys/kernel/sched_latency_ns:
> > > 
> > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > 
> > He would also need to lower min_granularity, otherwise, it'd be larger
> > than the whole latency target.
> > 
> > I'm testing right now, and one thing that is definitely a problem is the
> > amount of sleeper fairness we're giving.  A full latency is just too
> > much short term fairness in my testing.  While sleepers are catching up,
> > hogs languish.  That's the biggest issue going on.
> > 
> > I've also been doing some timings of make -j4 (looking at idle time),
> > and find that child_runs_first is mildly detrimental to fork/exec load,
> > as are buddies.
> > 
> > I'm running with the below at the moment.  (the kthread/workqueue thing
> > is just because I don't see any reason for it to exist, so consider it
> > to be a waste of perfectly good math;)
> 
> Using latt, it seems better than -rc9. The below are entries logged
> while running make -j128 on a 64 thread box. I did two runs on each, and
> latt is using 8 clients.
> 
> -rc9
>         Max                23772 usec
>         Avg                 1129 usec
>         Stdev               4328 usec
>         Stdev mean           117 usec
> 
>         Max                32709 usec
>         Avg                 1467 usec
>         Stdev               5095 usec
>         Stdev mean           136 usec
> 
> -rc9 + patch
> 
>         Max                11561 usec
>         Avg                 1532 usec
>         Stdev               1994 usec
>         Stdev mean            48 usec
> 
>         Max                 9590 usec
>         Avg                 1550 usec
>         Stdev               2051 usec
>         Stdev mean            50 usec
> 
> max latency is way down, and much smaller variation as well.

Things are much better with this patch on the notebook! I cannot compare
with BFS as that still doesn't run anywhere I want it to run, but it's
way better than -rc9-git stock. latt numbers on the notebook have 1/3
the max latency, average is lower, and stddev is much smaller too.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Wed, Sep 09 2009, Jens Axboe wrote:
> On Wed, Sep 09 2009, Jens Axboe wrote:
> > On Wed, Sep 09 2009, Mike Galbraith wrote:
> > > On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
> > > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > > 
> > > > > On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > > > > > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > > > > > And here's a newer version.
> > > > > > 
> > > > > > I tinkered a bit with your proglet and finally found the 
> > > > > > problem.
> > > > > > 
> > > > > > You used a single pipe per child, this means the loop in 
> > > > > > run_child() would consume what it just wrote out until it got 
> > > > > > force preempted by the parent which would also get woken.
> > > > > > 
> > > > > > This results in the child spinning a while (its full quota) and 
> > > > > > only reporting the last timestamp to the parent.
> > > > > 
> > > > > Oh doh, that's not well thought out. Well it was a quick hack :-) 
> > > > > Thanks for the fixup, now it's at least usable to some degree.
> > > > 
> > > > What kind of latencies does it report on your box?
> > > > 
> > > > Our vanilla scheduler default latency targets are:
> > > > 
> > > >   single-core: 20 msecs
> > > >     dual-core: 40 msecs
> > > >     quad-core: 60 msecs
> > > >     opto-core: 80 msecs
> > > > 
> > > > You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
> > > > /proc/sys/kernel/sched_latency_ns:
> > > > 
> > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > 
> > > He would also need to lower min_granularity, otherwise, it'd be larger
> > > than the whole latency target.
> > > 
> > > I'm testing right now, and one thing that is definitely a problem is the
> > > amount of sleeper fairness we're giving.  A full latency is just too
> > > much short term fairness in my testing.  While sleepers are catching up,
> > > hogs languish.  That's the biggest issue going on.
> > > 
> > > I've also been doing some timings of make -j4 (looking at idle time),
> > > and find that child_runs_first is mildly detrimental to fork/exec load,
> > > as are buddies.
> > > 
> > > I'm running with the below at the moment.  (the kthread/workqueue thing
> > > is just because I don't see any reason for it to exist, so consider it
> > > to be a waste of perfectly good math;)
> > 
> > Using latt, it seems better than -rc9. The below are entries logged
> > while running make -j128 on a 64 thread box. I did two runs on each, and
> > latt is using 8 clients.
> > 
> > -rc9
> >         Max                23772 usec
> >         Avg                 1129 usec
> >         Stdev               4328 usec
> >         Stdev mean           117 usec
> > 
> >         Max                32709 usec
> >         Avg                 1467 usec
> >         Stdev               5095 usec
> >         Stdev mean           136 usec
> > 
> > -rc9 + patch
> > 
> >         Max                11561 usec
> >         Avg                 1532 usec
> >         Stdev               1994 usec
> >         Stdev mean            48 usec
> > 
> >         Max                 9590 usec
> >         Avg                 1550 usec
> >         Stdev               2051 usec
> >         Stdev mean            50 usec
> > 
> > max latency is way down, and much smaller variation as well.
> 
> Things are much better with this patch on the notebook! I cannot compare
> with BFS as that still doesn't run anywhere I want it to run, but it's
> way better than -rc9-git stock. latt numbers on the notebook have 1/3
> the max latency, average is lower, and stddev is much smaller too.

BFS210 runs on the laptop (dual core intel core duo). With make -j4
running, I clock the following latt -c8 'sleep 10' latencies:

-rc9

        Max                17895 usec
        Avg                 8028 usec
        Stdev               5948 usec
        Stdev mean           405 usec

        Max                17896 usec
        Avg                 4951 usec
        Stdev               6278 usec
        Stdev mean           427 usec

        Max                17885 usec
        Avg                 5526 usec
        Stdev               6819 usec
        Stdev mean           464 usec

-rc9 + mike

        Max                 6061 usec
        Avg                 3797 usec
        Stdev               1726 usec
        Stdev mean           117 usec

        Max                 5122 usec
        Avg                 3958 usec
        Stdev               1697 usec
        Stdev mean           115 usec

        Max                 6691 usec
        Avg                 2130 usec
        Stdev               2165 usec
        Stdev mean           147 usec

-rc9 + bfs210

        Max                   92 usec
        Avg                   27 usec
        Stdev                 19 usec
        Stdev mean             1 usec

        Max                   80 usec
        Avg                   23 usec
        Stdev                 15 usec
        Stdev mean             1 usec

        Max                   97 usec
        Avg                   27 usec
        Stdev                 21 usec
        Stdev mean             1 usec

One thing I also noticed is that when I have logged in, I run xmodmap
manually to load some keymappings (I always tell myself to add this to
the log in scripts, but I suspend/resume this laptop for weeks at the
time and forget before the next boot). With the stock kernel, xmodmap
will halt X updates and take forever to run. With BFS, it returned
instantly. As I would expect.

So the BFS design may be lacking in the scalability end (which is
obviously true, if you look at the code), but I can understand the
appeal of the scheduler for "normal" desktop people.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 5521 bytes --]


* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Wed, Sep 09 2009, Jens Axboe wrote:
> > On Wed, Sep 09 2009, Jens Axboe wrote:
> > > On Wed, Sep 09 2009, Mike Galbraith wrote:
> > > > On Wed, 2009-09-09 at 08:13 +0200, Ingo Molnar wrote:
> > > > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > > > 
> > > > > > On Tue, Sep 08 2009, Peter Zijlstra wrote:
> > > > > > > On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
> > > > > > > > And here's a newer version.
> > > > > > > 
> > > > > > > I tinkered a bit with your proglet and finally found the 
> > > > > > > problem.
> > > > > > > 
> > > > > > > You used a single pipe per child, this means the loop in 
> > > > > > > run_child() would consume what it just wrote out until it got 
> > > > > > > force preempted by the parent which would also get woken.
> > > > > > > 
> > > > > > > This results in the child spinning a while (its full quota) and 
> > > > > > > only reporting the last timestamp to the parent.
> > > > > > 
> > > > > > Oh doh, that's not well thought out. Well it was a quick hack :-) 
> > > > > > Thanks for the fixup, now it's at least usable to some degree.
> > > > > 
> > > > > What kind of latencies does it report on your box?
> > > > > 
> > > > > Our vanilla scheduler default latency targets are:
> > > > > 
> > > > >   single-core: 20 msecs
> > > > >     dual-core: 40 msecs
> > > > >     quad-core: 60 msecs
> > > > >     opto-core: 80 msecs
> > > > > 
> > > > > You can enable CONFIG_SCHED_DEBUG=y and set it directly as well via 
> > > > > /proc/sys/kernel/sched_latency_ns:
> > > > > 
> > > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > > 
> > > > He would also need to lower min_granularity, otherwise, it'd be larger
> > > > than the whole latency target.
> > > > 
> > > > I'm testing right now, and one thing that is definitely a problem is the
> > > > amount of sleeper fairness we're giving.  A full latency is just too
> > > > much short term fairness in my testing.  While sleepers are catching up,
> > > > hogs languish.  That's the biggest issue going on.
> > > > 
> > > > I've also been doing some timings of make -j4 (looking at idle time),
> > > > and find that child_runs_first is mildly detrimental to fork/exec load,
> > > > as are buddies.
> > > > 
> > > > I'm running with the below at the moment.  (the kthread/workqueue thing
> > > > is just because I don't see any reason for it to exist, so consider it
> > > > to be a waste of perfectly good math;)
> > > 
> > > Using latt, it seems better than -rc9. The below are entries logged
> > > while running make -j128 on a 64 thread box. I did two runs on each, and
> > > latt is using 8 clients.
> > > 
> > > -rc9
> > >         Max                23772 usec
> > >         Avg                 1129 usec
> > >         Stdev               4328 usec
> > >         Stdev mean           117 usec
> > > 
> > >         Max                32709 usec
> > >         Avg                 1467 usec
> > >         Stdev               5095 usec
> > >         Stdev mean           136 usec
> > > 
> > > -rc9 + patch
> > > 
> > >         Max                11561 usec
> > >         Avg                 1532 usec
> > >         Stdev               1994 usec
> > >         Stdev mean            48 usec
> > > 
> > >         Max                 9590 usec
> > >         Avg                 1550 usec
> > >         Stdev               2051 usec
> > >         Stdev mean            50 usec
> > > 
> > > max latency is way down, and much smaller variation as well.
> > 
> > Things are much better with this patch on the notebook! I cannot compare
> > with BFS as that still doesn't run anywhere I want it to run, but it's
> > way better than -rc9-git stock. latt numbers on the notebook have 1/3
> > the max latency, average is lower, and stddev is much smaller too.
> 
> BFS210 runs on the laptop (dual core intel core duo). With make -j4
> running, I clock the following latt -c8 'sleep 10' latencies:
> 
> -rc9
> 
>         Max                17895 usec
>         Avg                 8028 usec
>         Stdev               5948 usec
>         Stdev mean           405 usec
> 
>         Max                17896 usec
>         Avg                 4951 usec
>         Stdev               6278 usec
>         Stdev mean           427 usec
> 
>         Max                17885 usec
>         Avg                 5526 usec
>         Stdev               6819 usec
>         Stdev mean           464 usec
> 
> -rc9 + mike
> 
>         Max                 6061 usec
>         Avg                 3797 usec
>         Stdev               1726 usec
>         Stdev mean           117 usec
> 
>         Max                 5122 usec
>         Avg                 3958 usec
>         Stdev               1697 usec
>         Stdev mean           115 usec
> 
>         Max                 6691 usec
>         Avg                 2130 usec
>         Stdev               2165 usec
>         Stdev mean           147 usec

At least in my tests these latencies were mainly due to a bug in 
latt.c - i've attached the fixed version.

The other reason was wakeup batching. If you do this:

   echo 0 > /proc/sys/kernel/sched_wakeup_granularity_ns 

... then you can switch on insta-wakeups on -tip too.

With a dual-core box and a make -j4 background job running, on 
latest -tip i get the following latencies:

 $ ./latt -c8 sleep 30
 Entries: 656 (clients=8)

 Averages:
 ------------------------------ 
 	Max	      158 usec 
	Avg	       12 usec
	Stdev	       10 usec

Thanks,

	Ingo

[-- Attachment #2: latt.c --]
[-- Type: text/plain, Size: 9067 bytes --]

/*
 * Simple latency tester that combines multiple processes.
 *
 * Compile: gcc -Wall -O2 -D_GNU_SOURCE -lrt -lm -o latt latt.c
 *
 * Run with: latt -c8 'program --args'
 *
 * Options:
 *
 *	-cX	Use X number of clients
 *	-fX	Use X msec as the minimum sleep time for the parent
 *	-tX	Use X msec as the maximum sleep time for the parent
 *	-v	Print all delays as they are logged
 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <getopt.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <time.h>
#include <math.h>
#include <poll.h>
#include <pthread.h>


/*
 * In msecs
 */
static unsigned int min_delay = 100;
static unsigned int max_delay = 500;
static unsigned int clients = 1;
static unsigned int verbose;

#define MAX_CLIENTS		512

struct stats
{
	double n, mean, M2, max;
	int max_pid;
};

static void update_stats(struct stats *stats, unsigned long long val)
{
	double delta, x = val;

	stats->n++;
	delta = x - stats->mean;
	stats->mean += delta / stats->n;
	stats->M2 += delta*(x - stats->mean);

	if (stats->max < x)
		stats->max = x;
}

static unsigned long nr_stats(struct stats *stats)
{
	return stats->n;
}

static double max_stats(struct stats *stats)
{
	return stats->max;
}

static double avg_stats(struct stats *stats)
{
	return stats->mean;
}

/*
 * http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
 *
 *       (\Sum n_i^2) - ((\Sum n_i)^2)/n
 * s^2 = -------------------------------
 *                  n - 1
 *
 * http://en.wikipedia.org/wiki/Stddev
 */
static double stddev_stats(struct stats *stats)
{
	double variance = stats->M2 / (stats->n - 1);

	return sqrt(variance);
}

struct stats delay_stats;

static int pipes[MAX_CLIENTS*2][2];

static pid_t app_pid;

#define CLOCKSOURCE		CLOCK_MONOTONIC

struct sem {
	pthread_mutex_t lock;
	pthread_cond_t cond;
	int value;
	int waiters;
};

static void init_sem(struct sem *sem)
{
	pthread_mutexattr_t attr;
	pthread_condattr_t cond;

	pthread_mutexattr_init(&attr);
	pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
	pthread_condattr_init(&cond);
	pthread_condattr_setpshared(&cond, PTHREAD_PROCESS_SHARED);
	pthread_cond_init(&sem->cond, &cond);
	pthread_mutex_init(&sem->lock, &attr);

	sem->value = 0;
	sem->waiters = 0;
}

static void sem_down(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);

	while (!sem->value) {
		sem->waiters++;
		pthread_cond_wait(&sem->cond, &sem->lock);
		sem->waiters--;
	}

	sem->value--;
	pthread_mutex_unlock(&sem->lock);
}

static void sem_up(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);
	if (!sem->value && sem->waiters)
		pthread_cond_signal(&sem->cond);
	sem->value++;
	pthread_mutex_unlock(&sem->lock);
}

static int parse_options(int argc, char *argv[])
{
	struct option l_opts[] = {
		{ "min-delay", 	1, 	NULL,	'f' },
		{ "max-delay",	1,	NULL,	't' },
		{ "clients",	1,	NULL,	'c' },
		{ "verbose",	1,	NULL,	'v' }
	};
	int c, res, index = 0;

	while ((c = getopt_long(argc, argv, "f:t:c:v", l_opts, &res)) != -1) {
		index++;
		switch (c) {
			case 'f':
				min_delay = atoi(optarg);
				break;
			case 't':
				max_delay = atoi(optarg);
				break;
			case 'c':
				clients = atoi(optarg);
				if (clients > MAX_CLIENTS)
					clients = MAX_CLIENTS;
				break;
			case 'v':
				verbose = 1;
				break;
		}
	}

	return index + 1;
}

static pid_t fork_off(const char *app)
{
	pid_t pid;

	pid = fork();
	if (pid)
		return pid;

	exit(system(app));
}

static unsigned long usec_since(struct timespec *start, struct timespec *end)
{
	unsigned long long s, e;

	s = start->tv_sec * 1000000000ULL + start->tv_nsec;
	e =   end->tv_sec * 1000000000ULL +   end->tv_nsec;

	return (e - s) / 1000;
}

static void log_delay(unsigned long delay)
{
	if (verbose) {
		fprintf(stderr, "log delay %8lu usec (pid %d)\n", delay, getpid());
		fflush(stderr);
	}

	update_stats(&delay_stats, delay);
}

/*
 * Reads a timestamp (which is ignored, it's just a wakeup call), and replies
 * with the timestamp of when we saw it
 */
static void run_child(int *in, int *out, struct sem *sem)
{
	struct timespec ts;

	if (verbose) {
		fprintf(stderr, "present: %d\n", getpid());
		fflush(stderr);
	}

	sem_up(sem);

	do {
		int ret;

		ret = read(in[0], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		if (ret != sizeof(ts))
			printf("bugger3\n");

		clock_gettime(CLOCKSOURCE, &ts);

		ret = write(out[1], &ts, sizeof(ts));
		if (ret <= 0)
			break;

		if (ret != sizeof(ts))
			printf("bugger4\n");

		if (verbose) {
			fprintf(stderr, "alife: %d\n", getpid());
			fflush(stderr);
		}
	} while (1);
}

/*
 * Do a random sleep between min and max delay
 */
static void do_rand_sleep(void)
{
	unsigned int msecs;

	msecs = min_delay + ((float) max_delay * (rand() / (RAND_MAX + 1.0)));
	if (verbose) {
		fprintf(stderr, "sleeping for: %u msec\n", msecs);
		fflush(stderr);
	}
	usleep(msecs * 1000);
}

static void kill_connection(void)
{
	int i;

	for (i = 0; i < 2*clients; i++) {
		if (pipes[i][0] != -1) {
			close(pipes[i][0]);
			pipes[i][0] = -1;
		}
		if (pipes[i][1] != -1) {
			close(pipes[i][1]);
			pipes[i][1] = -1;
		}
	}
}

static int __write_ts(int i, struct timespec *ts)
{
	int fd = pipes[2*i][1];

	clock_gettime(CLOCKSOURCE, ts);

	return write(fd, ts, sizeof(*ts)) != sizeof(*ts);
}

static long __read_ts(int i, struct timespec *ts, pid_t *cpids)
{
	int fd = pipes[2*i+1][0];
	struct timespec t;

	if (read(fd, &t, sizeof(t)) != sizeof(t))
		return -1;

	log_delay(usec_since(ts, &t));
	if (verbose)
		fprintf(stderr, "got delay %ld from child %d [pid %d]\n", usec_since(ts, &t), i, cpids[i]);

	return 0;
}

static int read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts,
		   pid_t *cpids)
{
	unsigned int i;

	for (i = 0; i < clients; i++) {
		if (pfd[i].revents & (POLLERR | POLLHUP | POLLNVAL))
			return -1L;
		if (pfd[i].revents & POLLIN) {
			pfd[i].events = 0;
			if (__read_ts(i, &ts[i], cpids))
				return -1L;
			nr--;
		}
		if (!nr)
			break;
	}

	return 0;
}

static int app_has_exited(void)
{
	int ret, status;

	/*
	 * If our app has exited, stop
	 */
	ret = waitpid(app_pid, &status, WNOHANG);
	if (ret < 0) {
		perror("waitpid");
		return 1;
	} else if (ret == app_pid &&
		   (WIFSIGNALED(status) || WIFEXITED(status))) {
		return 1;
	}

	return 0;
}

/*
 * While our given app is running, send a timestamp to each client and
 * log the maximum latency for each of them to wakeup and reply
 */
static void run_parent(pid_t *cpids)
{
	struct pollfd *ipfd;
	int do_exit = 0, i;
	struct timespec *t1;

	t1 = malloc(sizeof(struct timespec) * clients);
	ipfd = malloc(sizeof(struct pollfd) * clients);

	srand(1234);

	do {
		unsigned pending_events;

		do_rand_sleep();

		if (app_has_exited())
			break;

		for (i = 0; i < clients; i++) {
			ipfd[i].fd = pipes[2*i+1][0];
			ipfd[i].events = POLLIN;
		}

		/*
		 * Write wakeup calls
		 */
		for (i = 0; i < clients; i++) {
			if (verbose) {
				fprintf(stderr, "waking: %d\n", cpids[i]);
				fflush(stderr);
			}

			if (__write_ts(i, t1+i)) {
				do_exit = 1;
				break;
			}
		}

		if (do_exit)
			break;

		/*
		 * Poll and read replies
		 */
		pending_events = clients;
		while (pending_events) {
			int evts = poll(ipfd, clients, -1);

			if (evts < 0) {
				do_exit = 1;
				break;
			} else if (!evts) {
				printf("bugger2\n");
				continue;
			}

			if (read_ts(ipfd, evts, t1, cpids)) {
				do_exit = 1;
				break;
			}

			pending_events -= evts;
		}
	} while (!do_exit);

	free(t1);
	free(ipfd);
	kill_connection();
}

static void run_test(void)
{
	struct sem *sem;
	pid_t *cpids;
	int i, status;

	sem = mmap(NULL, sizeof(*sem), PROT_READ|PROT_WRITE,
			MAP_SHARED | MAP_ANONYMOUS, 0, 0);
	if (sem == MAP_FAILED) {
		perror("mmap");
		return;
	}

	init_sem(sem);

	for (i = 0; i < 2*clients; i++) {
		if (pipe(pipes[i])) {
			perror("pipe");
			return;
		}
	}

	cpids = malloc(sizeof(pid_t) * clients);

	for (i = 0; i < clients; i++) {
		cpids[i] = fork();
		if (cpids[i]) {
			sem_down(sem);
			continue;
		}

		run_child(pipes[2*i], pipes[2*i+1], sem);
		exit(0);
	}

	run_parent(cpids);

	for (i = 0; i < clients; i++)
		kill(cpids[i], SIGQUIT);
	for (i = 0; i < clients; i++)
		waitpid(cpids[i], &status, 0);

	free(cpids);
	munmap(sem, sizeof(*sem));
}

static void handle_sigint(int sig)
{
	kill(app_pid, SIGINT);
}

int main(int argc, char *argv[])
{
	int app_offset, off;
	char app[256];

	off = 0;
	app_offset = parse_options(argc, argv);
	while (app_offset < argc) {
		if (off) {
			app[off] = ' ';
			off++;
		}
		off += sprintf(app + off, "%s", argv[app_offset]);
		app_offset++;
	}

	signal(SIGINT, handle_sigint);

	/*
	 * Start app and start logging latencies
	 */
	app_pid = fork_off(app);
	run_test();

	printf("Entries: %lu (clients=%d)\n", nr_stats(&delay_stats), clients);
	printf("\nAverages:\n");
	printf("------------------------------\n");
	printf("\tMax\t %8.0f usec\n", max_stats(&delay_stats));
	printf("\tAvg\t %8.0f usec\n", avg_stats(&delay_stats));
	printf("\tStdev\t %8.0f usec\n", stddev_stats(&delay_stats));

	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/09/2009 09:04 PM, Ingo Molnar wrote:
> [...]
> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>
>> On Wed, Sep 09 2009, Jens Axboe wrote:
>>  [...]
>> BFS210 runs on the laptop (dual core intel core duo). With make -j4
>> running, I clock the following latt -c8 'sleep 10' latencies:
>>
>> -rc9
>>
>>          Max                17895 usec
>>          Avg                 8028 usec
>>          Stdev               5948 usec
>>          Stdev mean           405 usec
>>
>>          Max                17896 usec
>>          Avg                 4951 usec
>>          Stdev               6278 usec
>>          Stdev mean           427 usec
>>
>>          Max                17885 usec
>>          Avg                 5526 usec
>>          Stdev               6819 usec
>>          Stdev mean           464 usec
>>
>> -rc9 + mike
>>
>>          Max                 6061 usec
>>          Avg                 3797 usec
>>          Stdev               1726 usec
>>          Stdev mean           117 usec
>>
>>          Max                 5122 usec
>>          Avg                 3958 usec
>>          Stdev               1697 usec
>>          Stdev mean           115 usec
>>
>>          Max                 6691 usec
>>          Avg                 2130 usec
>>          Stdev               2165 usec
>>          Stdev mean           147 usec
>
> At least in my tests these latencies were mainly due to a bug in
> latt.c - i've attached the fixed version.
>
> The other reason was wakeup batching. If you do this:
>
>     echo 0>  /proc/sys/kernel/sched_wakeup_granularity_ns
>
> ... then you can switch on insta-wakeups on -tip too.
>
> With a dual-core box and a make -j4 background job running, on
> latest -tip i get the following latencies:
>
>   $ ./latt -c8 sleep 30
>   Entries: 656 (clients=8)
>
>   Averages:
>   ------------------------------
>   	Max	      158 usec
> 	Avg	       12 usec
> 	Stdev	       10 usec

With your version of latt.c, I get these results with 2.6-tip vs 
2.6.31-rc9-bfs:


(mainline)
Averages:
------------------------------
         Max            50 usec
         Avg            12 usec
         Stdev           3 usec


(BFS)
Averages:
------------------------------
         Max           474 usec
         Avg            11 usec
         Stdev          16 usec


However, the interactivity problems still remain.  Does that mean it's 
not a latency issue?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Wed, Sep 09 2009, Nikos Chantziaras wrote:
> On 09/09/2009 09:04 PM, Ingo Molnar wrote:
>> [...]
>> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>>
>>> On Wed, Sep 09 2009, Jens Axboe wrote:
>>>  [...]
>>> BFS210 runs on the laptop (dual core intel core duo). With make -j4
>>> running, I clock the following latt -c8 'sleep 10' latencies:
>>>
>>> -rc9
>>>
>>>          Max                17895 usec
>>>          Avg                 8028 usec
>>>          Stdev               5948 usec
>>>          Stdev mean           405 usec
>>>
>>>          Max                17896 usec
>>>          Avg                 4951 usec
>>>          Stdev               6278 usec
>>>          Stdev mean           427 usec
>>>
>>>          Max                17885 usec
>>>          Avg                 5526 usec
>>>          Stdev               6819 usec
>>>          Stdev mean           464 usec
>>>
>>> -rc9 + mike
>>>
>>>          Max                 6061 usec
>>>          Avg                 3797 usec
>>>          Stdev               1726 usec
>>>          Stdev mean           117 usec
>>>
>>>          Max                 5122 usec
>>>          Avg                 3958 usec
>>>          Stdev               1697 usec
>>>          Stdev mean           115 usec
>>>
>>>          Max                 6691 usec
>>>          Avg                 2130 usec
>>>          Stdev               2165 usec
>>>          Stdev mean           147 usec
>>
>> At least in my tests these latencies were mainly due to a bug in
>> latt.c - i've attached the fixed version.
>>
>> The other reason was wakeup batching. If you do this:
>>
>>     echo 0>  /proc/sys/kernel/sched_wakeup_granularity_ns
>>
>> ... then you can switch on insta-wakeups on -tip too.
>>
>> With a dual-core box and a make -j4 background job running, on
>> latest -tip i get the following latencies:
>>
>>   $ ./latt -c8 sleep 30
>>   Entries: 656 (clients=8)
>>
>>   Averages:
>>   ------------------------------
>>   	Max	      158 usec
>> 	Avg	       12 usec
>> 	Stdev	       10 usec
>
> With your version of latt.c, I get these results with 2.6-tip vs  
> 2.6.31-rc9-bfs:
>
>
> (mainline)
> Averages:
> ------------------------------
>         Max            50 usec
>         Avg            12 usec
>         Stdev           3 usec
>
>
> (BFS)
> Averages:
> ------------------------------
>         Max           474 usec
>         Avg            11 usec
>         Stdev          16 usec
>
>
> However, the interactivity problems still remain.  Does that mean it's  
> not a latency issue?

It probably just means that latt isn't a good measure of the problem.
Which isn't really too much of a surprise.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Con Kolivas]

On Thu, 10 Sep 2009 06:50:43 Jens Axboe wrote:
> On Wed, Sep 09 2009, Nikos Chantziaras wrote:
> > On 09/09/2009 09:04 PM, Ingo Molnar wrote:
> >> [...]
> >>
> >> * Jens Axboe<jens.axboe@oracle.com>  wrote:
> >>> On Wed, Sep 09 2009, Jens Axboe wrote:
> >>>  [...]
> >>> BFS210 runs on the laptop (dual core intel core duo). With make -j4
> >>> running, I clock the following latt -c8 'sleep 10' latencies:
> >>>
> >>> -rc9
> >>>
> >>>          Max                17895 usec
> >>>          Avg                 8028 usec
> >>>          Stdev               5948 usec
> >>>          Stdev mean           405 usec
> >>>
> >>>          Max                17896 usec
> >>>          Avg                 4951 usec
> >>>          Stdev               6278 usec
> >>>          Stdev mean           427 usec
> >>>
> >>>          Max                17885 usec
> >>>          Avg                 5526 usec
> >>>          Stdev               6819 usec
> >>>          Stdev mean           464 usec
> >>>
> >>> -rc9 + mike
> >>>
> >>>          Max                 6061 usec
> >>>          Avg                 3797 usec
> >>>          Stdev               1726 usec
> >>>          Stdev mean           117 usec
> >>>
> >>>          Max                 5122 usec
> >>>          Avg                 3958 usec
> >>>          Stdev               1697 usec
> >>>          Stdev mean           115 usec
> >>>
> >>>          Max                 6691 usec
> >>>          Avg                 2130 usec
> >>>          Stdev               2165 usec
> >>>          Stdev mean           147 usec
> >>
> >> At least in my tests these latencies were mainly due to a bug in
> >> latt.c - i've attached the fixed version.
> >>
> >> The other reason was wakeup batching. If you do this:
> >>
> >>     echo 0>  /proc/sys/kernel/sched_wakeup_granularity_ns
> >>
> >> ... then you can switch on insta-wakeups on -tip too.
> >>
> >> With a dual-core box and a make -j4 background job running, on
> >> latest -tip i get the following latencies:
> >>
> >>   $ ./latt -c8 sleep 30
> >>   Entries: 656 (clients=8)
> >>
> >>   Averages:
> >>   ------------------------------
> >>   	Max	      158 usec
> >> 	Avg	       12 usec
> >> 	Stdev	       10 usec
> >
> > With your version of latt.c, I get these results with 2.6-tip vs
> > 2.6.31-rc9-bfs:
> >
> >
> > (mainline)
> > Averages:
> > ------------------------------
> >         Max            50 usec
> >         Avg            12 usec
> >         Stdev           3 usec
> >
> >
> > (BFS)
> > Averages:
> > ------------------------------
> >         Max           474 usec
> >         Avg            11 usec
> >         Stdev          16 usec
> >
> >
> > However, the interactivity problems still remain.  Does that mean it's
> > not a latency issue?
>
> It probably just means that latt isn't a good measure of the problem.
> Which isn't really too much of a surprise.

And that's a real shame because this was one of the first real good attempts 
I've seen to actually measure the difference, and I thank you for your 
efforts Jens. I believe the reason it's limited is because all you're 
measuring is time from wakeup and the test app isn't actually doing any work. 
The issue is more than just waking up as fast as possible, it's then doing 
some meaningful amount of work within a reasonable time frame as well. What 
the "meaningful amount of work" and "reasonable time frame" are, remains a 
mystery, but I guess could be added on to this testing app.

What does please me now, though, is that this message thread is finally 
concentrating on what BFS was all about. The fact that it doesn't scale is no 
mystery whatsoever. The fact that that throughput and lack of scaling was 
what was given attention was missing the point entirely. To point that out I 
used the bluntest response possible, because I know that works on lkml (does 
it not?). Unfortunately I was so blunt that I ended up writing it in another 
language; Troll. So for that, I apologise.

The unfortunate part is that BFS is still far from a working, complete state, 
yet word got out that I had "released" something, which I had not, but 
obviously there's no great distinction between putting something on a server 
for testing, and a real release with an announce.

BFS is a scheduling experiment to demonstrate what effect the cpu scheduler 
really has on the desktop and how it might be able to perform if we design 
the scheduler for that one purpose.

It pleases me immensely to see that it has already spurred on a flood of 
changes to the interactivity side of mainline development in its few days of 
existence, including some ideas that BFS uses itself. That in itself, to me, 
means it has already started to accomplish its goal, which ultimately, one 
way or another, is to improve what the CPU scheduler can do for the linux 
desktop. I can't track all the sensitive areas of the mainline kernel 
scheduler changes without getting involved more deeply than I care to so it 
would be counterproductive of me to try and hack on mainline. I much prefer 
the quieter inbox.

If people want to use BFS for their own purposes or projects, or even better 
help hack on it, that would make me happy for different reasons. I will 
continue to work on my little project -in my own time- and hope that it 
continues to drive further development of the mainline kernel in its own way. 
We need more experiments like this to question what we currently have and 
accept. Other major kernel subsystems are no exception.

Regards,
-- 
-ck

<code before rhetoric>

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

On Thu, Sep 10 2009, Con Kolivas wrote:
> > It probably just means that latt isn't a good measure of the problem.
> > Which isn't really too much of a surprise.
> 
> And that's a real shame because this was one of the first real good attempts 
> I've seen to actually measure the difference, and I thank you for your 
> efforts Jens. I believe the reason it's limited is because all you're 
> measuring is time from wakeup and the test app isn't actually doing any work. 
> The issue is more than just waking up as fast as possible, it's then doing 
> some meaningful amount of work within a reasonable time frame as well. What 
> the "meaningful amount of work" and "reasonable time frame" are, remains a 
> mystery, but I guess could be added on to this testing app.

Here's a quickie addition that adds some work to the threads. The
latency measure is now 'when did I wake up and complete my work'. The
default work is filling a buffer with pseudo random data and then
compressing it with zlib. Default is 64kb of data, can be adjusted with
-x. -x0 turns off work processing.

-- 
Jens Axboe


[-- Attachment #2: latt.c --]
[-- Type: text/x-csrc, Size: 11245 bytes --]

/*
 * Simple latency tester that combines multiple processes.
 *
 * Compile: gcc -Wall -O2 -D_GNU_SOURCE -lrt -lm -lz -o latt latt.c
 *
 * Run with: latt -c8 'program --args'
 *
 * Options:
 *
 *	-cX	Use X number of clients
 *	-sX	Use X msec as the minimum sleep time for the parent
 *	-SX	Use X msec as the maximum sleep time for the parent
 *	-xX	Use X kb as the work buffer to randomize and compress
 *	-v	Print all delays as they are logged
 *
 * (C) Jens Axboe <jens.axboe@oracle.com> 2009
 * Fixes from Peter Zijlstra <a.p.zijlstra@chello.nl> to actually make it
 * measure what it was intended to measure.
 * Fix from Ingo Molnar for poll() using 0 as timeout (should be negative).
 *
 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <getopt.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <time.h>
#include <math.h>
#include <poll.h>
#include <pthread.h>
#include <zlib.h>
#include <string.h>

/*
 * In msecs
 */
static unsigned int min_delay = 100;
static unsigned int max_delay = 500;
static unsigned int clients = 1;
static unsigned int compress_size = 64 * 1024;
static unsigned int verbose;

#define print_if_verbose(args...)		\
	do {					\
	if (verbose) {				\
		fprintf(stderr, ##args);	\
		fflush(stderr);			\
	}					\
	} while (0)

#define MAX_CLIENTS		512

static int pipes[MAX_CLIENTS*2][2];
static pid_t app_pid;

#define CLOCKSOURCE		CLOCK_MONOTONIC

struct stats
{
	double n, mean, M2, max, max_client;
} delay_stats;


static void update_stats(struct stats *stats, unsigned long long val)
{
	double delta, x = val;

	stats->n++;
	delta = x - stats->mean;
	stats->mean += delta / stats->n;
	stats->M2 += delta*(x - stats->mean);

	if (stats->max < x)
		stats->max = x;
}

static unsigned long nr_stats(struct stats *stats)
{
	return stats->n;
}

static double max_stats(struct stats *stats)
{
	return stats->max;
}

static double avg_stats(struct stats *stats)
{
	return stats->mean;
}

/*
 * http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance
 *
 *       (\Sum n_i^2) - ((\Sum n_i)^2)/n
 * s^2 = -------------------------------
 *                  n - 1
 *
 * http://en.wikipedia.org/wiki/Stddev
 */
static double stddev_stats(struct stats *stats)
{
	double variance = stats->M2 / (stats->n - 1);

	return sqrt(variance);
}

/*
 * The std dev of the mean is related to the std dev by:
 *
 *             s
 * s_mean = -------
 *          sqrt(n)
 *
 */
static double stddev_mean_stats(struct stats *stats)
{
	double variance = stats->M2 / (stats->n - 1);
	double variance_mean = variance / stats->n;

	return sqrt(variance_mean);
}

struct sem {
	pthread_mutex_t lock;
	pthread_cond_t cond;
	int value;
	int waiters;
};

static void init_sem(struct sem *sem)
{
	pthread_mutexattr_t attr;
	pthread_condattr_t cond;

	pthread_mutexattr_init(&attr);
	pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED);
	pthread_condattr_init(&cond);
	pthread_condattr_setpshared(&cond, PTHREAD_PROCESS_SHARED);
	pthread_cond_init(&sem->cond, &cond);
	pthread_mutex_init(&sem->lock, &attr);

	sem->value = 0;
	sem->waiters = 0;
}

static void sem_down(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);

	while (!sem->value) {
		sem->waiters++;
		pthread_cond_wait(&sem->cond, &sem->lock);
		sem->waiters--;
	}

	sem->value--;
	pthread_mutex_unlock(&sem->lock);
}

static void sem_up(struct sem *sem)
{
	pthread_mutex_lock(&sem->lock);
	if (!sem->value && sem->waiters)
		pthread_cond_signal(&sem->cond);
	sem->value++;
	pthread_mutex_unlock(&sem->lock);
}

static int parse_options(int argc, char *argv[])
{
	struct option l_opts[] = {
		{ "min-delay (msec)", 	1, 	NULL,	's' },
		{ "max-delay (msec)",	1,	NULL,	'S' },
		{ "clients",		1,	NULL,	'c' },
		{ "compress-size (kb)",	1,	NULL,	'x' },
		{ "verbose",		0,	NULL,	'v' },
		{ "help",		0,	NULL,	'h' },
		{ NULL,					    },
	};
	int c, res, index = 0, i;

	while ((c = getopt_long(argc, argv, "s:S:c:vhx:", l_opts, &res)) != -1) {
		index++;
		switch (c) {
			case 's':
				min_delay = atoi(optarg);
				break;
			case 'S':
				max_delay = atoi(optarg);
				break;
			case 'c':
				clients = atoi(optarg);
				if (clients > MAX_CLIENTS) {
					clients = MAX_CLIENTS;
					printf("Capped clients to %d\n", clients);
				}
				break;
			case 'v':
				verbose = 1;
				break;
			case 'h':
				for (i = 0; l_opts[i].name; i++)
					printf("-%c: %s\n", l_opts[i].val, l_opts[i].name);
				break;
			case 'x':
				compress_size = atoi(optarg);
				compress_size *= 1024;
				break;
		}
	}

	return index + 1;
}

static pid_t fork_off(const char *app)
{
	pid_t pid;

	pid = fork();
	if (pid)
		return pid;

	exit(system(app));
}

static unsigned long usec_since(struct timespec *start, struct timespec *end)
{
	unsigned long long s, e;

	s = start->tv_sec * 1000000000ULL + start->tv_nsec;
	e =   end->tv_sec * 1000000000ULL +   end->tv_nsec;

	return (e - s) / 1000;
}

static void log_delay(unsigned long delay)
{
	print_if_verbose("log delay %8lu usec (pid=%d)\n", delay, getpid());

	update_stats(&delay_stats, delay);
}

static unsigned long pseed = 1;

static int get_rand(void)
{
	pseed = pseed * 1103515245 + 12345;
	return ((unsigned) (pseed / 65536) % 32768);
}

static void fill_random_data(int *buf, unsigned int nr_ints)
{
	int i;

	for (i = 0; i < nr_ints; i++)
		buf[i] = get_rand();
}

/*
 * Generate random data and compress it with zlib
 */
static void do_work(void)
{
	unsigned long work_size = compress_size;
	z_stream stream;
	int ret, bytes;
	void *zbuf;
	int *buf;

	memset(&stream, 0, sizeof(stream));
	deflateInit(&stream, 7);
	bytes = deflateBound(&stream, work_size);

	zbuf = malloc(bytes);
	buf = malloc(work_size);
	fill_random_data(buf, work_size / sizeof(int));

	stream.next_in = (void *) buf;
	stream.avail_in = work_size;
	stream.next_out = zbuf;
	stream.avail_out = bytes;

	do {
		ret = deflate(&stream, Z_FINISH);
	} while (ret == Z_OK);

	deflateEnd(&stream);
	free(buf);
	free(zbuf);
}

/*
 * Reads a timestamp (which is ignored, it's just a wakeup call), and replies
 * with the timestamp of when we saw it
 */
static void run_child(int *in, int *out, struct sem *sem)
{
	struct timespec ts;

	print_if_verbose("present: %d\n", getpid());

	sem_up(sem);

	do {
		int ret;

		ret = read(in[0], &ts, sizeof(ts));
		if (ret <= 0)
			break;
		else if (ret != sizeof(ts))
			break;

		if (compress_size)
			do_work();

		clock_gettime(CLOCKSOURCE, &ts);

		ret = write(out[1], &ts, sizeof(ts));
		if (ret <= 0)
			break;
		else if (ret != sizeof(ts))
			break;

		print_if_verbose("alive: %d\n", getpid());
	} while (1);
}

/*
 * Do a random sleep between min and max delay
 */
static void do_rand_sleep(void)
{
	unsigned int msecs;

	msecs = min_delay + ((float) max_delay * (rand() / (RAND_MAX + 1.0)));
	print_if_verbose("sleeping for: %u msec\n", msecs);
	usleep(msecs * 1000);
}

static void kill_connection(void)
{
	int i;

	for (i = 0; i < 2*clients; i++) {
		if (pipes[i][0] != -1) {
			close(pipes[i][0]);
			pipes[i][0] = -1;
		}
		if (pipes[i][1] != -1) {
			close(pipes[i][1]);
			pipes[i][1] = -1;
		}
	}
}

static int __write_ts(int i, struct timespec *ts)
{
	int fd = pipes[2*i][1];

	clock_gettime(CLOCKSOURCE, ts);

	return write(fd, ts, sizeof(*ts)) != sizeof(*ts);
}

static long __read_ts(int i, struct timespec *ts, pid_t *cpids)
{
	int fd = pipes[2*i+1][0];
	unsigned long delay;
	struct timespec t;

	if (read(fd, &t, sizeof(t)) != sizeof(t))
		return -1;

	delay = usec_since(ts, &t);
	log_delay(delay);
	print_if_verbose("got delay %ld from child %d [pid %d]\n", delay,
				i, cpids[i]);
	return 0;
}

static int read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts,
		   pid_t *cpids)
{
	unsigned int i;

	for (i = 0; i < clients; i++) {
		if (pfd[i].revents & (POLLERR | POLLHUP | POLLNVAL))
			return -1L;
		if (pfd[i].revents & POLLIN) {
			pfd[i].events = 0;
			if (__read_ts(i, &ts[i], cpids))
				return -1L;
			nr--;
		}
		if (!nr)
			break;
	}

	return 0;
}

static int app_has_exited(void)
{
	int ret, status;

	/*
	 * If our app has exited, stop
	 */
	ret = waitpid(app_pid, &status, WNOHANG);
	if (ret < 0) {
		perror("waitpid");
		return 1;
	} else if (ret == app_pid &&
		   (WIFSIGNALED(status) || WIFEXITED(status))) {
		return 1;
	}

	return 0;
}

/*
 * While our given app is running, send a timestamp to each client and
 * log the maximum latency for each of them to wakeup and reply
 */
static void run_parent(pid_t *cpids)
{
	struct pollfd *ipfd;
	int do_exit = 0, i;
	struct timespec *t1;

	t1 = malloc(sizeof(struct timespec) * clients);
	ipfd = malloc(sizeof(struct pollfd) * clients);

	srand(1234);

	do {
		unsigned pending_events;

		do_rand_sleep();

		if (app_has_exited())
			break;

		for (i = 0; i < clients; i++) {
			ipfd[i].fd = pipes[2*i+1][0];
			ipfd[i].events = POLLIN;
		}

		/*
		 * Write wakeup calls
		 */
		for (i = 0; i < clients; i++) {
			print_if_verbose("waking: %d\n", cpids[i]);

			if (__write_ts(i, t1+i)) {
				do_exit = 1;
				break;
			}
		}

		if (do_exit)
			break;

		/*
		 * Poll and read replies
		 */
		pending_events = clients;
		while (pending_events) {
			int evts = poll(ipfd, clients, -1);

			if (evts < 0) {
				do_exit = 1;
				break;
			} else if (!evts)
				continue;

			if (read_ts(ipfd, evts, t1, cpids)) {
				do_exit = 1;
				break;
			}

			pending_events -= evts;
		}
	} while (!do_exit);

	kill_connection();
	free(t1);
	free(ipfd);
}

static void run_test(void)
{
	struct sem *sem;
	pid_t *cpids;
	int i, status;

	sem = mmap(NULL, sizeof(*sem), PROT_READ|PROT_WRITE,
			MAP_SHARED | MAP_ANONYMOUS, 0, 0);
	if (sem == MAP_FAILED) {
		perror("mmap");
		return;
	}

	init_sem(sem);

	for (i = 0; i < 2*clients; i++) {
		if (pipe(pipes[i])) {
			perror("pipe");
			return;
		}
	}

	cpids = malloc(sizeof(pid_t) * clients);

	for (i = 0; i < clients; i++) {
		cpids[i] = fork();
		if (cpids[i]) {
			sem_down(sem);
			continue;
		}

		run_child(pipes[2*i], pipes[2*i+1], sem);
		exit(0);
	}

	run_parent(cpids);

	for (i = 0; i < clients; i++)
		kill(cpids[i], SIGQUIT);
	for (i = 0; i < clients; i++)
		waitpid(cpids[i], &status, 0);

	free(cpids);
	munmap(sem, sizeof(*sem));
}

static void handle_sigint(int sig)
{
	kill(app_pid, SIGINT);
}

int main(int argc, char *argv[])
{
	int app_offset, off;
	char app[256];

	off = 0;
	app_offset = parse_options(argc, argv);
	if (app_offset >= argc) {
		printf("%s: [options] 'app'\n", argv[0]);
		return 1;
	}

	while (app_offset < argc) {
		if (off) {
			app[off] = ' ';
			off++;
		}
		off += sprintf(app + off, "%s", argv[app_offset]);
		app_offset++;
	}

	signal(SIGINT, handle_sigint);

	/*
	 * Start app and start logging latencies
	 */
	app_pid = fork_off(app);
	run_test();

	printf("\nParameters: min_wait=%ums, max_wait=%ums, clients=%u\n",
			min_delay, max_delay, clients);
	printf("Entries logged: %lu\n", nr_stats(&delay_stats));
	printf("\n             Averages\n");
	printf("-------------------------------------\n");
	printf("\tMax\t\t%8.0f usec\n", max_stats(&delay_stats));
	printf("\tAvg\t\t%8.0f usec\n", avg_stats(&delay_stats));
	printf("\tStdev\t\t%8.0f usec\n", stddev_stats(&delay_stats));
	printf("\tStdev mean\t%8.0f usec\n", stddev_mean_stats(&delay_stats));

	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Wed, 2009-09-09 at 23:12 +0300, Nikos Chantziaras wrote:

> With your version of latt.c, I get these results with 2.6-tip vs 
> 2.6.31-rc9-bfs:
> 
> 
> (mainline)
> Averages:
> ------------------------------
>          Max            50 usec
>          Avg            12 usec
>          Stdev           3 usec
> 
> 
> (BFS)
> Averages:
> ------------------------------
>          Max           474 usec
>          Avg            11 usec
>          Stdev          16 usec
> 
> 
> However, the interactivity problems still remain.  Does that mean it's 
> not a latency issue?

Could be a fairness issue.  If X+client needs more than it's fair share
of CPU, there's nothing to do but use nice levels.  I'm stuck with
unaccelerated X (nvidia card), so if I want a good DVD watching or
whatever eye-candy experience while my box does a lot of other work, I
either have to use SCHED_IDLE/nice for the background stuff, or renice
X.  That's the down side of a fair scheduler.

There is another variant of latency related interactivity issue for the
desktop though, too LOW latency.  If X and clients are switching too
fast, redraw can look nasty, sliced/diced.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 8360 bytes --]


* Nikos Chantziaras <realnc@arcor.de> wrote:

> On 09/09/2009 09:04 PM, Ingo Molnar wrote:
>> [...]
>> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>>
>>> On Wed, Sep 09 2009, Jens Axboe wrote:
>>>  [...]
>>> BFS210 runs on the laptop (dual core intel core duo). With make -j4
>>> running, I clock the following latt -c8 'sleep 10' latencies:
>>>
>>> -rc9
>>>
>>>          Max                17895 usec
>>>          Avg                 8028 usec
>>>          Stdev               5948 usec
>>>          Stdev mean           405 usec
>>>
>>>          Max                17896 usec
>>>          Avg                 4951 usec
>>>          Stdev               6278 usec
>>>          Stdev mean           427 usec
>>>
>>>          Max                17885 usec
>>>          Avg                 5526 usec
>>>          Stdev               6819 usec
>>>          Stdev mean           464 usec
>>>
>>> -rc9 + mike
>>>
>>>          Max                 6061 usec
>>>          Avg                 3797 usec
>>>          Stdev               1726 usec
>>>          Stdev mean           117 usec
>>>
>>>          Max                 5122 usec
>>>          Avg                 3958 usec
>>>          Stdev               1697 usec
>>>          Stdev mean           115 usec
>>>
>>>          Max                 6691 usec
>>>          Avg                 2130 usec
>>>          Stdev               2165 usec
>>>          Stdev mean           147 usec
>>
>> At least in my tests these latencies were mainly due to a bug in
>> latt.c - i've attached the fixed version.
>>
>> The other reason was wakeup batching. If you do this:
>>
>>     echo 0>  /proc/sys/kernel/sched_wakeup_granularity_ns
>>
>> ... then you can switch on insta-wakeups on -tip too.
>>
>> With a dual-core box and a make -j4 background job running, on
>> latest -tip i get the following latencies:
>>
>>   $ ./latt -c8 sleep 30
>>   Entries: 656 (clients=8)
>>
>>   Averages:
>>   ------------------------------
>>   	Max	      158 usec
>> 	Avg	       12 usec
>> 	Stdev	       10 usec
>
> With your version of latt.c, I get these results with 2.6-tip vs  
> 2.6.31-rc9-bfs:
>
>
> (mainline)
> Averages:
> ------------------------------
>         Max            50 usec
>         Avg            12 usec
>         Stdev           3 usec
>
>
> (BFS)
> Averages:
> ------------------------------
>         Max           474 usec
>         Avg            11 usec
>         Stdev          16 usec
>
> However, the interactivity problems still remain.  Does that mean 
> it's not a latency issue?

It means that Jens's test-app, which demonstrated and helped us fix 
the issue for him does not help us fix it for you just yet.

The "fluidity problem" you described might not be a classic latency 
issue per se (which latt.c measures), but a timeslicing / CPU time 
distribution problem.

A slight shift in CPU time allocation can change the flow of tasks 
to result in a 'choppier' system.

Have you tried, in addition of the granularity tweaks you've done, 
to renice mplayer either up or down? (or compiz and Xorg for that 
matter)

I'm not necessarily suggesting this as a 'real' solution (we really 
prefer kernels that just get it right) - but it's an additional 
parameter dimension along which you can tweak CPU time distribution 
on your box.

Here's the general rule of thumb: mine one nice level gives plus 5% 
CPU time to a task and takes away 5% CPU time from another task - 
i.e. shifts the CPU allocation by 10%.
 
( this is modified by all sorts of dynamic conditions: by the number
  of tasks running and their wakeup patters so not a rule cast into 
  stone - but still a good ballpark figure for CPU intense tasks. )

Btw., i've read your descriptions about what you've tuned so far - 
have you seen/checked the wakeup_granularity tunable as well? 
Setting that to 0 will change the general balance of how CPU time is 
allocated between tasks too.

There's also a whole bunch of scheduler features you can turn on/off 
individually via /debug/sched_features. For example, to turn off 
NEW_FAIR_SLEEPERS, you can do:

  # cat /debug/sched_features 
  NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT 
  START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK 
  NO_DOUBLE_TICK ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD 
  NO_WAKEUP_OVERLAP LAST_BUDDY OWNER_SPIN 

  # echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features

Btw., NO_NEW_FAIR_SLEEPERS is something that will turn the scheduler 
into a more classic fair scheduler (like BFS is too).

NO_START_DEBIT might be another thing that improves (or worsens :-/) 
make -j type of kernel build workloads.

Note, these flags are all runtime, the new settings take effect 
almost immediately (and at the latest it takes effect when a task 
has started up) and safe to do runtime.

It basically gives us 32768 pluggable schedulers each with a 
slightly separate algorithm - each setting in essence creates a new 
scheduler. (this mechanism is how we introduce new scheduler 
features and allow their debugging / regression-testing.)

(okay, almost, so beware: turning on HRTICK might lock up your 
system.)

Plus, yet another dimension of tuning on SMP systems (such as 
dual-core) are the sched-domains tunable. There's a whole world of 
tuning in that area and BFS essentially implements a very agressive 
'always balance to other CPUs' policy.

I've attached my sched-tune-domains script which helps tune these 
parameters.

For example on a testbox of mine it outputs:

usage: tune-sched-domains <val>
{cpu0/domain0:SIBLING} SD flag: 239
+   1: SD_LOAD_BALANCE:          Do load balancing on this domain
+   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
+   4: SD_BALANCE_EXEC:          Balance on exec
+   8: SD_BALANCE_FORK:          Balance on fork, clone
-  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
+  32: SD_WAKE_AFFINE:           Wake task to waking CPU
+  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
+ 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
- 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
- 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
-1024: SD_SERIALIZE:             Only a single load balancing instance
-2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
-4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain
{cpu0/domain1:MC} SD flag: 4735
+   1: SD_LOAD_BALANCE:          Do load balancing on this domain
+   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
+   4: SD_BALANCE_EXEC:          Balance on exec
+   8: SD_BALANCE_FORK:          Balance on fork, clone
+  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
+  32: SD_WAKE_AFFINE:           Wake task to waking CPU
+  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
- 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
- 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
+ 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
-1024: SD_SERIALIZE:             Only a single load balancing instance
-2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
+4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain
{cpu0/domain2:NODE} SD flag: 3183
+   1: SD_LOAD_BALANCE:          Do load balancing on this domain
+   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
+   4: SD_BALANCE_EXEC:          Balance on exec
+   8: SD_BALANCE_FORK:          Balance on fork, clone
-  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
+  32: SD_WAKE_AFFINE:           Wake task to waking CPU
+  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
- 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
- 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
- 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
+1024: SD_SERIALIZE:             Only a single load balancing instance
+2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
-4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain

The way i can turn on say SD_WAKE_IDLE for the NODE domain is to:

   tune-sched-domains 239 4735 $((3183+16))

( This is a pretty stone-age script i admit ;-)

Thanks for all your testing so far,

	Ingo

[-- Attachment #2: tune-sched-domains --]
[-- Type: text/plain, Size: 2152 bytes --]


DIR=/proc/sys/kernel/sched_domain/

print_flags()
{
  flags[1]="SD_LOAD_BALANCE:          Do load balancing on this domain"
  flags[2]="SD_BALANCE_NEWIDLE:       Balance when about to become idle"
  flags[4]="SD_BALANCE_EXEC:          Balance on exec"
  flags[8]="SD_BALANCE_FORK:          Balance on fork, clone"
  flags[16]="SD_WAKE_IDLE:             Wake to idle CPU on task wakeup"
  flags[32]="SD_WAKE_AFFINE:           Wake task to waking CPU"
  flags[64]="SD_WAKE_BALANCE:          Perform balancing at task wakeup"
  flags[128]="SD_SHARE_CPUPOWER:        Domain members share cpu power"
  flags[256]="SD_POWERSAVINGS_BALANCE:  Balance for power savings"
  flags[512]="SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources"
  flags[1024]="SD_SERIALIZE:             Only a single load balancing instance"
  flags[2048]="SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit"
  flags[4096]="SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain"

  DEC=$1
  CPU=$2
  DOM=$3

  [ -d $DIR/$CPU/$DOM ] || { exit 0; }

  NAME=$(cat $DIR/$CPU/$DOM/name)

  [ $DEC = "-1" ] && DEC=$(cat $DIR/$CPU/$DOM/flags)

  echo "{$CPU/$DOM:$NAME} SD flag: $DEC"

  for ((mask=1;mask<=4096;mask*=2)); do
   if [ "$[$mask & $DEC]" != "0" ]; then
      printf "+%4d: %s\n" $mask "${flags[$mask]}"
   else
      if [ $mask -le 4096 ]; then
        printf "%c%4d: %s\n" "-" $mask "${flags[$mask]}"
      fi
   fi
  done
}


[ $# -lt "1" ] && {
  echo 'usage: tune-sched-domains <val>'

  print_flags -1 cpu0 domain0
  print_flags -1 cpu0 domain1
  print_flags -1 cpu0 domain2
  print_flags -1 cpu0 domain3

  exit -1;
}

DOM0=$1
DOM1=${2:-$DOM0}
DOM2=${3:-$DOM1}
DOM3=${4:-$DOM2}

cd $DIR

for CPU in *; do
 cd $CPU
 for DOM in *; do

  FLAGS=$DOM/flags

  VAL=`cat $FLAGS`
  case $FLAGS in
     "domain0/flags") NEW_VAL=$DOM0 ;;
     "domain1/flags") NEW_VAL=$DOM1 ;;
     "domain2/flags") NEW_VAL=$DOM2 ;;
     "domain3/flags") NEW_VAL=$DOM3 ;;
     *) echo "error!" ;;
  esac

  echo $NEW_VAL > $FLAGS

  [ "$CPU" = "cpu0" ] && {
    echo "changed $FLAGS: $VAL => $NEW_VAL"
    print_flags $NEW_VAL $CPU $DOM
  }
 done
 cd ..
done

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> > However, the interactivity problems still remain.  Does that 
> > mean it's not a latency issue?
> 
> It means that Jens's test-app, which demonstrated and helped us 
> fix the issue for him does not help us fix it for you just yet.

Lemme qualify that by saying that Jens's issues are improved not 
fixed [he has not re-run with latest latt.c yet] but not all things 
are fully fixed yet. For example the xmodmap thing sounds 
interesting - could that be a child-runs-first effect?

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > > However, the interactivity problems still remain.  Does that 
> > > mean it's not a latency issue?
> > 
> > It means that Jens's test-app, which demonstrated and helped us 
> > fix the issue for him does not help us fix it for you just yet.
> 
> Lemme qualify that by saying that Jens's issues are improved not 
> fixed [he has not re-run with latest latt.c yet] but not all things 
> are fully fixed yet. For example the xmodmap thing sounds 
> interesting - could that be a child-runs-first effect?

I thought so too, so when -tip failed to boot I pulled the patches from
Mike into 2.6.31. It doesn't change anything for xmodmap, though.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Thu, Sep 10 2009, Ingo Molnar wrote:
> > 
> > * Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > > > However, the interactivity problems still remain.  Does that 
> > > > mean it's not a latency issue?
> > > 
> > > It means that Jens's test-app, which demonstrated and helped us 
> > > fix the issue for him does not help us fix it for you just yet.
> > 
> > Lemme qualify that by saying that Jens's issues are improved not 
> > fixed [he has not re-run with latest latt.c yet] but not all things 
> > are fully fixed yet. For example the xmodmap thing sounds 
> > interesting - could that be a child-runs-first effect?
> 
> I thought so too, so when -tip failed to boot I pulled the patches 
> from Mike into 2.6.31. It doesn't change anything for xmodmap, 
> though.

Note, you can access just the pristine scheduler patches by checking 
out and testing tip:sched/core - no need to pull them out and apply.

Your crash looks like clocksource related - that's in a separate 
topic which you can thus isolate if you use sched/core.

Thanks,

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Thu, Sep 10 2009, Ingo Molnar wrote:
> > > 
> > > * Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > > > However, the interactivity problems still remain.  Does that 
> > > > > mean it's not a latency issue?
> > > > 
> > > > It means that Jens's test-app, which demonstrated and helped us 
> > > > fix the issue for him does not help us fix it for you just yet.
> > > 
> > > Lemme qualify that by saying that Jens's issues are improved not 
> > > fixed [he has not re-run with latest latt.c yet] but not all things 
> > > are fully fixed yet. For example the xmodmap thing sounds 
> > > interesting - could that be a child-runs-first effect?
> > 
> > I thought so too, so when -tip failed to boot I pulled the patches 
> > from Mike into 2.6.31. It doesn't change anything for xmodmap, 
> > though.
> 
> Note, you can access just the pristine scheduler patches by checking 
> out and testing tip:sched/core - no need to pull them out and apply.
> 
> Your crash looks like clocksource related - that's in a separate 
> topic which you can thus isolate if you use sched/core.

I'm building sched/core now and will run the xmodmap test there.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Jens Axboe wrote:
> On Thu, Sep 10 2009, Ingo Molnar wrote:
> > 
> > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > 
> > > On Thu, Sep 10 2009, Ingo Molnar wrote:
> > > > 
> > > > * Ingo Molnar <mingo@elte.hu> wrote:
> > > > 
> > > > > > However, the interactivity problems still remain.  Does that 
> > > > > > mean it's not a latency issue?
> > > > > 
> > > > > It means that Jens's test-app, which demonstrated and helped us 
> > > > > fix the issue for him does not help us fix it for you just yet.
> > > > 
> > > > Lemme qualify that by saying that Jens's issues are improved not 
> > > > fixed [he has not re-run with latest latt.c yet] but not all things 
> > > > are fully fixed yet. For example the xmodmap thing sounds 
> > > > interesting - could that be a child-runs-first effect?
> > > 
> > > I thought so too, so when -tip failed to boot I pulled the patches 
> > > from Mike into 2.6.31. It doesn't change anything for xmodmap, 
> > > though.
> > 
> > Note, you can access just the pristine scheduler patches by checking 
> > out and testing tip:sched/core - no need to pull them out and apply.
> > 
> > Your crash looks like clocksource related - that's in a separate 
> > topic which you can thus isolate if you use sched/core.
> 
> I'm building sched/core now and will run the xmodmap test there.

No difference. Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
I get:

 Performance counter stats for 'xmodmap .xmodmap-carl':

       9.009137  task-clock-msecs         #      0.447 CPUs 
             18  context-switches         #      0.002 M/sec
              1  CPU-migrations           #      0.000 M/sec
            315  page-faults              #      0.035 M/sec
  <not counted>  cycles                  
  <not counted>  instructions            
  <not counted>  cache-references        
  <not counted>  cache-misses            

    0.020167093  seconds time elapsed

Woot!

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Thu, 2009-09-10 at 12:28 +0200, Jens Axboe wrote:

> No difference. Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
> I get:
> 
>  Performance counter stats for 'xmodmap .xmodmap-carl':
> 
>        9.009137  task-clock-msecs         #      0.447 CPUs 
>              18  context-switches         #      0.002 M/sec
>               1  CPU-migrations           #      0.000 M/sec
>             315  page-faults              #      0.035 M/sec
>   <not counted>  cycles                  
>   <not counted>  instructions            
>   <not counted>  cache-references        
>   <not counted>  cache-misses            
> 
>     0.020167093  seconds time elapsed
> 
> Woot!

Something is very seriously hosed on that box... clock?

Can you turn it back on, and do..
	while sleep .1; do cat /proc/sched_debug >> foo; done 
..on one core, and (quickly;) xmodmap .xmodmap-carl, then send me a few
seconds worth (gzipped up) to eyeball?

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

On Thu, Sep 10 2009, Mike Galbraith wrote:
> On Thu, 2009-09-10 at 12:28 +0200, Jens Axboe wrote:
> 
> > No difference. Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
> > I get:
> > 
> >  Performance counter stats for 'xmodmap .xmodmap-carl':
> > 
> >        9.009137  task-clock-msecs         #      0.447 CPUs 
> >              18  context-switches         #      0.002 M/sec
> >               1  CPU-migrations           #      0.000 M/sec
> >             315  page-faults              #      0.035 M/sec
> >   <not counted>  cycles                  
> >   <not counted>  instructions            
> >   <not counted>  cache-references        
> >   <not counted>  cache-misses            
> > 
> >     0.020167093  seconds time elapsed
> > 
> > Woot!
> 
> Something is very seriously hosed on that box... clock?

model name      : Genuine Intel(R) CPU           T2400  @ 1.83GHz

Throttles down to 1.00GHz when idle.

> Can you turn it back on, and do..

I guess you mean turn NEW_FAIR_SLEEPERS back on, correct?

> 	while sleep .1; do cat /proc/sched_debug >> foo; done 
> ..on one core, and (quickly;) xmodmap .xmodmap-carl, then send me a few
> seconds worth (gzipped up) to eyeball?

Attached.

-- 
Jens Axboe


[-- Attachment #2: sched-debug-cat.bz2 --]
[-- Type: application/octet-stream, Size: 12209 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Thu, 2009-09-10 at 13:09 +0200, Jens Axboe wrote:
> On Thu, Sep 10 2009, Mike Galbraith wrote:
> > On Thu, 2009-09-10 at 12:28 +0200, Jens Axboe wrote:
> > 
> > > No difference. Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
> > > I get:
> > > 
> > >  Performance counter stats for 'xmodmap .xmodmap-carl':
> > > 
> > >        9.009137  task-clock-msecs         #      0.447 CPUs 
> > >              18  context-switches         #      0.002 M/sec
> > >               1  CPU-migrations           #      0.000 M/sec
> > >             315  page-faults              #      0.035 M/sec
> > >   <not counted>  cycles                  
> > >   <not counted>  instructions            
> > >   <not counted>  cache-references        
> > >   <not counted>  cache-misses            
> > > 
> > >     0.020167093  seconds time elapsed
> > > 
> > > Woot!
> > 
> > Something is very seriously hosed on that box... clock?
> 
> model name      : Genuine Intel(R) CPU           T2400  @ 1.83GHz
> 
> Throttles down to 1.00GHz when idle.
> 
> > Can you turn it back on, and do..
> 
> I guess you mean turn NEW_FAIR_SLEEPERS back on, correct?
> 
> > 	while sleep .1; do cat /proc/sched_debug >> foo; done 
> > ..on one core, and (quickly;) xmodmap .xmodmap-carl, then send me a few
> > seconds worth (gzipped up) to eyeball?
> 
> Attached.

xmodmap doesn't seem to be running in this sample.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Mike Galbraith wrote:
> On Thu, 2009-09-10 at 13:09 +0200, Jens Axboe wrote:
> > On Thu, Sep 10 2009, Mike Galbraith wrote:
> > > On Thu, 2009-09-10 at 12:28 +0200, Jens Axboe wrote:
> > > 
> > > > No difference. Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
> > > > I get:
> > > > 
> > > >  Performance counter stats for 'xmodmap .xmodmap-carl':
> > > > 
> > > >        9.009137  task-clock-msecs         #      0.447 CPUs 
> > > >              18  context-switches         #      0.002 M/sec
> > > >               1  CPU-migrations           #      0.000 M/sec
> > > >             315  page-faults              #      0.035 M/sec
> > > >   <not counted>  cycles                  
> > > >   <not counted>  instructions            
> > > >   <not counted>  cache-references        
> > > >   <not counted>  cache-misses            
> > > > 
> > > >     0.020167093  seconds time elapsed
> > > > 
> > > > Woot!
> > > 
> > > Something is very seriously hosed on that box... clock?
> > 
> > model name      : Genuine Intel(R) CPU           T2400  @ 1.83GHz
> > 
> > Throttles down to 1.00GHz when idle.
> > 
> > > Can you turn it back on, and do..
> > 
> > I guess you mean turn NEW_FAIR_SLEEPERS back on, correct?
> > 
> > > 	while sleep .1; do cat /proc/sched_debug >> foo; done 
> > > ..on one core, and (quickly;) xmodmap .xmodmap-carl, then send me a few
> > > seconds worth (gzipped up) to eyeball?
> > 
> > Attached.
> 
> xmodmap doesn't seem to be running in this sample.

That's weird, it was definitely running. I did:

sleep 1; xmodmap .xmodmap-carl

in one xterm, and then switched to the other and ran the sched_debug
dump. I have to do it this way, as X will not move focus once xmodmap
starts running. It could be that xmodmap is mostly idle, and the real
work is done by Xorg and/or xfwm4 (my window manager).

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Thu, 2009-09-10 at 13:24 +0200, Jens Axboe wrote:
> On Thu, Sep 10 2009, Mike Galbraith wrote:

> > xmodmap doesn't seem to be running in this sample.
> 
> That's weird, it was definitely running. I did:
> 
> sleep 1; xmodmap .xmodmap-carl
> 
> in one xterm, and then switched to the other and ran the sched_debug
> dump. I have to do it this way, as X will not move focus once xmodmap
> starts running. It could be that xmodmap is mostly idle, and the real
> work is done by Xorg and/or xfwm4 (my window manager).

Hm.  Ok, I'll crawl over it, see if anything falls out.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Mike Galbraith wrote:
> On Thu, 2009-09-10 at 13:24 +0200, Jens Axboe wrote:
> > On Thu, Sep 10 2009, Mike Galbraith wrote:
> 
> > > xmodmap doesn't seem to be running in this sample.
> > 
> > That's weird, it was definitely running. I did:
> > 
> > sleep 1; xmodmap .xmodmap-carl
> > 
> > in one xterm, and then switched to the other and ran the sched_debug
> > dump. I have to do it this way, as X will not move focus once xmodmap
> > starts running. It could be that xmodmap is mostly idle, and the real
> > work is done by Xorg and/or xfwm4 (my window manager).
> 
> Hm.  Ok, I'll crawl over it, see if anything falls out.

That seems to be confirmed with the low context switch rate of the perf
stat of xmodmap. If I run perf stat -a to get a system wide collection
for xmodmap, I get:

 Performance counter stats for 'xmodmap .xmodmap-carl':

   20112.060925  task-clock-msecs         #      1.998 CPUs 
         629360  context-switches         #      0.031 M/sec
              8  CPU-migrations           #      0.000 M/sec
          13489  page-faults              #      0.001 M/sec
  <not counted>  cycles                  
  <not counted>  instructions            
  <not counted>  cache-references        
  <not counted>  cache-misses            

   10.067532449  seconds time elapsed

And again, system is idle while this is happening. Can't rule out that
this is some kind of user space bug of course.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Thu, 2009-09-10 at 13:35 +0200, Jens Axboe wrote:
> On Thu, Sep 10 2009, Mike Galbraith wrote:
> > On Thu, 2009-09-10 at 13:24 +0200, Jens Axboe wrote:
> > > On Thu, Sep 10 2009, Mike Galbraith wrote:
> > 
> > > > xmodmap doesn't seem to be running in this sample.
> > > 
> > > That's weird, it was definitely running. I did:
> > > 
> > > sleep 1; xmodmap .xmodmap-carl
> > > 
> > > in one xterm, and then switched to the other and ran the sched_debug
> > > dump. I have to do it this way, as X will not move focus once xmodmap
> > > starts running. It could be that xmodmap is mostly idle, and the real
> > > work is done by Xorg and/or xfwm4 (my window manager).
> > 
> > Hm.  Ok, I'll crawl over it, see if anything falls out.
> 
> That seems to be confirmed with the low context switch rate of the perf
> stat of xmodmap. If I run perf stat -a to get a system wide collection
> for xmodmap, I get:
> 
>  Performance counter stats for 'xmodmap .xmodmap-carl':
> 
>    20112.060925  task-clock-msecs         #      1.998 CPUs 
>          629360  context-switches         #      0.031 M/sec
>               8  CPU-migrations           #      0.000 M/sec
>           13489  page-faults              #      0.001 M/sec
>   <not counted>  cycles                  
>   <not counted>  instructions            
>   <not counted>  cache-references        
>   <not counted>  cache-misses            
> 
>    10.067532449  seconds time elapsed
> 
> And again, system is idle while this is happening. Can't rule out that
> this is some kind of user space bug of course.

All I'm seeing so far is massive CPU usage for dinky job.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Bret Towe]

On Wed, Sep 9, 2009 at 11:08 PM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Nikos Chantziaras <realnc@arcor.de> wrote:
>
>> On 09/09/2009 09:04 PM, Ingo Molnar wrote:
>>> [...]
>>> * Jens Axboe<jens.axboe@oracle.com>  wrote:
>>>
>>>> On Wed, Sep 09 2009, Jens Axboe wrote:
>>>>  [...]
>>>> BFS210 runs on the laptop (dual core intel core duo). With make -j4
>>>> running, I clock the following latt -c8 'sleep 10' latencies:
>>>>
>>>> -rc9
>>>>
>>>>          Max                17895 usec
>>>>          Avg                 8028 usec
>>>>          Stdev               5948 usec
>>>>          Stdev mean           405 usec
>>>>
>>>>          Max                17896 usec
>>>>          Avg                 4951 usec
>>>>          Stdev               6278 usec
>>>>          Stdev mean           427 usec
>>>>
>>>>          Max                17885 usec
>>>>          Avg                 5526 usec
>>>>          Stdev               6819 usec
>>>>          Stdev mean           464 usec
>>>>
>>>> -rc9 + mike
>>>>
>>>>          Max                 6061 usec
>>>>          Avg                 3797 usec
>>>>          Stdev               1726 usec
>>>>          Stdev mean           117 usec
>>>>
>>>>          Max                 5122 usec
>>>>          Avg                 3958 usec
>>>>          Stdev               1697 usec
>>>>          Stdev mean           115 usec
>>>>
>>>>          Max                 6691 usec
>>>>          Avg                 2130 usec
>>>>          Stdev               2165 usec
>>>>          Stdev mean           147 usec
>>>
>>> At least in my tests these latencies were mainly due to a bug in
>>> latt.c - i've attached the fixed version.
>>>
>>> The other reason was wakeup batching. If you do this:
>>>
>>>     echo 0>  /proc/sys/kernel/sched_wakeup_granularity_ns
>>>
>>> ... then you can switch on insta-wakeups on -tip too.
>>>
>>> With a dual-core box and a make -j4 background job running, on
>>> latest -tip i get the following latencies:
>>>
>>>   $ ./latt -c8 sleep 30
>>>   Entries: 656 (clients=8)
>>>
>>>   Averages:
>>>   ------------------------------
>>>      Max           158 usec
>>>      Avg            12 usec
>>>      Stdev          10 usec
>>
>> With your version of latt.c, I get these results with 2.6-tip vs
>> 2.6.31-rc9-bfs:
>>
>>
>> (mainline)
>> Averages:
>> ------------------------------
>>         Max            50 usec
>>         Avg            12 usec
>>         Stdev           3 usec
>>
>>
>> (BFS)
>> Averages:
>> ------------------------------
>>         Max           474 usec
>>         Avg            11 usec
>>         Stdev          16 usec
>>
>> However, the interactivity problems still remain.  Does that mean
>> it's not a latency issue?
>
> It means that Jens's test-app, which demonstrated and helped us fix
> the issue for him does not help us fix it for you just yet.
>
> The "fluidity problem" you described might not be a classic latency
> issue per se (which latt.c measures), but a timeslicing / CPU time
> distribution problem.
>
> A slight shift in CPU time allocation can change the flow of tasks
> to result in a 'choppier' system.
>
> Have you tried, in addition of the granularity tweaks you've done,
> to renice mplayer either up or down? (or compiz and Xorg for that
> matter)
>
> I'm not necessarily suggesting this as a 'real' solution (we really
> prefer kernels that just get it right) - but it's an additional
> parameter dimension along which you can tweak CPU time distribution
> on your box.
>
> Here's the general rule of thumb: mine one nice level gives plus 5%
> CPU time to a task and takes away 5% CPU time from another task -
> i.e. shifts the CPU allocation by 10%.
>
> ( this is modified by all sorts of dynamic conditions: by the number
>  of tasks running and their wakeup patters so not a rule cast into
>  stone - but still a good ballpark figure for CPU intense tasks. )
>
> Btw., i've read your descriptions about what you've tuned so far -
> have you seen/checked the wakeup_granularity tunable as well?
> Setting that to 0 will change the general balance of how CPU time is
> allocated between tasks too.
>
> There's also a whole bunch of scheduler features you can turn on/off
> individually via /debug/sched_features. For example, to turn off
> NEW_FAIR_SLEEPERS, you can do:
>
>  # cat /debug/sched_features
>  NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT
>  START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK
>  NO_DOUBLE_TICK ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD
>  NO_WAKEUP_OVERLAP LAST_BUDDY OWNER_SPIN
>
>  # echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features
>
> Btw., NO_NEW_FAIR_SLEEPERS is something that will turn the scheduler
> into a more classic fair scheduler (like BFS is too).
>
> NO_START_DEBIT might be another thing that improves (or worsens :-/)
> make -j type of kernel build workloads.

thanks to this thread and others I've seen several kernel tunables
that can effect how the scheduler performs/acts
but what I don't see after a bit of looking is where all these are documented
perhaps thats also part of the reason there are unhappy people with
the current code in the kernel just because they don't know how
to tune it for their workload

> Note, these flags are all runtime, the new settings take effect
> almost immediately (and at the latest it takes effect when a task
> has started up) and safe to do runtime.
>
> It basically gives us 32768 pluggable schedulers each with a
> slightly separate algorithm - each setting in essence creates a new
> scheduler. (this mechanism is how we introduce new scheduler
> features and allow their debugging / regression-testing.)
>
> (okay, almost, so beware: turning on HRTICK might lock up your
> system.)
>
> Plus, yet another dimension of tuning on SMP systems (such as
> dual-core) are the sched-domains tunable. There's a whole world of
> tuning in that area and BFS essentially implements a very agressive
> 'always balance to other CPUs' policy.
>
> I've attached my sched-tune-domains script which helps tune these
> parameters.
>
> For example on a testbox of mine it outputs:
>
> usage: tune-sched-domains <val>
> {cpu0/domain0:SIBLING} SD flag: 239
> +   1: SD_LOAD_BALANCE:          Do load balancing on this domain
> +   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
> +   4: SD_BALANCE_EXEC:          Balance on exec
> +   8: SD_BALANCE_FORK:          Balance on fork, clone
> -  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
> +  32: SD_WAKE_AFFINE:           Wake task to waking CPU
> +  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
> + 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
> - 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
> - 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
> -1024: SD_SERIALIZE:             Only a single load balancing instance
> -2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
> -4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain
> {cpu0/domain1:MC} SD flag: 4735
> +   1: SD_LOAD_BALANCE:          Do load balancing on this domain
> +   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
> +   4: SD_BALANCE_EXEC:          Balance on exec
> +   8: SD_BALANCE_FORK:          Balance on fork, clone
> +  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
> +  32: SD_WAKE_AFFINE:           Wake task to waking CPU
> +  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
> - 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
> - 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
> + 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
> -1024: SD_SERIALIZE:             Only a single load balancing instance
> -2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
> +4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain
> {cpu0/domain2:NODE} SD flag: 3183
> +   1: SD_LOAD_BALANCE:          Do load balancing on this domain
> +   2: SD_BALANCE_NEWIDLE:       Balance when about to become idle
> +   4: SD_BALANCE_EXEC:          Balance on exec
> +   8: SD_BALANCE_FORK:          Balance on fork, clone
> -  16: SD_WAKE_IDLE:             Wake to idle CPU on task wakeup
> +  32: SD_WAKE_AFFINE:           Wake task to waking CPU
> +  64: SD_WAKE_BALANCE:          Perform balancing at task wakeup
> - 128: SD_SHARE_CPUPOWER:        Domain members share cpu power
> - 256: SD_POWERSAVINGS_BALANCE:  Balance for power savings
> - 512: SD_SHARE_PKG_RESOURCES:   Domain members share cpu pkg resources
> +1024: SD_SERIALIZE:             Only a single load balancing instance
> +2048: SD_WAKE_IDLE_FAR:         Gain latency sacrificing cache hit
> -4096: SD_PREFER_SIBLING:        Prefer to place tasks in a sibling domain
>
> The way i can turn on say SD_WAKE_IDLE for the NODE domain is to:
>
>   tune-sched-domains 239 4735 $((3183+16))
>
> ( This is a pretty stone-age script i admit ;-)
>
> Thanks for all your testing so far,
>
>        Ingo
>

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

On Thu, 2009-09-10 at 09:02 -0700, Bret Towe wrote:
> 
> thanks to this thread and others I've seen several kernel tunables
> that can effect how the scheduler performs/acts
> but what I don't see after a bit of looking is where all these are
> documented
> perhaps thats also part of the reason there are unhappy people with
> the current code in the kernel just because they don't know how
> to tune it for their workload

The thing is, ideally they should not need to poke at these. These knobs
are under CONFIG_SCHED_DEBUG, and that is exactly what they are for.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Bret Towe]

On Thu, Sep 10, 2009 at 9:05 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> On Thu, 2009-09-10 at 09:02 -0700, Bret Towe wrote:
>>
>> thanks to this thread and others I've seen several kernel tunables
>> that can effect how the scheduler performs/acts
>> but what I don't see after a bit of looking is where all these are
>> documented
>> perhaps thats also part of the reason there are unhappy people with
>> the current code in the kernel just because they don't know how
>> to tune it for their workload
>
> The thing is, ideally they should not need to poke at these. These knobs
> are under CONFIG_SCHED_DEBUG, and that is exactly what they are for.

even then I would think they should be documented so people can find out
what item is hurting their workload so they can better report the bug no?

>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Bret Towe <magnade@gmail.com> wrote:

> On Thu, Sep 10, 2009 at 9:05 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> > On Thu, 2009-09-10 at 09:02 -0700, Bret Towe wrote:
> >>
> >> thanks to this thread and others I've seen several kernel tunables
> >> that can effect how the scheduler performs/acts
> >> but what I don't see after a bit of looking is where all these are
> >> documented
> >> perhaps thats also part of the reason there are unhappy people with
> >> the current code in the kernel just because they don't know how
> >> to tune it for their workload
> >
> > The thing is, ideally they should not need to poke at these. 
> > These knobs are under CONFIG_SCHED_DEBUG, and that is exactly 
> > what they are for.
> 
> even then I would think they should be documented so people can 
> find out what item is hurting their workload so they can better 
> report the bug no?

Would be happy to apply such documentation patches. You could also 
help start adding a 'scheduler performance' wiki portion to 
perf.wiki.kernel.org, if you have time for that.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Bret Towe]

On Thu, Sep 10, 2009 at 9:26 AM, Ingo Molnar <mingo@elte.hu> wrote:
>
> * Bret Towe <magnade@gmail.com> wrote:
>
>> On Thu, Sep 10, 2009 at 9:05 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
>> > On Thu, 2009-09-10 at 09:02 -0700, Bret Towe wrote:
>> >>
>> >> thanks to this thread and others I've seen several kernel tunables
>> >> that can effect how the scheduler performs/acts
>> >> but what I don't see after a bit of looking is where all these are
>> >> documented
>> >> perhaps thats also part of the reason there are unhappy people with
>> >> the current code in the kernel just because they don't know how
>> >> to tune it for their workload
>> >
>> > The thing is, ideally they should not need to poke at these.
>> > These knobs are under CONFIG_SCHED_DEBUG, and that is exactly
>> > what they are for.
>>
>> even then I would think they should be documented so people can
>> find out what item is hurting their workload so they can better
>> report the bug no?
>
> Would be happy to apply such documentation patches. You could also
> help start adding a 'scheduler performance' wiki portion to
> perf.wiki.kernel.org, if you have time for that.

time isn't so much the issue but not having any clue as to what any
of the options do

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Bret Towe <magnade@gmail.com> wrote:

> On Thu, Sep 10, 2009 at 9:26 AM, Ingo Molnar <mingo@elte.hu> wrote:
> >
> > * Bret Towe <magnade@gmail.com> wrote:
> >
> >> On Thu, Sep 10, 2009 at 9:05 AM, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:
> >> > On Thu, 2009-09-10 at 09:02 -0700, Bret Towe wrote:
> >> >>
> >> >> thanks to this thread and others I've seen several kernel tunables
> >> >> that can effect how the scheduler performs/acts
> >> >> but what I don't see after a bit of looking is where all these are
> >> >> documented
> >> >> perhaps thats also part of the reason there are unhappy people with
> >> >> the current code in the kernel just because they don't know how
> >> >> to tune it for their workload
> >> >
> >> > The thing is, ideally they should not need to poke at these.
> >> > These knobs are under CONFIG_SCHED_DEBUG, and that is exactly
> >> > what they are for.
> >>
> >> even then I would think they should be documented so people can
> >> find out what item is hurting their workload so they can better
> >> report the bug no?
> >
> > Would be happy to apply such documentation patches. You could also
> > help start adding a 'scheduler performance' wiki portion to
> > perf.wiki.kernel.org, if you have time for that.
> 
> time isn't so much the issue but not having any clue as to what 
> any of the options do

One approach would be to list them in an email in this thread with 
question marks and let people here fill them in - then help by 
organizing and prettifying the result on the wiki.

Asking for clarifications when an explanation is unclear is also 
helpful - those who write this code are not the best people to judge 
whether technical descriptions are understandable enough.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/10/2009 09:08 AM, Ingo Molnar wrote:
>
> * Nikos Chantziaras<realnc@arcor.de>  wrote:
>>
>> With your version of latt.c, I get these results with 2.6-tip vs
>> 2.6.31-rc9-bfs:
>>
>>
>> (mainline)
>> Averages:
>> ------------------------------
>>          Max            50 usec
>>          Avg            12 usec
>>          Stdev           3 usec
>>
>>
>> (BFS)
>> Averages:
>> ------------------------------
>>          Max           474 usec
>>          Avg            11 usec
>>          Stdev          16 usec
>>
>> However, the interactivity problems still remain.  Does that mean
>> it's not a latency issue?
>
> It means that Jens's test-app, which demonstrated and helped us fix
> the issue for him does not help us fix it for you just yet.
>
> The "fluidity problem" you described might not be a classic latency
> issue per se (which latt.c measures), but a timeslicing / CPU time
> distribution problem.
>
> A slight shift in CPU time allocation can change the flow of tasks
> to result in a 'choppier' system.
>
> Have you tried, in addition of the granularity tweaks you've done,
> to renice mplayer either up or down? (or compiz and Xorg for that
> matter)

Yes.  It seems to do what one would expect, but only if two separate 
programs are competing for CPU time continuously.  For example, when 
running two glxgears instances, one with nice 0 the other with 19, the 
first will report ~5000 FPS, the other ~1000.  Renicing the second one 
from 19 to 0, will result in both reporting ~3000.  So nice values 
obviously work in distributing CPU time.  But the problem isn't the 
available CPU time it seems since even if running glxgears nice -20, it 
will still freeze during various other interactive taks (moving windows 
etc.)


> [...]
>    # echo NO_NEW_FAIR_SLEEPERS>  /debug/sched_features
>
> Btw., NO_NEW_FAIR_SLEEPERS is something that will turn the scheduler
> into a more classic fair scheduler (like BFS is too).

Setting NO_NEW_FAIR_SLEEPERS (with everything else at default values) 
pretty much solves all issues I raised in all my other posts!  With this 
setting, I can do "nice -n 19 make -j20" and still have a very smooth 
desktop and watch a movie at the same time.  Various other annoyances 
(like the "logout/shutdown/restart" dialog of KDE not appearing at all 
until the background fade-out effect has finished) are also gone.  So 
this seems to be the single most important setting that vastly improves 
desktop behavior, at least here.

In fact, I liked this setting so much that I went to 
kernel/sched_features.h of kernel 2.6.30.5 (the kernel I use normally 
right now) and set SCHED_FEAT(NEW_FAIR_SLEEPERS, 0) (default is 1) with 
absolutely no other tweaks (like sched_latency_ns, 
sched_wakeup_granularity_ns, etc.).  It pretty much behaves like BFS now 
from an interactivity point of view.  But I've used it only for about an 
hour or so, so I don't know if any ill effects will appear later on.


> NO_START_DEBIT might be another thing that improves (or worsens :-/)
> make -j type of kernel build workloads.

No effect with this one, at least not one I could observe.

I didn't have the opportunity yet to test and tweak all the other 
various settings you listed, but I will try to do so as soon as possible.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Nikos Chantziaras <realnc@arcor.de> wrote:

> On 09/10/2009 09:08 AM, Ingo Molnar wrote:
>>
>> * Nikos Chantziaras<realnc@arcor.de>  wrote:
>>>
>>> With your version of latt.c, I get these results with 2.6-tip vs
>>> 2.6.31-rc9-bfs:
>>>
>>>
>>> (mainline)
>>> Averages:
>>> ------------------------------
>>>          Max            50 usec
>>>          Avg            12 usec
>>>          Stdev           3 usec
>>>
>>>
>>> (BFS)
>>> Averages:
>>> ------------------------------
>>>          Max           474 usec
>>>          Avg            11 usec
>>>          Stdev          16 usec
>>>
>>> However, the interactivity problems still remain.  Does that mean
>>> it's not a latency issue?
>>
>> It means that Jens's test-app, which demonstrated and helped us fix
>> the issue for him does not help us fix it for you just yet.
>>
>> The "fluidity problem" you described might not be a classic latency
>> issue per se (which latt.c measures), but a timeslicing / CPU time
>> distribution problem.
>>
>> A slight shift in CPU time allocation can change the flow of tasks
>> to result in a 'choppier' system.
>>
>> Have you tried, in addition of the granularity tweaks you've done,
>> to renice mplayer either up or down? (or compiz and Xorg for that
>> matter)
>
> Yes.  It seems to do what one would expect, but only if two separate  
> programs are competing for CPU time continuously.  For example, when  
> running two glxgears instances, one with nice 0 the other with 19, the  
> first will report ~5000 FPS, the other ~1000.  Renicing the second one  
> from 19 to 0, will result in both reporting ~3000.  So nice values  
> obviously work in distributing CPU time.  But the problem isn't the  
> available CPU time it seems since even if running glxgears nice -20, it  
> will still freeze during various other interactive taks (moving windows  
> etc.)
>
>
>> [...]
>>    # echo NO_NEW_FAIR_SLEEPERS>  /debug/sched_features
>>
>> Btw., NO_NEW_FAIR_SLEEPERS is something that will turn the scheduler
>> into a more classic fair scheduler (like BFS is too).
>
> Setting NO_NEW_FAIR_SLEEPERS (with everything else at default 
> values) pretty much solves all issues I raised in all my other 
> posts!  With this setting, I can do "nice -n 19 make -j20" and 
> still have a very smooth desktop and watch a movie at the same 
> time.  Various other annoyances (like the 
> "logout/shutdown/restart" dialog of KDE not appearing at all until 
> the background fade-out effect has finished) are also gone.  So 
> this seems to be the single most important setting that vastly 
> improves desktop behavior, at least here.
>
> In fact, I liked this setting so much that I went to 
> kernel/sched_features.h of kernel 2.6.30.5 (the kernel I use 
> normally right now) and set SCHED_FEAT(NEW_FAIR_SLEEPERS, 0) 
> (default is 1) with absolutely no other tweaks (like 
> sched_latency_ns, sched_wakeup_granularity_ns, etc.).  It pretty 
> much behaves like BFS now from an interactivity point of view.  
> But I've used it only for about an hour or so, so I don't know if 
> any ill effects will appear later on.

ok, this is quite an important observation!

Either NEW_FAIR_SLEEPERS is broken, or if it works it's not what we 
want to do. Other measures in the scheduler protect us from fatal 
badness here, but all the finer wakeup behavior is out the window 
really.

Will check this. We'll probably start with a quick commit disabling 
it first - then re-enabling it if it's fixed (will Cc: you so that 
you can re-test with fixed-NEW_FAIR_SLEEPERS, if it's re-enabled).

Thanks a lot for the persistent testing!

	Ingo

[tip:sched/core] sched: Disable NEW_FAIR_SLEEPERS for now

[tip-bot for Ingo Molnar]

Commit-ID:  3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
Gitweb:     http://git.kernel.org/tip/3f2aa307c4d26b4ed6509d0a79e8254c9e07e921
Author:     Ingo Molnar <mingo@elte.hu>
AuthorDate: Thu, 10 Sep 2009 20:34:48 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 10 Sep 2009 20:34:48 +0200

sched: Disable NEW_FAIR_SLEEPERS for now

Nikos Chantziaras and Jens Axboe reported that turning off
NEW_FAIR_SLEEPERS improves desktop interactivity visibly.

Nikos described his experiences the following way:

  " With this setting, I can do "nice -n 19 make -j20" and
    still have a very smooth desktop and watch a movie at
    the same time.  Various other annoyances (like the
    "logout/shutdown/restart" dialog of KDE not appearing
    at all until the background fade-out effect has finished)
    are also gone.  So this seems to be the single most
    important setting that vastly improves desktop behavior,
    at least here. "

Jens described it the following way, referring to a 10-seconds
xmodmap scheduling delay he was trying to debug:

  " Then I tried switching NO_NEW_FAIR_SLEEPERS on, and then
    I get:

    Performance counter stats for 'xmodmap .xmodmap-carl':

         9.009137  task-clock-msecs         #      0.447 CPUs
               18  context-switches         #      0.002 M/sec
                1  CPU-migrations           #      0.000 M/sec
              315  page-faults              #      0.035 M/sec

    0.020167093  seconds time elapsed

    Woot! "

So disable it for now. In perf trace output i can see weird
delta timestamps:

  cc1-9943  [001]  2802.059479616: sched_stat_wait: task: as:9944 wait: 2801938766276 [ns]

That nsec field is not supposed to be that large. More digging
is needed - but lets turn it off while the real bug is found.

Reported-by: Nikos Chantziaras <realnc@arcor.de>
Tested-by: Nikos Chantziaras <realnc@arcor.de>
Reported-by: Jens Axboe <jens.axboe@oracle.com>
Tested-by: Jens Axboe <jens.axboe@oracle.com>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Mike Galbraith <efault@gmx.de>
LKML-Reference: <4AA93D34.8040500@arcor.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 kernel/sched_features.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/sched_features.h b/kernel/sched_features.h
index 4569bfa..e2dc63a 100644
--- a/kernel/sched_features.h
+++ b/kernel/sched_features.h
@@ -1,4 +1,4 @@
-SCHED_FEAT(NEW_FAIR_SLEEPERS, 1)
+SCHED_FEAT(NEW_FAIR_SLEEPERS, 0)
 SCHED_FEAT(NORMALIZED_SLEEPER, 0)
 SCHED_FEAT(ADAPTIVE_GRAN, 1)
 SCHED_FEAT(WAKEUP_PREEMPT, 1)

[tip:sched/core] sched: Fix sched::sched_stat_wait tracepoint field

[tip-bot for Ingo Molnar]

Commit-ID:  e1f8450854d69f0291882804406ea1bab3ca44b4
Gitweb:     http://git.kernel.org/tip/e1f8450854d69f0291882804406ea1bab3ca44b4
Author:     Ingo Molnar <mingo@elte.hu>
AuthorDate: Thu, 10 Sep 2009 20:52:09 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Thu, 10 Sep 2009 20:52:54 +0200

sched: Fix sched::sched_stat_wait tracepoint field

This weird perf trace output:

  cc1-9943  [001]  2802.059479616: sched_stat_wait: task: as:9944 wait: 2801938766276 [ns]

Is caused by setting one component field of the delta to zero
a bit too early. Move it to later.

( Note, this does not affect the NEW_FAIR_SLEEPERS interactivity bug,
  it's just a reporting bug in essence. )

Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Nikos Chantziaras <realnc@arcor.de>
Cc: Jens Axboe <jens.axboe@oracle.com>
Cc: Mike Galbraith <efault@gmx.de>
LKML-Reference: <4AA93D34.8040500@arcor.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 kernel/sched_fair.c |    3 +--
 1 files changed, 1 insertions(+), 2 deletions(-)

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index 26fadb4..aa7f841 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -545,14 +545,13 @@ update_stats_wait_end(struct cfs_rq *cfs_rq, struct sched_entity *se)
 	schedstat_set(se->wait_count, se->wait_count + 1);
 	schedstat_set(se->wait_sum, se->wait_sum +
 			rq_of(cfs_rq)->clock - se->wait_start);
-	schedstat_set(se->wait_start, 0);
-
 #ifdef CONFIG_SCHEDSTATS
 	if (entity_is_task(se)) {
 		trace_sched_stat_wait(task_of(se),
 			rq_of(cfs_rq)->clock - se->wait_start);
 	}
 #endif
+	schedstat_set(se->wait_start, 0);
 }
 
 static inline void

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Wed, Sep 09 2009, Ingo Molnar wrote:
> At least in my tests these latencies were mainly due to a bug in 
> latt.c - i've attached the fixed version.

What bug? I don't see any functional change between the version you
attach and the current one.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Wed, Sep 09 2009, Ingo Molnar wrote:
> > At least in my tests these latencies were mainly due to a bug in 
> > latt.c - i've attached the fixed version.
> 
> What bug? I don't see any functional change between the version 
> you attach and the current one.

Here's the diff of what i fixed yesterday over the last latt.c 
version i found in this thread. The poll() thing is the significant 
one.

	Ingo

--- latt.c.orig
+++ latt.c
@@ -39,6 +39,7 @@ static unsigned int verbose;
 struct stats
 {
 	double n, mean, M2, max;
+	int max_pid;
 };
 
 static void update_stats(struct stats *stats, unsigned long long val)
@@ -85,22 +86,6 @@ static double stddev_stats(struct stats 
 	return sqrt(variance);
 }
 
-/*
- * The std dev of the mean is related to the std dev by:
- *
- *             s
- * s_mean = -------
- *          sqrt(n)
- *
- */
-static double stddev_mean_stats(struct stats *stats)
-{
-	double variance = stats->M2 / (stats->n - 1);
-	double variance_mean = variance / stats->n;
-
-	return sqrt(variance_mean);
-}
-
 struct stats delay_stats;
 
 static int pipes[MAX_CLIENTS*2][2];
@@ -212,7 +197,7 @@ static unsigned long usec_since(struct t
 static void log_delay(unsigned long delay)
 {
 	if (verbose) {
-		fprintf(stderr, "log delay %8lu usec\n", delay);
+		fprintf(stderr, "log delay %8lu usec (pid %d)\n", delay, getpid());
 		fflush(stderr);
 	}
 
@@ -300,7 +285,7 @@ static int __write_ts(int i, struct time
 	return write(fd, ts, sizeof(*ts)) != sizeof(*ts);
 }
 
-static long __read_ts(int i, struct timespec *ts)
+static long __read_ts(int i, struct timespec *ts, pid_t *cpids)
 {
 	int fd = pipes[2*i+1][0];
 	struct timespec t;
@@ -309,11 +294,14 @@ static long __read_ts(int i, struct time
 		return -1;
 
 	log_delay(usec_since(ts, &t));
+	if (verbose)
+		fprintf(stderr, "got delay %ld from child %d [pid %d]\n", usec_since(ts, &t), i, cpids[i]);
 
 	return 0;
 }
 
-static int read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts)
+static int read_ts(struct pollfd *pfd, unsigned int nr, struct timespec *ts,
+		   pid_t *cpids)
 {
 	unsigned int i;
 
@@ -322,7 +310,7 @@ static int read_ts(struct pollfd *pfd, u
 			return -1L;
 		if (pfd[i].revents & POLLIN) {
 			pfd[i].events = 0;
-			if (__read_ts(i, &ts[i]))
+			if (__read_ts(i, &ts[i], cpids))
 				return -1L;
 			nr--;
 		}
@@ -368,7 +356,6 @@ static void run_parent(pid_t *cpids)
 	srand(1234);
 
 	do {
-		unsigned long delay;
 		unsigned pending_events;
 
 		do_rand_sleep();
@@ -404,17 +391,17 @@ static void run_parent(pid_t *cpids)
 		 */
 		pending_events = clients;
 		while (pending_events) {
-			int evts = poll(ipfd, clients, 0);
+			int evts = poll(ipfd, clients, -1);
 
 			if (evts < 0) {
 				do_exit = 1;
 				break;
 			} else if (!evts) {
-				/* printf("bugger2\n"); */
+				printf("bugger2\n");
 				continue;
 			}
 
-			if (read_ts(ipfd, evts, t1)) {
+			if (read_ts(ipfd, evts, t1, cpids)) {
 				do_exit = 1;
 				break;
 			}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Wed, Sep 09 2009, Ingo Molnar wrote:
> > > At least in my tests these latencies were mainly due to a bug in 
> > > latt.c - i've attached the fixed version.
> > 
> > What bug? I don't see any functional change between the version 
> > you attach and the current one.
> 
> Here's the diff of what i fixed yesterday over the last latt.c 
> version i found in this thread. The poll() thing is the significant 
> one.

Ah indeed, thanks Ingo! I'm tempted to add some actual work processing
into latt as well, to see if that helps improve it.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> 
> One thing I also noticed is that when I have logged in, I run xmodmap
> manually to load some keymappings (I always tell myself to add this to
> the log in scripts, but I suspend/resume this laptop for weeks at the
> time and forget before the next boot). With the stock kernel, xmodmap
> will halt X updates and take forever to run. With BFS, it returned
> instantly. As I would expect.

Can you provide a little more detail (I'm a xmodmap n00b), how does one
run xmodmap and maybe provide your xmodmap config?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Peter Zijlstra wrote:
> On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > 
> > One thing I also noticed is that when I have logged in, I run xmodmap
> > manually to load some keymappings (I always tell myself to add this to
> > the log in scripts, but I suspend/resume this laptop for weeks at the
> > time and forget before the next boot). With the stock kernel, xmodmap
> > will halt X updates and take forever to run. With BFS, it returned
> > instantly. As I would expect.
> 
> Can you provide a little more detail (I'm a xmodmap n00b), how does one
> run xmodmap and maybe provide your xmodmap config?

Will do, let me get the notebook and strace time it on both bfs and
mainline.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]


* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Thu, Sep 10 2009, Peter Zijlstra wrote:
> > On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > > 
> > > One thing I also noticed is that when I have logged in, I run xmodmap
> > > manually to load some keymappings (I always tell myself to add this to
> > > the log in scripts, but I suspend/resume this laptop for weeks at the
> > > time and forget before the next boot). With the stock kernel, xmodmap
> > > will halt X updates and take forever to run. With BFS, it returned
> > > instantly. As I would expect.
> > 
> > Can you provide a little more detail (I'm a xmodmap n00b), how 
> > does one run xmodmap and maybe provide your xmodmap config?
> 
> Will do, let me get the notebook and strace time it on both bfs 
> and mainline.

A 'perf stat' comparison would be nice as well - that will show us 
events strace doesnt show, and shows us the basic scheduler behavior 
as well.

A 'full' trace could be done as well via trace-cmd.c (attached), if 
you enable:

  CONFIG_CONTEXT_SWITCH_TRACER=y

and did something like:

  trace-cmd -s xmodmap ... > trace.txt

	Ingo

[-- Attachment #2: trace-cmd.c --]
[-- Type: text/plain, Size: 6541 bytes --]

/*
 * Copyright (C) 2008, Steven Rostedt <srostedt@redhat.com>
 *
 * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 *
 * This program is free software; you can redistribute it and/or modify
 * it under the terms of the GNU General Public License as published by
 * the Free Software Foundation; version 2 of the License (not later!)
 *
 * This program is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 * GNU General Public License for more details.
 *
 * You should have received a copy of the GNU General Public License
 * along with this program; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 *
 * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>

#define VERSION "0.2"

#define _STR(x) #x
#define STR(x) _STR(x)
#define MAX_PATH 256

#define TRACE_CTRL	"tracing_enabled"
#define TRACE		"latency_trace"
#define AVAILABLE	"available_tracers"
#define CURRENT		"current_tracer"
#define ITER_CTRL	"iter_ctrl"
#define MAX_LATENCY	"tracing_max_latency"
#define THRESH		"tracing_thresh"

static void die(char *fmt, ...)
{
	va_list ap;
	int ret = errno;

	if (errno)
		perror("trace-cmd");
	else
		ret = -1;

	va_start(ap, fmt);
	fprintf(stderr, "  ");
	vfprintf(stderr, fmt, ap);
	va_end(ap);

	fprintf(stderr, "\n");
	exit(ret);
}

static int search_mounts(char *path, int size)
{
	FILE *fp;
	static char debugfs[MAX_PATH+1];
	static int debugfs_size;

	if (!debugfs_size) {
		char type[100];

		if ((fp = fopen("/proc/mounts","r")) == NULL)
			die("Can't open /proc/mounts for read");

		while (fscanf(fp, "%*s %"
			      STR(MAX_PATH)
			      "s %99s %*s %*d %*d\n",
			      debugfs, type) == 2) {
			if (strcmp(type, "debugfs") == 0)
				break;
		}
		fclose(fp);

		if (strcmp(type, "debugfs") != 0)
			die("debugfs not mounted, please mount");
	}

	debugfs_size = strlen(debugfs)+1;

	if (size > debugfs_size)
		size = debugfs_size;

	memcpy(path, debugfs, size);

	return size;
}

/*
 * Finds the path to the debugfs/tracing
 * Allocates the string and stores it.
 */
static int tracing_dir(char *path, int size)
{
	static char debugfs[MAX_PATH];
	static int debugfs_size;
	int ret;

	if (!debugfs_size) {
		ret = search_mounts(debugfs, MAX_PATH);
		if (ret < 0)
			return ret;
		debugfs_size = MAX_PATH - ret;
		strncat(debugfs, "/tracing", debugfs_size);
		debugfs_size = strlen(debugfs)+1;
	}

	if (size > debugfs_size)
		size = debugfs_size;

	memcpy(path, debugfs, size);

	return size;
}

static int tracing_type(char *path, const char *type, int size)
{
	int len = strlen(type) + 1;
	int ret;

	ret = tracing_dir(path, size);
	size -= ret;
	if (len > size)
		die ("debugfs path is too big!");
	strcat(path, "/");
	strcat(path, type);

	return ret + len;
}

static void write_trace(const char *file, const char *val)
{
	char path[MAX_PATH+1];
	int fd;

	tracing_type(path, file, MAX_PATH);

	fd = open(path, O_WRONLY);
	if (fd < 0)
		die("writng %s", path);
	write(fd, val, strlen(val));
	close(fd);
}

static int find_trace_type(const char *type)
{
	FILE *fp;
	char path[MAX_PATH+1];
	char scan[100];
	int ret;

	tracing_type(path, AVAILABLE, MAX_PATH);
	fp = fopen(path, "r");
	if (!fp)
		die("reading %s", path);
	do {
		ret = fscanf(fp, "%99s", scan);
		if (ret > 0 && strcmp(scan, type) == 0)
			break;
	} while (ret > 0);
	fclose(fp);

	return ret > 0;
}

static void set_ftrace(int set)
{
	int fd;
	char *val = set ? "1" : "0";

	fd = open("/proc/sys/kernel/ftrace_enabled", O_WRONLY);
	if (fd < 0)
		die ("Can't %s ftrace", set ? "enable" : "disable");

	write(fd, val, 1);
	close(fd);
}

void run_cmd(int argc, char **argv)
{
	int status;
	int pid;

	if ((pid = fork()) < 0)
		die("failed to fork");
	if (!pid) {
		/* child */
		if (execvp(argv[0], argv))
			exit(-1);
	}
	waitpid(pid, &status, 0);
}

static void usage(char **argv)
{
	char *arg = argv[0];
	char *p = arg+strlen(arg);

	while (p >= arg && *p != '/')
		p--;
	p++;

	printf("\n"
	       "%s version %s\n\n"
	       "usage: %s OPTION [-f] command ...\n"
	       "\n"
	       "  -s  set context switch trace\n"
	       "  -p  set preemption off trace\n"
	       "  -i  set interrupts off trace\n"
	       "  -b  set preempt and interrupts off trace\n"
	       "  -w  set wakeup tracing\n"
	       "  -e  set event tracing\n"
	       "  -f  set function trace\n"
	       "\n"
	       "  Note: only -f may be set with any other trace\n"
	       "\n", p, VERSION, p);
	exit(-1);
}

int main (int argc, char **argv)
{
	const char *type = NULL;
	const char *config;
	int function = 0;
	int type_set = 0;
	int max = 0;
	int c;

	while ((c = getopt(argc, argv, "+hspibfew")) >= 0) {
		switch (c) {
		case 'h':
			usage(argv);
			break;
		case 's':
			type = "sched_switch";
			config = "CONFIG_CONTEXT_SWITCH_TRACER";
			type_set++;
			break;
		case 'p':
			type = "preemptoff";
			config = "CONFIG_CRITICAL_PREEMPT_TIMING";
			max = 1;
			type_set++;
			break;
		case 'i':
			type = "irqsoff";
			config = "CONFIG_CRITICAL_IRQSOFF_TIMING";
			max = 1;
			type_set++;
			break;
		case 'b':
			type = "preemptirqsoff";
			config = "CONFIG_CRITICAL_IRQSOFF_TIMING and"
				" CONFIG_CRITICAL_PREEMPT_TIMING";
			max = 1;
			type_set++;
			break;
		case 'w':
			type = "wakeup";
			config = "CONFIG_WAKEUP_TRACER";
			max = 1;
			type_set++;
			break;
		case 'e':
			type = "events";
			config = "CONFIG_EVENT_TRACER";
			type_set++;
			break;
		case 'f':
			if (!type) {
				type = "ftrace";
				config = "CONFIG_FTRACE";
			}
			function = 1;
			break;
		default:
			/* yeah yeah, I know this is a dup! */
			usage(argv);
			break;
		}
	}

	if (type_set > 1)
		usage(argv);

	if (!(argc - optind))
		usage(argv);

	if (!type)
		usage(argv);

	if (!find_trace_type(type))
		die("Trace type %s not found.\n"
		    " Please configure the kernel with %s set\n",
		    type, config);

	write_trace(TRACE_CTRL, "0");
	if (function)
		set_ftrace(1);
	if (max)
		write_trace(MAX_LATENCY, "0");

	write_trace(CURRENT, type);
	write_trace(TRACE_CTRL, "1");

	run_cmd(argc - optind, &argv[optind]);

	write_trace(TRACE_CTRL, "0");
	if (function)
		set_ftrace(0);

	system("cat /debug/tracing/trace");

	return 0;
}

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Thu, Sep 10 2009, Peter Zijlstra wrote:
> > > On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > > > 
> > > > One thing I also noticed is that when I have logged in, I run xmodmap
> > > > manually to load some keymappings (I always tell myself to add this to
> > > > the log in scripts, but I suspend/resume this laptop for weeks at the
> > > > time and forget before the next boot). With the stock kernel, xmodmap
> > > > will halt X updates and take forever to run. With BFS, it returned
> > > > instantly. As I would expect.
> > > 
> > > Can you provide a little more detail (I'm a xmodmap n00b), how 
> > > does one run xmodmap and maybe provide your xmodmap config?
> > 
> > Will do, let me get the notebook and strace time it on both bfs 
> > and mainline.
> 
> A 'perf stat' comparison would be nice as well - that will show us 
> events strace doesnt show, and shows us the basic scheduler behavior 
> as well.
> 
> A 'full' trace could be done as well via trace-cmd.c (attached), if 
> you enable:
> 
>   CONFIG_CONTEXT_SWITCH_TRACER=y
> 
> and did something like:
> 
>   trace-cmd -s xmodmap ... > trace.txt

trace.txt attached. Steven, you seem to go through a lot of trouble to
find the debugfs path, yet at the very end do:

> 	system("cat /debug/tracing/trace");

which doesn't seem quite right :-)

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 165 bytes --]

On Thu, Sep 10 2009, Jens Axboe wrote:
> trace.txt attached.

Now it really is, I very much need a more clever MUA to help me with
these things :-)

-- 
Jens Axboe


[-- Attachment #2: trace.txt.bz2 --]
[-- Type: application/octet-stream, Size: 241339 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Steven Rostedt]

On Thu, 2009-09-10 at 11:44 +0200, Jens Axboe wrote:
> On Thu, Sep 10 2009, Ingo Molnar wrote:

> trace.txt attached. Steven, you seem to go through a lot of trouble to
> find the debugfs path, yet at the very end do:
> 
> > 	system("cat /debug/tracing/trace");
> 
> which doesn't seem quite right :-)
> 

That's an older version of the tool. The newer version (still in alpha)
doesn't do that.

-- Steve

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

[-- Attachment #1: Type: text/plain, Size: 2179 bytes --]

On Thu, Sep 10 2009, Jens Axboe wrote:
> On Thu, Sep 10 2009, Peter Zijlstra wrote:
> > On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > > 
> > > One thing I also noticed is that when I have logged in, I run xmodmap
> > > manually to load some keymappings (I always tell myself to add this to
> > > the log in scripts, but I suspend/resume this laptop for weeks at the
> > > time and forget before the next boot). With the stock kernel, xmodmap
> > > will halt X updates and take forever to run. With BFS, it returned
> > > instantly. As I would expect.
> > 
> > Can you provide a little more detail (I'm a xmodmap n00b), how does one
> > run xmodmap and maybe provide your xmodmap config?
> 
> Will do, let me get the notebook and strace time it on both bfs and
> mainline.

Here's the result of running perf stat xmodmap .xmodmap-carl on the
notebook. I have attached the .xmodmap-carl file, it's pretty simple. I
have also attached the output of strace -o foo -f -tt xmodmap
.xmodmap-carl when run on 2.6.31-rc9.

2.6.31-rc9-bfs210

 Performance counter stats for 'xmodmap .xmodmap-carl':

     153.994976  task-clock-msecs         #      0.990 CPUs   (scaled from 99.86%)
              0  context-switches         #      0.000 M/sec  (scaled from 99.86%)
              0  CPU-migrations           #      0.000 M/sec  (scaled from 99.86%)
            315  page-faults              #      0.002 M/sec  (scaled from 99.86%)
  <not counted>  cycles                  
  <not counted>  instructions            
  <not counted>  cache-references        
  <not counted>  cache-misses            

    0.155573406  seconds time elapsed

2.6.31-rc9

 Performance counter stats for 'xmodmap .xmodmap-carl':

       8.529265  task-clock-msecs         #      0.001 CPUs 
             23  context-switches         #      0.003 M/sec
              1  CPU-migrations           #      0.000 M/sec
            315  page-faults              #      0.037 M/sec
  <not counted>  cycles                  
  <not counted>  instructions            
  <not counted>  cache-references        
  <not counted>  cache-misses            

   11.804293482  seconds time elapsed


-- 
Jens Axboe


[-- Attachment #2: .xmodmap-carl --]
[-- Type: text/plain, Size: 1365 bytes --]

!
! This is an `xmodmap' input file for 
!   PC 105 key, wide Delete, tall Enter (XFree86; US) keyboards.
! Automatically generated on Sun Aug 31 20:11:39 2008 by axboe with
! XKeyCaps 2.47; Copyright (c) 1991-1999 Jamie Zawinski; 2005-2006 Christoph Berg.
! http://www.jwz.org/xkeycaps/
!
! This file presupposes that the keyboard is in the default state, and
! may malfunction if it is not.
!
remove Lock    = Caps_Lock
remove Mod4    = Meta_L Meta_R
remove Mod5    = Scroll_Lock

keycode 0x43 =	Escape	XF86_Switch_VT_1
keycode 0x44 =	F2	XF86_Switch_VT_2
keycode 0x45 =	F3	XF86_Switch_VT_3
keycode 0x46 =	F4	XF86_Switch_VT_4
keycode 0x47 =	F5	XF86_Switch_VT_5
keycode 0x48 =	F6	XF86_Switch_VT_6
keycode 0x49 =	F7	XF86_Switch_VT_7
keycode 0x4A =	F8	XF86_Switch_VT_8
keycode 0x4B =	F9	XF86_Switch_VT_9
keycode 0x4C =	F10	XF86_Switch_VT_10
keycode 0x5F =	F11	XF86_Switch_VT_11
keycode 0x60 =	F12	XF86_Switch_VT_12
keycode 0x16 =	BackSpace	Terminate_Server
keycode 0x3F =	KP_Multiply	XF86_ClearGrab
keycode 0x52 =	KP_Subtract	XF86_Prev_VMode
keycode 0x56 =	KP_Add	XF86_Next_VMode
keycode 0x42 =	Control_L
keycode 0x5E =	less	greater	bar	brokenbar	bar	brokenbar
keycode 0x40 =	Alt_L	Meta_L

add    Control = Control_L
add    Mod1    = 0x009C
add    Mod4    = 0x007F 0x0080
add    Mod5    = Mode_switch ISO_Level3_Shift

keycode 0xa6 = Page_Up
keycode 0xa7 = Page_Down

[-- Attachment #3: strace-xmodmap.txt --]
[-- Type: text/plain, Size: 20480 bytes --]

4659  09:32:29.464949 execve("/usr/bin/xmodmap", ["xmodmap", ".xmodmap-carl"], [/* 44 vars */]) = 0
4659  09:32:29.465551 brk(0)            = 0x87e2000
4659  09:32:29.465664 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.465771 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff8a000
4659  09:32:29.465875 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
4659  09:32:29.466003 open("/etc/ld.so.cache", O_RDONLY) = 3
4659  09:32:29.466100 fstat64(3, {st_mode=S_IFREG|0644, st_size=80093, ...}) = 0
4659  09:32:29.466243 mmap2(NULL, 80093, PROT_READ, MAP_PRIVATE, 3, 0) = 0x6ff76000
4659  09:32:29.466355 close(3)          = 0
4659  09:32:29.466438 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.466551 open("/usr/lib/libX11.so.6", O_RDONLY) = 3
4659  09:32:29.466654 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2604\1\0004\0\0\0\234"..., 512) = 512
4659  09:32:29.466775 fstat64(3, {st_mode=S_IFREG|0644, st_size=971436, ...}) = 0
4659  09:32:29.466913 mmap2(NULL, 975508, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fe87000
4659  09:32:29.467000 mprotect(0x6ff71000, 4096, PROT_NONE) = 0
4659  09:32:29.467089 mmap2(0x6ff72000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xea) = 0x6ff72000
4659  09:32:29.467193 mmap2(0x6ff75000, 660, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6ff75000
4659  09:32:29.467290 close(3)          = 0
4659  09:32:29.467404 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.467504 open("/lib/libc.so.6", O_RDONLY) = 3
4659  09:32:29.467605 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320h\1\0004\0\0\0T"..., 512) = 512
4659  09:32:29.467722 fstat64(3, {st_mode=S_IFREG|0755, st_size=1331404, ...}) = 0
4659  09:32:29.467860 mmap2(NULL, 1336944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fd40000
4659  09:32:29.467951 mmap2(0x6fe81000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x141) = 0x6fe81000
4659  09:32:29.468055 mmap2(0x6fe84000, 9840, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x6fe84000
4659  09:32:29.468152 close(3)          = 0
4659  09:32:29.468238 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.468375 open("/usr/lib/libxcb.so.1", O_RDONLY) = 3
4659  09:32:29.468477 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200z\0\0004\0\0\0\250"..., 512) = 512
4659  09:32:29.468599 fstat64(3, {st_mode=S_IFREG|0644, st_size=99768, ...}) = 0
4659  09:32:29.468732 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6fd3f000
4659  09:32:29.468829 mmap2(NULL, 102676, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fd25000
4659  09:32:29.468920 mmap2(0x6fd3d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17) = 0x6fd3d000
4659  09:32:29.469026 close(3)          = 0
4659  09:32:29.469112 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.469213 open("/lib/libdl.so.2", O_RDONLY) = 3
4659  09:32:29.469324 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0 \n\0\0004\0\0\0D"..., 512) = 512
4659  09:32:29.469453 fstat64(3, {st_mode=S_IFREG|0644, st_size=9676, ...}) = 0
4659  09:32:29.469594 mmap2(NULL, 12408, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fd21000
4659  09:32:29.469685 mmap2(0x6fd23000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x6fd23000
4659  09:32:29.469790 close(3)          = 0
4659  09:32:29.469879 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.469976 open("/usr/lib/libXau.so.6", O_RDONLY) = 3
4659  09:32:29.470077 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\n\0\0004\0\0\0<"..., 512) = 512
4659  09:32:29.470192 fstat64(3, {st_mode=S_IFREG|0644, st_size=9508, ...}) = 0
4659  09:32:29.470341 mmap2(NULL, 12412, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fd1d000
4659  09:32:29.470445 mmap2(0x6fd1f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x6fd1f000
4659  09:32:29.470557 close(3)          = 0
4659  09:32:29.470655 access("/etc/ld.so.nohwcap", F_OK) = 0
4659  09:32:29.470753 open("/usr/lib/libXdmcp.so.6", O_RDONLY) = 3
4659  09:32:29.470855 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\16\0\0004\0\0\0\f"..., 512) = 512
4659  09:32:29.470976 fstat64(3, {st_mode=S_IFREG|0644, st_size=16628, ...}) = 0
4659  09:32:29.471112 mmap2(NULL, 19520, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x6fd18000
4659  09:32:29.471204 mmap2(0x6fd1c000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0x6fd1c000
4659  09:32:29.471320 close(3)          = 0
4659  09:32:29.471450 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6fd17000
4659  09:32:29.471555 set_thread_area({entry_number:-1 -> 6, base_addr:0x6fd17a00, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
4659  09:32:29.471658 open("/dev/urandom", O_RDONLY) = 3
4659  09:32:29.471755 read(3, "\6~R)"..., 4) = 4
4659  09:32:29.471844 close(3)          = 0
4659  09:32:29.471947 mprotect(0x6fd1f000, 4096, PROT_READ) = 0
4659  09:32:29.472055 mprotect(0x6fd23000, 4096, PROT_READ) = 0
4659  09:32:29.472153 mprotect(0x6fd3d000, 4096, PROT_READ) = 0
4659  09:32:29.472344 mprotect(0x6fe81000, 8192, PROT_READ) = 0
4659  09:32:29.472525 mprotect(0x6ff72000, 4096, PROT_READ) = 0
4659  09:32:29.472621 mprotect(0x804e000, 4096, PROT_READ) = 0
4659  09:32:29.472712 mprotect(0x6ffa7000, 4096, PROT_READ) = 0
4659  09:32:29.472794 munmap(0x6ff76000, 80093) = 0
4659  09:32:29.473047 brk(0)            = 0x87e2000
4659  09:32:29.473126 brk(0x8803000)    = 0x8803000
4659  09:32:29.473323 socket(PF_FILE, SOCK_STREAM, 0) = 3
4659  09:32:29.473445 connect(3, {sa_family=AF_FILE, path=@"/tmp/.X11-unix/X0"...}, 20) = 0
4659  09:32:29.473708 getpeername(3, {sa_family=AF_FILE, path=@"/tmp/.X11-unix/X0"...}, [20]) = 0
4659  09:32:29.473878 uname({sys="Linux", node="carl", ...}) = 0
4659  09:32:29.474146 access("/home/axboe/.Xauthority", R_OK) = 0
4659  09:32:29.474257 open("/home/axboe/.Xauthority", O_RDONLY) = 4
4659  09:32:29.474398 fstat64(4, {st_mode=S_IFREG|0600, st_size=115, ...}) = 0
4659  09:32:29.474536 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff89000
4659  09:32:29.474629 read(4, "\1\0\0\4carl\0\0010\0\22MIT-MAGIC-COOKIE-1\0\20"..., 4096) = 115
4659  09:32:29.474756 read(4, ""..., 4096) = 0
4659  09:32:29.474839 close(4)          = 0
4659  09:32:29.474918 munmap(0x6ff89000, 4096) = 0
4659  09:32:29.475036 fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR)
4659  09:32:29.475124 fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4659  09:32:29.475205 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
4659  09:32:29.475313 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.475437 writev(3, [{"l\0\v\0\0\0\22\0\20\0\0\0"..., 12}, {""..., 0}, {"MIT-MAGIC-COOKIE-1"..., 18}, {"\0\0"..., 2}, {"\232\346\374\251t\31M}(\306K\276\272\16\354\376"..., 16}, {""..., 0}], 6) = 48
4659  09:32:29.475719 read(3, "\1\0\v\0\0\0\263\1"..., 8) = 8
4659  09:32:29.475823 read(3, "@\276\241\0\0\0`\2\377\377\37\0\0\1\0\0\24\0\377\377\1\7\0\0  \10\377\1\0\0\0T"..., 1740) = 1740
4659  09:32:29.476021 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.476122 writev(3, [{"b\0\5\0\f\0\0\0BIG-REQUESTS"..., 20}], 1) = 20
4659  09:32:29.476314 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.476427 read(3, "\1\0\1\0\0\0\0\0\1\216\0\0\0\0\0\0\24\0\0\0\0\0\0\0p\343%\t\0\0\0\0"..., 4096) = 32
4659  09:32:29.476580 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.476680 writev(3, [{"\216\0\1\0"..., 4}], 1) = 4
4659  09:32:29.476843 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.476940 read(3, "\1\0\2\0\0\0\0\0\377\377?\0\0\0\0\1\0\0\0\0p\343%\t\0\0\0\0\364_\36\10"..., 4096) = 32
4659  09:32:29.477087 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.477193 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.477306 writev(3, [{"7\0\5\0\0\0`\2#\1\0\0\10\0\0\0\377\377\377\0\24\0\6\0#\1\0\0\27\0\0\0\37"..., 44}, {NULL, 0}, {""..., 0}], 3) = 44
4659  09:32:29.477566 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.477665 read(3, "\1\10\4\0x\0\0\0\37\0\0\0\0\0\0\0\335\1\0\0``\37\10xs\310w9\2\26\10*"..., 4096) = 512
4659  09:32:29.477802 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.477909 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.478008 writev(3, [{"b\0\5\0\t\0`\2"..., 8}, {"XKEYBOARD"..., 9}, {"\0\0\0"..., 3}], 3) = 20
4659  09:32:29.478200 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.478310 read(3, "\1\0\5\0\0\0\0\0\1\220`\231\0\0\0\0\24\0\0\0\0\0\0\0p\343%\t\0\0\0\0"..., 4096) = 32
4659  09:32:29.478459 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.478561 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.478660 writev(3, [{"\220\0\2\0\1\0\0\0"..., 8}, {NULL, 0}, {""..., 0}], 3) = 8
4659  09:32:29.478841 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.478938 read(3, "\1\1\6\0\0\0\0\0\1\0\0\0\230\270\30\t\2\0\0\0\10\0\0\0p\343%\t\30\260\350\10"..., 4096) = 32
4659  09:32:29.479072 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.479170 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.479269 writev(3, [{"w\0\1\0"..., 4}, {NULL, 0}, {""..., 0}], 3) = 4
4659  09:32:29.479475 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.479577 read(3, "\1\3\7\0\6\0\0\0p\343%\t\0\0\0\0\4\0\0\0\0\0\0\0p\343%\t\0\0\0\0002"..., 4096) = 56
4659  09:32:29.479712 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.479804 open(".xmodmap-carl", O_RDONLY) = 4
4659  09:32:29.479905 fstat64(4, {st_mode=S_IFREG|0644, st_size=1365, ...}) = 0
4659  09:32:29.480039 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff89000
4659  09:32:29.480126 read(4, "!\n! This is an `xmodmap' input fi"..., 4096) = 1365
4659  09:32:29.480305 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.480421 writev(3, [{"\220\10\7\0\0\1\7\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 28}, {NULL, 0}, {""..., 0}], 3) = 28
4659  09:32:29.480679 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.480778 read(3, "\1\1\10\0G\4\0\0\0\0\10\377\7\0\0\31\31\10(\1\370\0\0\0\0\0\0\0\0\0\0\10\370"..., 4096) = 4096
4659  09:32:29.480906 read(3, "\0\0\0\0\0\0\0\0\1\1\1\0006\377\10\20\0\0\0\0\0\0\0\0\0\0\0\0\1\1\1\0i"..., 316) = 316
4659  09:32:29.481043 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.481734 open("/usr/share/X11/XKeysymDB", O_RDONLY) = 6
4659  09:32:29.481858 fstat64(6, {st_mode=S_IFREG|0644, st_size=8934, ...}) = 0
4659  09:32:29.481999 read(6, "! $Xorg: XKeysymDB,v 1.3 2000/08/"..., 8934) = 8934
4659  09:32:29.482113 close(6)          = 0
4659  09:32:29.482440 open("/usr/share/X11/locale/locale.alias", O_RDONLY) = 6
4659  09:32:29.482570 fstat64(6, {st_mode=S_IFREG|0644, st_size=80646, ...}) = 0
4659  09:32:29.482705 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.482793 read(6, "#\t$XdotOrg: lib/X11/nls/locale.al"..., 4096) = 4096
4659  09:32:29.482975 read(6, "br_FR.iso88591\t\t\t\t\tbr_FR.ISO8859-"..., 4096) = 4096
4659  09:32:29.483150 read(6, "9-1\nde_DE.ISO_8859-1\t\t\t\tde_DE.ISO"..., 4096) = 4096
4659  09:32:29.483358 read(6, "-1\nen_ZW.ISO-8859-1\t\t\t\ten_ZW.ISO8"..., 4096) = 4096
4659  09:32:29.483553 read(6, "UTF-8\nfi\t\t\t\t\t\tfi_FI.ISO8859-15\nfi"..., 4096) = 4096
4659  09:32:29.483732 read(6, "1255\nhe_IL.MICROSOFT-CP1255\t\t\t\the"..., 4096) = 4096
4659  09:32:29.483832 read(6, "\tlt_LT.UTF-8\nlv\t\t\t\t\t\tlv_LV.ISO885"..., 4096) = 4096
4659  09:32:29.483930 read(6, "1\t\t\t\tpt_BR.ISO8859-1\npt_BR.ISO_88"..., 4096) = 4096
4659  09:32:29.484024 read(6, "O-8859-1\t\t\t\tsv_FI.ISO8859-1\nsv_FI"..., 4096) = 4096
4659  09:32:29.484124 read(6, "8\nZH_TW.UTF-8\t\t\t\t\tzh_TW.UTF-8\nzu\t"..., 4096) = 4096
4659  09:32:29.484222 read(6, "\tar_AE.ISO8859-6\nar_AE.utf8:\t\t\t\t\t"..., 4096) = 4096
4659  09:32:29.484340 read(6, "5\nca_ES.UTF-8@euro:\t\t\t\tca_ES.UTF-"..., 4096) = 4096
4659  09:32:29.484447 read(6, "_GR.ISO8859-7\nel_GR@euro:\t\t\t\t\tel_"..., 4096) = 4096
4659  09:32:29.484551 read(6, "\tes_DO.ISO8859-1\nes_DO.iso88591:\t"..., 4096) = 4096
4659  09:32:29.484652 read(6, "\t\t\t\tfr_BE.ISO8859-1\nfr_BE.88591.e"..., 4096) = 4096
4659  09:32:29.484751 read(6, "nesian (now id).  These lines sho"..., 4096) = 4096
4659  09:32:29.484853 read(6, "5\nmk_MK.cp1251:\t\t\t\t\tmk_MK.CP1251\n"..., 4096) = 4096
4659  09:32:29.484955 read(6, "15\npt_PT.ISO-8859-15@euro:\t\t\t\tpt_"..., 4096) = 4096
4659  09:32:29.485052 read(6, "E.ISO8859-1\nsv_SE.88591:\t\t\t\t\tsv_S"..., 4096) = 4096
4659  09:32:29.485156 read(6, "zu_ZA.utf8:\t\t\t\t\tzu_ZA.UTF-8\n\n# Th"..., 4096) = 2822
4659  09:32:29.485243 read(6, ""..., 4096) = 0
4659  09:32:29.485288 close(6)          = 0
4659  09:32:29.485349 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.485406 open("/usr/share/X11/locale/locale.dir", O_RDONLY) = 6
4659  09:32:29.485466 fstat64(6, {st_mode=S_IFREG|0644, st_size=39195, ...}) = 0
4659  09:32:29.485539 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.485590 read(6, "#\t$XdotOrg: lib/X11/nls/locale.di"..., 4096) = 4096
4659  09:32:29.485682 read(6, "ALE:\t\t\teo_EO.ISO8859-3\n# iso8859-"..., 4096) = 4096
4659  09:32:29.485773 read(6, "59-1\niso8859-1/XLC_LOCALE\t\t\tny_NO"..., 4096) = 4096
4659  09:32:29.485864 read(6, "\t\tar_TN.UTF-8\nen_US.UTF-8/XLC_LOC"..., 4096) = 4096
4659  09:32:29.485953 read(6, "/XLC_LOCALE\t\t\tmk_MK.UTF-8\nen_US.U"..., 4096) = 4096
4659  09:32:29.486035 close(6)          = 0
4659  09:32:29.486079 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.486128 access("/usr/share/X11/locale/C/XLC_LOCALE", R_OK) = 0
4659  09:32:29.486190 open("/usr/share/X11/locale/C/XLC_LOCALE", O_RDONLY) = 6
4659  09:32:29.486251 fstat64(6, {st_mode=S_IFREG|0644, st_size=772, ...}) = 0
4659  09:32:29.486342 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.486389 read(6, "#  $Xorg: C,v 1.3 2000/08/17 19:4"..., 4096) = 772
4659  09:32:29.486512 read(6, ""..., 4096) = 0
4659  09:32:29.486561 close(6)          = 0
4659  09:32:29.486605 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.486720 open("/usr/share/X11/locale/locale.alias", O_RDONLY) = 6
4659  09:32:29.486780 fstat64(6, {st_mode=S_IFREG|0644, st_size=80646, ...}) = 0
4659  09:32:29.486853 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.486901 read(6, "#\t$XdotOrg: lib/X11/nls/locale.al"..., 4096) = 4096
4659  09:32:29.487000 read(6, "br_FR.iso88591\t\t\t\t\tbr_FR.ISO8859-"..., 4096) = 4096
4659  09:32:29.487094 read(6, "9-1\nde_DE.ISO_8859-1\t\t\t\tde_DE.ISO"..., 4096) = 4096
4659  09:32:29.487189 read(6, "-1\nen_ZW.ISO-8859-1\t\t\t\ten_ZW.ISO8"..., 4096) = 4096
4659  09:32:29.487285 read(6, "UTF-8\nfi\t\t\t\t\t\tfi_FI.ISO8859-15\nfi"..., 4096) = 4096
4659  09:32:29.487398 read(6, "1255\nhe_IL.MICROSOFT-CP1255\t\t\t\the"..., 4096) = 4096
4659  09:32:29.487497 read(6, "\tlt_LT.UTF-8\nlv\t\t\t\t\t\tlv_LV.ISO885"..., 4096) = 4096
4659  09:32:29.487597 read(6, "1\t\t\t\tpt_BR.ISO8859-1\npt_BR.ISO_88"..., 4096) = 4096
4659  09:32:29.487690 read(6, "O-8859-1\t\t\t\tsv_FI.ISO8859-1\nsv_FI"..., 4096) = 4096
4659  09:32:29.487789 read(6, "8\nZH_TW.UTF-8\t\t\t\t\tzh_TW.UTF-8\nzu\t"..., 4096) = 4096
4659  09:32:29.487886 read(6, "\tar_AE.ISO8859-6\nar_AE.utf8:\t\t\t\t\t"..., 4096) = 4096
4659  09:32:29.487987 read(6, "5\nca_ES.UTF-8@euro:\t\t\t\tca_ES.UTF-"..., 4096) = 4096
4659  09:32:29.488085 read(6, "_GR.ISO8859-7\nel_GR@euro:\t\t\t\t\tel_"..., 4096) = 4096
4659  09:32:29.488185 read(6, "\tes_DO.ISO8859-1\nes_DO.iso88591:\t"..., 4096) = 4096
4659  09:32:29.488283 read(6, "\t\t\t\tfr_BE.ISO8859-1\nfr_BE.88591.e"..., 4096) = 4096
4659  09:32:29.488399 read(6, "nesian (now id).  These lines sho"..., 4096) = 4096
4659  09:32:29.488501 read(6, "5\nmk_MK.cp1251:\t\t\t\t\tmk_MK.CP1251\n"..., 4096) = 4096
4659  09:32:29.488604 read(6, "15\npt_PT.ISO-8859-15@euro:\t\t\t\tpt_"..., 4096) = 4096
4659  09:32:29.488709 read(6, "E.ISO8859-1\nsv_SE.88591:\t\t\t\t\tsv_S"..., 4096) = 4096
4659  09:32:29.488812 read(6, "zu_ZA.utf8:\t\t\t\t\tzu_ZA.UTF-8\n\n# Th"..., 4096) = 2822
4659  09:32:29.488899 read(6, ""..., 4096) = 0
4659  09:32:29.488943 close(6)          = 0
4659  09:32:29.488986 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.489038 open("/usr/share/X11/locale/locale.dir", O_RDONLY) = 6
4659  09:32:29.489097 fstat64(6, {st_mode=S_IFREG|0644, st_size=39195, ...}) = 0
4659  09:32:29.489170 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.489218 read(6, "#\t$XdotOrg: lib/X11/nls/locale.di"..., 4096) = 4096
4659  09:32:29.489318 read(6, "ALE:\t\t\teo_EO.ISO8859-3\n# iso8859-"..., 4096) = 4096
4659  09:32:29.489417 read(6, "59-1\niso8859-1/XLC_LOCALE\t\t\tny_NO"..., 4096) = 4096
4659  09:32:29.489507 read(6, "\t\tar_TN.UTF-8\nen_US.UTF-8/XLC_LOC"..., 4096) = 4096
4659  09:32:29.489599 read(6, "/XLC_LOCALE\t\t\tmk_MK.UTF-8\nen_US.U"..., 4096) = 4096
4659  09:32:29.489680 close(6)          = 0
4659  09:32:29.489723 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.489771 access("/usr/share/X11/locale/C/XLC_LOCALE", R_OK) = 0
4659  09:32:29.489831 open("/usr/share/X11/locale/C/XLC_LOCALE", O_RDONLY) = 6
4659  09:32:29.489890 fstat64(6, {st_mode=S_IFREG|0644, st_size=772, ...}) = 0
4659  09:32:29.489963 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x6ff88000
4659  09:32:29.490010 read(6, "#  $Xorg: C,v 1.3 2000/08/17 19:4"..., 4096) = 772
4659  09:32:29.490129 read(6, ""..., 4096) = 0
4659  09:32:29.490173 close(6)          = 0
4659  09:32:29.490217 munmap(0x6ff88000, 4096) = 0
4659  09:32:29.490608 read(4, ""..., 4096) = 0
4659  09:32:29.490665 close(4)          = 0
4659  09:32:29.490709 munmap(0x6ff89000, 4096) = 0
4659  09:32:29.490767 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.490823 writev(3, [{"\220\1\5\0\1\0\1\0\0\0\0\0\0\0\0\0\5\0\5\0\220\1\4\0\1\0\2\0\0\0\0\0\7"..., 356}, {NULL, 0}, {""..., 0}], 3) = 356
4659  09:32:29.496610 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.499184 read(3, "`\1\v\0\1\31\6\0\1\0\22\0\10\377\0\0C\1C\1\0\0\0\0\0\0\0\0\0\0\1C`"..., 4096) = 2464
4659  09:32:29.499699 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.499929 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.500036 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.500150 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.500339 writev(3, [{"\220\10\7\0\1\0\0\0\2\0\0\0\26K\0\0\0\0\0\0\0\0\0\0\0\0\2\0"..., 28}, {NULL, 0}, {""..., 0}], 3) = 28
4659  09:32:29.500496 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:29.504436 read(3, "\1\1\37\0+\1\0\0\0\0\10\377\2\0\0\0\31\26\223\0K\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 1228
4659  09:32:29.504621 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:29.505186 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:29.505244 writev(3, [{"d\1\3\0\246\1\0\0U\377\0\0d\1\3\0\247\1\0\0V\377\0\0v\3\7\0002>\0\0\0"..., 52}, {NULL, 0}, {""..., 0}], 3) = 52
4659  09:32:29.505331 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:35.629358 read(3, "`\1 \0\3720\6\0\1\0\22\0\10\377\0\0\246\1\246\1\0\0\0\0\0\0\0\0\0\0\1\246`"..., 4096) = 416
4659  09:32:35.629460 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:35.629527 select(4, [3], [3], NULL, NULL) = 1 (out [3])
4659  09:32:35.629592 writev(3, [{"<\1\2\0\0\0`\2+\377\1\0"..., 12}, {NULL, 0}, {""..., 0}], 3) = 12
4659  09:32:35.629658 select(4, [3], [], NULL, NULL) = 1 (in [3])
4659  09:32:36.638211 read(3, "\1\1$\0\0\0\0\0$\0\300\1\0\0\0\0\4\0\0\0p\0\0\0p\343%\t\0\0\0\0"..., 4096) = 32
4659  09:32:36.638304 read(3, 0x87e2850, 4096) = -1 EAGAIN (Resource temporarily unavailable)
4659  09:32:36.638373 close(3)          = 0
4659  09:32:36.638473 exit_group(1)     = ?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Thu, Sep 10 2009, Jens Axboe wrote:
> > On Thu, Sep 10 2009, Peter Zijlstra wrote:
> > > On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > > > 
> > > > One thing I also noticed is that when I have logged in, I 
> > > > run xmodmap manually to load some keymappings (I always tell 
> > > > myself to add this to the log in scripts, but I 
> > > > suspend/resume this laptop for weeks at the time and forget 
> > > > before the next boot). With the stock kernel, xmodmap will 
> > > > halt X updates and take forever to run. With BFS, it 
> > > > returned instantly. As I would expect.
> > > 
> > > Can you provide a little more detail (I'm a xmodmap n00b), how does one
> > > run xmodmap and maybe provide your xmodmap config?
> > 
> > Will do, let me get the notebook and strace time it on both bfs and
> > mainline.
> 
> Here's the result of running perf stat xmodmap .xmodmap-carl on 
> the notebook. I have attached the .xmodmap-carl file, it's pretty 
> simple. I have also attached the output of strace -o foo -f -tt 
> xmodmap .xmodmap-carl when run on 2.6.31-rc9.
> 
> 2.6.31-rc9-bfs210
> 
>  Performance counter stats for 'xmodmap .xmodmap-carl':
> 
>      153.994976  task-clock-msecs         #      0.990 CPUs   (scaled from 99.86%)
>               0  context-switches         #      0.000 M/sec  (scaled from 99.86%)
>               0  CPU-migrations           #      0.000 M/sec  (scaled from 99.86%)
>             315  page-faults              #      0.002 M/sec  (scaled from 99.86%)
>   <not counted>  cycles                  
>   <not counted>  instructions            
>   <not counted>  cache-references        
>   <not counted>  cache-misses            
> 
>     0.155573406  seconds time elapsed

(Side question: what hardware is this - why are there no hw 
counters? Could you post the /proc/cpuinfo?)

> 2.6.31-rc9
> 
>  Performance counter stats for 'xmodmap .xmodmap-carl':
> 
>        8.529265  task-clock-msecs         #      0.001 CPUs 
>              23  context-switches         #      0.003 M/sec
>               1  CPU-migrations           #      0.000 M/sec
>             315  page-faults              #      0.037 M/sec
>   <not counted>  cycles                  
>   <not counted>  instructions            
>   <not counted>  cache-references        
>   <not counted>  cache-misses            
> 
>    11.804293482  seconds time elapsed

Thanks - so we context-switch 23 times - possibly to Xorg. But 11 
seconds is extremely long. Will try to reproduce it.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > On Thu, Sep 10 2009, Jens Axboe wrote:
> > > On Thu, Sep 10 2009, Peter Zijlstra wrote:
> > > > On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > > > > 
> > > > > One thing I also noticed is that when I have logged in, I 
> > > > > run xmodmap manually to load some keymappings (I always tell 
> > > > > myself to add this to the log in scripts, but I 
> > > > > suspend/resume this laptop for weeks at the time and forget 
> > > > > before the next boot). With the stock kernel, xmodmap will 
> > > > > halt X updates and take forever to run. With BFS, it 
> > > > > returned instantly. As I would expect.
> > > > 
> > > > Can you provide a little more detail (I'm a xmodmap n00b), how does one
> > > > run xmodmap and maybe provide your xmodmap config?
> > > 
> > > Will do, let me get the notebook and strace time it on both bfs and
> > > mainline.
> > 
> > Here's the result of running perf stat xmodmap .xmodmap-carl on 
> > the notebook. I have attached the .xmodmap-carl file, it's pretty 
> > simple. I have also attached the output of strace -o foo -f -tt 
> > xmodmap .xmodmap-carl when run on 2.6.31-rc9.
> > 
> > 2.6.31-rc9-bfs210
> > 
> >  Performance counter stats for 'xmodmap .xmodmap-carl':
> > 
> >      153.994976  task-clock-msecs         #      0.990 CPUs   (scaled from 99.86%)
> >               0  context-switches         #      0.000 M/sec  (scaled from 99.86%)
> >               0  CPU-migrations           #      0.000 M/sec  (scaled from 99.86%)
> >             315  page-faults              #      0.002 M/sec  (scaled from 99.86%)
> >   <not counted>  cycles                  
> >   <not counted>  instructions            
> >   <not counted>  cache-references        
> >   <not counted>  cache-misses            
> > 
> >     0.155573406  seconds time elapsed
> 
> (Side question: what hardware is this - why are there no hw 
> counters? Could you post the /proc/cpuinfo?)

Sure, attached. It's a Thinkpad x60, core duo. Nothing fancy. The perf
may be a bit dated.

I went to try -tip btw, but it crashes on boot. Here's the backtrace,
typed manually, it's crashing in queue_work_on+0x28/0x60.

Call Trace:
        queue_work
        schedule_work
        clocksource_mark_unstable
        mark_tsc_unstable
        check_tsc_sync_source
        native_cpu_up
        relay_hotcpu_callback
        do_forK_idle
        _cpu_up
        cpu_up
        kernel_init
        kernel_thread_helper

> >  Performance counter stats for 'xmodmap .xmodmap-carl':
> > 
> >        8.529265  task-clock-msecs         #      0.001 CPUs 
> >              23  context-switches         #      0.003 M/sec
> >               1  CPU-migrations           #      0.000 M/sec
> >             315  page-faults              #      0.037 M/sec
> >   <not counted>  cycles                  
> >   <not counted>  instructions            
> >   <not counted>  cache-references        
> >   <not counted>  cache-misses            
> > 
> >    11.804293482  seconds time elapsed
> 
> Thanks - so we context-switch 23 times - possibly to Xorg. But 11 
> seconds is extremely long. Will try to reproduce it.

There's also the strace info with timings. Xorg is definitely involved,
during those 11s things stop updating completely.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> I went to try -tip btw, but it crashes on boot. Here's the 
> backtrace, typed manually, it's crashing in 
> queue_work_on+0x28/0x60.
> 
> Call Trace:
>         queue_work
>         schedule_work
>         clocksource_mark_unstable
>         mark_tsc_unstable
>         check_tsc_sync_source
>         native_cpu_up
>         relay_hotcpu_callback
>         do_forK_idle
>         _cpu_up
>         cpu_up
>         kernel_init
>         kernel_thread_helper

hm, that looks like an old bug i fixed days ago via:

  00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"

Have you tested tip:master - do you still know which sha1?

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Thu, Sep 10 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > I went to try -tip btw, but it crashes on boot. Here's the 
> > backtrace, typed manually, it's crashing in 
> > queue_work_on+0x28/0x60.
> > 
> > Call Trace:
> >         queue_work
> >         schedule_work
> >         clocksource_mark_unstable
> >         mark_tsc_unstable
> >         check_tsc_sync_source
> >         native_cpu_up
> >         relay_hotcpu_callback
> >         do_forK_idle
> >         _cpu_up
> >         cpu_up
> >         kernel_init
> >         kernel_thread_helper
> 
> hm, that looks like an old bug i fixed days ago via:
> 
>   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> 
> Have you tested tip:master - do you still know which sha1?

It was -tip pulled this morning, 2-3 hours ago. I don't have the sha
anymore, but it was a fresh pull today.

-- 
Jens Axboe

[crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > I went to try -tip btw, but it crashes on boot. Here's the 
> > backtrace, typed manually, it's crashing in 
> > queue_work_on+0x28/0x60.
> > 
> > Call Trace:
> >         queue_work
> >         schedule_work
> >         clocksource_mark_unstable
> >         mark_tsc_unstable
> >         check_tsc_sync_source
> >         native_cpu_up
> >         relay_hotcpu_callback
> >         do_forK_idle
> >         _cpu_up
> >         cpu_up
> >         kernel_init
> >         kernel_thread_helper
> 
> hm, that looks like an old bug i fixed days ago via:
> 
>   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> 
> Have you tested tip:master - do you still know which sha1?

Ok, i reproduced it on a testbox and bisected it, the crash is 
caused by:

 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16 is first bad commit
 commit 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16
 Author: Thomas Gleixner <tglx@linutronix.de>
 Date:   Fri Aug 28 20:25:24 2009 +0200

    clocksource: Resolve cpu hotplug dead lock with TSC unstable
    
    Martin Schwidefsky analyzed it:

I've reverted it in tip/master for now.

	Ingo

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > 
> > > I went to try -tip btw, but it crashes on boot. Here's the 
> > > backtrace, typed manually, it's crashing in 
> > > queue_work_on+0x28/0x60.
> > > 
> > > Call Trace:
> > >         queue_work
> > >         schedule_work
> > >         clocksource_mark_unstable
> > >         mark_tsc_unstable
> > >         check_tsc_sync_source
> > >         native_cpu_up
> > >         relay_hotcpu_callback
> > >         do_forK_idle
> > >         _cpu_up
> > >         cpu_up
> > >         kernel_init
> > >         kernel_thread_helper
> > 
> > hm, that looks like an old bug i fixed days ago via:
> > 
> >   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> > 
> > Have you tested tip:master - do you still know which sha1?
> 
> Ok, i reproduced it on a testbox and bisected it, the crash is 
> caused by:
> 
>  7285dd7fd375763bfb8ab1ac9cf3f1206f503c16 is first bad commit
>  commit 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16
>  Author: Thomas Gleixner <tglx@linutronix.de>
>  Date:   Fri Aug 28 20:25:24 2009 +0200
> 
>     clocksource: Resolve cpu hotplug dead lock with TSC unstable
>     
>     Martin Schwidefsky analyzed it:
> 
> I've reverted it in tip/master for now.

and that uncovers the circular locking bug that this commit was 
supposed to fix ...

Martin?

	Ingo

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Martin Schwidefsky]

On Fri, 11 Sep 2009 09:37:47 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > > 
> > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > I went to try -tip btw, but it crashes on boot. Here's the 
> > > > backtrace, typed manually, it's crashing in 
> > > > queue_work_on+0x28/0x60.
> > > > 
> > > > Call Trace:
> > > >         queue_work
> > > >         schedule_work
> > > >         clocksource_mark_unstable
> > > >         mark_tsc_unstable
> > > >         check_tsc_sync_source
> > > >         native_cpu_up
> > > >         relay_hotcpu_callback
> > > >         do_forK_idle
> > > >         _cpu_up
> > > >         cpu_up
> > > >         kernel_init
> > > >         kernel_thread_helper
> > > 
> > > hm, that looks like an old bug i fixed days ago via:
> > > 
> > >   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> > > 
> > > Have you tested tip:master - do you still know which sha1?
> > 
> > Ok, i reproduced it on a testbox and bisected it, the crash is 
> > caused by:
> > 
> >  7285dd7fd375763bfb8ab1ac9cf3f1206f503c16 is first bad commit
> >  commit 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16
> >  Author: Thomas Gleixner <tglx@linutronix.de>
> >  Date:   Fri Aug 28 20:25:24 2009 +0200
> > 
> >     clocksource: Resolve cpu hotplug dead lock with TSC unstable
> >     
> >     Martin Schwidefsky analyzed it:
> > 
> > I've reverted it in tip/master for now.
> 
> and that uncovers the circular locking bug that this commit was 
> supposed to fix ...
> 
> Martin?

Damn, back to running around in circles ..

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Martin Schwidefsky]

On Fri, 11 Sep 2009 09:37:47 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > > 
> > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > I went to try -tip btw, but it crashes on boot. Here's the 
> > > > backtrace, typed manually, it's crashing in 
> > > > queue_work_on+0x28/0x60.
> > > > 
> > > > Call Trace:
> > > >         queue_work
> > > >         schedule_work
> > > >         clocksource_mark_unstable
> > > >         mark_tsc_unstable
> > > >         check_tsc_sync_source
> > > >         native_cpu_up
> > > >         relay_hotcpu_callback
> > > >         do_forK_idle
> > > >         _cpu_up
> > > >         cpu_up
> > > >         kernel_init
> > > >         kernel_thread_helper
> > > 
> > > hm, that looks like an old bug i fixed days ago via:
> > > 
> > >   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> > > 
> > > Have you tested tip:master - do you still know which sha1?
> > 
> > Ok, i reproduced it on a testbox and bisected it, the crash is 
> > caused by:
> > 
> >  7285dd7fd375763bfb8ab1ac9cf3f1206f503c16 is first bad commit
> >  commit 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16
> >  Author: Thomas Gleixner <tglx@linutronix.de>
> >  Date:   Fri Aug 28 20:25:24 2009 +0200
> > 
> >     clocksource: Resolve cpu hotplug dead lock with TSC unstable
> >     
> >     Martin Schwidefsky analyzed it:
> > 
> > I've reverted it in tip/master for now.
> 
> and that uncovers the circular locking bug that this commit was 
> supposed to fix ...
> 
> Martin?

This patch should fix the obvious problem that the watchdog_work
structure is not yet initialized if the clocksource watchdog is not
running yet.
--
Subject: [PATCH] clocksource: statically initialize watchdog workqueue

From: Martin Schwidefsky <schwidefsky@de.ibm.com>

The watchdog timer is started after the watchdog clocksource and at least
one watched clocksource have been registered. The clocksource work element
watchdog_work is initialized just before the clocksource timer is started.
This is too late for the clocksource_mark_unstable call from native_cpu_up.
To fix this use a static initializer for watchdog_work.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
---
 kernel/time/clocksource.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Index: linux-2.6/kernel/time/clocksource.c
===================================================================
--- linux-2.6.orig/kernel/time/clocksource.c
+++ linux-2.6/kernel/time/clocksource.c
@@ -123,10 +123,12 @@ static DEFINE_MUTEX(clocksource_mutex);
 static char override_name[32];
 
 #ifdef CONFIG_CLOCKSOURCE_WATCHDOG
+static void clocksource_watchdog_work(struct work_struct *work);
+
 static LIST_HEAD(watchdog_list);
 static struct clocksource *watchdog;
 static struct timer_list watchdog_timer;
-static struct work_struct watchdog_work;
+static DECLARE_WORK(watchdog_work, clocksource_watchdog_work);
 static DEFINE_SPINLOCK(watchdog_lock);
 static cycle_t watchdog_last;
 static int watchdog_running;
@@ -230,7 +232,6 @@ static inline void clocksource_start_wat
 {
 	if (watchdog_running || !watchdog || list_empty(&watchdog_list))
 		return;
-	INIT_WORK(&watchdog_work, clocksource_watchdog_work);
 	init_timer(&watchdog_timer);
 	watchdog_timer.function = clocksource_watchdog;
 	watchdog_last = watchdog->read(watchdog);

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

[tip:timers/core] clocksource: Resolve cpu hotplug dead lock with TSC unstable, fix crash

[tip-bot for Martin Schwidefsky]

Commit-ID:  f79e0258ea1f04d63db499479b5fb855dff6dbc5
Gitweb:     http://git.kernel.org/tip/f79e0258ea1f04d63db499479b5fb855dff6dbc5
Author:     Martin Schwidefsky <schwidefsky@de.ibm.com>
AuthorDate: Fri, 11 Sep 2009 15:33:05 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Fri, 11 Sep 2009 20:17:18 +0200

clocksource: Resolve cpu hotplug dead lock with TSC unstable, fix crash

The watchdog timer is started after the watchdog clocksource
and at least one watched clocksource have been registered. The
clocksource work element watchdog_work is initialized just
before the clocksource timer is started. This is too late for
the clocksource_mark_unstable call from native_cpu_up. To fix
this use a static initializer for watchdog_work.

This resolves a boot crash reported by multiple people.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Jens Axboe <jens.axboe@oracle.com>
Cc: John Stultz <johnstul@us.ibm.com>
LKML-Reference: <20090911153305.3fe9a361@skybase>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 kernel/time/clocksource.c |    5 +++--
 1 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
index a0af4ff..5697155 100644
--- a/kernel/time/clocksource.c
+++ b/kernel/time/clocksource.c
@@ -123,10 +123,12 @@ static DEFINE_MUTEX(clocksource_mutex);
 static char override_name[32];
 
 #ifdef CONFIG_CLOCKSOURCE_WATCHDOG
+static void clocksource_watchdog_work(struct work_struct *work);
+
 static LIST_HEAD(watchdog_list);
 static struct clocksource *watchdog;
 static struct timer_list watchdog_timer;
-static struct work_struct watchdog_work;
+static DECLARE_WORK(watchdog_work, clocksource_watchdog_work);
 static DEFINE_SPINLOCK(watchdog_lock);
 static cycle_t watchdog_last;
 static int watchdog_running;
@@ -257,7 +259,6 @@ static inline void clocksource_start_watchdog(void)
 {
 	if (watchdog_running || !watchdog || list_empty(&watchdog_list))
 		return;
-	INIT_WORK(&watchdog_work, clocksource_watchdog_work);
 	init_timer(&watchdog_timer);
 	watchdog_timer.function = clocksource_watchdog;
 	watchdog_last = watchdog->read(watchdog);

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Ingo Molnar]

* Martin Schwidefsky <schwidefsky@de.ibm.com> wrote:

> On Fri, 11 Sep 2009 09:37:47 +0200
> Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > > 
> > > * Ingo Molnar <mingo@elte.hu> wrote:
> > > 
> > > > 
> > > > * Jens Axboe <jens.axboe@oracle.com> wrote:
> > > > 
> > > > > I went to try -tip btw, but it crashes on boot. Here's the 
> > > > > backtrace, typed manually, it's crashing in 
> > > > > queue_work_on+0x28/0x60.
> > > > > 
> > > > > Call Trace:
> > > > >         queue_work
> > > > >         schedule_work
> > > > >         clocksource_mark_unstable
> > > > >         mark_tsc_unstable
> > > > >         check_tsc_sync_source
> > > > >         native_cpu_up
> > > > >         relay_hotcpu_callback
> > > > >         do_forK_idle
> > > > >         _cpu_up
> > > > >         cpu_up
> > > > >         kernel_init
> > > > >         kernel_thread_helper
> > > > 
> > > > hm, that looks like an old bug i fixed days ago via:
> > > > 
> > > >   00a3273: Revert "x86: Make tsc=reliable override boot time stability checks"
> > > > 
> > > > Have you tested tip:master - do you still know which sha1?
> > > 
> > > Ok, i reproduced it on a testbox and bisected it, the crash is 
> > > caused by:
> > > 
> > >  7285dd7fd375763bfb8ab1ac9cf3f1206f503c16 is first bad commit
> > >  commit 7285dd7fd375763bfb8ab1ac9cf3f1206f503c16
> > >  Author: Thomas Gleixner <tglx@linutronix.de>
> > >  Date:   Fri Aug 28 20:25:24 2009 +0200
> > > 
> > >     clocksource: Resolve cpu hotplug dead lock with TSC unstable
> > >     
> > >     Martin Schwidefsky analyzed it:
> > > 
> > > I've reverted it in tip/master for now.
> > 
> > and that uncovers the circular locking bug that this commit was 
> > supposed to fix ...
> > 
> > Martin?
> 
> This patch should fix the obvious problem that the watchdog_work
> structure is not yet initialized if the clocksource watchdog is not
> running yet.
> --
> Subject: [PATCH] clocksource: statically initialize watchdog workqueue
> 
> From: Martin Schwidefsky <schwidefsky@de.ibm.com>
> 
> The watchdog timer is started after the watchdog clocksource and at least
> one watched clocksource have been registered. The clocksource work element
> watchdog_work is initialized just before the clocksource timer is started.
> This is too late for the clocksource_mark_unstable call from native_cpu_up.
> To fix this use a static initializer for watchdog_work.
> 
> Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
> ---
>  kernel/time/clocksource.c |    5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> Index: linux-2.6/kernel/time/clocksource.c
> ===================================================================
> --- linux-2.6.orig/kernel/time/clocksource.c
> +++ linux-2.6/kernel/time/clocksource.c
> @@ -123,10 +123,12 @@ static DEFINE_MUTEX(clocksource_mutex);
>  static char override_name[32];
>  
>  #ifdef CONFIG_CLOCKSOURCE_WATCHDOG
> +static void clocksource_watchdog_work(struct work_struct *work);
> +
>  static LIST_HEAD(watchdog_list);
>  static struct clocksource *watchdog;
>  static struct timer_list watchdog_timer;
> -static struct work_struct watchdog_work;
> +static DECLARE_WORK(watchdog_work, clocksource_watchdog_work);
>  static DEFINE_SPINLOCK(watchdog_lock);
>  static cycle_t watchdog_last;
>  static int watchdog_running;
> @@ -230,7 +232,6 @@ static inline void clocksource_start_wat
>  {
>  	if (watchdog_running || !watchdog || list_empty(&watchdog_list))
>  		return;
> -	INIT_WORK(&watchdog_work, clocksource_watchdog_work);
>  	init_timer(&watchdog_timer);
>  	watchdog_timer.function = clocksource_watchdog;
>  	watchdog_last = watchdog->read(watchdog);

Now another box crashes during bootup. Reverting these two:

 f79e025: clocksource: Resolve cpu hotplug dead lock with TSC unstable, fix crash
 7285dd7: clocksource: Resolve cpu hotplug dead lock with TSC unstable

allows me to boot it.

plain 32-bit defconfig.

	Ingo

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Martin Schwidefsky]

On Mon, 14 Sep 2009 17:19:58 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> Now another box crashes during bootup. Reverting these two:
> 
>  f79e025: clocksource: Resolve cpu hotplug dead lock with TSC unstable, fix crash
>  7285dd7: clocksource: Resolve cpu hotplug dead lock with TSC unstable
> 
> allows me to boot it.
> 
> plain 32-bit defconfig.

I've seen the bug report. init_workqueues comes after smp_init.
The idea I'm currently playing with is a simple check in the tsc
code if the tsc clocksource is already registered or not.
When smp_init is called the tsc is not yet registered, we could
just set the rating to zero.

-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: [crash, bisected] Re: clocksource: Resolve cpu hotplug dead lock with TSC unstable

[Martin Schwidefsky]

On Mon, 14 Sep 2009 17:19:58 +0200
Ingo Molnar <mingo@elte.hu> wrote:

> Now another box crashes during bootup. Reverting these two:
> 
>  f79e025: clocksource: Resolve cpu hotplug dead lock with TSC unstable, fix crash
>  7285dd7: clocksource: Resolve cpu hotplug dead lock with TSC unstable
> 
> allows me to boot it.
> 
> plain 32-bit defconfig.

Ok, I forced the situation where the bad thing happens. With the patch below
the crash goes away.

[    0.152056] checking TSC synchronization [CPU#0 -> CPU#1]:
[    0.156001] Measured 0 cycles TSC warp between CPUs, turning off TSC clock.
[    0.156001] Marking TSC unstable due to check_tsc_sync_source failed

Is there a reason why we need the TSC as a clocksource early in the boot
process?

--
Subject: clocksource: delay tsc clocksource registration

From: Martin Schwidefsky <schwidefsky@de.ibm.com>

Until the tsc clocksource has been registered it can be
downgraded by setting the CLOCK_SOURCE_UNSTABLE bit and the
rating to zero. Once the tsc clocksource is registered a
work queue is needed to change the rating.

Delay the registration of the tsc clocksource to a point in
the boot process after the work queues have been initialized.

This hopefully finally resolves the boot crash due to the
tsc downgrade.

Signed-off-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Jens Axboe <jens.axboe@oracle.com>
Cc: John Stultz <johnstul@us.ibm.com>
---

Index: linux-2.6-tip/arch/x86/kernel/tsc.c
===================================================================
--- linux-2.6-tip.orig/arch/x86/kernel/tsc.c	2009-09-14 19:25:02.000000000 +0200
+++ linux-2.6-tip/arch/x86/kernel/tsc.c	2009-09-14 19:30:13.000000000 +0200
@@ -853,9 +853,16 @@
 		clocksource_tsc.rating = 0;
 		clocksource_tsc.flags &= ~CLOCK_SOURCE_IS_CONTINUOUS;
 	}
+}
+
+static int __init register_tsc_clocksource(void)
+{
 	clocksource_register(&clocksource_tsc);
+	return 0;
 }
 
+core_initcall(register_tsc_clocksource);
+
 #ifdef CONFIG_X86_64
 /*
  * calibrate_cpu is used on systems with fixed rate TSCs to determine
-- 
blue skies,
   Martin.

"Reality continues to ruin my life." - Calvin.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> On Wed, 2009-09-09 at 14:20 +0200, Jens Axboe wrote:
> > 
> > One thing I also noticed is that when I have logged in, I run xmodmap
> > manually to load some keymappings (I always tell myself to add this to
> > the log in scripts, but I suspend/resume this laptop for weeks at the
> > time and forget before the next boot). With the stock kernel, xmodmap
> > will halt X updates and take forever to run. With BFS, it returned
> > instantly. As I would expect.
> 
> Can you provide a little more detail (I'm a xmodmap n00b), how 
> does one run xmodmap and maybe provide your xmodmap config?

(and which version did you use, just in case it matters.)

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mike Galbraith]

On Wed, 2009-09-09 at 13:54 +0200, Jens Axboe wrote:

> Things are much better with this patch on the notebook! I cannot compare
> with BFS as that still doesn't run anywhere I want it to run, but it's
> way better than -rc9-git stock. latt numbers on the notebook have 1/3
> the max latency, average is lower, and stddev is much smaller too.

That patch has a bit of bustage in it.

We definitely want to turn down sched_latency though, and LAST_BUDDY
also wants some examination it seems.

taskset -c 3 ./xx 1 (100% cpu 1 sec interval perturbation measurement proggy.  overhead is what it is not getting)
xx says
2392.52 MHZ CPU
perturbation threshold 0.057 usecs.
...
'nuther terminal
taskset -c 3 make -j2 vmlinux

xx output

current (fixed breakage) patched tip tree
pert/s:      153 >18842.18us:       11 min:  0.50 max:36010.37 avg:4354.06 sum/s:666171us overhead:66.62%
pert/s:      160 >18767.18us:       12 min:  0.13 max:32011.66 avg:4172.69 sum/s:667631us overhead:66.66%
pert/s:      156 >18499.43us:        9 min:  0.13 max:27883.24 avg:4296.08 sum/s:670189us overhead:66.49%
pert/s:      146 >18480.71us:       10 min:  0.50 max:32009.38 avg:4615.19 sum/s:673818us overhead:67.26%
pert/s:      154 >18433.20us:       17 min:  0.14 max:31537.12 avg:4474.14 sum/s:689018us overhead:67.68%
pert/s:      158 >18520.11us:        9 min:  0.50 max:34328.86 avg:4275.66 sum/s:675554us overhead:66.76%
pert/s:      154 >18683.74us:       12 min:  0.51 max:35949.23 avg:4363.67 sum/s:672005us overhead:67.04%
pert/s:      154 >18745.53us:        8 min:  0.51 max:34203.43 avg:4399.72 sum/s:677556us overhead:67.03%

bfs209
pert/s:      124 >18681.88us:       17 min:  0.15 max:27274.74 avg:4627.36 sum/s:573793us overhead:56.70%
pert/s:      106 >18702.52us:       20 min:  0.55 max:32022.07 avg:5754.48 sum/s:609975us overhead:59.80%
pert/s:      116 >19082.42us:       17 min:  0.15 max:39835.34 avg:5167.69 sum/s:599452us overhead:59.95%
pert/s:      109 >19289.41us:       22 min:  0.14 max:36818.95 avg:5485.79 sum/s:597951us overhead:59.64%
pert/s:      108 >19238.97us:       19 min:  0.14 max:32026.74 avg:5543.17 sum/s:598662us overhead:59.87%
pert/s:      106 >19415.76us:       20 min:  0.54 max:36011.78 avg:6001.89 sum/s:636201us overhead:62.95%
pert/s:      115 >19341.89us:       16 min:  0.08 max:32040.83 avg:5313.45 sum/s:611047us overhead:59.98%
pert/s:      101 >19527.53us:       24 min:  0.14 max:36018.37 avg:6378.06 sum/s:644184us overhead:64.42%

stock tip (ouch ouch ouch)
pert/s:      153 >48453.23us:        5 min:  0.12 max:144009.85 avg:4688.90 sum/s:717401us overhead:70.89%
pert/s:      172 >47209.49us:        3 min:  0.48 max:68009.05 avg:4022.55 sum/s:691879us overhead:67.05%
pert/s:      148 >51139.18us:        5 min:  0.53 max:168094.76 avg:4918.14 sum/s:727885us overhead:71.65%
pert/s:      171 >51350.64us:        6 min:  0.12 max:102202.79 avg:4304.77 sum/s:736115us overhead:69.24%
pert/s:      153 >57686.54us:        5 min:  0.12 max:224019.85 avg:5399.31 sum/s:826094us overhead:74.50%
pert/s:      172 >55886.47us:        2 min:  0.11 max:75378.18 avg:3993.52 sum/s:686885us overhead:67.67%
pert/s:      157 >58819.31us:        3 min:  0.12 max:165976.63 avg:4453.16 sum/s:699146us overhead:69.91%
pert/s:      149 >58410.21us:        5 min:  0.12 max:104663.89 avg:4792.73 sum/s:714116us overhead:71.41%

sched_latency=20ms min_granularity=4ms
pert/s:      162 >30152.07us:        2 min:  0.49 max:60011.85 avg:4272.97 sum/s:692221us overhead:68.13%
pert/s:      147 >29705.33us:        8 min:  0.14 max:46577.27 avg:4792.03 sum/s:704428us overhead:70.44%
pert/s:      162 >29344.16us:        2 min:  0.49 max:48010.50 avg:4176.75 sum/s:676633us overhead:67.40%
pert/s:      155 >29109.69us:        2 min:  0.49 max:49575.08 avg:4423.87 sum/s:685700us overhead:68.30%
pert/s:      153 >30627.66us:        3 min:  0.13 max:84005.71 avg:4573.07 sum/s:699680us overhead:69.42%
pert/s:      142 >30652.47us:        5 min:  0.49 max:56760.06 avg:4991.61 sum/s:708808us overhead:70.88%
pert/s:      152 >30101.12us:        2 min:  0.49 max:45757.88 avg:4519.92 sum/s:687028us overhead:67.89%
pert/s:      161 >29303.50us:        3 min:  0.12 max:40011.73 avg:4238.15 sum/s:682342us overhead:67.43%

NO_LAST_BUDDY
pert/s:      154 >15257.87us:       28 min:  0.13 max:42004.05 avg:4590.99 sum/s:707013us overhead:70.41%
pert/s:      162 >15392.05us:       34 min:  0.12 max:29021.79 avg:4177.47 sum/s:676750us overhead:66.81%
pert/s:      162 >15665.11us:       33 min:  0.13 max:32008.34 avg:4237.10 sum/s:686410us overhead:67.90%
pert/s:      159 >15914.89us:       31 min:  0.56 max:32056.86 avg:4268.87 sum/s:678751us overhead:67.47%
pert/s:      166 >15858.94us:       26 min:  0.13 max:26655.84 avg:4055.02 sum/s:673134us overhead:66.65%
pert/s:      165 >15878.96us:       32 min:  0.13 max:28010.44 avg:4107.86 sum/s:677798us overhead:66.68%
pert/s:      164 >16213.55us:       29 min:  0.14 max:34263.04 avg:4186.64 sum/s:686610us overhead:68.04%
pert/s:      149 >16764.54us:       20 min:  0.13 max:38688.64 avg:4758.26 sum/s:708981us overhead:70.23%

[tip:sched/core] sched: Turn off child_runs_first

[tip-bot for Mike Galbraith]

Commit-ID:  2bba22c50b06abe9fd0d23933b1e64d35b419262
Gitweb:     http://git.kernel.org/tip/2bba22c50b06abe9fd0d23933b1e64d35b419262
Author:     Mike Galbraith <efault@gmx.de>
AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Wed, 9 Sep 2009 17:30:05 +0200

sched: Turn off child_runs_first

Set child_runs_first default to off.

It hurts 'optimal' make -j<NR_CPUS> workloads as make jobs
get preempted by child tasks, reducing parallelism.

Note, this patch might make existing races in user
applications more prominent than before - so breakages
might be bisected to this commit.

Child-runs-first is broken on SMP to begin with, and we
already had it off briefly in v2.6.23 so most of the
offenders ought to be fixed. Would be nice not to revert
this commit but fix those apps finally ...

Signed-off-by: Mike Galbraith <efault@gmx.de>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1252486344.28645.18.camel@marge.simson.net>
[ made the sysctl independent of CONFIG_SCHED_DEBUG, in case
  people want to work around broken apps. ]
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 include/linux/sched.h |    2 +-
 kernel/sched_fair.c   |    4 ++--
 kernel/sysctl.c       |   16 ++++++++--------
 3 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 3b7f43e..3a50e82 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1820,8 +1820,8 @@ extern unsigned int sysctl_sched_min_granularity;
 extern unsigned int sysctl_sched_wakeup_granularity;
 extern unsigned int sysctl_sched_shares_ratelimit;
 extern unsigned int sysctl_sched_shares_thresh;
-#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_child_runs_first;
+#ifdef CONFIG_SCHED_DEBUG
 extern unsigned int sysctl_sched_features;
 extern unsigned int sysctl_sched_migration_cost;
 extern unsigned int sysctl_sched_nr_migrate;
diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index e386e5d..af325a3 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -48,10 +48,10 @@ unsigned int sysctl_sched_min_granularity = 4000000ULL;
 static unsigned int sched_nr_latency = 5;
 
 /*
- * After fork, child runs first. (default) If set to 0 then
+ * After fork, child runs first. If set to 0 (default) then
  * parent will (try to) run first.
  */
-const_debug unsigned int sysctl_sched_child_runs_first = 1;
+unsigned int sysctl_sched_child_runs_first __read_mostly;
 
 /*
  * sys_sched_yield() compat mode
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index 6c9836e..25d6bf3 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -246,6 +246,14 @@ static int max_wakeup_granularity_ns = NSEC_PER_SEC;	/* 1 second */
 #endif
 
 static struct ctl_table kern_table[] = {
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "sched_child_runs_first",
+		.data		= &sysctl_sched_child_runs_first,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec,
+	},
 #ifdef CONFIG_SCHED_DEBUG
 	{
 		.ctl_name	= CTL_UNNUMBERED,
@@ -300,14 +308,6 @@ static struct ctl_table kern_table[] = {
 	},
 	{
 		.ctl_name	= CTL_UNNUMBERED,
-		.procname	= "sched_child_runs_first",
-		.data		= &sysctl_sched_child_runs_first,
-		.maxlen		= sizeof(unsigned int),
-		.mode		= 0644,
-		.proc_handler	= &proc_dointvec,
-	},
-	{
-		.ctl_name	= CTL_UNNUMBERED,
 		.procname	= "sched_features",
 		.data		= &sysctl_sched_features,
 		.maxlen		= sizeof(unsigned int),

Re: [tip:sched/core] sched: Turn off child_runs_first

[Theodore Tso]

On Wed, Sep 09, 2009 at 03:37:07PM +0000, tip-bot for Mike Galbraith wrote:
> Commit-ID:  2bba22c50b06abe9fd0d23933b1e64d35b419262
> Gitweb:     http://git.kernel.org/tip/2bba22c50b06abe9fd0d23933b1e64d35b419262
> Author:     Mike Galbraith <efault@gmx.de>
> AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Wed, 9 Sep 2009 17:30:05 +0200
> 
> sched: Turn off child_runs_first
> 
> Set child_runs_first default to off.
> 
> It hurts 'optimal' make -j<NR_CPUS> workloads as make jobs
> get preempted by child tasks, reducing parallelism.

Wasn't one of the reasons why we historically did child_runs_first was
so that for fork/exit workloads, the child has a chance to exec the
new process?  If the parent runs first, then more pages will probably
need to be COW'ed.

					- Ted

Re: [tip:sched/core] sched: Turn off child_runs_first

[Ingo Molnar]

* Theodore Tso <tytso@mit.edu> wrote:

> On Wed, Sep 09, 2009 at 03:37:07PM +0000, tip-bot for Mike Galbraith wrote:
> > Commit-ID:  2bba22c50b06abe9fd0d23933b1e64d35b419262
> > Gitweb:     http://git.kernel.org/tip/2bba22c50b06abe9fd0d23933b1e64d35b419262
> > Author:     Mike Galbraith <efault@gmx.de>
> > AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
> > Committer:  Ingo Molnar <mingo@elte.hu>
> > CommitDate: Wed, 9 Sep 2009 17:30:05 +0200
> > 
> > sched: Turn off child_runs_first
> > 
> > Set child_runs_first default to off.
> > 
> > It hurts 'optimal' make -j<NR_CPUS> workloads as make jobs
> > get preempted by child tasks, reducing parallelism.
> 
> Wasn't one of the reasons why we historically did child_runs_first 
> was so that for fork/exit workloads, the child has a chance to 
> exec the new process?  If the parent runs first, then more pages 
> will probably need to be COW'ed.

That kind of workload should be using vfork() anyway, and be even 
faster because it can avoid the fork overhead, right?

Also, on SMP we do that anyway - there's good likelyhood on an idle 
system that we wake the child on the other core straight away.

	Ingo

Re: [tip:sched/core] sched: Turn off child_runs_first

[Chris Friesen]

On 09/09/2009 12:08 PM, Ingo Molnar wrote:
> 
> * Theodore Tso <tytso@mit.edu> wrote:

>> Wasn't one of the reasons why we historically did child_runs_first 
>> was so that for fork/exit workloads, the child has a chance to 
>> exec the new process?  If the parent runs first, then more pages 
>> will probably need to be COW'ed.
> 
> That kind of workload should be using vfork() anyway, and be even 
> faster because it can avoid the fork overhead, right?

According to my man page, POSIX.1-2008 removes the specification  of
vfork().

Chris

Re: [tip:sched/core] sched: Turn off child_runs_first

[Pavel Machek]

Hi!

> > > It hurts 'optimal' make -j<NR_CPUS> workloads as make jobs
> > > get preempted by child tasks, reducing parallelism.
> > 
> > Wasn't one of the reasons why we historically did child_runs_first 
> > was so that for fork/exit workloads, the child has a chance to 
> > exec the new process?  If the parent runs first, then more pages 
> > will probably need to be COW'ed.
> 
> That kind of workload should be using vfork() anyway, and be even 
> faster because it can avoid the fork overhead, right?

Well.. one should not have to update userspace to keep
performance.... and vfork is extremely ugly interface.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[tip:sched/core] sched: Re-tune the scheduler latency defaults to decrease worst-case latencies

[tip-bot for Mike Galbraith]

Commit-ID:  172e082a9111ea504ee34cbba26284a5ebdc53a7
Gitweb:     http://git.kernel.org/tip/172e082a9111ea504ee34cbba26284a5ebdc53a7
Author:     Mike Galbraith <efault@gmx.de>
AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Wed, 9 Sep 2009 17:30:06 +0200

sched: Re-tune the scheduler latency defaults to decrease worst-case latencies

Reduce the latency target from 20 msecs to 5 msecs.

Why? Larger latencies increase spread, which is good for scaling,
but bad for worst case latency.

We still have the ilog(nr_cpus) rule to scale up on bigger
server boxes.

Signed-off-by: Mike Galbraith <efault@gmx.de>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1252486344.28645.18.camel@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 kernel/sched_fair.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index af325a3..26fadb4 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -24,7 +24,7 @@
 
 /*
  * Targeted preemption latency for CPU-bound tasks:
- * (default: 20ms * (1 + ilog(ncpus)), units: nanoseconds)
+ * (default: 5ms * (1 + ilog(ncpus)), units: nanoseconds)
  *
  * NOTE: this latency value is not the same as the concept of
  * 'timeslice length' - timeslices in CFS are of variable length
@@ -34,13 +34,13 @@
  * (to see the precise effective timeslice length of your workload,
  *  run vmstat and monitor the context-switches (cs) field)
  */
-unsigned int sysctl_sched_latency = 20000000ULL;
+unsigned int sysctl_sched_latency = 5000000ULL;
 
 /*
  * Minimal preemption granularity for CPU-bound tasks:
- * (default: 4 msec * (1 + ilog(ncpus)), units: nanoseconds)
+ * (default: 1 msec * (1 + ilog(ncpus)), units: nanoseconds)
  */
-unsigned int sysctl_sched_min_granularity = 4000000ULL;
+unsigned int sysctl_sched_min_granularity = 1000000ULL;
 
 /*
  * is kept at sysctl_sched_latency / sysctl_sched_min_granularity
@@ -63,13 +63,13 @@ unsigned int __read_mostly sysctl_sched_compat_yield;
 
 /*
  * SCHED_OTHER wake-up granularity.
- * (default: 5 msec * (1 + ilog(ncpus)), units: nanoseconds)
+ * (default: 1 msec * (1 + ilog(ncpus)), units: nanoseconds)
  *
  * This option delays the preemption effects of decoupled workloads
  * and reduces their over-scheduling. Synchronous workloads will still
  * have immediate wakeup/sleep latencies.
  */
-unsigned int sysctl_sched_wakeup_granularity = 5000000UL;
+unsigned int sysctl_sched_wakeup_granularity = 1000000UL;
 
 const_debug unsigned int sysctl_sched_migration_cost = 500000UL;
 

Re: [tip:sched/core] sched: Re-tune the scheduler latency defaults to decrease worst-case latencies

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 4623 bytes --]

Am Mittwoch 09 September 2009 schrieb tip-bot for Mike Galbraith:
> Commit-ID:  172e082a9111ea504ee34cbba26284a5ebdc53a7
> Gitweb:    
>  http://git.kernel.org/tip/172e082a9111ea504ee34cbba26284a5ebdc53a7
>  Author:     Mike Galbraith <efault@gmx.de>
> AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
> Committer:  Ingo Molnar <mingo@elte.hu>
> CommitDate: Wed, 9 Sep 2009 17:30:06 +0200
> 
> sched: Re-tune the scheduler latency defaults to decrease worst-case
>  latencies
> 
> Reduce the latency target from 20 msecs to 5 msecs.
> 
> Why? Larger latencies increase spread, which is good for scaling,
> but bad for worst case latency.
> 
> We still have the ilog(nr_cpus) rule to scale up on bigger
> server boxes.
> 
> Signed-off-by: Mike Galbraith <efault@gmx.de>
> Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
> LKML-Reference: <1252486344.28645.18.camel@marge.simson.net>
> Signed-off-by: Ingo Molnar <mingo@elte.hu>
> 
> 
> ---
>  kernel/sched_fair.c |   12 ++++++------
>  1 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
> index af325a3..26fadb4 100644
> --- a/kernel/sched_fair.c
> +++ b/kernel/sched_fair.c
> @@ -24,7 +24,7 @@
> 
>  /*
>   * Targeted preemption latency for CPU-bound tasks:
> - * (default: 20ms * (1 + ilog(ncpus)), units: nanoseconds)
> + * (default: 5ms * (1 + ilog(ncpus)), units: nanoseconds)
>   *
>   * NOTE: this latency value is not the same as the concept of
>   * 'timeslice length' - timeslices in CFS are of variable length
> @@ -34,13 +34,13 @@
>   * (to see the precise effective timeslice length of your workload,
>   *  run vmstat and monitor the context-switches (cs) field)
>   */
> -unsigned int sysctl_sched_latency = 20000000ULL;
> +unsigned int sysctl_sched_latency = 5000000ULL;
> 
>  /*
>   * Minimal preemption granularity for CPU-bound tasks:
> - * (default: 4 msec * (1 + ilog(ncpus)), units: nanoseconds)
> + * (default: 1 msec * (1 + ilog(ncpus)), units: nanoseconds)
>   */
> -unsigned int sysctl_sched_min_granularity = 4000000ULL;
> +unsigned int sysctl_sched_min_granularity = 1000000ULL;

Needs to be lower for a fluid desktop experience here:

shambhala:/proc/sys/kernel> cat sched_min_granularity_ns
100000

> 
>  /*
>   * is kept at sysctl_sched_latency / sysctl_sched_min_granularity
> @@ -63,13 +63,13 @@ unsigned int __read_mostly
>  sysctl_sched_compat_yield;
> 
>  /*
>   * SCHED_OTHER wake-up granularity.
> - * (default: 5 msec * (1 + ilog(ncpus)), units: nanoseconds)
> + * (default: 1 msec * (1 + ilog(ncpus)), units: nanoseconds)
>   *
>   * This option delays the preemption effects of decoupled workloads
>   * and reduces their over-scheduling. Synchronous workloads will still
>   * have immediate wakeup/sleep latencies.
>   */
> -unsigned int sysctl_sched_wakeup_granularity = 5000000UL;
> +unsigned int sysctl_sched_wakeup_granularity = 1000000UL;

Dito:

shambhala:/proc/sys/kernel> cat sched_wakeup_granularity_ns
100000

With

shambhala:~> cat /proc/version
Linux version 2.6.31-rc7-tp42-toi-3.0.1-04741-g57e61c0 (martin@shambhala) 
(gcc version 4.3.3 (Debian 4.3.3-10) ) #6 PREEMPT Sun Aug 23 10:51:32 CEST 
2009

on my ThinkPad T42.

Otherwise compositing animations like switching desktops and zooming in 
newly opening windows still appear jerky. Even with:

shambhala:/sys/kernel/debug> cat sched_features
NO_NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT 
START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK 
NO_DOUBLE_TICK ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD 
NO_WAKEUP_OVERLAP LAST_BUDDY OWNER_SPIN

But NO_NEW_FAIR_SLEEPERS also gives a benefit. It makes those animation 
even more fluent.

In complete I am quity happy with

shambhala:/proc/sys/kernel> grep "" *sched*
sched_child_runs_first:0
sched_compat_yield:0
sched_features:113916
sched_latency_ns:5000000
sched_migration_cost:500000
sched_min_granularity_ns:100000
sched_nr_migrate:32
sched_rt_period_us:1000000
sched_rt_runtime_us:950000
sched_shares_ratelimit:250000
sched_shares_thresh:4
sched_wakeup_granularity_ns:100000

for now.

It really makes a *lot* of difference. But it appears that both 
sched_min_granularity_ns and sched_wakeup_granularity_ns have to be lower 
on my ThinkPad for best effect.

I would still prefer some autotuning, where I say "desktop!" or nothing at 
all. And thats it.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

[tip:sched/core] sched: Keep kthreads at default priority

[tip-bot for Mike Galbraith]

Commit-ID:  61cbe54d9479ad98283b2dda686deae4c34b2d59
Gitweb:     http://git.kernel.org/tip/61cbe54d9479ad98283b2dda686deae4c34b2d59
Author:     Mike Galbraith <efault@gmx.de>
AuthorDate: Wed, 9 Sep 2009 15:41:37 +0200
Committer:  Ingo Molnar <mingo@elte.hu>
CommitDate: Wed, 9 Sep 2009 17:30:06 +0200

sched: Keep kthreads at default priority

Removes kthread/workqueue priority boost, they increase worst-case
desktop latencies.

Signed-off-by: Mike Galbraith <efault@gmx.de>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <1252486344.28645.18.camel@marge.simson.net>
Signed-off-by: Ingo Molnar <mingo@elte.hu>


---
 kernel/kthread.c   |    4 ----
 kernel/workqueue.c |    2 --
 2 files changed, 0 insertions(+), 6 deletions(-)

diff --git a/kernel/kthread.c b/kernel/kthread.c
index eb8751a..5fe7099 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -16,8 +16,6 @@
 #include <linux/mutex.h>
 #include <trace/events/sched.h>
 
-#define KTHREAD_NICE_LEVEL (-5)
-
 static DEFINE_SPINLOCK(kthread_create_lock);
 static LIST_HEAD(kthread_create_list);
 struct task_struct *kthreadd_task;
@@ -145,7 +143,6 @@ struct task_struct *kthread_create(int (*threadfn)(void *data),
 		 * The kernel thread should not inherit these properties.
 		 */
 		sched_setscheduler_nocheck(create.result, SCHED_NORMAL, &param);
-		set_user_nice(create.result, KTHREAD_NICE_LEVEL);
 		set_cpus_allowed_ptr(create.result, cpu_all_mask);
 	}
 	return create.result;
@@ -221,7 +218,6 @@ int kthreadd(void *unused)
 	/* Setup a clean context for our children to inherit. */
 	set_task_comm(tsk, "kthreadd");
 	ignore_signals(tsk);
-	set_user_nice(tsk, KTHREAD_NICE_LEVEL);
 	set_cpus_allowed_ptr(tsk, cpu_all_mask);
 	set_mems_allowed(node_possible_map);
 
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 0668795..ea1b4e7 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -317,8 +317,6 @@ static int worker_thread(void *__cwq)
 	if (cwq->wq->freezeable)
 		set_freezable();
 
-	set_user_nice(current, -5);
-
 	for (;;) {
 		prepare_to_wait(&cwq->more_work, &wait, TASK_INTERRUPTIBLE);
 		if (!freezing(current) &&

Re: [tip:sched/core] sched: Keep kthreads at default priority

[Dmitry Torokhov]

On Wed, Sep 09, 2009 at 03:37:34PM +0000, tip-bot for Mike Galbraith wrote:
> 
> diff --git a/kernel/kthread.c b/kernel/kthread.c
> index eb8751a..5fe7099 100644
> --- a/kernel/kthread.c
> +++ b/kernel/kthread.c
> @@ -16,8 +16,6 @@
>  #include <linux/mutex.h>
>  #include <trace/events/sched.h>
>  
> -#define KTHREAD_NICE_LEVEL (-5)
> -

Why don't we just redefine it to 0? We may find out later that we'd
still prefer to have kernel threads have boost.

-- 
Dmitry

Re: [tip:sched/core] sched: Keep kthreads at default priority

[Peter Zijlstra]

On Wed, 2009-09-09 at 09:55 -0700, Dmitry Torokhov wrote:
> On Wed, Sep 09, 2009 at 03:37:34PM +0000, tip-bot for Mike Galbraith wrote:
> > 
> > diff --git a/kernel/kthread.c b/kernel/kthread.c
> > index eb8751a..5fe7099 100644
> > --- a/kernel/kthread.c
> > +++ b/kernel/kthread.c
> > @@ -16,8 +16,6 @@
> >  #include <linux/mutex.h>
> >  #include <trace/events/sched.h>
> >  
> > -#define KTHREAD_NICE_LEVEL (-5)
> > -
> 
> Why don't we just redefine it to 0? We may find out later that we'd
> still prefer to have kernel threads have boost.

Seems sensible, also the traditional reasoning behind this nice level is
that kernel threads do work on behalf of multiple tasks. Its a kind of
prio ceiling thing.

Re: [tip:sched/core] sched: Keep kthreads at default priority

[Mike Galbraith]

On Wed, 2009-09-09 at 19:06 +0200, Peter Zijlstra wrote:
> On Wed, 2009-09-09 at 09:55 -0700, Dmitry Torokhov wrote:
> > On Wed, Sep 09, 2009 at 03:37:34PM +0000, tip-bot for Mike Galbraith wrote:
> > > 
> > > diff --git a/kernel/kthread.c b/kernel/kthread.c
> > > index eb8751a..5fe7099 100644
> > > --- a/kernel/kthread.c
> > > +++ b/kernel/kthread.c
> > > @@ -16,8 +16,6 @@
> > >  #include <linux/mutex.h>
> > >  #include <trace/events/sched.h>
> > >  
> > > -#define KTHREAD_NICE_LEVEL (-5)
> > > -
> > 
> > Why don't we just redefine it to 0? We may find out later that we'd
> > still prefer to have kernel threads have boost.
> 
> Seems sensible, also the traditional reasoning behind this nice level is
> that kernel threads do work on behalf of multiple tasks. Its a kind of
> prio ceiling thing.

True.  None of our current threads are heavy enough to matter much.

	-Mike

Re: [tip:sched/core] sched: Keep kthreads at default priority

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 1251 bytes --]

Am Mittwoch 09 September 2009 schrieb Mike Galbraith:
> On Wed, 2009-09-09 at 19:06 +0200, Peter Zijlstra wrote:
> > On Wed, 2009-09-09 at 09:55 -0700, Dmitry Torokhov wrote:
> > > On Wed, Sep 09, 2009 at 03:37:34PM +0000, tip-bot for Mike Galbraith 
wrote:
> > > > diff --git a/kernel/kthread.c b/kernel/kthread.c
> > > > index eb8751a..5fe7099 100644
> > > > --- a/kernel/kthread.c
> > > > +++ b/kernel/kthread.c
> > > > @@ -16,8 +16,6 @@
> > > >  #include <linux/mutex.h>
> > > >  #include <trace/events/sched.h>
> > > >
> > > > -#define KTHREAD_NICE_LEVEL (-5)
> > > > -
> > >
> > > Why don't we just redefine it to 0? We may find out later that we'd
> > > still prefer to have kernel threads have boost.
> >
> > Seems sensible, also the traditional reasoning behind this nice level
> > is that kernel threads do work on behalf of multiple tasks. Its a
> > kind of prio ceiling thing.
> 
> True.  None of our current threads are heavy enough to matter much.

Does it make sense to have this as a tunable? Where does it matter? Server 
workloads?

(Oh no, not another tunable I can hear you yell;-).

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [tip:sched/core] sched: Keep kthreads at default priority

[Mike Galbraith]

On Sat, 2009-09-12 at 13:48 +0200, Martin Steigerwald wrote:
> Am Mittwoch 09 September 2009 schrieb Mike Galbraith:
> > On Wed, 2009-09-09 at 19:06 +0200, Peter Zijlstra wrote:
> > > On Wed, 2009-09-09 at 09:55 -0700, Dmitry Torokhov wrote:
> > > > On Wed, Sep 09, 2009 at 03:37:34PM +0000, tip-bot for Mike Galbraith 
> wrote:
> > > > > diff --git a/kernel/kthread.c b/kernel/kthread.c
> > > > > index eb8751a..5fe7099 100644
> > > > > --- a/kernel/kthread.c
> > > > > +++ b/kernel/kthread.c
> > > > > @@ -16,8 +16,6 @@
> > > > >  #include <linux/mutex.h>
> > > > >  #include <trace/events/sched.h>
> > > > >
> > > > > -#define KTHREAD_NICE_LEVEL (-5)
> > > > > -
> > > >
> > > > Why don't we just redefine it to 0? We may find out later that we'd
> > > > still prefer to have kernel threads have boost.
> > >
> > > Seems sensible, also the traditional reasoning behind this nice level
> > > is that kernel threads do work on behalf of multiple tasks. Its a
> > > kind of prio ceiling thing.
> > 
> > True.  None of our current threads are heavy enough to matter much.
> 
> Does it make sense to have this as a tunable? Where does it matter? Server 
> workloads?

I don't think it should be a knob.  It only makes a difference to
kthreads that are heavy CPU users.  If one pops up as a performance
problem, IMHO, it should be tweaked separately.  Running at default
weight saves a bit of unnecessary math for the common case.

	-Mike

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 06:23 PM, Peter Zijlstra wrote:
> On Tue, 2009-09-08 at 11:13 +0200, Jens Axboe wrote:
>> And here's a newer version.
>
> I tinkered a bit with your proglet and finally found the problem.
>
> You used a single pipe per child, this means the loop in run_child()
> would consume what it just wrote out until it got force preempted by the
> parent which would also get woken.
>
> This results in the child spinning a while (its full quota) and only
> reporting the last timestamp to the parent.
>
> Since consumer (parent) is a single thread the program basically
> measures the worst delay in a thundering herd wakeup of N children.
>
> The below version yields:
>
> idle
>
> [root@opteron sched]# ./latt -c8 sleep 30
> Entries: 664 (clients=8)
>
> Averages:
> ------------------------------
>          Max           128 usec
>          Avg            26 usec
>          Stdev          16 usec
>
>
> make -j4
>
> [root@opteron sched]# ./latt -c8 sleep 30
> Entries: 648 (clients=8)
>
> Averages:
> ------------------------------
>          Max         20861 usec
>          Avg          3763 usec
>          Stdev        4637 usec
>
>
> Mike's patch, make -j4
>
> [root@opteron sched]# ./latt -c8 sleep 30
> Entries: 648 (clients=8)
>
> Averages:
> ------------------------------
>          Max         17854 usec
>          Avg          6298 usec
>          Stdev        4735 usec

I've run two tests with this tool.  One with mainline (2.6.31-rc9) and 
one patched with 2.6.31-rc9-sched-bfs-210.patch.

Before running this test, I disabled the cron daemon in order not to 
have something pop-up in the background out of a sudden.

The test consisted of starting a "make -j2" in the kernel tree inside a 
3GB tmpfs mountpoint and then running 'latt "mplayer -vo gl2 -framedrop 
videofile.mkv"'  (mplayer in this case is a single-threaded 
application.)  Caches were warmed-up first; the results below are from 
the second run of each test.

The kernel .config file used by the running kernels and also for "make 
-j2" is:

   http://foss.math.aegean.gr/~realnc/kernel/config-2.6.31-rc9-latt-test

The video file used for mplayer is:

   http://foss.math.aegean.gr/~realnc/vids/3DMark2000.mkv (100MB)
   (The reason this was used is that it's a 60FPS video,
   therefore very smooth and makes all skips stand out
   clearly.)


Results for mainline:

   Averages:
   ------------------------------
           Max         29930 usec
           Avg         11043 usec
           Stdev        5752 usec


Results for BFS:

   Averages:
   ------------------------------
           Max         14017 usec
           Avg            49 usec
           Stdev         697 usec


One thing that's worth noting is that with mainline, mplayer would 
occasionally spit this out:

    YOUR SYSTEM IS TOO SLOW TO PLAY THIS

which doesn't happen with BFS.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Avi Kivity]

On 09/07/2009 12:49 PM, Jens Axboe wrote:
>
> I ran a simple test as well, since I was curious to see how it performed
> wrt interactiveness. One of my pet peeves with the current scheduler is
> that I have to nice compile jobs, or my X experience is just awful while
> the compile is running.
>    

I think the problem is that CFS is optimizing for the wrong thing.  It's 
trying to be fair to tasks, but these are meaningless building blocks of 
jobs, which is what the user sees and measures.  Your make -j128 
dominates your interactive task by two orders of magnitude.  If the 
scheduler attempts to bridge this gap using heuristics, it will fail 
badly when it misdetects since it will starve the really important 
100-thread job for a task that was misdetected as interactive.

I think that bash (and the GUI shell) should put any new job (for bash, 
a pipeline; for the GUI, an application launch from the menu) in a 
scheduling group of its own.  This way it will have equal weight in the 
scheduler's eyes with interactive tasks; one will not dominate the 
other.  Of course if the cpu is free the compile job is welcome to use 
all 128 threads.

(similarly, different login sessions should be placed in different jobs 
to avoid a heavily multithreaded screensaver from overwhelming ed).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Avi Kivity wrote:
> On 09/07/2009 12:49 PM, Jens Axboe wrote:
>>
>> I ran a simple test as well, since I was curious to see how it performed
>> wrt interactiveness. One of my pet peeves with the current scheduler is
>> that I have to nice compile jobs, or my X experience is just awful while
>> the compile is running.
>>    
>
> I think the problem is that CFS is optimizing for the wrong thing.  It's  
> trying to be fair to tasks, but these are meaningless building blocks of  
> jobs, which is what the user sees and measures.  Your make -j128  
> dominates your interactive task by two orders of magnitude.  If the  
> scheduler attempts to bridge this gap using heuristics, it will fail  
> badly when it misdetects since it will starve the really important  
> 100-thread job for a task that was misdetected as interactive.

Agree, I was actually looking into doing joint latency for X number of
tasks for the test app. I'll try and do that and see if we can detect
something from that.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> Agree, I was actually looking into doing joint latency for X 
> number of tasks for the test app. I'll try and do that and see if 
> we can detect something from that.

Could you please try latest -tip:

   http://people.redhat.com/mingo/tip.git/README

(c26f010 or later)

Does it get any better with make -j128 build jobs? Peter just fixed 
a bug in the SMP load-balancer that can cause interactivity problems 
on large CPU count systems.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Ingo Molnar wrote:
> 
> * Jens Axboe <jens.axboe@oracle.com> wrote:
> 
> > Agree, I was actually looking into doing joint latency for X 
> > number of tasks for the test app. I'll try and do that and see if 
> > we can detect something from that.
> 
> Could you please try latest -tip:
> 
>    http://people.redhat.com/mingo/tip.git/README
> 
> (c26f010 or later)
> 
> Does it get any better with make -j128 build jobs? Peter just fixed 

The compile 'problem' is on my workstation, which is a dual core Intel
core 2. I use -j4 on that typically. On the bigger boxes, I don't notice
any interactivity problems, largely because I don't run anything latency
sensitive on those :-)

> a bug in the SMP load-balancer that can cause interactivity problems 
> on large CPU count systems.

Worth trying on the dual core box?

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Peter Zijlstra]

On Mon, 2009-09-07 at 22:46 +0200, Jens Axboe wrote:
> > a bug in the SMP load-balancer that can cause interactivity problems 
> > on large CPU count systems.
> 
> Worth trying on the dual core box?

I debugged the issue on a dual core :-)

It should be more pronounced on larger machines, but its present on
dual-core too.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Jens Axboe]

On Mon, Sep 07 2009, Peter Zijlstra wrote:
> On Mon, 2009-09-07 at 22:46 +0200, Jens Axboe wrote:
> > > a bug in the SMP load-balancer that can cause interactivity problems 
> > > on large CPU count systems.
> > 
> > Worth trying on the dual core box?
> 
> I debugged the issue on a dual core :-)
> 
> It should be more pronounced on larger machines, but its present on
> dual-core too.

Alright, I'll upgrade that box to -tip tomorrow and see if it makes
a noticable difference. At -j4 or higher, I can literally see windows
slowly popping up when switching to a different virtual desktop.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Jens Axboe <jens.axboe@oracle.com> wrote:

> On Mon, Sep 07 2009, Peter Zijlstra wrote:
> > On Mon, 2009-09-07 at 22:46 +0200, Jens Axboe wrote:
> > > > a bug in the SMP load-balancer that can cause interactivity problems 
> > > > on large CPU count systems.
> > > 
> > > Worth trying on the dual core box?
> > 
> > I debugged the issue on a dual core :-)
> > 
> > It should be more pronounced on larger machines, but its present on
> > dual-core too.
> 
> Alright, I'll upgrade that box to -tip tomorrow and see if it 
> makes a noticable difference. At -j4 or higher, I can literally 
> see windows slowly popping up when switching to a different 
> virtual desktop.

btw., if you run -tip and have these enabled:

  CONFIG_PERF_COUNTER=y 
  CONFIG_EVENT_TRACING=y

  cd tools/perf/
  make -j install

... then you can use a couple of new perfcounters features to 
measure scheduler latencies. For example:

  perf stat -e sched:sched_stat_wait -e task-clock ./hackbench 20

Will tell you how many times this workload got delayed by waiting 
for CPU time.

You can repeat the workload as well and see the statistical 
properties of those metrics:

 aldebaran:/home/mingo> perf stat --repeat 10 -e \
              sched:sched_stat_wait:r -e task-clock ./hackbench 20
 Time: 0.251
 Time: 0.214
 Time: 0.254
 Time: 0.278
 Time: 0.245
 Time: 0.308
 Time: 0.242
 Time: 0.222
 Time: 0.268
 Time: 0.244

 Performance counter stats for './hackbench 20' (10 runs):

          59826  sched:sched_stat_wait    #      0.026 M/sec   ( +-   5.540% )
    2280.099643  task-clock-msecs         #      7.525 CPUs    ( +-   1.620% )

    0.303013390  seconds time elapsed   ( +-   3.189% )

To get scheduling events, do:

 # perf list 2>&1 | grep sched:
  sched:sched_kthread_stop                   [Tracepoint event]
  sched:sched_kthread_stop_ret               [Tracepoint event]
  sched:sched_wait_task                      [Tracepoint event]
  sched:sched_wakeup                         [Tracepoint event]
  sched:sched_wakeup_new                     [Tracepoint event]
  sched:sched_switch                         [Tracepoint event]
  sched:sched_migrate_task                   [Tracepoint event]
  sched:sched_process_free                   [Tracepoint event]
  sched:sched_process_exit                   [Tracepoint event]
  sched:sched_process_wait                   [Tracepoint event]
  sched:sched_process_fork                   [Tracepoint event]
  sched:sched_signal_send                    [Tracepoint event]
  sched:sched_stat_wait                      [Tracepoint event]
  sched:sched_stat_sleep                     [Tracepoint event]
  sched:sched_stat_iowait                    [Tracepoint event]

stat_wait/sleep/iowait would be the interesting ones, for latency 
analysis.

Or, if you want to see all the specific delays and want to see 
min/max/avg, you can do:

  perf record -e sched:sched_stat_wait:r -f -R -c 1 ./hackbench 20
  perf trace

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Pavel Machek]

Hi!

> > So ... to get to the numbers - i've tested both BFS and the tip of 
> > the latest upstream scheduler tree on a testbox of mine. I 
> > intentionally didnt test BFS on any really large box - because you 
> > described its upper limit like this in the announcement:
> 
> I ran a simple test as well, since I was curious to see how it performed
> wrt interactiveness. One of my pet peeves with the current scheduler is
> that I have to nice compile jobs, or my X experience is just awful while
> the compile is running.
> 
> Now, this test case is something that attempts to see what
> interactiveness would be like. It'll run a given command line while at
> the same time logging delays. The delays are measured as follows:
> 
> - The app creates a pipe, and forks a child that blocks on reading from
>   that pipe.
> - The app sleeps for a random period of time, anywhere between 100ms
>   and 2s. When it wakes up, it gets the current time and writes that to
>   the pipe.
> - The child then gets woken, checks the time on its own, and logs the
>   difference between the two.
> 
> The idea here being that the delay between writing to the pipe and the
> child reading the data and comparing should (in some way) be indicative
> of how responsive the system would seem to a user.
> 
> The test app was quickly hacked up, so don't put too much into it. The
> test run is a simple kernel compile, using -jX where X is the number of
> threads in the system. The files are cache hot, so little IO is done.
> The -x2 run is using the double number of processes as we have threads,
> eg -j128 on a 64 thread box.

Could you post the source? Someone else might get us
numbers... preferably on dualcore box or something...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

latt location (Was Re: BFS vs. mainline scheduler benchmarks and measurements)

[Jens Axboe]

On Wed, Sep 09 2009, Pavel Machek wrote:
> Could you post the source? Someone else might get us
> numbers... preferably on dualcore box or something...

Since it's posted in various places and by various people, I've put it
on the web now as well. Should always be the latest version.

http://kernel.dk/latt.c

Note that it requires zlib-devel packages to build now.

-- 
Jens Axboe

Re: BFS vs. mainline scheduler benchmarks and measurements

[Michael Buesch]

Here's a very simple test setup on an embedded singlecore bcm47xx machine (WL500GPv2)
It uses iperf for performance testing. The iperf server is run on the
embedded device. The device is so slow that the iperf test is completely
CPU bound. The network connection is a 100MBit on the device connected
via patch cable to a 1000MBit machine.

The kernel is openwrt-2.6.30.5.

Here are the results:



Mainline CFS scheduler:

mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 35793 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  27.4 MBytes  23.0 Mbits/sec
mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 35794 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  27.3 MBytes  22.9 Mbits/sec
mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 56147 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  27.3 MBytes  22.9 Mbits/sec


BFS scheduler:

mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 52489 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  38.2 MBytes  32.0 Mbits/sec
mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 52490 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  38.1 MBytes  31.9 Mbits/sec
mb@homer:~$ iperf -c 192.168.1.1
------------------------------------------------------------
Client connecting to 192.168.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.99 port 52491 connected with 192.168.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  38.1 MBytes  31.9 Mbits/sec


-- 
Greetings, Michael.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Michael Buesch <mb@bu3sch.de> wrote:

> Here's a very simple test setup on an embedded singlecore bcm47xx 
> machine (WL500GPv2) It uses iperf for performance testing. The 
> iperf server is run on the embedded device. The device is so slow 
> that the iperf test is completely CPU bound. The network 
> connection is a 100MBit on the device connected via patch cable to 
> a 1000MBit machine.
> 
> The kernel is openwrt-2.6.30.5.
> 
> Here are the results:
> 
> 
> 
> Mainline CFS scheduler:
> 
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 35793 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  27.4 MBytes  23.0 Mbits/sec
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 35794 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  27.3 MBytes  22.9 Mbits/sec
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 56147 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  27.3 MBytes  22.9 Mbits/sec
> 
> 
> BFS scheduler:
> 
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 52489 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  38.2 MBytes  32.0 Mbits/sec
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 52490 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  38.1 MBytes  31.9 Mbits/sec
> mb@homer:~$ iperf -c 192.168.1.1
> ------------------------------------------------------------
> Client connecting to 192.168.1.1, TCP port 5001
> TCP window size: 16.0 KByte (default)
> ------------------------------------------------------------
> [  3] local 192.168.1.99 port 52491 connected with 192.168.1.1 port 5001
> [ ID] Interval       Transfer     Bandwidth
> [  3]  0.0-10.0 sec  38.1 MBytes  31.9 Mbits/sec

That's interesting. I tried to reproduce it on x86, but the profile 
does not show any scheduler overhead at all on the server:

$ perf report

#
# Samples: 8369
#
# Overhead  Symbol
# ........  ......
#
     9.20%  [k] copy_user_generic_string
     3.80%  [k] e1000_clean
     3.58%  [k] ipt_do_table
     2.72%  [k] mwait_idle
     2.68%  [k] nf_iterate
     2.28%  [k] e1000_intr
     2.15%  [k] tcp_packet
     2.10%  [k] __hash_conntrack
     1.59%  [k] read_tsc
     1.52%  [k] _local_bh_enable_ip
     1.34%  [k] eth_type_trans
     1.29%  [k] __alloc_skb
     1.19%  [k] tcp_recvmsg
     1.19%  [k] ip_rcv
     1.17%  [k] e1000_clean_rx_irq
     1.12%  [k] apic_timer_interrupt
     0.99%  [k] vsnprintf
     0.96%  [k] nf_conntrack_in
     0.96%  [k] kmem_cache_free
     0.93%  [k] __kmalloc_track_caller


Could you profile it please? Also, what's the context-switch rate?

Below is the call-graph profile as well - all the overhead is in 
networking and SLAB.

	Ingo

 $ perf report --call-graph fractal,5

#
# Samples: 8947
#
# Overhead         Command                  Shared Object  Symbol
# ........  ..............  .............................  ......
#
     9.06%           iperf  [kernel]                       [k] copy_user_generic_string
                |          
                |--98.89%-- skb_copy_datagram_iovec
                |          |          
                |          |--77.18%-- tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --22.82%-- tcp_rcv_established
                |                     tcp_v4_do_rcv
                |                     tcp_prequeue_process
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                 --1.11%-- system_call_fastpath
                           __GI___libc_nanosleep

     3.62%          [init]  [kernel]                       [k] e1000_clean
     2.96%          [init]  [kernel]                       [k] ipt_do_table
     2.79%          [init]  [kernel]                       [k] mwait_idle
     2.22%          [init]  [kernel]                       [k] e1000_intr
     1.93%          [init]  [kernel]                       [k] nf_iterate
     1.65%          [init]  [kernel]                       [k] __hash_conntrack
     1.52%          [init]  [kernel]                       [k] tcp_packet
     1.29%          [init]  [kernel]                       [k] ip_rcv
     1.18%          [init]  [kernel]                       [k] __alloc_skb
     1.15%           iperf  [kernel]                       [k] tcp_recvmsg

     1.04%          [init]  [kernel]                       [k] _local_bh_enable_ip
     1.02%          [init]  [kernel]                       [k] apic_timer_interrupt
     1.02%          [init]  [kernel]                       [k] eth_type_trans
     1.01%          [init]  [kernel]                       [k] tcp_v4_rcv
     0.96%           iperf  [kernel]                       [k] kfree
                |          
                |--95.35%-- skb_release_data
                |          __kfree_skb
                |          |          
                |          |--79.27%-- tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --20.73%-- tcp_rcv_established
                |                     tcp_v4_do_rcv
                |                     tcp_prequeue_process
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                 --4.65%-- __kfree_skb
                           |          
                           |--75.00%-- tcp_rcv_established
                           |          tcp_v4_do_rcv
                           |          tcp_prequeue_process
                           |          tcp_recvmsg
                           |          sock_common_recvmsg
                           |          __sock_recvmsg
                           |          sock_recvmsg
                           |          sys_recvfrom
                           |          system_call_fastpath
                           |          __recv
                           |          
                            --25.00%-- tcp_recvmsg
                                      sock_common_recvmsg
                                      __sock_recvmsg
                                      sock_recvmsg
                                      sys_recvfrom
                                      system_call_fastpath
                                      __recv

     0.96%          [init]  [kernel]                       [k] read_tsc
     0.92%           iperf  [kernel]                       [k] tcp_v4_do_rcv
                |          
                |--95.12%-- tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --4.88%-- tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.92%          [init]  [kernel]                       [k] e1000_clean_rx_irq
     0.86%           iperf  [kernel]                       [k] tcp_rcv_established
                |          
                |--96.10%-- tcp_v4_do_rcv
                |          tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --3.90%-- tcp_prequeue_process
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.84%           iperf  [kernel]                       [k] kmem_cache_free
                |          
                |--93.33%-- __kfree_skb
                |          |          
                |          |--71.43%-- tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --28.57%-- tcp_rcv_established
                |                     tcp_v4_do_rcv
                |                     tcp_prequeue_process
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--4.00%-- tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --2.67%-- tcp_rcv_established
                           tcp_v4_do_rcv
                           tcp_prequeue_process
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.80%          [init]  [kernel]                       [k] netif_receive_skb
     0.79%           iperf  [kernel]                       [k] tcp_event_data_recv
                |          
                |--83.10%-- tcp_rcv_established
                |          tcp_v4_do_rcv
                |          tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                |--12.68%-- tcp_v4_do_rcv
                |          tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --4.23%-- tcp_data_queue
                           tcp_rcv_established
                           tcp_v4_do_rcv
                           tcp_prequeue_process
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.67%            perf  [kernel]                       [k] format_decode
                |          
                |--91.67%-- vsnprintf
                |          seq_printf
                |          |          
                |          |--67.27%-- show_map_vma
                |          |          show_map
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--23.64%-- render_sigset_t
                |          |          proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--7.27%-- proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |           --1.82%-- cpuset_task_status_allowed
                |                     proc_pid_status
                |                     proc_single_show
                |                     seq_read
                |                     vfs_read
                |                     sys_read
                |                     system_call_fastpath
                |                     __GI_read
                |          
                 --8.33%-- seq_printf
                           |          
                           |--60.00%-- proc_pid_status
                           |          proc_single_show
                           |          seq_read
                           |          vfs_read
                           |          sys_read
                           |          system_call_fastpath
                           |          __GI_read
                           |          
                            --40.00%-- show_map_vma
                                      show_map
                                      seq_read
                                      vfs_read
                                      sys_read
                                      system_call_fastpath
                                      __GI_read

     0.65%          [init]  [kernel]                       [k] __kmalloc_track_caller
     0.63%          [init]  [kernel]                       [k] nf_conntrack_in
     0.63%          [init]  [kernel]                       [k] ip_route_input
     0.58%            perf  [kernel]                       [k] vsnprintf
                |          
                |--98.08%-- seq_printf
                |          |          
                |          |--60.78%-- show_map_vma
                |          |          show_map
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--19.61%-- render_sigset_t
                |          |          proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--9.80%-- proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--3.92%-- task_mem
                |          |          proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |          |--3.92%-- cpuset_task_status_allowed
                |          |          proc_pid_status
                |          |          proc_single_show
                |          |          seq_read
                |          |          vfs_read
                |          |          sys_read
                |          |          system_call_fastpath
                |          |          __GI_read
                |          |          
                |           --1.96%-- render_cap_t
                |                     proc_pid_status
                |                     proc_single_show
                |                     seq_read
                |                     vfs_read
                |                     sys_read
                |                     system_call_fastpath
                |                     __GI_read
                |          
                 --1.92%-- snprintf
                           proc_task_readdir
                           vfs_readdir
                           sys_getdents
                           system_call_fastpath
                           __getdents64
                           0x69706565000a3430

     0.57%          [init]  [kernel]                       [k] ktime_get
     0.57%          [init]  [kernel]                       [k] nf_nat_fn
     0.56%           iperf  [kernel]                       [k] tcp_packet
                |          
                |--68.00%-- __tcp_ack_snd_check
                |          tcp_rcv_established
                |          tcp_v4_do_rcv
                |          tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --32.00%-- tcp_cleanup_rbuf
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.56%           iperf  /usr/bin/iperf                 [.] 0x000000000059f8
                |          
                |--8.00%-- 0x4059f8
                |          
                |--8.00%-- 0x405a16
                |          
                |--8.00%-- 0x4059fd
                |          
                |--4.00%-- 0x409d22
                |          
                |--4.00%-- 0x405871
                |          
                |--4.00%-- 0x406ee1
                |          
                |--4.00%-- 0x405726
                |          
                |--4.00%-- 0x4058db
                |          
                |--4.00%-- 0x406ee8
                |          
                |--2.00%-- 0x405b60
                |          
                |--2.00%-- 0x4058fd
                |          
                |--2.00%-- 0x4058d5
                |          
                |--2.00%-- 0x405490
                |          
                |--2.00%-- 0x4058bb
                |          
                |--2.00%-- 0x405b93
                |          
                |--2.00%-- 0x405b8e
                |          
                |--2.00%-- 0x405903
                |          
                |--2.00%-- 0x405ba8
                |          
                |--2.00%-- 0x406eae
                |          
                |--2.00%-- 0x405545
                |          
                |--2.00%-- 0x405870
                |          
                |--2.00%-- 0x405b67
                |          
                |--2.00%-- 0x4058ce
                |          
                |--2.00%-- 0x40570e
                |          
                |--2.00%-- 0x406ee4
                |          
                |--2.00%-- 0x405a02
                |          
                |--2.00%-- 0x406eec
                |          
                |--2.00%-- 0x405b82
                |          
                |--2.00%-- 0x40556a
                |          
                |--2.00%-- 0x405755
                |          
                |--2.00%-- 0x405a0a
                |          
                |--2.00%-- 0x405498
                |          
                |--2.00%-- 0x409d20
                |          
                |--2.00%-- 0x405b21
                |          
                 --2.00%-- 0x405a2c

     0.56%          [init]  [kernel]                       [k] kmem_cache_alloc
     0.56%          [init]  [kernel]                       [k] __inet_lookup_established
     0.55%            perf  [kernel]                       [k] number
                |          
                |--95.92%-- vsnprintf
                |          |          
                |          |--97.87%-- seq_printf
                |          |          |          
                |          |          |--56.52%-- show_map_vma
                |          |          |          show_map
                |          |          |          seq_read
                |          |          |          vfs_read
                |          |          |          sys_read
                |          |          |          system_call_fastpath
                |          |          |          __GI_read
                |          |          |          
                |          |          |--28.26%-- render_sigset_t
                |          |          |          proc_pid_status
                |          |          |          proc_single_show
                |          |          |          seq_read
                |          |          |          vfs_read
                |          |          |          sys_read
                |          |          |          system_call_fastpath
                |          |          |          __GI_read
                |          |          |          
                |          |          |--6.52%-- proc_pid_status
                |          |          |          proc_single_show
                |          |          |          seq_read
                |          |          |          vfs_read
                |          |          |          sys_read
                |          |          |          system_call_fastpath
                |          |          |          __GI_read
                |          |          |          
                |          |          |--4.35%-- render_cap_t
                |          |          |          proc_pid_status
                |          |          |          proc_single_show
                |          |          |          seq_read
                |          |          |          vfs_read
                |          |          |          sys_read
                |          |          |          system_call_fastpath
                |          |          |          __GI_read
                |          |          |          
                |          |           --4.35%-- task_mem
                |          |                     proc_pid_status
                |          |                     proc_single_show
                |          |                     seq_read
                |          |                     vfs_read
                |          |                     sys_read
                |          |                     system_call_fastpath
                |          |                     __GI_read
                |          |          
                |           --2.13%-- scnprintf
                |                     bitmap_scnlistprintf
                |                     seq_bitmap_list
                |                     cpuset_task_status_allowed
                |                     proc_pid_status
                |                     proc_single_show
                |                     seq_read
                |                     vfs_read
                |                     sys_read
                |                     system_call_fastpath
                |                     __GI_read
                |          
                 --4.08%-- seq_printf
                           |          
                           |--50.00%-- show_map_vma
                           |          show_map
                           |          seq_read
                           |          vfs_read
                           |          sys_read
                           |          system_call_fastpath
                           |          __GI_read
                           |          
                            --50.00%-- render_sigset_t
                                      proc_pid_status
                                      proc_single_show
                                      seq_read
                                      vfs_read
                                      sys_read
                                      system_call_fastpath
                                      __GI_read

     0.55%          [init]  [kernel]                       [k] native_sched_clock
     0.50%           iperf  [kernel]                       [k] e1000_xmit_frame
                |          
                |--71.11%-- __tcp_ack_snd_check
                |          tcp_rcv_established
                |          tcp_v4_do_rcv
                |          tcp_prequeue_process
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                 --28.89%-- tcp_cleanup_rbuf
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.50%           iperf  [kernel]                       [k] ipt_do_table
                |          
                |--37.78%-- ipt_local_hook
                |          nf_iterate
                |          nf_hook_slow
                |          __ip_local_out
                |          ip_local_out
                |          ip_queue_xmit
                |          tcp_transmit_skb
                |          tcp_send_ack
                |          |          
                |          |--58.82%-- __tcp_ack_snd_check
                |          |          tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          tcp_prequeue_process
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --41.18%-- tcp_cleanup_rbuf
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--31.11%-- ipt_post_routing_hook
                |          nf_iterate
                |          nf_hook_slow
                |          ip_output
                |          ip_local_out
                |          ip_queue_xmit
                |          tcp_transmit_skb
                |          tcp_send_ack
                |          |          
                |          |--64.29%-- __tcp_ack_snd_check
                |          |          tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          tcp_prequeue_process
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --35.71%-- tcp_cleanup_rbuf
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--20.00%-- ipt_local_out_hook
                |          nf_iterate
                |          nf_hook_slow
                |          __ip_local_out
                |          ip_local_out
                |          ip_queue_xmit
                |          tcp_transmit_skb
                |          tcp_send_ack
                |          |          
                |          |--88.89%-- __tcp_ack_snd_check
                |          |          tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          tcp_prequeue_process
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --11.11%-- tcp_cleanup_rbuf
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--6.67%-- nf_iterate
                |          nf_hook_slow
                |          |          
                |          |--66.67%-- ip_output
                |          |          ip_local_out
                |          |          ip_queue_xmit
                |          |          tcp_transmit_skb
                |          |          tcp_send_ack
                |          |          tcp_cleanup_rbuf
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --33.33%-- __ip_local_out
                |                     ip_local_out
                |                     ip_queue_xmit
                |                     tcp_transmit_skb
                |                     tcp_send_ack
                |                     __tcp_ack_snd_check
                |                     tcp_rcv_established
                |                     tcp_v4_do_rcv
                |                     tcp_prequeue_process
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--2.22%-- ipt_local_in_hook
                |          nf_iterate
                |          nf_hook_slow
                |          ip_local_deliver
                |          ip_rcv_finish
                |          ip_rcv
                |          netif_receive_skb
                |          napi_skb_finish
                |          napi_gro_receive
                |          e1000_receive_skb
                |          e1000_clean_rx_irq
                |          e1000_clean
                |          net_rx_action
                |          __do_softirq
                |          call_softirq
                |          do_softirq
                |          irq_exit
                |          do_IRQ
                |          ret_from_intr
                |          vgettimeofday
                |          
                 --2.22%-- ipt_pre_routing_hook
                           nf_iterate
                           nf_hook_slow
                           ip_rcv
                           netif_receive_skb
                           napi_skb_finish
                           napi_gro_receive
                           e1000_receive_skb
                           e1000_clean_rx_irq
                           e1000_clean
                           net_rx_action
                           __do_softirq
                           call_softirq
                           do_softirq
                           irq_exit
                           do_IRQ
                           ret_from_intr
                           __GI___libc_nanosleep

     0.50%           iperf  [kernel]                       [k] schedule
                |          
                |--57.78%-- do_nanosleep
                |          hrtimer_nanosleep
                |          sys_nanosleep
                |          system_call_fastpath
                |          __GI___libc_nanosleep
                |          
                |--33.33%-- schedule_timeout
                |          sk_wait_data
                |          tcp_recvmsg
                |          sock_common_recvmsg
                |          __sock_recvmsg
                |          sock_recvmsg
                |          sys_recvfrom
                |          system_call_fastpath
                |          __recv
                |          
                |--6.67%-- hrtimer_nanosleep
                |          sys_nanosleep
                |          system_call_fastpath
                |          __GI___libc_nanosleep
                |          
                 --2.22%-- sk_wait_data
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.49%           iperf  [kernel]                       [k] tcp_transmit_skb
                |          
                |--97.73%-- tcp_send_ack
                |          |          
                |          |--83.72%-- __tcp_ack_snd_check
                |          |          tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          |          
                |          |          |--97.22%-- tcp_prequeue_process
                |          |          |          tcp_recvmsg
                |          |          |          sock_common_recvmsg
                |          |          |          __sock_recvmsg
                |          |          |          sock_recvmsg
                |          |          |          sys_recvfrom
                |          |          |          system_call_fastpath
                |          |          |          __recv
                |          |          |          
                |          |           --2.78%-- release_sock
                |          |                     tcp_recvmsg
                |          |                     sock_common_recvmsg
                |          |                     __sock_recvmsg
                |          |                     sock_recvmsg
                |          |                     sys_recvfrom
                |          |                     system_call_fastpath
                |          |                     __recv
                |          |          
                |           --16.28%-- tcp_cleanup_rbuf
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                 --2.27%-- __tcp_ack_snd_check
                           tcp_rcv_established
                           tcp_v4_do_rcv
                           tcp_prequeue_process
                           tcp_recvmsg
                           sock_common_recvmsg
                           __sock_recvmsg
                           sock_recvmsg
                           sys_recvfrom
                           system_call_fastpath
                           __recv

     0.49%          [init]  [kernel]                       [k] nf_hook_slow
     0.48%           iperf  [kernel]                       [k] virt_to_head_page
                |          
                |--53.49%-- kfree
                |          skb_release_data
                |          __kfree_skb
                |          |          
                |          |--65.22%-- tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --34.78%-- tcp_rcv_established
                |                     tcp_v4_do_rcv
                |                     tcp_prequeue_process
                |                     tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--18.60%-- skb_release_data
                |          __kfree_skb
                |          |          
                |          |--62.50%-- tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          tcp_prequeue_process
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --37.50%-- tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                |--18.60%-- kmem_cache_free
                |          __kfree_skb
                |          |          
                |          |--62.50%-- tcp_rcv_established
                |          |          tcp_v4_do_rcv
                |          |          tcp_prequeue_process
                |          |          tcp_recvmsg
                |          |          sock_common_recvmsg
                |          |          __sock_recvmsg
                |          |          sock_recvmsg
                |          |          sys_recvfrom
                |          |          system_call_fastpath
                |          |          __recv
                |          |          
                |           --37.50%-- tcp_recvmsg
                |                     sock_common_recvmsg
                |                     __sock_recvmsg
                |                     sock_recvmsg
                |                     sys_recvfrom
                |                     system_call_fastpath
                |                     __recv
                |          
                 --9.30%-- __kfree_skb
                           |          
                           |--75.00%-- tcp_rcv_established
                           |          tcp_v4_do_rcv
                           |          tcp_prequeue_process
                           |          tcp_recvmsg
                           |          sock_common_recvmsg
                           |          __sock_recvmsg
                           |          sock_recvmsg
                           |          sys_recvfrom
                           |          system_call_fastpath
                           |          __recv
                           |          
                            --25.00%-- tcp_recvmsg
                                      sock_common_recvmsg
                                      __sock_recvmsg
                                      sock_recvmsg
                                      sys_recvfrom
                                      system_call_fastpath
                                      __recv
 ...

Re: BFS vs. mainline scheduler benchmarks and measurements

[Daniel Walker]

On Mon, 2009-09-07 at 20:26 +0200, Ingo Molnar wrote:
> That's interesting. I tried to reproduce it on x86, but the profile 
> does not show any scheduler overhead at all on the server:

If the scheduler isn't running the task which causes the lower
throughput , would that even show up in profiling output?

Daniel

Re: BFS vs. mainline scheduler benchmarks and measurements

[Michael Buesch]

On Monday 07 September 2009 20:26:29 Ingo Molnar wrote:
> Could you profile it please? Also, what's the context-switch rate?

As far as I can tell, the broadcom mips architecture does not have profiling support.
It does only have some proprietary profiling registers that nobody wrote kernel
support for, yet.

-- 
Greetings, Michael.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Michael Buesch <mb@bu3sch.de> wrote:

> On Monday 07 September 2009 20:26:29 Ingo Molnar wrote:
> > Could you profile it please? Also, what's the context-switch rate?
> 
> As far as I can tell, the broadcom mips architecture does not have 
> profiling support. It does only have some proprietary profiling 
> registers that nobody wrote kernel support for, yet.

Well, what does 'vmstat 1' show - how many context switches are 
there per second on the iperf server? In theory if it's a truly 
saturated box, there shouldnt be many - just a single iperf task 
running at 100% CPU utilization or so.

(Also, if there's hrtimer support for that board then perfcounters 
could be used to profile it.)

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Pekka Pietikainen]

On Mon, Sep 07, 2009 at 10:57:01PM +0200, Ingo Molnar wrote:
> > > Could you profile it please? Also, what's the context-switch rate?
> > 
> > As far as I can tell, the broadcom mips architecture does not have 
> > profiling support. It does only have some proprietary profiling 
> > registers that nobody wrote kernel support for, yet.
> Well, what does 'vmstat 1' show - how many context switches are 
> there per second on the iperf server? In theory if it's a truly 
> saturated box, there shouldnt be many - just a single iperf task 
Yay, finally something that's measurable in this thread \o/

Gigabit Ethernet iperf on an Atom or so might be something that 
shows similar effects yet is debuggable. Anyone feel like taking a shot?

That beast doing iperf probably ends up making it go quite close to it's
limits (IO, mem bw, cpu). IIRC the routing/bridging performance is
something like 40Mbps (depends a lot on the model, corresponds pretty
well with the Mhz of the beast). 

Maybe not totally unlike what make -j16 does to a 1-4 core box? 

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Pekka Pietikainen <pp@ee.oulu.fi> wrote:

> On Mon, Sep 07, 2009 at 10:57:01PM +0200, Ingo Molnar wrote:
> > > > Could you profile it please? Also, what's the context-switch rate?
> > > 
> > > As far as I can tell, the broadcom mips architecture does not have 
> > > profiling support. It does only have some proprietary profiling 
> > > registers that nobody wrote kernel support for, yet.
> > Well, what does 'vmstat 1' show - how many context switches are 
> > there per second on the iperf server? In theory if it's a truly 
> > saturated box, there shouldnt be many - just a single iperf task 
>
> Yay, finally something that's measurable in this thread \o/

My initial posting in this thread contains 6 separate types of 
measurements, rather extensive ones. Out of those, 4 measurements 
were latency oriented, two were throughput oriented. Plenty of data, 
plenty of results, and very good reproducability.

> Gigabit Ethernet iperf on an Atom or so might be something that 
> shows similar effects yet is debuggable. Anyone feel like taking a 
> shot?

I tried iperf on x86 and simulated saturation and no, there's no BFS 
versus mainline performance difference that i can measure - simply 
because a saturated iperf server does not schedule much - it's busy 
handling all that networking workload.

I did notice that iperf is somewhat noisy: it can easily have weird 
outliers regardless of which scheduler is used. That could be an 
effect of queueing/timing: depending on precisely what order packets 
arrive and they get queued by the networking stack, does get a 
cache-effective pathway of packets get opened - while with slightly 
different timings, that pathway closes and we get much worse 
queueing performance. I saw noise on the order of magnitude of 10%, 
so iperf has to be measured carefully before drawing conclusions.

> That beast doing iperf probably ends up making it go quite close 
> to it's limits (IO, mem bw, cpu). IIRC the routing/bridging 
> performance is something like 40Mbps (depends a lot on the model, 
> corresponds pretty well with the Mhz of the beast).
> 
> Maybe not totally unlike what make -j16 does to a 1-4 core box?

No, a single iperf session is very different from kbuild make -j16. 

Firstly, iperf server is just a single long-lived task - so we 
context-switch between that and the idle thread , [and perhaps a 
kernel thread such as ksoftirqd]. The scheduler essentially has no 
leeway what task to schedule and for how long: if there's work going 
on the iperf server task will run - if there's none, the idle task 
runs. [modulo ksoftirqd - depending on the driver model and 
dependent on precise timings.]

kbuild -j16 on the other hand is a complex hierarchy and mixture of 
thousands of short-lived and long-lived tasks. The scheduler has a 
lot of leeway to decide what to schedule and for how long.

>From a scheduler perspective the two workloads could not be any more 
different. Kbuild does test scheduler decisions in non-trivial ways 
- iperf server does not really.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 11:04 AM, Ingo Molnar wrote:
>
> * Pekka Pietikainen<pp@ee.oulu.fi>  wrote:
>
>> On Mon, Sep 07, 2009 at 10:57:01PM +0200, Ingo Molnar wrote:
>>>>> Could you profile it please? Also, what's the context-switch rate?
>>>>
>>>> As far as I can tell, the broadcom mips architecture does not have
>>>> profiling support. It does only have some proprietary profiling
>>>> registers that nobody wrote kernel support for, yet.
>>> Well, what does 'vmstat 1' show - how many context switches are
>>> there per second on the iperf server? In theory if it's a truly
>>> saturated box, there shouldnt be many - just a single iperf task
>>
>> Yay, finally something that's measurable in this thread \o/
>
> My initial posting in this thread contains 6 separate types of
> measurements, rather extensive ones. Out of those, 4 measurements
> were latency oriented, two were throughput oriented. Plenty of data,
> plenty of results, and very good reproducability.

None of which involve latency-prone GUI applications running on cheap 
commodity hardware though.  I listed examples where mainline seems to 
behave sub-optimal and ways to reproduce them but this doesn't seem to 
be an area of interest.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Nikos Chantziaras <realnc@arcor.de> wrote:

> On 09/08/2009 11:04 AM, Ingo Molnar wrote:
>>
>> * Pekka Pietikainen<pp@ee.oulu.fi>  wrote:
>>
>>> On Mon, Sep 07, 2009 at 10:57:01PM +0200, Ingo Molnar wrote:
>>>>>> Could you profile it please? Also, what's the context-switch rate?
>>>>>
>>>>> As far as I can tell, the broadcom mips architecture does not have
>>>>> profiling support. It does only have some proprietary profiling
>>>>> registers that nobody wrote kernel support for, yet.
>>>> Well, what does 'vmstat 1' show - how many context switches are
>>>> there per second on the iperf server? In theory if it's a truly
>>>> saturated box, there shouldnt be many - just a single iperf task
>>>
>>> Yay, finally something that's measurable in this thread \o/
>>
>> My initial posting in this thread contains 6 separate types of 
>> measurements, rather extensive ones. Out of those, 4 measurements 
>> were latency oriented, two were throughput oriented. Plenty of 
>> data, plenty of results, and very good reproducability.
>
> None of which involve latency-prone GUI applications running on 
> cheap commodity hardware though. [...]

The lat_tcp, lat_pipe and pipe-test numbers are all benchmarks that 
characterise such workloads - they show the latency of context 
switches.

I also tested where Con posted numbers that BFS has an edge over 
mainline: kbuild performance. Should i not have done that?

Also note the interbench latency measurements that Con posted:

   http://ck.kolivas.org/patches/bfs/interbench-bfs-cfs.txt

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.004 +/- 0.00436    0.006             100            100
Video     0.008 +/- 0.00879    0.015             100            100
X         0.006 +/- 0.0067     0.014             100            100
Burn      0.005 +/- 0.00563    0.009             100            100
Write     0.005 +/- 0.00887     0.16             100            100
Read      0.006 +/- 0.00696    0.018             100            100
Compile   0.007 +/- 0.00751    0.019             100            100

Versus the mainline scheduler:

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.005 +/- 0.00562    0.007             100            100
Video     0.003 +/- 0.00333    0.009             100            100
X         0.003 +/- 0.00409     0.01             100            100
Burn      0.004 +/- 0.00415    0.006             100            100
Write     0.005 +/- 0.00592    0.021             100            100
Read      0.004 +/- 0.00463    0.009             100            100
Compile   0.003 +/- 0.00426    0.014             100            100

look at those standard deviation numbers, their spread is way too 
high, often 50% or more - very hard to compare such noisy data. 

Furthermore, they happen to show the 2.6.30 mainline scheduler 
outperforming BFS in almost every interactivity metric.

Check it for yourself and compare the entries. I havent made those 
measurements, Con did.

For example 'Compile' latencies:

--- Benchmarking simulated cpu of Audio in the presence of simulated Load
                  Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
 v2.6.30: Compile   0.003 +/- 0.00426    0.014             100            100
     BFS: Compile   0.007 +/- 0.00751    0.019             100            100

but ... with a near 100% standard deviation that's pretty hard to 
judge. The Max Latency went from 14 usecs under v2.6.30 to 19 usecs 
on BFS.

> [...]  I listed examples where mainline seems to behave 
> sub-optimal and ways to reproduce them but this doesn't seem to be 
> an area of interest.

It is an area of interest of course. That's how the interactivity 
results above became possible.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 01:12 PM, Ingo Molnar wrote:
>
> * Nikos Chantziaras<realnc@arcor.de>  wrote:
>
>> On 09/08/2009 11:04 AM, Ingo Molnar wrote:
>>>
>>> * Pekka Pietikainen<pp@ee.oulu.fi>   wrote:
>>>
>>>> On Mon, Sep 07, 2009 at 10:57:01PM +0200, Ingo Molnar wrote:
>>>>>>> Could you profile it please? Also, what's the context-switch rate?
>>>>>>
>>>>>> As far as I can tell, the broadcom mips architecture does not have
>>>>>> profiling support. It does only have some proprietary profiling
>>>>>> registers that nobody wrote kernel support for, yet.
>>>>> Well, what does 'vmstat 1' show - how many context switches are
>>>>> there per second on the iperf server? In theory if it's a truly
>>>>> saturated box, there shouldnt be many - just a single iperf task
>>>>
>>>> Yay, finally something that's measurable in this thread \o/
>>>
>>> My initial posting in this thread contains 6 separate types of
>>> measurements, rather extensive ones. Out of those, 4 measurements
>>> were latency oriented, two were throughput oriented. Plenty of
>>> data, plenty of results, and very good reproducability.
>>
>> None of which involve latency-prone GUI applications running on
>> cheap commodity hardware though. [...]
>
> The lat_tcp, lat_pipe and pipe-test numbers are all benchmarks that
> characterise such workloads - they show the latency of context
> switches.
>
> I also tested where Con posted numbers that BFS has an edge over
> mainline: kbuild performance. Should i not have done that?

It's good that you did, of course.  However, when someone reports a 
problem/issue, the developer usually tries to reproduce the problem; he 
needs to see what the user sees.  This is how it's usually done, not 
only in most other development environments, but also here from I could 
gather by reading this list.  When getting reports about interactivity 
issues and with very specific examples of how to reproduce, I would have 
expected that most developers interested in identifying the issue would 
try to reproduce the same problem and work from there.  That would mean 
that you (or anyone else with an interest of tracking this down) would 
follow the examples given (by me and others, like enabling desktop 
compositing, firing up mplayer with a video and generally reproducing 
this using the quite detailed steps I posted as a recipe).

However, in this case, instead of the above, raw numbers are posted with 
batch jobs and benchmarks that aren't actually reproducing the issue as 
described by the reporter(s).  That way, the developer doesn't get to 
experience the issue firt-hand (and due to this possibly missing the 
real cause).  In most other bug reports or issues, the right thing seems 
to happen and the devs try to reproduce it exactly as described.  But 
not in this case.  I suspect this is due to most devs not using the 
software components on their machines that are necessary for this and 
therefore it would take too much time to reproduce the issue exactly as 
described?

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Nikos Chantziaras <realnc@arcor.de> wrote:

> [...] That would mean that you (or anyone else with an interest of 
> tracking this down) would follow the examples given (by me and 
> others, like enabling desktop compositing, firing up mplayer with 
> a video and generally reproducing this using the quite detailed 
> steps I posted as a recipe).

Could you follow up on Frederic's detailed tracing suggestions that 
would give us the source of the latency?

( Also, as per lkml etiquette, please try to keep the Cc: list 
  intact when replying to emails. I missed your first reply
  that you un-Cc:-ed. )

A quick look at the latencytop output suggests a scheduling latency. 
Could you send me the kernel .config that you are using?

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/08/2009 02:35 PM, Ingo Molnar wrote:
>
> * Nikos Chantziaras<realnc@arcor.de>  wrote:
>
>> [...] That would mean that you (or anyone else with an interest of
>> tracking this down) would follow the examples given (by me and
>> others, like enabling desktop compositing, firing up mplayer with
>> a video and generally reproducing this using the quite detailed
>> steps I posted as a recipe).
>
> Could you follow up on Frederic's detailed tracing suggestions that
> would give us the source of the latency?

I've set it up and ran the tests now.


> ( Also, as per lkml etiquette, please try to keep the Cc: list
>    intact when replying to emails. I missed your first reply
>    that you un-Cc:-ed. )

Sorry for that.


> A quick look at the latencytop output suggests a scheduling latency.
> Could you send me the kernel .config that you are using?

That would be this one:

     http://foss.math.aegean.gr/~realnc/kernel/config-2.6.31-rc9

Re: BFS vs. mainline scheduler benchmarks and measurements

[el_es]

Ingo Molnar <mingo <at> elte.hu> writes:


> For example 'Compile' latencies:
> 
> --- Benchmarking simulated cpu of Audio in the presence of simulated Load
>                   Latency +/- SD (ms)  Max Latency   % Desired CPU  %
Deadlines Met
>  v2.6.30: Compile   0.003 +/- 0.00426    0.014             100            100
>      BFS: Compile   0.007 +/- 0.00751    0.019             100            100
> 
> but ... with a near 100% standard deviation that's pretty hard to 
> judge. The Max Latency went from 14 usecs under v2.6.30 to 19 usecs 
> on BFS.
> 
[...]
> 	Ingo
> 

This just struck me : maybe what desktop users *feel* is exactly that : current
approach is too fine-grained, trying to achieve the minimum latency with *most*
reproductible result (less stddev) at all cost ? And BFS just doesn't care? 
I know this sounds like heresy.

[ the space below is to satisfy the brain-dead GMane posting engine].










Lukasz

Re: BFS vs. mainline scheduler benchmarks and measurements

[Michael Buesch]

On Monday 07 September 2009 22:57:01 Ingo Molnar wrote:
> 
> * Michael Buesch <mb@bu3sch.de> wrote:
> 
> > On Monday 07 September 2009 20:26:29 Ingo Molnar wrote:
> > > Could you profile it please? Also, what's the context-switch rate?
> > 
> > As far as I can tell, the broadcom mips architecture does not have 
> > profiling support. It does only have some proprietary profiling 
> > registers that nobody wrote kernel support for, yet.
> 
> Well, what does 'vmstat 1' show - how many context switches are 
> there per second on the iperf server? In theory if it's a truly 
> saturated box, there shouldnt be many - just a single iperf task 
> running at 100% CPU utilization or so.
> 
> (Also, if there's hrtimer support for that board then perfcounters 
> could be used to profile it.)

CFS:

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0  15892   1684   5868    0    0     0     0  268    6 31 69  0  0
 1  0      0  15892   1684   5868    0    0     0     0  266    2 34 66  0  0
 1  0      0  15892   1684   5868    0    0     0     0  266    6 33 67  0  0
 1  0      0  15892   1684   5868    0    0     0     0  267    4 37 63  0  0
 1  0      0  15892   1684   5868    0    0     0     0  267    6 34 66  0  0
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.99 port 47278
 2  0      0  15756   1684   5868    0    0     0     0 1655   68 26 74  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1945   88 20 80  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1882   85 20 80  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1923   86 18 82  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1986   87 23 77  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1923   87 17 83  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1951   84 19 81  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1970   87 18 82  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1972   85 23 77  0  0
 2  0      0  15756   1684   5868    0    0     0     0 1961   87 18 82  0  0
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  28.6 MBytes  23.9 Mbits/sec
 1  0      0  15752   1684   5868    0    0     0     0  599   22 22 78  0  0
 1  0      0  15752   1684   5868    0    0     0     0  269    4 32 68  0  0
 1  0      0  15752   1684   5868    0    0     0     0  266    4 29 71  0  0
 1  0      0  15764   1684   5868    0    0     0     0  267    6 37 63  0  0
 1  0      0  15764   1684   5868    0    0     0     0  267    4 31 69  0  0
 1  0      0  15768   1684   5868    0    0     0     0  266    4 51 49  0  0


I'm currently unable to test BFS, because the device throws strange flash errors.
Maybe the flash is broken :(

-- 
Greetings, Michael.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> That's interesting. I tried to reproduce it on x86, but the 
> profile does not show any scheduler overhead at all on the server:

I've now simulated a saturated iperf server by adding an 
udelay(3000) to e1000_intr() in via the patch below.

There's no idle time left that way:

 Cpu(s):  0.0%us,  2.6%sy,  0.0%ni,  0.0%id,  0.0%wa, 93.2%hi,  4.2%si,  0.0%st
 Mem:   1021044k total,    93400k used,   927644k free,     5068k buffers
 Swap:  8193140k total,        0k used,  8193140k free,    25404k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                    
 1604 mingo     20   0 38300  956  724 S 99.4  0.1   3:15.07 iperf                      
  727 root      15  -5     0    0    0 S  0.2  0.0   0:00.41 kondemand/0                
 1226 root      20   0  6452  336  240 S  0.2  0.0   0:00.06 irqbalance                 
 1387 mingo     20   0 78872 1988 1300 S  0.2  0.2   0:00.23 sshd                       
 1657 mingo     20   0 12752 1128  800 R  0.2  0.1   0:01.34 top                        
    1 root      20   0 10320  684  572 S  0.0  0.1   0:01.79 init                       
    2 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd                   

And the server is only able to saturate half of the 1 gigabit 
bandwidth:

 Client connecting to t, TCP port 5001
 TCP window size: 16.0 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.0.1.19 port 50836 connected with 10.0.1.14 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec    504 MBytes    423 Mbits/sec
 ------------------------------------------------------------
 Client connecting to t, TCP port 5001
 TCP window size: 16.0 KByte (default)
 ------------------------------------------------------------
 [  3] local 10.0.1.19 port 50837 connected with 10.0.1.14 port 5001
 [ ID] Interval       Transfer     Bandwidth
 [  3]  0.0-10.0 sec    502 MBytes    420 Mbits/sec


perf top is showing:

------------------------------------------------------------------------------
   PerfTop:   28517 irqs/sec  kernel:99.4% [100000 cycles],  (all, 1 CPUs)
------------------------------------------------------------------------------

             samples    pcnt   kernel function
             _______   _____   _______________

           139553.00 - 93.2% : delay_tsc
             2098.00 -  1.4% : hmac_digest
              561.00 -  0.4% : ip_call_ra_chain
              335.00 -  0.2% : neigh_alloc
              279.00 -  0.2% : __hash_conntrack
              257.00 -  0.2% : dev_activate
              186.00 -  0.1% : proc_tcp_available_congestion_control
              178.00 -  0.1% : e1000_get_regs
              167.00 -  0.1% : tcp_event_data_recv

delay_tsc() dominates, as expected. Still zero scheduler overhead 
and the contex-switch rate is well below 1000 per sec.

Then i booted v2.6.30 vanilla, added the udelay(3000) and got:

 [  5] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 47026
 [  5]  0.0-10.0 sec    493 MBytes    412 Mbits/sec
 [  4] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 47027
 [  4]  0.0-10.0 sec    520 MBytes    436 Mbits/sec
 [  5] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 47028
 [  5]  0.0-10.0 sec    506 MBytes    424 Mbits/sec
 [  4] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 47029
 [  4]  0.0-10.0 sec    496 MBytes    415 Mbits/sec

i.e. essentially the same throughput. (and this shows that using .30 
versus .31 did not materially impact iperf performance in this test, 
under these conditions and with this hardware)

The i applied the BFS patch to v2.6.30 and used the same 
udelay(3000) hack and got:

No measurable change in throughput.

Obviously, this test is not equivalent to your test - but it does 
show that even saturated iperf is getting scheduled just fine. (or, 
rather, does not get scheduled all that much.)

[  5] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 38505
[  5]  0.0-10.1 sec    481 MBytes    401 Mbits/sec
[  4] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 38506
[  4]  0.0-10.0 sec    505 MBytes    423 Mbits/sec
[  5] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 38507
[  5]  0.0-10.0 sec    508 MBytes    426 Mbits/sec
[  4] local 10.0.1.14 port 5001 connected with 10.0.1.19 port 38508
[  4]  0.0-10.0 sec    486 MBytes    406 Mbits/sec

So either your MIPS system has some unexpected dependency on the 
scheduler, or there's something weird going on.

Mind poking on this one to figure out whether it's all repeatable 
and why that slowdown happens? Multiple attempts to reproduce it 
failed here for me.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

On Tue, 2009-09-08 at 09:48 +0200, Ingo Molnar wrote:
> So either your MIPS system has some unexpected dependency on the 
> scheduler, or there's something weird going on.
> 
> Mind poking on this one to figure out whether it's all repeatable 
> and why that slowdown happens? Multiple attempts to reproduce it 
> failed here for me.

Could it be the scheduler using constructs that don't do well on MIPS ? 

I remember at some stage we spotted an expensive multiply in there,
maybe there's something similar, or some unaligned or non-cache friendly
vs. the MIPS cache line size data structure, that sort of thing ...

Is this a SW loaded TLB ? Does it misses on kernel space ? That could
also be some differences in how many pages are touched by each scheduler
causing more TLB pressure. This will be mostly invisible on x86.

At this stage, it will be hard to tell without some profile data I
suppose. Maybe next week I can try on a small SW loaded TLB embedded PPC
see if I can reproduce some of that, but no promises here.

Cheers,
Ben.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ralf Baechle]

On Tue, Sep 08, 2009 at 07:50:00PM +1000, Benjamin Herrenschmidt wrote:

> On Tue, 2009-09-08 at 09:48 +0200, Ingo Molnar wrote:
> > So either your MIPS system has some unexpected dependency on the 
> > scheduler, or there's something weird going on.
> > 
> > Mind poking on this one to figure out whether it's all repeatable 
> > and why that slowdown happens? Multiple attempts to reproduce it 
> > failed here for me.
> 
> Could it be the scheduler using constructs that don't do well on MIPS ? 

It would surprise me.

I'm wondering if BFS has properties that make it perform better on a very
low memory system; I guess the BCM74xx system will have like 32MB or 64MB
only.

> I remember at some stage we spotted an expensive multiply in there,
> maybe there's something similar, or some unaligned or non-cache friendly
> vs. the MIPS cache line size data structure, that sort of thing ...
> 
> Is this a SW loaded TLB ? Does it misses on kernel space ? That could
> also be some differences in how many pages are touched by each scheduler
> causing more TLB pressure. This will be mostly invisible on x86.

Software refilled.  No misses ever for kernel space or low-mem; think of
it as low-mem and kernel executable living in a 512MB page that is mapped
by a mechanism outside the TLB.  Vmalloc ranges are TLB mapped.  Ioremap
address ranges only if above physical address 512MB.

An emulated unaligned load/store is very expensive; one that is encoded
properly by GCC for __attribute__((packed)) is only 1 cycle and 1
instruction ( = 4 bytes) extra.

> At this stage, it will be hard to tell without some profile data I
> suppose. Maybe next week I can try on a small SW loaded TLB embedded PPC
> see if I can reproduce some of that, but no promises here.

  Ralf

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Ralf Baechle wrote:
>> I remember at some stage we spotted an expensive multiply in there,
>> maybe there's something similar, or some unaligned or non-cache friendly
>> vs. the MIPS cache line size data structure, that sort of thing ...
>> 
>> Is this a SW loaded TLB ? Does it misses on kernel space ? That could
>> also be some differences in how many pages are touched by each scheduler
>> causing more TLB pressure. This will be mostly invisible on x86.
> 
> Software refilled.  No misses ever for kernel space or low-mem; think of
> it as low-mem and kernel executable living in a 512MB page that is mapped
> by a mechanism outside the TLB.  Vmalloc ranges are TLB mapped.  Ioremap
> address ranges only if above physical address 512MB.
> 
> An emulated unaligned load/store is very expensive; one that is encoded
> properly by GCC for __attribute__((packed)) is only 1 cycle and 1
> instruction ( = 4 bytes) extra.
CFS definitely isn't causing any emulated unaligned load/stores on these
devices, we've tested that.

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Benjamin Herrenschmidt wrote:
> On Tue, 2009-09-08 at 09:48 +0200, Ingo Molnar wrote:
>> So either your MIPS system has some unexpected dependency on the 
>> scheduler, or there's something weird going on.
>> 
>> Mind poking on this one to figure out whether it's all repeatable 
>> and why that slowdown happens? Multiple attempts to reproduce it 
>> failed here for me.
> 
> Could it be the scheduler using constructs that don't do well on MIPS ? 
> 
> I remember at some stage we spotted an expensive multiply in there,
> maybe there's something similar, or some unaligned or non-cache friendly
> vs. the MIPS cache line size data structure, that sort of thing ...
> 
> Is this a SW loaded TLB ? Does it misses on kernel space ? That could
> also be some differences in how many pages are touched by each scheduler
> causing more TLB pressure. This will be mostly invisible on x86.
The TLB is SW loaded, yes. However it should not do any misses on kernel
space, since the whole segment is in a wired TLB entry.

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Benjamin Herrenschmidt]

> The TLB is SW loaded, yes. However it should not do any misses on kernel
> space, since the whole segment is in a wired TLB entry.

Including vmalloc space ?

Ben.

Re: BFS vs. mainline scheduler benchmarks and measurements

[David Miller]

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: Wed, 09 Sep 2009 10:28:22 +1000

>> The TLB is SW loaded, yes. However it should not do any misses on kernel
>> space, since the whole segment is in a wired TLB entry.
> 
> Including vmalloc space ?

No, MIPS does take SW tlb misses on vmalloc space. :-)

Re: BFS vs. mainline scheduler benchmarks and measurements

[Michael Buesch]

On Tuesday 08 September 2009 09:48:25 Ingo Molnar wrote:
> Mind poking on this one to figure out whether it's all repeatable 
> and why that slowdown happens?

I repeated the test several times, because I couldn't really believe that
there's such a big difference for me, but the results were the same.
I don't really know what's going on nor how to find out what's going on.

-- 
Greetings, Michael.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Michael Buesch <mb@bu3sch.de> wrote:

> On Tuesday 08 September 2009 09:48:25 Ingo Molnar wrote:
> > Mind poking on this one to figure out whether it's all repeatable 
> > and why that slowdown happens?
> 
> I repeated the test several times, because I couldn't really believe 
> that there's such a big difference for me, but the results were the 
> same. I don't really know what's going on nor how to find out what's 
> going on.

Well that's a really memory constrained MIPS device with like 16 MB of 
RAM or so? So having effects from small things like changing details in 
a kernel image is entirely plausible.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Ingo Molnar wrote:
> * Michael Buesch <mb@bu3sch.de> wrote:
> 
>> On Tuesday 08 September 2009 09:48:25 Ingo Molnar wrote:
>> > Mind poking on this one to figure out whether it's all repeatable 
>> > and why that slowdown happens?
>> 
>> I repeated the test several times, because I couldn't really believe 
>> that there's such a big difference for me, but the results were the 
>> same. I don't really know what's going on nor how to find out what's 
>> going on.
> 
> Well that's a really memory constrained MIPS device with like 16 MB of 
> RAM or so? So having effects from small things like changing details in 
> a kernel image is entirely plausible.
Normally changing small details doesn't have much of an effect. While 16
MB is indeed not that much, we do usually have around 8 MB free with a
full user space running. Changes to other subsystems normally produce
consistent and repeatable differences that seem entirely unrelated to
memory use, so any measurable difference related to scheduler changes is
unlikely to be related to the low amount of RAM.
By the way, we do frequently also test the same software with devices
that have more RAM, e.g. 32 or 64 MB and it usually behaves in a very
similar way.

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Felix Fietkau <nbd@openwrt.org> wrote:

> Ingo Molnar wrote:
> > * Michael Buesch <mb@bu3sch.de> wrote:
> > 
> >> On Tuesday 08 September 2009 09:48:25 Ingo Molnar wrote:
> >> > Mind poking on this one to figure out whether it's all repeatable 
> >> > and why that slowdown happens?
> >> 
> >> I repeated the test several times, because I couldn't really believe 
> >> that there's such a big difference for me, but the results were the 
> >> same. I don't really know what's going on nor how to find out what's 
> >> going on.
> > 
> > Well that's a really memory constrained MIPS device with like 16 MB of 
> > RAM or so? So having effects from small things like changing details in 
> > a kernel image is entirely plausible.
>
> Normally changing small details doesn't have much of an effect. While 
> 16 MB is indeed not that much, we do usually have around 8 MB free 
> with a full user space running. Changes to other subsystems normally 
> produce consistent and repeatable differences that seem entirely 
> unrelated to memory use, so any measurable difference related to 
> scheduler changes is unlikely to be related to the low amount of RAM. 
> By the way, we do frequently also test the same software with devices 
> that have more RAM, e.g. 32 or 64 MB and it usually behaves in a very 
> similar way.

Well, Michael Buesch posted vmstat results, and they show what i have 
found with my x86 simulated reproducer as well (these are Michael's 
numbers):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0	   0  15892   1684   5868    0    0     0     0  268    6 31 69  0  0
 1  0	   0  15892   1684   5868    0    0     0     0  266    2 34 66  0  0
 1  0	   0  15892   1684   5868    0    0     0     0  266    6 33 67  0  0
 1  0	   0  15892   1684   5868    0    0     0     0  267    4 37 63  0  0
 1  0	   0  15892   1684   5868    0    0     0     0  267    6 34 66  0  0

on average 4 context switches _per second_. The scheduler is not a 
factor on this box.

Furthermore:

 | I'm currently unable to test BFS, because the device throws strange 
 | flash errors. Maybe the flash is broken :(

So maybe those flash errors somehow impacted the measurements as well?

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Ingo Molnar wrote:
> * Felix Fietkau <nbd@openwrt.org> wrote:
> 
>> Ingo Molnar wrote:
>> > Well that's a really memory constrained MIPS device with like 16 MB of 
>> > RAM or so? So having effects from small things like changing details in 
>> > a kernel image is entirely plausible.
>>
>> Normally changing small details doesn't have much of an effect. While 
>> 16 MB is indeed not that much, we do usually have around 8 MB free 
>> with a full user space running. Changes to other subsystems normally 
>> produce consistent and repeatable differences that seem entirely 
>> unrelated to memory use, so any measurable difference related to 
>> scheduler changes is unlikely to be related to the low amount of RAM. 
>> By the way, we do frequently also test the same software with devices 
>> that have more RAM, e.g. 32 or 64 MB and it usually behaves in a very 
>> similar way.
> 
> Well, Michael Buesch posted vmstat results, and they show what i have 
> found with my x86 simulated reproducer as well (these are Michael's 
> numbers):
> 
> procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
>  1  0	   0  15892   1684   5868    0    0     0     0  268    6 31 69  0  0
>  1  0	   0  15892   1684   5868    0    0     0     0  266    2 34 66  0  0
>  1  0	   0  15892   1684   5868    0    0     0     0  266    6 33 67  0  0
>  1  0	   0  15892   1684   5868    0    0     0     0  267    4 37 63  0  0
>  1  0	   0  15892   1684   5868    0    0     0     0  267    6 34 66  0  0
> 
> on average 4 context switches _per second_. The scheduler is not a 
> factor on this box.
> 
> Furthermore:
> 
>  | I'm currently unable to test BFS, because the device throws strange 
>  | flash errors. Maybe the flash is broken :(
> 
> So maybe those flash errors somehow impacted the measurements as well?
I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
iperf tests, I consistently get the following results when running the
transfer from the device to my laptop:

CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec

The transfer speed from my laptop to the device are the same with BFS
and CFS. I repeated the tests a few times just to be sure, and I will
check vmstat later.
The difference here cannot be flash related, as I ran a kernel image
with the whole userland contained in initramfs. No on-flash filesystem
was mounted or accessed.

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Felix Fietkau <nbd@openwrt.org> wrote:

> Ingo Molnar wrote:
> > * Felix Fietkau <nbd@openwrt.org> wrote:
> > 
> >> Ingo Molnar wrote:
> >> > Well that's a really memory constrained MIPS device with like 16 MB of 
> >> > RAM or so? So having effects from small things like changing details in 
> >> > a kernel image is entirely plausible.
> >>
> >> Normally changing small details doesn't have much of an effect. While 
> >> 16 MB is indeed not that much, we do usually have around 8 MB free 
> >> with a full user space running. Changes to other subsystems normally 
> >> produce consistent and repeatable differences that seem entirely 
> >> unrelated to memory use, so any measurable difference related to 
> >> scheduler changes is unlikely to be related to the low amount of RAM. 
> >> By the way, we do frequently also test the same software with devices 
> >> that have more RAM, e.g. 32 or 64 MB and it usually behaves in a very 
> >> similar way.
> > 
> > Well, Michael Buesch posted vmstat results, and they show what i have 
> > found with my x86 simulated reproducer as well (these are Michael's 
> > numbers):
> > 
> > procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
> >  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
> >  1  0	   0  15892   1684   5868    0    0     0     0  268    6 31 69  0  0
> >  1  0	   0  15892   1684   5868    0    0     0     0  266    2 34 66  0  0
> >  1  0	   0  15892   1684   5868    0    0     0     0  266    6 33 67  0  0
> >  1  0	   0  15892   1684   5868    0    0     0     0  267    4 37 63  0  0
> >  1  0	   0  15892   1684   5868    0    0     0     0  267    6 34 66  0  0
> > 
> > on average 4 context switches _per second_. The scheduler is not a 
> > factor on this box.
> > 
> > Furthermore:
> > 
> >  | I'm currently unable to test BFS, because the device throws strange 
> >  | flash errors. Maybe the flash is broken :(
> > 
> > So maybe those flash errors somehow impacted the measurements as well?
> I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
> MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
> iperf tests, I consistently get the following results when running the
> transfer from the device to my laptop:
> 
> CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
> BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec
> 
> The transfer speed from my laptop to the device are the same with BFS
> and CFS. I repeated the tests a few times just to be sure, and I will
> check vmstat later.

Which exact mainline kernel have you tried? For anything performance 
related running latest upstream -git (currently at 202c467) would be 
recommended.

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Ingo Molnar wrote:
> * Felix Fietkau <nbd@openwrt.org> wrote:
>> I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
>> MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
>> iperf tests, I consistently get the following results when running the
>> transfer from the device to my laptop:
>> 
>> CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
>> BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec
>> 
>> The transfer speed from my laptop to the device are the same with BFS
>> and CFS. I repeated the tests a few times just to be sure, and I will
>> check vmstat later.
> 
> Which exact mainline kernel have you tried? For anything performance 
> related running latest upstream -git (currently at 202c467) would be 
> recommended.
I used the OpenWrt-patched 2.6.30. Support for the hardware that I
tested with hasn't been merged upstream yet. Do you think that the
scheduler related changes after 2.6.30 are relevant for non-SMP
performance as well? If so, I'll work on a test with latest upstream
-git with the necessary patches when I have time for it.

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Felix Fietkau <nbd@openwrt.org> wrote:

> Ingo Molnar wrote:
> > * Felix Fietkau <nbd@openwrt.org> wrote:
> >> I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
> >> MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
> >> iperf tests, I consistently get the following results when running the
> >> transfer from the device to my laptop:
> >> 
> >> CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
> >> BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec
> >> 
> >> The transfer speed from my laptop to the device are the same with BFS
> >> and CFS. I repeated the tests a few times just to be sure, and I will
> >> check vmstat later.
> > 
> > Which exact mainline kernel have you tried? For anything performance 
> > related running latest upstream -git (currently at 202c467) would be 
> > recommended.
>
> I used the OpenWrt-patched 2.6.30. Support for the hardware that I 
> tested with hasn't been merged upstream yet. Do you think that the 
> scheduler related changes after 2.6.30 are relevant for non-SMP 
> performance as well? If so, I'll work on a test with latest upstream 
> -git with the necessary patches when I have time for it.

Dont know - it's hard to tell what happens without basic analysis tools. 
Is there _any_ way to profile what happens on that system? (Do hrtimers 
work on it that could be used to profile it?)

	Ingo

Re: BFS vs. mainline scheduler benchmarks and measurements

[Felix Fietkau]

Ingo Molnar wrote:
> * Felix Fietkau <nbd@openwrt.org> wrote:
> 
>> Ingo Molnar wrote:
>> > * Felix Fietkau <nbd@openwrt.org> wrote:
>> >> I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
>> >> MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
>> >> iperf tests, I consistently get the following results when running the
>> >> transfer from the device to my laptop:
>> >> 
>> >> CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
>> >> BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec
>> >> 
>> >> The transfer speed from my laptop to the device are the same with BFS
>> >> and CFS. I repeated the tests a few times just to be sure, and I will
>> >> check vmstat later.
>> > 
>> > Which exact mainline kernel have you tried? For anything performance 
>> > related running latest upstream -git (currently at 202c467) would be 
>> > recommended.
>>
>> I used the OpenWrt-patched 2.6.30. Support for the hardware that I 
>> tested with hasn't been merged upstream yet. Do you think that the 
>> scheduler related changes after 2.6.30 are relevant for non-SMP 
>> performance as well? If so, I'll work on a test with latest upstream 
>> -git with the necessary patches when I have time for it.
> 
> Dont know - it's hard to tell what happens without basic analysis tools. 
> Is there _any_ way to profile what happens on that system? (Do hrtimers 
> work on it that could be used to profile it?)
oprofile doesn't have any support for it (mips r4k, no generic
perfcounters), the only usable clock source is a simple cpu cycle
counter (which is also used for the timer interrupt).

- Felix

Re: BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Felix Fietkau <nbd@openwrt.org> wrote:

> Ingo Molnar wrote:
> > * Felix Fietkau <nbd@openwrt.org> wrote:
> > 
> >> Ingo Molnar wrote:
> >> > * Felix Fietkau <nbd@openwrt.org> wrote:
> >> >> I did some tests with BFS v230 vs CFS on Linux 2.6.30 on a different
> >> >> MIPS device (Atheros AR2317) with 180 MHz and 16 MB RAM. When running
> >> >> iperf tests, I consistently get the following results when running the
> >> >> transfer from the device to my laptop:
> >> >> 
> >> >> CFS: [  5]  0.0-60.0 sec    107 MBytes  15.0 Mbits/sec
> >> >> BFS: [  5]  0.0-60.0 sec    119 MBytes  16.6 Mbits/sec
> >> >> 
> >> >> The transfer speed from my laptop to the device are the same with BFS
> >> >> and CFS. I repeated the tests a few times just to be sure, and I will
> >> >> check vmstat later.
> >> > 
> >> > Which exact mainline kernel have you tried? For anything performance 
> >> > related running latest upstream -git (currently at 202c467) would be 
> >> > recommended.
> >>
> >> I used the OpenWrt-patched 2.6.30. Support for the hardware that I 
> >> tested with hasn't been merged upstream yet. Do you think that the 
> >> scheduler related changes after 2.6.30 are relevant for non-SMP 
> >> performance as well? If so, I'll work on a test with latest upstream 
> >> -git with the necessary patches when I have time for it.
> > 
> > Dont know - it's hard to tell what happens without basic analysis tools. 
> > Is there _any_ way to profile what happens on that system? (Do hrtimers 
> > work on it that could be used to profile it?)
>
> oprofile doesn't have any support for it (mips r4k, no generic 
> perfcounters), the only usable clock source is a simple cpu cycle 
> counter (which is also used for the timer interrupt).

A simple cpu cycle counter ought to be enough to get pretty good 
perfcounters support going on that box.

It takes a surprisingly small amount of code to do that, and a large 
portion of the perf tooling should then work out of box. Here's a few 
example commits of minimal perfcounters support, on other architectures:

 310d6b6: [S390] wire up sys_perf_counter_open
 2d4618d: parisc: perf: wire up sys_perf_counter_open
 19470e1: sh: Wire up sys_perf_counter_open.

Takes about 15 well placed lines of code, if there are no other 
complications on MIPS ;-)

	Ingo

Epic regression in throughput since v2.6.23

[Serge Belyshev]

Hi. I've done measurments of time taken by make -j4 kernel build
on a quadcore box.  Results are interesting: mainline kernel
has regressed since v2.6.23 release by more than 10%.

The following graph is time taken by "make -j4" (median over 9 runs)
versus kernel version.  The huge (10%) regressions since v2.6.23 is
apparent.  Note that tip/master c26f010 is better than current mainline.
Also note that BFS is significantly better than both and shows the same
throughput as vanilla v2.6.23:

http://img403.imageshack.us/img403/7029/epicmakej4.png


The following plot is a detailed comparison of time taken versus number
of parallel jobs. Note that at "make -j4" (which equals number of hardware
threads), BFS has the minimum (best performance), 
and tip/master -- maximum (worst).  I've also tested mainline v2.6.31
(not shown on the graph) which produces similar, albeit a bit slower,
results as the tip/master.

http://img179.imageshack.us/img179/5335/epicbfstip.png


Conclusion are
1) mainline has severely regressed since v2.6.23
2) BFS shows optimal performance at make -jN where N equals number of
   h/w threads, while current mainline scheduler performance is far from
   optimal in this case.

Re: Epic regression in throughput since v2.6.23

[Jesse Brandeburg]

On Tue, Sep 8, 2009 at 5:57 AM, Serge
Belyshev<belyshev@depni.sinp.msu.ru> wrote:
>
> Hi. I've done measurments of time taken by make -j4 kernel build
> on a quadcore box.  Results are interesting: mainline kernel
> has regressed since v2.6.23 release by more than 10%.

Is this related to why I now have to double the amount of threads X I
pass to make -jX, in order to use all my idle time for a kernel
compile?  I had noticed (without measuring exactly) that it seems with
each kernel released in this series mentioned, I had to increase my
number of worker threads, my common working model now is (cpus * 2) in
order to get zero idle time.

Sorry I haven't tested BFS yet, but am interested to see if it helps
interactivity when playing flash videos on my dual core laptop.

Re: Epic regression in throughput since v2.6.23

[Nikos Chantziaras]

On 09/08/2009 08:47 PM, Jesse Brandeburg wrote:
>[...]
> Sorry I haven't tested BFS yet, but am interested to see if it helps
> interactivity when playing flash videos on my dual core laptop.

Interactivity: yes (Flash will not result in the rest of the system 
lagging).

Flash videos: they will still play as bad as before.  BFS has no way to 
fix broken code inside Flash :P

Re: Epic regression in throughput since v2.6.23

[Jeff Garzik]

On 09/08/2009 01:47 PM, Jesse Brandeburg wrote:
> On Tue, Sep 8, 2009 at 5:57 AM, Serge
> Belyshev<belyshev@depni.sinp.msu.ru>  wrote:
>>
>> Hi. I've done measurments of time taken by make -j4 kernel build
>> on a quadcore box.  Results are interesting: mainline kernel
>> has regressed since v2.6.23 release by more than 10%.
>
> Is this related to why I now have to double the amount of threads X I
> pass to make -jX, in order to use all my idle time for a kernel
> compile?  I had noticed (without measuring exactly) that it seems with
> each kernel released in this series mentioned, I had to increase my
> number of worker threads, my common working model now is (cpus * 2) in
> order to get zero idle time.

You will almost certainly see idle CPUs/threads with "make -jN_CPUS" due 
to processes waiting for I/O.

If you're curious, there is also room for experimenting with make's "-l" 
argument, which caps the number of jobs based on load average rather 
than a static number of job slots.

	Jeff

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

Jeff Garzik <jeff@garzik.org> writes:

> You will almost certainly see idle CPUs/threads with "make -jN_CPUS"
> due to processes waiting for I/O.

Just to clarify: I have excluded all I/O effects in my plots completely
by building completely from tmpfs. Also, before each actual measurment
there was a thrown-off "pre-caching" one.  And my box has 8GB of RAM.

Re: Epic regression in throughput since v2.6.23

[Jeff Garzik]

On 09/08/2009 03:20 PM, Serge Belyshev wrote:
> Jeff Garzik<jeff@garzik.org>  writes:
>
>> You will almost certainly see idle CPUs/threads with "make -jN_CPUS"
>> due to processes waiting for I/O.
>
> Just to clarify: I have excluded all I/O effects in my plots completely
> by building completely from tmpfs. Also, before each actual measurment
> there was a thrown-off "pre-caching" one.  And my box has 8GB of RAM.

You could always one-up that by using ramfs ;)

	Jeff

Re: Epic regression in throughput since v2.6.23

[Nikos Chantziaras]

On 09/08/2009 03:57 PM, Serge Belyshev wrote:
>
> Hi. I've done measurments of time taken by make -j4 kernel build
> on a quadcore box.  Results are interesting: mainline kernel
> has regressed since v2.6.23 release by more than 10%.

It seems more people are starting to confirm this issue:

   http://foldingforum.org/viewtopic.php?f=44&t=11336

IMHO it's not *that* dramatic as some people there describe it ("Is it 
the holy grail?") but if something makes your desktop "smooth as silk" 
just like that, it might seem as a holy grail ;)  In any case, there 
clearly seems to be a performance problem with the mainline scheduler on 
many people's desktops that are being solved by BFS.

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

Serge Belyshev <belyshev@depni.sinp.msu.ru> writes:
>[snip]

I've updated the graphs, added kernels 2.6.24..2.6.29:
http://img186.imageshack.us/img186/7029/epicmakej4.png

And added comparison with best-performing 2.6.23 kernel:
http://img34.imageshack.us/img34/7563/epicbfstips.png

>
> Conclusions are
> 1) mainline has severely regressed since v2.6.23
> 2) BFS shows optimal performance at make -jN where N equals number of
>    h/w threads, while current mainline scheduler performance is far from
>    optimal in this case.

Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

* Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:

> Serge Belyshev <belyshev@depni.sinp.msu.ru> writes:
> >[snip]
> 
> I've updated the graphs, added kernels 2.6.24..2.6.29:
> http://img186.imageshack.us/img186/7029/epicmakej4.png
> 
> And added comparison with best-performing 2.6.23 kernel:
> http://img34.imageshack.us/img34/7563/epicbfstips.png

Thanks!

I think we found the reason for that regression - would you mind 
to re-test with latest -tip, e157986 or later?

If that works for you i'll describe our theory.

	Ingo

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

Ingo Molnar <mingo@elte.hu> writes:

> Thanks!
>
> I think we found the reason for that regression - would you mind 
> to re-test with latest -tip, e157986 or later?
>
> If that works for you i'll describe our theory.
>

Good job -- seems to work, thanks.  Regression is still about 3% though:
http://img3.imageshack.us/img3/5335/epicbfstip.png

Re: Epic regression in throughput since v2.6.23

[Cory Fields]

I've noticed the same regression since around 2.6.23, mainly in
multi-core video decoding. A git bisect reveals the guilty commit to
be: 33b0c4217dcd67b788318c3192a2912b530e4eef

It is easily visible because with the guilty commit included, one core
of the cpu remains pegged while the other(s) are severly
underutilized.

Hope this helps

Cory Fields

Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

* Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:

> Ingo Molnar <mingo@elte.hu> writes:
> 
> > Thanks!
> >
> > I think we found the reason for that regression - would you mind 
> > to re-test with latest -tip, e157986 or later?
> >
> > If that works for you i'll describe our theory.
> >
> 
> Good job -- seems to work, thanks.  Regression is still about 3% 
> though: http://img3.imageshack.us/img3/5335/epicbfstip.png

Ok, thanks for the update. The problem is that i've run out of 
testsystems that can reproduce this. So we need your help to debug 
this directly ...

A good start would be to post the -tip versus BFS "perf stat" 
measurement results:

   perf stat --repeat 3 make -j4 bzImage

And also the -j8 perf stat result, so that we can see what the 
difference is between -j4 and -j8.

Note: please check out latest tip and do:

  cd tools/perf/
  make -j install

To pick up the latest 'perf' tool. In particular the precision of 
--repeat has been improved recently so you want that binary from 
-tip even if you measure vanilla .31 or .31 based BFS.

Also, it would be nice if you could send me your kernel config - 
maybe it's some config detail that keeps me from being able to 
reproduce these results. I havent seen a link to a config in your 
mails (maybe i missed it - these threads are voluminous).

	Ingo

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

Ingo Molnar <mingo@elte.hu> writes:

>    perf stat --repeat 3 make -j4 bzImage

BFS hangs here:

[  128.859000] BUG: soft lockup - CPU#2 stuck for 61s! [sh:7946]
[  128.859016] Modules linked in:
[  128.859016] CPU 2:
[  128.859016] Modules linked in:
[  128.859016] Pid: 7946, comm: sh Not tainted 2.6.31-bfs211-dirty #4 GA-MA790FX-DQ6
[  128.859016] RIP: 0010:[<ffffffff81055a52>]  [<ffffffff81055a52>] task_oncpu_function_call+0x22/0x40
[  128.859016] RSP: 0018:ffff880205967e18  EFLAGS: 00000246
[  128.859016] RAX: 0000000000000002 RBX: ffff880205964cc0 RCX: 000000000000dd00
[  128.859016] RDX: ffff880211138c00 RSI: ffffffff8108d3f0 RDI: ffff88022e42a100
[  128.859016] RBP: ffffffff8102d76e R08: ffff880028066000 R09: 0000000000000000
[  128.859016] R10: 0000000000000000 R11: 0000000000000058 R12: ffffffff8108d3f0
[  128.859016] R13: ffff880211138c00 R14: 0000000000000001 R15: 000000000000e260
[  128.859016] FS:  00002b9ba0924e00(0000) GS:ffff880028066000(0000) knlGS:0000000000000000
[  128.859016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  128.859016] CR2: 00002b9ba091e4a8 CR3: 0000000001001000 CR4: 00000000000006e0
[  128.859016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  128.859016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  128.859016] Call Trace:
[  128.859016]  [<ffffffff8108ee3b>] ? perf_counter_remove_from_context+0x3b/0x90
[  128.859016]  [<ffffffff810904b4>] ? perf_counter_exit_task+0x114/0x340
[  128.859016]  [<ffffffff810c3f66>] ? filp_close+0x56/0x90
[  128.859016]  [<ffffffff8105d3ac>] ? do_exit+0x14c/0x6f0
[  128.859016]  [<ffffffff8105d991>] ? do_group_exit+0x41/0xb0
[  128.859016]  [<ffffffff8105da12>] ? sys_exit_group+0x12/0x20
[  128.859016]  [<ffffffff8102cceb>] ? system_call_fastpath+0x16/0x1b

So, got nothing to compare with.

> Also, it would be nice if you could send me your kernel config - 
> maybe it's some config detail that keeps me from being able to 
> reproduce these results. I havent seen a link to a config in your 
> mails (maybe i missed it - these threads are voluminous).

Attached.

[-- Attachment #2: .config --]
[-- Type: text/plain, Size: 52628 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.31-rc9
# Wed Sep  9 18:35:46 2009
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_HAVE_INTEL_TXT=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
CONFIG_LOCALVERSION_AUTO=y
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_BSD_PROCESS_ACCT=y
CONFIG_BSD_PROCESS_ACCT_V3=y
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
CONFIG_TREE_RCU=y
# CONFIG_TREE_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
# CONFIG_CGROUPS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_RELAY=y
CONFIG_NAMESPACES=y
# CONFIG_UTS_NS is not set
# CONFIG_IPC_NS is not set
# CONFIG_USER_NS is not set
# CONFIG_PID_NS is not set
# CONFIG_NET_NS is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
# CONFIG_EMBEDDED is not set
CONFIG_UID16=y
CONFIG_SYSCTL_SYSCALL=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
# CONFIG_KALLSYMS_EXTRA_PASS is not set
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_COUNTERS=y

#
# Performance Counters
#
CONFIG_PERF_COUNTERS=y
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
CONFIG_SLUB_DEBUG=y
# CONFIG_STRIP_ASM_SYMS is not set
# CONFIG_COMPAT_BRK is not set
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
CONFIG_PROFILING=y
# CONFIG_MARKERS is not set
# CONFIG_OPROFILE is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_DMA_API_DEBUG=y
CONFIG_HAVE_HW_BREAKPOINT=y

#
# GCOV-based kernel profiling
#
# CONFIG_SLOW_WORK is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_SLABINFO=y
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
# CONFIG_MODULE_FORCE_UNLOAD is not set
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
# CONFIG_IOSCHED_AS is not set
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
# CONFIG_FREEZER is not set

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
# CONFIG_SPARSE_IRQ is not set
CONFIG_X86_MPPARSE=y
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_SCHED_OMIT_FRAME_POINTER=y
# CONFIG_PARAVIRT_GUEST is not set
CONFIG_MEMTEST=y
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_CPU_SUP_INTEL=y
CONFIG_CPU_SUP_AMD=y
CONFIG_CPU_SUP_CENTAUR=y
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
# CONFIG_AMD_IOMMU is not set
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
# CONFIG_MAXSMP is not set
CONFIG_NR_CPUS=4
# CONFIG_SCHED_SMT is not set
# CONFIG_SCHED_MC is not set
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
CONFIG_X86_MCE=y
# CONFIG_X86_MCE_INTEL is not set
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
# CONFIG_X86_CPU_DEBUG is not set
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y
# CONFIG_MEMORY_HOTPLUG is not set
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HAVE_MLOCK=y
CONFIG_HAVE_MLOCKED_PAGE_BIT=y
CONFIG_MMU_NOTIFIER=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=0
# CONFIG_X86_CHECK_BIOS_CORRUPTION is not set
CONFIG_X86_RESERVE_LOW_64K=y
CONFIG_MTRR=y
# CONFIG_MTRR_SANITIZER is not set
CONFIG_X86_PAT=y
CONFIG_ARCH_USES_PG_UNCACHED=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100
CONFIG_SCHED_HRTICK=y
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PHYSICAL_START=0x200000
CONFIG_RELOCATABLE=y
CONFIG_PHYSICAL_ALIGN=0x1000000
# CONFIG_HOTPLUG_CPU is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_PM=y
# CONFIG_PM_DEBUG is not set
# CONFIG_SUSPEND is not set
# CONFIG_HIBERNATION is not set
CONFIG_ACPI=y
# CONFIG_ACPI_PROCFS is not set
# CONFIG_ACPI_PROCFS_POWER is not set
CONFIG_ACPI_SYSFS_POWER=y
# CONFIG_ACPI_PROC_EVENT is not set
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
# CONFIG_ACPI_FAN is not set
# CONFIG_ACPI_DOCK is not set
# CONFIG_ACPI_PROCESSOR is not set
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
# CONFIG_ACPI_PCI_SLOT is not set
CONFIG_X86_PM_TIMER=y
# CONFIG_ACPI_CONTAINER is not set
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
# CONFIG_CPU_FREQ is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_DMAR=y
CONFIG_DMAR_DEFAULT_ON=y
# CONFIG_DMAR_BROKEN_GFX_WA is not set
CONFIG_DMAR_FLOPPY_WA=y
# CONFIG_INTR_REMAP is not set
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
CONFIG_PCIEASPM=y
# CONFIG_PCIEASPM_DEBUG is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
# CONFIG_BINFMT_MISC is not set
CONFIG_IA32_EMULATION=y
# CONFIG_IA32_AOUT is not set
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
# CONFIG_IP_ROUTE_MULTIPATH is not set
CONFIG_IP_ROUTE_VERBOSE=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
CONFIG_NET_IPGRE=y
# CONFIG_NET_IPGRE_BROADCAST is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_SYN_COOKIES is not set
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
CONFIG_INET_TUNNEL=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
# CONFIG_INET_LRO is not set
# CONFIG_INET_DIAG is not set
# CONFIG_TCP_CONG_ADVANCED is not set
CONFIG_TCP_CONG_CUBIC=y
CONFIG_DEFAULT_TCP_CONG="cubic"
# CONFIG_TCP_MD5SIG is not set
CONFIG_IPV6=y
CONFIG_IPV6_PRIVACY=y
CONFIG_IPV6_ROUTER_PREF=y
CONFIG_IPV6_ROUTE_INFO=y
CONFIG_IPV6_OPTIMISTIC_DAD=y
# CONFIG_INET6_AH is not set
# CONFIG_INET6_ESP is not set
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET6_XFRM_MODE_TUNNEL is not set
# CONFIG_INET6_XFRM_MODE_BEET is not set
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_NETWORK_SECMARK is not set
CONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_NETFILTER_ADVANCED is not set

#
# Core Netfilter Configuration
#
CONFIG_NETFILTER_NETLINK=y
CONFIG_NETFILTER_NETLINK_LOG=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_CONNTRACK_FTP=y
CONFIG_NF_CONNTRACK_IRC=y
CONFIG_NF_CONNTRACK_SIP=y
CONFIG_NF_CT_NETLINK=y
CONFIG_NETFILTER_XTABLES=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER_XT_TARGET_NFLOG=y
CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
CONFIG_NETFILTER_XT_MATCH_MARK=y
CONFIG_NETFILTER_XT_MATCH_STATE=y
# CONFIG_IP_VS is not set

#
# IP: Netfilter Configuration
#
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_CONNTRACK_IPV4=y
CONFIG_NF_CONNTRACK_PROC_COMPAT=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_TARGET_REJECT=y
CONFIG_IP_NF_TARGET_LOG=y
CONFIG_IP_NF_TARGET_ULOG=y
CONFIG_NF_NAT=y
CONFIG_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=y
CONFIG_NF_NAT_FTP=y
CONFIG_NF_NAT_IRC=y
# CONFIG_NF_NAT_TFTP is not set
# CONFIG_NF_NAT_AMANDA is not set
# CONFIG_NF_NAT_PPTP is not set
# CONFIG_NF_NAT_H323 is not set
CONFIG_NF_NAT_SIP=y
CONFIG_IP_NF_MANGLE=y

#
# IPv6: Netfilter Configuration
#
CONFIG_NF_CONNTRACK_IPV6=y
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_MATCH_IPV6HEADER=y
CONFIG_IP6_NF_TARGET_LOG=y
CONFIG_IP6_NF_FILTER=y
CONFIG_IP6_NF_TARGET_REJECT=y
CONFIG_IP6_NF_MANGLE=y
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
CONFIG_STP=y
CONFIG_BRIDGE=y
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=y
# CONFIG_VLAN_8021Q_GVRP is not set
# CONFIG_DECNET is not set
CONFIG_LLC=y
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set

#
# Network testing
#
CONFIG_NET_PKTGEN=y
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
# CONFIG_BT is not set
# CONFIG_AF_RXRPC is not set
CONFIG_FIB_RULES=y
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
# CONFIG_RFKILL is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_DEBUG_DRIVER is not set
# CONFIG_DEBUG_DEVRES is not set
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=m
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=y
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
CONFIG_BLK_DEV_NBD=y
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
CONFIG_BLK_DEV_RAM=y
CONFIG_BLK_DEV_RAM_COUNT=16
CONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_XIP is not set
CONFIG_CDROM_PKTCDVD=y
CONFIG_CDROM_PKTCDVD_BUFFERS=64
CONFIG_CDROM_PKTCDVD_WCACHE=y
CONFIG_ATA_OVER_ETH=y
# CONFIG_VIRTIO_BLK is not set
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_ISL29003 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
# CONFIG_EEPROM_AT24 is not set
CONFIG_EEPROM_LEGACY=y
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
CONFIG_SCSI_TGT=y
# CONFIG_SCSI_NETLINK is not set
# CONFIG_SCSI_PROC_FS is not set

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=y
# CONFIG_BLK_DEV_SR_VENDOR is not set
CONFIG_CHR_DEV_SG=y
# CONFIG_CHR_DEV_SCH is not set
# CONFIG_SCSI_MULTI_LUN is not set
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
# CONFIG_SCSI_SPI_ATTRS is not set
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
CONFIG_SCSI_SAS_ATTRS=y
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
# CONFIG_SCSI_LOWLEVEL is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
# CONFIG_ATA_ACPI is not set
CONFIG_SATA_PMP=y
CONFIG_SATA_AHCI=y
CONFIG_SATA_SIL24=y
CONFIG_ATA_SFF=y
# CONFIG_SATA_SVW is not set
# CONFIG_ATA_PIIX is not set
# CONFIG_SATA_MV is not set
# CONFIG_SATA_NV is not set
# CONFIG_PDC_ADMA is not set
# CONFIG_SATA_QSTOR is not set
# CONFIG_SATA_PROMISE is not set
# CONFIG_SATA_SX4 is not set
# CONFIG_SATA_SIL is not set
# CONFIG_SATA_SIS is not set
# CONFIG_SATA_ULI is not set
# CONFIG_SATA_VIA is not set
# CONFIG_SATA_VITESSE is not set
# CONFIG_SATA_INIC162X is not set
# CONFIG_PATA_ALI is not set
# CONFIG_PATA_AMD is not set
# CONFIG_PATA_ARTOP is not set
# CONFIG_PATA_ATIIXP is not set
# CONFIG_PATA_CMD640_PCI is not set
# CONFIG_PATA_CMD64X is not set
# CONFIG_PATA_CS5520 is not set
# CONFIG_PATA_CS5530 is not set
# CONFIG_PATA_CYPRESS is not set
# CONFIG_PATA_EFAR is not set
# CONFIG_ATA_GENERIC is not set
# CONFIG_PATA_HPT366 is not set
# CONFIG_PATA_HPT37X is not set
# CONFIG_PATA_HPT3X2N is not set
# CONFIG_PATA_HPT3X3 is not set
# CONFIG_PATA_IT821X is not set
# CONFIG_PATA_IT8213 is not set
CONFIG_PATA_JMICRON=y
# CONFIG_PATA_TRIFLEX is not set
# CONFIG_PATA_MARVELL is not set
# CONFIG_PATA_MPIIX is not set
# CONFIG_PATA_OLDPIIX is not set
# CONFIG_PATA_NETCELL is not set
# CONFIG_PATA_NINJA32 is not set
# CONFIG_PATA_NS87410 is not set
# CONFIG_PATA_NS87415 is not set
# CONFIG_PATA_OPTI is not set
# CONFIG_PATA_OPTIDMA is not set
# CONFIG_PATA_PDC_OLD is not set
# CONFIG_PATA_RADISYS is not set
# CONFIG_PATA_RZ1000 is not set
# CONFIG_PATA_SC1200 is not set
# CONFIG_PATA_SERVERWORKS is not set
# CONFIG_PATA_PDC2027X is not set
# CONFIG_PATA_SIL680 is not set
# CONFIG_PATA_SIS is not set
# CONFIG_PATA_VIA is not set
# CONFIG_PATA_WINBOND is not set
# CONFIG_PATA_SCH is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
# CONFIG_MD_LINEAR is not set
CONFIG_MD_RAID0=y
CONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not set
# CONFIG_MD_RAID456 is not set
# CONFIG_MD_MULTIPATH is not set
CONFIG_MD_FAULTY=y
CONFIG_BLK_DEV_DM=y
# CONFIG_DM_DEBUG is not set
CONFIG_DM_CRYPT=y
# CONFIG_DM_SNAPSHOT is not set
# CONFIG_DM_MIRROR is not set
CONFIG_DM_ZERO=y
# CONFIG_DM_MULTIPATH is not set
# CONFIG_DM_DELAY is not set
# CONFIG_DM_UEVENT is not set
CONFIG_FUSION=y
# CONFIG_FUSION_SPI is not set
# CONFIG_FUSION_FC is not set
CONFIG_FUSION_SAS=y
CONFIG_FUSION_MAX_SGE=128
CONFIG_FUSION_CTL=m
CONFIG_FUSION_LOGGING=y

#
# IEEE 1394 (FireWire) support
#

#
# You can enable one or both FireWire driver stacks.
#

#
# See the help texts for more information.
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
CONFIG_DUMMY=y
# CONFIG_BONDING is not set
CONFIG_MACVLAN=y
# CONFIG_EQUALIZER is not set
CONFIG_TUN=y
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_NET_ETHERNET is not set
CONFIG_MII=y
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=y
CONFIG_R8169_VLAN=y
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_JME is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
# CONFIG_WLAN_80211 is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
CONFIG_PPP=y
CONFIG_PPP_MULTILINK=y
CONFIG_PPP_FILTER=y
CONFIG_PPP_ASYNC=y
CONFIG_PPP_SYNC_TTY=y
CONFIG_PPP_DEFLATE=y
CONFIG_PPP_BSDCOMP=y
CONFIG_PPP_MPPE=y
CONFIG_PPPOE=y
CONFIG_PPPOL2TP=y
# CONFIG_SLIP is not set
CONFIG_SLHC=y
# CONFIG_NET_FC is not set
CONFIG_NETCONSOLE=y
CONFIG_NETCONSOLE_DYNAMIC=y
CONFIG_NETPOLL=y
CONFIG_NETPOLL_TRAP=y
CONFIG_NET_POLL_CONTROLLER=y
# CONFIG_VIRTIO_NET is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
# CONFIG_INPUT_FF_MEMLESS is not set
# CONFIG_INPUT_POLLDEV is not set

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1280
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=800
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
# CONFIG_INPUT_JOYSTICK is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=y
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
# CONFIG_SERIO_SERPORT is not set
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
# CONFIG_GAMEPORT is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
CONFIG_DEVKMEM=y
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_FIX_EARLYCON_MEM=y
CONFIG_SERIAL_8250_PCI=y
CONFIG_SERIAL_8250_PNP=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_8250_RUNTIME_UARTS=4
# CONFIG_SERIAL_8250_EXTENDED is not set

#
# Non-8250 serial port support
#
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_VIRTIO_CONSOLE is not set
# CONFIG_IPMI_HANDLER is not set
CONFIG_HW_RANDOM=y
# CONFIG_HW_RANDOM_TIMERIOMEM is not set
# CONFIG_HW_RANDOM_INTEL is not set
CONFIG_HW_RANDOM_AMD=y
# CONFIG_HW_RANDOM_VIA is not set
# CONFIG_HW_RANDOM_VIRTIO is not set
# CONFIG_NVRAM is not set
CONFIG_RTC=y
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
# CONFIG_HANGCHECK_TIMER is not set
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_HELPER_AUTO=y

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_PIIX4=y
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Graphics adapter I2C/DDC channel drivers
#
# CONFIG_I2C_VOODOO3 is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_STUB is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
# CONFIG_SPI is not set

#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=y
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7473 is not set
# CONFIG_SENSORS_ADT7475 is not set
CONFIG_SENSORS_K8TEMP=y
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATK0110 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
CONFIG_SENSORS_IT87=y
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
# CONFIG_SENSORS_W83627EHF is not set
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
# CONFIG_THERMAL is not set
# CONFIG_THERMAL_HWMON is not set
CONFIG_WATCHDOG=y
# CONFIG_WATCHDOG_NOWAYOUT is not set

#
# Watchdog Device Drivers
#
# CONFIG_SOFT_WATCHDOG is not set
# CONFIG_ACQUIRE_WDT is not set
# CONFIG_ADVANTECH_WDT is not set
# CONFIG_ALIM1535_WDT is not set
# CONFIG_ALIM7101_WDT is not set
# CONFIG_SC520_WDT is not set
# CONFIG_EUROTECH_WDT is not set
# CONFIG_IB700_WDT is not set
# CONFIG_IBMASR is not set
# CONFIG_WAFER_WDT is not set
# CONFIG_I6300ESB_WDT is not set
# CONFIG_ITCO_WDT is not set
# CONFIG_IT8712F_WDT is not set
# CONFIG_IT87_WDT is not set
# CONFIG_HP_WATCHDOG is not set
# CONFIG_SC1200_WDT is not set
# CONFIG_PC87413_WDT is not set
# CONFIG_60XX_WDT is not set
# CONFIG_SBC8360_WDT is not set
# CONFIG_CPU5_WDT is not set
# CONFIG_SMSC_SCH311X_WDT is not set
# CONFIG_SMSC37B787_WDT is not set
# CONFIG_W83627HF_WDT is not set
# CONFIG_W83697HF_WDT is not set
# CONFIG_W83697UG_WDT is not set
# CONFIG_W83877F_WDT is not set
# CONFIG_W83977F_WDT is not set
# CONFIG_MACHZ_WDT is not set
# CONFIG_SBC_EPX_C3_WATCHDOG is not set

#
# PCI-based Watchdog Cards
#
# CONFIG_PCIPCWATCHDOG is not set
# CONFIG_WDTPCI is not set

#
# USB-based Watchdog Cards
#
# CONFIG_USBPCWATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_AB3100_CORE is not set
# CONFIG_REGULATOR is not set
# CONFIG_MEDIA_SUPPORT is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
# CONFIG_AGP_INTEL is not set
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
# CONFIG_DRM is not set
# CONFIG_VGASTATE is not set
# CONFIG_VIDEO_OUTPUT_CONTROL is not set
# CONFIG_FB is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=2048
CONFIG_DUMMY_CONSOLE=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=y
CONFIG_SND_PCM=y
CONFIG_SND_HWDEP=y
CONFIG_SND_RAWMIDI=y
CONFIG_SND_SEQUENCER=y
# CONFIG_SND_SEQ_DUMMY is not set
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=y
CONFIG_SND_PCM_OSS=y
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
# CONFIG_SND_HRTIMER is not set
# CONFIG_SND_RTCTIMER is not set
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_RAWMIDI_SEQ=y
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
# CONFIG_SND_EMU10K1_SEQ is not set
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
# CONFIG_SND_VIRMIDI is not set
# CONFIG_SND_SERIAL_U16550 is not set
# CONFIG_SND_MPU401 is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
# CONFIG_SND_EMU10K1 is not set
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=m
# CONFIG_SND_HDA_HWDEP is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
# CONFIG_SND_HDA_CODEC_ANALOG is not set
# CONFIG_SND_HDA_CODEC_SIGMATEL is not set
# CONFIG_SND_HDA_CODEC_VIA is not set
# CONFIG_SND_HDA_CODEC_ATIHDMI is not set
CONFIG_SND_HDA_CODEC_NVHDMI=y
CONFIG_SND_HDA_CODEC_INTELHDMI=y
CONFIG_SND_HDA_ELD=y
# CONFIG_SND_HDA_CODEC_CONEXANT is not set
CONFIG_SND_HDA_CODEC_CA0110=y
# CONFIG_SND_HDA_CODEC_CMEDIA is not set
# CONFIG_SND_HDA_CODEC_SI3054 is not set
CONFIG_SND_HDA_GENERIC=y
CONFIG_SND_HDA_POWER_SAVE=y
CONFIG_SND_HDA_POWER_SAVE_DEFAULT=0
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LX6464ES is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=y
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
# CONFIG_HID_DEBUG is not set
# CONFIG_HIDRAW is not set

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_CYPRESS=y
CONFIG_HID_DRAGONRISE=y
# CONFIG_DRAGONRISE_FF is not set
CONFIG_HID_EZKEY=y
CONFIG_HID_KYE=y
CONFIG_HID_GYRATION=y
CONFIG_HID_KENSINGTON=y
CONFIG_HID_LOGITECH=y
# CONFIG_LOGITECH_FF is not set
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
# CONFIG_PANTHERLORD_FF is not set
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
CONFIG_HID_GREENASIA=y
# CONFIG_GREENASIA_FF is not set
CONFIG_HID_SMARTJOYPLUS=y
# CONFIG_SMARTJOYPLUS_FF is not set
CONFIG_HID_TOPSEED=y
CONFIG_HID_THRUSTMASTER=y
# CONFIG_THRUSTMASTER_FF is not set
CONFIG_HID_ZEROPLUS=y
# CONFIG_ZEROPLUS_FF is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
# CONFIG_USB_DEBUG is not set
# CONFIG_USB_ANNOUNCE_NEW_DEVICES is not set

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
CONFIG_USB_DYNAMIC_MINORS=y
# CONFIG_USB_SUSPEND is not set
# CONFIG_USB_OTG is not set
# CONFIG_USB_MON is not set
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_EHCI_ROOT_HUB_TT=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
CONFIG_USB_UHCI_HCD=m
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
CONFIG_USB_STORAGE_DATAFAB=m
CONFIG_USB_STORAGE_FREECOM=m
CONFIG_USB_STORAGE_ISD200=m
CONFIG_USB_STORAGE_USBAT=m
CONFIG_USB_STORAGE_SDDR09=m
CONFIG_USB_STORAGE_SDDR55=m
CONFIG_USB_STORAGE_JUMPSHOT=m
CONFIG_USB_STORAGE_ALAUDA=m
CONFIG_USB_STORAGE_ONETOUCH=m
CONFIG_USB_STORAGE_KARMA=m
CONFIG_USB_STORAGE_CYPRESS_ATACB=m
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
CONFIG_USB_SERIAL=m
# CONFIG_USB_EZUSB is not set
CONFIG_USB_SERIAL_GENERIC=y
# CONFIG_USB_SERIAL_AIRCABLE is not set
# CONFIG_USB_SERIAL_ARK3116 is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_CH341 is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_CP210X is not set
# CONFIG_USB_SERIAL_CYPRESS_M8 is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_FUNSOFT is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_GARMIN is not set
# CONFIG_USB_SERIAL_IPW is not set
# CONFIG_USB_SERIAL_IUU is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_MOS7720 is not set
# CONFIG_USB_SERIAL_MOS7840 is not set
# CONFIG_USB_SERIAL_MOTOROLA is not set
# CONFIG_USB_SERIAL_NAVMAN is not set
CONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_OTI6858 is not set
# CONFIG_USB_SERIAL_QUALCOMM is not set
# CONFIG_USB_SERIAL_SPCP8X5 is not set
# CONFIG_USB_SERIAL_HP4X is not set
# CONFIG_USB_SERIAL_SAFE is not set
# CONFIG_USB_SERIAL_SIEMENS_MPI is not set
# CONFIG_USB_SERIAL_SIERRAWIRELESS is not set
# CONFIG_USB_SERIAL_SYMBOL is not set
# CONFIG_USB_SERIAL_TI is not set
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OPTION is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_SERIAL_OPTICON is not set
# CONFIG_USB_SERIAL_DEBUG is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set

#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
# CONFIG_NEW_LEDS is not set
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
CONFIG_EDAC=y

#
# Reporting subsystems
#
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_MM_EDAC=y
CONFIG_EDAC_AMD64=y
# CONFIG_EDAC_AMD64_ERROR_INJECTION is not set
# CONFIG_EDAC_E752X is not set
# CONFIG_EDAC_I82975X is not set
# CONFIG_EDAC_I3000 is not set
# CONFIG_EDAC_X38 is not set
# CONFIG_EDAC_I5400 is not set
# CONFIG_EDAC_I5000 is not set
# CONFIG_EDAC_I5100 is not set
# CONFIG_RTC_CLASS is not set
# CONFIG_DMADEVICES is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set

#
# TI VLYNQ
#
CONFIG_STAGING=y
# CONFIG_STAGING_EXCLUDE_BUILD is not set
# CONFIG_ET131X is not set
# CONFIG_SLICOSS is not set
# CONFIG_ME4000 is not set
# CONFIG_MEILHAUS is not set
# CONFIG_USB_IP_COMMON is not set
# CONFIG_ECHO is not set
# CONFIG_COMEDI is not set
# CONFIG_ASUS_OLED is not set
# CONFIG_ALTERA_PCIE_CHDMA is not set
# CONFIG_INPUT_MIMIO is not set
# CONFIG_TRANZPORT is not set
# CONFIG_EPL is not set

#
# Android
#
# CONFIG_ANDROID is not set
# CONFIG_DST is not set
CONFIG_POHMELFS=m
# CONFIG_POHMELFS_DEBUG is not set
# CONFIG_POHMELFS_CRYPTO is not set
# CONFIG_B3DFG is not set
# CONFIG_IDE_PHISON is not set
# CONFIG_PLAN9AUTH is not set
# CONFIG_HECI is not set
# CONFIG_LINE6_USB is not set
# CONFIG_USB_SERIAL_QUATECH2 is not set
# CONFIG_VT6655 is not set
# CONFIG_USB_CPC is not set
# CONFIG_RDC_17F3101X is not set
# CONFIG_X86_PLATFORM_DEVICES is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=m
CONFIG_EXT2_FS_XATTR=y
CONFIG_EXT2_FS_POSIX_ACL=y
CONFIG_EXT2_FS_SECURITY=y
# CONFIG_EXT2_FS_XIP is not set
CONFIG_EXT3_FS=m
CONFIG_EXT3_DEFAULTS_TO_ORDERED=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_EXT3_FS_POSIX_ACL=y
CONFIG_EXT3_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_EXT4DEV_COMPAT=y
CONFIG_EXT4_FS_XATTR=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_JBD=m
# CONFIG_JBD_DEBUG is not set
CONFIG_JBD2=y
# CONFIG_JBD2_DEBUG is not set
CONFIG_FS_MBCACHE=y
# CONFIG_REISERFS_FS is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
CONFIG_BTRFS_FS=m
CONFIG_BTRFS_FS_POSIX_ACL=y
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
CONFIG_FUSE_FS=y
# CONFIG_CUSE is not set

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=y
# CONFIG_MSDOS_FS is not set
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=1251
CONFIG_FAT_DEFAULT_IOCHARSET="utf8"
# CONFIG_NTFS_FS is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_VMCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
# CONFIG_TMPFS_POSIX_ACL is not set
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
CONFIG_CONFIGFS_FS=y
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set

#
# Partition Types
#
CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
CONFIG_OSF_PARTITION=y
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not set
CONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_KARMA_PARTITION is not set
# CONFIG_EFI_PARTITION is not set
# CONFIG_SYSV68_PARTITION is not set
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
CONFIG_NLS_CODEPAGE_866=y
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
CONFIG_NLS_CODEPAGE_1251=y
# CONFIG_NLS_ASCII is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
CONFIG_NLS_ISO8859_5=y
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
CONFIG_NLS_KOI8_R=y
CONFIG_NLS_KOI8_U=y
CONFIG_NLS_UTF8=y
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
CONFIG_ALLOW_WARNINGS=y
CONFIG_ENABLE_WARN_DEPRECATED=y
CONFIG_ENABLE_MUST_CHECK=y
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_SECTION_MISMATCH is not set
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
# CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC is not set
CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC_VALUE=0
CONFIG_DETECT_HUNG_TASK=y
# CONFIG_BOOTPARAM_HUNG_TASK_PANIC is not set
CONFIG_BOOTPARAM_HUNG_TASK_PANIC_VALUE=0
# CONFIG_SCHED_DEBUG is not set
# CONFIG_SCHEDSTATS is not set
# CONFIG_TIMER_STATS is not set
# CONFIG_DEBUG_OBJECTS is not set
# CONFIG_SLUB_DEBUG_ON is not set
# CONFIG_SLUB_STATS is not set
# CONFIG_DEBUG_KMEMLEAK is not set
# CONFIG_DEBUG_RT_MUTEXES is not set
# CONFIG_RT_MUTEX_TESTER is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_DEBUG_MUTEXES is not set
# CONFIG_DEBUG_LOCK_ALLOC is not set
# CONFIG_PROVE_LOCKING is not set
# CONFIG_LOCK_STAT is not set
# CONFIG_DEBUG_SPINLOCK_SLEEP is not set
# CONFIG_DEBUG_LOCKING_API_SELFTESTS is not set
# CONFIG_DEBUG_KOBJECT is not set
CONFIG_DEBUG_BUGVERBOSE=y
CONFIG_DEBUG_INFO=y
# CONFIG_DEBUG_VM is not set
# CONFIG_DEBUG_VIRTUAL is not set
# CONFIG_DEBUG_WRITECOUNT is not set
CONFIG_DEBUG_MEMORY_INIT=y
# CONFIG_DEBUG_LIST is not set
# CONFIG_DEBUG_SG is not set
# CONFIG_DEBUG_NOTIFIERS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
# CONFIG_FRAME_POINTER is not set
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
# CONFIG_BACKTRACE_SELF_TEST is not set
# CONFIG_DEBUG_BLOCK_EXT_DEVT is not set
# CONFIG_FAULT_INJECTION is not set
# CONFIG_LATENCYTOP is not set
CONFIG_SYSCTL_SYSCALL_CHECK=y
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
# CONFIG_FUNCTION_TRACER is not set
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
# CONFIG_SCHED_TRACER is not set
# CONFIG_ENABLE_DEFAULT_TRACERS is not set
# CONFIG_FTRACE_SYSCALLS is not set
# CONFIG_BOOT_TRACER is not set
CONFIG_BRANCH_PROFILE_NONE=y
# CONFIG_PROFILE_ANNOTATED_BRANCHES is not set
# CONFIG_PROFILE_ALL_BRANCHES is not set
# CONFIG_POWER_TRACER is not set
# CONFIG_KSYM_TRACER is not set
# CONFIG_STACK_TRACER is not set
# CONFIG_KMEMTRACE is not set
# CONFIG_WORKQUEUE_TRACER is not set
# CONFIG_BLK_DEV_IO_TRACE is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
# CONFIG_KGDB is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_KMEMCHECK is not set
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_NX_TEST is not set
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
CONFIG_IO_DELAY_0X80=y
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
# CONFIG_IO_DELAY_NONE is not set
CONFIG_DEFAULT_IO_DELAY_TYPE=0
# CONFIG_DEBUG_BOOT_PARAMS is not set
# CONFIG_CPA_DEBUG is not set
# CONFIG_OPTIMIZE_INLINING is not set

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_INTEL_TXT is not set
# CONFIG_IMA is not set
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
# CONFIG_CRYPTO_FIPS is not set
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
CONFIG_CRYPTO_GF128MUL=y
# CONFIG_CRYPTO_NULL is not set
CONFIG_CRYPTO_WORKQUEUE=y
CONFIG_CRYPTO_CRYPTD=y
# CONFIG_CRYPTO_AUTHENC is not set
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
CONFIG_CRYPTO_LRW=y
# CONFIG_CRYPTO_PCBC is not set
CONFIG_CRYPTO_XTS=y

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=m
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=y
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_MICHAEL_MIC=y
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
CONFIG_CRYPTO_WP512=y

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_AES_X86_64=y
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
CONFIG_CRYPTO_TWOFISH=y
CONFIG_CRYPTO_TWOFISH_COMMON=y
CONFIG_CRYPTO_TWOFISH_X86_64=y

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
# CONFIG_CRYPTO_HW is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
# CONFIG_KVM_INTEL is not set
CONFIG_KVM_AMD=y
# CONFIG_KVM_TRACE is not set
CONFIG_VIRTIO=y
CONFIG_VIRTIO_RING=y
CONFIG_VIRTIO_PCI=y
CONFIG_VIRTIO_BALLOON=y
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=y
CONFIG_CRC16=y
# CONFIG_CRC_T10DIF is not set
CONFIG_CRC_ITU_T=y
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=y
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y

Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

* Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:

> Ingo Molnar <mingo@elte.hu> writes:
> 
> >    perf stat --repeat 3 make -j4 bzImage
> 
> BFS hangs here:
> 
> [  128.859000] BUG: soft lockup - CPU#2 stuck for 61s! [sh:7946]
> [  128.859016] Modules linked in:
> [  128.859016] CPU 2:
> [  128.859016] Modules linked in:
> [  128.859016] Pid: 7946, comm: sh Not tainted 2.6.31-bfs211-dirty #4 GA-MA790FX-DQ6
> [  128.859016] RIP: 0010:[<ffffffff81055a52>]  [<ffffffff81055a52>] task_oncpu_function_call+0x22/0x40
> [  128.859016] RSP: 0018:ffff880205967e18  EFLAGS: 00000246
> [  128.859016] RAX: 0000000000000002 RBX: ffff880205964cc0 RCX: 000000000000dd00
> [  128.859016] RDX: ffff880211138c00 RSI: ffffffff8108d3f0 RDI: ffff88022e42a100
> [  128.859016] RBP: ffffffff8102d76e R08: ffff880028066000 R09: 0000000000000000
> [  128.859016] R10: 0000000000000000 R11: 0000000000000058 R12: ffffffff8108d3f0
> [  128.859016] R13: ffff880211138c00 R14: 0000000000000001 R15: 000000000000e260
> [  128.859016] FS:  00002b9ba0924e00(0000) GS:ffff880028066000(0000) knlGS:0000000000000000
> [  128.859016] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [  128.859016] CR2: 00002b9ba091e4a8 CR3: 0000000001001000 CR4: 00000000000006e0
> [  128.859016] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  128.859016] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [  128.859016] Call Trace:
> [  128.859016]  [<ffffffff8108ee3b>] ? perf_counter_remove_from_context+0x3b/0x90
> [  128.859016]  [<ffffffff810904b4>] ? perf_counter_exit_task+0x114/0x340
> [  128.859016]  [<ffffffff810c3f66>] ? filp_close+0x56/0x90
> [  128.859016]  [<ffffffff8105d3ac>] ? do_exit+0x14c/0x6f0
> [  128.859016]  [<ffffffff8105d991>] ? do_group_exit+0x41/0xb0
> [  128.859016]  [<ffffffff8105da12>] ? sys_exit_group+0x12/0x20
> [  128.859016]  [<ffffffff8102cceb>] ? system_call_fastpath+0x16/0x1b
> 
> So, got nothing to compare with.

Could still compare -j5 to -j4 on -tip, to see why -j4 is 3% short 
of -j5's throughput.

(Plus maybe the NEW_FAIR_SLEEPERS change in -tip fixes the 3% drop.)

	Ingo

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

Ingo Molnar <mingo@elte.hu> writes:

> Could still compare -j5 to -j4 on -tip, to see why -j4 is 3% short 
> of -j5's throughput.
>
> (Plus maybe the NEW_FAIR_SLEEPERS change in -tip fixes the 3% drop.)

Will do in about 12 hours or so.

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

[-- Attachment #1: Type: text/plain, Size: 5259 bytes --]

Ingo Molnar <mingo@elte.hu> writes:

> Could still compare -j5 to -j4 on -tip, to see why -j4 is 3% short 
> of -j5's throughput.
>
> (Plus maybe the NEW_FAIR_SLEEPERS change in -tip fixes the 3% drop.)

I've hacked the "perf" tool a bit (see the attached patch) and obtained
the following results with "perf stat -r9 make -j4 -- make clean":

--------------------------------------------------------------------------------
2.6.31-tip-01381-gf4c92b6:

 Performance counter stats for 'make -j4' (9 runs):

   94730.871493  task-clock-msecs         #      3.503 CPUs    ( +-   0.022% )
          17499  context-switches         #      0.000 M/sec   ( +-   0.270% )
           1127  CPU-migrations           #      0.000 M/sec   ( +-   1.159% )
        4601950  page-faults              #      0.049 M/sec   ( +-   0.001% )
   237853044260  cycles                   #   2510.829 M/sec   ( +-   0.022% )
   200554338937  instructions             #      0.843 IPC     ( +-   0.000% )
    70615226021  cache-references         #    745.430 M/sec   ( +-   0.002% )
     2268068445  cache-misses             #     23.942 M/sec   ( +-   0.020% )

   27.046390652  seconds time elapsed   ( +-   0.085% )


 Performance counter stats for 'make -j5' (9 runs):

   95283.766024  task-clock-msecs         #      3.541 CPUs    ( +-   0.024% )
          20612  context-switches         #      0.000 M/sec   ( +-   0.297% )
           1644  CPU-migrations           #      0.000 M/sec   ( +-   0.711% )
        4601733  page-faults              #      0.048 M/sec   ( +-   0.001% )
   239234709890  cycles                   #   2510.760 M/sec   ( +-   0.024% )
   200562340756  instructions             #      0.838 IPC     ( +-   0.000% )
    70575215911  cache-references         #    740.685 M/sec   ( +-   0.004% )
     2266361085  cache-misses             #     23.785 M/sec   ( +-   0.017% )

   26.909772228  seconds time elapsed   ( +-   0.041% )


 Performance counter stats for 'make -j6' (9 runs):

   95664.586804  task-clock-msecs         #      3.568 CPUs    ( +-   0.020% )
          22358  context-switches         #      0.000 M/sec   ( +-   0.194% )
           1835  CPU-migrations           #      0.000 M/sec   ( +-   0.659% )
        4601777  page-faults              #      0.048 M/sec   ( +-   0.001% )
   240188102235  cycles                   #   2510.732 M/sec   ( +-   0.020% )
   200569036210  instructions             #      0.835 IPC     ( +-   0.000% )
    70565619805  cache-references         #    737.636 M/sec   ( +-   0.006% )
     2265714056  cache-misses             #     23.684 M/sec   ( +-   0.017% )

   26.808072908  seconds time elapsed   ( +-   0.063% )
--------------------------------------------------------------------------------
2.6.31-tip-01385-g1ca1afc:

 Performance counter stats for 'make -j4' (9 runs):

   94873.128287  task-clock-msecs         #      3.422 CPUs    ( +-   0.020% )
          13196  context-switches         #      0.000 M/sec   ( +-   0.181% )
           1777  CPU-migrations           #      0.000 M/sec   ( +-   0.664% )
        4601784  page-faults              #      0.049 M/sec   ( +-   0.001% )
   238126976192  cycles                   #   2509.952 M/sec   ( +-   0.020% )
   200545291785  instructions             #      0.842 IPC     ( +-   0.000% )
    70607104279  cache-references         #    744.227 M/sec   ( +-   0.005% )
     2266980390  cache-misses             #     23.895 M/sec   ( +-   0.022% )

   27.723375828  seconds time elapsed   ( +-   0.134% )


 Performance counter stats for 'make -j5' (9 runs):

   95363.595017  task-clock-msecs         #      3.519 CPUs    ( +-   0.015% )
          15003  context-switches         #      0.000 M/sec   ( +-   0.294% )
           1880  CPU-migrations           #      0.000 M/sec   ( +-   1.017% )
        4601648  page-faults              #      0.048 M/sec   ( +-   0.001% )
   239351549481  cycles                   #   2509.884 M/sec   ( +-   0.015% )
   200552379266  instructions             #      0.838 IPC     ( +-   0.000% )
    70576977633  cache-references         #    740.083 M/sec   ( +-   0.006% )
     2265294365  cache-misses             #     23.754 M/sec   ( +-   0.012% )

   27.096710006  seconds time elapsed   ( +-   0.074% )


 Performance counter stats for 'make -j6' (9 runs):

   95739.941233  task-clock-msecs         #      3.568 CPUs    ( +-   0.024% )
          16327  context-switches         #      0.000 M/sec   ( +-   0.230% )
           1863  CPU-migrations           #      0.000 M/sec   ( +-   0.623% )
        4601715  page-faults              #      0.048 M/sec   ( +-   0.002% )
   240292969273  cycles                   #   2509.851 M/sec   ( +-   0.024% )
   200554772612  instructions             #      0.835 IPC     ( +-   0.000% )
    70566092747  cache-references         #    737.060 M/sec   ( +-   0.004% )
     2264738533  cache-misses             #     23.655 M/sec   ( +-   0.021% )

   26.836474777  seconds time elapsed   ( +-   0.181% )
--------------------------------------------------------------------------------

Note that the disabling NEW_FAIR_SLEEPERS doesn't fix 3% regression from
v2.6.23, but instead makes "make -j4" runtime another 2% worse (27.05 -> 27.72).


[-- Attachment #2: perf cleanup-command patch --]
[-- Type: text/plain, Size: 1716 bytes --]

---
 tools/perf/builtin-stat.c |   18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

Index: linux/tools/perf/builtin-stat.c
===================================================================
--- linux.orig/tools/perf/builtin-stat.c
+++ linux/tools/perf/builtin-stat.c
@@ -44,6 +44,7 @@
 #include "util/parse-events.h"
 #include "util/event.h"
 #include "util/debug.h"
+#include "util/run-command.h"
 
 #include <sys/prctl.h>
 #include <math.h>
@@ -458,7 +459,7 @@ static const struct option options[] = {
 
 int cmd_stat(int argc, const char **argv, const char *prefix __used)
 {
-	int status;
+	int status, j, cleanup, cleanup_argc;
 
 	argc = parse_options(argc, argv, options, stat_usage,
 		PARSE_OPT_STOP_AT_NON_OPTION);
@@ -467,6 +468,19 @@ int cmd_stat(int argc, const char **argv
 	if (run_count <= 0)
 		usage_with_options(stat_usage, options);
 
+	// quick ugly hack: if a "--" appears in the command, treat is as
+	// a delimiter and use remaining part as a "cleanup command",
+	// not affecting performance counters.
+	cleanup = cleanup_argc = 0;
+	for (j = 1; j < (argc-1); j ++) {
+		if (!strcmp (argv[j], "--")) {
+			cleanup = j + 1;
+			cleanup_argc = argc - j - 1;
+			argv[j] = NULL;
+			argc = j;
+		}
+	}
+
 	/* Set attrs and nr_counters if no event is selected and !null_run */
 	if (!null_run && !nr_counters) {
 		memcpy(attrs, default_attrs, sizeof(default_attrs));
@@ -493,6 +507,8 @@ int cmd_stat(int argc, const char **argv
 		if (run_count != 1 && verbose)
 			fprintf(stderr, "[ perf stat: executing run #%d ... ]\n", run_idx + 1);
 		status = run_perf_stat(argc, argv);
+		if (cleanup)
+			run_command_v_opt (&argv [cleanup], 0);
 	}
 
 	print_stat(argc, argv);

Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

* Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:

> Note that the disabling NEW_FAIR_SLEEPERS doesn't fix 3% 
> regression from v2.6.23, but instead makes "make -j4" runtime 
> another 2% worse (27.05 -> 27.72).

ok - thanks for the numbers, will have a look.

> ---
>  tools/perf/builtin-stat.c |   18 +++++++++++++++++-
>  1 file changed, 17 insertions(+), 1 deletion(-)

> +	// quick ugly hack: if a "--" appears in the command, treat is as
> +	// a delimiter and use remaining part as a "cleanup command",
> +	// not affecting performance counters.
> +	cleanup = cleanup_argc = 0;
> +	for (j = 1; j < (argc-1); j ++) {
> +		if (!strcmp (argv[j], "--")) {
> +			cleanup = j + 1;
> +			cleanup_argc = argc - j - 1;
> +			argv[j] = NULL;
> +			argc = j;
> +		}
> +	}

Nice feature!

How about doing it a bit cleaner, as '--repeat-prepare' and 
'--repeat-cleanup' options, to allow both pre-repeat and post-repeat 
cleanup ops to be done outside of the measured period?

	Ingo

Re: Epic regression in throughput since v2.6.23

[Mike Galbraith]

On Sun, 2009-09-13 at 17:47 +0200, Ingo Molnar wrote:
> * Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:
> 
> > Note that the disabling NEW_FAIR_SLEEPERS doesn't fix 3% 
> > regression from v2.6.23, but instead makes "make -j4" runtime 
> > another 2% worse (27.05 -> 27.72).
> 
> ok - thanks for the numbers, will have a look.

Seems NEXT_BUDDY is hurting the -j4 build.

LAST_BUDDY helps, which makes some sense.. if a task has heated up
cache, and is wakeup preempted by a fast mover (kthread, make..), it can
get the CPU back with still toasty data.  Hm.  If NEXT_BUDDY is on, that
benefit would likely be frequently destroyed too, because NEXT_BUDDY is
preferred over LAST_BUDDY.

Anyway, I'm thinking of tracking forks/sec as a means of detecting the
fork/exec load.  Or, maybe just enable it when there's > 1 buddy pair
running.. or something.  After all, NEXT_BUDDY is about scalability, and
make -j4 on a quad surely doesn't need any scalability help :)

 Performance counter stats for 'make -j4 vmlinux':

stock
  111.625198810  seconds time elapsed  avg 112.120  1.00
  112.209501685  seconds time elapsed
  112.528258240  seconds time elapsed

NO_NEXT_BUDDY NO_LAST_BUDDY
  109.405064078  seconds time elapsed  avg 109.351  .975
  108.708076118  seconds time elapsed
  109.942346026  seconds time elapsed

NO_NEXT_BUDDY
  108.005756718  seconds time elapsed  avg 108.064  .963
  107.689862679  seconds time elapsed
  108.497117555  seconds time elapsed

NO_LAST_BUDDY
  110.208717063  seconds time elapsed  avg 110.120  .982
  110.362412902  seconds time elapsed
  109.791359601  seconds time elapsed


diff --git a/kernel/sched_fair.c b/kernel/sched_fair.c
index aa7f841..7cfea64 100644
--- a/kernel/sched_fair.c
+++ b/kernel/sched_fair.c
@@ -1501,7 +1501,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int sync)
 	 */
 	if (sched_feat(LAST_BUDDY) && likely(se->on_rq && curr != rq->idle))
 		set_last_buddy(se);
-	set_next_buddy(pse);
+	if (sched_feat(NEXT_BUDDY))
+		set_next_buddy(pse);
 
 	/*
 	 * We can come here with TIF_NEED_RESCHED already set from new task
diff --git a/kernel/sched_features.h b/kernel/sched_features.h
index e2dc63a..6e7070b 100644
--- a/kernel/sched_features.h
+++ b/kernel/sched_features.h
@@ -13,5 +13,6 @@ SCHED_FEAT(LB_BIAS, 1)
 SCHED_FEAT(LB_WAKEUP_UPDATE, 1)
 SCHED_FEAT(ASYM_EFF_LOAD, 1)
 SCHED_FEAT(WAKEUP_OVERLAP, 0)
+SCHED_FEAT(NEXT_BUDDY, 1)
 SCHED_FEAT(LAST_BUDDY, 1)
 SCHED_FEAT(OWNER_SPIN, 1)

Re: Epic regression in throughput since v2.6.23

[Mike Galbraith]

On Sun, 2009-09-13 at 21:17 +0200, Mike Galbraith wrote:

> Anyway, I'm thinking of tracking forks/sec as a means of detecting the
> fork/exec load.  Or, maybe just enable it when there's > 1 buddy pair
> running.. or something.  After all, NEXT_BUDDY is about scalability, and
> make -j4 on a quad surely doesn't need any scalability help :)

But, this buddy vs fork/exec thing is not at all cut and dried.  Even
with fork/exec load being the primary CPU consumer, there are genuine
buddies to worry about when you've got a GUI running, next/last buddy
can reduce the chances that an oinker slips in between X and client.

Ponder...

(oil for rusty old ponder machine welcome, gears grinding)

	-Mike

Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

* Serge Belyshev <belyshev@depni.sinp.msu.ru> wrote:

> Note that the disabling NEW_FAIR_SLEEPERS doesn't fix 3% regression 
> from v2.6.23, but instead makes "make -j4" runtime another 2% worse 
> (27.05 -> 27.72).

Ok, i think we've got a handle on that finally - mind checking latest 
-tip?

	Ingo

Re: Epic regression in throughput since v2.6.23

[Serge Belyshev]

Ingo Molnar <mingo@elte.hu> writes:

> Ok, i think we've got a handle on that finally - mind checking latest 
> -tip?

Kernel build benchmark:
http://img11.imageshack.us/img11/4544/makej20090916.png

I have also repeated video encode benchmarks described here:
http://article.gmane.org/gmane.linux.kernel/889444

"x264 --preset ultrafast":
http://img11.imageshack.us/img11/9020/ultrafast20090916.png

"x264 --preset medium":
http://img11.imageshack.us/img11/7729/medium20090916.png

[patchlet] Re: Epic regression in throughput since v2.6.23

[Mike Galbraith]

On Wed, 2009-09-16 at 23:18 +0000, Serge Belyshev wrote:
> Ingo Molnar <mingo@elte.hu> writes:
> 
> > Ok, i think we've got a handle on that finally - mind checking latest 
> > -tip?
> 
> Kernel build benchmark:
> http://img11.imageshack.us/img11/4544/makej20090916.png
> 
> I have also repeated video encode benchmarks described here:
> http://article.gmane.org/gmane.linux.kernel/889444
> 
> "x264 --preset ultrafast":
> http://img11.imageshack.us/img11/9020/ultrafast20090916.png
> 
> "x264 --preset medium":
> http://img11.imageshack.us/img11/7729/medium20090916.png

Pre-ramble..
Most of the performance differences I've examined in all these CFS vs
BFS threads boil down to fair scheduler vs unfair scheduler.  If you
favor hogs, naturally, hogs getting more bandwidth perform better than
hogs getting their fair share.  That's wonderful for hogs, somewhat less
than wonderful for their competition.  That fairness is not necessarily
the best thing for throughput is well known.  If you've got a single
dissimilar task load running alone, favoring hogs may perform better..
or not.  What about mixed loads though?  Is the throughput of frequent
switchers less important than hog throughput?

Moving right along..

That x264 thing uncovered an interesting issue within CFS.  That load is
a frequent clone() customer, and when it has to compete against a not so
fork/clone happy load, it suffers mightily.  Even when running solo, ie
only competing against it's own siblings, IFF sleeper fairness is
enabled, the pain of thread startup latency is quite visible.  With
concurrent loads, it is agonizingly painful.

concurrent load test
tbench 8 vs
x264 --preset ultrafast --no-scenecut --sync-lookahead 0 --qp 20 -o /dev/null --threads 8 soccer_4cif.y4m

(i can turn knobs and get whatever numbers i want, including
outperforming bfs, concurrent or solo.. not the point)

START_DEBIT
encoded 600 frames, 44.29 fps, 22096.60 kb/s
encoded 600 frames, 43.59 fps, 22096.60 kb/s
encoded 600 frames, 43.78 fps, 22096.60 kb/s
encoded 600 frames, 43.77 fps, 22096.60 kb/s
encoded 600 frames, 45.67 fps, 22096.60 kb/s

8   1068214   672.35 MB/sec  execute  57 sec
8   1083785   672.16 MB/sec  execute  58 sec
8   1099188   672.18 MB/sec  execute  59 sec
8   1114626   672.00 MB/sec  cleanup  60 sec
8   1114626   671.96 MB/sec  cleanup  60 sec

NO_START_DEBIT
encoded 600 frames, 123.19 fps, 22096.60 kb/s
encoded 600 frames, 123.85 fps, 22096.60 kb/s
encoded 600 frames, 120.05 fps, 22096.60 kb/s
encoded 600 frames, 123.43 fps, 22096.60 kb/s
encoded 600 frames, 121.27 fps, 22096.60 kb/s

8    848135   533.79 MB/sec  execute  57 sec
8    860829   534.08 MB/sec  execute  58 sec
8    872840   533.74 MB/sec  execute  59 sec
8    885036   533.66 MB/sec  cleanup  60 sec
8    885036   533.64 MB/sec  cleanup  60 sec

2.6.31-bfs221-smp
encoded 600 frames, 169.00 fps, 22096.60 kb/s
encoded 600 frames, 163.85 fps, 22096.60 kb/s
encoded 600 frames, 161.00 fps, 22096.60 kb/s
encoded 600 frames, 155.57 fps, 22096.60 kb/s
encoded 600 frames, 162.01 fps, 22096.60 kb/s

8    458328   287.67 MB/sec  execute  57 sec
8    464442   288.68 MB/sec  execute  58 sec
8    471129   288.71 MB/sec  execute  59 sec
8    477643   288.61 MB/sec  cleanup  60 sec
8    477643   288.60 MB/sec  cleanup  60 sec

patchlet:

sched: disable START_DEBIT.

START_DEBIT induces unfairness to loads which fork/clone frequently when they
must compete against loads which do not.


Signed-off-by: Mike Galbraith <efault@gmx.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
LKML-Reference: <new-submission>

 kernel/sched_features.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/kernel/sched_features.h b/kernel/sched_features.h
index d5059fd..2fc94a0 100644
--- a/kernel/sched_features.h
+++ b/kernel/sched_features.h
@@ -23,7 +23,7 @@ SCHED_FEAT(NORMALIZED_SLEEPER, 0)
  * Place new tasks ahead so that they do not starve already running
  * tasks
  */
-SCHED_FEAT(START_DEBIT, 1)
+SCHED_FEAT(START_DEBIT, 0)
 
 /*
  * Should wakeups try to preempt running tasks.

Re: [patchlet] Re: Epic regression in throughput since v2.6.23

[Mike Galbraith]

Aw poo, forgot to add Peter to CC list before poking xmit.

On Thu, 2009-09-17 at 06:55 +0200, Mike Galbraith wrote:
> On Wed, 2009-09-16 at 23:18 +0000, Serge Belyshev wrote:
> > Ingo Molnar <mingo@elte.hu> writes:
> > 
> > > Ok, i think we've got a handle on that finally - mind checking latest 
> > > -tip?
> > 
> > Kernel build benchmark:
> > http://img11.imageshack.us/img11/4544/makej20090916.png
> > 
> > I have also repeated video encode benchmarks described here:
> > http://article.gmane.org/gmane.linux.kernel/889444
> > 
> > "x264 --preset ultrafast":
> > http://img11.imageshack.us/img11/9020/ultrafast20090916.png
> > 
> > "x264 --preset medium":
> > http://img11.imageshack.us/img11/7729/medium20090916.png
> 
> Pre-ramble..
> Most of the performance differences I've examined in all these CFS vs
> BFS threads boil down to fair scheduler vs unfair scheduler.  If you
> favor hogs, naturally, hogs getting more bandwidth perform better than
> hogs getting their fair share.  That's wonderful for hogs, somewhat less
> than wonderful for their competition.  That fairness is not necessarily
> the best thing for throughput is well known.  If you've got a single
> dissimilar task load running alone, favoring hogs may perform better..
> or not.  What about mixed loads though?  Is the throughput of frequent
> switchers less important than hog throughput?
> 
> Moving right along..
> 
> That x264 thing uncovered an interesting issue within CFS.  That load is
> a frequent clone() customer, and when it has to compete against a not so
> fork/clone happy load, it suffers mightily.  Even when running solo, ie
> only competing against it's own siblings, IFF sleeper fairness is
> enabled, the pain of thread startup latency is quite visible.  With
> concurrent loads, it is agonizingly painful.
> 
> concurrent load test
> tbench 8 vs
> x264 --preset ultrafast --no-scenecut --sync-lookahead 0 --qp 20 -o /dev/null --threads 8 soccer_4cif.y4m
> 
> (i can turn knobs and get whatever numbers i want, including
> outperforming bfs, concurrent or solo.. not the point)
> 
> START_DEBIT
> encoded 600 frames, 44.29 fps, 22096.60 kb/s
> encoded 600 frames, 43.59 fps, 22096.60 kb/s
> encoded 600 frames, 43.78 fps, 22096.60 kb/s
> encoded 600 frames, 43.77 fps, 22096.60 kb/s
> encoded 600 frames, 45.67 fps, 22096.60 kb/s
> 
> 8   1068214   672.35 MB/sec  execute  57 sec
> 8   1083785   672.16 MB/sec  execute  58 sec
> 8   1099188   672.18 MB/sec  execute  59 sec
> 8   1114626   672.00 MB/sec  cleanup  60 sec
> 8   1114626   671.96 MB/sec  cleanup  60 sec
> 
> NO_START_DEBIT
> encoded 600 frames, 123.19 fps, 22096.60 kb/s
> encoded 600 frames, 123.85 fps, 22096.60 kb/s
> encoded 600 frames, 120.05 fps, 22096.60 kb/s
> encoded 600 frames, 123.43 fps, 22096.60 kb/s
> encoded 600 frames, 121.27 fps, 22096.60 kb/s
> 
> 8    848135   533.79 MB/sec  execute  57 sec
> 8    860829   534.08 MB/sec  execute  58 sec
> 8    872840   533.74 MB/sec  execute  59 sec
> 8    885036   533.66 MB/sec  cleanup  60 sec
> 8    885036   533.64 MB/sec  cleanup  60 sec
> 
> 2.6.31-bfs221-smp
> encoded 600 frames, 169.00 fps, 22096.60 kb/s
> encoded 600 frames, 163.85 fps, 22096.60 kb/s
> encoded 600 frames, 161.00 fps, 22096.60 kb/s
> encoded 600 frames, 155.57 fps, 22096.60 kb/s
> encoded 600 frames, 162.01 fps, 22096.60 kb/s
> 
> 8    458328   287.67 MB/sec  execute  57 sec
> 8    464442   288.68 MB/sec  execute  58 sec
> 8    471129   288.71 MB/sec  execute  59 sec
> 8    477643   288.61 MB/sec  cleanup  60 sec
> 8    477643   288.60 MB/sec  cleanup  60 sec
> 
> patchlet:
> 
> sched: disable START_DEBIT.
> 
> START_DEBIT induces unfairness to loads which fork/clone frequently when they
> must compete against loads which do not.
> 
> 
> Signed-off-by: Mike Galbraith <efault@gmx.de>
> Cc: Ingo Molnar <mingo@elte.hu>
> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
> LKML-Reference: <new-submission>
> 
>  kernel/sched_features.h |    2 +-
>  1 files changed, 1 insertions(+), 1 deletions(-)
> 
> diff --git a/kernel/sched_features.h b/kernel/sched_features.h
> index d5059fd..2fc94a0 100644
> --- a/kernel/sched_features.h
> +++ b/kernel/sched_features.h
> @@ -23,7 +23,7 @@ SCHED_FEAT(NORMALIZED_SLEEPER, 0)
>   * Place new tasks ahead so that they do not starve already running
>   * tasks
>   */
> -SCHED_FEAT(START_DEBIT, 1)
> +SCHED_FEAT(START_DEBIT, 0)
>  
>  /*
>   * Should wakeups try to preempt running tasks.
> 

Re: [patchlet] Re: Epic regression in throughput since v2.6.23

[Ingo Molnar]

here's some start-debit versus non-start-debit numbers.

The workload: on a dual-core box start and kill 10 loops, once every 
second. PID 23137 is a shell doing interactive stuff. (running a loop of 
usleep 100000 and echo)

START_DEBIT:

europe:~> perf sched lat | grep 23137
  bash:23137            |     34.380 ms |      187 | avg:    0.005 ms | max:    0.017 ms |
  bash:23137            |     36.410 ms |      188 | avg:    0.005 ms | max:    0.011 ms |
  bash:23137            |     36.680 ms |      183 | avg:    0.007 ms | max:    0.333 ms |

NO_START_DEBIT:

europe:~> perf sched lat | grep 23137
  bash:23137            |     35.531 ms |      183 | avg:    0.005 ms | max:    0.019 ms |
  bash:23137            |     35.511 ms |      188 | avg:    0.007 ms | max:    0.334 ms |
  bash:23137            |     35.774 ms |      185 | avg:    0.005 ms | max:    0.019 ms |

Seems very similar at first sight.

	Ingo

[updated] BFS vs. mainline scheduler benchmarks and measurements

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

>   OLTP performance (postgresql + sysbench)
>      http://redhat.com/~mingo/misc/bfs-vs-tip-oltp.jpg

To everyone who might care about this, i've updated the sysbench 
results to latest -tip:

    http://redhat.com/~mingo/misc/bfs-vs-tip-oltp-v2.jpg

This double checks the effects of the various interactivity fixlets 
in the scheduler tree (whose interactivity effects 
mentioned/documented in the various threads on lkml) in the 
throughput space too and they also improved sysbench performance.

Con, i'd also like to thank you for raising general interest in 
scheduler latencies once more by posting the BFS patch. It gave us 
more bugreports upstream and gave us desktop users willing to test 
patches which in turn helps us improve the code. When users choose 
to suffer in silence that is never helpful.

BFS isnt particularly strong in this graph - from having looked at 
the workload under BFS my impression was that this is primarily due 
to you having cut out much of the sched-domains SMP load-balancer 
code. BFS 'insta-balances' very agressively, which hurts cache 
affine workloads rather visibly.

You might want to have a look at that design detail if you care - 
load-balancing is in significant parts orthogonal to the basic 
design of a fair scheduler.

For example we kept much of the existing load-balancer when we went 
to CFS in v2.6.23 - the fairness engine and the load-balancer are in 
large parts independent units of code and can be improved/tweaked 
separately.

There's interactions, but the concepts are largely separate.

Thanks,

	Ingo

Phoronix CFS vs BFS bencharks

[Nikos Chantziaras]

Phoronix has published some benchmarks, including some "non-synthetic" 
real-life applications:

   http://www.phoronix.com/vr.php?view=14179

The benchmarks are:

  * World of Padman
  * Timed Apache Compilation
  * Timed PHP Compilation
  * 7-Zip Compression
  * GraphicsMagick
  * Apache Benchmark
  * Threaded I/O Tester
  * PostMark

The test was performed on an Ubuntu 9.10 daily snapshot from 2009-09-10 
with the GNOME 2.27.91 desktop, X Server 1.6.3, NVIDIA 190.32 display 
driver, GCC 4.4.1, and an EXT4 file-system.

Re: Phoronix CFS vs BFS bencharks

[Mike Galbraith]

On Mon, 2009-09-14 at 12:46 +0300, Nikos Chantziaras wrote:
> Phoronix has published some benchmarks, including some "non-synthetic" 
> real-life applications:
> 
>    http://www.phoronix.com/vr.php?view=14179
> 
> The benchmarks are:
> 
>   * World of Padman
>   * Timed Apache Compilation
>   * Timed PHP Compilation
>   * 7-Zip Compression
>   * GraphicsMagick
>   * Apache Benchmark
>   * Threaded I/O Tester
>   * PostMark
> 
> The test was performed on an Ubuntu 9.10 daily snapshot from 2009-09-10 
> with the GNOME 2.27.91 desktop, X Server 1.6.3, NVIDIA 190.32 display 
> driver, GCC 4.4.1, and an EXT4 file-system.

Interesting results.

It'd be nice to see what difference the changes since .31 have made to
these comparisons.  In particular, child_runs_first was found to have a
substantial negative impact on parallel compiles, and has been turned
off.  The reduction of sched_latency has a rather large effect on worst
case latency for CPU hogs, so will likely affect some results markedly.

Hohum, back to the grindstone.

	-Mike

Re: Phoronix CFS vs BFS bencharks

[Mike Galbraith]

On Mon, 2009-09-14 at 16:27 +0200, Marcin Letyns wrote:
> Hello,
> 
> Disabling NEW_FAIR_SLEEPERS makes a lot of difference here in the
> Apache benchmark:
> 
> 2.6.30.6-bfs: 7311.05
> 
> 2.6.30.6-cfs-fair_sl_disabled: 8249.17
> 
> 2.6.30.6-cfs-fair_sl_enabled: 4894.99

Wow.

Some loads like wakeup preemption (mysql+oltp), and some hate it.  This
load appears to REALLY hate it (as does volanomark, but that thing is
extremely overloaded).  How many threads does that benchmark run
concurrently?

In any case, it's currently disabled in tip.  Time will tell which
benchmarks gain, and which lose.  With it disabled, anything light loses
when competing with hog(s).  There _are_ one heck of a lot of hogs out
there though, so maybe it _should_ be disabled by default.  Dunno.

	-Mike

Re: Phoronix CFS vs BFS bencharks

[Marcin Letyns]

2009/9/14 Mike Galbraith <efault@gmx.de>
>
> On Mon, 2009-09-14 at 16:27 +0200, Marcin Letyns wrote:
> > Hello,
> >
> > Disabling NEW_FAIR_SLEEPERS makes a lot of difference here in the
> > Apache benchmark:
> >
> > 2.6.30.6-bfs: 7311.05
> >
> > 2.6.30.6-cfs-fair_sl_disabled: 8249.17
> >
> > 2.6.30.6-cfs-fair_sl_enabled: 4894.99
>
> Wow.
>
> Some loads like wakeup preemption (mysql+oltp), and some hate it.  This
> load appears to REALLY hate it (as does volanomark, but that thing is
> extremely overloaded).  How many threads does that benchmark run
> concurrently?

>From the benchmark description:

This is a test of ab, which is the Apache Benchmark program. This test
profile measures how many requests per second a given system can
sustain when carrying out 500,000 requests with 100 requests being
carried out concurrently.

Re: Phoronix CFS vs BFS bencharks

[Willy Tarreau]

On Mon, Sep 14, 2009 at 09:14:35PM +0200, Marcin Letyns wrote:
> 2009/9/14 Mike Galbraith <efault@gmx.de>
> >
> > On Mon, 2009-09-14 at 16:27 +0200, Marcin Letyns wrote:
> > > Hello,
> > >
> > > Disabling NEW_FAIR_SLEEPERS makes a lot of difference here in the
> > > Apache benchmark:
> > >
> > > 2.6.30.6-bfs: 7311.05
> > >
> > > 2.6.30.6-cfs-fair_sl_disabled: 8249.17
> > >
> > > 2.6.30.6-cfs-fair_sl_enabled: 4894.99
> >
> > Wow.
> >
> > Some loads like wakeup preemption (mysql+oltp), and some hate it.  This
> > load appears to REALLY hate it (as does volanomark, but that thing is
> > extremely overloaded).  How many threads does that benchmark run
> > concurrently?
> 
> >From the benchmark description:
> 
> This is a test of ab, which is the Apache Benchmark program. This test
> profile measures how many requests per second a given system can
> sustain when carrying out 500,000 requests with 100 requests being
> carried out concurrently.

Be careful not to run ab on the same machine as you run apache, otherwise
the numerous apache processes can limit ab's throughput. This is the same
reason as why I educate people so that they don't run a single-process
proxy in front of a multi-process/multi-thread web server. Apparently
it's not obvious to everyone.

Regards,
Willy

Re: Phoronix CFS vs BFS bencharks

[Mike Galbraith]

On Mon, 2009-09-14 at 22:49 +0200, Willy Tarreau wrote:
> On Mon, Sep 14, 2009 at 09:14:35PM +0200, Marcin Letyns wrote:
> > 2009/9/14 Mike Galbraith <efault@gmx.de>
> > >
> > > On Mon, 2009-09-14 at 16:27 +0200, Marcin Letyns wrote:
> > > > Hello,
> > > >
> > > > Disabling NEW_FAIR_SLEEPERS makes a lot of difference here in the
> > > > Apache benchmark:
> > > >
> > > > 2.6.30.6-bfs: 7311.05
> > > >
> > > > 2.6.30.6-cfs-fair_sl_disabled: 8249.17
> > > >
> > > > 2.6.30.6-cfs-fair_sl_enabled: 4894.99
> > >
> > > Wow.
> > >
> > > Some loads like wakeup preemption (mysql+oltp), and some hate it.  This
> > > load appears to REALLY hate it (as does volanomark, but that thing is
> > > extremely overloaded).  How many threads does that benchmark run
> > > concurrently?
> > 
> > >From the benchmark description:
> > 
> > This is a test of ab, which is the Apache Benchmark program. This test
> > profile measures how many requests per second a given system can
> > sustain when carrying out 500,000 requests with 100 requests being
> > carried out concurrently.
> 
> Be careful not to run ab on the same machine as you run apache, otherwise
> the numerous apache processes can limit ab's throughput. This is the same
> reason as why I educate people so that they don't run a single-process
> proxy in front of a multi-process/multi-thread web server. Apparently
> it's not obvious to everyone.

I turned on apache, and played with ab a bit, and yup, ab is a hog, so
any fairness hurts it a badly.  Ergo, running ab on the same box as
apache suffers with CFS when NEW_FAIR_SLEEPERS are turned on.  Issuing
ab bandwidth to match it's 1:N pig nature brings throughput right back.

(In all the comparison testing I've done, BFS favors hogs, and with
NEW_FAIR_SLEEPERS turned off, so does CFS, though not as much.)

Running apache on one core and ab on another (with shared cache tho),
something went south with BFS. I would have expected it to be much
closer (shrug).  

Some likely not very uninteresting numbers below.  I wasted a lot more
of my time generating them than anyone will downloading them :)

ab -n 500000 -c 100 http://localhost/openSUSE.org.html

2.6.31-bfs221-smp
Concurrency Level:      100
Time taken for tests:   43.556 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158558404 bytes
HTML transferred:       7027047358 bytes
Requests per second:    11479.50 [#/sec] (mean)
Time per request:       8.711 [ms] (mean)
Time per request:       0.087 [ms] (mean, across all concurrent requests)
Transfer rate:          160501.38 [Kbytes/sec] received

2.6.32-tip-smp NO_NEW_FAIR_SLEEPERS
Concurrency Level:      100
Time taken for tests:   42.834 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158429480 bytes
HTML transferred:       7026921590 bytes
Requests per second:    11672.84 [#/sec] (mean)
Time per request:       8.567 [ms] (mean)
Time per request:       0.086 [ms] (mean, across all concurrent requests)
Transfer rate:          163201.63 [Kbytes/sec] received

2.6.32-tip-smp NEW_FAIR_SLEEPERS
Concurrency Level:      100
Time taken for tests:   68.221 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158357900 bytes
HTML transferred:       7026851325 bytes
Requests per second:    7329.12 [#/sec] (mean)
Time per request:       13.644 [ms] (mean)
Time per request:       0.136 [ms] (mean, across all concurrent requests)
Transfer rate:          102469.65 [Kbytes/sec] received

2.6.32-tip-smp NEW_FAIR_SLEEPERS + ab at nice -15
Concurrency Level:      100
Time taken for tests:   42.824 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158451988 bytes
HTML transferred:       7026943572 bytes
Requests per second:    11675.68 [#/sec] (mean)
Time per request:       8.565 [ms] (mean)
Time per request:       0.086 [ms] (mean, across all concurrent requests)
Transfer rate:          163241.78 [Kbytes/sec] received

taskset -c 2 /etc/init.d/apache2 restart
taskset -c 3 ab -n 500000 -c 100 http://localhost/openSUSE.org.html

2.6.31-bfs221-smp
Concurrency Level:      100
Time taken for tests:   86.590 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158000000 bytes
HTML transferred:       7026500000 bytes
Requests per second:    5774.37 [#/sec] (mean)
Time per request:       17.318 [ms] (mean)
Time per request:       0.173 [ms] (mean, across all concurrent requests)
Transfer rate:          80728.41 [Kbytes/sec] received

2.6.32-tip-smp
Concurrency Level:      100
Time taken for tests:   48.640 seconds
Complete requests:      500000
Failed requests:        0
Write errors:           0
Total transferred:      7158000000 bytes
HTML transferred:       7026500000 bytes
Requests per second:    10279.71 [#/sec] (mean)
Time per request:       9.728 [ms] (mean)
Time per request:       0.097 [ms] (mean, across all concurrent requests)
Transfer rate:          143715.15 [Kbytes/sec] received

Re: BFS vs. mainline scheduler benchmarks and measurements

[Martin Steigerwald]

[-- Attachment #1: Type: Text/Plain, Size: 7736 bytes --]


Uhoh, it seems I configured my kernel's scheduler to produce keyboard 
failure. The many o's in my last mails where not intended.

Let's see whether its better with:

shambhala:/proc/sys/kernel> echo 2000000 > 
/proc/sys/kernel/sched_wakeup_granularity_ns

First time I just lost the keyboard in X for a while in such a way that 
even a Strg-Alt-F1 did not yield any effect. None of what I typed appeared 
anywhere, not in the mail composer window nor in a Konsole terminal or in 
the Kickoff menu search field. Mouse still worked okay. I was able to log 
out via mouse. Then even in KDM the keyboard did not work. Suddenly it 
produced repeating key input events without me typing anything anymore. 
Like the second time with those o's where I pressed on send instead of 
save as draft accidentally.

I did not have any keyboard issues like this recently nor in the last year 
or longer. Let's see whether that raised wakeup_granulaty helps. Desktop 
experience seems still quite fluid, maybe not as fluid as with the settings 
below. But I prefer a working keyboard in order to finish up this mail.

Am Donnerstag 10 September 2009 schrieb Ingo Molnar:
> * Martin Steigerwald <Martin@lichtvoll.de> wrote:
> > Am Mittwoch 09 September 2009 schrieb Peter Zijlstra:
> > > On Wed, 2009-09-09 at 12:05 +0300, Nikos Chantziaras wrote:
> > > > Thank you for mentioning min_granularity.  After:
> > > >
> > > >    echo 10000000 > /proc/sys/kernel/sched_latency_ns
> > > >    echo 2000000 > /proc/sys/kernel/sched_min_granularity_ns
> > >
> > > You might also want to do:
> > >
> > >      echo 2000000 > /proc/sys/kernel/sched_wakeup_granularity_ns
> > >
> > > That affects when a newly woken task will preempt an already
> > > running task.
> >
> > Heh that scheduler thing again... and unfortunately Col appearing
> > to feel hurt while I am think that Ingo is honest on his offer on
> > collaboration...
> >
> > While it makes fun playing with that numbers and indeed
> > experiencing subjectively a more fluid deskopt how about just a
> >
> > echo "This is a f* desktop!" > /proc/sys/kernel/sched_workload
> 
> No need to do that, that's supposed to be the default :-) The knobs
> are really just there to help us make it even more so - i.e. you
> dont need to tune them. But it really relies on people helping us
> out and tell us which combinations work best ...

Well currently I have:

shambhala:/proc/sys/kernel> grep "" sched_latency_ns 
sched_min_granularity_ns sched_wakeup_granularity_ns
sched_latency_ns:100000
sched_min_granularity_ns:200000
sched_wakeup_granularity_ns:0

And this give me *a completely different* desktop experience.

I am using KDE 4.3.1 on a mixture of Debian Squeeze/Sid/Experimental, with 
compositing. And now when I flip desktops or open a window I can *actually 
see* the animation. Before I just saw two to five steps of the animation, 
now its really a lot more fluid. 

perceived _latency--! Well its like opening the eyes again cause I tended 
to take the jerky behavior as normal and possibly related to having KDE 
4.3.1 with compositing enabled on a ThinkPad T42 with 


01:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV350 
[Mobility Radeon 9600 M10] [1002:4e50]
(with OSS Radeon driver)

which I consider to be low end for that workload. But then why actually? 
Next to me is a Sam440ep with PPC 440 667 MHz and and even older Radeon M9 
with AmigaOS 4.1 and some simple transparency effects with compositing. And 
well this combo does feel like it wheel spins cause the hardware is 
actually to fast [2nd keyboard borkage, somewhere before where the first 
where I saved as draft] for that operating system. So actually I knew 
there could be less waiting and less latency. (Granted AmigaOS 4.1 is much 
more minimalistic also in terms on features and no complete memory 
protection and message passing by exchanging pointers and whatnot).

At least to summarize this: With those settings I just keep switching 
desktops and opening windows to enjoy the effects. Desktops flip over 
fluently and windows zoom in on opening fluently either.

All those experiences are with:

shambhala:~> cat /proc/version
Linux version 2.6.31-rc7-tp42-toi-3.0.1-04741-g57e61c0 (martin@shambhala) 
(gcc version 4.3.3 (Debian 4.3.3-10) ) #6 PREEMPT Sun Aug 23 10:51:32 CEST 
2009

(Nigel Cunningham's tuxonice-head git from about 10 days ago)

Keyboard still working. Possibly it really has got broken with zero as 
wakeup granularity.

> > Or to say it in other words: The Linux kernel should not require
> > me to fine-tune three or more values to have the scheduler act in
> > a way that matches my workload.
> >
> > I am willing to test stuff on my work thinkpad and my Amarok
> > thinkpad in order to help improving with that.
> 
> It would be great if you could check latest -tip:
> 
>   http://people.redhat.com/mingo/tip.git/README
> 
> and compare it to vanilla .31?
> 
> Also, could you outline the interactivity problems/complaints you
> have?

Hmmm, would there be a simple possibilty to somehow merge tuxonice git and 
your tip.git into a nice TuxOnIce + scheduler enhanced kernel? I do tend 
not to stick with non TuxOnIce enabled kernels for too long. At least not 
on my work thinkpad and my Amarok thinkpad cause I believe that reboots 
are just for kernel upgrades (with API changes that is) ;-).

Apart from that I lack time to compile a kernel a day at the moment like 
in good old RSDL, SD and CFS testing times ;-). But next kernel, 2.6.31-
not-a-rc, is due and I take suggestions for that one. Preferably I would 
like to have it with TuxOnIce tough.

Problems I faced:

1) Well those effect issues. Jerky at best. Animations which should have 
had 25 frames per second at least, showed 2-5 frames a second. Above 
tuning helped a lot with that. On the other hand DVD playback with Dragon 
Player (and Xine) seems just fine - I thought at least. Maybe I should 
compare watching StarTrek NG with and without scheduler latency fixes. And 
maybe I find some additional frames per duration there too.

2) Some jerks here and there. Difficult to categorize. It sometimes just 
happens that the machine does not follow where I put my attention. Like a 
distracted human who has other things to do than listening to me. This 
could be trying to enter some text in a Qt text input widget. But I need 
to look more carefully as to when, where and why. This is just too fuzzy.

3) Sometimes even typing as a visible latency. Its difficult to spot the 
cause for that stuff. When I type I expect to see each letter as I type it.

4) I/O latencies causing the machine to actually stall for seconds. But 
this one got much better when I switched to 2.6.31 - I had to skip 2.6.30 
cause it didn't tuxonice nicely, even 2.6.31 did not until it reached rc5. 
It seems even way better after switching from XFS to Ext4. But well that 
is a different issue. And at the moment I am quite happy with that.

5) Some window manager operations like resizing windows take very long 
with compositing. But I think this issue may lie elsewhere. Cause these 
did not improve with above settings while many compositing effects did. I 
dunno where that slow window resizing comes from. Whether its a 
compositing / KWin / Qt refresh issue or something scheduler or something 
gfx driver related.

Keyboard is still working, yay! So the X.org keyboard driver might get 
irritated with zero as wakeup granularity.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: BFS vs. mainline scheduler benchmarks and measurements

[Mat]

Martin Steigerwald <Martin <at> lichtvoll.de> writes:

>
> Am Donnerstag 10 September 2009 schrieb Ingo Molnar:

[snip]

> > what is /debug/sched_features - is NO_NEW_FAIR_SLEEPERS set? If not
> > set yet then try it:
> >
> >   echo NO_NEW_FAIR_SLEEPERS > /debug/sched_features
> >
> > that too might make things more fluid.

Hi Martin,

it made an tremendous difference which still has to be tested out :)

Hi Ingo,

which adverse effect could

cat /proc/sys/kernel/sched_wakeup_granularity_ns
0

have on the throughput ?

Concerning that "NO_NEW_FAIR_SLEEPERS" switch - isn't it as easy as to

do the following ? (I'm not sure if there's supposed to be another debug)

echo NO_NEW_FAIR_SLEEPERS > /sys/kernel/debug/sched_features

which after the change says:

cat /sys/kernel/debug/sched_features
NO_NEW_FAIR_SLEEPERS NO_NORMALIZED_SLEEPER ADAPTIVE_GRAN WAKEUP_PREEMPT
START_DEBIT AFFINE_WAKEUPS CACHE_HOT_BUDDY SYNC_WAKEUPS NO_HRTICK NO_DOUBLE_TICK
ASYM_GRAN LB_BIAS LB_WAKEUP_UPDATE ASYM_EFF_LOAD NO_WAKEUP_OVERLAP LAST_BUDDY
OWNER_SPIN

I hope that's the correct switch ^^

Greetings and please keep on improving the scheduler (especially with regards to
the desktop crowd)

Regards

Mat


(Sorry for the "double-post" - this one is including all of the CC
which GMane left out :) )

Re: BFS vs. mainline scheduler benchmarks and measurements

[Volker Armin Hemmann]

[-- Attachment #1: Type: Text/Plain, Size: 56668 bytes --]

Hi,

this is with 2.6.31+reiser4+fglrx
Phenom II X4 955

KDE 4.3.1, composite temporary disabled.
tvtime running.

load: 
fat emerge with make -j5 running in one konsole tab (xulrunner being 
compiled).

without NO_NEW_FAIR_SLEEPERS:

tvtime is smooth most of the time

with NO_NEW_FAIR_SLEEPERS:

tvtime is more jerky. Very visible in scenes with movement.

without background load:

both settings act the same. tvtime is smooth, video is smooth, games are nice. 
No real difference.

config is attached.

Glück Auf,
Volker

dmesg:

[    0.000000] Linux version 2.6.31r4 (root@energy) (gcc version 4.4.1 (Gentoo 
4.4.1 p1.0) ) #1 SMP Thu Sep 10 10:48:07 CEST 2009
[    0.000000] Command line: root=/dev/md1 md=3,/dev/sda3,/dev/sdb3,/dev/sdc3 
nmi_watchdog=0 mtrr_spare_reg_nr=1
[    0.000000] KERNEL supported cpus:
[    0.000000]   AMD AuthenticAMD
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
[    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000c7eb0000 (usable)
[    0.000000]  BIOS-e820: 00000000c7eb0000 - 00000000c7ec0000 (ACPI data)
[    0.000000]  BIOS-e820: 00000000c7ec0000 - 00000000c7ef0000 (ACPI NVS)
[    0.000000]  BIOS-e820: 00000000c7ef0000 - 00000000c7f00000 (reserved)
[    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 0000000238000000 (usable)
[    0.000000] DMI present.
[    0.000000] AMI BIOS detected: BIOS may corrupt low RAM, working around it.
[    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable) 
==> (reserved)
[    0.000000] last_pfn = 0x238000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: uncachable
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-EFFFF uncachable
[    0.000000]   F0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
[    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
[    0.000000]   2 base 0000C0000000 mask FFFFF8000000 write-back
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] TOM2: 0000000238000000 aka 9088M
[    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new 
0x7010600070106
[    0.000000] e820 update range: 00000000c8000000 - 0000000100000000 (usable) 
==> (reserved)
[    0.000000] last_pfn = 0xc7eb0 max_arch_pfn = 0x400000000
[    0.000000] Scanning 0 areas for low memory corruption
[    0.000000] modified physical RAM map:
[    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
[    0.000000]  modified: 0000000000010000 - 000000000009fc00 (usable)
[    0.000000]  modified: 000000000009fc00 - 00000000000a0000 (reserved)
[    0.000000]  modified: 00000000000e6000 - 0000000000100000 (reserved)
[    0.000000]  modified: 0000000000100000 - 00000000c7eb0000 (usable)
[    0.000000]  modified: 00000000c7eb0000 - 00000000c7ec0000 (ACPI data)
[    0.000000]  modified: 00000000c7ec0000 - 00000000c7ef0000 (ACPI NVS)
[    0.000000]  modified: 00000000c7ef0000 - 00000000c7f00000 (reserved)
[    0.000000]  modified: 00000000fff00000 - 0000000100000000 (reserved)
[    0.000000]  modified: 0000000100000000 - 0000000238000000 (usable)
[    0.000000] initial memory mapped : 0 - 20000000
[    0.000000] Using GB pages for direct mapping
[    0.000000] init_memory_mapping: 0000000000000000-00000000c7eb0000
[    0.000000]  0000000000 - 00c0000000 page 1G
[    0.000000]  00c0000000 - 00c7e00000 page 2M
[    0.000000]  00c7e00000 - 00c7eb0000 page 4k
[    0.000000] kernel direct mapping tables up to c7eb0000 @ 10000-13000
[    0.000000] init_memory_mapping: 0000000100000000-0000000238000000
[    0.000000]  0100000000 - 0200000000 page 1G
[    0.000000]  0200000000 - 0238000000 page 2M
[    0.000000] kernel direct mapping tables up to 238000000 @ 12000-14000
[    0.000000] ACPI: RSDP 00000000000fa7c0 00014 (v00 ACPIAM)
[    0.000000] ACPI: RSDT 00000000c7eb0000 00040 (v01 050609 RSDT2000 20090506 
MSFT 00000097)
[    0.000000] ACPI: FACP 00000000c7eb0200 00084 (v02 A M I  OEMFACP  12000601 
MSFT 00000097)
[    0.000000] ACPI: DSDT 00000000c7eb0440 08512 (v01  AS140 AS140121 00000121 
INTL 20051117)
[    0.000000] ACPI: FACS 00000000c7ec0000 00040
[    0.000000] ACPI: APIC 00000000c7eb0390 0006C (v01 050609 APIC2000 20090506 
MSFT 00000097)
[    0.000000] ACPI: MCFG 00000000c7eb0400 0003C (v01 050609 OEMMCFG  20090506 
MSFT 00000097)
[    0.000000] ACPI: OEMB 00000000c7ec0040 00071 (v01 050609 OEMB2000 20090506 
MSFT 00000097)
[    0.000000] ACPI: AAFT 00000000c7eb8960 00027 (v01 050609 OEMAAFT  20090506 
MSFT 00000097)
[    0.000000] ACPI: HPET 00000000c7eb8990 00038 (v01 050609 OEMHPET  20090506 
MSFT 00000097)
[    0.000000] ACPI: SSDT 00000000c7eb89d0 0088C (v01 A M I  POWERNOW 00000001 
AMD  00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] (7 early reservations) ==> bootmem [0000000000 - 0238000000]
[    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 
- 0000001000]
[    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 
- 0000008000]
[    0.000000]   #2 [0001000000 - 00015fb8c0]    TEXT DATA BSS ==> [0001000000 
- 00015fb8c0]
[    0.000000]   #3 [000009fc00 - 0000100000]    BIOS reserved ==> [000009fc00 
- 0000100000]
[    0.000000]   #4 [00015fc000 - 00015fc133]              BRK ==> [00015fc000 
- 00015fc133]
[    0.000000]   #5 [0000010000 - 0000012000]          PGTABLE ==> [0000010000 
- 0000012000]
[    0.000000]   #6 [0000012000 - 0000013000]          PGTABLE ==> [0000012000 
- 0000013000]
[    0.000000]  [ffffea0000000000-ffffea0007dfffff] PMD -> [ffff880028600000-
ffff88002f7fffff] on node 0
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x00238000
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[3] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x0000009f
[    0.000000]     0: 0x00000100 -> 0x000c7eb0
[    0.000000]     0: 0x00100000 -> 0x00238000
[    0.000000] On node 0 totalpages: 2096703
[    0.000000]   DMA zone: 56 pages used for memmap
[    0.000000]   DMA zone: 102 pages reserved
[    0.000000]   DMA zone: 3825 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 14280 pages used for memmap
[    0.000000]   DMA32 zone: 800488 pages, LIFO batch:31
[    0.000000]   Normal zone: 17472 pages used for memmap
[    0.000000]   Normal zone: 1260480 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0x808
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
[    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
[    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8300 base: 0xfed00000
[    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
[    0.000000] nr_irqs_gsi: 24
[    0.000000] PM: Registered nosave memory: 000000000009f000 - 
00000000000a0000
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 
00000000000e6000
[    0.000000] PM: Registered nosave memory: 00000000000e6000 - 
0000000000100000
[    0.000000] PM: Registered nosave memory: 00000000c7eb0000 - 
00000000c7ec0000
[    0.000000] PM: Registered nosave memory: 00000000c7ec0000 - 
00000000c7ef0000
[    0.000000] PM: Registered nosave memory: 00000000c7ef0000 - 
00000000c7f00000
[    0.000000] PM: Registered nosave memory: 00000000c7f00000 - 
00000000fff00000
[    0.000000] PM: Registered nosave memory: 00000000fff00000 - 
0000000100000000
[    0.000000] Allocating PCI resources starting at c7f00000 (gap: 
c7f00000:38000000)
[    0.000000] NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
[    0.000000] PERCPU: Embedded 25 pages at ffff880028034000, static data 72160 
bytes
[    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total 
pages: 2064793
[    0.000000] Kernel command line: root=/dev/md1 
md=3,/dev/sda3,/dev/sdb3,/dev/sdc3 nmi_watchdog=0 mtrr_spare_reg_nr=1
[    0.000000] md: Will configure md3 (super-block) from 
/dev/sda3,/dev/sdb3,/dev/sdc3, below.
[    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
[    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 
bytes)
[    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 
bytes)
[    0.000000] Initializing CPU#0
[    0.000000] Checking aperture...
[    0.000000] No AGP bridge found
[    0.000000] Node 0: aperture @ 2a42000000 size 32 MB
[    0.000000] Aperture beyond 4GB. Ignoring.
[    0.000000] Your BIOS doesn't leave a aperture memory hole
[    0.000000] Please enable the IOMMU option in the BIOS setup
[    0.000000] This costs you 64 MB of RAM
[    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
[    0.000000] PM: Registered nosave memory: 0000000020000000 - 
0000000024000000
[    0.000000] Memory: 8184476k/9306112k available (3500k kernel code, 919300k 
absent, 201380k reserved, 1751k data, 376k init)
[    0.000000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4, 
Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS:4352 nr_irqs:440
[    0.000000] Fast TSC calibration using PIT
[    0.000000] Detected 3200.214 MHz processor.
[    0.000609] Console: colour VGA+ 80x25
[    0.000611] console [tty0] enabled
[    0.003333] hpet clockevent registered
[    0.003333]   alloc irq_desc for 24 on node 0
[    0.003333]   alloc kstat_irqs on node 0
[    0.003333] HPET: 4 timers in total, 1 timers will be used for per-cpu 
timer
[    0.003339] Calibrating delay loop (skipped), value calculated using timer 
frequency.. 6402.10 BogoMIPS (lpj=10667366)
[    0.003421] Mount-cache hash table entries: 256
[    0.003543] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 
bytes/line)
[    0.003578] CPU: L2 Cache: 512K (64 bytes/line)
[    0.003613] tseg: 0000000000
[    0.003618] CPU: Physical Processor ID: 0
[    0.003652] CPU: Processor Core ID: 0
[    0.003686] mce: CPU supports 6 MCE banks
[    0.003725] using C1E aware idle routine
[    0.003768] ACPI: Core revision 20090521
[    0.016704] Setting APIC routing to flat
[    0.017026] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.050860] CPU0: AMD Phenom(tm) II X4 955 Processor stepping 02
[    0.053333] Booting processor 1 APIC 0x1 ip 0x6000
[    0.003333] Initializing CPU#1
[    0.003333] Calibrating delay using timer specific routine.. 6402.85 
BogoMIPS (lpj=10666966)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 
bytes/line)
[    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 1
[    0.003333] mce: CPU supports 6 MCE banks
[    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new 
0x7010600070106
[    0.144161] CPU1: AMD Phenom(tm) II X4 955 Processor stepping 02
[    0.144507] checking TSC synchronization [CPU#0 -> CPU#1]: passed.
[    0.146699] Booting processor 2 APIC 0x2 ip 0x6000
[    0.003333] Initializing CPU#2
[    0.003333] Calibrating delay using timer specific routine.. 6402.85 
BogoMIPS (lpj=10666970)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 
bytes/line)
[    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 2
[    0.003333] mce: CPU supports 6 MCE banks
[    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new 
0x7010600070106
[    0.240822] CPU2: AMD Phenom(tm) II X4 955 Processor stepping 02
[    0.241168] checking TSC synchronization [CPU#0 -> CPU#2]: passed.
[    0.243373] Booting processor 3 APIC 0x3 ip 0x6000
[    0.003333] Initializing CPU#3
[    0.003333] Calibrating delay using timer specific routine.. 6402.85 
BogoMIPS (lpj=10666972)
[    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 
bytes/line)
[    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
[    0.003333] CPU: Physical Processor ID: 0
[    0.003333] CPU: Processor Core ID: 3
[    0.003333] mce: CPU supports 6 MCE banks
[    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new 
0x7010600070106
[    0.337491] CPU3: AMD Phenom(tm) II X4 955 Processor stepping 02
[    0.337836] checking TSC synchronization [CPU#0 -> CPU#3]: passed.
[    0.340006] Brought up 4 CPUs
[    0.340040] Total of 4 processors activated (25611.67 BogoMIPS).
[    0.340109] CPU0 attaching sched-domain:
[    0.340111]  domain 0: span 0-3 level MC
[    0.340112]   groups: 0 1 2 3
[    0.340116] CPU1 attaching sched-domain:
[    0.340117]  domain 0: span 0-3 level MC
[    0.340118]   groups: 1 2 3 0
[    0.340120] CPU2 attaching sched-domain:
[    0.340121]  domain 0: span 0-3 level MC
[    0.340122]   groups: 2 3 0 1
[    0.340125] CPU3 attaching sched-domain:
[    0.340126]  domain 0: span 0-3 level MC
[    0.340127]   groups: 3 0 1 2
[    0.340162] xor: automatically using best checksumming function: 
generic_sse
[    0.356667]    generic_sse: 12835.200 MB/sec
[    0.356700] xor: using function: generic_sse (12835.200 MB/sec)
[    0.356754] Time:  7:07:42  Date: 09/11/09
[    0.356802] NET: Registered protocol family 16
[    0.356851] node 0 link 0: io port [1000, ffffff]
[    0.356851] TOM: 00000000c8000000 aka 3200M
[    0.356851] Fam 10h mmconf [e0000000, efffffff]
[    0.356851] node 0 link 0: mmio [e0000000, efffffff] ==> none
[    0.356851] node 0 link 0: mmio [f0000000, ffffffff]
[    0.356851] node 0 link 0: mmio [a0000, bffff]
[    0.356851] node 0 link 0: mmio [c8000000, dfffffff]
[    0.356851] TOM2: 0000000238000000 aka 9088M
[    0.356851] bus: [00,07] on node 0 link 0
[    0.356851] bus: 00 index 0 io port: [0, ffff]
[    0.356851] bus: 00 index 1 mmio: [f0000000, ffffffff]
[    0.356851] bus: 00 index 2 mmio: [a0000, bffff]
[    0.356851] bus: 00 index 3 mmio: [c8000000, dfffffff]
[    0.356851] bus: 00 index 4 mmio: [238000000, fcffffffff]
[    0.356851] ACPI: bus type pci registered
[    0.356851] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[    0.356851] PCI: Not using MMCONFIG.
[    0.356851] PCI: Using configuration type 1 for base access
[    0.356851] PCI: Using configuration type 1 for extended access
[    0.356851] bio: create slab <bio-0> at 0
[    0.356978] ACPI: EC: Look up EC in DSDT
[    0.367031] ACPI: Interpreter enabled
[    0.367538] ACPI: (supports S0 S1 S3 S4 S5)
[    0.367652] ACPI: Using IOAPIC for interrupt routing
[    0.367725] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
[    0.370531] PCI: MCFG area at e0000000 reserved in ACPI motherboard 
resources
[    0.375947] PCI: Using MMCONFIG at e0000000 - efffffff
[    0.380367] ACPI: No dock devices found.
[    0.380451] ACPI: PCI Root Bridge [PCI0] (0000:00)
[    0.380520] pci 0000:00:00.0: reg 1c 64bit mmio: [0xe0000000-0xffffffff]
[    0.380520] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
[    0.380520] pci 0000:00:02.0: PME# disabled
[    0.380520] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
[    0.380520] pci 0000:00:09.0: PME# disabled
[    0.380520] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
[    0.380520] pci 0000:00:0a.0: PME# disabled
[    0.380520] pci 0000:00:11.0: reg 10 io port: [0xa000-0xa007]
[    0.380520] pci 0000:00:11.0: reg 14 io port: [0x9000-0x9003]
[    0.380520] pci 0000:00:11.0: reg 18 io port: [0x8000-0x8007]
[    0.380520] pci 0000:00:11.0: reg 1c io port: [0x7000-0x7003]
[    0.380520] pci 0000:00:11.0: reg 20 io port: [0x6000-0x600f]
[    0.380520] pci 0000:00:11.0: reg 24 32bit mmio: [0xfddff800-0xfddffbff]
[    0.380520] pci 0000:00:12.0: reg 10 32bit mmio: [0xfddfe000-0xfddfefff]
[    0.380520] pci 0000:00:12.1: reg 10 32bit mmio: [0xfddfd000-0xfddfdfff]
[    0.380576] pci 0000:00:12.2: reg 10 32bit mmio: [0xfddff000-0xfddff0ff]
[    0.380625] pci 0000:00:12.2: supports D1 D2
[    0.380626] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
[    0.380663] pci 0000:00:12.2: PME# disabled
[    0.380724] pci 0000:00:13.0: reg 10 32bit mmio: [0xfddfc000-0xfddfcfff]
[    0.380775] pci 0000:00:13.1: reg 10 32bit mmio: [0xfddf7000-0xfddf7fff]
[    0.380843] pci 0000:00:13.2: reg 10 32bit mmio: [0xfddf6800-0xfddf68ff]
[    0.380893] pci 0000:00:13.2: supports D1 D2
[    0.380894] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
[    0.380930] pci 0000:00:13.2: PME# disabled
[    0.381072] pci 0000:00:14.1: reg 10 io port: [0x00-0x07]
[    0.381078] pci 0000:00:14.1: reg 14 io port: [0x00-0x03]
[    0.381084] pci 0000:00:14.1: reg 18 io port: [0x00-0x07]
[    0.381089] pci 0000:00:14.1: reg 1c io port: [0x00-0x03]
[    0.381095] pci 0000:00:14.1: reg 20 io port: [0xff00-0xff0f]
[    0.381219] pci 0000:00:14.5: reg 10 32bit mmio: [0xfddf5000-0xfddf5fff]
[    0.381353] pci 0000:02:00.0: reg 10 64bit mmio: [0xd0000000-0xdfffffff]
[    0.381360] pci 0000:02:00.0: reg 18 64bit mmio: [0xfdff0000-0xfdffffff]
[    0.381365] pci 0000:02:00.0: reg 20 io port: [0xc000-0xc0ff]
[    0.381372] pci 0000:02:00.0: reg 30 32bit mmio: [0xfdfc0000-0xfdfdffff]
[    0.381387] pci 0000:02:00.0: supports D1 D2
[    0.381415] pci 0000:02:00.1: reg 10 64bit mmio: [0xfdfec000-0xfdfeffff]
[    0.381445] pci 0000:02:00.1: supports D1 D2
[    0.381495] pci 0000:00:02.0: bridge io port: [0xc000-0xcfff]
[    0.381497] pci 0000:00:02.0: bridge 32bit mmio: [0xfdf00000-0xfdffffff]
[    0.381500] pci 0000:00:02.0: bridge 64bit mmio pref: [0xd0000000-0xdfffffff]
[    0.381543] pci 0000:00:09.0: bridge io port: [0xd000-0xdfff]
[    0.381545] pci 0000:00:09.0: bridge 32bit mmio: [0xfe000000-0xfebfffff]
[    0.381548] pci 0000:00:09.0: bridge 64bit mmio pref: [0xfa000000-0xfcefffff]
[    0.383347] pci 0000:01:00.0: reg 10 io port: [0xb800-0xb8ff]
[    0.383360] pci 0000:01:00.0: reg 18 64bit mmio: [0xcffff000-0xcfffffff]
[    0.383370] pci 0000:01:00.0: reg 20 64bit mmio: [0xcffe0000-0xcffeffff]
[    0.383375] pci 0000:01:00.0: reg 30 32bit mmio: [0xfdef0000-0xfdefffff]
[    0.383402] pci 0000:01:00.0: supports D1 D2
[    0.383403] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
[    0.383440] pci 0000:01:00.0: PME# disabled
[    0.383525] pci 0000:00:0a.0: bridge io port: [0xb000-0xbfff]
[    0.383527] pci 0000:00:0a.0: bridge 32bit mmio: [0xfde00000-0xfdefffff]
[    0.383530] pci 0000:00:0a.0: bridge 64bit mmio pref: [0xcff00000-0xcfffffff]
[    0.383565] pci 0000:05:06.0: reg 10 32bit mmio: [0xfcfff000-0xfcffffff]
[    0.383644] pci 0000:05:08.0: reg 10 io port: [0xe800-0xe83f]
[    0.383701] pci 0000:05:08.0: supports D1 D2
[    0.383743] pci 0000:00:14.4: transparent bridge
[    0.383779] pci 0000:00:14.4: bridge io port: [0xe000-0xefff]
[    0.383785] pci 0000:00:14.4: bridge 32bit mmio pref: [0xfcf00000-0xfcffffff]
[    0.383797] pci_bus 0000:00: on NUMA node 0
[    0.383800] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
[    0.383936] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE2._PRT]
[    0.383981] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCEA._PRT]
[    0.384025] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
[    0.384091] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE9._PRT]
[    0.386726] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 *7 10 11 12 14 15)
[    0.386953] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 *11 12 14 15)
[    0.387183] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 7 *10 11 12 14 15)
[    0.387408] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 7 *10 11 12 14 15)
[    0.387643] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 11 12 14 15) *0, 
disabled.
[    0.387914] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 *11 12 14 15)
[    0.388140] ACPI: PCI Interrupt Link [LNKG] (IRQs *4 10 11 12 14 15)
[    0.388352] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 *10 11 12 14 15)
[    0.388538] SCSI subsystem initialized
[    0.388538] libata version 3.00 loaded.
[    0.388538] usbcore: registered new interface driver usbfs
[    0.388538] usbcore: registered new interface driver hub
[    0.388538] usbcore: registered new device driver usb
[    0.443347] raid6: int64x1   2755 MB/s
[    0.500010] raid6: int64x2   3858 MB/s
[    0.556669] raid6: int64x4   2850 MB/s
[    0.613353] raid6: int64x8   2537 MB/s
[    0.670007] raid6: sse2x1    3999 MB/s
[    0.726666] raid6: sse2x2    7012 MB/s
[    0.783343] raid6: sse2x4    7975 MB/s
[    0.783377] raid6: using algorithm sse2x4 (7975 MB/s)
[    0.783423] PCI: Using ACPI for IRQ routing
[    0.783423] pci 0000:00:00.0: BAR 3: address space collision on of device 
[0xe0000000-0xffffffff]
[    0.783425] pci 0000:00:00.0: BAR 3: can't allocate resource
[    0.793433] PCI-DMA: Disabling AGP.
[    0.793518] PCI-DMA: aperture base @ 20000000 size 65536 KB
[    0.793518] PCI-DMA: using GART IOMMU.
[    0.793518] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
[    0.795205] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 24, 0
[    0.795312] hpet0: 4 comparators, 32-bit 14.318180 MHz counter
[    0.800030] hpet: hpet2 irq 24 for MSI
[    0.810015] Switched to high resolution mode on CPU 0
[    0.811193] Switched to high resolution mode on CPU 2
[    0.811196] Switched to high resolution mode on CPU 1
[    0.811200] Switched to high resolution mode on CPU 3
[    0.820035] pnp: PnP ACPI init
[    0.820085] ACPI: bus type pnp registered
[    0.822351] pnp 00:0b: mem resource (0x0-0x9ffff) overlaps 0000:00:00.0 BAR 3 
(0x0-0x1fffffff), disabling
[    0.822406] pnp 00:0b: mem resource (0xc0000-0xcffff) overlaps 0000:00:00.0 
BAR 3 (0x0-0x1fffffff), disabling
[    0.822461] pnp 00:0b: mem resource (0xe0000-0xfffff) overlaps 0000:00:00.0 
BAR 3 (0x0-0x1fffffff), disabling
[    0.822515] pnp 00:0b: mem resource (0x100000-0xc7efffff) overlaps 
0000:00:00.0 BAR 3 (0x0-0x1fffffff), disabling
[    0.822837] pnp: PnP ACPI: found 12 devices
[    0.822871] ACPI: ACPI bus type pnp unregistered
[    0.822911] system 00:06: iomem range 0xfec00000-0xfec00fff could not be 
reserved
[    0.822965] system 00:06: iomem range 0xfee00000-0xfee00fff has been 
reserved
[    0.823002] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
[    0.823037] system 00:07: ioport range 0x40b-0x40b has been reserved
[    0.823072] system 00:07: ioport range 0x4d6-0x4d6 has been reserved
[    0.823106] system 00:07: ioport range 0xc00-0xc01 has been reserved
[    0.823141] system 00:07: ioport range 0xc14-0xc14 has been reserved
[    0.823176] system 00:07: ioport range 0xc50-0xc51 has been reserved
[    0.823211] system 00:07: ioport range 0xc52-0xc52 has been reserved
[    0.823245] system 00:07: ioport range 0xc6c-0xc6c has been reserved
[    0.823280] system 00:07: ioport range 0xc6f-0xc6f has been reserved
[    0.823315] system 00:07: ioport range 0xcd0-0xcd1 has been reserved
[    0.823359] system 00:07: ioport range 0xcd2-0xcd3 has been reserved
[    0.823394] system 00:07: ioport range 0xcd4-0xcd5 has been reserved
[    0.823429] system 00:07: ioport range 0xcd6-0xcd7 has been reserved
[    0.823464] system 00:07: ioport range 0xcd8-0xcdf has been reserved
[    0.823499] system 00:07: ioport range 0x800-0x89f has been reserved
[    0.823533] system 00:07: ioport range 0xb00-0xb0f has been reserved
[    0.823568] system 00:07: ioport range 0xb20-0xb3f has been reserved
[    0.823603] system 00:07: ioport range 0x900-0x90f has been reserved
[    0.823638] system 00:07: ioport range 0x910-0x91f has been reserved
[    0.823673] system 00:07: ioport range 0xfe00-0xfefe has been reserved
[    0.823708] system 00:07: iomem range 0xffb80000-0xffbfffff has been reserved
[    0.823743] system 00:07: iomem range 0xfec10000-0xfec1001f has been 
reserved
[    0.823780] system 00:09: ioport range 0x290-0x29f has been reserved
[    0.823816] system 00:0a: iomem range 0xe0000000-0xefffffff has been reserved
[    0.823852] system 00:0b: iomem range 0xfec00000-0xffffffff could not be 
reserved
[    0.828757] pci 0000:00:02.0: PCI bridge, secondary bus 0000:02
[    0.828792] pci 0000:00:02.0:   IO window: 0xc000-0xcfff
[    0.828828] pci 0000:00:02.0:   MEM window: 0xfdf00000-0xfdffffff
[    0.828863] pci 0000:00:02.0:   PREFETCH window: 
0x000000d0000000-0x000000dfffffff
[    0.828918] pci 0000:00:09.0: PCI bridge, secondary bus 0000:03
[    0.828953] pci 0000:00:09.0:   IO window: 0xd000-0xdfff
[    0.828988] pci 0000:00:09.0:   MEM window: 0xfe000000-0xfebfffff
[    0.829023] pci 0000:00:09.0:   PREFETCH window: 
0x000000fa000000-0x000000fcefffff
[    0.829078] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:01
[    0.829112] pci 0000:00:0a.0:   IO window: 0xb000-0xbfff
[    0.829148] pci 0000:00:0a.0:   MEM window: 0xfde00000-0xfdefffff
[    0.829183] pci 0000:00:0a.0:   PREFETCH window: 
0x000000cff00000-0x000000cfffffff
[    0.829237] pci 0000:00:14.4: PCI bridge, secondary bus 0000:05
[    0.829273] pci 0000:00:14.4:   IO window: 0xe000-0xefff
[    0.829310] pci 0000:00:14.4:   MEM window: disabled
[    0.829346] pci 0000:00:14.4:   PREFETCH window: 0xfcf00000-0xfcffffff
[    0.829387]   alloc irq_desc for 18 on node -1
[    0.829388]   alloc kstat_irqs on node -1
[    0.829392] pci 0000:00:02.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    0.829428] pci 0000:00:02.0: setting latency timer to 64
[    0.829432]   alloc irq_desc for 17 on node -1
[    0.829433]   alloc kstat_irqs on node -1
[    0.829435] pci 0000:00:09.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[    0.829471] pci 0000:00:09.0: setting latency timer to 64
[    0.829474] pci 0000:00:0a.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[    0.829509] pci 0000:00:0a.0: setting latency timer to 64
[    0.829516] pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
[    0.829517] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffffffffffff]
[    0.829519] pci_bus 0000:02: resource 0 io:  [0xc000-0xcfff]
[    0.829521] pci_bus 0000:02: resource 1 mem: [0xfdf00000-0xfdffffff]
[    0.829522] pci_bus 0000:02: resource 2 pref mem [0xd0000000-0xdfffffff]
[    0.829523] pci_bus 0000:03: resource 0 io:  [0xd000-0xdfff]
[    0.829525] pci_bus 0000:03: resource 1 mem: [0xfe000000-0xfebfffff]
[    0.829526] pci_bus 0000:03: resource 2 pref mem [0xfa000000-0xfcefffff]
[    0.829527] pci_bus 0000:01: resource 0 io:  [0xb000-0xbfff]
[    0.829529] pci_bus 0000:01: resource 1 mem: [0xfde00000-0xfdefffff]
[    0.829530] pci_bus 0000:01: resource 2 pref mem [0xcff00000-0xcfffffff]
[    0.829531] pci_bus 0000:05: resource 0 io:  [0xe000-0xefff]
[    0.829533] pci_bus 0000:05: resource 2 pref mem [0xfcf00000-0xfcffffff]
[    0.829534] pci_bus 0000:05: resource 3 io:  [0x00-0xffff]
[    0.829535] pci_bus 0000:05: resource 4 mem: [0x000000-0xffffffffffffffff]
[    0.829547] NET: Registered protocol family 2
[    0.829602] IP route cache hash table entries: 262144 (order: 9, 2097152 
bytes)
[    0.830084] TCP established hash table entries: 262144 (order: 10, 4194304 
bytes)
[    0.831067] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.831468] TCP: Hash tables configured (established 262144 bind 65536)
[    0.831504] TCP reno registered
[    0.831582] NET: Registered protocol family 1
[    0.832803] Scanning for low memory corruption every 60 seconds
[    0.833572] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.833696] Loading Reiser4. See www.namesys.com for a description of 
Reiser4.
[    0.833781] msgmni has been set to 15987
[    0.834082] alg: No test for stdrng (krng)
[    0.834123] async_tx: api initialized (sync-only)
[    0.834227] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 
253)
[    0.834280] io scheduler noop registered
[    0.834315] io scheduler cfq registered (default)
[    0.834448] pci 0000:02:00.0: Boot video device
[    0.834538]   alloc irq_desc for 25 on node -1
[    0.834540]   alloc kstat_irqs on node -1
[    0.834545] pcieport-driver 0000:00:02.0: irq 25 for MSI/MSI-X
[    0.834550] pcieport-driver 0000:00:02.0: setting latency timer to 64
[    0.834642]   alloc irq_desc for 26 on node -1
[    0.834643]   alloc kstat_irqs on node -1
[    0.834646] pcieport-driver 0000:00:09.0: irq 26 for MSI/MSI-X
[    0.834650] pcieport-driver 0000:00:09.0: setting latency timer to 64
[    0.834741]   alloc irq_desc for 27 on node -1
[    0.834742]   alloc kstat_irqs on node -1
[    0.834744] pcieport-driver 0000:00:0a.0: irq 27 for MSI/MSI-X
[    0.834748] pcieport-driver 0000:00:0a.0: setting latency timer to 64
[    0.834953] input: Power Button as 
/devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    0.835007] ACPI: Power Button [PWRF]
[    0.835098] input: Power Button as 
/devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input1
[    0.835152] ACPI: Power Button [PWRB]
[    0.835302] processor LNXCPU:00: registered as cooling_device0
[    0.835337] ACPI: Processor [CPU0] (supports 8 throttling states)
[    0.835434] processor LNXCPU:01: registered as cooling_device1
[    0.835504] processor LNXCPU:02: registered as cooling_device2
[    0.835577] processor LNXCPU:03: registered as cooling_device3
[    0.839315] Linux agpgart interface v0.103
[    0.839511] ahci 0000:00:11.0: version 3.0
[    0.839521]   alloc irq_desc for 22 on node -1
[    0.839522]   alloc kstat_irqs on node -1
[    0.839525] ahci 0000:00:11.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
[    0.839673] ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f 
impl SATA mode
[    0.839727] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio 
slum part 
[    0.840272] scsi0 : ahci
[    0.840403] scsi1 : ahci
[    0.840501] scsi2 : ahci
[    0.840598] scsi3 : ahci
[    0.840697] scsi4 : ahci
[    0.840795] scsi5 : ahci
[    0.840926] ata1: SATA max UDMA/133 irq_stat 0x00400000, PHY RDY changed
[    0.840962] ata2: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddff980 irq 
22
[    0.841016] ata3: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffa00 irq 
22
[    0.841070] ata4: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffa80 irq 
22
[    0.841124] ata5: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffb00 irq 
22
[    0.841178] ata6: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffb80 irq 
22
[    0.841459] PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
[    0.841493] PNP: PS/2 appears to have AUX port disabled, if this is 
incorrect please boot with i8042.nopnp
[    0.841943] serio: i8042 KBD port at 0x60,0x64 irq 1
[    0.842074] mice: PS/2 mouse device common for all mice
[    0.842243] rtc_cmos 00:02: RTC can wake from S4
[    0.842313] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[    0.842369] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
[    0.842428] md: raid1 personality registered for level 1
[    0.842462] md: raid6 personality registered for level 6
[    0.842496] md: raid5 personality registered for level 5
[    0.842530] md: raid4 personality registered for level 4
[    0.843117] cpuidle: using governor ladder
[    0.843151] cpuidle: using governor menu
[    0.843689] usbcore: registered new interface driver hiddev
[    0.843746] usbcore: registered new interface driver usbhid
[    0.843780] usbhid: v2.6:USB HID core driver
[    0.843843] Advanced Linux Sound Architecture Driver Version 1.0.20.
[    0.843878] ALSA device list:
[    0.843911]   No soundcards found.
[    0.843979] TCP cubic registered
[    0.844019] NET: Registered protocol family 10
[    0.844148] IPv6 over IPv4 tunneling driver
[    0.844265] NET: Registered protocol family 17
[    0.844315] powernow-k8: Found 1 AMD Phenom(tm) II X4 955 Processor 
processors (4 cpu cores) (version 2.20.00)
[    0.844391] powernow-k8:    0 : pstate 0 (3200 MHz)
[    0.844425] powernow-k8:    1 : pstate 1 (2500 MHz)
[    0.844459] powernow-k8:    2 : pstate 2 (2100 MHz)
[    0.844492] powernow-k8:    3 : pstate 3 (800 MHz)
[    0.844886] PM: Resume from disk failed.
[    0.844966]   Magic number: 9:648:116
[    0.866018] input: AT Translated Set 2 keyboard as 
/devices/platform/i8042/serio0/input/input2
[    1.160036] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.160097] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.160150] ata6: SATA link down (SStatus 0 SControl 300)
[    1.160205] ata5: SATA link down (SStatus 0 SControl 300)
[    1.160259] ata3: SATA link down (SStatus 0 SControl 300)
[    1.166391] ata4.00: ATA-7: SAMSUNG HD753LJ, 1AA01113, max UDMA7
[    1.166432] ata4.00: 1465149168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    1.166480] ata2.00: ATA-7: SAMSUNG HD502IJ, 1AA01110, max UDMA7
[    1.166514] ata2.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    1.172888] ata4.00: configured for UDMA/133
[    1.172943] ata2.00: configured for UDMA/133
[    1.560035] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[    1.566394] ata1.00: ATA-7: SAMSUNG HD502IJ, 1AA01109, max UDMA7
[    1.566430] ata1.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
[    1.572855] ata1.00: configured for UDMA/133
[    1.583424] scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HD502IJ  1AA0 
PQ: 0 ANSI: 5
[    1.583684] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500 
GB/465 GiB)
[    1.583756] sd 0:0:0:0: [sda] Write Protect is off
[    1.583791] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
[    1.583800] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, 
doesn't support DPO or FUA
[    1.583911]  sda:
[    1.583952] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    1.584094] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD502IJ  1AA0 
PQ: 0 ANSI: 5
[    1.584283] sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500 
GB/465 GiB)
[    1.584354] sd 1:0:0:0: [sdb] Write Protect is off
[    1.584389] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
[    1.584398] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, 
doesn't support DPO or FUA
[    1.584507]  sdb:
[    1.584577] sd 1:0:0:0: Attached scsi generic sg1 type 0
[    1.584709] scsi 3:0:0:0: Direct-Access     ATA      SAMSUNG HD753LJ  1AA0 
PQ: 0 ANSI: 5
[    1.584865] sd 3:0:0:0: [sdc] 1465149168 512-byte logical blocks: (750 
GB/698 GiB)
[    1.584934] sd 3:0:0:0: [sdc] Write Protect is off
[    1.584969] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
[    1.584977] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, 
doesn't support DPO or FUA
[    1.585071]  sdc:
[    1.585123] sd 3:0:0:0: Attached scsi generic sg2 type 0
[    1.588354]  sdc1 sdc2 sdc3 sdc4 < sdb1 sdb2 sdb3 sdb4 < sda1 sda2 sda3 
sda4 < sdb5 sdc5 sda5 sdc6 sdb6 >
[    1.606850] sd 1:0:0:0: [sdb] Attached SCSI disk
[    1.610449]  sda6 >
[    1.610814] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.617957]  sdc7 >
[    1.618341] sd 3:0:0:0: [sdc] Attached SCSI disk
[    1.618382] md: Waiting for all devices to be available before autodetect
[    1.618417] md: If you don't use raid, use raid=noautodetect
[    1.618533] md: Autodetecting RAID arrays.
[    1.724723] md: Scanned 12 and added 12 devices.
[    1.724758] md: autorun ...
[    1.724792] md: considering sdc6 ...
[    1.724827] md:  adding sdc6 ...
[    1.724862] md: sdc5 has different UUID to sdc6
[    1.724897] md: sdc3 has different UUID to sdc6
[    1.724931] md: sdc1 has different UUID to sdc6
[    1.724967] md:  adding sda6 ...
[    1.725001] md: sda5 has different UUID to sdc6
[    1.725036] md: sda3 has different UUID to sdc6
[    1.725070] md: sda1 has different UUID to sdc6
[    1.725106] md:  adding sdb6 ...
[    1.725140] md: sdb5 has different UUID to sdc6
[    1.725175] md: sdb3 has different UUID to sdc6
[    1.725209] md: sdb1 has different UUID to sdc6
[    1.725349] md: created md3
[    1.725382] md: bind<sdb6>
[    1.725419] md: bind<sda6>
[    1.725456] md: bind<sdc6>
[    1.725492] md: running: <sdc6><sda6><sdb6>
[    1.725626] raid5: device sdc6 operational as raid disk 2
[    1.725661] raid5: device sda6 operational as raid disk 0
[    1.725695] raid5: device sdb6 operational as raid disk 1
[    1.725846] raid5: allocated 3220kB for md3
[    1.725910] raid5: raid level 5 set md3 active with 3 out of 3 devices, 
algorithm 2
[    1.725963] RAID5 conf printout:
[    1.725996]  --- rd:3 wd:3
[    1.726029]  disk 0, o:1, dev:sda6
[    1.726062]  disk 1, o:1, dev:sdb6
[    1.726095]  disk 2, o:1, dev:sdc6
[    1.726142] md3: detected capacity change from 0 to 864065421312
[    1.726213] md: considering sdc5 ...
[    1.726249] md:  adding sdc5 ...
[    1.726283] md: sdc3 has different UUID to sdc5
[    1.726318] md: sdc1 has different UUID to sdc5
[    1.726353] md:  adding sda5 ...
[    1.726388] md: sda3 has different UUID to sdc5
[    1.726422] md: sda1 has different UUID to sdc5
[    1.726458] md:  adding sdb5 ...
[    1.726492] md: sdb3 has different UUID to sdc5
[    1.726526] md: sdb1 has different UUID to sdc5
[    1.726630] md: created md2
[    1.726663] md: bind<sdb5>
[    1.726700] md: bind<sda5>
[    1.726738] md: bind<sdc5>
[    1.726774] md: running: <sdc5><sda5><sdb5>
[    1.726901] raid5: device sdc5 operational as raid disk 2
[    1.726935] raid5: device sda5 operational as raid disk 0
[    1.726969] raid5: device sdb5 operational as raid disk 1
[    1.727126] raid5: allocated 3220kB for md2
[    1.727190] raid5: raid level 5 set md2 active with 3 out of 3 devices, 
algorithm 2
[    1.727243] RAID5 conf printout:
[    1.727276]  --- rd:3 wd:3
[    1.727309]  disk 0, o:1, dev:sda5
[    1.727342]  disk 1, o:1, dev:sdb5
[    1.727376]  disk 2, o:1, dev:sdc5
[    1.727420] md2: detected capacity change from 0 to 40007499776
[    1.727490] md: considering sdc3 ...
[    1.727526] md:  adding sdc3 ...
[    1.727560] md: sdc1 has different UUID to sdc3
[    1.727595] md:  adding sda3 ...
[    1.727629] md: sda1 has different UUID to sdc3
[    1.727664] md:  adding sdb3 ...
[    1.727698] md: sdb1 has different UUID to sdc3
[    1.727799] md: created md1
[    1.727832] md: bind<sdb3>
[    1.727869] md: bind<sda3>
[    1.727905] md: bind<sdc3>
[    1.727945] md: running: <sdc3><sda3><sdb3>
[    1.728090] raid5: device sdc3 operational as raid disk 2
[    1.728125] raid5: device sda3 operational as raid disk 0
[    1.728159] raid5: device sdb3 operational as raid disk 1
[    1.728320] raid5: allocated 3220kB for md1
[    1.728370] raid5: raid level 5 set md1 active with 3 out of 3 devices, 
algorithm 2
[    1.728423] RAID5 conf printout:
[    1.728455]  --- rd:3 wd:3
[    1.728488]  disk 0, o:1, dev:sda3
[    1.728522]  disk 1, o:1, dev:sdb3
[    1.728555]  disk 2, o:1, dev:sdc3
[    1.728604] md1: detected capacity change from 0 to 79998877696
[    1.728674] md: considering sdc1 ...
[    1.728710] md:  adding sdc1 ...
[    1.728745] md:  adding sda1 ...
[    1.728779] md:  adding sdb1 ...
[    1.728813] md: created md0
[    1.728846] md: bind<sdb1>
[    1.728882] md: bind<sda1>
[    1.728919] md: bind<sdc1>
[    1.728955] md: running: <sdc1><sda1><sdb1>
[    1.729133] raid1: raid set md0 active with 3 out of 3 mirrors
[    1.729176] md0: detected capacity change from 0 to 65667072
[    1.729232] md: ... autorun DONE.
[    1.729284] md: Loading md3: /dev/sda3
[    1.729322]  md3: unknown partition table
[    1.729481] md: couldn't update array info. -22
[    1.729518] md: could not bd_claim sda3.
[    1.729552] md: md_import_device returned -16
[    1.729588] md: could not bd_claim sdb3.
[    1.729621] md: md_import_device returned -16
[    1.729657] md: could not bd_claim sdc3.
[    1.729690] md: md_import_device returned -16
[    1.729725] md: starting md3 failed
[    1.729800]  md1: unknown partition table
[    1.767199] reiser4: md1: found disk format 4.0.0.
[    5.790318] VFS: Mounted root (reiser4 filesystem) readonly on device 9:1.
[    5.790370] Freeing unused kernel memory: 376k freed
[    9.037775] udev: starting version 145
[    9.217043]  md2:
[    9.217072]  md0: unknown partition table
[    9.282015]  unknown partition table
[   10.420576] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   10.420591] r8169 0000:01:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[   10.420840] r8169 0000:01:00.0: setting latency timer to 64
[   10.420871]   alloc irq_desc for 28 on node -1
[   10.420872]   alloc kstat_irqs on node -1
[   10.420882] r8169 0000:01:00.0: irq 28 for MSI/MSI-X
[   10.420988] eth0: RTL8168c/8111c at 0xffffc90012f18000, 00:19:66:86:ce:12, 
XID 3c4000c0 IRQ 28
[   10.440462] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[   10.440465] ehci_hcd: block sizes: qh 192 qtd 96 itd 192 sitd 96
[   10.440491] ehci_hcd 0000:00:12.2: PCI INT B -> GSI 17 (level, low) -> IRQ 
17
[   10.440518] ehci_hcd 0000:00:12.2: EHCI Host Controller
[   10.440532] drivers/usb/core/inode.c: creating file 'devices'
[   10.440534] drivers/usb/core/inode.c: creating file '001'
[   10.440566] ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus 
number 1
[   10.440572] ehci_hcd 0000:00:12.2: reset hcs_params 0x102306 dbg=1 cc=2 
pcc=3 ordered !ppc ports=6
[   10.440575] ehci_hcd 0000:00:12.2: reset hcc_params a072 thresh 7 uframes 
256/512/1024
[   10.440596] ehci_hcd 0000:00:12.2: applying AMD SB600/SB700 USB freeze 
workaround
[   10.440602] ehci_hcd 0000:00:12.2: reset command 080002 (park)=0 ithresh=8 
period=1024 Reset HALT
[   10.440615] ehci_hcd 0000:00:12.2: debug port 1
[   10.440619] ehci_hcd 0000:00:12.2: MWI active
[   10.440620] ehci_hcd 0000:00:12.2: supports USB remote wakeup
[   10.440633] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfddff000
[   10.440637] ehci_hcd 0000:00:12.2: reset command 080002 (park)=0 ithresh=8 
period=1024 Reset HALT
[   10.440642] ehci_hcd 0000:00:12.2: init command 010009 (park)=0 ithresh=1 
period=256 RUN
[   10.447931] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
[   10.447961] usb usb1: default language 0x0409
[   10.447965] usb usb1: udev 1, busnum 1, minor = 0
[   10.447967] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   10.447968] usb usb1: New USB device strings: Mfr=3, Product=2, 
SerialNumber=1
[   10.447969] usb usb1: Product: EHCI Host Controller
[   10.447970] usb usb1: Manufacturer: Linux 2.6.31r4 ehci_hcd
[   10.447971] usb usb1: SerialNumber: 0000:00:12.2
[   10.447998] usb usb1: uevent
[   10.448007] usb usb1: usb_probe_device
[   10.448009] usb usb1: configuration #1 chosen from 1 choice
[   10.448014] usb usb1: adding 1-0:1.0 (config #1, interface 0)
[   10.448021] usb 1-0:1.0: uevent
[   10.448028] hub 1-0:1.0: usb_probe_interface
[   10.448029] hub 1-0:1.0: usb_probe_interface - got id
[   10.448031] hub 1-0:1.0: USB hub found
[   10.448035] hub 1-0:1.0: 6 ports detected
[   10.448036] hub 1-0:1.0: standalone hub
[   10.448037] hub 1-0:1.0: no power switching (usb 1.0)
[   10.448038] hub 1-0:1.0: individual port over-current protection
[   10.448039] hub 1-0:1.0: power on to power good time: 20ms
[   10.448042] hub 1-0:1.0: local power source is good
[   10.448043] hub 1-0:1.0: trying to enable port power on non-switchable hub
[   10.448067] drivers/usb/core/inode.c: creating file '001'
[   10.448085]   alloc irq_desc for 19 on node -1
[   10.448087]   alloc kstat_irqs on node -1
[   10.448091] ehci_hcd 0000:00:13.2: PCI INT B -> GSI 19 (level, low) -> IRQ 
19
[   10.448101] ehci_hcd 0000:00:13.2: EHCI Host Controller
[   10.448105] drivers/usb/core/inode.c: creating file '002'
[   10.448122] ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus 
number 2
[   10.448127] ehci_hcd 0000:00:13.2: reset hcs_params 0x102306 dbg=1 cc=2 
pcc=3 ordered !ppc ports=6
[   10.448130] ehci_hcd 0000:00:13.2: reset hcc_params a072 thresh 7 uframes 
256/512/1024
[   10.448143] ehci_hcd 0000:00:13.2: applying AMD SB600/SB700 USB freeze 
workaround
[   10.448148] ehci_hcd 0000:00:13.2: reset command 080002 (park)=0 ithresh=8 
period=1024 Reset HALT
[   10.448161] ehci_hcd 0000:00:13.2: debug port 1
[   10.448164] ehci_hcd 0000:00:13.2: MWI active
[   10.448165] ehci_hcd 0000:00:13.2: supports USB remote wakeup
[   10.448173] ehci_hcd 0000:00:13.2: irq 19, io mem 0xfddf6800
[   10.448176] ehci_hcd 0000:00:13.2: reset command 080002 (park)=0 ithresh=8 
period=1024 Reset HALT
[   10.448181] ehci_hcd 0000:00:13.2: init command 010009 (park)=0 ithresh=1 
period=256 RUN
[   10.457930] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
[   10.457945] usb usb2: default language 0x0409
[   10.457949] usb usb2: udev 1, busnum 2, minor = 128
[   10.457950] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
[   10.457952] usb usb2: New USB device strings: Mfr=3, Product=2, 
SerialNumber=1
[   10.457953] usb usb2: Product: EHCI Host Controller
[   10.457954] usb usb2: Manufacturer: Linux 2.6.31r4 ehci_hcd
[   10.457955] usb usb2: SerialNumber: 0000:00:13.2
[   10.457977] usb usb2: uevent
[   10.457984] usb usb2: usb_probe_device
[   10.457986] usb usb2: configuration #1 chosen from 1 choice
[   10.457989] usb usb2: adding 2-0:1.0 (config #1, interface 0)
[   10.457997] usb 2-0:1.0: uevent
[   10.458003] hub 2-0:1.0: usb_probe_interface
[   10.458004] hub 2-0:1.0: usb_probe_interface - got id
[   10.458005] hub 2-0:1.0: USB hub found
[   10.458009] hub 2-0:1.0: 6 ports detected
[   10.458010] hub 2-0:1.0: standalone hub
[   10.458010] hub 2-0:1.0: no power switching (usb 1.0)
[   10.458011] hub 2-0:1.0: individual port over-current protection
[   10.458013] hub 2-0:1.0: power on to power good time: 20ms
[   10.458015] hub 2-0:1.0: local power source is good
[   10.458016] hub 2-0:1.0: trying to enable port power on non-switchable hub
[   10.458038] drivers/usb/core/inode.c: creating file '001'
[   10.474718] Linux video capture interface: v2.00
[   10.489875] bttv: driver version 0.9.18 loaded
[   10.489877] bttv: using 8 buffers with 2080k (520 pages) each for capture
[   10.489914] bttv: Bt8xx card found (0).
[   10.489923]   alloc irq_desc for 21 on node -1
[   10.489924]   alloc kstat_irqs on node -1
[   10.489928] bttv 0000:05:06.0: PCI INT A -> GSI 21 (level, low) -> IRQ 21
[   10.489938] bttv0: Bt848 (rev 18) at 0000:05:06.0, irq: 21, latency: 128, 
mmio: 0xfcfff000
[   10.489971] bttv0: using: Terratec TerraTV+ Version 1.0 (Bt848)/ Terra 
TValue Version 1.0/ Vobis TV-Boostar [card=25,insmod option]
[   10.489974] IRQ 21/bttv0: IRQF_DISABLED is not guaranteed on shared IRQs
[   10.490007] bttv0: gpio: en=00000000, out=00000000 in=00ffffff [init]
[   10.547936] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001803 POWER 
sig=j CSC CONNECT
[   10.547939] hub 1-0:1.0: port 2: status 0501 change 0001
[   10.557954] hub 2-0:1.0: state 7 ports 6 chg 0000 evt 0000
[   10.647931] hub 1-0:1.0: state 7 ports 6 chg 0004 evt 0000
[   10.647940] hub 1-0:1.0: port 2, status 0501, change 0000, 480 Mb/s
[   10.700045] ehci_hcd 0000:00:12.2: port 2 high speed
[   10.700049] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001005 POWER 
sig=se0 PE CONNECT
[   10.757104] usb 1-2: new high speed USB device using ehci_hcd and address 2
[   10.810036] ehci_hcd 0000:00:12.2: port 2 high speed
[   10.810039] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001005 POWER 
sig=se0 PE CONNECT
[   10.881474] usb 1-2: default language 0x0409
[   10.881724] usb 1-2: udev 2, busnum 1, minor = 1
[   10.881725] usb 1-2: New USB device found, idVendor=05e3, idProduct=0608
[   10.881726] usb 1-2: New USB device strings: Mfr=0, Product=1, 
SerialNumber=0
[   10.881728] usb 1-2: Product: USB2.0 Hub
[   10.881764] usb 1-2: uevent
[   10.881773] usb 1-2: usb_probe_device
[   10.881775] usb 1-2: configuration #1 chosen from 1 choice
[   10.882188] usb 1-2: adding 1-2:1.0 (config #1, interface 0)
[   10.882199] usb 1-2:1.0: uevent
[   10.882206] hub 1-2:1.0: usb_probe_interface
[   10.882207] hub 1-2:1.0: usb_probe_interface - got id
[   10.882209] hub 1-2:1.0: USB hub found
[   10.882473] hub 1-2:1.0: 4 ports detected
[   10.882474] hub 1-2:1.0: standalone hub
[   10.882476] hub 1-2:1.0: individual port power switching
[   10.882477] hub 1-2:1.0: individual port over-current protection
[   10.882478] hub 1-2:1.0: Single TT
[   10.882479] hub 1-2:1.0: TT requires at most 32 FS bit times (2664 ns)
[   10.882480] hub 1-2:1.0: Port indicators are supported
[   10.882481] hub 1-2:1.0: power on to power good time: 100ms
[   10.882848] hub 1-2:1.0: local power source is good
[   10.882849] hub 1-2:1.0: enabling power on all ports
[   10.883860] drivers/usb/core/inode.c: creating file '002'
[   10.983961] hub 1-2:1.0: port 2: status 0301 change 0001
[   11.083356] usb 1-2: link qh256-0001/ffff8800c7800180 start 1 [1/0 us]
[   11.083364] hub 1-2:1.0: state 7 ports 4 chg 0004 evt 0000
[   11.083700] hub 1-2:1.0: port 2, status 0301, change 0000, 1.5 Mb/s
[   11.152198] usb 1-2.2: new low speed USB device using ehci_hcd and address 
3
[   11.241187] usb 1-2.2: skipped 1 descriptor after interface
[   11.241189] usb 1-2.2: skipped 1 descriptor after interface
[   11.241685] usb 1-2.2: default language 0x0409
[   11.243941] usb 1-2.2: udev 3, busnum 1, minor = 2
[   11.243943] usb 1-2.2: New USB device found, idVendor=046d, idProduct=c518
[   11.243944] usb 1-2.2: New USB device strings: Mfr=1, Product=2, 
SerialNumber=0
[   11.243945] usb 1-2.2: Product: USB Receiver
[   11.243947] usb 1-2.2: Manufacturer: Logitech
[   11.243972] usb 1-2.2: uevent
[   11.243980] usb 1-2.2: usb_probe_device
[   11.243981] usb 1-2.2: configuration #1 chosen from 1 choice
[   11.251309] usb 1-2.2: adding 1-2.2:1.0 (config #1, interface 0)
[   11.251324] usb 1-2.2:1.0: uevent
[   11.251334] usbhid 1-2.2:1.0: usb_probe_interface
[   11.251335] usbhid 1-2.2:1.0: usb_probe_interface - got id
[   11.251796] usb 1-2: clear tt buffer port 2, a3 ep0 t80008d42
[   11.254692] input: Logitech USB Receiver as 
/devices/pci0000:00/0000:00:12.2/usb1/1-2/1-2.2/1-2.2:1.0/input/input3
[   11.254732] generic-usb 0003:046D:C518.0001: input,hidraw0: USB HID v1.11 
Mouse [Logitech USB Receiver] on usb-0000:00:12.2-2.2/input0
[   11.254740] usb 1-2.2: adding 1-2.2:1.1 (config #1, interface 1)
[   11.254749] usb 1-2.2:1.1: uevent
[   11.254755] usbhid 1-2.2:1.1: usb_probe_interface
[   11.254757] usbhid 1-2.2:1.1: usb_probe_interface - got id
[   11.255046] usb 1-2: clear tt buffer port 2, a3 ep0 t80008d42
[   11.260359] input: Logitech USB Receiver as 
/devices/pci0000:00/0000:00:12.2/usb1/1-2/1-2.2/1-2.2:1.1/input/input4
[   11.260368] usb 1-2.2: link qh8-0601/ffff8800c7800300 start 2 [1/2 us]
[   11.260384] drivers/usb/core/file.c: looking for a minor, starting at 96
[   11.260414] generic-usb 0003:046D:C518.0002: input,hiddev96,hidraw1: USB 
HID v1.11 Device [Logitech USB Receiver] on usb-0000:00:12.2-2.2/input1
[   11.260427] drivers/usb/core/inode.c: creating file '003'
[   11.260438] hub 1-2:1.0: state 7 ports 4 chg 0000 evt 0004
[   11.280770] usb 1-2.2:1.0: uevent
[   11.280827] usb 1-2.2: uevent
[   11.280844] usb 1-2.2:1.0: uevent
[   11.280903] usb 1-2.2: uevent
[   11.281405] usb 1-2.2:1.1: uevent
[   11.281467] usb 1-2.2: uevent
[   11.281517] usb 1-2.2:1.0: uevent
[   11.281529] usb 1-2.2:1.0: uevent
[   11.282164] usb 1-2.2:1.1: uevent
[   11.493831] bttv0: tea5757: read timeout
[   11.493832] bttv0: tuner type=5
[   11.505445] bttv0: audio absent, no audio device found!
[   11.525030] TUNER: Unable to find symbol tea5767_autodetection()
[   11.525033] tuner 0-0060: chip found @ 0xc0 (bt848 #0 [sw])
[   11.527843] tuner-simple 0-0060: creating new instance
[   11.527846] tuner-simple 0-0060: type set to 5 (Philips PAL_BG (FI1216 and 
compatibles))
[   11.528664] bttv0: registered device video0
[   11.528695] bttv0: registered device vbi0
[   11.814508] reiser4: md2: found disk format 4.0.0.
[   13.337101] hub 2-0:1.0: hub_suspend
[   13.337107] usb usb2: bus auto-suspend
[   13.337109] ehci_hcd 0000:00:13.2: suspend root hub
[   13.591267] reiser4: md3: found disk format 4.0.0.
[   55.020583] reiser4: sdc7: found disk format 4.0.0.
[   67.386552] Adding 7815612k swap on /dev/sda2.  Priority:1 extents:1 
across:7815612k 
[   67.408915] Adding 7815612k swap on /dev/sdb2.  Priority:1 extents:1 
across:7815612k 
[   67.501498] Adding 7815612k swap on /dev/sdc2.  Priority:1 extents:1 
across:7815612k 
[   68.205087] w83627ehf: Found W83627EHG chip at 0x290
[   68.421646] r8169: eth0: link up
[   68.421650] r8169: eth0: link up
[   70.281241] usb usb1: uevent
[   70.281269] usb 1-0:1.0: uevent
[   70.281293] usb 1-2: uevent
[   70.281320] usb 1-2.2: uevent
[   70.281346] usb 1-2.2:1.0: uevent
[   70.281533] usb 1-2.2:1.1: uevent
[   70.281706] usb 1-2:1.0: uevent
[   70.281804] usb usb2: uevent
[   70.281830] usb 2-0:1.0: uevent
[   71.272886]   alloc irq_desc for 23 on node -1
[   71.272889]   alloc kstat_irqs on node -1
[   71.272895] EMU10K1_Audigy 0000:05:08.0: PCI INT A -> GSI 23 (level, low) -
> IRQ 23
[   71.278891] Audigy2 value: Special config.
[   72.473408] fglrx: module license 'Proprietary. (C) 2002 - ATI 
Technologies, Starnberg, GERMANY' taints kernel.
[   72.473417] Disabling lock debugging due to kernel taint
[   72.490481] [fglrx] Maximum main memory to use for locked dma buffers: 7760 
MBytes.
[   72.490559] [fglrx]   vendor: 1002 device: 9501 count: 1
[   72.490739] [fglrx] ioport: bar 4, base 0xc000, size: 0x100
[   72.490750] pci 0000:02:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
[   72.490754] pci 0000:02:00.0: setting latency timer to 64
[   72.490884] [fglrx] Kernel PAT support is enabled
[   72.490901] [fglrx] module loaded - fglrx 8.66.2 [Sep  1 2009] with 1 
minors
[   72.668413]   alloc irq_desc for 29 on node -1
[   72.668416]   alloc kstat_irqs on node -1
[   72.668424] fglrx_pci 0000:02:00.0: irq 29 for MSI/MSI-X
[   72.668759] [fglrx] Firegl kernel thread PID: 3800
[   74.787429] [fglrx] Gart USWC size:1279 M.
[   74.787431] [fglrx] Gart cacheable size:508 M.
[   74.787435] [fglrx] Reserved FB block: Shared offset:0, size:1000000 
[   74.787437] [fglrx] Reserved FB block: Unshared offset:fbff000, size:401000 
[   74.787438] [fglrx] Reserved FB block: Unshared offset:1fffc000, size:4000 
[   75.947153] usb 1-2.2: link qh8-0601/ffff8800c78003c0 start 3 [1/2 us]
[  616.849440] reiser4[ktxnmgrd:md1:ru(581)]: disable_write_barrier 
(fs/reiser4/wander.c:235)[zam-1055]:
[  616.849445] NOTICE: md1 does not support write barriers, using synchronous 
write instead.
[  671.813536] reiser4[ktxnmgrd:md2:ru(2774)]: disable_write_barrier 
(fs/reiser4/wander.c:235)[zam-1055]:
[  671.813541] NOTICE: md2 does not support write barriers, using synchronous 
write instead.
[  703.842289] reiser4[ktxnmgrd:md3:ru(2776)]: disable_write_barrier 
(fs/reiser4/wander.c:235)[zam-1055]:
[  703.842293] NOTICE: md3 does not support write barriers, using synchronous 
write instead.

ps. I have to disable c1e in bios, or video is very jerky - unless something 
cpu heavy runs on at least one core...




[-- Attachment #2: .config --]
[-- Type: text/plain, Size: 55251 bytes --]

#
# Automatically generated make config: don't edit
# Linux kernel version: 2.6.31
# Thu Sep 10 10:45:50 2009
#
CONFIG_64BIT=y
# CONFIG_X86_32 is not set
CONFIG_X86_64=y
CONFIG_X86=y
CONFIG_OUTPUT_FORMAT="elf64-x86-64"
CONFIG_ARCH_DEFCONFIG="arch/x86/configs/x86_64_defconfig"
CONFIG_GENERIC_TIME=y
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
CONFIG_FAST_CMPXCHG_LOCAL=y
CONFIG_MMU=y
CONFIG_ZONE_DMA=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_GENERIC_BUG=y
CONFIG_GENERIC_BUG_RELATIVE_POINTERS=y
CONFIG_GENERIC_HWEIGHT=y
CONFIG_ARCH_MAY_HAVE_PC_FDC=y
CONFIG_RWSEM_GENERIC_SPINLOCK=y
# CONFIG_RWSEM_XCHGADD_ALGORITHM is not set
CONFIG_ARCH_HAS_CPU_IDLE_WAIT=y
CONFIG_GENERIC_CALIBRATE_DELAY=y
CONFIG_GENERIC_TIME_VSYSCALL=y
CONFIG_ARCH_HAS_CPU_RELAX=y
CONFIG_ARCH_HAS_DEFAULT_IDLE=y
CONFIG_ARCH_HAS_CACHE_LINE_SIZE=y
CONFIG_HAVE_SETUP_PER_CPU_AREA=y
CONFIG_HAVE_DYNAMIC_PER_CPU_AREA=y
CONFIG_HAVE_CPUMASK_OF_CPU_MAP=y
CONFIG_ARCH_HIBERNATION_POSSIBLE=y
CONFIG_ARCH_SUSPEND_POSSIBLE=y
CONFIG_ZONE_DMA32=y
CONFIG_ARCH_POPULATES_NODE_MAP=y
CONFIG_AUDIT_ARCH=y
CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING=y
CONFIG_ARCH_SUPPORTS_DEBUG_PAGEALLOC=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_GENERIC_PENDING_IRQ=y
CONFIG_USE_GENERIC_SMP_HELPERS=y
CONFIG_X86_64_SMP=y
CONFIG_X86_HT=y
CONFIG_X86_TRAMPOLINE=y
# CONFIG_KTIME_SCALAR is not set
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
CONFIG_CONSTRUCTORS=y

#
# General setup
#
CONFIG_EXPERIMENTAL=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION="r4"
# CONFIG_LOCALVERSION_AUTO is not set
CONFIG_HAVE_KERNEL_GZIP=y
CONFIG_HAVE_KERNEL_BZIP2=y
CONFIG_HAVE_KERNEL_LZMA=y
CONFIG_KERNEL_GZIP=y
# CONFIG_KERNEL_BZIP2 is not set
# CONFIG_KERNEL_LZMA is not set
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_SYSVIPC_SYSCTL=y
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
# CONFIG_BSD_PROCESS_ACCT is not set
# CONFIG_TASKSTATS is not set
# CONFIG_AUDIT is not set

#
# RCU Subsystem
#
# CONFIG_CLASSIC_RCU is not set
CONFIG_TREE_RCU=y
# CONFIG_PREEMPT_RCU is not set
# CONFIG_RCU_TRACE is not set
CONFIG_RCU_FANOUT=64
# CONFIG_RCU_FANOUT_EXACT is not set
# CONFIG_TREE_RCU_TRACE is not set
# CONFIG_PREEMPT_RCU_TRACE is not set
# CONFIG_IKCONFIG is not set
CONFIG_LOG_BUF_SHIFT=18
CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
# CONFIG_GROUP_SCHED is not set
# CONFIG_CGROUPS is not set
# CONFIG_SYSFS_DEPRECATED_V2 is not set
CONFIG_RELAY=y
# CONFIG_NAMESPACES is not set
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
CONFIG_SYSCTL=y
CONFIG_ANON_INODES=y
CONFIG_EMBEDDED=y
# CONFIG_UID16 is not set
# CONFIG_SYSCTL_SYSCALL is not set
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_HOTPLUG=y
CONFIG_PRINTK=y
CONFIG_BUG=y
CONFIG_ELF_CORE=y
CONFIG_PCSPKR_PLATFORM=y
CONFIG_BASE_FULL=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SIGNALFD=y
CONFIG_TIMERFD=y
CONFIG_EVENTFD=y
CONFIG_SHMEM=y
CONFIG_AIO=y
CONFIG_HAVE_PERF_COUNTERS=y

#
# Performance Counters
#
# CONFIG_PERF_COUNTERS is not set
CONFIG_VM_EVENT_COUNTERS=y
CONFIG_PCI_QUIRKS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_STRIP_ASM_SYMS is not set
CONFIG_COMPAT_BRK=y
# CONFIG_SLAB is not set
CONFIG_SLUB=y
# CONFIG_SLOB is not set
# CONFIG_PROFILING is not set
# CONFIG_MARKERS is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
CONFIG_HAVE_IOREMAP_PROT=y
CONFIG_HAVE_KPROBES=y
CONFIG_HAVE_KRETPROBES=y
CONFIG_HAVE_ARCH_TRACEHOOK=y
CONFIG_HAVE_DMA_ATTRS=y
CONFIG_HAVE_DMA_API_DEBUG=y

#
# GCOV-based kernel profiling
#
# CONFIG_GCOV_KERNEL is not set
# CONFIG_SLOW_WORK is not set
# CONFIG_HAVE_GENERIC_DMA_COHERENT is not set
CONFIG_RT_MUTEXES=y
CONFIG_BASE_SMALL=0
CONFIG_MODULES=y
# CONFIG_MODULE_FORCE_LOAD is not set
CONFIG_MODULE_UNLOAD=y
CONFIG_MODULE_FORCE_UNLOAD=y
# CONFIG_MODVERSIONS is not set
# CONFIG_MODULE_SRCVERSION_ALL is not set
CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_BLK_DEV_BSG=y
# CONFIG_BLK_DEV_INTEGRITY is not set
CONFIG_BLOCK_COMPAT=y

#
# IO Schedulers
#
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=m
CONFIG_IOSCHED_DEADLINE=m
CONFIG_IOSCHED_CFQ=y
# CONFIG_DEFAULT_AS is not set
# CONFIG_DEFAULT_DEADLINE is not set
CONFIG_DEFAULT_CFQ=y
# CONFIG_DEFAULT_NOOP is not set
CONFIG_DEFAULT_IOSCHED="cfq"
CONFIG_PREEMPT_NOTIFIERS=y
CONFIG_FREEZER=y

#
# Processor type and features
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
CONFIG_SMP=y
CONFIG_SPARSE_IRQ=y
# CONFIG_X86_MPPARSE is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
# CONFIG_SCHED_OMIT_FRAME_POINTER is not set
# CONFIG_PARAVIRT_GUEST is not set
# CONFIG_MEMTEST is not set
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
# CONFIG_MPENTIUMM is not set
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
CONFIG_MK8=y
# CONFIG_MCRUSOE is not set
# CONFIG_MEFFICEON is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MGEODEGX1 is not set
# CONFIG_MGEODE_LX is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
# CONFIG_MCORE2 is not set
# CONFIG_GENERIC_CPU is not set
CONFIG_X86_CPU=y
CONFIG_X86_L1_CACHE_BYTES=64
CONFIG_X86_INTERNODE_CACHE_BYTES=64
CONFIG_X86_CMPXCHG=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_TSC=y
CONFIG_X86_CMPXCHG64=y
CONFIG_X86_CMOV=y
CONFIG_X86_MINIMUM_CPU_FAMILY=64
CONFIG_X86_DEBUGCTLMSR=y
CONFIG_PROCESSOR_SELECT=y
# CONFIG_CPU_SUP_INTEL is not set
CONFIG_CPU_SUP_AMD=y
# CONFIG_CPU_SUP_CENTAUR is not set
# CONFIG_X86_DS is not set
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_DMI=y
CONFIG_GART_IOMMU=y
# CONFIG_CALGARY_IOMMU is not set
CONFIG_AMD_IOMMU=y
CONFIG_AMD_IOMMU_STATS=y
CONFIG_SWIOTLB=y
CONFIG_IOMMU_HELPER=y
CONFIG_IOMMU_API=y
CONFIG_NR_CPUS=4
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS=y
CONFIG_X86_MCE=y
CONFIG_X86_NEW_MCE=y
CONFIG_X86_MCE_INTEL=y
CONFIG_X86_MCE_AMD=y
CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
CONFIG_X86_THERMAL_VECTOR=y
# CONFIG_I8K is not set
# CONFIG_MICROCODE is not set
CONFIG_X86_MSR=y
CONFIG_X86_CPUID=y
CONFIG_X86_CPU_DEBUG=m
CONFIG_ARCH_PHYS_ADDR_T_64BIT=y
CONFIG_DIRECT_GBPAGES=y
# CONFIG_NUMA is not set
CONFIG_ARCH_SPARSEMEM_DEFAULT=y
CONFIG_ARCH_SPARSEMEM_ENABLE=y
CONFIG_ARCH_SELECT_MEMORY_MODEL=y
CONFIG_SELECT_MEMORY_MODEL=y
# CONFIG_FLATMEM_MANUAL is not set
# CONFIG_DISCONTIGMEM_MANUAL is not set
CONFIG_SPARSEMEM_MANUAL=y
CONFIG_SPARSEMEM=y
CONFIG_HAVE_MEMORY_PRESENT=y
CONFIG_SPARSEMEM_EXTREME=y
CONFIG_SPARSEMEM_VMEMMAP_ENABLE=y
CONFIG_SPARSEMEM_VMEMMAP=y

#
# Memory hotplug is currently incompatible with Software Suspend
#
CONFIG_PAGEFLAGS_EXTENDED=y
CONFIG_SPLIT_PTLOCK_CPUS=4
CONFIG_PHYS_ADDR_T_64BIT=y
CONFIG_ZONE_DMA_FLAG=1
CONFIG_BOUNCE=y
CONFIG_VIRT_TO_BUS=y
CONFIG_HAVE_MLOCK=y
CONFIG_HAVE_MLOCKED_PAGE_BIT=y
CONFIG_MMU_NOTIFIER=y
CONFIG_DEFAULT_MMAP_MIN_ADDR=4096
CONFIG_X86_CHECK_BIOS_CORRUPTION=y
CONFIG_X86_BOOTPARAM_MEMORY_CORRUPTION_CHECK=y
CONFIG_X86_RESERVE_LOW_64K=y
CONFIG_MTRR=y
CONFIG_MTRR_SANITIZER=y
CONFIG_MTRR_SANITIZER_ENABLE_DEFAULT=1
CONFIG_MTRR_SANITIZER_SPARE_REG_NR_DEFAULT=1
CONFIG_X86_PAT=y
# CONFIG_EFI is not set
# CONFIG_SECCOMP is not set
# CONFIG_CC_STACKPROTECTOR is not set
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
CONFIG_HZ_300=y
# CONFIG_HZ_1000 is not set
CONFIG_HZ=300
CONFIG_SCHED_HRTICK=y
# CONFIG_KEXEC is not set
# CONFIG_CRASH_DUMP is not set
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x1000000
CONFIG_HOTPLUG_CPU=y
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y

#
# Power management and ACPI options
#
CONFIG_ARCH_HIBERNATION_HEADER=y
CONFIG_PM=y
CONFIG_PM_DEBUG=y
# CONFIG_PM_VERBOSE is not set
CONFIG_CAN_PM_TRACE=y
CONFIG_PM_TRACE=y
CONFIG_PM_TRACE_RTC=y
CONFIG_PM_SLEEP_SMP=y
CONFIG_PM_SLEEP=y
CONFIG_SUSPEND=y
# CONFIG_PM_TEST_SUSPEND is not set
CONFIG_SUSPEND_FREEZER=y
CONFIG_HIBERNATION_NVS=y
CONFIG_HIBERNATION=y
CONFIG_PM_STD_PARTITION=""
CONFIG_ACPI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_PROCFS=y
CONFIG_ACPI_PROCFS_POWER=y
CONFIG_ACPI_SYSFS_POWER=y
CONFIG_ACPI_PROC_EVENT=y
# CONFIG_ACPI_AC is not set
# CONFIG_ACPI_BATTERY is not set
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_VIDEO=m
CONFIG_ACPI_FAN=y
CONFIG_ACPI_DOCK=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_HOTPLUG_CPU=y
CONFIG_ACPI_THERMAL=y
# CONFIG_ACPI_CUSTOM_DSDT is not set
CONFIG_ACPI_BLACKLIST_YEAR=0
# CONFIG_ACPI_DEBUG is not set
CONFIG_ACPI_PCI_SLOT=m
CONFIG_X86_PM_TIMER=y
CONFIG_ACPI_CONTAINER=y
# CONFIG_ACPI_SBS is not set

#
# CPU Frequency scaling
#
CONFIG_CPU_FREQ=y
CONFIG_CPU_FREQ_TABLE=y
CONFIG_CPU_FREQ_DEBUG=y
CONFIG_CPU_FREQ_STAT=m
CONFIG_CPU_FREQ_STAT_DETAILS=y
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=y
# CONFIG_CPU_FREQ_DEFAULT_GOV_POWERSAVE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_USERSPACE is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND is not set
# CONFIG_CPU_FREQ_DEFAULT_GOV_CONSERVATIVE is not set
CONFIG_CPU_FREQ_GOV_PERFORMANCE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
# CONFIG_CPU_FREQ_GOV_USERSPACE is not set
CONFIG_CPU_FREQ_GOV_ONDEMAND=y
# CONFIG_CPU_FREQ_GOV_CONSERVATIVE is not set

#
# CPUFreq processor drivers
#
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_POWERNOW_K8=y
# CONFIG_X86_SPEEDSTEP_CENTRINO is not set
# CONFIG_X86_P4_CLOCKMOD is not set

#
# shared options
#
# CONFIG_X86_SPEEDSTEP_LIB is not set
CONFIG_CPU_IDLE=y
CONFIG_CPU_IDLE_GOV_LADDER=y
CONFIG_CPU_IDLE_GOV_MENU=y

#
# Memory power savings
#
# CONFIG_I7300_IDLE is not set

#
# Bus options (PCI etc.)
#
CONFIG_PCI=y
CONFIG_PCI_DIRECT=y
CONFIG_PCI_MMCONFIG=y
CONFIG_PCI_DOMAINS=y
CONFIG_DMAR=y
# CONFIG_DMAR_DEFAULT_ON is not set
# CONFIG_DMAR_BROKEN_GFX_WA is not set
CONFIG_DMAR_FLOPPY_WA=y
# CONFIG_INTR_REMAP is not set
CONFIG_PCIEPORTBUS=y
CONFIG_PCIEAER=y
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
# CONFIG_PCIEASPM is not set
CONFIG_ARCH_SUPPORTS_MSI=y
CONFIG_PCI_MSI=y
# CONFIG_PCI_LEGACY is not set
# CONFIG_PCI_STUB is not set
CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_K8_NB=y
# CONFIG_PCCARD is not set
# CONFIG_HOTPLUG_PCI is not set

#
# Executable file formats / Emulations
#
CONFIG_BINFMT_ELF=y
CONFIG_COMPAT_BINFMT_ELF=y
# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
# CONFIG_HAVE_AOUT is not set
CONFIG_BINFMT_MISC=m
CONFIG_IA32_EMULATION=y
CONFIG_IA32_AOUT=m
CONFIG_COMPAT=y
CONFIG_COMPAT_FOR_U64_ALIGNMENT=y
CONFIG_SYSVIPC_COMPAT=y
CONFIG_NET=y

#
# Networking options
#
CONFIG_PACKET=y
CONFIG_PACKET_MMAP=y
CONFIG_UNIX=y
CONFIG_XFRM=y
# CONFIG_XFRM_USER is not set
# CONFIG_XFRM_SUB_POLICY is not set
# CONFIG_XFRM_MIGRATE is not set
# CONFIG_XFRM_STATISTICS is not set
# CONFIG_NET_KEY is not set
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
# CONFIG_IP_ADVANCED_ROUTER is not set
CONFIG_IP_FIB_HASH=y
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
CONFIG_SYN_COOKIES=y
# CONFIG_INET_AH is not set
# CONFIG_INET_ESP is not set
# CONFIG_INET_IPCOMP is not set
# CONFIG_INET_XFRM_TUNNEL is not set
CONFIG_INET_TUNNEL=y
# CONFIG_INET_XFRM_MODE_TRANSPORT is not set
# CONFIG_INET_XFRM_MODE_TUNNEL is not set
# CONFIG_INET_XFRM_MODE_BEET is not set
CONFIG_INET_LRO=y
CONFIG_INET_DIAG=m
CONFIG_INET_TCP_DIAG=m
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
CONFIG_TCP_CONG_CUBIC=y
# CONFIG_TCP_CONG_WESTWOOD is not set
# CONFIG_TCP_CONG_HTCP is not set
# CONFIG_TCP_CONG_HSTCP is not set
# CONFIG_TCP_CONG_HYBLA is not set
# CONFIG_TCP_CONG_VEGAS is not set
# CONFIG_TCP_CONG_SCALABLE is not set
# CONFIG_TCP_CONG_LP is not set
# CONFIG_TCP_CONG_VENO is not set
# CONFIG_TCP_CONG_YEAH is not set
# CONFIG_TCP_CONG_ILLINOIS is not set
# CONFIG_DEFAULT_BIC is not set
CONFIG_DEFAULT_CUBIC=y
# CONFIG_DEFAULT_HTCP is not set
# CONFIG_DEFAULT_VEGAS is not set
# CONFIG_DEFAULT_WESTWOOD is not set
# CONFIG_DEFAULT_RENO is not set
CONFIG_DEFAULT_TCP_CONG="cubic"
CONFIG_TCP_MD5SIG=y
CONFIG_IPV6=y
# CONFIG_IPV6_PRIVACY is not set
# CONFIG_IPV6_ROUTER_PREF is not set
# CONFIG_IPV6_OPTIMISTIC_DAD is not set
CONFIG_INET6_AH=y
CONFIG_INET6_ESP=y
# CONFIG_INET6_IPCOMP is not set
# CONFIG_IPV6_MIP6 is not set
# CONFIG_INET6_XFRM_TUNNEL is not set
# CONFIG_INET6_TUNNEL is not set
CONFIG_INET6_XFRM_MODE_TRANSPORT=y
CONFIG_INET6_XFRM_MODE_TUNNEL=y
CONFIG_INET6_XFRM_MODE_BEET=y
# CONFIG_INET6_XFRM_MODE_ROUTEOPTIMIZATION is not set
CONFIG_IPV6_SIT=y
CONFIG_IPV6_NDISC_NODETYPE=y
# CONFIG_IPV6_TUNNEL is not set
# CONFIG_IPV6_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set
# CONFIG_NETWORK_SECMARK is not set
# CONFIG_NETFILTER is not set
# CONFIG_IP_DCCP is not set
# CONFIG_IP_SCTP is not set
# CONFIG_TIPC is not set
# CONFIG_ATM is not set
# CONFIG_BRIDGE is not set
# CONFIG_NET_DSA is not set
CONFIG_VLAN_8021Q=m
# CONFIG_VLAN_8021Q_GVRP is not set
# CONFIG_DECNET is not set
# CONFIG_LLC2 is not set
# CONFIG_IPX is not set
# CONFIG_ATALK is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_PHONET is not set
# CONFIG_IEEE802154 is not set
# CONFIG_NET_SCHED is not set
# CONFIG_DCB is not set

#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
# CONFIG_HAMRADIO is not set
# CONFIG_CAN is not set
# CONFIG_IRDA is not set
CONFIG_BT=m
CONFIG_BT_L2CAP=m
CONFIG_BT_SCO=m
CONFIG_BT_RFCOMM=m
CONFIG_BT_RFCOMM_TTY=y
CONFIG_BT_BNEP=m
CONFIG_BT_BNEP_MC_FILTER=y
CONFIG_BT_BNEP_PROTO_FILTER=y
CONFIG_BT_HIDP=m

#
# Bluetooth device drivers
#
CONFIG_BT_HCIBTUSB=m
CONFIG_BT_HCIUART=m
CONFIG_BT_HCIUART_H4=y
CONFIG_BT_HCIUART_BCSP=y
CONFIG_BT_HCIUART_LL=y
# CONFIG_BT_HCIBCM203X is not set
# CONFIG_BT_HCIBPA10X is not set
# CONFIG_BT_HCIBFUSB is not set
CONFIG_BT_HCIVHCI=m
# CONFIG_AF_RXRPC is not set
# CONFIG_WIRELESS is not set
# CONFIG_WIMAX is not set
CONFIG_RFKILL=y
CONFIG_RFKILL_LEDS=y
# CONFIG_RFKILL_INPUT is not set
# CONFIG_NET_9P is not set

#
# Device Drivers
#

#
# Generic Driver Options
#
CONFIG_UEVENT_HELPER_PATH="/sbin/hotplug"
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_FW_LOADER=y
CONFIG_FIRMWARE_IN_KERNEL=y
CONFIG_EXTRA_FIRMWARE=""
# CONFIG_SYS_HYPERVISOR is not set
CONFIG_CONNECTOR=y
CONFIG_PROC_EVENTS=y
# CONFIG_MTD is not set
# CONFIG_PARPORT is not set
CONFIG_PNP=y
CONFIG_PNP_DEBUG_MESSAGES=y

#
# Protocols
#
CONFIG_PNPACPI=y
CONFIG_BLK_DEV=y
# CONFIG_BLK_DEV_FD is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not set
# CONFIG_BLK_DEV_COW_COMMON is not set
CONFIG_BLK_DEV_LOOP=m
# CONFIG_BLK_DEV_CRYPTOLOOP is not set
# CONFIG_BLK_DEV_NBD is not set
# CONFIG_BLK_DEV_SX8 is not set
# CONFIG_BLK_DEV_UB is not set
# CONFIG_BLK_DEV_RAM is not set
CONFIG_CDROM_PKTCDVD=m
CONFIG_CDROM_PKTCDVD_BUFFERS=32
# CONFIG_CDROM_PKTCDVD_WCACHE is not set
# CONFIG_ATA_OVER_ETH is not set
# CONFIG_BLK_DEV_HD is not set
CONFIG_MISC_DEVICES=y
# CONFIG_IBM_ASM is not set
# CONFIG_PHANTOM is not set
# CONFIG_SGI_IOC4 is not set
# CONFIG_TIFM_CORE is not set
# CONFIG_ICS932S401 is not set
# CONFIG_ENCLOSURE_SERVICES is not set
# CONFIG_HP_ILO is not set
# CONFIG_ISL29003 is not set
# CONFIG_C2PORT is not set

#
# EEPROM support
#
CONFIG_EEPROM_AT24=m
# CONFIG_EEPROM_AT25 is not set
CONFIG_EEPROM_LEGACY=m
# CONFIG_EEPROM_MAX6875 is not set
# CONFIG_EEPROM_93CX6 is not set
# CONFIG_CB710_CORE is not set
CONFIG_HAVE_IDE=y
# CONFIG_IDE is not set

#
# SCSI device support
#
# CONFIG_RAID_ATTRS is not set
CONFIG_SCSI=y
CONFIG_SCSI_DMA=y
# CONFIG_SCSI_TGT is not set
# CONFIG_SCSI_NETLINK is not set
CONFIG_SCSI_PROC_FS=y

#
# SCSI support type (disk, tape, CD-ROM)
#
CONFIG_BLK_DEV_SD=y
CONFIG_CHR_DEV_ST=m
# CONFIG_CHR_DEV_OSST is not set
CONFIG_BLK_DEV_SR=m
CONFIG_BLK_DEV_SR_VENDOR=y
CONFIG_CHR_DEV_SG=y
CONFIG_CHR_DEV_SCH=m
CONFIG_SCSI_MULTI_LUN=y
# CONFIG_SCSI_CONSTANTS is not set
# CONFIG_SCSI_LOGGING is not set
# CONFIG_SCSI_SCAN_ASYNC is not set
CONFIG_SCSI_WAIT_SCAN=m

#
# SCSI Transports
#
CONFIG_SCSI_SPI_ATTRS=y
# CONFIG_SCSI_FC_ATTRS is not set
# CONFIG_SCSI_ISCSI_ATTRS is not set
# CONFIG_SCSI_SAS_ATTRS is not set
# CONFIG_SCSI_SAS_LIBSAS is not set
# CONFIG_SCSI_SRP_ATTRS is not set
CONFIG_SCSI_LOWLEVEL=y
# CONFIG_ISCSI_TCP is not set
# CONFIG_SCSI_BNX2_ISCSI is not set
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_3W_9XXX is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AACRAID is not set
CONFIG_SCSI_AIC7XXX=m
CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
CONFIG_AIC7XXX_RESET_DELAY_MS=5000
CONFIG_AIC7XXX_DEBUG_ENABLE=y
CONFIG_AIC7XXX_DEBUG_MASK=0
CONFIG_AIC7XXX_REG_PRETTY_PRINT=y
# CONFIG_SCSI_AIC7XXX_OLD is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_AIC94XX is not set
# CONFIG_SCSI_MVSAS is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_ARCMSR is not set
# CONFIG_MEGARAID_NEWGEN is not set
# CONFIG_MEGARAID_LEGACY is not set
# CONFIG_MEGARAID_SAS is not set
# CONFIG_SCSI_MPT2SAS is not set
# CONFIG_SCSI_HPTIOP is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_LIBFC is not set
# CONFIG_LIBFCOE is not set
# CONFIG_FCOE is not set
# CONFIG_FCOE_FNIC is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_IPS is not set
CONFIG_SCSI_INITIO=m
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_STEX is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_IPR is not set
# CONFIG_SCSI_QLOGIC_1280 is not set
# CONFIG_SCSI_QLA_FC is not set
# CONFIG_SCSI_QLA_ISCSI is not set
# CONFIG_SCSI_LPFC is not set
# CONFIG_SCSI_DC395x is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_DEBUG is not set
# CONFIG_SCSI_SRP is not set
# CONFIG_SCSI_DH is not set
# CONFIG_SCSI_OSD_INITIATOR is not set
CONFIG_ATA=y
# CONFIG_ATA_NONSTANDARD is not set
CONFIG_ATA_ACPI=y
# CONFIG_SATA_PMP is not set
CONFIG_SATA_AHCI=y
# CONFIG_SATA_SIL24 is not set
# CONFIG_ATA_SFF is not set
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_AUTODETECT=y
# CONFIG_MD_LINEAR is not set
# CONFIG_MD_RAID0 is not set
CONFIG_MD_RAID1=y
# CONFIG_MD_RAID10 is not set
CONFIG_MD_RAID456=y
CONFIG_MD_RAID6_PQ=y
# CONFIG_MD_MULTIPATH is not set
# CONFIG_MD_FAULTY is not set
# CONFIG_BLK_DEV_DM is not set
# CONFIG_FUSION is not set

#
# IEEE 1394 (FireWire) support
#

#
# You can enable one or both FireWire driver stacks.
#

#
# See the help texts for more information.
#
# CONFIG_FIREWIRE is not set
# CONFIG_IEEE1394 is not set
# CONFIG_I2O is not set
# CONFIG_MACINTOSH_DRIVERS is not set
CONFIG_NETDEVICES=y
# CONFIG_DUMMY is not set
# CONFIG_BONDING is not set
CONFIG_MACVLAN=m
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_VETH is not set
# CONFIG_NET_SB1000 is not set
# CONFIG_ARCNET is not set
# CONFIG_NET_ETHERNET is not set
CONFIG_MII=m
CONFIG_NETDEV_1000=y
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set
# CONFIG_IP1000 is not set
# CONFIG_IGB is not set
# CONFIG_IGBVF is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
CONFIG_R8169=m
CONFIG_R8169_VLAN=y
# CONFIG_SIS190 is not set
# CONFIG_SKGE is not set
# CONFIG_SKY2 is not set
# CONFIG_VIA_VELOCITY is not set
# CONFIG_TIGON3 is not set
# CONFIG_BNX2 is not set
# CONFIG_CNIC is not set
# CONFIG_QLA3XXX is not set
# CONFIG_ATL1 is not set
# CONFIG_ATL1E is not set
# CONFIG_ATL1C is not set
# CONFIG_JME is not set
# CONFIG_NETDEV_10000 is not set
# CONFIG_TR is not set

#
# Wireless LAN
#
# CONFIG_WLAN_PRE80211 is not set
CONFIG_WLAN_80211=y
# CONFIG_LIBERTAS is not set
# CONFIG_AIRO is not set
# CONFIG_ATMEL is not set
# CONFIG_PRISM54 is not set
# CONFIG_USB_ZD1201 is not set
# CONFIG_IPW2100 is not set
# CONFIG_IPW2200 is not set
# CONFIG_HOSTAP is not set
# CONFIG_HERMES is not set

#
# Enable WiMAX (Networking options) to see the WiMAX drivers
#

#
# USB Network Adapters
#
# CONFIG_USB_CATC is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_HSO is not set
# CONFIG_WAN is not set
# CONFIG_FDDI is not set
# CONFIG_HIPPI is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
# CONFIG_NET_FC is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_NETPOLL is not set
# CONFIG_NET_POLL_CONTROLLER is not set
# CONFIG_ISDN is not set
# CONFIG_PHONE is not set

#
# Input device support
#
CONFIG_INPUT=y
CONFIG_INPUT_FF_MEMLESS=y
CONFIG_INPUT_POLLDEV=y

#
# Userland interfaces
#
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1680
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=1050
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=y
# CONFIG_INPUT_EVBUG is not set

#
# Input Device Drivers
#
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
# CONFIG_KEYBOARD_LKKBD is not set
# CONFIG_KEYBOARD_LM8323 is not set
# CONFIG_KEYBOARD_NEWTON is not set
# CONFIG_KEYBOARD_STOWAWAY is not set
# CONFIG_KEYBOARD_SUNKBD is not set
# CONFIG_KEYBOARD_XTKBD is not set
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_MOUSE_PS2_ALPS=y
CONFIG_MOUSE_PS2_LOGIPS2PP=y
CONFIG_MOUSE_PS2_SYNAPTICS=y
CONFIG_MOUSE_PS2_LIFEBOOK=y
CONFIG_MOUSE_PS2_TRACKPOINT=y
# CONFIG_MOUSE_PS2_ELANTECH is not set
# CONFIG_MOUSE_PS2_TOUCHKIT is not set
# CONFIG_MOUSE_SERIAL is not set
# CONFIG_MOUSE_APPLETOUCH is not set
# CONFIG_MOUSE_BCM5974 is not set
# CONFIG_MOUSE_VSXXXAA is not set
# CONFIG_MOUSE_SYNAPTICS_I2C is not set
CONFIG_INPUT_JOYSTICK=y
CONFIG_JOYSTICK_ANALOG=m
# CONFIG_JOYSTICK_A3D is not set
# CONFIG_JOYSTICK_ADI is not set
# CONFIG_JOYSTICK_COBRA is not set
# CONFIG_JOYSTICK_GF2K is not set
# CONFIG_JOYSTICK_GRIP is not set
# CONFIG_JOYSTICK_GRIP_MP is not set
# CONFIG_JOYSTICK_GUILLEMOT is not set
# CONFIG_JOYSTICK_INTERACT is not set
# CONFIG_JOYSTICK_SIDEWINDER is not set
# CONFIG_JOYSTICK_TMDC is not set
# CONFIG_JOYSTICK_IFORCE is not set
# CONFIG_JOYSTICK_WARRIOR is not set
# CONFIG_JOYSTICK_MAGELLAN is not set
# CONFIG_JOYSTICK_SPACEORB is not set
# CONFIG_JOYSTICK_SPACEBALL is not set
# CONFIG_JOYSTICK_STINGER is not set
# CONFIG_JOYSTICK_TWIDJOY is not set
# CONFIG_JOYSTICK_ZHENHUA is not set
# CONFIG_JOYSTICK_JOYDUMP is not set
# CONFIG_JOYSTICK_XPAD is not set
# CONFIG_INPUT_TABLET is not set
# CONFIG_INPUT_TOUCHSCREEN is not set
CONFIG_INPUT_MISC=y
CONFIG_INPUT_PCSPKR=m
# CONFIG_INPUT_APANEL is not set
# CONFIG_INPUT_ATLAS_BTNS is not set
# CONFIG_INPUT_ATI_REMOTE is not set
# CONFIG_INPUT_ATI_REMOTE2 is not set
# CONFIG_INPUT_KEYSPAN_REMOTE is not set
# CONFIG_INPUT_POWERMATE is not set
# CONFIG_INPUT_YEALINK is not set
# CONFIG_INPUT_CM109 is not set
# CONFIG_INPUT_UINPUT is not set

#
# Hardware I/O ports
#
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_SERIO_SERPORT=y
# CONFIG_SERIO_CT82C710 is not set
# CONFIG_SERIO_PCIPS2 is not set
CONFIG_SERIO_LIBPS2=y
# CONFIG_SERIO_RAW is not set
CONFIG_GAMEPORT=m
CONFIG_GAMEPORT_NS558=m
# CONFIG_GAMEPORT_L4 is not set
CONFIG_GAMEPORT_EMU10K1=m
# CONFIG_GAMEPORT_FM801 is not set

#
# Character devices
#
CONFIG_VT=y
CONFIG_CONSOLE_TRANSLATIONS=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_VT_HW_CONSOLE_BINDING=y
# CONFIG_DEVKMEM is not set
# CONFIG_SERIAL_NONSTANDARD is not set
# CONFIG_NOZOMI is not set

#
# Serial drivers
#
# CONFIG_SERIAL_8250 is not set
CONFIG_FIX_EARLYCON_MEM=y

#
# Non-8250 serial port support
#
# CONFIG_SERIAL_MAX3100 is not set
# CONFIG_SERIAL_JSM is not set
CONFIG_UNIX98_PTYS=y
# CONFIG_DEVPTS_MULTIPLE_INSTANCES is not set
# CONFIG_LEGACY_PTYS is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_HW_RANDOM is not set
CONFIG_NVRAM=m
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_MWAVE is not set
# CONFIG_PC8736x_GPIO is not set
# CONFIG_RAW_DRIVER is not set
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_HANGCHECK_TIMER=m
# CONFIG_TCG_TPM is not set
# CONFIG_TELCLOCK is not set
CONFIG_DEVPORT=y
CONFIG_I2C=y
CONFIG_I2C_BOARDINFO=y
CONFIG_I2C_CHARDEV=m
CONFIG_I2C_HELPER_AUTO=y
CONFIG_I2C_ALGOBIT=m

#
# I2C Hardware Bus support
#

#
# PC SMBus host controller drivers
#
# CONFIG_I2C_ALI1535 is not set
# CONFIG_I2C_ALI1563 is not set
# CONFIG_I2C_ALI15X3 is not set
# CONFIG_I2C_AMD756 is not set
# CONFIG_I2C_AMD8111 is not set
# CONFIG_I2C_I801 is not set
# CONFIG_I2C_ISCH is not set
CONFIG_I2C_PIIX4=m
# CONFIG_I2C_NFORCE2 is not set
# CONFIG_I2C_SIS5595 is not set
# CONFIG_I2C_SIS630 is not set
# CONFIG_I2C_SIS96X is not set
# CONFIG_I2C_VIA is not set
# CONFIG_I2C_VIAPRO is not set

#
# I2C system bus drivers (mostly embedded / system-on-chip)
#
# CONFIG_I2C_OCORES is not set
# CONFIG_I2C_SIMTEC is not set

#
# External I2C/SMBus adapter drivers
#
# CONFIG_I2C_PARPORT_LIGHT is not set
# CONFIG_I2C_TAOS_EVM is not set
# CONFIG_I2C_TINY_USB is not set

#
# Graphics adapter I2C/DDC channel drivers
#
# CONFIG_I2C_VOODOO3 is not set

#
# Other I2C/SMBus bus drivers
#
# CONFIG_I2C_PCA_PLATFORM is not set
# CONFIG_I2C_STUB is not set

#
# Miscellaneous I2C Chip support
#
# CONFIG_DS1682 is not set
# CONFIG_SENSORS_PCF8574 is not set
# CONFIG_PCF8575 is not set
# CONFIG_SENSORS_PCA9539 is not set
# CONFIG_SENSORS_TSL2550 is not set
# CONFIG_I2C_DEBUG_CORE is not set
# CONFIG_I2C_DEBUG_ALGO is not set
# CONFIG_I2C_DEBUG_BUS is not set
# CONFIG_I2C_DEBUG_CHIP is not set
CONFIG_SPI=y
CONFIG_SPI_MASTER=y

#
# SPI Master Controller Drivers
#
CONFIG_SPI_BITBANG=m

#
# SPI Protocol Masters
#
# CONFIG_SPI_SPIDEV is not set
# CONFIG_SPI_TLE62X0 is not set

#
# PPS support
#
# CONFIG_PPS is not set
CONFIG_ARCH_WANT_OPTIONAL_GPIOLIB=y
# CONFIG_GPIOLIB is not set
# CONFIG_W1 is not set
CONFIG_POWER_SUPPLY=y
# CONFIG_POWER_SUPPLY_DEBUG is not set
# CONFIG_PDA_POWER is not set
# CONFIG_BATTERY_DS2760 is not set
# CONFIG_BATTERY_DS2782 is not set
# CONFIG_BATTERY_BQ27x00 is not set
# CONFIG_BATTERY_MAX17040 is not set
CONFIG_HWMON=y
CONFIG_HWMON_VID=m
# CONFIG_SENSORS_ABITUGURU is not set
# CONFIG_SENSORS_ABITUGURU3 is not set
# CONFIG_SENSORS_AD7414 is not set
# CONFIG_SENSORS_AD7418 is not set
# CONFIG_SENSORS_ADCXX is not set
# CONFIG_SENSORS_ADM1021 is not set
# CONFIG_SENSORS_ADM1025 is not set
# CONFIG_SENSORS_ADM1026 is not set
# CONFIG_SENSORS_ADM1029 is not set
# CONFIG_SENSORS_ADM1031 is not set
# CONFIG_SENSORS_ADM9240 is not set
# CONFIG_SENSORS_ADT7462 is not set
# CONFIG_SENSORS_ADT7470 is not set
# CONFIG_SENSORS_ADT7473 is not set
# CONFIG_SENSORS_ADT7475 is not set
CONFIG_SENSORS_K8TEMP=m
# CONFIG_SENSORS_ASB100 is not set
# CONFIG_SENSORS_ATK0110 is not set
# CONFIG_SENSORS_ATXP1 is not set
# CONFIG_SENSORS_DS1621 is not set
# CONFIG_SENSORS_I5K_AMB is not set
# CONFIG_SENSORS_F71805F is not set
# CONFIG_SENSORS_F71882FG is not set
# CONFIG_SENSORS_F75375S is not set
# CONFIG_SENSORS_FSCHER is not set
# CONFIG_SENSORS_FSCPOS is not set
# CONFIG_SENSORS_FSCHMD is not set
# CONFIG_SENSORS_G760A is not set
# CONFIG_SENSORS_GL518SM is not set
# CONFIG_SENSORS_GL520SM is not set
# CONFIG_SENSORS_CORETEMP is not set
# CONFIG_SENSORS_IT87 is not set
# CONFIG_SENSORS_LM63 is not set
# CONFIG_SENSORS_LM70 is not set
# CONFIG_SENSORS_LM75 is not set
# CONFIG_SENSORS_LM77 is not set
# CONFIG_SENSORS_LM78 is not set
# CONFIG_SENSORS_LM80 is not set
# CONFIG_SENSORS_LM83 is not set
# CONFIG_SENSORS_LM85 is not set
# CONFIG_SENSORS_LM87 is not set
# CONFIG_SENSORS_LM90 is not set
# CONFIG_SENSORS_LM92 is not set
# CONFIG_SENSORS_LM93 is not set
# CONFIG_SENSORS_LTC4215 is not set
# CONFIG_SENSORS_LTC4245 is not set
# CONFIG_SENSORS_LM95241 is not set
# CONFIG_SENSORS_MAX1111 is not set
# CONFIG_SENSORS_MAX1619 is not set
# CONFIG_SENSORS_MAX6650 is not set
# CONFIG_SENSORS_PC87360 is not set
# CONFIG_SENSORS_PC87427 is not set
# CONFIG_SENSORS_PCF8591 is not set
# CONFIG_SENSORS_SIS5595 is not set
# CONFIG_SENSORS_DME1737 is not set
# CONFIG_SENSORS_SMSC47M1 is not set
# CONFIG_SENSORS_SMSC47M192 is not set
# CONFIG_SENSORS_SMSC47B397 is not set
# CONFIG_SENSORS_ADS7828 is not set
# CONFIG_SENSORS_THMC50 is not set
# CONFIG_SENSORS_TMP401 is not set
# CONFIG_SENSORS_VIA686A is not set
# CONFIG_SENSORS_VT1211 is not set
# CONFIG_SENSORS_VT8231 is not set
# CONFIG_SENSORS_W83781D is not set
# CONFIG_SENSORS_W83791D is not set
# CONFIG_SENSORS_W83792D is not set
# CONFIG_SENSORS_W83793 is not set
# CONFIG_SENSORS_W83L785TS is not set
# CONFIG_SENSORS_W83L786NG is not set
# CONFIG_SENSORS_W83627HF is not set
CONFIG_SENSORS_W83627EHF=m
# CONFIG_SENSORS_HDAPS is not set
# CONFIG_SENSORS_LIS3LV02D is not set
# CONFIG_SENSORS_APPLESMC is not set
# CONFIG_HWMON_DEBUG_CHIP is not set
CONFIG_THERMAL=y
CONFIG_THERMAL_HWMON=y
# CONFIG_WATCHDOG is not set
CONFIG_SSB_POSSIBLE=y

#
# Sonics Silicon Backplane
#
# CONFIG_SSB is not set

#
# Multifunction device drivers
#
# CONFIG_MFD_CORE is not set
# CONFIG_MFD_SM501 is not set
# CONFIG_HTC_PASIC3 is not set
# CONFIG_TWL4030_CORE is not set
# CONFIG_MFD_TMIO is not set
# CONFIG_PMIC_DA903X is not set
# CONFIG_MFD_WM8400 is not set
# CONFIG_MFD_WM8350_I2C is not set
# CONFIG_MFD_PCF50633 is not set
# CONFIG_AB3100_CORE is not set
# CONFIG_EZX_PCAP is not set
# CONFIG_REGULATOR is not set
CONFIG_MEDIA_SUPPORT=m

#
# Multimedia core support
#
CONFIG_VIDEO_DEV=m
CONFIG_VIDEO_V4L2_COMMON=m
CONFIG_VIDEO_ALLOW_V4L1=y
CONFIG_VIDEO_V4L1_COMPAT=y
CONFIG_DVB_CORE=m
CONFIG_VIDEO_MEDIA=m

#
# Multimedia drivers
#
CONFIG_MEDIA_ATTACH=y
CONFIG_MEDIA_TUNER=m
CONFIG_MEDIA_TUNER_CUSTOMISE=y
CONFIG_MEDIA_TUNER_SIMPLE=m
# CONFIG_MEDIA_TUNER_TDA8290 is not set
# CONFIG_MEDIA_TUNER_TDA827X is not set
# CONFIG_MEDIA_TUNER_TDA18271 is not set
CONFIG_MEDIA_TUNER_TDA9887=m
# CONFIG_MEDIA_TUNER_TEA5761 is not set
# CONFIG_MEDIA_TUNER_TEA5767 is not set
# CONFIG_MEDIA_TUNER_MT20XX is not set
# CONFIG_MEDIA_TUNER_MT2060 is not set
# CONFIG_MEDIA_TUNER_MT2266 is not set
# CONFIG_MEDIA_TUNER_MT2131 is not set
# CONFIG_MEDIA_TUNER_QT1010 is not set
CONFIG_MEDIA_TUNER_XC2028=m
CONFIG_MEDIA_TUNER_XC5000=m
# CONFIG_MEDIA_TUNER_MXL5005S is not set
# CONFIG_MEDIA_TUNER_MXL5007T is not set
# CONFIG_MEDIA_TUNER_MC44S803 is not set
CONFIG_VIDEO_V4L2=m
CONFIG_VIDEO_V4L1=m
CONFIG_VIDEOBUF_GEN=m
CONFIG_VIDEOBUF_DMA_SG=m
CONFIG_VIDEOBUF_VMALLOC=m
CONFIG_VIDEOBUF_DVB=m
CONFIG_VIDEO_BTCX=m
CONFIG_VIDEO_IR=m
CONFIG_VIDEO_TVEEPROM=m
CONFIG_VIDEO_TUNER=m
CONFIG_VIDEO_CAPTURE_DRIVERS=y
# CONFIG_VIDEO_ADV_DEBUG is not set
# CONFIG_VIDEO_FIXED_MINOR_RANGES is not set
CONFIG_VIDEO_HELPER_CHIPS_AUTO=y
CONFIG_VIDEO_IR_I2C=m
CONFIG_VIDEO_TVAUDIO=m
CONFIG_VIDEO_TDA7432=m
CONFIG_VIDEO_MSP3400=m
CONFIG_VIDEO_SAA6588=m
CONFIG_VIDEO_MT9V011=m
CONFIG_VIDEO_SAA711X=m
CONFIG_VIDEO_TVP5150=m
# CONFIG_VIDEO_VIVI is not set
CONFIG_VIDEO_BT848=m
# CONFIG_VIDEO_BT848_DVB is not set
# CONFIG_VIDEO_CPIA is not set
# CONFIG_VIDEO_CPIA2 is not set
# CONFIG_VIDEO_SAA5246A is not set
# CONFIG_VIDEO_SAA5249 is not set
# CONFIG_VIDEO_STRADIS is not set
# CONFIG_VIDEO_ZORAN is not set
# CONFIG_VIDEO_SAA7134 is not set
# CONFIG_VIDEO_MXB is not set
# CONFIG_VIDEO_HEXIUM_ORION is not set
# CONFIG_VIDEO_HEXIUM_GEMINI is not set
# CONFIG_VIDEO_CX88 is not set
# CONFIG_VIDEO_CX23885 is not set
# CONFIG_VIDEO_AU0828 is not set
# CONFIG_VIDEO_IVTV is not set
# CONFIG_VIDEO_CX18 is not set
# CONFIG_VIDEO_CAFE_CCIC is not set
# CONFIG_SOC_CAMERA is not set
CONFIG_V4L_USB_DRIVERS=y
# CONFIG_USB_VIDEO_CLASS is not set
CONFIG_USB_VIDEO_CLASS_INPUT_EVDEV=y
# CONFIG_USB_GSPCA is not set
# CONFIG_VIDEO_PVRUSB2 is not set
# CONFIG_VIDEO_HDPVR is not set
CONFIG_VIDEO_EM28XX=m
CONFIG_VIDEO_EM28XX_ALSA=m
CONFIG_VIDEO_EM28XX_DVB=m
# CONFIG_VIDEO_CX231XX is not set
# CONFIG_VIDEO_USBVISION is not set
# CONFIG_USB_VICAM is not set
# CONFIG_USB_IBMCAM is not set
# CONFIG_USB_KONICAWC is not set
# CONFIG_USB_QUICKCAM_MESSENGER is not set
# CONFIG_USB_ET61X251 is not set
# CONFIG_VIDEO_OVCAMCHIP is not set
# CONFIG_USB_OV511 is not set
# CONFIG_USB_SE401 is not set
# CONFIG_USB_SN9C102 is not set
# CONFIG_USB_STV680 is not set
# CONFIG_USB_ZC0301 is not set
# CONFIG_USB_PWC is not set
# CONFIG_USB_PWC_INPUT_EVDEV is not set
# CONFIG_USB_ZR364XX is not set
# CONFIG_USB_STKWEBCAM is not set
# CONFIG_USB_S2255 is not set
# CONFIG_RADIO_ADAPTERS is not set
# CONFIG_DVB_DYNAMIC_MINORS is not set
CONFIG_DVB_CAPTURE_DRIVERS=y

#
# Supported SAA7146 based PCI Adapters
#
# CONFIG_TTPCI_EEPROM is not set
# CONFIG_DVB_AV7110 is not set
# CONFIG_DVB_BUDGET_CORE is not set

#
# Supported USB Adapters
#
# CONFIG_DVB_USB is not set
# CONFIG_DVB_TTUSB_BUDGET is not set
# CONFIG_DVB_TTUSB_DEC is not set
# CONFIG_SMS_SIANO_MDTV is not set

#
# Supported FlexCopII (B2C2) Adapters
#
# CONFIG_DVB_B2C2_FLEXCOP is not set

#
# Supported BT878 Adapters
#
# CONFIG_DVB_BT8XX is not set

#
# Supported Pluto2 Adapters
#
# CONFIG_DVB_PLUTO2 is not set

#
# Supported SDMC DM1105 Adapters
#
# CONFIG_DVB_DM1105 is not set

#
# Supported DVB Frontends
#
# CONFIG_DVB_FE_CUSTOMISE is not set
CONFIG_DVB_ZL10353=m
CONFIG_DVB_LGDT330X=m
# CONFIG_DAB is not set

#
# Graphics support
#
CONFIG_AGP=y
CONFIG_AGP_AMD64=y
CONFIG_AGP_INTEL=y
# CONFIG_AGP_SIS is not set
# CONFIG_AGP_VIA is not set
CONFIG_DRM=m
# CONFIG_DRM_TDFX is not set
# CONFIG_DRM_R128 is not set
# CONFIG_DRM_RADEON is not set
# CONFIG_DRM_I810 is not set
# CONFIG_DRM_I830 is not set
CONFIG_DRM_I915=m
CONFIG_DRM_I915_KMS=y
# CONFIG_DRM_MGA is not set
# CONFIG_DRM_SIS is not set
# CONFIG_DRM_VIA is not set
# CONFIG_DRM_SAVAGE is not set
# CONFIG_VGASTATE is not set
CONFIG_VIDEO_OUTPUT_CONTROL=m
CONFIG_FB=m
# CONFIG_FIRMWARE_EDID is not set
# CONFIG_FB_DDC is not set
# CONFIG_FB_BOOT_VESA_SUPPORT is not set
CONFIG_FB_CFB_FILLRECT=m
CONFIG_FB_CFB_COPYAREA=m
CONFIG_FB_CFB_IMAGEBLIT=m
# CONFIG_FB_CFB_REV_PIXELS_IN_BYTE is not set
# CONFIG_FB_SYS_FILLRECT is not set
# CONFIG_FB_SYS_COPYAREA is not set
# CONFIG_FB_SYS_IMAGEBLIT is not set
# CONFIG_FB_FOREIGN_ENDIAN is not set
# CONFIG_FB_SYS_FOPS is not set
# CONFIG_FB_SVGALIB is not set
# CONFIG_FB_MACMODES is not set
# CONFIG_FB_BACKLIGHT is not set
CONFIG_FB_MODE_HELPERS=y
CONFIG_FB_TILEBLITTING=y

#
# Frame buffer hardware drivers
#
# CONFIG_FB_CIRRUS is not set
# CONFIG_FB_PM2 is not set
# CONFIG_FB_CYBER2000 is not set
# CONFIG_FB_ARC is not set
# CONFIG_FB_VGA16 is not set
# CONFIG_FB_UVESA is not set
# CONFIG_FB_N411 is not set
# CONFIG_FB_HGA is not set
# CONFIG_FB_S1D13XXX is not set
# CONFIG_FB_NVIDIA is not set
# CONFIG_FB_RIVA is not set
# CONFIG_FB_LE80578 is not set
# CONFIG_FB_INTEL is not set
# CONFIG_FB_MATROX is not set
# CONFIG_FB_RADEON is not set
# CONFIG_FB_ATY128 is not set
# CONFIG_FB_ATY is not set
# CONFIG_FB_S3 is not set
# CONFIG_FB_SAVAGE is not set
# CONFIG_FB_SIS is not set
# CONFIG_FB_VIA is not set
# CONFIG_FB_NEOMAGIC is not set
# CONFIG_FB_KYRO is not set
# CONFIG_FB_3DFX is not set
# CONFIG_FB_VOODOO1 is not set
# CONFIG_FB_VT8623 is not set
# CONFIG_FB_TRIDENT is not set
# CONFIG_FB_ARK is not set
# CONFIG_FB_PM3 is not set
# CONFIG_FB_CARMINE is not set
# CONFIG_FB_GEODE is not set
# CONFIG_FB_VIRTUAL is not set
# CONFIG_FB_METRONOME is not set
# CONFIG_FB_MB862XX is not set
# CONFIG_FB_BROADSHEET is not set
# CONFIG_BACKLIGHT_LCD_SUPPORT is not set
CONFIG_BACKLIGHT_CLASS_DEVICE=m
CONFIG_BACKLIGHT_GENERIC=m
# CONFIG_BACKLIGHT_PROGEAR is not set
# CONFIG_BACKLIGHT_MBP_NVIDIA is not set
# CONFIG_BACKLIGHT_SAHARA is not set

#
# Display device support
#
# CONFIG_DISPLAY_SUPPORT is not set

#
# Console display driver support
#
CONFIG_VGA_CONSOLE=y
CONFIG_VGACON_SOFT_SCROLLBACK=y
CONFIG_VGACON_SOFT_SCROLLBACK_SIZE=128
CONFIG_DUMMY_CONSOLE=y
# CONFIG_FRAMEBUFFER_CONSOLE is not set
CONFIG_LOGO=y
# CONFIG_LOGO_LINUX_MONO is not set
# CONFIG_LOGO_LINUX_VGA16 is not set
CONFIG_LOGO_LINUX_CLUT224=y
CONFIG_SOUND=y
CONFIG_SOUND_OSS_CORE=y
CONFIG_SND=y
CONFIG_SND_TIMER=m
CONFIG_SND_PCM=m
CONFIG_SND_HWDEP=m
CONFIG_SND_RAWMIDI=m
CONFIG_SND_SEQUENCER=m
CONFIG_SND_SEQ_DUMMY=m
CONFIG_SND_OSSEMUL=y
CONFIG_SND_MIXER_OSS=m
CONFIG_SND_PCM_OSS=m
CONFIG_SND_PCM_OSS_PLUGINS=y
CONFIG_SND_SEQUENCER_OSS=y
CONFIG_SND_HRTIMER=m
CONFIG_SND_SEQ_HRTIMER_DEFAULT=y
CONFIG_SND_DYNAMIC_MINORS=y
# CONFIG_SND_SUPPORT_OLD_API is not set
CONFIG_SND_VERBOSE_PROCFS=y
# CONFIG_SND_VERBOSE_PRINTK is not set
# CONFIG_SND_DEBUG is not set
CONFIG_SND_VMASTER=y
CONFIG_SND_RAWMIDI_SEQ=m
# CONFIG_SND_OPL3_LIB_SEQ is not set
# CONFIG_SND_OPL4_LIB_SEQ is not set
# CONFIG_SND_SBAWE_SEQ is not set
CONFIG_SND_EMU10K1_SEQ=m
CONFIG_SND_MPU401_UART=m
CONFIG_SND_AC97_CODEC=m
CONFIG_SND_DRIVERS=y
# CONFIG_SND_PCSP is not set
# CONFIG_SND_DUMMY is not set
CONFIG_SND_VIRMIDI=m
# CONFIG_SND_MTPAV is not set
# CONFIG_SND_SERIAL_U16550 is not set
CONFIG_SND_MPU401=m
# CONFIG_SND_AC97_POWER_SAVE is not set
CONFIG_SND_PCI=y
# CONFIG_SND_AD1889 is not set
# CONFIG_SND_ALS300 is not set
# CONFIG_SND_ALS4000 is not set
# CONFIG_SND_ALI5451 is not set
# CONFIG_SND_ATIIXP is not set
# CONFIG_SND_ATIIXP_MODEM is not set
# CONFIG_SND_AU8810 is not set
# CONFIG_SND_AU8820 is not set
# CONFIG_SND_AU8830 is not set
# CONFIG_SND_AW2 is not set
# CONFIG_SND_AZT3328 is not set
# CONFIG_SND_BT87X is not set
# CONFIG_SND_CA0106 is not set
# CONFIG_SND_CMIPCI is not set
# CONFIG_SND_OXYGEN is not set
# CONFIG_SND_CS4281 is not set
# CONFIG_SND_CS46XX is not set
# CONFIG_SND_CS5530 is not set
# CONFIG_SND_CTXFI is not set
# CONFIG_SND_DARLA20 is not set
# CONFIG_SND_GINA20 is not set
# CONFIG_SND_LAYLA20 is not set
# CONFIG_SND_DARLA24 is not set
# CONFIG_SND_GINA24 is not set
# CONFIG_SND_LAYLA24 is not set
# CONFIG_SND_MONA is not set
# CONFIG_SND_MIA is not set
# CONFIG_SND_ECHO3G is not set
# CONFIG_SND_INDIGO is not set
# CONFIG_SND_INDIGOIO is not set
# CONFIG_SND_INDIGODJ is not set
# CONFIG_SND_INDIGOIOX is not set
# CONFIG_SND_INDIGODJX is not set
CONFIG_SND_EMU10K1=m
# CONFIG_SND_EMU10K1X is not set
# CONFIG_SND_ENS1370 is not set
# CONFIG_SND_ENS1371 is not set
# CONFIG_SND_ES1938 is not set
# CONFIG_SND_ES1968 is not set
# CONFIG_SND_FM801 is not set
CONFIG_SND_HDA_INTEL=m
CONFIG_SND_HDA_HWDEP=y
# CONFIG_SND_HDA_RECONFIG is not set
# CONFIG_SND_HDA_INPUT_BEEP is not set
# CONFIG_SND_HDA_INPUT_JACK is not set
CONFIG_SND_HDA_CODEC_REALTEK=y
# CONFIG_SND_HDA_CODEC_ANALOG is not set
# CONFIG_SND_HDA_CODEC_SIGMATEL is not set
# CONFIG_SND_HDA_CODEC_VIA is not set
CONFIG_SND_HDA_CODEC_ATIHDMI=y
# CONFIG_SND_HDA_CODEC_NVHDMI is not set
# CONFIG_SND_HDA_CODEC_INTELHDMI is not set
# CONFIG_SND_HDA_CODEC_CONEXANT is not set
# CONFIG_SND_HDA_CODEC_CA0110 is not set
# CONFIG_SND_HDA_CODEC_CMEDIA is not set
# CONFIG_SND_HDA_CODEC_SI3054 is not set
CONFIG_SND_HDA_GENERIC=y
# CONFIG_SND_HDA_POWER_SAVE is not set
# CONFIG_SND_HDSP is not set
# CONFIG_SND_HDSPM is not set
# CONFIG_SND_HIFIER is not set
# CONFIG_SND_ICE1712 is not set
# CONFIG_SND_ICE1724 is not set
# CONFIG_SND_INTEL8X0 is not set
# CONFIG_SND_INTEL8X0M is not set
# CONFIG_SND_KORG1212 is not set
# CONFIG_SND_LX6464ES is not set
# CONFIG_SND_MAESTRO3 is not set
# CONFIG_SND_MIXART is not set
# CONFIG_SND_NM256 is not set
# CONFIG_SND_PCXHR is not set
# CONFIG_SND_RIPTIDE is not set
# CONFIG_SND_RME32 is not set
# CONFIG_SND_RME96 is not set
# CONFIG_SND_RME9652 is not set
# CONFIG_SND_SONICVIBES is not set
# CONFIG_SND_TRIDENT is not set
# CONFIG_SND_VIA82XX is not set
# CONFIG_SND_VIA82XX_MODEM is not set
# CONFIG_SND_VIRTUOSO is not set
# CONFIG_SND_VX222 is not set
# CONFIG_SND_YMFPCI is not set
# CONFIG_SND_SPI is not set
CONFIG_SND_USB=y
CONFIG_SND_USB_AUDIO=m
# CONFIG_SND_USB_USX2Y is not set
# CONFIG_SND_USB_CAIAQ is not set
# CONFIG_SND_USB_US122L is not set
# CONFIG_SND_SOC is not set
# CONFIG_SOUND_PRIME is not set
CONFIG_AC97_BUS=m
CONFIG_HID_SUPPORT=y
CONFIG_HID=y
CONFIG_HID_DEBUG=y
CONFIG_HIDRAW=y

#
# USB Input Devices
#
CONFIG_USB_HID=y
CONFIG_HID_PID=y
CONFIG_USB_HIDDEV=y

#
# Special HID drivers
#
CONFIG_HID_A4TECH=y
CONFIG_HID_APPLE=y
CONFIG_HID_BELKIN=y
CONFIG_HID_CHERRY=y
CONFIG_HID_CHICONY=y
CONFIG_HID_CYPRESS=y
# CONFIG_HID_DRAGONRISE is not set
CONFIG_HID_EZKEY=y
# CONFIG_HID_KYE is not set
CONFIG_HID_GYRATION=y
# CONFIG_HID_KENSINGTON is not set
CONFIG_HID_LOGITECH=y
CONFIG_LOGITECH_FF=y
# CONFIG_LOGIRUMBLEPAD2_FF is not set
CONFIG_HID_MICROSOFT=y
CONFIG_HID_MONTEREY=y
CONFIG_HID_NTRIG=y
CONFIG_HID_PANTHERLORD=y
CONFIG_PANTHERLORD_FF=y
CONFIG_HID_PETALYNX=y
CONFIG_HID_SAMSUNG=y
CONFIG_HID_SONY=y
CONFIG_HID_SUNPLUS=y
# CONFIG_HID_GREENASIA is not set
# CONFIG_HID_SMARTJOYPLUS is not set
CONFIG_HID_TOPSEED=y
# CONFIG_HID_THRUSTMASTER is not set
# CONFIG_HID_WACOM is not set
# CONFIG_HID_ZEROPLUS is not set
CONFIG_USB_SUPPORT=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_ARCH_HAS_EHCI=y
CONFIG_USB=y
CONFIG_USB_DEBUG=y
CONFIG_USB_ANNOUNCE_NEW_DEVICES=y

#
# Miscellaneous USB options
#
CONFIG_USB_DEVICEFS=y
# CONFIG_USB_DEVICE_CLASS is not set
# CONFIG_USB_DYNAMIC_MINORS is not set
CONFIG_USB_SUSPEND=y
# CONFIG_USB_OTG is not set
# CONFIG_USB_OTG_WHITELIST is not set
# CONFIG_USB_OTG_BLACKLIST_HUB is not set
CONFIG_USB_MON=y
# CONFIG_USB_WUSB is not set
# CONFIG_USB_WUSB_CBAF is not set

#
# USB Host Controller Drivers
#
# CONFIG_USB_C67X00_HCD is not set
# CONFIG_USB_XHCI_HCD is not set
CONFIG_USB_EHCI_HCD=m
# CONFIG_USB_EHCI_ROOT_HUB_TT is not set
# CONFIG_USB_EHCI_TT_NEWSCHED is not set
# CONFIG_USB_OXU210HP_HCD is not set
# CONFIG_USB_ISP116X_HCD is not set
# CONFIG_USB_ISP1760_HCD is not set
CONFIG_USB_OHCI_HCD=m
# CONFIG_USB_OHCI_BIG_ENDIAN_DESC is not set
# CONFIG_USB_OHCI_BIG_ENDIAN_MMIO is not set
CONFIG_USB_OHCI_LITTLE_ENDIAN=y
# CONFIG_USB_UHCI_HCD is not set
# CONFIG_USB_SL811_HCD is not set
# CONFIG_USB_R8A66597_HCD is not set
# CONFIG_USB_WHCI_HCD is not set
# CONFIG_USB_HWA_HCD is not set

#
# USB Device Class drivers
#
CONFIG_USB_ACM=m
CONFIG_USB_PRINTER=m
# CONFIG_USB_WDM is not set
# CONFIG_USB_TMC is not set

#
# NOTE: USB_STORAGE depends on SCSI but BLK_DEV_SD may
#

#
# also be needed; see USB_STORAGE Help for more info
#
CONFIG_USB_STORAGE=m
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_USBAT is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_STORAGE_ALAUDA is not set
# CONFIG_USB_STORAGE_ONETOUCH is not set
# CONFIG_USB_STORAGE_KARMA is not set
# CONFIG_USB_STORAGE_CYPRESS_ATACB is not set
# CONFIG_USB_LIBUSUAL is not set

#
# USB Imaging devices
#
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_MICROTEK is not set

#
# USB port drivers
#
# CONFIG_USB_SERIAL is not set

#
# USB Miscellaneous drivers
#
# CONFIG_USB_EMI62 is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_ADUTUX is not set
# CONFIG_USB_SEVSEG is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_LEGOTOWER is not set
# CONFIG_USB_LCD is not set
# CONFIG_USB_BERRY_CHARGE is not set
# CONFIG_USB_LED is not set
# CONFIG_USB_CYPRESS_CY7C63 is not set
# CONFIG_USB_CYTHERM is not set
# CONFIG_USB_IDMOUSE is not set
# CONFIG_USB_FTDI_ELAN is not set
# CONFIG_USB_APPLEDISPLAY is not set
# CONFIG_USB_SISUSBVGA is not set
# CONFIG_USB_LD is not set
# CONFIG_USB_TRANCEVIBRATOR is not set
# CONFIG_USB_IOWARRIOR is not set
# CONFIG_USB_TEST is not set
# CONFIG_USB_ISIGHTFW is not set
# CONFIG_USB_VST is not set
# CONFIG_USB_GADGET is not set

#
# OTG and related infrastructure
#
# CONFIG_NOP_USB_XCEIV is not set
# CONFIG_UWB is not set
# CONFIG_MMC is not set
# CONFIG_MEMSTICK is not set
CONFIG_NEW_LEDS=y
CONFIG_LEDS_CLASS=y

#
# LED drivers
#
# CONFIG_LEDS_ALIX2 is not set
# CONFIG_LEDS_PCA9532 is not set
# CONFIG_LEDS_LP3944 is not set
# CONFIG_LEDS_CLEVO_MAIL is not set
# CONFIG_LEDS_PCA955X is not set
# CONFIG_LEDS_DAC124S085 is not set
# CONFIG_LEDS_BD2802 is not set

#
# LED Triggers
#
CONFIG_LEDS_TRIGGERS=y
# CONFIG_LEDS_TRIGGER_TIMER is not set
# CONFIG_LEDS_TRIGGER_HEARTBEAT is not set
# CONFIG_LEDS_TRIGGER_BACKLIGHT is not set
# CONFIG_LEDS_TRIGGER_DEFAULT_ON is not set

#
# iptables trigger is under Netfilter config (LED target)
#
# CONFIG_ACCESSIBILITY is not set
# CONFIG_INFINIBAND is not set
# CONFIG_EDAC is not set
CONFIG_RTC_LIB=y
CONFIG_RTC_CLASS=y
# CONFIG_RTC_HCTOSYS is not set
# CONFIG_RTC_DEBUG is not set

#
# RTC interfaces
#
CONFIG_RTC_INTF_SYSFS=y
CONFIG_RTC_INTF_PROC=y
CONFIG_RTC_INTF_DEV=y
# CONFIG_RTC_INTF_DEV_UIE_EMUL is not set
# CONFIG_RTC_DRV_TEST is not set

#
# I2C RTC drivers
#
# CONFIG_RTC_DRV_DS1307 is not set
# CONFIG_RTC_DRV_DS1374 is not set
# CONFIG_RTC_DRV_DS1672 is not set
# CONFIG_RTC_DRV_MAX6900 is not set
# CONFIG_RTC_DRV_RS5C372 is not set
# CONFIG_RTC_DRV_ISL1208 is not set
# CONFIG_RTC_DRV_X1205 is not set
# CONFIG_RTC_DRV_PCF8563 is not set
# CONFIG_RTC_DRV_PCF8583 is not set
# CONFIG_RTC_DRV_M41T80 is not set
# CONFIG_RTC_DRV_S35390A is not set
# CONFIG_RTC_DRV_FM3130 is not set
# CONFIG_RTC_DRV_RX8581 is not set
# CONFIG_RTC_DRV_RX8025 is not set

#
# SPI RTC drivers
#
# CONFIG_RTC_DRV_M41T94 is not set
# CONFIG_RTC_DRV_DS1305 is not set
# CONFIG_RTC_DRV_DS1390 is not set
# CONFIG_RTC_DRV_MAX6902 is not set
# CONFIG_RTC_DRV_R9701 is not set
# CONFIG_RTC_DRV_RS5C348 is not set
# CONFIG_RTC_DRV_DS3234 is not set

#
# Platform RTC drivers
#
CONFIG_RTC_DRV_CMOS=y
# CONFIG_RTC_DRV_DS1286 is not set
# CONFIG_RTC_DRV_DS1511 is not set
# CONFIG_RTC_DRV_DS1553 is not set
# CONFIG_RTC_DRV_DS1742 is not set
# CONFIG_RTC_DRV_STK17TA8 is not set
# CONFIG_RTC_DRV_M48T86 is not set
# CONFIG_RTC_DRV_M48T35 is not set
# CONFIG_RTC_DRV_M48T59 is not set
# CONFIG_RTC_DRV_BQ4802 is not set
# CONFIG_RTC_DRV_V3020 is not set

#
# on-CPU RTC drivers
#
CONFIG_DMADEVICES=y

#
# DMA Devices
#
# CONFIG_INTEL_IOATDMA is not set
# CONFIG_AUXDISPLAY is not set
# CONFIG_UIO is not set

#
# TI VLYNQ
#
# CONFIG_STAGING is not set
CONFIG_X86_PLATFORM_DEVICES=y
# CONFIG_ACER_WMI is not set
# CONFIG_ACERHDF is not set
# CONFIG_ASUS_LAPTOP is not set
# CONFIG_FUJITSU_LAPTOP is not set
# CONFIG_MSI_LAPTOP is not set
# CONFIG_PANASONIC_LAPTOP is not set
# CONFIG_COMPAL_LAPTOP is not set
# CONFIG_SONY_LAPTOP is not set
# CONFIG_THINKPAD_ACPI is not set
# CONFIG_INTEL_MENLOW is not set
# CONFIG_ACPI_WMI is not set
# CONFIG_ACPI_ASUS is not set
# CONFIG_ACPI_TOSHIBA is not set

#
# Firmware Drivers
#
# CONFIG_EDD is not set
CONFIG_FIRMWARE_MEMMAP=y
# CONFIG_DELL_RBU is not set
# CONFIG_DCDBAS is not set
CONFIG_DMIID=y
# CONFIG_ISCSI_IBFT_FIND is not set

#
# File systems
#
CONFIG_EXT2_FS=y
# CONFIG_EXT2_FS_XATTR is not set
# CONFIG_EXT2_FS_XIP is not set
# CONFIG_EXT3_FS is not set
# CONFIG_EXT4_FS is not set
CONFIG_REISER4_FS=y
# CONFIG_REISER4_DEBUG is not set
CONFIG_REISERFS_FS=m
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_REISERFS_FS_XATTR is not set
# CONFIG_JFS_FS is not set
CONFIG_FS_POSIX_ACL=y
# CONFIG_XFS_FS is not set
# CONFIG_GFS2_FS is not set
# CONFIG_OCFS2_FS is not set
# CONFIG_BTRFS_FS is not set
CONFIG_FILE_LOCKING=y
CONFIG_FSNOTIFY=y
CONFIG_DNOTIFY=y
CONFIG_INOTIFY=y
CONFIG_INOTIFY_USER=y
# CONFIG_QUOTA is not set
# CONFIG_AUTOFS_FS is not set
# CONFIG_AUTOFS4_FS is not set
# CONFIG_FUSE_FS is not set
CONFIG_GENERIC_ACL=y

#
# Caches
#
# CONFIG_FSCACHE is not set

#
# CD-ROM/DVD Filesystems
#
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_UDF_FS=m
CONFIG_UDF_NLS=y

#
# DOS/FAT/NT Filesystems
#
CONFIG_FAT_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=m
CONFIG_FAT_DEFAULT_CODEPAGE=850
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-15"
CONFIG_NTFS_FS=m
# CONFIG_NTFS_DEBUG is not set
# CONFIG_NTFS_RW is not set

#
# Pseudo filesystems
#
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_PROC_SYSCTL=y
CONFIG_PROC_PAGE_MONITOR=y
CONFIG_SYSFS=y
CONFIG_TMPFS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_HUGETLBFS=y
CONFIG_HUGETLB_PAGE=y
# CONFIG_CONFIGFS_FS is not set
CONFIG_MISC_FILESYSTEMS=y
# CONFIG_ADFS_FS is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BFS_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_CRAMFS is not set
# CONFIG_SQUASHFS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_OMFS_FS is not set
# CONFIG_HPFS_FS is not set
# CONFIG_QNX4FS_FS is not set
# CONFIG_ROMFS_FS is not set
# CONFIG_SYSV_FS is not set
# CONFIG_UFS_FS is not set
# CONFIG_NILFS2_FS is not set
# CONFIG_NETWORK_FILESYSTEMS is not set

#
# Partition Types
#
# CONFIG_PARTITION_ADVANCED is not set
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="utf8"
CONFIG_NLS_CODEPAGE_437=m
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
CONFIG_NLS_CODEPAGE_850=m
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
CONFIG_NLS_ASCII=m
CONFIG_NLS_ISO8859_1=m
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
CONFIG_NLS_ISO8859_15=m
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
CONFIG_NLS_UTF8=m
# CONFIG_DLM is not set

#
# Kernel hacking
#
CONFIG_TRACE_IRQFLAGS_SUPPORT=y
CONFIG_PRINTK_TIME=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FRAME_WARN=2048
CONFIG_MAGIC_SYSRQ=y
# CONFIG_UNUSED_SYMBOLS is not set
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
# CONFIG_DEBUG_KERNEL is not set
CONFIG_SCHED_DEBUG=y
CONFIG_SCHEDSTATS=y
CONFIG_STACKTRACE=y
# CONFIG_DEBUG_BUGVERBOSE is not set
# CONFIG_DEBUG_MEMORY_INIT is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
CONFIG_FRAME_POINTER=y
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
CONFIG_LATENCYTOP=y
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
CONFIG_HAVE_FUNCTION_TRACE_MCOUNT_TEST=y
CONFIG_HAVE_DYNAMIC_FTRACE=y
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_FTRACE_SYSCALLS=y
CONFIG_TRACING_SUPPORT=y
# CONFIG_FTRACE is not set
# CONFIG_PROVIDE_OHCI1394_DMA_INIT is not set
# CONFIG_DYNAMIC_DEBUG is not set
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
CONFIG_EARLY_PRINTK=y
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_IOMMU_STRESS is not set
CONFIG_HAVE_MMIOTRACE_SUPPORT=y
CONFIG_IO_DELAY_TYPE_0X80=0
CONFIG_IO_DELAY_TYPE_0XED=1
CONFIG_IO_DELAY_TYPE_UDELAY=2
CONFIG_IO_DELAY_TYPE_NONE=3
# CONFIG_IO_DELAY_0X80 is not set
# CONFIG_IO_DELAY_0XED is not set
# CONFIG_IO_DELAY_UDELAY is not set
CONFIG_IO_DELAY_NONE=y
CONFIG_DEFAULT_IO_DELAY_TYPE=3
CONFIG_OPTIMIZE_INLINING=y

#
# Security options
#
# CONFIG_KEYS is not set
# CONFIG_SECURITY is not set
# CONFIG_SECURITYFS is not set
# CONFIG_SECURITY_FILE_CAPABILITIES is not set
# CONFIG_IMA is not set
CONFIG_XOR_BLOCKS=y
CONFIG_ASYNC_CORE=y
CONFIG_ASYNC_MEMCPY=y
CONFIG_ASYNC_XOR=y
CONFIG_CRYPTO=y

#
# Crypto core or helper
#
# CONFIG_CRYPTO_FIPS is not set
CONFIG_CRYPTO_ALGAPI=y
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=y
CONFIG_CRYPTO_AEAD2=y
CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
CONFIG_CRYPTO_RNG2=y
CONFIG_CRYPTO_PCOMP=y
CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_MANAGER2=y
# CONFIG_CRYPTO_GF128MUL is not set
# CONFIG_CRYPTO_NULL is not set
CONFIG_CRYPTO_WORKQUEUE=y
# CONFIG_CRYPTO_CRYPTD is not set
CONFIG_CRYPTO_AUTHENC=y
# CONFIG_CRYPTO_TEST is not set

#
# Authenticated Encryption with Associated Data
#
# CONFIG_CRYPTO_CCM is not set
# CONFIG_CRYPTO_GCM is not set
# CONFIG_CRYPTO_SEQIV is not set

#
# Block modes
#
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_CTR is not set
# CONFIG_CRYPTO_CTS is not set
CONFIG_CRYPTO_ECB=y
# CONFIG_CRYPTO_LRW is not set
# CONFIG_CRYPTO_PCBC is not set
# CONFIG_CRYPTO_XTS is not set

#
# Hash modes
#
CONFIG_CRYPTO_HMAC=y
# CONFIG_CRYPTO_XCBC is not set

#
# Digest
#
CONFIG_CRYPTO_CRC32C=m
# CONFIG_CRYPTO_CRC32C_INTEL is not set
# CONFIG_CRYPTO_MD4 is not set
CONFIG_CRYPTO_MD5=y
# CONFIG_CRYPTO_MICHAEL_MIC is not set
# CONFIG_CRYPTO_RMD128 is not set
# CONFIG_CRYPTO_RMD160 is not set
# CONFIG_CRYPTO_RMD256 is not set
# CONFIG_CRYPTO_RMD320 is not set
CONFIG_CRYPTO_SHA1=y
# CONFIG_CRYPTO_SHA256 is not set
# CONFIG_CRYPTO_SHA512 is not set
# CONFIG_CRYPTO_TGR192 is not set
# CONFIG_CRYPTO_WP512 is not set

#
# Ciphers
#
CONFIG_CRYPTO_AES=y
# CONFIG_CRYPTO_AES_X86_64 is not set
# CONFIG_CRYPTO_AES_NI_INTEL is not set
# CONFIG_CRYPTO_ANUBIS is not set
CONFIG_CRYPTO_ARC4=y
# CONFIG_CRYPTO_BLOWFISH is not set
# CONFIG_CRYPTO_CAMELLIA is not set
# CONFIG_CRYPTO_CAST5 is not set
# CONFIG_CRYPTO_CAST6 is not set
CONFIG_CRYPTO_DES=y
# CONFIG_CRYPTO_FCRYPT is not set
# CONFIG_CRYPTO_KHAZAD is not set
# CONFIG_CRYPTO_SALSA20 is not set
# CONFIG_CRYPTO_SALSA20_X86_64 is not set
# CONFIG_CRYPTO_SEED is not set
# CONFIG_CRYPTO_SERPENT is not set
# CONFIG_CRYPTO_TEA is not set
# CONFIG_CRYPTO_TWOFISH is not set
# CONFIG_CRYPTO_TWOFISH_X86_64 is not set

#
# Compression
#
# CONFIG_CRYPTO_DEFLATE is not set
# CONFIG_CRYPTO_ZLIB is not set
# CONFIG_CRYPTO_LZO is not set

#
# Random Number Generation
#
# CONFIG_CRYPTO_ANSI_CPRNG is not set
CONFIG_CRYPTO_HW=y
# CONFIG_CRYPTO_DEV_PADLOCK is not set
# CONFIG_CRYPTO_DEV_HIFN_795X is not set
CONFIG_HAVE_KVM=y
CONFIG_HAVE_KVM_IRQCHIP=y
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=m
# CONFIG_KVM_INTEL is not set
CONFIG_KVM_AMD=m
# CONFIG_KVM_TRACE is not set
# CONFIG_VIRTIO_PCI is not set
# CONFIG_VIRTIO_BALLOON is not set
# CONFIG_BINARY_PRINTF is not set

#
# Library routines
#
CONFIG_BITREVERSE=y
CONFIG_GENERIC_FIND_FIRST_BIT=y
CONFIG_GENERIC_FIND_NEXT_BIT=y
CONFIG_GENERIC_FIND_LAST_BIT=y
CONFIG_CRC_CCITT=m
CONFIG_CRC16=m
CONFIG_CRC_T10DIF=y
CONFIG_CRC_ITU_T=m
CONFIG_CRC32=y
# CONFIG_CRC7 is not set
CONFIG_LIBCRC32C=m
CONFIG_ZLIB_INFLATE=y
CONFIG_ZLIB_DEFLATE=y
CONFIG_LZO_COMPRESS=y
CONFIG_LZO_DECOMPRESS=y
CONFIG_HAS_IOMEM=y
CONFIG_HAS_IOPORT=y
CONFIG_HAS_DMA=y
CONFIG_NLATTR=y

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

(Volker stripped all CCs from his posts; I restored them manually.)

On 09/11/2009 09:33 PM, Volker Armin Hemmann wrote:
> Hi,
>
> this is with 2.6.31+reiser4+fglrx
> Phenom II X4 955
>
> KDE 4.3.1, composite temporary disabled.
> tvtime running.
>
> load:
> fat emerge with make -j5 running in one konsole tab (xulrunner being
> compiled).
>
> without NO_NEW_FAIR_SLEEPERS:
>
> tvtime is smooth most of the time
>
> with NO_NEW_FAIR_SLEEPERS:
>
> tvtime is more jerky. Very visible in scenes with movement.

Is the make -j5 running niced 0?  If yes, that would be actually the 
correct behavior.  Unfortunately, I can't test tvtime specifically (I 
don't have a TV card), but other applications displaying video continue 
to work smooth on my dual core machine (Core 2 Duo E6600) even if I do 
"nice -n 19 make -j20".  If I don't nice it, the video is skippy here 
too though.

Question to Ingo:
Would posting perf results help in any way with finding differences 
between mainline NEW_FAIR_SLEEPERS/NO_NEW_FAIR_SLEEPERS and BFS?


> without background load:
>
> both settings act the same. tvtime is smooth, video is smooth, games are nice.
> No real difference.
>
> config is attached.
>
> Glück Auf,
> Volker
>
> dmesg:
>
> [    0.000000] Linux version 2.6.31r4 (root@energy) (gcc version 4.4.1 (Gentoo
> 4.4.1 p1.0) ) #1 SMP Thu Sep 10 10:48:07 CEST 2009
> [    0.000000] Command line: root=/dev/md1 md=3,/dev/sda3,/dev/sdb3,/dev/sdc3
> nmi_watchdog=0 mtrr_spare_reg_nr=1
> [    0.000000] KERNEL supported cpus:
> [    0.000000]   AMD AuthenticAMD
> [    0.000000] BIOS-provided physical RAM map:
> [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
> [    0.000000]  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
> [    0.000000]  BIOS-e820: 00000000000e6000 - 0000000000100000 (reserved)
> [    0.000000]  BIOS-e820: 0000000000100000 - 00000000c7eb0000 (usable)
> [    0.000000]  BIOS-e820: 00000000c7eb0000 - 00000000c7ec0000 (ACPI data)
> [    0.000000]  BIOS-e820: 00000000c7ec0000 - 00000000c7ef0000 (ACPI NVS)
> [    0.000000]  BIOS-e820: 00000000c7ef0000 - 00000000c7f00000 (reserved)
> [    0.000000]  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved)
> [    0.000000]  BIOS-e820: 0000000100000000 - 0000000238000000 (usable)
> [    0.000000] DMI present.
> [    0.000000] AMI BIOS detected: BIOS may corrupt low RAM, working around it.
> [    0.000000] e820 update range: 0000000000000000 - 0000000000010000 (usable)
> ==>  (reserved)
> [    0.000000] last_pfn = 0x238000 max_arch_pfn = 0x400000000
> [    0.000000] MTRR default type: uncachable
> [    0.000000] MTRR fixed ranges enabled:
> [    0.000000]   00000-9FFFF write-back
> [    0.000000]   A0000-EFFFF uncachable
> [    0.000000]   F0000-FFFFF write-protect
> [    0.000000] MTRR variable ranges enabled:
> [    0.000000]   0 base 000000000000 mask FFFF80000000 write-back
> [    0.000000]   1 base 000080000000 mask FFFFC0000000 write-back
> [    0.000000]   2 base 0000C0000000 mask FFFFF8000000 write-back
> [    0.000000]   3 disabled
> [    0.000000]   4 disabled
> [    0.000000]   5 disabled
> [    0.000000]   6 disabled
> [    0.000000]   7 disabled
> [    0.000000] TOM2: 0000000238000000 aka 9088M
> [    0.000000] x86 PAT enabled: cpu 0, old 0x7040600070406, new
> 0x7010600070106
> [    0.000000] e820 update range: 00000000c8000000 - 0000000100000000 (usable)
> ==>  (reserved)
> [    0.000000] last_pfn = 0xc7eb0 max_arch_pfn = 0x400000000
> [    0.000000] Scanning 0 areas for low memory corruption
> [    0.000000] modified physical RAM map:
> [    0.000000]  modified: 0000000000000000 - 0000000000010000 (reserved)
> [    0.000000]  modified: 0000000000010000 - 000000000009fc00 (usable)
> [    0.000000]  modified: 000000000009fc00 - 00000000000a0000 (reserved)
> [    0.000000]  modified: 00000000000e6000 - 0000000000100000 (reserved)
> [    0.000000]  modified: 0000000000100000 - 00000000c7eb0000 (usable)
> [    0.000000]  modified: 00000000c7eb0000 - 00000000c7ec0000 (ACPI data)
> [    0.000000]  modified: 00000000c7ec0000 - 00000000c7ef0000 (ACPI NVS)
> [    0.000000]  modified: 00000000c7ef0000 - 00000000c7f00000 (reserved)
> [    0.000000]  modified: 00000000fff00000 - 0000000100000000 (reserved)
> [    0.000000]  modified: 0000000100000000 - 0000000238000000 (usable)
> [    0.000000] initial memory mapped : 0 - 20000000
> [    0.000000] Using GB pages for direct mapping
> [    0.000000] init_memory_mapping: 0000000000000000-00000000c7eb0000
> [    0.000000]  0000000000 - 00c0000000 page 1G
> [    0.000000]  00c0000000 - 00c7e00000 page 2M
> [    0.000000]  00c7e00000 - 00c7eb0000 page 4k
> [    0.000000] kernel direct mapping tables up to c7eb0000 @ 10000-13000
> [    0.000000] init_memory_mapping: 0000000100000000-0000000238000000
> [    0.000000]  0100000000 - 0200000000 page 1G
> [    0.000000]  0200000000 - 0238000000 page 2M
> [    0.000000] kernel direct mapping tables up to 238000000 @ 12000-14000
> [    0.000000] ACPI: RSDP 00000000000fa7c0 00014 (v00 ACPIAM)
> [    0.000000] ACPI: RSDT 00000000c7eb0000 00040 (v01 050609 RSDT2000 20090506
> MSFT 00000097)
> [    0.000000] ACPI: FACP 00000000c7eb0200 00084 (v02 A M I  OEMFACP  12000601
> MSFT 00000097)
> [    0.000000] ACPI: DSDT 00000000c7eb0440 08512 (v01  AS140 AS140121 00000121
> INTL 20051117)
> [    0.000000] ACPI: FACS 00000000c7ec0000 00040
> [    0.000000] ACPI: APIC 00000000c7eb0390 0006C (v01 050609 APIC2000 20090506
> MSFT 00000097)
> [    0.000000] ACPI: MCFG 00000000c7eb0400 0003C (v01 050609 OEMMCFG  20090506
> MSFT 00000097)
> [    0.000000] ACPI: OEMB 00000000c7ec0040 00071 (v01 050609 OEMB2000 20090506
> MSFT 00000097)
> [    0.000000] ACPI: AAFT 00000000c7eb8960 00027 (v01 050609 OEMAAFT  20090506
> MSFT 00000097)
> [    0.000000] ACPI: HPET 00000000c7eb8990 00038 (v01 050609 OEMHPET  20090506
> MSFT 00000097)
> [    0.000000] ACPI: SSDT 00000000c7eb89d0 0088C (v01 A M I  POWERNOW 00000001
> AMD  00000001)
> [    0.000000] ACPI: Local APIC address 0xfee00000
> [    0.000000] (7 early reservations) ==>  bootmem [0000000000 - 0238000000]
> [    0.000000]   #0 [0000000000 - 0000001000]   BIOS data page ==>  [0000000000
> - 0000001000]
> [    0.000000]   #1 [0000006000 - 0000008000]       TRAMPOLINE ==>  [0000006000
> - 0000008000]
> [    0.000000]   #2 [0001000000 - 00015fb8c0]    TEXT DATA BSS ==>  [0001000000
> - 00015fb8c0]
> [    0.000000]   #3 [000009fc00 - 0000100000]    BIOS reserved ==>  [000009fc00
> - 0000100000]
> [    0.000000]   #4 [00015fc000 - 00015fc133]              BRK ==>  [00015fc000
> - 00015fc133]
> [    0.000000]   #5 [0000010000 - 0000012000]          PGTABLE ==>  [0000010000
> - 0000012000]
> [    0.000000]   #6 [0000012000 - 0000013000]          PGTABLE ==>  [0000012000
> - 0000013000]
> [    0.000000]  [ffffea0000000000-ffffea0007dfffff] PMD ->  [ffff880028600000-
> ffff88002f7fffff] on node 0
> [    0.000000] Zone PFN ranges:
> [    0.000000]   DMA      0x00000010 ->  0x00001000
> [    0.000000]   DMA32    0x00001000 ->  0x00100000
> [    0.000000]   Normal   0x00100000 ->  0x00238000
> [    0.000000] Movable zone start PFN for each node
> [    0.000000] early_node_map[3] active PFN ranges
> [    0.000000]     0: 0x00000010 ->  0x0000009f
> [    0.000000]     0: 0x00000100 ->  0x000c7eb0
> [    0.000000]     0: 0x00100000 ->  0x00238000
> [    0.000000] On node 0 totalpages: 2096703
> [    0.000000]   DMA zone: 56 pages used for memmap
> [    0.000000]   DMA zone: 102 pages reserved
> [    0.000000]   DMA zone: 3825 pages, LIFO batch:0
> [    0.000000]   DMA32 zone: 14280 pages used for memmap
> [    0.000000]   DMA32 zone: 800488 pages, LIFO batch:31
> [    0.000000]   Normal zone: 17472 pages used for memmap
> [    0.000000]   Normal zone: 1260480 pages, LIFO batch:31
> [    0.000000] ACPI: PM-Timer IO Port: 0x808
> [    0.000000] ACPI: Local APIC address 0xfee00000
> [    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x02] enabled)
> [    0.000000] ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] enabled)
> [    0.000000] ACPI: IOAPIC (id[0x04] address[0xfec00000] gsi_base[0])
> [    0.000000] IOAPIC[0]: apic_id 4, version 33, address 0xfec00000, GSI 0-23
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
> [    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 low level)
> [    0.000000] ACPI: IRQ0 used by override.
> [    0.000000] ACPI: IRQ2 used by override.
> [    0.000000] ACPI: IRQ9 used by override.
> [    0.000000] Using ACPI (MADT) for SMP configuration information
> [    0.000000] ACPI: HPET id: 0x8300 base: 0xfed00000
> [    0.000000] SMP: Allowing 4 CPUs, 0 hotplug CPUs
> [    0.000000] nr_irqs_gsi: 24
> [    0.000000] PM: Registered nosave memory: 000000000009f000 -
> 00000000000a0000
> [    0.000000] PM: Registered nosave memory: 00000000000a0000 -
> 00000000000e6000
> [    0.000000] PM: Registered nosave memory: 00000000000e6000 -
> 0000000000100000
> [    0.000000] PM: Registered nosave memory: 00000000c7eb0000 -
> 00000000c7ec0000
> [    0.000000] PM: Registered nosave memory: 00000000c7ec0000 -
> 00000000c7ef0000
> [    0.000000] PM: Registered nosave memory: 00000000c7ef0000 -
> 00000000c7f00000
> [    0.000000] PM: Registered nosave memory: 00000000c7f00000 -
> 00000000fff00000
> [    0.000000] PM: Registered nosave memory: 00000000fff00000 -
> 0000000100000000
> [    0.000000] Allocating PCI resources starting at c7f00000 (gap:
> c7f00000:38000000)
> [    0.000000] NR_CPUS:4 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1
> [    0.000000] PERCPU: Embedded 25 pages at ffff880028034000, static data 72160
> bytes
> [    0.000000] Built 1 zonelists in Zone order, mobility grouping on.  Total
> pages: 2064793
> [    0.000000] Kernel command line: root=/dev/md1
> md=3,/dev/sda3,/dev/sdb3,/dev/sdc3 nmi_watchdog=0 mtrr_spare_reg_nr=1
> [    0.000000] md: Will configure md3 (super-block) from
> /dev/sda3,/dev/sdb3,/dev/sdc3, below.
> [    0.000000] PID hash table entries: 4096 (order: 12, 32768 bytes)
> [    0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608
> bytes)
> [    0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304
> bytes)
> [    0.000000] Initializing CPU#0
> [    0.000000] Checking aperture...
> [    0.000000] No AGP bridge found
> [    0.000000] Node 0: aperture @ 2a42000000 size 32 MB
> [    0.000000] Aperture beyond 4GB. Ignoring.
> [    0.000000] Your BIOS doesn't leave a aperture memory hole
> [    0.000000] Please enable the IOMMU option in the BIOS setup
> [    0.000000] This costs you 64 MB of RAM
> [    0.000000] Mapping aperture over 65536 KB of RAM @ 20000000
> [    0.000000] PM: Registered nosave memory: 0000000020000000 -
> 0000000024000000
> [    0.000000] Memory: 8184476k/9306112k available (3500k kernel code, 919300k
> absent, 201380k reserved, 1751k data, 376k init)
> [    0.000000] SLUB: Genslabs=13, HWalign=64, Order=0-3, MinObjects=0, CPUs=4,
> Nodes=1
> [    0.000000] Hierarchical RCU implementation.
> [    0.000000] NR_IRQS:4352 nr_irqs:440
> [    0.000000] Fast TSC calibration using PIT
> [    0.000000] Detected 3200.214 MHz processor.
> [    0.000609] Console: colour VGA+ 80x25
> [    0.000611] console [tty0] enabled
> [    0.003333] hpet clockevent registered
> [    0.003333]   alloc irq_desc for 24 on node 0
> [    0.003333]   alloc kstat_irqs on node 0
> [    0.003333] HPET: 4 timers in total, 1 timers will be used for per-cpu
> timer
> [    0.003339] Calibrating delay loop (skipped), value calculated using timer
> frequency.. 6402.10 BogoMIPS (lpj=10667366)
> [    0.003421] Mount-cache hash table entries: 256
> [    0.003543] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64
> bytes/line)
> [    0.003578] CPU: L2 Cache: 512K (64 bytes/line)
> [    0.003613] tseg: 0000000000
> [    0.003618] CPU: Physical Processor ID: 0
> [    0.003652] CPU: Processor Core ID: 0
> [    0.003686] mce: CPU supports 6 MCE banks
> [    0.003725] using C1E aware idle routine
> [    0.003768] ACPI: Core revision 20090521
> [    0.016704] Setting APIC routing to flat
> [    0.017026] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
> [    0.050860] CPU0: AMD Phenom(tm) II X4 955 Processor stepping 02
> [    0.053333] Booting processor 1 APIC 0x1 ip 0x6000
> [    0.003333] Initializing CPU#1
> [    0.003333] Calibrating delay using timer specific routine.. 6402.85
> BogoMIPS (lpj=10666966)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64
> bytes/line)
> [    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
> [    0.003333] CPU: Physical Processor ID: 0
> [    0.003333] CPU: Processor Core ID: 1
> [    0.003333] mce: CPU supports 6 MCE banks
> [    0.003333] x86 PAT enabled: cpu 1, old 0x7040600070406, new
> 0x7010600070106
> [    0.144161] CPU1: AMD Phenom(tm) II X4 955 Processor stepping 02
> [    0.144507] checking TSC synchronization [CPU#0 ->  CPU#1]: passed.
> [    0.146699] Booting processor 2 APIC 0x2 ip 0x6000
> [    0.003333] Initializing CPU#2
> [    0.003333] Calibrating delay using timer specific routine.. 6402.85
> BogoMIPS (lpj=10666970)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64
> bytes/line)
> [    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
> [    0.003333] CPU: Physical Processor ID: 0
> [    0.003333] CPU: Processor Core ID: 2
> [    0.003333] mce: CPU supports 6 MCE banks
> [    0.003333] x86 PAT enabled: cpu 2, old 0x7040600070406, new
> 0x7010600070106
> [    0.240822] CPU2: AMD Phenom(tm) II X4 955 Processor stepping 02
> [    0.241168] checking TSC synchronization [CPU#0 ->  CPU#2]: passed.
> [    0.243373] Booting processor 3 APIC 0x3 ip 0x6000
> [    0.003333] Initializing CPU#3
> [    0.003333] Calibrating delay using timer specific routine.. 6402.85
> BogoMIPS (lpj=10666972)
> [    0.003333] CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64
> bytes/line)
> [    0.003333] CPU: L2 Cache: 512K (64 bytes/line)
> [    0.003333] CPU: Physical Processor ID: 0
> [    0.003333] CPU: Processor Core ID: 3
> [    0.003333] mce: CPU supports 6 MCE banks
> [    0.003333] x86 PAT enabled: cpu 3, old 0x7040600070406, new
> 0x7010600070106
> [    0.337491] CPU3: AMD Phenom(tm) II X4 955 Processor stepping 02
> [    0.337836] checking TSC synchronization [CPU#0 ->  CPU#3]: passed.
> [    0.340006] Brought up 4 CPUs
> [    0.340040] Total of 4 processors activated (25611.67 BogoMIPS).
> [    0.340109] CPU0 attaching sched-domain:
> [    0.340111]  domain 0: span 0-3 level MC
> [    0.340112]   groups: 0 1 2 3
> [    0.340116] CPU1 attaching sched-domain:
> [    0.340117]  domain 0: span 0-3 level MC
> [    0.340118]   groups: 1 2 3 0
> [    0.340120] CPU2 attaching sched-domain:
> [    0.340121]  domain 0: span 0-3 level MC
> [    0.340122]   groups: 2 3 0 1
> [    0.340125] CPU3 attaching sched-domain:
> [    0.340126]  domain 0: span 0-3 level MC
> [    0.340127]   groups: 3 0 1 2
> [    0.340162] xor: automatically using best checksumming function:
> generic_sse
> [    0.356667]    generic_sse: 12835.200 MB/sec
> [    0.356700] xor: using function: generic_sse (12835.200 MB/sec)
> [    0.356754] Time:  7:07:42  Date: 09/11/09
> [    0.356802] NET: Registered protocol family 16
> [    0.356851] node 0 link 0: io port [1000, ffffff]
> [    0.356851] TOM: 00000000c8000000 aka 3200M
> [    0.356851] Fam 10h mmconf [e0000000, efffffff]
> [    0.356851] node 0 link 0: mmio [e0000000, efffffff] ==>  none
> [    0.356851] node 0 link 0: mmio [f0000000, ffffffff]
> [    0.356851] node 0 link 0: mmio [a0000, bffff]
> [    0.356851] node 0 link 0: mmio [c8000000, dfffffff]
> [    0.356851] TOM2: 0000000238000000 aka 9088M
> [    0.356851] bus: [00,07] on node 0 link 0
> [    0.356851] bus: 00 index 0 io port: [0, ffff]
> [    0.356851] bus: 00 index 1 mmio: [f0000000, ffffffff]
> [    0.356851] bus: 00 index 2 mmio: [a0000, bffff]
> [    0.356851] bus: 00 index 3 mmio: [c8000000, dfffffff]
> [    0.356851] bus: 00 index 4 mmio: [238000000, fcffffffff]
> [    0.356851] ACPI: bus type pci registered
> [    0.356851] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> [    0.356851] PCI: Not using MMCONFIG.
> [    0.356851] PCI: Using configuration type 1 for base access
> [    0.356851] PCI: Using configuration type 1 for extended access
> [    0.356851] bio: create slab<bio-0>  at 0
> [    0.356978] ACPI: EC: Look up EC in DSDT
> [    0.367031] ACPI: Interpreter enabled
> [    0.367538] ACPI: (supports S0 S1 S3 S4 S5)
> [    0.367652] ACPI: Using IOAPIC for interrupt routing
> [    0.367725] PCI: MCFG configuration 0: base e0000000 segment 0 buses 0 - 255
> [    0.370531] PCI: MCFG area at e0000000 reserved in ACPI motherboard
> resources
> [    0.375947] PCI: Using MMCONFIG at e0000000 - efffffff
> [    0.380367] ACPI: No dock devices found.
> [    0.380451] ACPI: PCI Root Bridge [PCI0] (0000:00)
> [    0.380520] pci 0000:00:00.0: reg 1c 64bit mmio: [0xe0000000-0xffffffff]
> [    0.380520] pci 0000:00:02.0: PME# supported from D0 D3hot D3cold
> [    0.380520] pci 0000:00:02.0: PME# disabled
> [    0.380520] pci 0000:00:09.0: PME# supported from D0 D3hot D3cold
> [    0.380520] pci 0000:00:09.0: PME# disabled
> [    0.380520] pci 0000:00:0a.0: PME# supported from D0 D3hot D3cold
> [    0.380520] pci 0000:00:0a.0: PME# disabled
> [    0.380520] pci 0000:00:11.0: reg 10 io port: [0xa000-0xa007]
> [    0.380520] pci 0000:00:11.0: reg 14 io port: [0x9000-0x9003]
> [    0.380520] pci 0000:00:11.0: reg 18 io port: [0x8000-0x8007]
> [    0.380520] pci 0000:00:11.0: reg 1c io port: [0x7000-0x7003]
> [    0.380520] pci 0000:00:11.0: reg 20 io port: [0x6000-0x600f]
> [    0.380520] pci 0000:00:11.0: reg 24 32bit mmio: [0xfddff800-0xfddffbff]
> [    0.380520] pci 0000:00:12.0: reg 10 32bit mmio: [0xfddfe000-0xfddfefff]
> [    0.380520] pci 0000:00:12.1: reg 10 32bit mmio: [0xfddfd000-0xfddfdfff]
> [    0.380576] pci 0000:00:12.2: reg 10 32bit mmio: [0xfddff000-0xfddff0ff]
> [    0.380625] pci 0000:00:12.2: supports D1 D2
> [    0.380626] pci 0000:00:12.2: PME# supported from D0 D1 D2 D3hot
> [    0.380663] pci 0000:00:12.2: PME# disabled
> [    0.380724] pci 0000:00:13.0: reg 10 32bit mmio: [0xfddfc000-0xfddfcfff]
> [    0.380775] pci 0000:00:13.1: reg 10 32bit mmio: [0xfddf7000-0xfddf7fff]
> [    0.380843] pci 0000:00:13.2: reg 10 32bit mmio: [0xfddf6800-0xfddf68ff]
> [    0.380893] pci 0000:00:13.2: supports D1 D2
> [    0.380894] pci 0000:00:13.2: PME# supported from D0 D1 D2 D3hot
> [    0.380930] pci 0000:00:13.2: PME# disabled
> [    0.381072] pci 0000:00:14.1: reg 10 io port: [0x00-0x07]
> [    0.381078] pci 0000:00:14.1: reg 14 io port: [0x00-0x03]
> [    0.381084] pci 0000:00:14.1: reg 18 io port: [0x00-0x07]
> [    0.381089] pci 0000:00:14.1: reg 1c io port: [0x00-0x03]
> [    0.381095] pci 0000:00:14.1: reg 20 io port: [0xff00-0xff0f]
> [    0.381219] pci 0000:00:14.5: reg 10 32bit mmio: [0xfddf5000-0xfddf5fff]
> [    0.381353] pci 0000:02:00.0: reg 10 64bit mmio: [0xd0000000-0xdfffffff]
> [    0.381360] pci 0000:02:00.0: reg 18 64bit mmio: [0xfdff0000-0xfdffffff]
> [    0.381365] pci 0000:02:00.0: reg 20 io port: [0xc000-0xc0ff]
> [    0.381372] pci 0000:02:00.0: reg 30 32bit mmio: [0xfdfc0000-0xfdfdffff]
> [    0.381387] pci 0000:02:00.0: supports D1 D2
> [    0.381415] pci 0000:02:00.1: reg 10 64bit mmio: [0xfdfec000-0xfdfeffff]
> [    0.381445] pci 0000:02:00.1: supports D1 D2
> [    0.381495] pci 0000:00:02.0: bridge io port: [0xc000-0xcfff]
> [    0.381497] pci 0000:00:02.0: bridge 32bit mmio: [0xfdf00000-0xfdffffff]
> [    0.381500] pci 0000:00:02.0: bridge 64bit mmio pref: [0xd0000000-0xdfffffff]
> [    0.381543] pci 0000:00:09.0: bridge io port: [0xd000-0xdfff]
> [    0.381545] pci 0000:00:09.0: bridge 32bit mmio: [0xfe000000-0xfebfffff]
> [    0.381548] pci 0000:00:09.0: bridge 64bit mmio pref: [0xfa000000-0xfcefffff]
> [    0.383347] pci 0000:01:00.0: reg 10 io port: [0xb800-0xb8ff]
> [    0.383360] pci 0000:01:00.0: reg 18 64bit mmio: [0xcffff000-0xcfffffff]
> [    0.383370] pci 0000:01:00.0: reg 20 64bit mmio: [0xcffe0000-0xcffeffff]
> [    0.383375] pci 0000:01:00.0: reg 30 32bit mmio: [0xfdef0000-0xfdefffff]
> [    0.383402] pci 0000:01:00.0: supports D1 D2
> [    0.383403] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
> [    0.383440] pci 0000:01:00.0: PME# disabled
> [    0.383525] pci 0000:00:0a.0: bridge io port: [0xb000-0xbfff]
> [    0.383527] pci 0000:00:0a.0: bridge 32bit mmio: [0xfde00000-0xfdefffff]
> [    0.383530] pci 0000:00:0a.0: bridge 64bit mmio pref: [0xcff00000-0xcfffffff]
> [    0.383565] pci 0000:05:06.0: reg 10 32bit mmio: [0xfcfff000-0xfcffffff]
> [    0.383644] pci 0000:05:08.0: reg 10 io port: [0xe800-0xe83f]
> [    0.383701] pci 0000:05:08.0: supports D1 D2
> [    0.383743] pci 0000:00:14.4: transparent bridge
> [    0.383779] pci 0000:00:14.4: bridge io port: [0xe000-0xefff]
> [    0.383785] pci 0000:00:14.4: bridge 32bit mmio pref: [0xfcf00000-0xfcffffff]
> [    0.383797] pci_bus 0000:00: on NUMA node 0
> [    0.383800] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT]
> [    0.383936] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE2._PRT]
> [    0.383981] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCEA._PRT]
> [    0.384025] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.P0PC._PRT]
> [    0.384091] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCE9._PRT]
> [    0.386726] ACPI: PCI Interrupt Link [LNKA] (IRQs 4 *7 10 11 12 14 15)
> [    0.386953] ACPI: PCI Interrupt Link [LNKB] (IRQs 4 7 10 *11 12 14 15)
> [    0.387183] ACPI: PCI Interrupt Link [LNKC] (IRQs 4 7 *10 11 12 14 15)
> [    0.387408] ACPI: PCI Interrupt Link [LNKD] (IRQs 4 7 *10 11 12 14 15)
> [    0.387643] ACPI: PCI Interrupt Link [LNKE] (IRQs 4 7 10 11 12 14 15) *0,
> disabled.
> [    0.387914] ACPI: PCI Interrupt Link [LNKF] (IRQs 4 7 10 *11 12 14 15)
> [    0.388140] ACPI: PCI Interrupt Link [LNKG] (IRQs *4 10 11 12 14 15)
> [    0.388352] ACPI: PCI Interrupt Link [LNKH] (IRQs 4 7 *10 11 12 14 15)
> [    0.388538] SCSI subsystem initialized
> [    0.388538] libata version 3.00 loaded.
> [    0.388538] usbcore: registered new interface driver usbfs
> [    0.388538] usbcore: registered new interface driver hub
> [    0.388538] usbcore: registered new device driver usb
> [    0.443347] raid6: int64x1   2755 MB/s
> [    0.500010] raid6: int64x2   3858 MB/s
> [    0.556669] raid6: int64x4   2850 MB/s
> [    0.613353] raid6: int64x8   2537 MB/s
> [    0.670007] raid6: sse2x1    3999 MB/s
> [    0.726666] raid6: sse2x2    7012 MB/s
> [    0.783343] raid6: sse2x4    7975 MB/s
> [    0.783377] raid6: using algorithm sse2x4 (7975 MB/s)
> [    0.783423] PCI: Using ACPI for IRQ routing
> [    0.783423] pci 0000:00:00.0: BAR 3: address space collision on of device
> [0xe0000000-0xffffffff]
> [    0.783425] pci 0000:00:00.0: BAR 3: can't allocate resource
> [    0.793433] PCI-DMA: Disabling AGP.
> [    0.793518] PCI-DMA: aperture base @ 20000000 size 65536 KB
> [    0.793518] PCI-DMA: using GART IOMMU.
> [    0.793518] PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> [    0.795205] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 24, 0
> [    0.795312] hpet0: 4 comparators, 32-bit 14.318180 MHz counter
> [    0.800030] hpet: hpet2 irq 24 for MSI
> [    0.810015] Switched to high resolution mode on CPU 0
> [    0.811193] Switched to high resolution mode on CPU 2
> [    0.811196] Switched to high resolution mode on CPU 1
> [    0.811200] Switched to high resolution mode on CPU 3
> [    0.820035] pnp: PnP ACPI init
> [    0.820085] ACPI: bus type pnp registered
> [    0.822351] pnp 00:0b: mem resource (0x0-0x9ffff) overlaps 0000:00:00.0 BAR 3
> (0x0-0x1fffffff), disabling
> [    0.822406] pnp 00:0b: mem resource (0xc0000-0xcffff) overlaps 0000:00:00.0
> BAR 3 (0x0-0x1fffffff), disabling
> [    0.822461] pnp 00:0b: mem resource (0xe0000-0xfffff) overlaps 0000:00:00.0
> BAR 3 (0x0-0x1fffffff), disabling
> [    0.822515] pnp 00:0b: mem resource (0x100000-0xc7efffff) overlaps
> 0000:00:00.0 BAR 3 (0x0-0x1fffffff), disabling
> [    0.822837] pnp: PnP ACPI: found 12 devices
> [    0.822871] ACPI: ACPI bus type pnp unregistered
> [    0.822911] system 00:06: iomem range 0xfec00000-0xfec00fff could not be
> reserved
> [    0.822965] system 00:06: iomem range 0xfee00000-0xfee00fff has been
> reserved
> [    0.823002] system 00:07: ioport range 0x4d0-0x4d1 has been reserved
> [    0.823037] system 00:07: ioport range 0x40b-0x40b has been reserved
> [    0.823072] system 00:07: ioport range 0x4d6-0x4d6 has been reserved
> [    0.823106] system 00:07: ioport range 0xc00-0xc01 has been reserved
> [    0.823141] system 00:07: ioport range 0xc14-0xc14 has been reserved
> [    0.823176] system 00:07: ioport range 0xc50-0xc51 has been reserved
> [    0.823211] system 00:07: ioport range 0xc52-0xc52 has been reserved
> [    0.823245] system 00:07: ioport range 0xc6c-0xc6c has been reserved
> [    0.823280] system 00:07: ioport range 0xc6f-0xc6f has been reserved
> [    0.823315] system 00:07: ioport range 0xcd0-0xcd1 has been reserved
> [    0.823359] system 00:07: ioport range 0xcd2-0xcd3 has been reserved
> [    0.823394] system 00:07: ioport range 0xcd4-0xcd5 has been reserved
> [    0.823429] system 00:07: ioport range 0xcd6-0xcd7 has been reserved
> [    0.823464] system 00:07: ioport range 0xcd8-0xcdf has been reserved
> [    0.823499] system 00:07: ioport range 0x800-0x89f has been reserved
> [    0.823533] system 00:07: ioport range 0xb00-0xb0f has been reserved
> [    0.823568] system 00:07: ioport range 0xb20-0xb3f has been reserved
> [    0.823603] system 00:07: ioport range 0x900-0x90f has been reserved
> [    0.823638] system 00:07: ioport range 0x910-0x91f has been reserved
> [    0.823673] system 00:07: ioport range 0xfe00-0xfefe has been reserved
> [    0.823708] system 00:07: iomem range 0xffb80000-0xffbfffff has been reserved
> [    0.823743] system 00:07: iomem range 0xfec10000-0xfec1001f has been
> reserved
> [    0.823780] system 00:09: ioport range 0x290-0x29f has been reserved
> [    0.823816] system 00:0a: iomem range 0xe0000000-0xefffffff has been reserved
> [    0.823852] system 00:0b: iomem range 0xfec00000-0xffffffff could not be
> reserved
> [    0.828757] pci 0000:00:02.0: PCI bridge, secondary bus 0000:02
> [    0.828792] pci 0000:00:02.0:   IO window: 0xc000-0xcfff
> [    0.828828] pci 0000:00:02.0:   MEM window: 0xfdf00000-0xfdffffff
> [    0.828863] pci 0000:00:02.0:   PREFETCH window:
> 0x000000d0000000-0x000000dfffffff
> [    0.828918] pci 0000:00:09.0: PCI bridge, secondary bus 0000:03
> [    0.828953] pci 0000:00:09.0:   IO window: 0xd000-0xdfff
> [    0.828988] pci 0000:00:09.0:   MEM window: 0xfe000000-0xfebfffff
> [    0.829023] pci 0000:00:09.0:   PREFETCH window:
> 0x000000fa000000-0x000000fcefffff
> [    0.829078] pci 0000:00:0a.0: PCI bridge, secondary bus 0000:01
> [    0.829112] pci 0000:00:0a.0:   IO window: 0xb000-0xbfff
> [    0.829148] pci 0000:00:0a.0:   MEM window: 0xfde00000-0xfdefffff
> [    0.829183] pci 0000:00:0a.0:   PREFETCH window:
> 0x000000cff00000-0x000000cfffffff
> [    0.829237] pci 0000:00:14.4: PCI bridge, secondary bus 0000:05
> [    0.829273] pci 0000:00:14.4:   IO window: 0xe000-0xefff
> [    0.829310] pci 0000:00:14.4:   MEM window: disabled
> [    0.829346] pci 0000:00:14.4:   PREFETCH window: 0xfcf00000-0xfcffffff
> [    0.829387]   alloc irq_desc for 18 on node -1
> [    0.829388]   alloc kstat_irqs on node -1
> [    0.829392] pci 0000:00:02.0: PCI INT A ->  GSI 18 (level, low) ->  IRQ 18
> [    0.829428] pci 0000:00:02.0: setting latency timer to 64
> [    0.829432]   alloc irq_desc for 17 on node -1
> [    0.829433]   alloc kstat_irqs on node -1
> [    0.829435] pci 0000:00:09.0: PCI INT A ->  GSI 17 (level, low) ->  IRQ 17
> [    0.829471] pci 0000:00:09.0: setting latency timer to 64
> [    0.829474] pci 0000:00:0a.0: PCI INT A ->  GSI 18 (level, low) ->  IRQ 18
> [    0.829509] pci 0000:00:0a.0: setting latency timer to 64
> [    0.829516] pci_bus 0000:00: resource 0 io:  [0x00-0xffff]
> [    0.829517] pci_bus 0000:00: resource 1 mem: [0x000000-0xffffffffffffffff]
> [    0.829519] pci_bus 0000:02: resource 0 io:  [0xc000-0xcfff]
> [    0.829521] pci_bus 0000:02: resource 1 mem: [0xfdf00000-0xfdffffff]
> [    0.829522] pci_bus 0000:02: resource 2 pref mem [0xd0000000-0xdfffffff]
> [    0.829523] pci_bus 0000:03: resource 0 io:  [0xd000-0xdfff]
> [    0.829525] pci_bus 0000:03: resource 1 mem: [0xfe000000-0xfebfffff]
> [    0.829526] pci_bus 0000:03: resource 2 pref mem [0xfa000000-0xfcefffff]
> [    0.829527] pci_bus 0000:01: resource 0 io:  [0xb000-0xbfff]
> [    0.829529] pci_bus 0000:01: resource 1 mem: [0xfde00000-0xfdefffff]
> [    0.829530] pci_bus 0000:01: resource 2 pref mem [0xcff00000-0xcfffffff]
> [    0.829531] pci_bus 0000:05: resource 0 io:  [0xe000-0xefff]
> [    0.829533] pci_bus 0000:05: resource 2 pref mem [0xfcf00000-0xfcffffff]
> [    0.829534] pci_bus 0000:05: resource 3 io:  [0x00-0xffff]
> [    0.829535] pci_bus 0000:05: resource 4 mem: [0x000000-0xffffffffffffffff]
> [    0.829547] NET: Registered protocol family 2
> [    0.829602] IP route cache hash table entries: 262144 (order: 9, 2097152
> bytes)
> [    0.830084] TCP established hash table entries: 262144 (order: 10, 4194304
> bytes)
> [    0.831067] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
> [    0.831468] TCP: Hash tables configured (established 262144 bind 65536)
> [    0.831504] TCP reno registered
> [    0.831582] NET: Registered protocol family 1
> [    0.832803] Scanning for low memory corruption every 60 seconds
> [    0.833572] HugeTLB registered 2 MB page size, pre-allocated 0 pages
> [    0.833696] Loading Reiser4. See www.namesys.com for a description of
> Reiser4.
> [    0.833781] msgmni has been set to 15987
> [    0.834082] alg: No test for stdrng (krng)
> [    0.834123] async_tx: api initialized (sync-only)
> [    0.834227] Block layer SCSI generic (bsg) driver version 0.4 loaded (major
> 253)
> [    0.834280] io scheduler noop registered
> [    0.834315] io scheduler cfq registered (default)
> [    0.834448] pci 0000:02:00.0: Boot video device
> [    0.834538]   alloc irq_desc for 25 on node -1
> [    0.834540]   alloc kstat_irqs on node -1
> [    0.834545] pcieport-driver 0000:00:02.0: irq 25 for MSI/MSI-X
> [    0.834550] pcieport-driver 0000:00:02.0: setting latency timer to 64
> [    0.834642]   alloc irq_desc for 26 on node -1
> [    0.834643]   alloc kstat_irqs on node -1
> [    0.834646] pcieport-driver 0000:00:09.0: irq 26 for MSI/MSI-X
> [    0.834650] pcieport-driver 0000:00:09.0: setting latency timer to 64
> [    0.834741]   alloc irq_desc for 27 on node -1
> [    0.834742]   alloc kstat_irqs on node -1
> [    0.834744] pcieport-driver 0000:00:0a.0: irq 27 for MSI/MSI-X
> [    0.834748] pcieport-driver 0000:00:0a.0: setting latency timer to 64
> [    0.834953] input: Power Button as
> /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
> [    0.835007] ACPI: Power Button [PWRF]
> [    0.835098] input: Power Button as
> /devices/LNXSYSTM:00/device:00/PNP0C0C:00/input/input1
> [    0.835152] ACPI: Power Button [PWRB]
> [    0.835302] processor LNXCPU:00: registered as cooling_device0
> [    0.835337] ACPI: Processor [CPU0] (supports 8 throttling states)
> [    0.835434] processor LNXCPU:01: registered as cooling_device1
> [    0.835504] processor LNXCPU:02: registered as cooling_device2
> [    0.835577] processor LNXCPU:03: registered as cooling_device3
> [    0.839315] Linux agpgart interface v0.103
> [    0.839511] ahci 0000:00:11.0: version 3.0
> [    0.839521]   alloc irq_desc for 22 on node -1
> [    0.839522]   alloc kstat_irqs on node -1
> [    0.839525] ahci 0000:00:11.0: PCI INT A ->  GSI 22 (level, low) ->  IRQ 22
> [    0.839673] ahci 0000:00:11.0: AHCI 0001.0100 32 slots 6 ports 3 Gbps 0x3f
> impl SATA mode
> [    0.839727] ahci 0000:00:11.0: flags: 64bit ncq sntf ilck pm led clo pmp pio
> slum part
> [    0.840272] scsi0 : ahci
> [    0.840403] scsi1 : ahci
> [    0.840501] scsi2 : ahci
> [    0.840598] scsi3 : ahci
> [    0.840697] scsi4 : ahci
> [    0.840795] scsi5 : ahci
> [    0.840926] ata1: SATA max UDMA/133 irq_stat 0x00400000, PHY RDY changed
> [    0.840962] ata2: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddff980 irq
> 22
> [    0.841016] ata3: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffa00 irq
> 22
> [    0.841070] ata4: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffa80 irq
> 22
> [    0.841124] ata5: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffb00 irq
> 22
> [    0.841178] ata6: SATA max UDMA/133 abar m1024@0xfddff800 port 0xfddffb80 irq
> 22
> [    0.841459] PNP: PS/2 Controller [PNP0303:PS2K] at 0x60,0x64 irq 1
> [    0.841493] PNP: PS/2 appears to have AUX port disabled, if this is
> incorrect please boot with i8042.nopnp
> [    0.841943] serio: i8042 KBD port at 0x60,0x64 irq 1
> [    0.842074] mice: PS/2 mouse device common for all mice
> [    0.842243] rtc_cmos 00:02: RTC can wake from S4
> [    0.842313] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
> [    0.842369] rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
> [    0.842428] md: raid1 personality registered for level 1
> [    0.842462] md: raid6 personality registered for level 6
> [    0.842496] md: raid5 personality registered for level 5
> [    0.842530] md: raid4 personality registered for level 4
> [    0.843117] cpuidle: using governor ladder
> [    0.843151] cpuidle: using governor menu
> [    0.843689] usbcore: registered new interface driver hiddev
> [    0.843746] usbcore: registered new interface driver usbhid
> [    0.843780] usbhid: v2.6:USB HID core driver
> [    0.843843] Advanced Linux Sound Architecture Driver Version 1.0.20.
> [    0.843878] ALSA device list:
> [    0.843911]   No soundcards found.
> [    0.843979] TCP cubic registered
> [    0.844019] NET: Registered protocol family 10
> [    0.844148] IPv6 over IPv4 tunneling driver
> [    0.844265] NET: Registered protocol family 17
> [    0.844315] powernow-k8: Found 1 AMD Phenom(tm) II X4 955 Processor
> processors (4 cpu cores) (version 2.20.00)
> [    0.844391] powernow-k8:    0 : pstate 0 (3200 MHz)
> [    0.844425] powernow-k8:    1 : pstate 1 (2500 MHz)
> [    0.844459] powernow-k8:    2 : pstate 2 (2100 MHz)
> [    0.844492] powernow-k8:    3 : pstate 3 (800 MHz)
> [    0.844886] PM: Resume from disk failed.
> [    0.844966]   Magic number: 9:648:116
> [    0.866018] input: AT Translated Set 2 keyboard as
> /devices/platform/i8042/serio0/input/input2
> [    1.160036] ata4: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [    1.160097] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [    1.160150] ata6: SATA link down (SStatus 0 SControl 300)
> [    1.160205] ata5: SATA link down (SStatus 0 SControl 300)
> [    1.160259] ata3: SATA link down (SStatus 0 SControl 300)
> [    1.166391] ata4.00: ATA-7: SAMSUNG HD753LJ, 1AA01113, max UDMA7
> [    1.166432] ata4.00: 1465149168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> [    1.166480] ata2.00: ATA-7: SAMSUNG HD502IJ, 1AA01110, max UDMA7
> [    1.166514] ata2.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> [    1.172888] ata4.00: configured for UDMA/133
> [    1.172943] ata2.00: configured for UDMA/133
> [    1.560035] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
> [    1.566394] ata1.00: ATA-7: SAMSUNG HD502IJ, 1AA01109, max UDMA7
> [    1.566430] ata1.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32)
> [    1.572855] ata1.00: configured for UDMA/133
> [    1.583424] scsi 0:0:0:0: Direct-Access     ATA      SAMSUNG HD502IJ  1AA0
> PQ: 0 ANSI: 5
> [    1.583684] sd 0:0:0:0: [sda] 976773168 512-byte logical blocks: (500
> GB/465 GiB)
> [    1.583756] sd 0:0:0:0: [sda] Write Protect is off
> [    1.583791] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
> [    1.583800] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled,
> doesn't support DPO or FUA
> [    1.583911]  sda:
> [    1.583952] sd 0:0:0:0: Attached scsi generic sg0 type 0
> [    1.584094] scsi 1:0:0:0: Direct-Access     ATA      SAMSUNG HD502IJ  1AA0
> PQ: 0 ANSI: 5
> [    1.584283] sd 1:0:0:0: [sdb] 976773168 512-byte logical blocks: (500
> GB/465 GiB)
> [    1.584354] sd 1:0:0:0: [sdb] Write Protect is off
> [    1.584389] sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
> [    1.584398] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled,
> doesn't support DPO or FUA
> [    1.584507]  sdb:
> [    1.584577] sd 1:0:0:0: Attached scsi generic sg1 type 0
> [    1.584709] scsi 3:0:0:0: Direct-Access     ATA      SAMSUNG HD753LJ  1AA0
> PQ: 0 ANSI: 5
> [    1.584865] sd 3:0:0:0: [sdc] 1465149168 512-byte logical blocks: (750
> GB/698 GiB)
> [    1.584934] sd 3:0:0:0: [sdc] Write Protect is off
> [    1.584969] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
> [    1.584977] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled,
> doesn't support DPO or FUA
> [    1.585071]  sdc:
> [    1.585123] sd 3:0:0:0: Attached scsi generic sg2 type 0
> [    1.588354]  sdc1 sdc2 sdc3 sdc4<  sdb1 sdb2 sdb3 sdb4<  sda1 sda2 sda3
> sda4<  sdb5 sdc5 sda5 sdc6 sdb6>
> [    1.606850] sd 1:0:0:0: [sdb] Attached SCSI disk
> [    1.610449]  sda6>
> [    1.610814] sd 0:0:0:0: [sda] Attached SCSI disk
> [    1.617957]  sdc7>
> [    1.618341] sd 3:0:0:0: [sdc] Attached SCSI disk
> [    1.618382] md: Waiting for all devices to be available before autodetect
> [    1.618417] md: If you don't use raid, use raid=noautodetect
> [    1.618533] md: Autodetecting RAID arrays.
> [    1.724723] md: Scanned 12 and added 12 devices.
> [    1.724758] md: autorun ...
> [    1.724792] md: considering sdc6 ...
> [    1.724827] md:  adding sdc6 ...
> [    1.724862] md: sdc5 has different UUID to sdc6
> [    1.724897] md: sdc3 has different UUID to sdc6
> [    1.724931] md: sdc1 has different UUID to sdc6
> [    1.724967] md:  adding sda6 ...
> [    1.725001] md: sda5 has different UUID to sdc6
> [    1.725036] md: sda3 has different UUID to sdc6
> [    1.725070] md: sda1 has different UUID to sdc6
> [    1.725106] md:  adding sdb6 ...
> [    1.725140] md: sdb5 has different UUID to sdc6
> [    1.725175] md: sdb3 has different UUID to sdc6
> [    1.725209] md: sdb1 has different UUID to sdc6
> [    1.725349] md: created md3
> [    1.725382] md: bind<sdb6>
> [    1.725419] md: bind<sda6>
> [    1.725456] md: bind<sdc6>
> [    1.725492] md: running:<sdc6><sda6><sdb6>
> [    1.725626] raid5: device sdc6 operational as raid disk 2
> [    1.725661] raid5: device sda6 operational as raid disk 0
> [    1.725695] raid5: device sdb6 operational as raid disk 1
> [    1.725846] raid5: allocated 3220kB for md3
> [    1.725910] raid5: raid level 5 set md3 active with 3 out of 3 devices,
> algorithm 2
> [    1.725963] RAID5 conf printout:
> [    1.725996]  --- rd:3 wd:3
> [    1.726029]  disk 0, o:1, dev:sda6
> [    1.726062]  disk 1, o:1, dev:sdb6
> [    1.726095]  disk 2, o:1, dev:sdc6
> [    1.726142] md3: detected capacity change from 0 to 864065421312
> [    1.726213] md: considering sdc5 ...
> [    1.726249] md:  adding sdc5 ...
> [    1.726283] md: sdc3 has different UUID to sdc5
> [    1.726318] md: sdc1 has different UUID to sdc5
> [    1.726353] md:  adding sda5 ...
> [    1.726388] md: sda3 has different UUID to sdc5
> [    1.726422] md: sda1 has different UUID to sdc5
> [    1.726458] md:  adding sdb5 ...
> [    1.726492] md: sdb3 has different UUID to sdc5
> [    1.726526] md: sdb1 has different UUID to sdc5
> [    1.726630] md: created md2
> [    1.726663] md: bind<sdb5>
> [    1.726700] md: bind<sda5>
> [    1.726738] md: bind<sdc5>
> [    1.726774] md: running:<sdc5><sda5><sdb5>
> [    1.726901] raid5: device sdc5 operational as raid disk 2
> [    1.726935] raid5: device sda5 operational as raid disk 0
> [    1.726969] raid5: device sdb5 operational as raid disk 1
> [    1.727126] raid5: allocated 3220kB for md2
> [    1.727190] raid5: raid level 5 set md2 active with 3 out of 3 devices,
> algorithm 2
> [    1.727243] RAID5 conf printout:
> [    1.727276]  --- rd:3 wd:3
> [    1.727309]  disk 0, o:1, dev:sda5
> [    1.727342]  disk 1, o:1, dev:sdb5
> [    1.727376]  disk 2, o:1, dev:sdc5
> [    1.727420] md2: detected capacity change from 0 to 40007499776
> [    1.727490] md: considering sdc3 ...
> [    1.727526] md:  adding sdc3 ...
> [    1.727560] md: sdc1 has different UUID to sdc3
> [    1.727595] md:  adding sda3 ...
> [    1.727629] md: sda1 has different UUID to sdc3
> [    1.727664] md:  adding sdb3 ...
> [    1.727698] md: sdb1 has different UUID to sdc3
> [    1.727799] md: created md1
> [    1.727832] md: bind<sdb3>
> [    1.727869] md: bind<sda3>
> [    1.727905] md: bind<sdc3>
> [    1.727945] md: running:<sdc3><sda3><sdb3>
> [    1.728090] raid5: device sdc3 operational as raid disk 2
> [    1.728125] raid5: device sda3 operational as raid disk 0
> [    1.728159] raid5: device sdb3 operational as raid disk 1
> [    1.728320] raid5: allocated 3220kB for md1
> [    1.728370] raid5: raid level 5 set md1 active with 3 out of 3 devices,
> algorithm 2
> [    1.728423] RAID5 conf printout:
> [    1.728455]  --- rd:3 wd:3
> [    1.728488]  disk 0, o:1, dev:sda3
> [    1.728522]  disk 1, o:1, dev:sdb3
> [    1.728555]  disk 2, o:1, dev:sdc3
> [    1.728604] md1: detected capacity change from 0 to 79998877696
> [    1.728674] md: considering sdc1 ...
> [    1.728710] md:  adding sdc1 ...
> [    1.728745] md:  adding sda1 ...
> [    1.728779] md:  adding sdb1 ...
> [    1.728813] md: created md0
> [    1.728846] md: bind<sdb1>
> [    1.728882] md: bind<sda1>
> [    1.728919] md: bind<sdc1>
> [    1.728955] md: running:<sdc1><sda1><sdb1>
> [    1.729133] raid1: raid set md0 active with 3 out of 3 mirrors
> [    1.729176] md0: detected capacity change from 0 to 65667072
> [    1.729232] md: ... autorun DONE.
> [    1.729284] md: Loading md3: /dev/sda3
> [    1.729322]  md3: unknown partition table
> [    1.729481] md: couldn't update array info. -22
> [    1.729518] md: could not bd_claim sda3.
> [    1.729552] md: md_import_device returned -16
> [    1.729588] md: could not bd_claim sdb3.
> [    1.729621] md: md_import_device returned -16
> [    1.729657] md: could not bd_claim sdc3.
> [    1.729690] md: md_import_device returned -16
> [    1.729725] md: starting md3 failed
> [    1.729800]  md1: unknown partition table
> [    1.767199] reiser4: md1: found disk format 4.0.0.
> [    5.790318] VFS: Mounted root (reiser4 filesystem) readonly on device 9:1.
> [    5.790370] Freeing unused kernel memory: 376k freed
> [    9.037775] udev: starting version 145
> [    9.217043]  md2:
> [    9.217072]  md0: unknown partition table
> [    9.282015]  unknown partition table
> [   10.420576] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> [   10.420591] r8169 0000:01:00.0: PCI INT A ->  GSI 18 (level, low) ->  IRQ 18
> [   10.420840] r8169 0000:01:00.0: setting latency timer to 64
> [   10.420871]   alloc irq_desc for 28 on node -1
> [   10.420872]   alloc kstat_irqs on node -1
> [   10.420882] r8169 0000:01:00.0: irq 28 for MSI/MSI-X
> [   10.420988] eth0: RTL8168c/8111c at 0xffffc90012f18000, 00:19:66:86:ce:12,
> XID 3c4000c0 IRQ 28
> [   10.440462] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
> [   10.440465] ehci_hcd: block sizes: qh 192 qtd 96 itd 192 sitd 96
> [   10.440491] ehci_hcd 0000:00:12.2: PCI INT B ->  GSI 17 (level, low) ->  IRQ
> 17
> [   10.440518] ehci_hcd 0000:00:12.2: EHCI Host Controller
> [   10.440532] drivers/usb/core/inode.c: creating file 'devices'
> [   10.440534] drivers/usb/core/inode.c: creating file '001'
> [   10.440566] ehci_hcd 0000:00:12.2: new USB bus registered, assigned bus
> number 1
> [   10.440572] ehci_hcd 0000:00:12.2: reset hcs_params 0x102306 dbg=1 cc=2
> pcc=3 ordered !ppc ports=6
> [   10.440575] ehci_hcd 0000:00:12.2: reset hcc_params a072 thresh 7 uframes
> 256/512/1024
> [   10.440596] ehci_hcd 0000:00:12.2: applying AMD SB600/SB700 USB freeze
> workaround
> [   10.440602] ehci_hcd 0000:00:12.2: reset command 080002 (park)=0 ithresh=8
> period=1024 Reset HALT
> [   10.440615] ehci_hcd 0000:00:12.2: debug port 1
> [   10.440619] ehci_hcd 0000:00:12.2: MWI active
> [   10.440620] ehci_hcd 0000:00:12.2: supports USB remote wakeup
> [   10.440633] ehci_hcd 0000:00:12.2: irq 17, io mem 0xfddff000
> [   10.440637] ehci_hcd 0000:00:12.2: reset command 080002 (park)=0 ithresh=8
> period=1024 Reset HALT
> [   10.440642] ehci_hcd 0000:00:12.2: init command 010009 (park)=0 ithresh=1
> period=256 RUN
> [   10.447931] ehci_hcd 0000:00:12.2: USB 2.0 started, EHCI 1.00
> [   10.447961] usb usb1: default language 0x0409
> [   10.447965] usb usb1: udev 1, busnum 1, minor = 0
> [   10.447967] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
> [   10.447968] usb usb1: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [   10.447969] usb usb1: Product: EHCI Host Controller
> [   10.447970] usb usb1: Manufacturer: Linux 2.6.31r4 ehci_hcd
> [   10.447971] usb usb1: SerialNumber: 0000:00:12.2
> [   10.447998] usb usb1: uevent
> [   10.448007] usb usb1: usb_probe_device
> [   10.448009] usb usb1: configuration #1 chosen from 1 choice
> [   10.448014] usb usb1: adding 1-0:1.0 (config #1, interface 0)
> [   10.448021] usb 1-0:1.0: uevent
> [   10.448028] hub 1-0:1.0: usb_probe_interface
> [   10.448029] hub 1-0:1.0: usb_probe_interface - got id
> [   10.448031] hub 1-0:1.0: USB hub found
> [   10.448035] hub 1-0:1.0: 6 ports detected
> [   10.448036] hub 1-0:1.0: standalone hub
> [   10.448037] hub 1-0:1.0: no power switching (usb 1.0)
> [   10.448038] hub 1-0:1.0: individual port over-current protection
> [   10.448039] hub 1-0:1.0: power on to power good time: 20ms
> [   10.448042] hub 1-0:1.0: local power source is good
> [   10.448043] hub 1-0:1.0: trying to enable port power on non-switchable hub
> [   10.448067] drivers/usb/core/inode.c: creating file '001'
> [   10.448085]   alloc irq_desc for 19 on node -1
> [   10.448087]   alloc kstat_irqs on node -1
> [   10.448091] ehci_hcd 0000:00:13.2: PCI INT B ->  GSI 19 (level, low) ->  IRQ
> 19
> [   10.448101] ehci_hcd 0000:00:13.2: EHCI Host Controller
> [   10.448105] drivers/usb/core/inode.c: creating file '002'
> [   10.448122] ehci_hcd 0000:00:13.2: new USB bus registered, assigned bus
> number 2
> [   10.448127] ehci_hcd 0000:00:13.2: reset hcs_params 0x102306 dbg=1 cc=2
> pcc=3 ordered !ppc ports=6
> [   10.448130] ehci_hcd 0000:00:13.2: reset hcc_params a072 thresh 7 uframes
> 256/512/1024
> [   10.448143] ehci_hcd 0000:00:13.2: applying AMD SB600/SB700 USB freeze
> workaround
> [   10.448148] ehci_hcd 0000:00:13.2: reset command 080002 (park)=0 ithresh=8
> period=1024 Reset HALT
> [   10.448161] ehci_hcd 0000:00:13.2: debug port 1
> [   10.448164] ehci_hcd 0000:00:13.2: MWI active
> [   10.448165] ehci_hcd 0000:00:13.2: supports USB remote wakeup
> [   10.448173] ehci_hcd 0000:00:13.2: irq 19, io mem 0xfddf6800
> [   10.448176] ehci_hcd 0000:00:13.2: reset command 080002 (park)=0 ithresh=8
> period=1024 Reset HALT
> [   10.448181] ehci_hcd 0000:00:13.2: init command 010009 (park)=0 ithresh=1
> period=256 RUN
> [   10.457930] ehci_hcd 0000:00:13.2: USB 2.0 started, EHCI 1.00
> [   10.457945] usb usb2: default language 0x0409
> [   10.457949] usb usb2: udev 1, busnum 2, minor = 128
> [   10.457950] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002
> [   10.457952] usb usb2: New USB device strings: Mfr=3, Product=2,
> SerialNumber=1
> [   10.457953] usb usb2: Product: EHCI Host Controller
> [   10.457954] usb usb2: Manufacturer: Linux 2.6.31r4 ehci_hcd
> [   10.457955] usb usb2: SerialNumber: 0000:00:13.2
> [   10.457977] usb usb2: uevent
> [   10.457984] usb usb2: usb_probe_device
> [   10.457986] usb usb2: configuration #1 chosen from 1 choice
> [   10.457989] usb usb2: adding 2-0:1.0 (config #1, interface 0)
> [   10.457997] usb 2-0:1.0: uevent
> [   10.458003] hub 2-0:1.0: usb_probe_interface
> [   10.458004] hub 2-0:1.0: usb_probe_interface - got id
> [   10.458005] hub 2-0:1.0: USB hub found
> [   10.458009] hub 2-0:1.0: 6 ports detected
> [   10.458010] hub 2-0:1.0: standalone hub
> [   10.458010] hub 2-0:1.0: no power switching (usb 1.0)
> [   10.458011] hub 2-0:1.0: individual port over-current protection
> [   10.458013] hub 2-0:1.0: power on to power good time: 20ms
> [   10.458015] hub 2-0:1.0: local power source is good
> [   10.458016] hub 2-0:1.0: trying to enable port power on non-switchable hub
> [   10.458038] drivers/usb/core/inode.c: creating file '001'
> [   10.474718] Linux video capture interface: v2.00
> [   10.489875] bttv: driver version 0.9.18 loaded
> [   10.489877] bttv: using 8 buffers with 2080k (520 pages) each for capture
> [   10.489914] bttv: Bt8xx card found (0).
> [   10.489923]   alloc irq_desc for 21 on node -1
> [   10.489924]   alloc kstat_irqs on node -1
> [   10.489928] bttv 0000:05:06.0: PCI INT A ->  GSI 21 (level, low) ->  IRQ 21
> [   10.489938] bttv0: Bt848 (rev 18) at 0000:05:06.0, irq: 21, latency: 128,
> mmio: 0xfcfff000
> [   10.489971] bttv0: using: Terratec TerraTV+ Version 1.0 (Bt848)/ Terra
> TValue Version 1.0/ Vobis TV-Boostar [card=25,insmod option]
> [   10.489974] IRQ 21/bttv0: IRQF_DISABLED is not guaranteed on shared IRQs
> [   10.490007] bttv0: gpio: en=00000000, out=00000000 in=00ffffff [init]
> [   10.547936] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001803 POWER
> sig=j CSC CONNECT
> [   10.547939] hub 1-0:1.0: port 2: status 0501 change 0001
> [   10.557954] hub 2-0:1.0: state 7 ports 6 chg 0000 evt 0000
> [   10.647931] hub 1-0:1.0: state 7 ports 6 chg 0004 evt 0000
> [   10.647940] hub 1-0:1.0: port 2, status 0501, change 0000, 480 Mb/s
> [   10.700045] ehci_hcd 0000:00:12.2: port 2 high speed
> [   10.700049] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001005 POWER
> sig=se0 PE CONNECT
> [   10.757104] usb 1-2: new high speed USB device using ehci_hcd and address 2
> [   10.810036] ehci_hcd 0000:00:12.2: port 2 high speed
> [   10.810039] ehci_hcd 0000:00:12.2: GetStatus port 2 status 001005 POWER
> sig=se0 PE CONNECT
> [   10.881474] usb 1-2: default language 0x0409
> [   10.881724] usb 1-2: udev 2, busnum 1, minor = 1
> [   10.881725] usb 1-2: New USB device found, idVendor=05e3, idProduct=0608
> [   10.881726] usb 1-2: New USB device strings: Mfr=0, Product=1,
> SerialNumber=0
> [   10.881728] usb 1-2: Product: USB2.0 Hub
> [   10.881764] usb 1-2: uevent
> [   10.881773] usb 1-2: usb_probe_device
> [   10.881775] usb 1-2: configuration #1 chosen from 1 choice
> [   10.882188] usb 1-2: adding 1-2:1.0 (config #1, interface 0)
> [   10.882199] usb 1-2:1.0: uevent
> [   10.882206] hub 1-2:1.0: usb_probe_interface
> [   10.882207] hub 1-2:1.0: usb_probe_interface - got id
> [   10.882209] hub 1-2:1.0: USB hub found
> [   10.882473] hub 1-2:1.0: 4 ports detected
> [   10.882474] hub 1-2:1.0: standalone hub
> [   10.882476] hub 1-2:1.0: individual port power switching
> [   10.882477] hub 1-2:1.0: individual port over-current protection
> [   10.882478] hub 1-2:1.0: Single TT
> [   10.882479] hub 1-2:1.0: TT requires at most 32 FS bit times (2664 ns)
> [   10.882480] hub 1-2:1.0: Port indicators are supported
> [   10.882481] hub 1-2:1.0: power on to power good time: 100ms
> [   10.882848] hub 1-2:1.0: local power source is good
> [   10.882849] hub 1-2:1.0: enabling power on all ports
> [   10.883860] drivers/usb/core/inode.c: creating file '002'
> [   10.983961] hub 1-2:1.0: port 2: status 0301 change 0001
> [   11.083356] usb 1-2: link qh256-0001/ffff8800c7800180 start 1 [1/0 us]
> [   11.083364] hub 1-2:1.0: state 7 ports 4 chg 0004 evt 0000
> [   11.083700] hub 1-2:1.0: port 2, status 0301, change 0000, 1.5 Mb/s
> [   11.152198] usb 1-2.2: new low speed USB device using ehci_hcd and address
> 3
> [   11.241187] usb 1-2.2: skipped 1 descriptor after interface
> [   11.241189] usb 1-2.2: skipped 1 descriptor after interface
> [   11.241685] usb 1-2.2: default language 0x0409
> [   11.243941] usb 1-2.2: udev 3, busnum 1, minor = 2
> [   11.243943] usb 1-2.2: New USB device found, idVendor=046d, idProduct=c518
> [   11.243944] usb 1-2.2: New USB device strings: Mfr=1, Product=2,
> SerialNumber=0
> [   11.243945] usb 1-2.2: Product: USB Receiver
> [   11.243947] usb 1-2.2: Manufacturer: Logitech
> [   11.243972] usb 1-2.2: uevent
> [   11.243980] usb 1-2.2: usb_probe_device
> [   11.243981] usb 1-2.2: configuration #1 chosen from 1 choice
> [   11.251309] usb 1-2.2: adding 1-2.2:1.0 (config #1, interface 0)
> [   11.251324] usb 1-2.2:1.0: uevent
> [   11.251334] usbhid 1-2.2:1.0: usb_probe_interface
> [   11.251335] usbhid 1-2.2:1.0: usb_probe_interface - got id
> [   11.251796] usb 1-2: clear tt buffer port 2, a3 ep0 t80008d42
> [   11.254692] input: Logitech USB Receiver as
> /devices/pci0000:00/0000:00:12.2/usb1/1-2/1-2.2/1-2.2:1.0/input/input3
> [   11.254732] generic-usb 0003:046D:C518.0001: input,hidraw0: USB HID v1.11
> Mouse [Logitech USB Receiver] on usb-0000:00:12.2-2.2/input0
> [   11.254740] usb 1-2.2: adding 1-2.2:1.1 (config #1, interface 1)
> [   11.254749] usb 1-2.2:1.1: uevent
> [   11.254755] usbhid 1-2.2:1.1: usb_probe_interface
> [   11.254757] usbhid 1-2.2:1.1: usb_probe_interface - got id
> [   11.255046] usb 1-2: clear tt buffer port 2, a3 ep0 t80008d42
> [   11.260359] input: Logitech USB Receiver as
> /devices/pci0000:00/0000:00:12.2/usb1/1-2/1-2.2/1-2.2:1.1/input/input4
> [   11.260368] usb 1-2.2: link qh8-0601/ffff8800c7800300 start 2 [1/2 us]
> [   11.260384] drivers/usb/core/file.c: looking for a minor, starting at 96
> [   11.260414] generic-usb 0003:046D:C518.0002: input,hiddev96,hidraw1: USB
> HID v1.11 Device [Logitech USB Receiver] on usb-0000:00:12.2-2.2/input1
> [   11.260427] drivers/usb/core/inode.c: creating file '003'
> [   11.260438] hub 1-2:1.0: state 7 ports 4 chg 0000 evt 0004
> [   11.280770] usb 1-2.2:1.0: uevent
> [   11.280827] usb 1-2.2: uevent
> [   11.280844] usb 1-2.2:1.0: uevent
> [   11.280903] usb 1-2.2: uevent
> [   11.281405] usb 1-2.2:1.1: uevent
> [   11.281467] usb 1-2.2: uevent
> [   11.281517] usb 1-2.2:1.0: uevent
> [   11.281529] usb 1-2.2:1.0: uevent
> [   11.282164] usb 1-2.2:1.1: uevent
> [   11.493831] bttv0: tea5757: read timeout
> [   11.493832] bttv0: tuner type=5
> [   11.505445] bttv0: audio absent, no audio device found!
> [   11.525030] TUNER: Unable to find symbol tea5767_autodetection()
> [   11.525033] tuner 0-0060: chip found @ 0xc0 (bt848 #0 [sw])
> [   11.527843] tuner-simple 0-0060: creating new instance
> [   11.527846] tuner-simple 0-0060: type set to 5 (Philips PAL_BG (FI1216 and
> compatibles))
> [   11.528664] bttv0: registered device video0
> [   11.528695] bttv0: registered device vbi0
> [   11.814508] reiser4: md2: found disk format 4.0.0.
> [   13.337101] hub 2-0:1.0: hub_suspend
> [   13.337107] usb usb2: bus auto-suspend
> [   13.337109] ehci_hcd 0000:00:13.2: suspend root hub
> [   13.591267] reiser4: md3: found disk format 4.0.0.
> [   55.020583] reiser4: sdc7: found disk format 4.0.0.
> [   67.386552] Adding 7815612k swap on /dev/sda2.  Priority:1 extents:1
> across:7815612k
> [   67.408915] Adding 7815612k swap on /dev/sdb2.  Priority:1 extents:1
> across:7815612k
> [   67.501498] Adding 7815612k swap on /dev/sdc2.  Priority:1 extents:1
> across:7815612k
> [   68.205087] w83627ehf: Found W83627EHG chip at 0x290
> [   68.421646] r8169: eth0: link up
> [   68.421650] r8169: eth0: link up
> [   70.281241] usb usb1: uevent
> [   70.281269] usb 1-0:1.0: uevent
> [   70.281293] usb 1-2: uevent
> [   70.281320] usb 1-2.2: uevent
> [   70.281346] usb 1-2.2:1.0: uevent
> [   70.281533] usb 1-2.2:1.1: uevent
> [   70.281706] usb 1-2:1.0: uevent
> [   70.281804] usb usb2: uevent
> [   70.281830] usb 2-0:1.0: uevent
> [   71.272886]   alloc irq_desc for 23 on node -1
> [   71.272889]   alloc kstat_irqs on node -1
> [   71.272895] EMU10K1_Audigy 0000:05:08.0: PCI INT A ->  GSI 23 (level, low) -
>> IRQ 23
> [   71.278891] Audigy2 value: Special config.
> [   72.473408] fglrx: module license 'Proprietary. (C) 2002 - ATI
> Technologies, Starnberg, GERMANY' taints kernel.
> [   72.473417] Disabling lock debugging due to kernel taint
> [   72.490481] [fglrx] Maximum main memory to use for locked dma buffers: 7760
> MBytes.
> [   72.490559] [fglrx]   vendor: 1002 device: 9501 count: 1
> [   72.490739] [fglrx] ioport: bar 4, base 0xc000, size: 0x100
> [   72.490750] pci 0000:02:00.0: PCI INT A ->  GSI 18 (level, low) ->  IRQ 18
> [   72.490754] pci 0000:02:00.0: setting latency timer to 64
> [   72.490884] [fglrx] Kernel PAT support is enabled
> [   72.490901] [fglrx] module loaded - fglrx 8.66.2 [Sep  1 2009] with 1
> minors
> [   72.668413]   alloc irq_desc for 29 on node -1
> [   72.668416]   alloc kstat_irqs on node -1
> [   72.668424] fglrx_pci 0000:02:00.0: irq 29 for MSI/MSI-X
> [   72.668759] [fglrx] Firegl kernel thread PID: 3800
> [   74.787429] [fglrx] Gart USWC size:1279 M.
> [   74.787431] [fglrx] Gart cacheable size:508 M.
> [   74.787435] [fglrx] Reserved FB block: Shared offset:0, size:1000000
> [   74.787437] [fglrx] Reserved FB block: Unshared offset:fbff000, size:401000
> [   74.787438] [fglrx] Reserved FB block: Unshared offset:1fffc000, size:4000
> [   75.947153] usb 1-2.2: link qh8-0601/ffff8800c78003c0 start 3 [1/2 us]
> [  616.849440] reiser4[ktxnmgrd:md1:ru(581)]: disable_write_barrier
> (fs/reiser4/wander.c:235)[zam-1055]:
> [  616.849445] NOTICE: md1 does not support write barriers, using synchronous
> write instead.
> [  671.813536] reiser4[ktxnmgrd:md2:ru(2774)]: disable_write_barrier
> (fs/reiser4/wander.c:235)[zam-1055]:
> [  671.813541] NOTICE: md2 does not support write barriers, using synchronous
> write instead.
> [  703.842289] reiser4[ktxnmgrd:md3:ru(2776)]: disable_write_barrier
> (fs/reiser4/wander.c:235)[zam-1055]:
> [  703.842293] NOTICE: md3 does not support write barriers, using synchronous
> write instead.
>
> ps. I have to disable c1e in bios, or video is very jerky - unless something
> cpu heavy runs on at least one core...
>
>
>

Re: BFS vs. mainline scheduler benchmarks and measurements

[Arjan van de Ven]

On Sat, 12 Sep 2009 10:37:45 +0300
Nikos Chantziaras <realnc@arcor.de> wrote:

> (Volker stripped all CCs from his posts; I restored them manually.)
> 
> On 09/11/2009 09:33 PM, Volker Armin Hemmann wrote:
> > Hi,
> >
> > this is with 2.6.31+reiser4+fglrx
> > Phenom II X4 955
> >
> > KDE 4.3.1, composite temporary disabled.
> > tvtime running.
> >
> > load:
> > fat emerge with make -j5 running in one konsole tab (xulrunner being
> > compiled).
> >
> > without NO_NEW_FAIR_SLEEPERS:
> >
> > tvtime is smooth most of the time
> >
> > with NO_NEW_FAIR_SLEEPERS:
> >
> > tvtime is more jerky. Very visible in scenes with movement.
> 
> Is the make -j5 running niced 0?  If yes, that would be actually the 
> correct behavior.  Unfortunately, I can't test tvtime specifically (I 
> don't have a TV card), but other applications displaying video
> continue to work smooth on my dual core machine (Core 2 Duo E6600)
> even if I do "nice -n 19 make -j20".  If I don't nice it, the video
> is skippy here too though.
> 
> Question to Ingo:
> Would posting perf results help in any way with finding differences 
> between mainline NEW_FAIR_SLEEPERS/NO_NEW_FAIR_SLEEPERS and BFS?

please also post latencytop output for the app you care about
(the system wide latencytop numbers aren't as relevant; to some large
degree what is happening is that if you oversubscribe, you need to pay
the price for that period, all you can do is move the cost around to
those tasks you don't care about. For that reason, latencytop output
for the task you care about is relevant ;-)


-- 
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: BFS vs. mainline scheduler benchmarks and measurements

[Volker Armin Hemmann]

Hi,

On Samstag 12 September 2009, Nikos Chantziaras wrote:
> (Volker stripped all CCs from his posts; I restored them manually.)

stripping is not entirely correct. marc does not list all recipients ;)
Thank you.

> 
> On 09/11/2009 09:33 PM, Volker Armin Hemmann wrote:
> > Hi,
> >
> > this is with 2.6.31+reiser4+fglrx
> > Phenom II X4 955
> >
> > KDE 4.3.1, composite temporary disabled.
> > tvtime running.
> >
> > load:
> > fat emerge with make -j5 running in one konsole tab (xulrunner being
> > compiled).
> >
> > without NO_NEW_FAIR_SLEEPERS:
> >
> > tvtime is smooth most of the time
> >
> > with NO_NEW_FAIR_SLEEPERS:
> >
> > tvtime is more jerky. Very visible in scenes with movement.
> 
> Is the make -j5 running niced 0? 

yes. It always is.


> If yes, that would be actually the
> correct behavior. 

maybe. But I do not complain about jerks at all. I have 

[ 3618.305918] hpet1: lost 1 rtc interrupts

with tvtime running since I switched cpus - so something is wrong anyway.

I just wanted to report that for _me_ the behaviour is worse with 
NO_NEW_FAIR_SLEEPERS and plain 2.6.31.

I tried it yesterday when the firefox update came in and I switched between 
NO_NEW_FAIR... and NEW_FAIR... several times and with NO_NEW_FAIR... tvtime 
was just _more_ jerky, I am not saying that it wasn't jerky without it nor do 
I complain about it at all. ;)

Glück Auf,
Volker

Re: BFS vs. mainline scheduler benchmarks and measurements

[Nikos Chantziaras]

On 09/12/2009 11:27 AM, Volker Armin Hemmann wrote:
> Hi,
>
> On Samstag 12 September 2009, Nikos Chantziaras wrote:
>>
>> On 09/11/2009 09:33 PM, Volker Armin Hemmann wrote:
>>>
>>> this is with 2.6.31+reiser4+fglrx
>>> Phenom II X4 955
>>>
>>> KDE 4.3.1, composite temporary disabled.
>>> tvtime running.
>>>
>>> load:
>>> fat emerge with make -j5 running in one konsole tab (xulrunner being
>>> compiled).
>>>
>>> without NO_NEW_FAIR_SLEEPERS:
>>>
>>> tvtime is smooth most of the time
>>>
>>> with NO_NEW_FAIR_SLEEPERS:
>>>
>>> tvtime is more jerky. Very visible in scenes with movement.
>>
>> Is the make -j5 running niced 0?
>
> yes. It always is.
>
>> If yes, that would be actually the
>> correct behavior.
>
> maybe. But I do not complain about jerks at all. I have
>
> [ 3618.305918] hpet1: lost 1 rtc interrupts
>
> with tvtime running since I switched cpus - so something is wrong anyway.

Seeing the "lost 1 rtc interrupts" message makes me wonder if this could 
possibly relate to problems with the C1E state on AMD systems (missing 
timer interrupts):

   http://lkml.org/lkml/2008/6/12/127

That thread is one year old though and your Phenom II CPU was released 7 
months later.

Re: BFS vs. mainline scheduler benchmarks and measurements

[Volker Armin Hemmann]

On Samstag 12 September 2009, Nikos Chantziaras wrote:

> Seeing the "lost 1 rtc interrupts" message makes me wonder if this could
> possibly relate to problems with the C1E state on AMD systems (missing
> timer interrupts):
> 
>    http://lkml.org/lkml/2008/6/12/127
> 
> That thread is one year old though and your Phenom II CPU was released 7
> months later.
> 

thanks for the link!