linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
@ 2002-09-23  3:04 Con Kolivas
  2002-09-23  3:10 ` Robert Love
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Con Kolivas @ 2002-09-23  3:04 UTC (permalink / raw)
  To: linux-kernel

I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with
gcc3.2 to gcc2.95.3

warning: The following benchmarks may be disturbing to some viewers:

No Load:
Kernel                  Time            CPU
2.5.38                  68.25           99%
2.5.38-gcc32            103.03          99%

Process Load:
Kernel                  Time            CPU
2.5.38                  71.60           95%
2.5.38-gcc32            112.98          91%

IO Half Load:
Kernel                  Time            CPU
2.5.38                  81.26           90%
2.5.38-gcc32            168.25          70%

IO Full Load:
Kernel                  Time            CPU
2.5.38                  170.21          42%
2.5.38-gcc32            1405.25         8%

Mem Load:
Kernel                  Time            CPU
2.5.38                  104.22          70%
2.5.38-gcc32            153.00          74%

Both kernels had identical configs and contest was run using gcc2.95.3 (I can't
see how this would affect it but tell me if you think it will). The only
difference was the original 2.5.38 was compiled with gcc2.95.3 and -gcc32 was
compiled with gcc3.2

Note the lower performance even with noload, and the totally abysmal performance
under full load. See the logs below for the page fault differences too.

Full logs follow:
2.5.38:
noload Time: 68.25  CPU: 99%  Major Faults: 204613  Minor Faults: 255906
process_load Time: 71.60  CPU: 95%  Major Faults: 204019  Minor Faults: 255238
io_halfmem Time: 81.26  CPU: 90%  Major Faults: 204019  Minor Faults: 255325
Was writing number 4 of a 112Mb sized io_load file after 90 seconds
io_fullmem Time: 170.21  CPU: 42%  Major Faults: 204019  Minor Faults: 255272
Was writing number 6 of a 224Mb sized io_load file after 194 seconds
mem_load Time: 104.22  CPU: 70%  Major Faults: 204120  Minor Faults: 256271

2.5.38-gcc32:
noload Time: 103.03  CPU: 99%  Major Faults: 237750  Minor Faults: 439982
process_load Time: 112.98  CPU: 91%  Major Faults: 236958  Minor Faults: 439030
io_halfmem Time: 168.25  CPU: 70%  Major Faults: 236958  Minor Faults: 439261
Was writing number 16 of a 112Mb sized io_load file after 202 seconds
io_fullmem Time: 1405.25  CPU: 8%  Major Faults: 237199  Minor Faults: 440233
Was writing number 59 of a 224Mb sized io_load file after 1496 seconds
mem_load Time: 153.00  CPU: 74%  Major Faults: 237863  Minor Faults: 440524

These are disturbing to me. 

Comments?
Con.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:04 [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38) Con Kolivas
@ 2002-09-23  3:10 ` Robert Love
  2002-09-23  3:16   ` Con Kolivas
  2002-09-23  3:28 ` Robert Love
  2002-09-23  3:41 ` Andrew Morton
  2 siblings, 1 reply; 12+ messages in thread
From: Robert Love @ 2002-09-23  3:10 UTC (permalink / raw)
  To: Con Kolivas; +Cc: linux-kernel

On Sun, 2002-09-22 at 23:04, Con Kolivas wrote:

> IO Full Load:
> Kernel                  Time            CPU
> 2.5.38                  170.21          42%
> 2.5.38-gcc32            1405.25         8%

Ugh??  Something is _seriously_ messed up here.

The CPU utilization is only 8% but the time is nearly 10x worse.  You
sure the only difference was the compiler?  I could think gcc-3.2 makes
some poorer choices wrt code optimization, but nothing feasible can come
to mind that would produce such terrible results.

Also, I believe RedHat is compiling their kernel in 8.0 with gcc-3.2,
unless they reintroduced kgcc.  Surely that are not seeing these abysmal
numbers.

	Robert Love


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:10 ` Robert Love
@ 2002-09-23  3:16   ` Con Kolivas
  2002-09-23 11:06     ` Mark Veltzer
  0 siblings, 1 reply; 12+ messages in thread
From: Con Kolivas @ 2002-09-23  3:16 UTC (permalink / raw)
  To: Robert Love; +Cc: linux-kernel

Quoting Robert Love <rml@tech9.net>:

> On Sun, 2002-09-22 at 23:04, Con Kolivas wrote:
> 
> > IO Full Load:
> > Kernel                  Time            CPU
> > 2.5.38                  170.21          42%
> > 2.5.38-gcc32            1405.25         8%
> 
> Ugh??  Something is _seriously_ messed up here.

Agreed!

> The CPU utilization is only 8% but the time is nearly 10x worse.  You
> sure the only difference was the compiler?  I could think gcc-3.2 makes
> some poorer choices wrt code optimization, but nothing feasible can come
> to mind that would produce such terrible results.

Absolutely certain. I'm shaking from the results still... hard to type...

> Also, I believe RedHat is compiling their kernel in 8.0 with gcc-3.2,
> unless they reintroduced kgcc.  Surely that are not seeing these abysmal
> numbers.

contest is a new benchmark. Noone has ever done anything like this before so it
wouldn't have shown up in ordinary benchmarks. Mandrake has done the same with
LM9.0 I believe

Con

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:04 [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38) Con Kolivas
  2002-09-23  3:10 ` Robert Love
@ 2002-09-23  3:28 ` Robert Love
  2002-09-23  3:41 ` Andrew Morton
  2 siblings, 0 replies; 12+ messages in thread
From: Robert Love @ 2002-09-23  3:28 UTC (permalink / raw)
  To: Con Kolivas; +Cc: linux-kernel

On Sun, 2002-09-22 at 23:04, Con Kolivas wrote:

> I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with
> gcc3.2 to gcc2.95.3
> 
> warning: The following benchmarks may be disturbing to some viewers:

Daniel Jacobowitz, a stunning gcc hacker, suggests you retry with
gcc-3.2 from CVS.  Apparently some fixes have gone in since 3.2 release.

If the problem still occurs, there is either a serious problem in gcc or
we have some code that is miscompiled by gcc-3.2 (perhaps not its fault
but the kernel's).

	Robert Love


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:04 [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38) Con Kolivas
  2002-09-23  3:10 ` Robert Love
  2002-09-23  3:28 ` Robert Love
@ 2002-09-23  3:41 ` Andrew Morton
  2002-09-23  3:46   ` Daniel Jacobowitz
  2002-09-23  3:47   ` Robert Love
  2 siblings, 2 replies; 12+ messages in thread
From: Andrew Morton @ 2002-09-23  3:41 UTC (permalink / raw)
  To: Con Kolivas; +Cc: linux-kernel

Con Kolivas wrote:
> 
> I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with
> gcc3.2 to gcc2.95.3
> 
> warning: The following benchmarks may be disturbing to some viewers:
> 
> No Load:
> Kernel                  Time            CPU
> 2.5.38                  68.25           99%
> 2.5.38-gcc32            103.03          99%
> 

Try gcc-2.91.66.  It might break the 45 second mark.

> IO Full Load:
> Kernel                  Time            CPU
> 2.5.38                  170.21          42%
> 2.5.38-gcc32            1405.25         8%

The streaming write is stalling gcc's read for long enough for gcc's
working set to be evicted.  And the working set cannot be reestablished
because the streaming write prevents it.  Meltdown.

I have fixed this.  Hang around.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:41 ` Andrew Morton
@ 2002-09-23  3:46   ` Daniel Jacobowitz
  2002-09-23  3:50     ` Con Kolivas
  2002-09-23  3:47   ` Robert Love
  1 sibling, 1 reply; 12+ messages in thread
From: Daniel Jacobowitz @ 2002-09-23  3:46 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Con Kolivas, linux-kernel

On Sun, Sep 22, 2002 at 08:41:51PM -0700, Andrew Morton wrote:
> Con Kolivas wrote:
> > 
> > I performed contest benchmarks on kernel 2.5.38 when the kernel is compiled with
> > gcc3.2 to gcc2.95.3
> > 
> > warning: The following benchmarks may be disturbing to some viewers:
> > 
> > No Load:
> > Kernel                  Time            CPU
> > 2.5.38                  68.25           99%
> > 2.5.38-gcc32            103.03          99%
> > 
> 
> Try gcc-2.91.66.  It might break the 45 second mark.
> 
> > IO Full Load:
> > Kernel                  Time            CPU
> > 2.5.38                  170.21          42%
> > 2.5.38-gcc32            1405.25         8%
> 
> The streaming write is stalling gcc's read for long enough for gcc's
> working set to be evicted.  And the working set cannot be reestablished
> because the streaming write prevents it.  Meltdown.
> 
> I have fixed this.  Hang around.

That sounds like a slowdown in _running_ GCC... The worrisome part of
Con's post is that he was talking about which compiler he'd built the
kernel with.  Right?

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:41 ` Andrew Morton
  2002-09-23  3:46   ` Daniel Jacobowitz
@ 2002-09-23  3:47   ` Robert Love
  1 sibling, 0 replies; 12+ messages in thread
From: Robert Love @ 2002-09-23  3:47 UTC (permalink / raw)
  To: Andrew Morton; +Cc: Con Kolivas, linux-kernel

On Sun, 2002-09-22 at 23:41, Andrew Morton wrote:

> Try gcc-2.91.66.  It might break the 45 second mark.
> 
> > IO Full Load:
> > Kernel                  Time            CPU
> > 2.5.38                  170.21          42%
> > 2.5.38-gcc32            1405.25         8%
> 
> The streaming write is stalling gcc's read for long enough for gcc's
> working set to be evicted.  And the working set cannot be reestablished
> because the streaming write prevents it.  Meltdown.
> 
> I have fixed this.  Hang around.

Ehh, I was under the impression he was benchmarking kernels compiled
WITH these compilers, using contest?

Your post seems to imply he was using the compilers as the benchmark. 
If so, I retract my previous post - I know gcc 3.x is slow as puke.  I
think, however, he is comparing the resulting kernels, in which case
there is a serious issue at hand...

	Robert Love


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:46   ` Daniel Jacobowitz
@ 2002-09-23  3:50     ` Con Kolivas
       [not found]       ` <3D8E9158.4E3DE029@digeo.com>
  0 siblings, 1 reply; 12+ messages in thread
From: Con Kolivas @ 2002-09-23  3:50 UTC (permalink / raw)
  To: Daniel Jacobowitz; +Cc: linux-kernel, Andrew Morton

Quoting Daniel Jacobowitz <dan@debian.org>:

> On Sun, Sep 22, 2002 at 08:41:51PM -0700, Andrew Morton wrote:
> > Con Kolivas wrote:
> > > 
> > > I performed contest benchmarks on kernel 2.5.38 when the kernel is
> compiled with
> > > gcc3.2 to gcc2.95.3
> > > 
> > > warning: The following benchmarks may be disturbing to some viewers:
> > > 
> > > No Load:
> > > Kernel                  Time            CPU
> > > 2.5.38                  68.25           99%
> > > 2.5.38-gcc32            103.03          99%
> > > 
> > 
> > Try gcc-2.91.66.  It might break the 45 second mark.
> > 
> > > IO Full Load:
> > > Kernel                  Time            CPU
> > > 2.5.38                  170.21          42%
> > > 2.5.38-gcc32            1405.25         8%
> > 
> > The streaming write is stalling gcc's read for long enough for gcc's
> > working set to be evicted.  And the working set cannot be reestablished
> > because the streaming write prevents it.  Meltdown.
> > 
> > I have fixed this.  Hang around.
> 
> That sounds like a slowdown in _running_ GCC... The worrisome part of
> Con's post is that he was talking about which compiler he'd built the
> kernel with.  Right?

Correct. contest was run with gcc2.95.3 only. The kernels were compiled with
2.95.3 and 3.2 respectively.

Con.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
       [not found]           ` <3D8E988F.DCB3196D@digeo.com>
@ 2002-09-23  5:13             ` Con Kolivas
  2002-09-23  7:20               ` Axel H. Siebenwirth
  0 siblings, 1 reply; 12+ messages in thread
From: Con Kolivas @ 2002-09-23  5:13 UTC (permalink / raw)
  To: Andrew Morton; +Cc: linux-kernel, Ville Herva, Daniel Jacobowitz, Robert Love

Quoting Andrew Morton <akpm@digeo.com>:

> Con Kolivas wrote:
> > 
> > Quoting Andrew Morton <akpm@digeo.com>:
> > 
> > > Con Kolivas wrote:
> > > >
> > > > Correct. contest was run with gcc2.95.3 only. The kernels were
> compiled
> > > with
> > > > 2.95.3 and 3.2 respectively.
> > >
> > > I think you made a mistake.  Please rerun. Just one data point will do.
> > >
> > 
> > Ok here are two points to confirm the results and their reproducibility:
> > 
> > No Load:
> > 2.5.38                  68.25           99%
> > 2.5.38-gcc32            103.03          99%
> > 2.5.38-gcc32a           103.47          99%
> > 
> > Process Load:
> > 2.5.38                  71.60           95%
> > 2.5.38-gcc32            112.98          91%
> > 2.5.38-gcc32a           113.60          91%
> > 
> 
> beats the hell out of me.  Frankly, I *still* think
> you made a mistake (at least on the io load thing)
> because the CPU time went down by a mile - it was
> waiting on disk all the time.

If you think I've made a mistake then you're probably correct. I'm investigating
this further. Please do NOT pass judgement on these benchmarks until I
completely retest everything, ensuring gcc is fixed for everything except the
kernel being tested. Disregard until I have a fresh set of confirmed results.

Con.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  5:13             ` Con Kolivas
@ 2002-09-23  7:20               ` Axel H. Siebenwirth
  0 siblings, 0 replies; 12+ messages in thread
From: Axel H. Siebenwirth @ 2002-09-23  7:20 UTC (permalink / raw)
  To: Con Kolivas
  Cc: Andrew Morton, linux-kernel, Ville Herva, Daniel Jacobowitz, Robert Love


Con Kolivas wrote:

> If you think I've made a mistake then you're probably correct. I'm investigating
> this further. Please do NOT pass judgement on these benchmarks until I
> completely retest everything, ensuring gcc is fixed for everything except the
> kernel being tested. Disregard until I have a fresh set of confirmed results.

And do not forget to use gcc from CVS, they're at 3.2.1 (prerelease) 
right now. Or even checkout 3.3 while you're at it.

Regards,
Axel


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23  3:16   ` Con Kolivas
@ 2002-09-23 11:06     ` Mark Veltzer
  2002-09-23 13:56       ` Daniel Jacobowitz
  0 siblings, 1 reply; 12+ messages in thread
From: Mark Veltzer @ 2002-09-23 11:06 UTC (permalink / raw)
  To: Con Kolivas, Linux kernel mailing list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Monday 23 September 2002 06:16, Con Kolivas wrote:

> >
> > Ugh??  Something is _seriously_ messed up here.
>

The most important question to ask here is: What flags did you compile both 
?!? I wouldn't count on the flags that were designed for gcc 2.95 to be any 
good for 3.2... Could the original poster comment on this ?

Any GCC maintainers on this list to comment ? Is there any set of flags to be 
passed to gcc 3.2 to replicate 2.95 behaviour ? I wouldn't rule out gcc 3.2 
having a totaly different set of optimizations geared towards user space C++. 
Again, any gcc maintainers comments ?!? 

Since most of the code in gcc is for C++ most of the changes in gcc should 
have been geared towards C++ (yes - quite a monstrous language). It seems to 
me that the changes in C compilation between 2.95 and 3.2 should be minor 
EXCEPT in terms of C optimization. Can anyone with assembly knowledge take 
apart two identical drivers and see the better machine code produced by 2.95 
as compared to 3.2 ? If so - can this be reported to the gcc folk ?

It seems to me that the difference is so huge that even user space 
applications could show the difference. I suggest compiling a large C program 
(emphasis on the C) in user space and doing the comparison... I would guess 
that this should have been done by the gcc folk but because of the 
hideousness of the C++ language I would guess that they mostly concentrated 
on C++ and didn't bother to benchmark regular C optimization. This is quite 
awful as the bulk of lower level open source code is in C and not C++ so this 
kind of test has a lot of meaning for any distribution that is going to be 
based on gcc 3.2...

If this benchmark turns out to be right then it seems to me that the only 
conclusion is that the gcc folk let their interest in aesoteric features of 
C++ (which has about 1/2 a billion of those) override the basic need for 
strong C optimization. Yes - it now seems that the C++ language (which is 
quite an abomination in terms of engineering and the KISS principle) is 
actually hurting open source (which has been my conclusion for quite some 
time).

Mark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQE9jvWZxlxDIcceXTgRAnxpAKDYz61RWvceqD13Z889rwtZLOaomwCggmmj
ixt6x1e+zXewlrYCCHbiN9Y=
=snOl
-----END PGP SIGNATURE-----

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38)
  2002-09-23 11:06     ` Mark Veltzer
@ 2002-09-23 13:56       ` Daniel Jacobowitz
  0 siblings, 0 replies; 12+ messages in thread
From: Daniel Jacobowitz @ 2002-09-23 13:56 UTC (permalink / raw)
  To: Mark Veltzer; +Cc: Con Kolivas, Linux kernel mailing list

On Mon, Sep 23, 2002 at 02:06:01PM +0300, Mark Veltzer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On Monday 23 September 2002 06:16, Con Kolivas wrote:
> 
> > >
> > > Ugh??  Something is _seriously_ messed up here.
> >
> 
> The most important question to ask here is: What flags did you compile both 
> ?!? I wouldn't count on the flags that were designed for gcc 2.95 to be any 
> good for 3.2... Could the original poster comment on this ?
> 
> Any GCC maintainers on this list to comment ? Is there any set of flags to be 
> passed to gcc 3.2 to replicate 2.95 behaviour ? I wouldn't rule out gcc 3.2 
> having a totaly different set of optimizations geared towards user space C++. 
> Again, any gcc maintainers comments ?!? 
> 
> Since most of the code in gcc is for C++ most of the changes in gcc should 
> have been geared towards C++ (yes - quite a monstrous language). It seems to 
> me that the changes in C compilation between 2.95 and 3.2 should be minor 
> EXCEPT in terms of C optimization. Can anyone with assembly knowledge take 
> apart two identical drivers and see the better machine code produced by 2.95 
> as compared to 3.2 ? If so - can this be reported to the gcc folk ?
> 
> It seems to me that the difference is so huge that even user space 
> applications could show the difference. I suggest compiling a large C program 
> (emphasis on the C) in user space and doing the comparison... I would guess 
> that this should have been done by the gcc folk but because of the 
> hideousness of the C++ language I would guess that they mostly concentrated 
> on C++ and didn't bother to benchmark regular C optimization. This is quite 
> awful as the bulk of lower level open source code is in C and not C++ so this 
> kind of test has a lot of meaning for any distribution that is going to be 
> based on gcc 3.2...
> 
> If this benchmark turns out to be right then it seems to me that the only 
> conclusion is that the gcc folk let their interest in aesoteric features of 
> C++ (which has about 1/2 a billion of those) override the basic need for 
> strong C optimization. Yes - it now seems that the C++ language (which is 
> quite an abomination in terms of engineering and the KISS principle) is 
> actually hurting open source (which has been my conclusion for quite some 
> time).

Mark, if you followed the GCC development process you'd realize that
all of your above ranting about C++ is completely unfounded.  Most
people doing performance work - and there are a good number of them -
focus on language-independent optimizations and check them primarily in
C.

And I've no idea what you mean by "EXCEPT in terms of C optimization". 
First of all you're completely wrong - 3.2 supports most of C99, which
is substantial - and secondly, of course the bulk of changes in support
for a language are optimization.  And those are substantial too.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2002-09-23 13:51 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2002-09-23  3:04 [BENCHMARK] gcc3.2 v 2.95.3 (contest and linux-2.5.38) Con Kolivas
2002-09-23  3:10 ` Robert Love
2002-09-23  3:16   ` Con Kolivas
2002-09-23 11:06     ` Mark Veltzer
2002-09-23 13:56       ` Daniel Jacobowitz
2002-09-23  3:28 ` Robert Love
2002-09-23  3:41 ` Andrew Morton
2002-09-23  3:46   ` Daniel Jacobowitz
2002-09-23  3:50     ` Con Kolivas
     [not found]       ` <3D8E9158.4E3DE029@digeo.com>
     [not found]         ` <1032754853.3d8e96a520836@kolivas.net>
     [not found]           ` <3D8E988F.DCB3196D@digeo.com>
2002-09-23  5:13             ` Con Kolivas
2002-09-23  7:20               ` Axel H. Siebenwirth
2002-09-23  3:47   ` Robert Love

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).