Re: [BENCHMARK] contest 0.50 results to date

Re: [BENCHMARK] contest 0.50 results to date

[Paolo Ciarrocchi]

From: Andrew Morton <akpm@digeo.com>
> I think I'm going to have to be reminded what "Loads" and "LCPU"
> mean, please.
>From an email of Con:
The "loads" variable presented is an internal number (the absolute value is not important) and makes comparisons easier. The LCPU% is the cpu%
the load used while running. 
Note if you look for example at process_load the CPU% + LCPU% can be >100 because the load runs for longer than the kernel compile. 
However, this has been accounted for in the "loads" result, to take into account the variable extra duration the load runs relative to the kernel compile. "


> What is "2.4.19-0.24pre4"?
My fault ;-(
2.4.19-0.24pre4 is a compressed cache kernel.

Ciao,
Paolo
-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [BENCHMARK] contest 0.50 results to date

[Con Kolivas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sunday 06 Oct 2002 5:15 am, Andrew Morton wrote:
> Paolo Ciarrocchi wrote:
> > And here are my results:
> I think I'm going to have to be reminded what "Loads" and "LCPU"
> mean, please.

Loads for process_load is the number of iterations each load manages to 
succeed doing divided by 10000. Loads for mem_load is the number of times 
mem_load manages to access the ram divided by 1000. Loads for io_load is the 
approximate number of megabytes divided by 100 it writes during the kernel 
compile.  The denominator for loads was chosen to easily represent the data, 
and also correlates well with significant digits.

LCPU% is the load's cpu% usage while it is running. The load is started 3 
seconds before the kernel compile and takes a variable length of time to 
finish, so it can be more than 100-cpu%

> For these sorts of tests, I think system-wide CPU% is an interesting
> thing to track - both user and system.  If it is high then we're doing
> well - doing real work.

So total cpu% being used during the kernel compile? The cpu% + lcpu% should be 
very close to this.  However I'm not sure of the most accurate way to find 
out average total cpu% used during just the kernel compile - suggestion?

> The same isn't necessarily true of the compressed-cache kernel, because
> it's doing extra work in-kernel, so CPU load comparisons there need
> to be made with some caution.

That is clear, and also the reason I have a measure of work done by the load 
as well as just the lcpu% (which by itself is not very helpful).

> Apart from observing overall CPU occupancy, we also need to monitor
> fairness - one way of doing that is to measure the total kernel build
> elapsed time.  Another way would be to observe how much actual progress
> the streaming IO makes during the kernel build.

I believe that is what I'm already showing with the time for each load == 
kernel build time, and loads==io work done.

> What is "2.4.19-0.24pre4"?

Latest version of compressed cache. Note that in my testing of cc I used the 
optional LZO compression.

> I'd suggest that more tests be added.  Perhaps
>
> - one competing streaming read
>
> - several competing streaming reads
>
> - competing "tar cf foo ./linux"
>
> - competing "tar xf foo"
>
> - competing "ls -lR > /dev/null"
>

Sure, adding loads is easy enough. Just exactly what to add I wasn't sure 
about. I'll give those a shot soon.

> It would be interesting to test -aa kernels as well - Andrea's kernels
> tend to be well tuned.

Where time permits sure.

Regards,
Con.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9n4viF6dfvkL3i1gRArn8AJ9c+jKc/CMPxV0GWaXbVJasmBNX5QCghVYX
dbvST9mdltwuwmqEk0HHXYY=
=pcOu
-----END PGP SIGNATURE-----

Re: [BENCHMARK] contest 0.50 results to date

[Rodrigo Souza de Castro]

On Sat, Oct 05, 2002 at 12:15:30PM -0700, Andrew Morton wrote:
> Paolo Ciarrocchi wrote:
> > 
[snip]
> > mem_load:
> > Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
> > 2.4.19 [3]              161.1   80      38      2       1.26
> > 2.4.19-0.24pre4 [3]     181.2   76      253     19      1.41
> > 2.5.40 [3]              163.0   80      34      2       1.27
> > 2.5.40-nopree [3]       161.7   80      34      2       1.26
> > 
> 
> I think I'm going to have to be reminded what "Loads" and "LCPU"
> mean, please.
> 
> For these sorts of tests, I think system-wide CPU% is an interesting
> thing to track - both user and system.  If it is high then we're doing
> well - doing real work.
> 
> The same isn't necessarily true of the compressed-cache kernel, because
> it's doing extra work in-kernel, so CPU load comparisons there need
> to be made with some caution.

Agreed. 

I guess the scheduling is another important point here. Firstly, this
extra work, usually not substantial, may change a little the
scheduling in the system.

Secondly, given that compressed cache usually reduces the IO performed
by the system, it may change the scheduling depending on how much IO
it saves and on what the applications do. For example, mem_load
doesn't swap any page when compressed cache is enabled (those data are
highly compressible), turning out to use most of its CPU time
slice. In vanilla, mem_load is scheduled all the time to service page
faults.

-- 
Rodrigo

Re: [BENCHMARK] contest 0.50 results to date

[Andrew Morton]

Paolo Ciarrocchi wrote:
> 
> And here are my results:
> noload:
> Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
> 2.4.19 [3]              128.8   97      0       0       1.01
> 2.4.19-0.24pre4 [3]     127.4   98      0       0       0.99
> 2.5.40 [3]              134.4   96      0       0       1.05
> 2.5.40-nopree [3]       133.7   96      0       0       1.04
> 
> process_load:
> Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
> 2.4.19 [3]              194.1   60      134     40      1.52
> 2.4.19-0.24pre4 [3]     193.2   60      133     40      1.51
> 2.5.40 [3]              184.5   70      53      31      1.44
> 2.5.40-nopree [3]       286.4   45      163     55      2.24
> 
> io_load:
> Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
> 2.4.19 [3]              461.0   28      46      8       3.60
> 2.4.19-0.24pre4 [3]     235.4   55      26      10      1.84
> 2.5.40 [3]              293.6   45      25      8       2.29
> 2.5.40-nopree [3]       269.4   50      20      7       2.10
> 
> mem_load:
> Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
> 2.4.19 [3]              161.1   80      38      2       1.26
> 2.4.19-0.24pre4 [3]     181.2   76      253     19      1.41
> 2.5.40 [3]              163.0   80      34      2       1.27
> 2.5.40-nopree [3]       161.7   80      34      2       1.26
> 

I think I'm going to have to be reminded what "Loads" and "LCPU"
mean, please.

For these sorts of tests, I think system-wide CPU% is an interesting
thing to track - both user and system.  If it is high then we're doing
well - doing real work.

The same isn't necessarily true of the compressed-cache kernel, because
it's doing extra work in-kernel, so CPU load comparisons there need
to be made with some caution.

Apart from observing overall CPU occupancy, we also need to monitor
fairness - one way of doing that is to measure the total kernel build
elapsed time.  Another way would be to observe how much actual progress
the streaming IO makes during the kernel build.

What is "2.4.19-0.24pre4"?

I'd suggest that more tests be added.  Perhaps

- one competing streaming read

- several competing streaming reads

- competing "tar cf foo ./linux"

- competing "tar xf foo"

- competing "ls -lR > /dev/null"

It would be interesting to test -aa kernels as well - Andrea's kernels
tend to be well tuned.

Re: [BENCHMARK] contest 0.50 results to date

[Paolo Ciarrocchi]

And here are my results:
noload:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              128.8   97      0       0       1.01
2.4.19-0.24pre4 [3]     127.4   98      0       0       0.99
2.5.40 [3]              134.4   96      0       0       1.05
2.5.40-nopree [3]       133.7   96      0       0       1.04

process_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              194.1   60      134     40      1.52
2.4.19-0.24pre4 [3]     193.2   60      133     40      1.51
2.5.40 [3]              184.5   70      53      31      1.44
2.5.40-nopree [3]       286.4   45      163     55      2.24

io_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              461.0   28      46      8       3.60
2.4.19-0.24pre4 [3]     235.4   55      26      10      1.84
2.5.40 [3]              293.6   45      25      8       2.29
2.5.40-nopree [3]       269.4   50      20      7       2.10

mem_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              161.1   80      38      2       1.26
2.4.19-0.24pre4 [3]     181.2   76      253     19      1.41
2.5.40 [3]              163.0   80      34      2       1.27
2.5.40-nopree [3]       161.7   80      34      2       1.26

Comments ?

Paolo
-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

[BENCHMARK] contest 0.50 results to date

[Con Kolivas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Here are the updated contest (http://contest.kolivas.net) benchmarks with 
version 0.50

noload:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              67.7    98      0       0       1.01
2.4.19-cc [3]           67.9    97      0       0       1.01
2.5.38 [3]              72.0    93      0       0       1.07
2.5.38-mm3 [2]          71.8    93      0       0       1.07
2.5.39 [2]              72.2    93      0       0       1.07
2.5.39-mm1 [2]          72.3    93      0       0       1.08
2.5.40 [1]              72.5    93      0       0       1.08
2.5.40-mm1 [1]          72.9    93      0       0       1.09

process_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              106.5   59      112     43      1.59
2.4.19-cc [3]           105.0   59      110     42      1.56
2.5.38 [3]              89.5    74      34      28      1.33
2.5.38-mm3 [1]          86.0    78      29      25      1.28
2.5.39 [2]              91.2    73      36      28      1.36
2.5.39-mm1 [2]          92.0    73      37      29      1.37
2.5.40 [2]              82.8    80      25      23      1.23
2.5.40-mm1 [2]          86.9    77      30      25      1.29

io_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              492.6   14      38      10      7.33
2.4.19-cc [3]           156.0   48      12      10      2.32
2.5.38 [1]              4000.0  1       500     1       59.55
2.5.38-mm3 [1]          303.5   25      23      11      4.52
2.5.39 [2]              423.9   18      30      11      6.31
2.5.39-mm1 [2]          550.7   14      44      12      8.20
2.5.40 [1]              315.7   25      22      10      4.70
2.5.40-mm1 [1]          326.2   24      23      11      4.86

mem_load:
Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.4.19 [3]              100.0   72      33      3       1.49
2.4.19-cc [3]           92.7    76      146     21      1.38
2.5.38 [3]              107.3   70      34      3       1.60
2.5.38-mm3 [1]          100.3   72      27      2       1.49
2.5.39 [2]              103.1   72      31      3       1.53
2.5.39-mm1 [2]          103.3   72      32      3       1.54
2.5.40 [2]              102.5   72      31      3       1.53
2.5.40-mm1 [2]          107.7   68      29      2       1.60

Note the io_load value for 2.5.38 was an estimate as every time I tried to run 
it it took too long and I stopped it (the longest I waited was 4000 seconds); 
showing very clearly the write starves read problem.

Of most interest is the performance of 2.4.19 with the latest version of 
compressed cache under mem_load (2.4.19-cc). Note that although the 
performance is only slightly better timewise, the difference in actual work 
done by the background load during that time is _enormous_. This demonstrates 
most clearly the limitations in previous versions of contest.

Comments?
Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9nn+8F6dfvkL3i1gRApHxAJ9CANpp1CA+chu+DxEghiNXgP0VjwCfdHsm
qf7yp7W6sBOnkNx/cmTLPQY=
=7oEd
-----END PGP SIGNATURE-----