[chatroom benchmark version 1.0.1] Results

[chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

Hi all,
yesterday I played a bit with the chatroom benchmark.
I don't want to discuss about number (you are the right people to do that, not me).

--- What is the chat bench ---
Usage: chat_c ip_addr [num_rooms] [num_messages] [server_port]

       ip_addr:		server ip address that the client connects to.
       [num_rooms]:	the number of chat rooms.
			default is 10.
       [num_messages]:	the number of messages sent by each chat room member.
			default is 100.	
       [port]:		server port number the client connects to.
			default is 9999.

This benchmark includes both a client and server.  The benchmark is
modeled after a chat room.  This benchmark will create a lot of threads,
tcp connections, and send and receive a lot of messages.  The client side
of the benchmark will report the number of messages sent per second.

The number of chat rooms and messages will control the workload.
The default number of chat rooms is 10.
The default number of messages is 100.  The size of each message is 100 bytes.
Both of these parameters are specified on the client side.

(1) Each chat room has 20 users.
(2) 10 chat rooms represents 20*10 or 200 users.
(3) For each user in the chat room, the client will make a connection
    to the server.
(4) 10 chat rooms will create 10*20 or 200 connections between the client
    and server.
(5) For each user (or connection) in the chat room, 1 'send' thread is created
    and 1 'receive' thread is created.
(6) 10 chat rooms will create 10*20*2 or 400 client threads and 400 server
    threads for a total of 800 threads.
(7) Each client 'send' thread will send the specified number of messages
    to the server.
(8) For 10 chat rooms and 100 messages, the client will send 10*20*100 or
    20,000 messages.  The server 'receive' thread will receive the
    corresponding number of messages.
(9) The chat room server will echo each of the messages back to the other
    users in the chat room.
(10) For 10 chat rooms and 100 messages, the server 'send' thread will send
     10*20*100*19 or 380,000 messages.  The client 'receive' thread will
     receive the corresponding number of messages.
(11) The client will report the message throughput at the end of the test.
(12) The server loops and accepts another start message from the client.

What I did:
Boot in single mode with apm off
./chat_s 127.0.0.1 &
./chat_c 127.0.0.1 30 1000 9999

And now the results:
2.4.19-ck7.results:Average throughput : 55928 messages per second
2.4.19.results:Average throughput : 44851 messages per second
2.5.33.results:Average throughput : 59522 messages per second
2.5.34.results:Average throughput : 62941 messages per second
2.5.36.results:Average throughput : 60858 messages per second

2.4.19-ck7 is preemption ON
2.5.33 and 2.5.34 are preemption ON
2.5.36 is preemption OFF (Robert, 2.5.36 with preemption ON oops at boot)

Ciao,
            Paolo


-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [chatroom benchmark version 1.0.1] Results

[Daniel Phillips]

Just thought I'd bounce this one over to Andrew as a warm-n-fuzzy.
There's also some kind of hint that preemption improves throughput
here.

On Thursday 19 September 2002 10:06, Paolo Ciarrocchi wrote:
> What I did:
> Boot in single mode with apm off
> ./chat_s 127.0.0.1 &
> ./chat_c 127.0.0.1 30 1000 9999
> 
> And now the results:
> 2.4.19-ck7.results:Average throughput : 55928 messages per second
> 2.4.19.results:Average throughput : 44851 messages per second
> 2.5.33.results:Average throughput : 59522 messages per second
> 2.5.34.results:Average throughput : 62941 messages per second
> 2.5.36.results:Average throughput : 60858 messages per second
> 
> 2.4.19-ck7 is preemption ON
> 2.5.33 and 2.5.34 are preemption ON
> 2.5.36 is preemption OFF (Robert, 2.5.36 with preemption ON oops at boot)

-- 
Daniel

Re: [chatroom benchmark version 1.0.1] Results

[Andrew Morton]

Daniel Phillips wrote:
> 
> Just thought I'd bounce this one over to Andrew as a warm-n-fuzzy.

Is a scheduler test, isn't it?

> There's also some kind of hint that preemption improves throughput
> here.

One needs to treat any test which involves networking to localhost
with caution.  They tend to show large (+/- 10% or more) swings
in throughput from one run to the next.  Some sort of cache
associativity thing; not sure.

Running the test between separate machines is much, much more repeatable.

We'll get the cpu-local hot pages list code going soon; that may
provide some benefit to this sort of thing.  Even on uniprocessor.

Re: [chatroom benchmark version 1.0.1] Results

[Daniel Phillips]

On Thursday 19 September 2002 10:29, Andrew Morton wrote:
> Daniel Phillips wrote:
> > 
> > Just thought I'd bounce this one over to Andrew as a warm-n-fuzzy.
> 
> Is a scheduler test, isn't it?

Looks like it's inspired by volano, and yes, mainly scheduling, but I
suppose the messages put a little load on the VM as well.

-- 
Daniel

Re: [chatroom benchmark version 1.0.1] Results

[Rusty Russell]

On Thu, 19 Sep 2002 01:29:30 -0700
Andrew Morton <akpm@digeo.com> wrote:

> Daniel Phillips wrote:
> > 
> > Just thought I'd bounce this one over to Andrew as a warm-n-fuzzy.
> 
> Is a scheduler test, isn't it?

Mixed.  Before the O(1) scheduler, I distilled this test into "hackbench"
(which I now call schedbench).  Do well on this, and you'll do well on the
lots-of-threads-wakingup case: [BTW, 2.5 rocks at this 8)].  Ideally
"hackbench n" should scale with n:

/* Test groups of 20 processes spraying to 20 receivers */
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <sys/time.h>
#include <sys/select.h>

#define DATASIZE 100
static unsigned int loops = 100;
static int use_pipes = 0;

static void barf(const char *msg)
{
	fprintf(stderr, "%s (error: %s)\n", msg, strerror(errno));
	exit(1);
}

static void fdpair(int fds[2])
{
	if (use_pipes) {
		if (pipe(fds) == 0)
			return;
	} else {
		if (socketpair(AF_UNIX, SOCK_STREAM, 0, fds) == 0)
			return;
	}
	barf("Creating fdpair");
}

/* Block until we're ready to go */
static void ready(int ready_out, int wakefd)
{
	char dummy;
	fd_set fds;

	FD_ZERO(&fds);
	FD_SET(wakefd, &fds);

	/* Tell them we're ready. */
	if (write(ready_out, &dummy, 1) != 1)
		barf("CLIENT: ready write");

	/* Wait for "GO" signal */
	if (select(wakefd+1, &fds, NULL, NULL, NULL) != 1)
		barf("select");
}

/* Sender sprays loops messages down each file descriptor */
static void sender(unsigned int num_fds,
		   int out_fd[num_fds],
		   int ready_out,
		   int wakefd)
{
	char data[DATASIZE];
	unsigned int i, j;

	ready(ready_out, wakefd);

	/* Now pump to every receiver. */
	for (i = 0; i < loops; i++) {
		for (j = 0; j < num_fds; j++) {
			int ret, done = 0;

		again:
			ret = write(out_fd[j], data + done, sizeof(data)-done);
			if (ret < 0)
				barf("SENDER: write");
			done += ret;
			if (done < sizeof(data))
				goto again;
		}
	}
}

/* One receiver per fd */
static void receiver(unsigned int num_packets,
		     int in_fd,
		     int ready_out,
		     int wakefd)
{
	unsigned int i;

	/* Wait for start... */
	ready(ready_out, wakefd);

	/* Receive them all */
	for (i = 0; i < num_packets; i++) {
		char data[DATASIZE];
		int ret, done = 0;

	again:
		ret = read(in_fd, data + done, DATASIZE - done);
		if (ret < 0)
			barf("SERVER: read");
		done += ret;
		if (done < DATASIZE)
			goto again;
	}
}

/* One group of senders and receivers */
static unsigned int group(unsigned int num_fds,
			  int ready_out,
			  int wakefd)
{
	unsigned int i;
	unsigned int out_fds[num_fds];

	for (i = 0; i < num_fds; i++) {
		int fds[2];

		/* Create the pipe between client and server */
		fdpair(fds);

		/* Fork the receiver. */
		switch (fork()) {
		case -1: barf("fork()");
		case 0:
			close(fds[1]);
			receiver(num_fds*loops, fds[0], ready_out, wakefd);
			exit(0);
		}

		out_fds[i] = fds[1];
		close(fds[0]);
	}

	/* Now we have all the fds, fork the senders */
	for (i = 0; i < num_fds; i++) {
		switch (fork()) {
		case -1: barf("fork()");
		case 0:
			sender(num_fds, out_fds, ready_out, wakefd);
			exit(0);
		}
	}

	/* Close the fds we have left */
	for (i = 0; i < num_fds; i++)
		close(out_fds[i]);

	/* Return number of children to reap */
	return num_fds * 2;
}

int main(int argc, char *argv[])
{
	unsigned int i, num_groups, total_children;
	struct timeval start, stop, diff;
	unsigned int num_fds = 20;
	int readyfds[2], wakefds[2];
	char dummy;

	if (argv[1] && strcmp(argv[1], "-pipe") == 0) {
		use_pipes = 1;
		argc--;
		argv++;
	}

	if (argc != 2 || (num_groups = atoi(argv[1])) == 0)
		barf("Usage: hackbench [-pipe] <num groups>\n");

	fdpair(readyfds);
	fdpair(wakefds);

	total_children = 0;
	for (i = 0; i < num_groups; i++)
		total_children += group(num_fds, readyfds[1], wakefds[0]);

	/* Wait for everyone to be ready */
	for (i = 0; i < total_children; i++)
		if (read(readyfds[0], &dummy, 1) != 1)
			barf("Reading for readyfds");

	gettimeofday(&start, NULL);

	/* Kick them off */
	if (write(wakefds[1], &dummy, 1) != 1)
		barf("Writing to start them");

	/* Reap them all */
	for (i = 0; i < total_children; i++) {
		int status;
		wait(&status);
		if (!WIFEXITED(status))
			exit(1);
	}

	gettimeofday(&stop, NULL);

	/* Print time... */
	timersub(&stop, &start, &diff);
	printf("Time: %lu.%03lu\n", diff.tv_sec, diff.tv_usec/1000);
	exit(0);
}



-- 
   there are those who do and those who hang on and you don't see too
   many doers quoting their contemporaries.  -- Larry McVoy

Re: [chatroom benchmark version 1.0.1] Results

[Martin J. Bligh]

>> And now the results:
>> 2.4.19-ck7.results:Average throughput : 55928 messages per second
>> 2.4.19.results:Average throughput : 44851 messages per second
>> 2.5.33.results:Average throughput : 59522 messages per second
>> 2.5.34.results:Average throughput : 62941 messages per second
>> 2.5.36.results:Average throughput : 60858 messages per second
>> 
>> 2.4.19-ck7 is preemption ON
>> 2.5.33 and 2.5.34 are preemption ON
>> 2.5.36 is preemption OFF (Robert, 2.5.36 with preemption ON oops at boot)
>
> There's also some kind of hint that preemption improves throughput
> here.

Any chance of doing two runs on exactly the same kernel, one with 
preempt on, and the other with preempt off? That's a much nicer 
hint ;-)

M.

Re: [chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

From: Daniel Phillips <phillips@arcor.de>

[...]
> There's also some kind of hint that preemption improves throughput
> here.
I'm not so sure, but I'm not a developer,
just looking to the number:
 
> > 2.5.34.results:Average throughput : 62941 messages per second
> > 2.5.36.results:Average throughput : 60858 messages per second
2.5.34 and 2.5.36 have more or less the same performances... (2.5.34 is preemption ON, 2.5.36 is OFF)

There is an improovement in throughput in 2.5.3* against 2.4.*, and this is _good_ ;-)

Ciao,
       Paolo
-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

From: "Martin J. Bligh" <mbligh@aracnet.com>
[...]
> > There's also some kind of hint that preemption improves throughput
> > here.
> 
> Any chance of doing two runs on exactly the same kernel, one with 
> preempt on, and the other with preempt off? That's a much nicer 
> hint ;-)
Yep! you are rigth, but I can performe this test only with 2.5.33 and 2.5.34.
2.5.36 oops at boot if the preemption is ON. Robert!!

But, do this test using local network produce results very variable (5-10 %).

Ciao,
           Paolo
-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

From: "Martin J. Bligh" <mbligh@aracnet.com>
[...]
> Any chance of doing two runs on exactly the same kernel, one with 
> preempt on, and the other with preempt off? That's a much nicer 
> hint ;-)

Here is the result from 2.5.33

2.5.33-no_preemption.results:Average throughput :59695 messages per second

2.5.33.results:Average throughput : 59522 messages per second

No benefit from preemption.

Anyway, I'll back tomorrow with the results from 2.5.36 (with and withou preemption) running the test 
10 times and evaluating the average.
I'm gonna using this script:
#!/bin/sh
> `uname -r`_total.results
for i in `seq 1 1 10`
do
	./chat_c 127.0.0.1 30 1000 9999 >>`uname -r`_total.results
done
grep Average `uname -r`_total.results |cut -d " " -f 4| awk '{tot+=$1}; END {print "Average throughput: " tot/NR " messages per second"}'

If you have any suggestion, just let me know.

Ciao,
        Paolo


-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

Hi all,
I'm back with the results of the following script:
#!/bin/sh
> `uname -r`_total.results
for i in `seq 1 1 10`
do
	./chat_c 127.0.0.1 30 1000 9999 >>`uname -r`_total.results
done
grep Average `uname -r`_total.results | awk '{tot+=$4}; END {print "Average throughput: " tot/NR " messages per second"}' > `uname -r`.average

Here the results:
2.5.33-preemption.average:Average throughput: 60943.9 messages per second
2.5.33.average:Average throughput: 61779.8 messages per second
2.5.36-preemption.average:Average throughput: 60877.2 messages per second
2.5.36.average:Average throughput: 60858.7 messages per second
2.5.37-preemption.average: Average throughput: 61896.1 messages per second

Comments?

Ciao,
            Paolo
-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze

Re: [chatroom benchmark version 1.0.1] Results

[Paolo Ciarrocchi]

Hi all,
someone suggested me to report other information and not only the average (I have no time to write a script in order to evaluate the dev...any help?)

Therefore here it goes the full log:

2.5.33_total.results:Average throughput : 60402 messages per second
2.5.33_total.results:Average throughput : 63981 messages per second
2.5.33_total.results:Average throughput : 60271 messages per second
2.5.33_total.results:Average throughput : 61403 messages per second
2.5.33_total.results:Average throughput : 61235 messages per second
2.5.33_total.results:Average throughput : 60603 messages per second
2.5.33_total.results:Average throughput : 61258 messages per second
2.5.33_total.results:Average throughput : 61675 messages per second
2.5.33_total.results:Average throughput : 62447 messages per second
2.5.33_total.results:Average throughput : 64523 messages per second

2.5.36-preemption_total.results:Average throughput : 58717 messages per second
2.5.36-preemption_total.results:Average throughput : 62171 messages per second
2.5.36-preemption_total.results:Average throughput : 63255 messages per second
2.5.36-preemption_total.results:Average throughput : 60643 messages per second
2.5.36-preemption_total.results:Average throughput : 61854 messages per second
2.5.36-preemption_total.results:Average throughput : 63547 messages per second
2.5.36-preemption_total.results:Average throughput : 55841 messages per second
2.5.36-preemption_total.results:Average throughput : 62911 messages per second
2.5.36-preemption_total.results:Average throughput : 55490 messages per second
2.5.36-preemption_total.results:Average throughput : 64343 messages per second

2.5.36_total.results:Average throughput : 60695 messages per second
2.5.36_total.results:Average throughput : 60356 messages per second
2.5.36_total.results:Average throughput : 59959 messages per second
2.5.36_total.results:Average throughput : 61132 messages per second
2.5.36_total.results:Average throughput : 60486 messages per second
2.5.36_total.results:Average throughput : 60378 messages per second
2.5.36_total.results:Average throughput : 63030 messages per second
2.5.36_total.results:Average throughput : 60431 messages per second
2.5.36_total.results:Average throughput : 60063 messages per second
2.5.36_total.results:Average throughput : 62057 messages per second

2.5.37-preemption_total.results:Average throughput : 60660 messages per second
2.5.37-preemption_total.results:Average throughput : 62788 messages per second
2.5.37-preemption_total.results:Average throughput : 61604 messages per second
2.5.37-preemption_total.results:Average throughput : 62324 messages per second
2.5.37-preemption_total.results:Average throughput : 63011 messages per second
2.5.37-preemption_total.results:Average throughput : 61596 messages per second
2.5.37-preemption_total.results:Average throughput : 61944 messages per second
2.5.37-preemption_total.results:Average throughput : 59562 messages per second
2.5.37-preemption_total.results:Average throughput : 62385 messages per second
2.5.37-preemption_total.results:Average throughput : 63087 messages per second

2.5.38-preemption_total.results:Average throughput : 63051 messages per second
2.5.38-preemption_total.results:Average throughput : 60065 messages per second
2.5.38-preemption_total.results:Average throughput : 61488 messages per second
2.5.38-preemption_total.results:Average throughput : 63471 messages per second
2.5.38-preemption_total.results:Average throughput : 62200 messages per second
2.5.38-preemption_total.results:Average throughput : 61631 messages per second
2.5.38-preemption_total.results:Average throughput : 64894 messages per second
2.5.38-preemption_total.results:Average throughput : 63893 messages per second
2.5.38-preemption_total.results:Average throughput : 63127 messages per second
2.5.38-preemption_total.results:Average throughput : 61565 messages per second

And here it goes the average:
2.5.33-preemption.average:Average throughput: 60943.9 messages per second
2.5.33.average:Average throughput: 61779.8 messages per second
2.5.36-preemption.average:Average throughput: 60877.2 messages per second
2.5.36.average:Average throughput: 60858.7 messages per second
2.5.37-preemption.average:Average throughput: 61896.1 messages per second
2.5.38-preemption.average:Average throughput: 62538.5 messages per second

Ciao,
          Paolo

-- 
Get your free email from www.linuxmail.org 


Powered by Outblaze