Re: [khttpd-users] khttpd vs tux

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

> I don't mean an cluster of PC's, depending on the volume for the download
> exist some pretty "cache-pox" that mean no load balancing but the box can
> cache up to 1-4 GB ob data in the ram and the "web-server" there is
> running
> in hard wired asic's i think P5, Cisco and another Producer build it.
> Becasue in my opinion the traffic of up to 4gbit is not handable on an
> linux box (thinking on x86 achritekture)

hm...

how much do you think you can get out of a server with several 1Gb
ethernet cards, multiple 66MHz/64bit PCI busses, multiple SCSI busses or
perhaps some sort of SAN solution based on FibreChannel 2?

---
Computers are like air conditioners.
They stop working when you open Windows.

Re: [khttpd-users] khttpd vs tux

[Thomas Lussnig]

>
>
>how much do you think you can get out of a server with several 1Gb
>ethernet cards, multiple 66MHz/64bit PCI busses, multiple SCSI busses or
>perhaps some sort of SAN solution based on FibreChannel 2?
>
Ok,
on this hardware i think that the problem is the that the Kernel and
Webserver need to suport that ( each of the 1Gbit card is bound to its
own process and on Multiprozessor machine that the prozess is fixed to
one CPU to minimize the siwtch overhead, also im not firm with the 
FibreChannel2
spezifikation i think that there can some trouble with the load, but much
more important is to know how much different data is served, because then
you talk about khttpd i think that it is definit static data and so the 
question
is how much, because on an ideal case the whole set of files is cached 
in the
ram, with 500 hundred Users i think there is only minmal patch in the 
kernel to
do for higher file handles. So if there is only there the choice left open
tux or khttpd i think you should use tux

- more defelopment
- more tuning/config/log options
- better code ( khttpd soud's a little bit of try and error )

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

>> how much do you think you can get out of a server with several 1Gb
>> ethernet cards, multiple 66MHz/64bit PCI busses, multiple SCSI busses or
>> perhaps some sort of SAN solution based on FibreChannel 2?
>
> Ok,
> on this hardware i think that the problem is the that the Kernel and
> Webserver need to suport that ( each of the 1Gbit card is bound to its
> own process and on Multiprozessor machine that the prozess is fixed to
> one CPU to minimize the siwtch overhead, also im not firm with the
> FibreChannel2
> spezifikation i think that there can some trouble with the load, but much
> more important is to know how much different data is served, because then
> you talk about khttpd i think that it is definit static data and so the
> question
> is how much, because on an ideal case the whole set of files is cached
> in the
> ram, with 500 hundred Users i think there is only minmal patch in the
> kernel to
> do for higher file handles. So if there is only there the choice left open
> tux or khttpd i think you should use tux

What's this patch thing?
Do I need to patch up or rewrite parts of the kernel to support <1000 file
handles?

---
Computers are like air conditioners.
They stop working when you open Windows.

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

> Each GigE card will need its own 66MHz PCI bus. Each PCI bridge will need
> to be coming off a memory bus that can sustain all of these and the CPU
> at once.
>
> At that point it really doesnt look much like a PC.

How much raw speed do you think I can manage to get out of a really cool
n-way server from Compaq? I beleive we'll go for a Compaq server, as
that's what's been decided some time ago.

I read something by Linus about linux scalability, and I beleive he said
that 'linux [2.4] scales good up to 4 cpus, but not that good futher on
[to 8?]'. Can anyone fill in the holes here?

thanks

roy

---
Computers are like air conditioners.
They stop working when you open Windows.

Re: [khttpd-users] khttpd vs tux

[Alan Cox]

> on this hardware i think that the problem is the that the Kernel and
> Webserver need to suport that ( each of the 1Gbit card is bound to its
> own process and on Multiprozessor machine that the prozess is fixed to
> one CPU to minimize the siwtch overhead, also im not firm with the 
> FibreChannel2

Each GigE card will need its own 66MHz PCI bus. Each PCI bridge will need
to be coming off a memory bus that can sustain all of these and the CPU
at once.

At that point it really doesnt look much like a PC.

Re: [khttpd-users] khttpd vs tux

[J Sloan]

Roy Sigurd Karlsbakk wrote:

> I read something by Linus about linux scalability, and I beleive he said
> that 'linux [2.4] scales good up to 4 cpus, but not that good futher on
> [to 8?]'. Can anyone fill in the holes here?

Nobody scales better 1-4 CPUs, as indicated
by specweb99 - at 8 CPUs linux is OK, but not
as dominating....

When the high end specialists from IBM etc
can send in patches that enhance high end
performance without hurting the low end case
the numbers on 8-32 CPUs should really start
to shine. (There has been progress on that
front seen on lkml)

cu

jjs

Re: [khttpd-users] khttpd vs tux

[Alan Cox]

> > At that point it really doesnt look much like a PC.
> 
> How much raw speed do you think I can manage to get out of a really cool
> n-way server from Compaq? I beleive we'll go for a Compaq server, as
> that's what's been decided some time ago.

Take a look at the tux benchmark numbers. Thats pushing the limit of the
hardware

Re: [khttpd-users] khttpd vs tux

[Erik Mouw]

On Sat, Nov 03, 2001 at 08:18:19PM +0100, Roy Sigurd Karlsbakk wrote:
> > Each GigE card will need its own 66MHz PCI bus. Each PCI bridge will need
> > to be coming off a memory bus that can sustain all of these and the CPU
> > at once.
> >
> > At that point it really doesnt look much like a PC.
> 
> How much raw speed do you think I can manage to get out of a really cool
> n-way server from Compaq? I beleive we'll go for a Compaq server, as
> that's what's been decided some time ago.

Not that much. Alan's point is that you're pushing the limit of the
memory bandwidth, not the number of CPUs. This is the single reason
that high traffic websites either use some serious non-PC hardware (IBM
Z-series, for example) or a large number of PCs in parallel to share
the load.

> I read something by Linus about linux scalability, and I beleive he said
> that 'linux [2.4] scales good up to 4 cpus, but not that good futher on
> [to 8?]'. Can anyone fill in the holes here?

The number of CPUs really doesn't matter in this case. With several
GigE cards memory bandwidth and latency is your main problem.


Erik

-- 
J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: J.A.K.Mouw@its.tudelft.nl
WWW: http://www-ict.its.tudelft.nl/~erik/

Re: [khttpd-users] khttpd vs tux

[Alan Cox]

> Nobody scales better 1-4 CPUs, as indicated
> by specweb99 - at 8 CPUs linux is OK, but not
> as dominating....

At specweb. For some 2 and a large number of 4 processor workloads our
scheduler does not make good decisions

Re: [khttpd-users] khttpd vs tux

[John Alvord]

On Sun, 4 Nov 2001, Erik Mouw wrote:

> On Sat, Nov 03, 2001 at 08:18:19PM +0100, Roy Sigurd Karlsbakk wrote:
> > > Each GigE card will need its own 66MHz PCI bus. Each PCI bridge will need
> > > to be coming off a memory bus that can sustain all of these and the CPU
> > > at once.
> > >
> > > At that point it really doesnt look much like a PC.
> > 
> > How much raw speed do you think I can manage to get out of a really cool
> > n-way server from Compaq? I beleive we'll go for a Compaq server, as
> > that's what's been decided some time ago.
> 
> Not that much. Alan's point is that you're pushing the limit of the
> memory bandwidth, not the number of CPUs. This is the single reason
> that high traffic websites either use some serious non-PC hardware (IBM
> Z-series, for example) or a large number of PCs in parallel to share
> the load.
> 
> > I read something by Linus about linux scalability, and I beleive he said
> > that 'linux [2.4] scales good up to 4 cpus, but not that good futher on
> > [to 8?]'. Can anyone fill in the holes here?
> 
> The number of CPUs really doesn't matter in this case. With several
> GigE cards memory bandwidth and latency is your main problem.

Interesting parallel...

In the last few years there have been multiple cases where people reported
benchmarks where a dual processir gave less thruput then a single
processor. In most cases, the single processor benchmark had saturated the
memory bandwidth and a second processor didn't make much difference.

This was on "cheap" multi-processors.

john alvord

Re: [khttpd-users] khttpd vs tux

[Ingo Molnar]

On Sat, 3 Nov 2001, J Sloan wrote:

> Nobody scales better 1-4 CPUs, as indicated
> by specweb99 - at 8 CPUs linux is OK, but not
> as dominating....

This is a common misinterpretation of the TUX SPECweb99 numbers.
Performance and scalability are two distinct things. Also, maximum
performance on a given hardware, and the true scalability of the software
running on it are two different things as well. You can have a very slow
webserver that scales very well - and you can have a fast webserver that
scales poorly, but if the fast one beats the slow one even with the
highest number of CPUs used, the 'good' scalability of the slow webserver
does not matter much, does it? Also, TUX will max out an i486 pretty
quickly, and it will scale very badly on 4-way i486 systems (yes such
beasts do exist), simply because the hardware itself is pushed to the
maximum, more CPUs simply do not help - performance does not increase.

Ideally we want to have a very fast and very scalable webserver - TUX is
an attempt to be just that, and nothing more.

TUX maxes out the hardware on all systems tested so far - so the true
'scalability' of the Linux kernel and TUX simply cannot be measured: it's
the hardware (CPU, networking card, etc.) that is slowing TUX down, not
TUX's scalability faults. Algorithmically and SMP caching/locking-wise the
kernel and TUX is doing the right thing already, under these read-mostly
pagecache & TCP/IP loads. [well, this is not some black art, we simply
fixed every limit that showed up on the way.]

TUX maxes out 2-way and 4-way systems as well, while IIS does not appear
to do a good job there. So we can say that it's proven that IIS does not
scale well. I can still not say whether Linux+TUX scales well, i can only
say that it's too fast for the given hardware :-)

why does it look like as if TUX scaled well on 1, 2, 4 CPUs? Because
hardware designers are sizing up systems with more CPUs, so the true
limits of the hardware show a similar scalability graph as the scalability
graph would be of a scalable webserver.

Scalability of the software can only be judged on hardware where every
component (CPU, system board, cards) is faster than what TUX can push - so
it can be measured exactly how TUX (and the kernel) reacts to the addition
of more CPUs. Once a webserver pushes to the limits of the hardware, the
true scalability of the code gets distorted.

> When the high end specialists from IBM etc
> can send in patches that enhance high end
> performance without hurting the low end case
> the numbers on 8-32 CPUs should really start
> to shine. [...]

sadly, the TUX workloads scale 'perfectly' already both within TUX and
within the kernel (to the best of my knowledge), from an algorithmic point
of view - i dont think anyone could claim to be able to improve that
significantly, even on 32 way systems. My main development box is an 8-way
ia32 box (and a fair number of other kernel hackers have such boxes as
well), so we know the 8-way limits pretty well. Note that the TUX patches
include 3 extra scalability patches to the stock kernel:

 - the pagecache SMP-scalability patch [gets rid of pagecache_lock]
 - the smptimers patch [makes timers completely per-CPU.]
 - the per-CPU page allocator

There might be other areas in the kernel that could scale better under
non-TUX workloads (especially the block IO code has some scalability
problems), but none of them affects TUX in any measurable way on the
systems we measured. I'd say that TUX should scale pretty well to 16 or 32
CPUs, and SGI's tests appear to prove this in part: the pagecache
scalability patch alone helped their (non-TUX) NUMA cached-dbench
performance measurably. [on an 8-way system the pagecache scalability
patch is only a small but measurable win.] And if any kernel scalability
limit pops up on bigger boxes, we can fix it - there are few fundamental
issues left.

	Ingo

Re: [khttpd-users] khttpd vs tux

[J Sloan]

Ingo,

Thanks for commenting on this -

Ingo Molnar wrote:

> On Sat, 3 Nov 2001, J Sloan wrote:
>
> > Nobody scales better 1-4 CPUs, as indicated
> > by specweb99 - at 8 CPUs linux is OK, but not
> > as dominating....
>
> This is a common misinterpretation of the TUX SPECweb99 numbers.
> Performance and scalability are two distinct things.

Absolutely correct, I spoke sloppily.
I should have said, "nobody performs better...".

But the scalability certainly _appears_
to be better than average -

> TUX maxes out 2-way and 4-way systems as well, while IIS does not appear
> to do a good job there. So we can say that it's proven that IIS does not
> scale well. I can still not say whether Linux+TUX scales well, i can only
> say that it's too fast for the given hardware :-)

indeed...

> why does it look like as if TUX scaled well on 1, 2, 4 CPUs? Because
> hardware designers are sizing up systems with more CPUs, so the true
> limits of the hardware show a similar scalability graph as the scalability
> graph would be of a scalable webserver.

Excellent point, thanks for making the distinction.

Thanks as well for the other excellent insights,
it was informative to hear what you had to say.

cu

jjs

Re: [khttpd-users] khttpd vs tux

[Chul Lee]

You had better refer a paper whose tile is High-Performance Memory-Based Web Servers: Kernel and User-Space Performance, by Philippe  at IBM.T.J Watson
They also have experiments on performance between various kernel mode web server and user mode web server.

Chul

----- Original Message ----- 
From: "Roy Sigurd Karlsbakk" <roy@karlsbakk.net>
To: <linux-kernel@vger.kernel.org>
Cc: "khttpd mailing list" <khttpd-users@zgp.org>
Sent: Sunday, November 04, 2001 1:21 AM
Subject: [khttpd-users] khttpd vs tux


> hi
> 
> can someone tell me what the difference is, in functionality, speed etc.
> between the tux (2.0?) webserver and khttpd? I'm working on a project
> where all I need is raw speed - really raw speed - and I really don't know
> which to choose.
> 
> roy
> 
> ---
> Computers are like air conditioners.
> They stop working when you open Windows.
> 
> _______________________________________________
> khttpd-users maillist  -  khttpd-users@zgp.org
> http://zgp.org/mailman/listinfo/khttpd-users

Re: [khttpd-users] khttpd vs tux

[Dirk Moerenhout]

> Are there any good reasons why to run khttpd, then?
> What I need is a server serving something between 50 and 500 concurrent
> clients - each downloading at 4-8Mbps.

Are those people doing tons of requests or are they just downloading large
files? If they are downloading large files the type of serversoftware will
be the least of your problems. About anything can give you the full
bandwidth you can put out on your networkcards when you are serving few
requests and are just pushing out mass amount of data.

When it comes to serving people such huge amount of data you should also
take in mind that buying one big machine is not allways the right road to
take.

As an example say that the data you're serving is less than 36GB in total.
In that case you can easily buy 4 typical 2U rackmount servers with 9GB
RAID1/36GB RAID1, Dual CPU, enough RAM, 1Gb NIC and pay less than a 8-Way
system.  Furthermore those 4 servers give you more redundancy (one can
literally go up in smoke and you still lose only 25% power), they will in
scale better and so on and so on.

In the end, unless you are handling tons of requests, your concern should
be what hardware servers/switches/routers you need and certainly not what
software. That discussion by itself would off course get quite off topic
for lkml.

Dirk Moerenhout ///// System Administrator ///// Planet Internet NV

Re: [khttpd-users] khttpd vs tux

[Miquel van Smoorenburg]

In article <Pine.LNX.4.30.0111031740300.8812-100000@mustard.heime.net>,
Roy Sigurd Karlsbakk  <roy@karlsbakk.net> wrote:
>> tux is more advanced than khttpd. It's also more intrusive to the kernel as
>> far as core changes are concerned. These changes allow for higher
>> performance, but you'll only notice that if you want to fill a gigabit line
>> or more.....
>
>Are there any good reasons why to run khttpd, then?
>What I need is a server serving something between 50 and 500 concurrent
>clients - each downloading at 4-8Mbps.
>Which one would be best? Anyone have an idea?

Seriously? 500*8 Mbit/sec = 4 Gbit/sec

In that case you need at least 10 boxes, each with a gigabit card,
with loadbalancing through DNS. Each box will do max. 400 mbit/sec
and have 50 clients on it - standard apache will do fine, I think.
Otherwise just add a few boxes.

You will need a Juniper M20 or a Cisco 124xx series with 2xSTM16 (OC64)
or 1x10GigE upload capacity and 10xGigE slots in it. That will cost
as much 100-200 of the Linux boxes so the Linux boxes are the least
of your worries. Not to mention the cost of 4 Gbit/sec of Internet
bandwidth.

Mike.
-- 
"Only two things are infinite, the universe and human stupidity,
 and I'm not sure about the former" -- Albert Einstein.

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

> Hi do you know about what traffic you are talking ?

Yes. I know.

> Between 200Mps and 4Gps this means you would on the
> max Limit Transver abaout 400MB each second.
> And on the lower limit 20MB.
>
> I think that you shouldn't look for an software web server
> but an GOOD hardware server ( or load balancer ), but i think
> then there is khttpd the right choice.

I don't think good hardware is enough. I mean - I can probably strap up
twenty cool servers from some hardware producer, cluster them together
with some cool software and blah blah blah, but I can't use that money. I
need a good solution - not low-cost, but still lower...

So - good hardware and good software.

---
Computers are like air conditioners.
They stop working when you open Windows.

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

> TUX has been tested more (by me, Ingo and the other Red Hat folks and on
> several live sites) and is more advanced in that it also has more features
> (virtual hosting etc etc).....

Will this eventually result in tux being merged into the official kernel
and khttpd will cease to exist?

---
Computers are like air conditioners.
They stop working when you open Windows.

Re: [khttpd-users] khttpd vs tux

[Roy Sigurd Karlsbakk]

> tux is more advanced than khttpd. It's also more intrusive to the kernel as
> far as core changes are concerned. These changes allow for higher
> performance, but you'll only notice that if you want to fill a gigabit line
> or more.....

Are there any good reasons why to run khttpd, then?
What I need is a server serving something between 50 and 500 concurrent
clients - each downloading at 4-8Mbps.

Which one would be best? Anyone have an idea?

thanks

roy

---
Computers are like air conditioners.
They stop working when you open Windows.