linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* RE: Scaling noise
@ 2003-09-03 17:07 Brown, Len
  2003-09-03 17:32 ` Larry McVoy
  0 siblings, 1 reply; 149+ messages in thread
From: Brown, Len @ 2003-09-03 17:07 UTC (permalink / raw)
  To: Larry McVoy; +Cc: Giuliano Pochini, linux-kernel

Fortunately seek time on RAM is lower than disk;-)  Sure, parallel
systems are a waste of effort for running a single copy of a single
threaded app, but when you have multiple apps, or better yet MT apps,
you win.  If system performance were limited over time to the rate of
decrease in RAM latency, then we'd be in sorry shape.

Back to the original off-topic...
An OEM can spin their motivation to focus on smaller systems in 3 ways:

1. large server sales are a small % of industry units
2. large server sales are a small % of industry revenue
3. large server sales are a small % of industry profits

Only 1 is true.

Cheers,
-Len

#include <std/disclaimer.h>


> -----Original Message-----
> From: Larry McVoy [mailto:lm@bitmover.com] 
> Sent: Wednesday, September 03, 2003 7:20 AM
> To: Brown, Len
> Cc: Giuliano Pochini; Larry McVoy; linux-kernel@vger.kernel.org
> Subject: Re: Scaling noise
> 
> 
> On Wed, Sep 03, 2003 at 05:41:39AM -0400, Brown, Len wrote:
> > > Latency is not bandwidth.
> > 
> > Bingo.
> > 
> > The way to address memory latency is by increasing bandwidth and
> > increasing parallelism to use it -- thus amortizing the latency.  
> 
> And if the app is a pointer chasing app, as many apps are, 
> that doesn't
> help at all.
> 
> It's pretty much analogous to file systems.  If bandwidth was 
> the answer
> then we'd all be seeing data moving at 60MB/sec off the disk. 
>  Instead 
> we see about 4 or 5MB/sec.
> 
> Expecting more bandwidth to help your app is like expecting 
> more platter
> speed to help your file system.  It's not the platter speed, it's the
> seeks which are the problem.  Same thing in system doesn't, 
> it's not the
> bcopy speed, it's the cache misses that are the problem.  
> More bandwidth
> doesn't do much for that.
> -- 
> ---
> Larry McVoy              lm at bitmover.com          
> http://www.bitmover.com/lm
> 

^ permalink raw reply	[flat|nested] 149+ messages in thread
* Re: Scaling noise
@ 2003-09-10 15:14 John Bradford
  0 siblings, 0 replies; 149+ messages in thread
From: John Bradford @ 2003-09-10 15:14 UTC (permalink / raw)
  To: davidsen, john; +Cc: linux-kernel

[snip most of discussion]

You have changed the topic completely.

> > The hardware is fault tollerant by design.  Only extreme events like a
> > fire or flood at the datacentre are likely to cause downtime of the
> > whole machine.  I don't consider that any less secure than a rack of
> > small servers.
>
> And power failure, loss of ISP connectivity, loss of phone service...
> As said originally, reliability is *hard* and *expensive* to do right.

Not sure how any of the above is related to the mainframe vs rack of
micros discussion.  Redundant ISP connectivity can be provided to
both.

You seem to be suggesting that I'm saying that it's a good idea to
replace geographically distributed, separate microcomputer servers
with a _single_ mainframe.  I am not.  I am only concerned with the
per-site situation.  As long as the geographically separate machine
stays geographically separate it is no longer part of the equation.

The only thing I am suggesting is that a single rack of machines in a
datacentre  somewhere could be replaced by a single machine with
virtualisation technology, and the same or better hardware
reliability as the rack of discrete machines, and there would be no
loss of availability.

Infact, increased availability may result, due to the ease of
administrating virtual machines as opposed to physical machines, and
the reduction in network hardware.

> This would be a good topic for comp.arch, it's getting a bit OT here.

Agreed, except that I brought it up to re-inforce Larry's point that
scaling above ~4 CPUs and hurting <=4 CPUs performance was a bad
thing.

John.

^ permalink raw reply	[flat|nested] 149+ messages in thread
* Re: Scaling noise
@ 2003-09-10 10:01 John Bradford
  2003-09-10 11:35 ` Alan Cox
  2003-09-10 13:46 ` Bill Davidsen
  0 siblings, 2 replies; 149+ messages in thread
From: John Bradford @ 2003-09-10 10:01 UTC (permalink / raw)
  To: davidsen; +Cc: linux-kernel

> | Once the option of running a firewall, a hot spare firewall, a
> | customer webserver, a hot spare customer webserver, mail server,
> | backup mail server, and a few virtual machines for customers, all on a
> | 1U box, why are you going to want to pay for seven or more Us in a
> | datacentre, plus extra network hardware?
>
> If you plan to run anything else on your firewall, and use the same
> machine as a hot spare for itself, I don't want you as my ISP.
> Reliability is expensive, and what you describe is known as a single
> point of failure.

Of course, you would be right if we were talking about current
microcomputer architectures.  I was talking about the possibility of
current mainframe technologies being implemented in future
microcomputer architectures.

Today, it is perfectly acceptable, normal, and commonplace to run hot
spares of various images on a single Z/Series box.  Infact, the
ability to do that is often a large factor in budgeting for the
initial investment.

The hardware is fault tollerant by design.  Only extreme events like a
fire or flood at the datacentre are likely to cause downtime of the
whole machine.  I don't consider that any less secure than a rack of
small servers.

Different images running in their own LPARs, or under Z/Vm are
separated from each other.  Assessments of their isolation have been
done, and ratings are available.

You absolutely _can_ use the same physical hardware to run a hot
spare, and protect yourself against software failiures.  A process can
monitor the virtual machine, and switch to the hot spare if it fails.

Add to that the fact that physical LAN cabling is reduced.  The amount
of network hardware is also reduced.  That adds to reliability.

John.

^ permalink raw reply	[flat|nested] 149+ messages in thread
* RE: Scaling noise
@ 2003-09-08  6:21 Brown, Len
  2003-09-08  9:21 ` Eric W. Biederman
  0 siblings, 1 reply; 149+ messages in thread
From: Brown, Len @ 2003-09-08  6:21 UTC (permalink / raw)
  To: Eric W. Biederman, Larry McVoy
  Cc: Martin J. Bligh, William Lee Irwin III, Alan Cox,
	Giuliano Pochini, Linux Kernel Mailing List

> 5) NUMA machines are slow.  There is not a single NUMA machine in the
>    top 10 of the top500 supercomputers list.  Likely this has more to
>    do with system sizes supported by the manufacture than inherent
>    process inferiority, but it makes a difference.

Hardware that is good at running linpack (all you gotta run to get onto
http://www.top500.org/ )isn't necessarily hardware that is any good at,
say, http://www.tpc.org/

^ permalink raw reply	[flat|nested] 149+ messages in thread
[parent not found: <rx83.88x.5@gated-at.bofh.it>]
* RE: Scaling noise
@ 2003-09-03  9:41 Brown, Len
  2003-09-03 11:02 ` Geert Uytterhoeven
  2003-09-03 11:19 ` Larry McVoy
  0 siblings, 2 replies; 149+ messages in thread
From: Brown, Len @ 2003-09-03  9:41 UTC (permalink / raw)
  To: Giuliano Pochini, Larry McVoy; +Cc: linux-kernel

> Latency is not bandwidth.

Bingo.

The way to address memory latency is by increasing bandwidth and
increasing parallelism to use it -- thus amortizing the latency.  HT is
one of many ways to do this.  If systems are to grow faster at a rate
better than memory speeds, then plan on more parallelism, not less.

-Len

^ permalink raw reply	[flat|nested] 149+ messages in thread
* Re: Scaling noise
@ 2003-09-03  7:10 John Bradford
  2003-09-03  7:38 ` Mike Fedyk
  2003-09-08 20:05 ` bill davidsen
  0 siblings, 2 replies; 149+ messages in thread
From: John Bradford @ 2003-09-03  7:10 UTC (permalink / raw)
  To: linux-kernel, lm

> Tell me again that it is a good idea to screw up uniprocessor performance
> for 64 way machines.  Great idea, that.  Go Dinosaurs!

I suspect Larry is actually right that uniprocessor and $smallnum CPUs
SMP performance will remain the most important, but I don't agree with
his reasoning:

Once true virtualisation becomes part of a mainstream microprocessor
architecture, we'll start to see a lot of small ISPs wanting to move 4
or so 1U servers on to a single, 1U SMP box.  Server consolidation
saves money in many ways - physical LAN cabling is replaced by virtual
LANs, less network hardware such as switches is required, there is
less hardware to break, you can add a new Linux image in seconds using
spare capacity rather than going out and buying a new box.

Once the option of running a firewall, a hot spare firewall, a
customer webserver, a hot spare customer webserver, mail server,
backup mail server, and a few virtual machines for customers, all on a
1U box, why are you going to want to pay for seven or more Us in a
datacentre, plus extra network hardware?

You can do this today on Z/Series, but you need to consolidate a lot
of machines to make it financially viable.  Once virtualisation is
available on cheaper hardware, everybody will want $bignum way SMP
boxes, but no Linux image will run on more than $smallnum virtual
CPUs.

John.

^ permalink raw reply	[flat|nested] 149+ messages in thread
* Re: Scaling noise
@ 2003-09-03  5:02 Samium Gromoff
  0 siblings, 0 replies; 149+ messages in thread
From: Samium Gromoff @ 2003-09-03  5:02 UTC (permalink / raw)
  To: Mikael Abrahamsson; +Cc: linux-kernel


> > It's called asymptotic behavior.  After a while you can look at the graph
> > and see that more CPUs on the same memory doesn't make sense.  It hasn't
> > made sense for a decade, what makes anyone think that is changing?
> 
> It didnt make sense two decades ago either, the VAX 8300 could be made to 
> go 6way and it stopped going faster around the third processor added.

  It doesn`t dismiss the increasing disbalance between cpu/memory speeds, which is
  way more fundamental than the possible technical glitches with experimental
  SMP on VAXes.

regards, Samium Gromoff

^ permalink raw reply	[flat|nested] 149+ messages in thread
* Scaling noise
@ 2003-09-03  4:03 Larry McVoy
  2003-09-03  4:12 ` Roland Dreier
  2003-09-03  4:18 ` Anton Blanchard
  0 siblings, 2 replies; 149+ messages in thread
From: Larry McVoy @ 2003-09-03  4:03 UTC (permalink / raw)
  To: linux-kernel

I've frequently tried to make the point that all the scaling for lots of
processors is nonsense.  Mr Dell says it better:

    "Eight-way (servers) are less than 1 percent of the market and shrinking
    pretty dramatically," Dell said. "If our competitors want to claim
    they're No. 1 in eight-ways, that's fine. We want to lead the market
    with two-way and four-way (processor machines)."

Tell me again that it is a good idea to screw up uniprocessor performance
for 64 way machines.  Great idea, that.  Go Dinosaurs!
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

^ permalink raw reply	[flat|nested] 149+ messages in thread

end of thread, other threads:[~2003-10-22  4:35 UTC | newest]

Thread overview: 149+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-09-03 17:07 Scaling noise Brown, Len
2003-09-03 17:32 ` Larry McVoy
2003-09-03 18:07   ` William Lee Irwin III
2003-09-03 18:07     ` Larry McVoy
2003-09-03 18:25       ` William Lee Irwin III
2003-09-03 23:47         ` Larry McVoy
2003-09-03 23:52           ` William Lee Irwin III
2003-09-03 23:55           ` Martin J. Bligh
2003-09-03 18:28       ` Valdis.Kletnieks
2003-09-03 18:31       ` Alan Cox
2003-09-03 20:11       ` Diego Calleja García
2003-09-03 18:11   ` Alan Cox
2003-09-03 19:56     ` Daniel Gryniewicz
2003-09-03 18:17   ` Martin J. Bligh
2003-09-04  0:36     ` Larry McVoy
2003-09-04  2:21       ` Martin J. Bligh
2003-09-04  2:34         ` Larry McVoy
2003-09-04  2:48           ` Martin J. Bligh
2003-09-04  3:02             ` Larry McVoy
2003-09-04  3:46               ` Gerrit Huizenga
2003-09-04  4:41               ` Martin J. Bligh
2003-09-10 15:02               ` Timothy Miller
2003-09-10 15:12                 ` Larry McVoy
2003-09-28  1:51                   ` Paul Jakma
2003-09-28  3:13                     ` Steven Cole
2003-09-29  0:47                       ` Paul Jakma
2003-10-22  1:22                       ` Paul Jakma
2003-10-22  3:46                         ` Steven Cole
2003-09-04  3:16             ` David Lang
2003-09-04  3:45               ` William Lee Irwin III
2003-09-04  4:51               ` Martin J. Bligh
2003-09-04  3:47           ` Davide Libenzi
2003-09-04  4:16             ` Larry McVoy
2003-09-04  7:43               ` Davide Libenzi
  -- strict thread matches above, loose matches on Subject: below --
2003-09-10 15:14 John Bradford
2003-09-10 10:01 John Bradford
2003-09-10 11:35 ` Alan Cox
2003-09-10 13:46 ` Bill Davidsen
2003-09-08  6:21 Brown, Len
2003-09-08  9:21 ` Eric W. Biederman
     [not found] <rx83.88x.5@gated-at.bofh.it>
     [not found] ` <rxrp.8wt.1@gated-at.bofh.it>
     [not found]   ` <rxB3.gg.1@gated-at.bofh.it>
     [not found]     ` <rxB6.gg.5@gated-at.bofh.it>
     [not found]       ` <rydL.17V.1@gated-at.bofh.it>
     [not found]         ` <rGXO.5g9.7@gated-at.bofh.it>
2003-09-03 15:33           ` Ihar 'Philips' Filipau
2003-09-03  9:41 Brown, Len
2003-09-03 11:02 ` Geert Uytterhoeven
2003-09-03 11:19 ` Larry McVoy
2003-09-03 11:47   ` Matthias Andree
2003-09-03 18:00   ` William Lee Irwin III
2003-09-03 18:05     ` Larry McVoy
2003-09-03 18:15       ` William Lee Irwin III
2003-09-03 18:15         ` Larry McVoy
2003-09-03 18:26           ` William Lee Irwin III
2003-09-03 18:32         ` Alan Cox
2003-09-03 19:46           ` William Lee Irwin III
2003-09-03 20:13             ` Alan Cox
2003-09-03 20:31               ` William Lee Irwin III
2003-09-03 20:48             ` Martin J. Bligh
2003-09-03 21:21               ` William Lee Irwin III
2003-09-03 21:29                 ` Martin J. Bligh
2003-09-03 21:51                   ` William Lee Irwin III
2003-09-03 21:46                     ` Martin J. Bligh
2003-09-04  0:07                       ` Mike Fedyk
2003-09-04  1:06                       ` Larry McVoy
2003-09-04  1:10                         ` Larry McVoy
2003-09-04  1:32                         ` William Lee Irwin III
2003-09-04  1:46                           ` David Lang
2003-09-04  1:51                             ` William Lee Irwin III
2003-09-04  2:31                           ` Martin J. Bligh
2003-09-04  2:40                             ` Mike Fedyk
2003-09-04  2:50                               ` Martin J. Bligh
2003-09-04  3:49                                 ` Mike Fedyk
2003-09-04  2:48                             ` Steven Cole
2003-09-04 17:05                             ` Daniel Phillips
2003-09-07 21:18                         ` Eric W. Biederman
2003-09-07 23:07                           ` Larry McVoy
2003-09-07 23:47                             ` Eric W. Biederman
2003-09-08  0:57                               ` Larry McVoy
2003-09-08  3:55                                 ` Eric W. Biederman
2003-09-08  4:47                                 ` Stephen Satchell
2003-09-08  5:25                                   ` Larry McVoy
2003-09-08  8:32                                     ` Eric W. Biederman
2003-09-04  0:58                     ` Larry McVoy
2003-09-04  1:12                       ` William Lee Irwin III
2003-09-04  2:49                         ` Larry McVoy
2003-09-04  3:15                           ` William Lee Irwin III
2003-09-04  3:38                           ` Nick Piggin
2003-09-05  1:34         ` Robert White
2003-09-03 19:11     ` Steven Cole
2003-09-03 19:36       ` William Lee Irwin III
2003-09-03  7:10 John Bradford
2003-09-03  7:38 ` Mike Fedyk
2003-09-03 11:14   ` Larry McVoy
2003-09-08 20:05 ` bill davidsen
2003-09-03  5:02 Samium Gromoff
2003-09-03  4:03 Larry McVoy
2003-09-03  4:12 ` Roland Dreier
2003-09-03  4:20   ` Larry McVoy
2003-09-03 15:12   ` Martin J. Bligh
2003-09-03  4:18 ` Anton Blanchard
2003-09-03  4:29   ` Larry McVoy
2003-09-03  4:33     ` CaT
2003-09-03  5:08       ` Larry McVoy
2003-09-03  5:44         ` Mikael Abrahamsson
2003-09-03  6:12         ` Bernd Eckenfels
2003-09-03 12:09           ` Alan Cox
2003-09-03 15:10             ` Martin J. Bligh
2003-09-03 16:01               ` Jörn Engel
2003-09-03 16:21                 ` Martin J. Bligh
2003-09-03 19:41                   ` Mike Fedyk
2003-09-03 20:11                     ` Martin J. Bligh
2003-09-04 20:36               ` Rik van Riel
2003-09-04 20:47                 ` Martin J. Bligh
2003-09-04 21:30                 ` William Lee Irwin III
2003-09-03  8:11         ` Giuliano Pochini
2003-09-03 14:25         ` Steven Cole
2003-09-03 12:47           ` Antonio Vargas
2003-09-03 15:31             ` Steven Cole
2003-09-04  1:50               ` Daniel Phillips
2003-09-04  1:52                 ` Larry McVoy
2003-09-04  4:42                   ` David S. Miller
2003-09-08 19:40                     ` bill davidsen
2003-09-04  2:18                 ` William Lee Irwin III
2003-09-04  2:19                 ` Steven Cole
2003-09-04  2:35                   ` William Lee Irwin III
2003-09-04  2:40                     ` Steven Cole
2003-09-04  3:20                       ` Nick Piggin
2003-09-04  3:07                   ` Daniel Phillips
2003-09-08 19:27                 ` bill davidsen
2003-09-08 19:12           ` bill davidsen
2003-09-03 16:37         ` Kurt Wall
2003-09-06 15:08         ` Pavel Machek
2003-09-08 13:38           ` Alan Cox
2003-09-09  6:11             ` Rob Landley
2003-09-09 16:07               ` Ricardo Bugalho
2003-09-10  5:14                 ` Rob Landley
2003-09-10  5:45                   ` David Mosberger
2003-09-10 10:10                   ` Ricardo Bugalho
2003-09-03  6:28     ` Anton Blanchard
2003-09-03  6:55       ` Nick Piggin
2003-09-03 15:23         ` Martin J. Bligh
2003-09-03 15:39           ` Larry McVoy
2003-09-03 15:50             ` Martin J. Bligh
2003-09-04  0:49               ` Larry McVoy
2003-09-04  2:21                 ` Daniel Phillips
2003-09-04  2:35                   ` Martin J. Bligh
2003-09-04  2:46                   ` Larry McVoy
2003-09-04  4:58                     ` David S. Miller
2003-09-04  4:49             ` David S. Miller
2003-09-08 19:50             ` bill davidsen
2003-09-08 23:39               ` Peter Chubb
2003-09-03 17:16           ` William Lee Irwin III

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).