Re: TOE brain dump

[David Lang <david.lang@digitalinsight.com>]

On Mon, 4 Aug 2003, Werner Almesberger wrote:

> David Lang wrote:
> > I would think that it's much more difficult to run NUMA across different
> > types of CPU's
>
> I'd view this as a new and interesting challenge :-) Besides,
> if one would use Alan's idea, and just use an amd64, or such,
> the CPUs wouldn't be all that different in the end.

you missed Alan's point, he was saying you don't do TOE on the NIC, you
just add another CPU to your main system and use non-TOE NIC's the way you
do today.

> > then it would be to run a seperate kernel on the NIC.
>
> Yes, but that separate kernel would need new administrative
> interfaces, and things like route changes would be difficult
> to handle. (That is, if you still want this to appear as a
> single system to user space.) It would certainly be better
> that running a completely proprietary solution, but you still
> get a few nasty problems.

Any time you create a cluster of machines you want to create som nice
administrative interfaces for them to maintain your own sanity (you don't
think sysadmins login to every machine on a 1000 node beowolf cluster do
you :-)

in trying to run a single kernel across different types of CPU's you run
into some really nasty problems (different machine code, and even if it's
the same family of processor it could require very different
optimizations, imagine the two processor types useing different word
lengths or endian order)

Larry McVoy has the right general idea when he says buy another box to do
the job, he is just missing the idea that there are some advantages of
coupling the cluster more tightly then you can do with a seperate box.

David Lang