Re: Alan Shih: "TCP IP Offloading Interface"

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

IMHO, there are several cases for some type of TCP/IP offload.  One is for 
embedded systems that are just not capable of doing 1Gbps+.  Another is with 
10GbE, even high end servers will not be able keep up with TCP 
processing/data movement at these speeds.  Not being proactive in adopting 
TCP/IP offload will force Linux into accepting some scheme that will not 
necissarily be best.


>Alan Shih wrote:
>>Has anyone worked on a standard interface between TOE and Linux? (ie. 
>>something like Trapeze/Myrinet's GMS?)
>>
>>Or TOE is a forbidden discussion? Any effort in making Linux the OS for 
>>TOE at all even though Linux is a little too heavy for it?
>
>
>
>I do not forsee there _ever_ being a TOE interface for Linux.
>
>
>It's not a forbidden discussion, but, the networking developers tend to
>ignore people who mention TOE because it's been discussed to death, and
>no evidence has ever been presented to prove it has advantages where it
>matters, and it has significant _dis_advantages from the get-go.
>
>
>I really should write an LKML FAQ entry for TOE.
>
>
>        Jeff
>
>
>
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at http://www.tux.org/lkml/
>

_________________________________________________________________
STOP MORE SPAM with the new MSN 8 and get 2 months FREE*  
http://join.msn.com/?page=features/junkmail

Re: Alan Shih: "TCP IP Offloading Interface"

[Jeff Garzik]

David griego wrote:
> IMHO, there are several cases for some type of TCP/IP offload.  One is 
> for embedded systems that are just not capable of doing 1Gbps+.  Another 
> is with 10GbE, even high end servers will not be able keep up with TCP 
> processing/data movement at these speeds.  Not being proactive in 
> adopting TCP/IP offload will force Linux into accepting some scheme that 
> will not necissarily be best.


How does one evaluate a TOE stack to be sure that all the security fixes 
in Linux are also in that stack?

How does one evaluate a TOE stack to be sure it doesn't add new security 
holes that Linux never had?

	Jeff

Re: Alan Shih: "TCP IP Offloading Interface"

[Deepak Saxena]

On Jul 14 2003, at 15:02, Jeff Garzik was caught saying:
> David griego wrote:
> >IMHO, there are several cases for some type of TCP/IP offload.  One is 
> >for embedded systems that are just not capable of doing 1Gbps+.  Another 
> >is with 10GbE, even high end servers will not be able keep up with TCP 
> >processing/data movement at these speeds.  Not being proactive in 
> >adopting TCP/IP offload will force Linux into accepting some scheme that 
> >will not necissarily be best.
> 
> 
> How does one evaluate a TOE stack to be sure that all the security fixes 
> in Linux are also in that stack?
> 
> How does one evaluate a TOE stack to be sure it doesn't add new security 
> holes that Linux never had?

I just replied to Jeff privately regarding this on a side thread, but
here it is again:

My question back is how do you evaluate a high-end SCSI RAID controller
to make sure that there is no bug in the firmware that causes you to loose 
all your data? You test it and if it fails miserably, you go yell at the
HW manufacturer. There's no argueing that the question needs to be answered 
b/c opening up a security hole is a dangerous thing to do, but perhaps some 
sort of standardized test could be developed by the community, HW vendors, 
and OSDL?  I agree that TOE has problems and many of the points addressed 
by others in this thread are valid concerns, but simply saying that
because of these issues "TOE will never happen" or "TOE is Evil" is not 
going to make the desire of TOE from HW vendors go away. There needs to 
be an open discussion between HW vendors and the community to determine 
the best way to move forward. This includes addressing questions such as
do we want TOE + non-TOE stacks running at the same time? (I propose
a big no for that since the level of complexity has just increased
many times). Do we want to support multiple TOEs? How do we handle
routing between TOEs? Etc... We either need to start thinking about
these issues now or we'll be stuck with crap implementation requirements
due to already existing TOE support in other OSes.  In a perfect academic 
world TOE might be a horrid idea that should die, but the HW vendors have 
already decided to move in this direction? How is linux going to react to 
this: Just ignore it until it's too late or be pro-active about it?

A minimum step would be moving in the direction of FreeBSD and providing
hooks for alternate stacks. That way if an embedded developer wanted
to provide a different stack, he could do so and take full responsibility 
for supporting that kernel.

Finally, I would like to point out that just b/c something is considered
bad does not preclude it from being in the kernel. I think most kernel
developers agree that PAE, I2O, ISAPNP and other technologies are broken
and we wish they would die, yet Linux still supports them. Why? B/C the
HW requires it.  TOE is going to be no different. 

~Deepak

-- 
Deepak Saxena
MontaVista Software - Powering the Embedded Revolution - www.mvista.com

Re: Alan Shih: "TCP IP Offloading Interface"

[Jeff Garzik]

Deepak Saxena wrote:
> My question back is how do you evaluate a high-end SCSI RAID controller
> to make sure that there is no bug in the firmware that causes you to loose 
> all your data? You test it and if it fails miserably, you go yell at the
> HW manufacturer. There's no argueing that the question needs to be answered 

Hardware RAID is not remotely accessible to the entire world.


> and OSDL?  I agree that TOE has problems and many of the points addressed 
> by others in this thread are valid concerns, but simply saying that
> because of these issues "TOE will never happen" or "TOE is Evil" is not 
> going to make the desire of TOE from HW vendors go away. There needs to 
> be an open discussion between HW vendors and the community to determine 
> the best way to move forward. This includes addressing questions such as
> do we want TOE + non-TOE stacks running at the same time? (I propose
> a big no for that since the level of complexity has just increased
> many times). Do we want to support multiple TOEs? How do we handle
> routing between TOEs? Etc... We either need to start thinking about

Answering those questions implies that TOE is a good idea.  That still 
is still an open question.

The community has been _trying_ to dialogue with people interested in a 
progressive solution that actually addresses the problems we raise.  I 
haven't seen a single response.

I don't see any dialogue at all.  The examples I have seen so far are HW 
vendors saying "we are doing TOE" not "should we be doing TOE?"

DaveM has been dropping ever-more-blatant hints about an efficient 
design.  Who has listened?


> these issues now or we'll be stuck with crap implementation requirements
> due to already existing TOE support in other OSes.  In a perfect academic 
> world TOE might be a horrid idea that should die, but the HW vendors have 
> already decided to move in this direction? How is linux going to react to 
> this: Just ignore it until it's too late or be pro-active about it?

Not all hardware vendors.  One specific hardware vendor, with decades of 
experience in TCP/IP and Unix, realizes that TOE is not the answer.


> A minimum step would be moving in the direction of FreeBSD and providing
> hooks for alternate stacks. That way if an embedded developer wanted
> to provide a different stack, he could do so and take full responsibility 
> for supporting that kernel.

This is trivial.  Just create your own character device driver and go to 
town.


> Finally, I would like to point out that just b/c something is considered
> bad does not preclude it from being in the kernel. I think most kernel
> developers agree that PAE, I2O, ISAPNP and other technologies are broken
> and we wish they would die, yet Linux still supports them. Why? B/C the
> HW requires it.  TOE is going to be no different. 

TOE is vastly different.

As Alan said, nobody is stopping developers from maintaining their own 
TOE fork of Linux.  Under Linux, forks are _encouraged_, remember.

	Jeff

Re: Alan Shih: "TCP IP Offloading Interface"

[Werner Almesberger]

Deepak Saxena wrote:
> My question back is how do you evaluate a high-end SCSI RAID controller
> to make sure that there is no bug in the firmware that causes you to loose 
> all your data?

This may be a great moment for relaxing with a good book, e.g. Bruce
Schneier's "Secrets & Lies", chapter 22, where he describes why
functional testing and discovering security bugs are so different.

> Finally, I would like to point out that just b/c something is considered
> bad does not preclude it from being in the kernel.

Well, in the case of TOE, we should at least be able to politely ask
all the useless silicon to step aside :-)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 19:46, David griego wrote:
> IMHO, there are several cases for some type of TCP/IP offload.  One is for 
> embedded systems that are just not capable of doing 1Gbps+.  Another is with 

My fridge doesn't need to do 10Gbit a second, and for most other
embedded the constraints are ram bandwidth and nothing else. Since
deeply embedded stuff also doesn't run with MMUs or runs 'partially
trusted' most of the VM games and the socket api games also go away.

I've done deeply embedded tcp/ip. I don't buy the argument, embedded
gains the least of all from ToE. 

> 10GbE, even high end servers will not be able keep up with TCP 
> processing/data movement at these speeds.  Not being proactive in adopting 

They said that about 10Mbit until Van showed them a thing or two. They
said it about 100Mbit, they said it about gigabit.

> TCP/IP offload will force Linux into accepting some scheme that will not 
> necissarily be best.

TCP/IP is an exercise in two things when you are running at speed

1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
	sg does, on card target buffers do with decent chipsets.

2.	Handling in order perfectly predicted data streams. ToE is 
	overkill for this. Thats about latency to memory and touching
	as little as possible. The main CPU has a rather good connection
	to main memory.

ToE is also horribly vulnerable to attack because putting it on a card
dictates relatively low CPU power and low power consumption as well as
rather nasty pricing issues. Historically low power devices have 
repeatedly been screwed by attackers hitting software or other slow
paths in the device to attack it.

This is before we get into the delights of multipath routing across
different vendors cards, firewalling, traffic shaping, retrofitting new
features, questions about what happens with an old ToE card when its
got a hole...

The internet land speed record is held by a non ToE system, let me know
when that changes.

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

How does one measure the reliability and security of current software TCP/IP 
stacks?  Some standard set of test would have to be identified and the TOEs 
would need to be tested against this to ensure that they meet some minimum 
standard.  I would suggest offloading the minimum amount from the OS so that 
most of the control could be maintaind by the OS stack.  This also would 
make failover/routing changes between TOE -TOE, and TOE-NIC easier.  Current 
offloads such as checksum and segmentation will not be enough for 10GbE 
processing, so it would have to be something more than we have today.
David


>From: Jeff Garzik <jgarzik@pobox.com>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,  linux-kernel@vger.kernel.org
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: Mon, 14 Jul 2003 15:02:35 -0400
>
>David griego wrote:
>>IMHO, there are several cases for some type of TCP/IP offload.  One is for 
>>embedded systems that are just not capable of doing 1Gbps+.  Another is 
>>with 10GbE, even high end servers will not be able keep up with TCP 
>>processing/data movement at these speeds.  Not being proactive in adopting 
>>TCP/IP offload will force Linux into accepting some scheme that will not 
>>necissarily be best.
>
>
>How does one evaluate a TOE stack to be sure that all the security fixes in 
>Linux are also in that stack?
>
>How does one evaluate a TOE stack to be sure it doesn't add new security 
>holes that Linux never had?
>
>	Jeff
>
>
>

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*.  
http://join.msn.com/?page=features/virus

Re: Alan Shih: "TCP IP Offloading Interface"

[Jeff Garzik]

David griego wrote:
> How does one measure the reliability and security of current software 
> TCP/IP stacks?  Some standard set of test would have to be identified 
> and the TOEs would need to be tested against this to ensure that they 
> meet some minimum standard.  I would suggest offloading the minimum 
> amount from the OS so that most of the control could be maintaind by the 
> OS stack.  This also would make failover/routing changes between TOE 
> -TOE, and TOE-NIC easier.

Anything beyond basic host-only TOE adds massive complexity for very 
little gain:  interfacing netfilter and routing code with a black box we 
_hope_ will act properly sounds like suicide.


 >  Current offloads such as checksum and
> segmentation will not be enough for 10GbE processing, so it would have 
> to be something more than we have today.

All this is vague handwaving without supporting evidence.  So far we get 
stuff like Internet2 speed records _without_ TOE.  And Linux currently 
supports 10gige...  and hosts are just going to keep getting faster and 
faster.

	Jeff

Re: Alan Shih: "TCP IP Offloading Interface"

[Jesse Pollard]

On Monday 14 July 2003 14:26, Jeff Garzik wrote:
> David griego wrote:
> > How does one measure the reliability and security of current software
> > TCP/IP stacks?  Some standard set of test would have to be identified
> > and the TOEs would need to be tested against this to ensure that they
> > meet some minimum standard.  I would suggest offloading the minimum
> > amount from the OS so that most of the control could be maintaind by the
> > OS stack.  This also would make failover/routing changes between TOE
> > -TOE, and TOE-NIC easier.
>
> Anything beyond basic host-only TOE adds massive complexity for very
> little gain:  interfacing netfilter and routing code with a black box we
> _hope_ will act properly sounds like suicide.
>
>  >  Current offloads such as checksum and
> >
> > segmentation will not be enough for 10GbE processing, so it would have
> > to be something more than we have today.
>
> All this is vague handwaving without supporting evidence.  So far we get
> stuff like Internet2 speed records _without_ TOE.  And Linux currently
> supports 10gige...  and hosts are just going to keep getting faster and
> faster.
>
> 	Jeff

Not to mention the problems IPSec would have with such a device.

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 20:14, David griego wrote:
> How does one measure the reliability and security of current software TCP/IP 
> stacks?  

You stick them on irc servers, porn sites and unpopular news sites and
wait. Alternatively you can use the fact you have the source to do
formal verifications on them looking for everything from bugs to NSA
backdoors to the IPSEC. People have been doing both.

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

>Jeff Garzik wrote:
>Anything beyond basic host-only TOE adds massive complexity for very little 
>gain:  interfacing netfilter and routing code with a black box we _hope_ 
>will act properly sounds like suicide.
Keep most of this on the host, offload only performance path like the 
Alacritech TOE.

>All this is vague handwaving without supporting evidence.  So far we get 
>stuff like Internet2 speed records _without_ TOE.  And Linux currently 
>supports 10gige...  and hosts are just going to keep getting faster and 
>faster.

Intel Clusters and Network Storage Volume Platforms Lab reported that it 
takes about 1MHz to process 1Mbps on a PIII.  Using this rule of thumb (they 
showed it scaling from 400MHz to 800MHz) it would take 10GHz to process 
10Mbps.  Well you might say "what about multi-processers?"  This would be 
good for people that have multi-processors, but there is a large segment of 
embedded processors that are not going have SMP, or be at 10GHz anytime 
soon.  Besides that processing interrupts does not scale across MPs 
liniarly.  The truth is that communication speeds are outpacing processor 
speeds at this time.
David

>
>	Jeff
>
>
>

_________________________________________________________________
Help STOP SPAM with the new MSN 8 and get 2 months FREE*  
http://join.msn.com/?page=features/junkmail

Re: Alan Shih: "TCP IP Offloading Interface"

[Jeff Garzik]

David griego wrote:
> Intel Clusters and Network Storage Volume Platforms Lab reported that it 
> takes about 1MHz to process 1Mbps on a PIII.  Using this rule of thumb 
> (they showed it scaling from 400MHz to 800MHz) it would take 10GHz to 
> process 10Mbps.  Well you might say "what about multi-processers?"  This 

Um.  It doesn't take nearly 10Ghz to handle 10Mbps, or even 100Mbps.


> would be good for people that have multi-processors, but there is a 
> large segment of embedded processors that are not going have SMP, or be 
> at 10GHz anytime soon.  Besides that processing interrupts does not 
> scale across MPs liniarly.  The truth is that communication speeds are 
> outpacing processor speeds at this time.

If the host CPU is a bottleneck after large-send and checksums have been 
offloaded, then logically you aren't getting any work done _anyway_. 
You have to interface with the net stack at some point, in which case 
you incur a fixed cost, for socket handling, TCP exception handling, etc.

Maybe somebody needs to be looking into AMP (asymmetric 
multiprocessing), too.

	Jeff

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 21:03, Jeff Garzik wrote:
> If the host CPU is a bottleneck after large-send and checksums have been 
> offloaded, then logically you aren't getting any work done _anyway_. 
> You have to interface with the net stack at some point, in which case 
> you incur a fixed cost, for socket handling, TCP exception handling, etc.
> 
> Maybe somebody needs to be looking into AMP (asymmetric 
> multiprocessing), too.

There isnt currently any evidence it buy you anything, although HT may
change that equation a bit. Its still the same RAM bandwidth and you've
not really gotten rid of most of the socket handling/event/wakeup
overhead either.

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 20:43, David griego wrote:
> Intel Clusters and Network Storage Volume Platforms Lab reported that it 
> takes about 1MHz to process 1Mbps on a PIII.  Using this rule of thumb (they 

1MHz to proces 1Mbit doing what - file I/O to and from disk, web serving
- because ToE or otherwise I still have to process the data I receive
and do something useful with it unless I'm just a router, firewall or
load balancer. If you want to argue about using gate arrays and hardware
to accelerate IP routing, balancing and firewall filter cams then you
might get somewhere - but they dont need to talk TCP.

Also if its 1MHz per 1Mbit worse case and your ToE engine isnt entirely
hardware paths capable of sustaining 10Gbit/sec, what happens when I hit
you with 10Gbit of carefully chosen non optimal frames ?

Re: Alan Shih: "TCP IP Offloading Interface"

[Shawn]

Don't push him... he'll do it! We're talking "floor sweepings" frames!
"Too rank for sausage" frames!

On Mon, 2003-07-14 at 15:05, Alan Cox wrote:
> Also if its 1MHz per 1Mbit worse case and your ToE engine isnt entirely
> hardware paths capable of sustaining 10Gbit/sec, what happens when I hit
> you with 10Gbit of carefully chosen non optimal frames ?

Re: Alan Shih: "TCP IP Offloading Interface"

[Werner Almesberger]

Alan Cox wrote:
> load balancer. If you want to argue about using gate arrays and hardware
> to accelerate IP routing, balancing and firewall filter cams then you
> might get somewhere - but they dont need to talk TCP.

One thing that sounds right about TOE is that per-packet overhead
is becoming an issue, too. At 10 Gbps, the critters come flying in
at almost 1 MHz if you're using standard MTU sizes.

On the other hand, replicating the entire infrastructure on some
non-Linux hardware has several problems, even if we don't consider
performance:

 - where is the configuration interface ? In the kernel or in
   user space ? What about existing interfaces ?
 - you'll never get exactly the same semantics. Just identifying
   the differences is a very painful process. And again, what
   about existing interfaces ?
 - testing has just become a lot harder

What I think would be more promising is to investigate in the
direction of NUMA-style architectures, where some CPUs are closer
to NICs and whatever data source/sink those TCP streams go to.

Licensing issues, the classical reason for using independent
stacks, can be elegantly avoided on Linux.

Another area are network processors. They could help with fancy
things like Dave's flow cache, but also with fine-grained timing
needed for traffic control. One problem there is that they're
locked away behind walls of NDAs and proprietary development
environments, so one couldn't even begin to properly support them
in Linux. (What can be done is to treat NP+software as a black
box, but I wouldn't consider this a satisfying choice.)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, Buenos Aires, Argentina         wa@almesberger.net /
/_http://www.almesberger.net/____________________________________________/

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

Embedded does not simply include toasters and fridges, it also includes NAS 
and SAN appliances as well as telco gear.  These types of devices have 
advanced memory subsystems and run processors such as PPC and ARM.  One of 
the most limiting factors in these types of devices is power consumption.  
This usually limits the number of cores and frequency these cores.  
Offloading the processing of protocol stacks to ASICS would have a great 
impact in performance.  If you are going to embed a high frequency chip in 
your embedded devices I would recommend developing a heater not a fridge.


>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,   Linux Kernel Mailing List 
><linux-kernel@vger.kernel.org>
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: 14 Jul 2003 20:42:53 +0100
>
>On Llu, 2003-07-14 at 19:46, David griego wrote:
> > IMHO, there are several cases for some type of TCP/IP offload.  One is 
>for
> > embedded systems that are just not capable of doing 1Gbps+.  Another is 
>with
>
>My fridge doesn't need to do 10Gbit a second, and for most other
>embedded the constraints are ram bandwidth and nothing else. Since
>deeply embedded stuff also doesn't run with MMUs or runs 'partially
>trusted' most of the VM games and the socket api games also go away.

See PPC and ARM architecture for the use of MMUs in embedded systems

>
>I've done deeply embedded tcp/ip. I don't buy the argument, embedded
>gains the least of all from ToE.
>
> > 10GbE, even high end servers will not be able keep up with TCP
> > processing/data movement at these speeds.  Not being proactive in 
>adopting
>
>They said that about 10Mbit until Van showed them a thing or two. They
>said it about 100Mbit, they said it about gigabit.

Not the case for embedded.  I understand your viewpoint from the server 
space though.
>
> > TCP/IP offload will force Linux into accepting some scheme that will not
> > necessarily be best.
>
>TCP/IP is an exercise in two things when you are running at speed
>
>1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
>	sg does, on card target buffers do with decent chipsets.

A TOE enabled with RDDP would help eliminate the kernel to user space copy 
(and in the case of SAMBA the copy back to the kernel).  This would reduce 
the memory system loading by a third to a half.
>
>2.	Handling in order perfectly predicted data streams. ToE is
>	overkill for this. Thats about latency to memory and touching
>	as little as possible. The main CPU has a rather good connection
>	to main memory.
>
Yes, RDDP would be nice to have though for the reason stated for #1, so the 
hardware would need to at least be TCP aware.

>ToE is also horribly vulnerable to attack because putting it on a card
>dictates relatively low CPU power and low power consumption as well as
>rather nasty pricing issues. Historically low power devices have
>repeatedly been screwed by attackers hitting software or other slow
>paths in the device to attack it.
The use of ASICs could ensure that TCP processing is as quick as wire speed

>
>This is before we get into the delights of multipath routing across
>different vendors cards, firewalling, traffic shaping, retrofitting new
>features, questions about what happens with an old ToE card when its
>got a hole...
Try to keep the datapath processing on the TOE, and everything else in the 
OS.  Also give the API the ability to turn of the TOE if a hole exists and 
use it like a regular NIC.
>
>The internet land speed record is held by a non ToE system, let me know
>when that changes.
>
Layer one network processing is often handled by ASICS, also some of the 
fastest encryption engines are hardware.  I suggest we don't wait until your 
proven wrong before making a decision on TOE.

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*.  
http://join.msn.com/?page=features/virus

RE: Alan Shih: "TCP IP Offloading Interface"

[Alan Shih]

In a typical application of NAS ASIC, it is easier to place 2 300MHz
processor than a 1 GHz processor in cost. This line of reasoning forces me
to consider making 1 of the processor to be TOE while the other one deals
with disk/FS manipulations.

Alan


-----Original Message-----
From: David griego [mailto:dagriego@hotmail.com]
Sent: Monday, July 14, 2003 1:19 PM
To: alan@lxorguk.ukuu.org.uk
Cc: alan@storlinksemi.com; linux-kernel@vger.kernel.org
Subject: Re: Alan Shih: "TCP IP Offloading Interface"


Embedded does not simply include toasters and fridges, it also includes NAS
and SAN appliances as well as telco gear.  These types of devices have
advanced memory subsystems and run processors such as PPC and ARM.  One of
the most limiting factors in these types of devices is power consumption.
This usually limits the number of cores and frequency these cores.
Offloading the processing of protocol stacks to ASICS would have a great
impact in performance.  If you are going to embed a high frequency chip in
your embedded devices I would recommend developing a heater not a fridge.


>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,   Linux Kernel Mailing List
><linux-kernel@vger.kernel.org>
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: 14 Jul 2003 20:42:53 +0100
>
>On Llu, 2003-07-14 at 19:46, David griego wrote:
> > IMHO, there are several cases for some type of TCP/IP offload.  One is
>for
> > embedded systems that are just not capable of doing 1Gbps+.  Another is
>with
>
>My fridge doesn't need to do 10Gbit a second, and for most other
>embedded the constraints are ram bandwidth and nothing else. Since
>deeply embedded stuff also doesn't run with MMUs or runs 'partially
>trusted' most of the VM games and the socket api games also go away.

See PPC and ARM architecture for the use of MMUs in embedded systems

>
>I've done deeply embedded tcp/ip. I don't buy the argument, embedded
>gains the least of all from ToE.
>
> > 10GbE, even high end servers will not be able keep up with TCP
> > processing/data movement at these speeds.  Not being proactive in
>adopting
>
>They said that about 10Mbit until Van showed them a thing or two. They
>said it about 100Mbit, they said it about gigabit.

Not the case for embedded.  I understand your viewpoint from the server
space though.
>
> > TCP/IP offload will force Linux into accepting some scheme that will not
> > necessarily be best.
>
>TCP/IP is an exercise in two things when you are running at speed
>
>1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
>	sg does, on card target buffers do with decent chipsets.

A TOE enabled with RDDP would help eliminate the kernel to user space copy
(and in the case of SAMBA the copy back to the kernel).  This would reduce
the memory system loading by a third to a half.
>
>2.	Handling in order perfectly predicted data streams. ToE is
>	overkill for this. Thats about latency to memory and touching
>	as little as possible. The main CPU has a rather good connection
>	to main memory.
>
Yes, RDDP would be nice to have though for the reason stated for #1, so the
hardware would need to at least be TCP aware.

>ToE is also horribly vulnerable to attack because putting it on a card
>dictates relatively low CPU power and low power consumption as well as
>rather nasty pricing issues. Historically low power devices have
>repeatedly been screwed by attackers hitting software or other slow
>paths in the device to attack it.
The use of ASICs could ensure that TCP processing is as quick as wire speed

>
>This is before we get into the delights of multipath routing across
>different vendors cards, firewalling, traffic shaping, retrofitting new
>features, questions about what happens with an old ToE card when its
>got a hole...
Try to keep the datapath processing on the TOE, and everything else in the
OS.  Also give the API the ability to turn of the TOE if a hole exists and
use it like a regular NIC.
>
>The internet land speed record is held by a non ToE system, let me know
>when that changes.
>
Layer one network processing is often handled by ASICS, also some of the
fastest encryption engines are hardware.  I suggest we don't wait until your
proven wrong before making a decision on TOE.

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*.
http://join.msn.com/?page=features/virus

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 21:19, David griego wrote:
> Embedded does not simply include toasters and fridges, it also includes NAS 
> and SAN appliances as well as telco gear.  These types of devices have 

I have some sympathy with SAN developers, because you can treat the link
as dedicated and just for the SAN so you can implement iSCSI optimised
code. The right engineering approach probably consists of removing iSCSI
and doing the job right but I appreciate there are non engineering
issues.

> advanced memory subsystems and run processors such as PPC and ARM.  One of 
> the most limiting factors in these types of devices is power consumption.  

Memory bandwidth is your killer quite often, you have to keep the CPU
away from the data and often even off the memory bus holding most of the
data.

> >deeply embedded stuff also doesn't run with MMUs or runs 'partially
> >trusted' most of the VM games and the socket api games also go away.
> 
> See PPC and ARM architecture for the use of MMUs in embedded systems

You still have the partial trust stuff and the ability to violate the
socket api in the interest of speed - that helps.

> >TCP/IP is an exercise in two things when you are running at speed
> >
> >1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
> >	sg does, on card target buffers do with decent chipsets.
> 
> A TOE enabled with RDDP would help eliminate the kernel to user space copy 
> (and in the case of SAMBA the copy back to the kernel).  This would reduce 
> the memory system loading by a third to a half.

Thats mostly an API issue. Socket API found non optimal after 20 years.
Hardly shocking news - the cluster people already changed API, Larry
McVoy proposed stuff like splice years ago because he saw it coming.

> >2.	Handling in order perfectly predicted data streams. ToE is
> >	overkill for this. Thats about latency to memory and touching
> >	as little as possible. The main CPU has a rather good connection
> >	to main memory.
> >
> Yes, RDDP would be nice to have though for the reason stated for #1, so the 
> hardware would need to at least be TCP aware.

TCP aware hardware is good, segmentation, prediction, buffer/sequence
matchers etc but you don't want the policy parts in silicon

> >repeatedly been screwed by attackers hitting software or other slow
> >paths in the device to attack it.
> The use of ASICs could ensure that TCP processing is as quick as wire speed

Only if your ASIC worst case is wire speed. If your ASIC has one path it
must drop to software for and has a low powered internal CPU to fix up,
you just lost and you'll plow like a cisco with too many filter rules to
do in silicon.

> Try to keep the datapath processing on the TOE, and everything else in the 
> OS.  Also give the API the ability to turn of the TOE if a hole exists and 
> use it like a regular NIC.

Some of this is certainly about how you partition the load - its no
different to things like RAID. We've had hardware raid, software raid,
and finally people are popping up with neat hybrids. We've had software
audio, hardware audio and again now we are seeing hybrids that do
different bits in hardware and software - ditto modems.

> >The internet land speed record is held by a non ToE system, let me know
> >when that changes.
> >
> Layer one network processing is often handled by ASICS, also some of the 
> fastest encryption engines are hardware.  I suggest we don't wait until your 
> proven wrong before making a decision on TOE.

You don't have to. You can go build and test and maintain a set of TOE patches.
Nobody is stopping you. Lots of Linux code exists because someone decided that
the official story was wrong and proved it so.

Alan

Re: Alan Shih: "TCP IP Offloading Interface"

[Deepak Saxena]

On Jul 14 2003, at 21:34, Alan Cox was caught saying:
> 
> You don't have to. You can go build and test and maintain a set of TOE patches.
> Nobody is stopping you. Lots of Linux code exists because someone decided that
> the official story was wrong and proved it so.

Alan,

I agree with your basic sentiment, but the issue here is that supporting
TOE requires changes that are very intimate to the kernel. This is not
like developing I2O which is an edge driver layer, but a core portion
of the kernel.  Some support from the community is going to be needed. 
Currently, any time someone mentions the idea of discussing a TOE 
interface, it's shot down as being evil and bad. 

/me thinks that the HW vendors that really want TOE support need
 to fund some Linux networking folks to go look at the problem
 in detail.

~Deepak

-- 
Deepak Saxena
MontaVista Software - Powering the Embedded Revolution - www.mvista.com

Re: Alan Shih: "TCP IP Offloading Interface"

[Bill Davidsen]

On Mon, 14 Jul 2003, Deepak Saxena wrote:

> On Jul 14 2003, at 21:34, Alan Cox was caught saying:
> > 
> > You don't have to. You can go build and test and maintain a set of TOE patches.
> > Nobody is stopping you. Lots of Linux code exists because someone decided that
> > the official story was wrong and proved it so.
> 
> Alan,
> 
> I agree with your basic sentiment, but the issue here is that supporting
> TOE requires changes that are very intimate to the kernel. This is not
> like developing I2O which is an edge driver layer, but a core portion
> of the kernel.  Some support from the community is going to be needed. 
> Currently, any time someone mentions the idea of discussing a TOE 
> interface, it's shot down as being evil and bad. 
> 
> /me thinks that the HW vendors that really want TOE support need
>  to fund some Linux networking folks to go look at the problem
>  in detail.

My impression is that they have looked at the problem and think it's evil
and bad. Or perhaps more accurately, impractical and undesirable.

Based on my slight understanding, I think that doing TOE would require
vast changes in the way the kernel passes data and status, and given that
2.6 is a butterfly clawing its way out of the cocoon, there's no way a
major change is going to be made until 2.7.

I'm not suggesting that people will like the idea better then, but at
least the concept of major change might not be rejected, no matter what
the change might be.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

>From: Jeff Garzik <jgarzik@pobox.com>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,  linux-kernel@vger.kernel.org
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: Mon, 14 Jul 2003 16:03:01 -0400
>
>David griego wrote:
>>Intel Clusters and Network Storage Volume Platforms Lab reported that it 
>>takes about 1MHz to process 1Mbps on a PIII.  Using this rule of thumb 
>>(they showed it scaling from 400MHz to 800MHz) it would take 10GHz to 
>>process 10Mbps.  Well you might say "what about multi-processers?"  This
>
>Um.  It doesn't take nearly 10Ghz to handle 10Mbps, or even 100Mbps.
Err.  Make that 10GHz for 10Gbps :-)
>
>
>>would be good for people that have multi-processors, but there is a large 
>>segment of embedded processors that are not going have SMP, or be at 10GHz 
>>anytime soon.  Besides that processing interrupts does not scale across 
>>MPs liniarly.  The truth is that communication speeds are outpacing 
>>processor speeds at this time.
>
>If the host CPU is a bottleneck after large-send and checksums have been 
>offloaded, then logically you aren't getting any work done _anyway_. You 
>have to interface with the net stack at some point, in which case you incur 
>a fixed cost, for socket handling, TCP exception handling, etc.
Still other processing going on like RAID, NFS, or CIFS.
>
>Maybe somebody needs to be looking into AMP (asymmetric multiprocessing), 
>too.
Nice artical on AMP for ATM.  I'll try to find a pointer.
>
>	Jeff
>
>
>

_________________________________________________________________
MSN 8 with e-mail virus protection service: 2 months FREE*  
http://join.msn.com/?page=features/virus

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: David griego <dagriego@hotmail.com>
>CC: jgarzik@pobox.com, alan@storlinksemi.com,   Linux Kernel Mailing List 
><linux-kernel@vger.kernel.org>
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: 14 Jul 2003 21:05:53 +0100
>
>On Llu, 2003-07-14 at 20:43, David griego wrote:
> > Intel Clusters and Network Storage Volume Platforms Lab reported that it
> > takes about 1MHz to process 1Mbps on a PIII.  Using this rule of thumb 
>(they
>
>1MHz to proces 1Mbit doing what - file I/O to and from disk, web serving
>- because ToE or otherwise I still have to process the data I receive
>and do something useful with it unless I'm just a router, firewall or
>load balancer. If you want to argue about using gate arrays and hardware
>to accelerate IP routing, balancing and firewall filter cams then you
>might get somewhere - but they dont need to talk TCP.
This was stream testing nTTCP, so no other IO work being done.  Freedom from 
TCP processing would allow for other tasks such as RAID and storage 
virtualization.
>
>Also if its 1MHz per 1Mbit worse case and your ToE engine isnt entirely
>hardware paths capable of sustaining 10Gbit/sec, what happens when I hit
>you with 10Gbit of carefully chosen non optimal frames ?
I'll let the hardware teams figure that out.  From my understanding it going 
to be done.

_________________________________________________________________
MSN 8 with e-mail virus protection service: 2 months FREE*  
http://join.msn.com/?page=features/virus

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
> > Layer one network processing is often handled by ASICS, also some of the
> > fastest encryption engines are hardware.  I suggest we don't wait until 
>your
> > proven wrong before making a decision on TOE.
>
>You don't have to. You can go build and test and maintain a set of TOE 
>patches.
>Nobody is stopping you. Lots of Linux code exists because someone decided 
>that
>the official story was wrong and proved it so.
>
Our team has done this twice aready for Linux (one TOE in software one as an 
ASIC).  It can be hard to make decisions on tradeoffs when the general 
consinsus in Linux is to not support TOE.  I'm sure that once everything is 
said and done we will provide a driver for our TOE to the community.  
Support from other OS venders has been better and feedback from them will 
defenitly influance our hardware design.
>
>Alan
>

_________________________________________________________________
The new MSN 8: smart spam protection and 2 months FREE*  
http://join.msn.com/?page=features/junkmail

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Maw, 2003-07-15 at 00:26, David griego wrote:
> >Are these common cases to be optimized for latency or throughput?
> I would personaly see the common case optimized for throughput on large 
> packets, and allow the smaller packets to be processed by the OS.

Its very very application dependant. Latency is critical to a good file
server, although storage people often like to handwave those numbers
away (not all of them thankfully)

Cluster people want low latency at all times, and latency is the one
thing that is almost impossible to recover once you lose time to it