Re: Alan Shih: "TCP IP Offloading Interface"

Re: Alan Shih: "TCP IP Offloading Interface"

[David griego]

Embedded does not simply include toasters and fridges, it also includes NAS 
and SAN appliances as well as telco gear.  These types of devices have 
advanced memory subsystems and run processors such as PPC and ARM.  One of 
the most limiting factors in these types of devices is power consumption.  
This usually limits the number of cores and frequency these cores.  
Offloading the processing of protocol stacks to ASICS would have a great 
impact in performance.  If you are going to embed a high frequency chip in 
your embedded devices I would recommend developing a heater not a fridge.


>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,   Linux Kernel Mailing List 
><linux-kernel@vger.kernel.org>
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: 14 Jul 2003 20:42:53 +0100
>
>On Llu, 2003-07-14 at 19:46, David griego wrote:
> > IMHO, there are several cases for some type of TCP/IP offload.  One is 
>for
> > embedded systems that are just not capable of doing 1Gbps+.  Another is 
>with
>
>My fridge doesn't need to do 10Gbit a second, and for most other
>embedded the constraints are ram bandwidth and nothing else. Since
>deeply embedded stuff also doesn't run with MMUs or runs 'partially
>trusted' most of the VM games and the socket api games also go away.

See PPC and ARM architecture for the use of MMUs in embedded systems

>
>I've done deeply embedded tcp/ip. I don't buy the argument, embedded
>gains the least of all from ToE.
>
> > 10GbE, even high end servers will not be able keep up with TCP
> > processing/data movement at these speeds.  Not being proactive in 
>adopting
>
>They said that about 10Mbit until Van showed them a thing or two. They
>said it about 100Mbit, they said it about gigabit.

Not the case for embedded.  I understand your viewpoint from the server 
space though.
>
> > TCP/IP offload will force Linux into accepting some scheme that will not
> > necessarily be best.
>
>TCP/IP is an exercise in two things when you are running at speed
>
>1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
>	sg does, on card target buffers do with decent chipsets.

A TOE enabled with RDDP would help eliminate the kernel to user space copy 
(and in the case of SAMBA the copy back to the kernel).  This would reduce 
the memory system loading by a third to a half.
>
>2.	Handling in order perfectly predicted data streams. ToE is
>	overkill for this. Thats about latency to memory and touching
>	as little as possible. The main CPU has a rather good connection
>	to main memory.
>
Yes, RDDP would be nice to have though for the reason stated for #1, so the 
hardware would need to at least be TCP aware.

>ToE is also horribly vulnerable to attack because putting it on a card
>dictates relatively low CPU power and low power consumption as well as
>rather nasty pricing issues. Historically low power devices have
>repeatedly been screwed by attackers hitting software or other slow
>paths in the device to attack it.
The use of ASICs could ensure that TCP processing is as quick as wire speed

>
>This is before we get into the delights of multipath routing across
>different vendors cards, firewalling, traffic shaping, retrofitting new
>features, questions about what happens with an old ToE card when its
>got a hole...
Try to keep the datapath processing on the TOE, and everything else in the 
OS.  Also give the API the ability to turn of the TOE if a hole exists and 
use it like a regular NIC.
>
>The internet land speed record is held by a non ToE system, let me know
>when that changes.
>
Layer one network processing is often handled by ASICS, also some of the 
fastest encryption engines are hardware.  I suggest we don't wait until your 
proven wrong before making a decision on TOE.

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*.  
http://join.msn.com/?page=features/virus

RE: Alan Shih: "TCP IP Offloading Interface"

[Alan Shih]

In a typical application of NAS ASIC, it is easier to place 2 300MHz
processor than a 1 GHz processor in cost. This line of reasoning forces me
to consider making 1 of the processor to be TOE while the other one deals
with disk/FS manipulations.

Alan


-----Original Message-----
From: David griego [mailto:dagriego@hotmail.com]
Sent: Monday, July 14, 2003 1:19 PM
To: alan@lxorguk.ukuu.org.uk
Cc: alan@storlinksemi.com; linux-kernel@vger.kernel.org
Subject: Re: Alan Shih: "TCP IP Offloading Interface"


Embedded does not simply include toasters and fridges, it also includes NAS
and SAN appliances as well as telco gear.  These types of devices have
advanced memory subsystems and run processors such as PPC and ARM.  One of
the most limiting factors in these types of devices is power consumption.
This usually limits the number of cores and frequency these cores.
Offloading the processing of protocol stacks to ASICS would have a great
impact in performance.  If you are going to embed a high frequency chip in
your embedded devices I would recommend developing a heater not a fridge.


>From: Alan Cox <alan@lxorguk.ukuu.org.uk>
>To: David griego <dagriego@hotmail.com>
>CC: alan@storlinksemi.com,   Linux Kernel Mailing List
><linux-kernel@vger.kernel.org>
>Subject: Re: Alan Shih: "TCP IP Offloading Interface"
>Date: 14 Jul 2003 20:42:53 +0100
>
>On Llu, 2003-07-14 at 19:46, David griego wrote:
> > IMHO, there are several cases for some type of TCP/IP offload.  One is
>for
> > embedded systems that are just not capable of doing 1Gbps+.  Another is
>with
>
>My fridge doesn't need to do 10Gbit a second, and for most other
>embedded the constraints are ram bandwidth and nothing else. Since
>deeply embedded stuff also doesn't run with MMUs or runs 'partially
>trusted' most of the VM games and the socket api games also go away.

See PPC and ARM architecture for the use of MMUs in embedded systems

>
>I've done deeply embedded tcp/ip. I don't buy the argument, embedded
>gains the least of all from ToE.
>
> > 10GbE, even high end servers will not be able keep up with TCP
> > processing/data movement at these speeds.  Not being proactive in
>adopting
>
>They said that about 10Mbit until Van showed them a thing or two. They
>said it about 100Mbit, they said it about gigabit.

Not the case for embedded.  I understand your viewpoint from the server
space though.
>
> > TCP/IP offload will force Linux into accepting some scheme that will not
> > necessarily be best.
>
>TCP/IP is an exercise in two things when you are running at speed
>
>1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
>	sg does, on card target buffers do with decent chipsets.

A TOE enabled with RDDP would help eliminate the kernel to user space copy
(and in the case of SAMBA the copy back to the kernel).  This would reduce
the memory system loading by a third to a half.
>
>2.	Handling in order perfectly predicted data streams. ToE is
>	overkill for this. Thats about latency to memory and touching
>	as little as possible. The main CPU has a rather good connection
>	to main memory.
>
Yes, RDDP would be nice to have though for the reason stated for #1, so the
hardware would need to at least be TCP aware.

>ToE is also horribly vulnerable to attack because putting it on a card
>dictates relatively low CPU power and low power consumption as well as
>rather nasty pricing issues. Historically low power devices have
>repeatedly been screwed by attackers hitting software or other slow
>paths in the device to attack it.
The use of ASICs could ensure that TCP processing is as quick as wire speed

>
>This is before we get into the delights of multipath routing across
>different vendors cards, firewalling, traffic shaping, retrofitting new
>features, questions about what happens with an old ToE card when its
>got a hole...
Try to keep the datapath processing on the TOE, and everything else in the
OS.  Also give the API the ability to turn of the TOE if a hole exists and
use it like a regular NIC.
>
>The internet land speed record is held by a non ToE system, let me know
>when that changes.
>
Layer one network processing is often handled by ASICS, also some of the
fastest encryption engines are hardware.  I suggest we don't wait until your
proven wrong before making a decision on TOE.

_________________________________________________________________
MSN 8 helps eliminate e-mail viruses. Get 2 months FREE*.
http://join.msn.com/?page=features/virus

Re: Alan Shih: "TCP IP Offloading Interface"

[Alan Cox]

On Llu, 2003-07-14 at 21:19, David griego wrote:
> Embedded does not simply include toasters and fridges, it also includes NAS 
> and SAN appliances as well as telco gear.  These types of devices have 

I have some sympathy with SAN developers, because you can treat the link
as dedicated and just for the SAN so you can implement iSCSI optimised
code. The right engineering approach probably consists of removing iSCSI
and doing the job right but I appreciate there are non engineering
issues.

> advanced memory subsystems and run processors such as PPC and ARM.  One of 
> the most limiting factors in these types of devices is power consumption.  

Memory bandwidth is your killer quite often, you have to keep the CPU
away from the data and often even off the memory bus holding most of the
data.

> >deeply embedded stuff also doesn't run with MMUs or runs 'partially
> >trusted' most of the VM games and the socket api games also go away.
> 
> See PPC and ARM architecture for the use of MMUs in embedded systems

You still have the partial trust stuff and the ability to violate the
socket api in the interest of speed - that helps.

> >TCP/IP is an exercise in two things when you are running at speed
> >
> >1.	Finding the memory bandwidth - ToE doesn't help, checksums do,
> >	sg does, on card target buffers do with decent chipsets.
> 
> A TOE enabled with RDDP would help eliminate the kernel to user space copy 
> (and in the case of SAMBA the copy back to the kernel).  This would reduce 
> the memory system loading by a third to a half.

Thats mostly an API issue. Socket API found non optimal after 20 years.
Hardly shocking news - the cluster people already changed API, Larry
McVoy proposed stuff like splice years ago because he saw it coming.

> >2.	Handling in order perfectly predicted data streams. ToE is
> >	overkill for this. Thats about latency to memory and touching
> >	as little as possible. The main CPU has a rather good connection
> >	to main memory.
> >
> Yes, RDDP would be nice to have though for the reason stated for #1, so the 
> hardware would need to at least be TCP aware.

TCP aware hardware is good, segmentation, prediction, buffer/sequence
matchers etc but you don't want the policy parts in silicon

> >repeatedly been screwed by attackers hitting software or other slow
> >paths in the device to attack it.
> The use of ASICs could ensure that TCP processing is as quick as wire speed

Only if your ASIC worst case is wire speed. If your ASIC has one path it
must drop to software for and has a low powered internal CPU to fix up,
you just lost and you'll plow like a cisco with too many filter rules to
do in silicon.

> Try to keep the datapath processing on the TOE, and everything else in the 
> OS.  Also give the API the ability to turn of the TOE if a hole exists and 
> use it like a regular NIC.

Some of this is certainly about how you partition the load - its no
different to things like RAID. We've had hardware raid, software raid,
and finally people are popping up with neat hybrids. We've had software
audio, hardware audio and again now we are seeing hybrids that do
different bits in hardware and software - ditto modems.

> >The internet land speed record is held by a non ToE system, let me know
> >when that changes.
> >
> Layer one network processing is often handled by ASICS, also some of the 
> fastest encryption engines are hardware.  I suggest we don't wait until your 
> proven wrong before making a decision on TOE.

You don't have to. You can go build and test and maintain a set of TOE patches.
Nobody is stopping you. Lots of Linux code exists because someone decided that
the official story was wrong and proved it so.

Alan

Re: Alan Shih: "TCP IP Offloading Interface"

[Deepak Saxena]

On Jul 14 2003, at 21:34, Alan Cox was caught saying:
> 
> You don't have to. You can go build and test and maintain a set of TOE patches.
> Nobody is stopping you. Lots of Linux code exists because someone decided that
> the official story was wrong and proved it so.

Alan,

I agree with your basic sentiment, but the issue here is that supporting
TOE requires changes that are very intimate to the kernel. This is not
like developing I2O which is an edge driver layer, but a core portion
of the kernel.  Some support from the community is going to be needed. 
Currently, any time someone mentions the idea of discussing a TOE 
interface, it's shot down as being evil and bad. 

/me thinks that the HW vendors that really want TOE support need
 to fund some Linux networking folks to go look at the problem
 in detail.

~Deepak

-- 
Deepak Saxena
MontaVista Software - Powering the Embedded Revolution - www.mvista.com

Error compiling, scsi 2.6.0-test1

[backblue]

I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?

...

  CC      drivers/pnp/quirks.o
  CC      drivers/pnp/names.o
  CC      drivers/pnp/system.o
  LD      drivers/pnp/built-in.o
  CC      drivers/scsi/ini9100u.o
drivers/scsi/ini9100u.c:111:2: #error Please convert me to Documentation/DMA-mapping.txt
drivers/scsi/ini9100u.c:146: warning: initialization from incompatible pointer type
drivers/scsi/ini9100u.c:151: warning: initialization from incompatible pointer type
drivers/scsi/ini9100u.c:152: warning: initialization from incompatible pointer type
drivers/scsi/ini9100u.c: In function `i91uAppendSRBToQueue':
drivers/scsi/ini9100u.c:241: error: structure has no member named `next'
drivers/scsi/ini9100u.c:246: error: structure has no member named `next'
drivers/scsi/ini9100u.c: In function `i91uPopSRBFromQueue':
drivers/scsi/ini9100u.c:268: error: structure has no member named `next'
drivers/scsi/ini9100u.c:269: error: structure has no member named `next'
drivers/scsi/ini9100u.c: In function `i91uBuildSCB':
drivers/scsi/ini9100u.c:507: error: structure has no member named `address'
drivers/scsi/ini9100u.c:516: error: structure has no member named `address'
make[2]: *** [drivers/scsi/ini9100u.o] Error 1
make[1]: *** [drivers/scsi] Error 2
make: *** [drivers] Error 2

Re: Error compiling, scsi 2.6.0-test1

[Rahul Karnik]

backblue wrote:
<snip>
> drivers/scsi/ini9100u.c:111:2: #error Please convert me to Documentation/DMA-mapping.txt
<snip>

Looks like the driver has not been converted to work in 2.5.

-Rahul

Re: Error compiling, scsi 2.6.0-test1

[Alan Cox]

On Mer, 2003-07-16 at 23:28, backblue wrote:
> I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?

Nobody has coverted this driver to 2.6 yet. If someone does then it will
get merged in, if not the initio support will get deleted in time.

Re: Error compiling, scsi 2.6.0-test1

[backblue]

hello Alan,

I need this driver, but i dont know anouth of C, to code a new one, that old's on 2.6.0 :(, where to start? i really need it, i have everything scsi on my computer, with this controler, and i dont like the idea, of dont have suport to it!!
On 17 Jul 2003 12:51:42 +0100
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> On Mer, 2003-07-16 at 23:28, backblue wrote:
> > I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?
> 
> Nobody has coverted this driver to 2.6 yet. If someone does then it will
> get merged in, if not the initio support will get deleted in time.
> 

Re: Alan Shih: "TCP IP Offloading Interface"

[Bill Davidsen]

On Mon, 14 Jul 2003, Deepak Saxena wrote:

> On Jul 14 2003, at 21:34, Alan Cox was caught saying:
> > 
> > You don't have to. You can go build and test and maintain a set of TOE patches.
> > Nobody is stopping you. Lots of Linux code exists because someone decided that
> > the official story was wrong and proved it so.
> 
> Alan,
> 
> I agree with your basic sentiment, but the issue here is that supporting
> TOE requires changes that are very intimate to the kernel. This is not
> like developing I2O which is an edge driver layer, but a core portion
> of the kernel.  Some support from the community is going to be needed. 
> Currently, any time someone mentions the idea of discussing a TOE 
> interface, it's shot down as being evil and bad. 
> 
> /me thinks that the HW vendors that really want TOE support need
>  to fund some Linux networking folks to go look at the problem
>  in detail.

My impression is that they have looked at the problem and think it's evil
and bad. Or perhaps more accurately, impractical and undesirable.

Based on my slight understanding, I think that doing TOE would require
vast changes in the way the kernel passes data and status, and given that
2.6 is a butterfly clawing its way out of the cocoon, there's no way a
major change is going to be made until 2.7.

I'm not suggesting that people will like the idea better then, but at
least the concept of major change might not be rejected, no matter what
the change might be.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: Error compiling, scsi 2.6.0-test1

[Joshua Schmidlkofer]

Not to seem heartless, but it does work from 2.4, you may have to stick
with that till someone ports the driver forward.  Linux remains a
volunteer effort.  The kernel maintainers do not have a card to develop
against, and they don't need the hardware.  If no one else does either,
then the card becomes un-updated.   I would recommend finding people who
use the card, and trying to get some support for updating it.

have a nice day,


joshua

On Thu, 2003-07-17 at 11:59, backblue wrote:
> hello Alan,
> 
> I need this driver, but i dont know anouth of C, to code a new one, that old's on 2.6.0 :(, where to start? i really need it, i have everything scsi on my computer, with this controler, and i dont like the idea, of dont have suport to it!!
> On 17 Jul 2003 12:51:42 +0100
> Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> 
> > On Mer, 2003-07-16 at 23:28, backblue wrote:
> > > I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?
> > 
> > Nobody has coverted this driver to 2.6 yet. If someone does then it will
> > get merged in, if not the initio support will get deleted in time.
> > 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Error compiling, scsi 2.6.0-test1

[Steven Cole]

On Thu, 2003-07-17 at 12:59, backblue wrote:
> hello Alan,
> 
> I need this driver, but i dont know anouth of C, to code a new one, that old's on 2.6.0 :(, where to start? i really need it, i have everything scsi on my computer, with this controler, and i dont like the idea, of dont have suport to it!!
> On 17 Jul 2003 12:51:42 +0100
> Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> 
> > On Mer, 2003-07-16 at 23:28, backblue wrote:
> > > I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?
> > 
> > Nobody has coverted this driver to 2.6 yet. If someone does then it will
> > get merged in, if not the initio support will get deleted in time.
> > 

It looks like this came up a year ago:
http://marc.theaimsgroup.com/?l=linux-kernel&m=102526383927111&w=2

You might plead with Doug to finish the conversion.

Steven

Re: Error compiling, scsi 2.6.0-test1

[Doug Ledford]

Steven Cole wrote:
> On Thu, 2003-07-17 at 12:59, backblue wrote:
> 
>>hello Alan,
>>
>>I need this driver, but i dont know anouth of C, to code a new one, that old's on 2.6.0 :(, where to start? i really need it, i have everything scsi on my computer, with this controler, and i dont like the idea, of dont have suport to it!!
>>On 17 Jul 2003 12:51:42 +0100
>>Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
>>
>>
>>>On Mer, 2003-07-16 at 23:28, backblue wrote:
>>>
>>>>I have gcc 3.3, on x86 machine, i have this error, compiling the suport for my scsi card, someone know the problem?
>>>
>>>Nobody has coverted this driver to 2.6 yet. If someone does then it will
>>>get merged in, if not the initio support will get deleted in time.
>>>
> 
> 
> It looks like this came up a year ago:
> http://marc.theaimsgroup.com/?l=linux-kernel&m=102526383927111&w=2
> 
> You might plead with Doug to finish the conversion.
> 
> Steven

I wretched horribly when I read that driver.  Then I started to renounce 
my computer ways to become a Tibetten Monk and live out the remainder of 
my days in peace and solice.  Unfortunately, about that time, a space 
thread passed another thread, opened up a short lived wormhole, and I 
got a futuristic glimpse of myself in a robe and bald and I immediately 
came back to my senses.  I still couldn't handle that driver code 
though.  The urge to fix massive brokenness other than just PCI DMA code 
issues was overwhelming to my right hand's obsessive/compulsive disorder.

However, as a hint to anyone out there, I fixed the other initio driver 
a long ways back.  A person could pull those 3 (I think) changesets from 
the kernel tree and see what I did.  If you ignore all the changes to 
eliminate the fixed array of hosts and that kind of crap, and just 
concentrate on the changes that involve the PCI DMA mapping, they should 
apply almost perfectly into this other initio driver as they are very 
close copies of each other.


-- 
   Doug Ledford <dledford@redhat.com>     919-754-3700 x44233
            Red Hat, Inc.
            1801 Varsity Dr.
            Raleigh, NC 27606