RE: Kernel 2.6.12-rc6-mm1 & Chelsio driver

RE: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Scott Bardone]

Lukas,

You can download the N210/N110 (ver 2.1.1) from the Chelsio website and use that driver for the T110 with a newer kernel. I have tested that driver up to the 2.6.11 kernel release. It will provide you NIC mode functinoality on your T110 TOE card, you can use it as a module, or try to patch it into a later kernel. If patching it into a kernel, you may need to modify the patch a bit.

-Scott


-----Original Message-----
From: Lukas Hejtmanek [mailto:xhejtman@mail.muni.cz]
Sent: Tue 6/7/2005 11:50 PM
To: Scott Bardone
Cc: Francois Romieu; Jeff Garzik; linux-kernel@vger.kernel.org
Subject: Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver
 
On Tue, Jun 07, 2005 at 07:19:46PM -0700, Scott Bardone wrote:
> It looks like you have a T110 card (10Gb TOE) by the device ID 0006.
> The Chelsio driver which is in the 2.6 mm tree only supports the NIC model 
> cards (N110 & N210).
> 
> We currently don't have the TOE API in the Linux kernel so the TOE 
> functionality does not exist, therefore you can only use the Chelsio 
> modified 2.6.6 kernel for TOE.
> 
> You will need to download the driver from Chelsio's website for the T110. 
> Please send me an email if you don't have a login.

Thanks, we have an account. But I wonder whether T110 card could be used in
newer kernel (as 2.6.6 is rather old and cat /proc/iomap segfaults in kernel).

We do not need TOE functionality, UDP transfer is just fine.

-- 
Lukás Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Wed, Jun 08, 2005 at 10:33:09AM -0700, Scott Bardone wrote:
> You can download the N210/N110 (ver 2.1.1) from the Chelsio website and use
> that driver for the T110 with a newer kernel. I have tested that driver up to
> the 2.6.11 kernel release. It will provide you NIC mode functinoality on your
> T110 TOE card, you can use it as a module, or try to patch it into a later
> kernel. If patching it into a kernel, you may need to modify the patch a bit.

Thanks, however, without CONFIG_CHELSIO_T1_OFFLOAD card is not detected (no
wonder, driver enables T110 card only if offload is used). I do not need TCP
offload engine. With T1 Offload it cannot be compiled - it reject
cxgbtoe-2.1.1-linux-2.6.6-toe_api.patch

So, do I really need Offloading in kernel or should it work with just enableing
card in sources even without Offloading?

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Scott Bardone]

Lukas,

You would need to use the cxgb-2.1.1 driver (NIC only). It is near the bottom of 
the webpage, under "Chelsio N210 / N110 10Gb Ethernet Server Adapter",  Chelsio 
N210 / N110 Linux Driver - Version 2.1.1 (05/17/2005).
<https://service.chelsio.com/drivers/linux/n210/cxgb-2.1.1.tar.gz>
Use the above driver for the T110 in NIC mode. This driver will work for the 
T110 and T210 but only in NIC mode.

The NIC driver does not have CONFIG_CHELSIO_T1_OFFLOAD, so the card will work in 
  NIC only mode.

You would not have success in trying to modify the TOE driver (cxgbtoe-2.1.1) 
because this requires a lot of modifications to the TOM (TCP Offload Module) and 
the TOE API in order to work with a newer kernel.

In the future, we plan on trying to get our TOE API into the Linux kernel, but 
this requires a lot of work and acceptance of TOE by the community first.

-Scott

Lukas Hejtmanek wrote:
> On Wed, Jun 08, 2005 at 10:33:09AM -0700, Scott Bardone wrote:
> 
>>You can download the N210/N110 (ver 2.1.1) from the Chelsio website and use
>>that driver for the T110 with a newer kernel. I have tested that driver up to
>>the 2.6.11 kernel release. It will provide you NIC mode functinoality on your
>>T110 TOE card, you can use it as a module, or try to patch it into a later
>>kernel. If patching it into a kernel, you may need to modify the patch a bit.
> 
> 
> Thanks, however, without CONFIG_CHELSIO_T1_OFFLOAD card is not detected (no
> wonder, driver enables T110 card only if offload is used). I do not need TCP
> offload engine. With T1 Offload it cannot be compiled - it reject
> cxgbtoe-2.1.1-linux-2.6.6-toe_api.patch
> 
> So, do I really need Offloading in kernel or should it work with just enableing
> card in sources even without Offloading?
> 

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Wed, Jun 08, 2005 at 12:11:59PM -0700, Scott Bardone wrote:
> You would need to use the cxgb-2.1.1 driver (NIC only). It is near the 
> bottom of the webpage, under "Chelsio N210 / N110 10Gb Ethernet Server 
> Adapter",  Chelsio N210 / N110 Linux Driver - Version 2.1.1 (05/17/2005).
> <https://service.chelsio.com/drivers/linux/n210/cxgb-2.1.1.tar.gz>
> Use the above driver for the T110 in NIC mode. This driver will work for 
> the T110 and T210 but only in NIC mode.

Unfortunately, this driver does not contain 
CH_DEVICE(6, 0|1.. )

Should I just add it?

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Scott Bardone]

Lukas,

I had to investigate the driver a bit and I am mistaken in saying that the NIC 
driver will support the T110 card. It does not.

The reason for this is that the T110 will use the TCAM even in NIC mode and that 
code is missing from the NIC driver. The TCAM needs to be initialized for using 
TCP traffic in tunneled (non-offload) mode. However, if you just add the 
enumeration for the T110, UDP should work but TCP will not. This would not be 
very useful.

The T210 does not have this issue.

Unfortunately, you will need to use the TOE driver for 2.6.6 kernel.

I apologize for the confusion.

-Scott

Lukas Hejtmanek wrote:
> On Wed, Jun 08, 2005 at 12:11:59PM -0700, Scott Bardone wrote:
> 
>>You would need to use the cxgb-2.1.1 driver (NIC only). It is near the 
>>bottom of the webpage, under "Chelsio N210 / N110 10Gb Ethernet Server 
>>Adapter",  Chelsio N210 / N110 Linux Driver - Version 2.1.1 (05/17/2005).
>><https://service.chelsio.com/drivers/linux/n210/cxgb-2.1.1.tar.gz>
>>Use the above driver for the T110 in NIC mode. This driver will work for 
>>the T110 and T210 but only in NIC mode.
> 
> 
> Unfortunately, this driver does not contain 
> CH_DEVICE(6, 0|1.. )
> 
> Should I just add it?
> 

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Wed, Jun 08, 2005 at 01:05:28PM -0700, Scott Bardone wrote:
> There is one thing you could do though. If want, you could hack the TOE 
> driver to work with the latest kernel in NIC mode. You would need to 
> disable all the offload functions. Copy the source into the kernel tree, 
> then when you build it you will see the unresolved symbols. Disable those 
> sections of code, then you should end up with a NIC driver which will run 
> on the latest kernel.

I have ported your driver for kernel 2.6.6 to kernel 2.6.8 and 2.6.11.12, it
seems that it works (including TOE).

test3:~ # cat /proc/net/toe/devices 
Device           Offload Module       Interfaces
toe0             Chelsio T1           eth0
test3:~ # uname -a
Linux test3 2.6.11.12 #3 SMP Fri Jun 17 01:15:03 CEST 2005 x86_64 x86_64 x86_64
GNU/Linux
test3:~ # 

I can provide patch to mailing list if it is allowed.

-- 
Lukáš Hejtmánek

Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

Hello,

should chelsio 10GE driver work in this kernel? If I do modprobe cxgb, then it
silently returns. No messages in log (dmesg) nor terminal and no new ethX 
device is discoverred.

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Jeff Garzik]

Lukas Hejtmanek wrote:
> Hello,
> 
> should chelsio 10GE driver work in this kernel? If I do modprobe cxgb, then it
> silently returns. No messages in log (dmesg) nor terminal and no new ethX 
> device is discoverred.

I suppose you have Chelsio hardware?

	Jeff

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Tue, Jun 07, 2005 at 02:50:36PM -0400, Jeff Garzik wrote:
> >should chelsio 10GE driver work in this kernel? If I do modprobe cxgb, 
> >then it
> >silently returns. No messages in log (dmesg) nor terminal and no new ethX 
> >device is discoverred.
> 
> I suppose you have Chelsio hardware?

Yes :) 

Bus  2, device   3, function  0:
    Ethernet controller: PCI device 1425:0006 (ASIC Designers Inc) (rev 0).
      IRQ 24.
      Master Capable.  Latency=248.  
      Non-prefetchable 64 bit memory at 0xf6042000 [0xf6042fff].


kernel 2.6.6 and driver from web site:
Chelsio TOE Network Driver - version 2.1.0
eth0: Chelsio T110 1x10GBaseX TOE (rev 1), PCIX 100MHz/64-bit
eth0: 512MB SDRAM, 128MB FCRAM

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Jeff Garzik]

Lukas Hejtmanek wrote:
> On Tue, Jun 07, 2005 at 02:50:36PM -0400, Jeff Garzik wrote:
> 
>>>should chelsio 10GE driver work in this kernel? If I do modprobe cxgb, 
>>>then it
>>>silently returns. No messages in log (dmesg) nor terminal and no new ethX 
>>>device is discoverred.
>>
>>I suppose you have Chelsio hardware?
> 
> 
> Yes :) 
> 
> Bus  2, device   3, function  0:
>     Ethernet controller: PCI device 1425:0006 (ASIC Designers Inc) (rev 0).
>       IRQ 24.
>       Master Capable.  Latency=248.  
>       Non-prefetchable 64 bit memory at 0xf6042000 [0xf6042fff].
> 
> 
> kernel 2.6.6 and driver from web site:
> Chelsio TOE Network Driver - version 2.1.0
> eth0: Chelsio T110 1x10GBaseX TOE (rev 1), PCIX 100MHz/64-bit
> eth0: 512MB SDRAM, 128MB FCRAM

I can't help much beyond this then :(  There should be Chelsio email 
addresses in the driver...

	Jeff

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Tue, Jun 07, 2005 at 03:27:23PM -0400, Jeff Garzik wrote:
> >Bus  2, device   3, function  0:
> >    Ethernet controller: PCI device 1425:0006 (ASIC Designers Inc) (rev 0).
> >      IRQ 24.
> >      Master Capable.  Latency=248.  
> >      Non-prefetchable 64 bit memory at 0xf6042000 [0xf6042fff].
> >
> >
> >kernel 2.6.6 and driver from web site:
> >Chelsio TOE Network Driver - version 2.1.0
> >eth0: Chelsio T110 1x10GBaseX TOE (rev 1), PCIX 100MHz/64-bit
> >eth0: 512MB SDRAM, 128MB FCRAM
> 
> I can't help much beyond this then :(  There should be Chelsio email 
> addresses in the driver...

I'am afraid that they oficially support only 2.6.6 kernel :(

I wonder whether this driver works for someone. If not, it should be removed.

And also, who does it maintain?

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Francois Romieu]

Lukas Hejtmanek <xhejtman@mail.muni.cz> :
[...]
> > >    Ethernet controller: PCI device 1425:0006 (ASIC Designers Inc) (rev 0).
> > >      IRQ 24.
> > >      Master Capable.  Latency=248.  
> > >      Non-prefetchable 64 bit memory at 0xf6042000 [0xf6042fff].

/me greps in -mm

#define CH_DEVICE(devid, ssid, idx) \
	{ PCI_VENDOR_ID_CHELSIO, devid, PCI_ANY_ID, ssid, 0, 0, idx }
[...]
enum {
       CH_BRD_N110_1F,
       CH_BRD_N210_1F,
       CH_BRD_T210_1F,
};
[...]
struct pci_device_id t1_pci_tbl[] = {
	CH_DEVICE(7, 0, CH_BRD_N110_1F),
	CH_DEVICE(10, 1, CH_BRD_N210_1F),
	{ 0, }
};

-> it does not match your 0006 revision. No wonder nothing gets detected.

As a quick hack, one could cross fingers and add a
CH_DEVICE(6, 0, CH_BRD_N110_1F)

or, if it does not work:

CH_DEVICE(6, 1, CH_BRD_N210_1F)

(the drivers code suggests some deep differences between CH_BRD_N210_1F
and CH_BRD_N110_1F wrt irq management for instance)

If it does not work at all, someone will have to dissect the whole
thing. Please fill an entry at bugzilla.kernel.org, add it your lspci info
and make it link the 2.6.6 driver from Chelsio's website.

--
Ueimor

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Tue, Jun 07, 2005 at 10:08:20PM +0200, Francois Romieu wrote:
> -> it does not match your 0006 revision. No wonder nothing gets detected.
> 
> As a quick hack, one could cross fingers and add a
> CH_DEVICE(6, 0, CH_BRD_N110_1F)
> 
> or, if it does not work:
> 
> CH_DEVICE(6, 1, CH_BRD_N210_1F)

I added:

enum {
        CH_BRD_T110_1F,
        CH_BRD_N110_1F,
        CH_BRD_N210_1F,
        CH_BRD_T210_1F,
};

struct pci_device_id t1_pci_tbl[] = {
        CH_DEVICE(6, 0, CH_BRD_T110_1F),
        CH_DEVICE(6, 1, CH_BRD_T110_1F),
        CH_DEVICE(7, 0, CH_BRD_N110_1F),
        CH_DEVICE(10, 1, CH_BRD_N210_1F),
        { 0, }
};

according to 2.6.6 driver. 

However, it seems to be highly unstable. Using iperf it gets broken. Card 
receives packets but it does not transmit after some iperf tests.

Using tcpdump I see things like this:
tcpdump -ni eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
23:05:03.854587 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 1
23:05:04.853853 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 2
23:05:05.853965 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 3
23:05:06.854079 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 4
23:05:07.854193 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 5
 

> If it does not work at all, someone will have to dissect the whole
> thing. Please fill an entry at bugzilla.kernel.org, add it your lspci info
> and make it link the 2.6.6 driver from Chelsio's website.

Should I still add an bugzilla entry? Unfortunately, 2.6.6. driver from website
is accessible only through password.

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Francois Romieu]

Lukas Hejtmanek <xhejtman@mail.muni.cz> :
[...]
> However, it seems to be highly unstable. Using iperf it gets broken. Card 
> receives packets but it does not transmit after some iperf tests.

Hard to tell without further investigation if there is some kind of
wraparound or some unordered acesses to the asic (or worse).

[...]
> Should I still add an bugzilla entry? Unfortunately, 2.6.6. driver from
> website is accessible only through password.

Is it available under anything like a GPL license ?

--
Ueimor

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Scott Bardone]

Hi Lukas,

It looks like you have a T110 card (10Gb TOE) by the device ID 0006.
The Chelsio driver which is in the 2.6 mm tree only supports the NIC model cards 
(N110 & N210).

We currently don't have the TOE API in the Linux kernel so the TOE functionality 
does not exist, therefore you can only use the Chelsio modified 2.6.6 kernel for 
TOE.

You will need to download the driver from Chelsio's website for the T110. Please 
send me an email if you don't have a login.

Thanks,
Scott Bardone
Chelsio Communications
http://www.chelsio.com

Lukas Hejtmanek wrote:
> On Tue, Jun 07, 2005 at 10:08:20PM +0200, Francois Romieu wrote:
> 
>>-> it does not match your 0006 revision. No wonder nothing gets detected.
>>
>>As a quick hack, one could cross fingers and add a
>>CH_DEVICE(6, 0, CH_BRD_N110_1F)
>>
>>or, if it does not work:
>>
>>CH_DEVICE(6, 1, CH_BRD_N210_1F)
> 
> 
> I added:
> 
> enum {
>         CH_BRD_T110_1F,
>         CH_BRD_N110_1F,
>         CH_BRD_N210_1F,
>         CH_BRD_T210_1F,
> };
> 
> struct pci_device_id t1_pci_tbl[] = {
>         CH_DEVICE(6, 0, CH_BRD_T110_1F),
>         CH_DEVICE(6, 1, CH_BRD_T110_1F),
>         CH_DEVICE(7, 0, CH_BRD_N110_1F),
>         CH_DEVICE(10, 1, CH_BRD_N210_1F),
>         { 0, }
> };
> 
> according to 2.6.6 driver. 
> 
> However, it seems to be highly unstable. Using iperf it gets broken. Card 
> receives packets but it does not transmit after some iperf tests.
> 
> Using tcpdump I see things like this:
> tcpdump -ni eth0
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
> 23:05:03.854587 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 1
> 23:05:04.853853 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 2
> 23:05:05.853965 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 3
> 23:05:06.854079 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 4
> 23:05:07.854193 IP 10.0.0.1 > 10.0.0.2: icmp 64: echo request seq 5
>  
> 
> 
>>If it does not work at all, someone will have to dissect the whole
>>thing. Please fill an entry at bugzilla.kernel.org, add it your lspci info
>>and make it link the 2.6.6 driver from Chelsio's website.
> 
> 
> Should I still add an bugzilla entry? Unfortunately, 2.6.6. driver from website
> is accessible only through password.
> 

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Lukas Hejtmanek]

On Tue, Jun 07, 2005 at 07:19:46PM -0700, Scott Bardone wrote:
> It looks like you have a T110 card (10Gb TOE) by the device ID 0006.
> The Chelsio driver which is in the 2.6 mm tree only supports the NIC model 
> cards (N110 & N210).
> 
> We currently don't have the TOE API in the Linux kernel so the TOE 
> functionality does not exist, therefore you can only use the Chelsio 
> modified 2.6.6 kernel for TOE.
> 
> You will need to download the driver from Chelsio's website for the T110. 
> Please send me an email if you don't have a login.

Thanks, we have an account. But I wonder whether T110 card could be used in
newer kernel (as 2.6.6 is rather old and cat /proc/iomap segfaults in kernel).

We do not need TOE functionality, UDP transfer is just fine.

-- 
Lukáš Hejtmánek

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Christoph Hellwig]

On Tue, Jun 07, 2005 at 07:19:46PM -0700, Scott Bardone wrote:
> We currently don't have the TOE API in the Linux kernel so the TOE 
> functionality does not exist, therefore you can only use the Chelsio 
> modified 2.6.6 kernel for TOE.

Care to explain what modifications you do, and whether or not you consider
your card firmware a derived work of the TCP stack because of them?

Re: Kernel 2.6.12-rc6-mm1 & Chelsio driver

[Scott Bardone]

We modify the existing Linux TCP stack to add "hooks" so that our card can 
perform TCP offload and HW based checksum, thus making it possible to see high 
throughput with multiple connections and low CPU utilization.

We add 2 source files to the kernel, toedev.c and offload.c. We also modify 
neighbor.c, tcp.c, tcp_diag.c, tcp_ipv4.c, tcp_timer.c to add functions for our 
TOE capabilities so that the offloaded packet can be sent to our hardware 
(offload) path instead of going through the software (TCP stack) path.

Our processing engine is an ASIC with a HW based TCP stack which processes 
packets with Chelsio's CPL messages (Chelsio Protocol Language). I would not 
consider it a derived work.

-Scott


Christoph Hellwig wrote:
> On Tue, Jun 07, 2005 at 07:19:46PM -0700, Scott Bardone wrote:
> 
>>We currently don't have the TOE API in the Linux kernel so the TOE 
>>functionality does not exist, therefore you can only use the Chelsio 
>>modified 2.6.6 kernel for TOE.
> 
> 
> Care to explain what modifications you do, and whether or not you consider
> your card firmware a derived work of the TCP stack because of them?
>