ACK behaviour difference LRO/GRO

ACK behaviour difference LRO/GRO

[Markus Stockhausen]

[-- Attachment #1: Type: text/plain, Size: 3471 bytes --]

Hello,

about two month we had some problems with IPoIB transfer speeds . 
See more http://marc.info/?l=linux-rdma&m=137823326109158&w=2
After some quite hard test iterations the problem seems to come from the 
IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38. 

I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
ib_ipoib module against it. This way I can run a direct comparison
between the old and new module. The major difference between the 
two version is inside the ipoib_ib_handle_rx_wc() function:

2.6.37: lro_receive_skb(&priv->lro.lro_mgr, skb, NULL);
2.6.38: napi_gro_receive(&priv->napi, skb);

As in the last post we use ConnectX cards in datagram mode with a
2044 MTU.  We read a file sequentially from a NFS server into /dev/null. 
We just want to get the wire speed neglecting hard drives. The 
hardware is slightly newer so we get different transfer speeds but 
the overall effect should be evident. The server uses a 3.5 kernel and 
is not changed during the tests.

With 2.6.37 IPoIB module on the client side and LRO enabled the 
speed is 950 MByte/sec. On the NFS server side a tcpdump trace 
reads like:

19:51:51.432630 IP 10.10.30.251.nfs > 10.10.30.1.781: 
  Flags [P.], seq 1008434065:1008497161, ack 617432, 
  win 688, options [nop,nop,TS val 133047292 ecr 429568], 
  length 63096
19:51:51.432672 IP 10.10.30.1.781 > 10.10.30.251.nfs: 
  Flags [.], ack 1008241041, win 24576, options 
  [nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432677 IP 10.10.30.251.nfs > 10.10.30.1.781: 
  Flags [.], seq 1008497161:1008560905, ack 617432, 
  win 688, options [nop,nop,TS val 133047292 ecr 429568], 
  length 63744
19:51:51.432725 IP 10.10.30.1.781 > 10.10.30.251.nfs: 
  Flags [.], ack 1008304585, win 24576, options 
  [nop,nop,TS val 429568 ecr 133047292], length 0
19:51:51.432729 IP 10.10.30.251.nfs > 10.10.30.1.781: 
  Flags [.], seq 1008560905:1008624649, ack 617432, 
  win 688, options [nop,nop,TS val 133047292 ecr 429568], 
length 63744

With some slight differences here and there the client sends only
1 ack for about 60k of transferred data. With 2.6.38 module and
onwards (GRO enabled) the speed drops down to 380 MByte/sec 
and a different transfer pattern.

19:58:14.631430 IP 10.10.30.251.nfs > 10.10.30.1.ircs: 
  Flags [.], seq 722492293:722502253, ack 442312, win 537, 
  options [nop,nop,TS val 133143092 ecr 467889], length 9960
19:58:14.631460 IP 10.10.30.1.ircs > 10.10.30.251.nfs: 
  Flags [.], ack 722478181, win 24562, options 
  [nop,nop,TS val 467889 ecr 133143092], length 0
19:58:14.631485 IP 10.10.30.1.ircs > 10.10.30.251.nfs: 
  Flags [.], ack 722478181, win 24562, options 
  [nop,nop,TS val 467889 ecr 133143092,nop,nop,sack 1 
  {722480117:722482333}], length 0
19:58:14.631510 IP 10.10.30.1.ircs > 10.10.30.251.nfs: 
  Flags [.], ack 722488197, win 24562, options [nop,nop,TS 
  val 467889 ecr 133143092], length 0
19:58:14.631534 IP 10.10.30.1.ircs > 10.10.30.251.nfs: 
  Flags [.], ack 722494229, win 24562, options 
  [nop,nop,TS val 467889 ecr 133143092], length 0

It seems as if the NFS client acknowledges every 2K packet
separately. I thought that it may come from missing 
coalescing parameters and tried a  "ethtool -C ib0 rx-usecs 5"
on both machines but without success.

I'm quite lost now maybe someone can give a tip if I'm 
missing something.

Best regards.

Markus=

[-- Attachment #2: InterScan_Disclaimer.txt --]
[-- Type: text/plain, Size: 1650 bytes --]

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

Re: ACK behaviour difference LRO/GRO

[Wendy Cheng]

On Mon, Oct 28, 2013 at 12:34 PM, Markus Stockhausen
<stockhausen-FhFLwxQf9jkb1SvskN2V4Q@public.gmane.org> wrote:
> Hello,
>
> about two month we had some problems with IPoIB transfer speeds .
> See more http://marc.info/?l=linux-rdma&m=137823326109158&w=2
> After some quite hard test iterations the problem seems to come from the
> IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38.
>
> I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
> ib_ipoib module against it. This way I can run a direct comparison
> between the old and new module. The major difference between the
> two version is inside the ipoib_ib_handle_rx_wc() function:
>
> 2.6.37: lro_receive_skb(&priv->lro.lro_mgr, skb, NULL);
> 2.6.38: napi_gro_receive(&priv->napi, skb);
>
> As in the last post we use ConnectX cards in datagram mode with a
> 2044 MTU.  We read a file sequentially from a NFS server into /dev/null.
> We just want to get the wire speed neglecting hard drives. The
> hardware is slightly newer so we get different transfer speeds but
> the overall effect should be evident. The server uses a 3.5 kernel and
> is not changed during the tests.
>
> With 2.6.37 IPoIB module on the client side and LRO enabled the
> speed is 950 MByte/sec. On the NFS server side a tcpdump trace
> reads like:
>
> 19:51:51.432630 IP 10.10.30.251.nfs > 10.10.30.1.781:
>   Flags [P.], seq 1008434065:1008497161, ack 617432,
>   win 688, options [nop,nop,TS val 133047292 ecr 429568],
>   length 63096
> 19:51:51.432672 IP 10.10.30.1.781 > 10.10.30.251.nfs:
>   Flags [.], ack 1008241041, win 24576, options
>   [nop,nop,TS val 429568 ecr 133047292], length 0
> 19:51:51.432677 IP 10.10.30.251.nfs > 10.10.30.1.781:
>   Flags [.], seq 1008497161:1008560905, ack 617432,
>   win 688, options [nop,nop,TS val 133047292 ecr 429568],
>   length 63744
> 19:51:51.432725 IP 10.10.30.1.781 > 10.10.30.251.nfs:
>   Flags [.], ack 1008304585, win 24576, options
>   [nop,nop,TS val 429568 ecr 133047292], length 0
> 19:51:51.432729 IP 10.10.30.251.nfs > 10.10.30.1.781:
>   Flags [.], seq 1008560905:1008624649, ack 617432,
>   win 688, options [nop,nop,TS val 133047292 ecr 429568],
> length 63744
>
> With some slight differences here and there the client sends only
> 1 ack for about 60k of transferred data. With 2.6.38 module and
> onwards (GRO enabled) the speed drops down to 380 MByte/sec
> and a different transfer pattern.
>
> 19:58:14.631430 IP 10.10.30.251.nfs > 10.10.30.1.ircs:
>   Flags [.], seq 722492293:722502253, ack 442312, win 537,
>   options [nop,nop,TS val 133143092 ecr 467889], length 9960
> 19:58:14.631460 IP 10.10.30.1.ircs > 10.10.30.251.nfs:
>   Flags [.], ack 722478181, win 24562, options
>   [nop,nop,TS val 467889 ecr 133143092], length 0
> 19:58:14.631485 IP 10.10.30.1.ircs > 10.10.30.251.nfs:
>   Flags [.], ack 722478181, win 24562, options
>   [nop,nop,TS val 467889 ecr 133143092,nop,nop,sack 1
>   {722480117:722482333}], length 0
> 19:58:14.631510 IP 10.10.30.1.ircs > 10.10.30.251.nfs:
>   Flags [.], ack 722488197, win 24562, options [nop,nop,TS
>   val 467889 ecr 133143092], length 0
> 19:58:14.631534 IP 10.10.30.1.ircs > 10.10.30.251.nfs:
>   Flags [.], ack 722494229, win 24562, options
>   [nop,nop,TS val 467889 ecr 133143092], length 0
>
> It seems as if the NFS client acknowledges every 2K packet
> separately. I thought that it may come from missing
> coalescing parameters and tried a  "ethtool -C ib0 rx-usecs 5"
> on both machines but without success.
>
> I'm quite lost now maybe someone can give a tip if I'm
> missing something.
>

Nice work! Look like napi_gro_receive() does not do the work it is
supposed to do ?! My (embedded NFS client) system was on 2.6.38 kernel
but we use ipoib kmod from OFED 1.5.4.1 - so we're still on
lro_receive_skb() path that does not have this issue.

I'll try it out later this week to see what is going on. Mellanox
folks or Roland may have more to say.

-- Wendy
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ACK behaviour difference LRO/GRO

[Or Gerlitz]

On 28/10/2013 21:34, Markus Stockhausen wrote:
> After some quite hard test iterations the problem seems to come from the
> IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38.
>
> I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
> ib_ipoib module against it. This way I can run a direct comparison
> between the old and new module. The major difference between the
> two version is inside the ipoib_ib_handle_rx_wc() function:
>
> 2.6.37: lro_receive_skb(&priv->lro.lro_mgr, skb, NULL);
> 2.6.38: napi_gro_receive(&priv->napi, skb);

These two commits that went in 3.3

936d7de IPoIB: Stop lying about hard_header_len and use skb->cb to stash 
LL addresses
a0417fa net: Make qdisc_skb_cb upper size bound explicit

were supposed to make IPoIB/GRO to work properly, specifically with 
them, you should see aggregation coming into play

I think Yishai Hadas from Mellanox was looking on that too, do we have 
any insights on the matter?

Or.


Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

AW: ACK behaviour difference LRO/GRO

[Markus Stockhausen]

[-- Attachment #1: Type: text/plain, Size: 1957 bytes --]

> ________________________________________
> Von: Or Gerlitz [ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
> Gesendet: Dienstag, 29. Oktober 2013 09:31
> An: Markus Stockhausen; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Yishai Hadas
> Cc: s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org; Erez Shitrit; Saeed Mahameed
> Betreff: Re: ACK behaviour difference LRO/GRO
> 
> On 28/10/2013 21:34, Markus Stockhausen wrote:
> > After some quite hard test iterations the problem seems to come from the
> > IPoIB switch from LRO to GRO between kernels 2.6.37 and 2.6.38.
> >
> > I built a test setup with a 2.6.38 kernel and additionaly compiled a 2.6.37
> > ib_ipoib module against it. This way I can run a direct comparison
> > between the old and new module. The major difference between the
> > two version is inside the ipoib_ib_handle_rx_wc() function:
> >
> > 2.6.37: lro_receive_skb(&priv->lro.lro_mgr, skb, NULL);
> > 2.6.38: napi_gro_receive(&priv->napi, skb);
> 
> These two commits that went in 3.3
> 
> 936d7de IPoIB: Stop lying about hard_header_len and use skb->cb to stash
> LL addresses
> a0417fa net: Make qdisc_skb_cb upper size bound explicit
> 
> were supposed to make IPoIB/GRO to work properly, specifically with
> them, you should see aggregation coming into play
> 
> I think Yishai Hadas from Mellanox was looking on that too, do we have
> any insights on the matter?
> 
> Or.

At least for the 2.6.38 that sounds clear. My initial post was about
3.5. and 3.10 test kernels that showed the missing aggregation. So
I'm still a bit away from a solution. I will try to get the test machine
back to 3.10/3.11 to validate it once again. 

Just to be on the right way: What are the basics to get GRO working 
with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode? 

- enable GRO with ethtool.
- Activate Coalescing with ethtool? If yes how?

Thanks for the help.

Markus
=

[-- Attachment #2: InterScan_Disclaimer.txt --]
[-- Type: text/plain, Size: 1650 bytes --]

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

Re: AW: ACK behaviour difference LRO/GRO

[Or Gerlitz]

On 29/10/2013 13:10, Markus Stockhausen wrote:
> Just to be on the right way: What are the basics to get GRO working
> with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode?
>
> - enable GRO with ethtool.
> - Activate Coalescing with ethtool? If yes how?
GRO is SW element of the network stack, so the HCA doesn't matter, you 
need IPoIB to support that correct and enable it with ethtool
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AW: ACK behaviour difference LRO/GRO

[Erez Shitrit]

בתאריך 10/29/2013 1:43 PM, ציטוט Or Gerlitz:
> On 29/10/2013 13:10, Markus Stockhausen wrote:
>> Just to be on the right way: What are the basics to get GRO working
>> with a ConnectX (not 2 or 3) card in 2044 MTU datagram mode?
>>
>> - enable GRO with ethtool.
>> - Activate Coalescing with ethtool? If yes how?
> GRO is SW element of the network stack, so the HCA doesn't matter, you 
> need IPoIB to support that correct and enable it with ethtool
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
In addition to what Or just wrote,
GRO currently doesn't work on ipoib interfaces, that according to bad 
handling mac address that are not 6 bytes (we have plans to fix that in 
the near future), that is the reason you don't see 64k packets on 
tcpdump (like you see in LRO).

you have few options:
1. use old kernels and enable LRO (which works fine with ipoib)
2. use CM mode instead of UD, that probably will improve the performance.

Thanks, Erez
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AW: ACK behaviour difference LRO/GRO

[Or Gerlitz]

On 29/10/2013 14:55, Erez Shitrit wrote:
> In addition to what Or just wrote,
> GRO currently doesn't work on ipoib interfaces, that according to bad 
> handling mac address that are not 6 bytes (we have plans to fix that 
> in the near future), that is the reason you don't see 64k packets on 
> tcpdump (like you see in LRO). 

I just checked with net-next which is 3.12-rc6+ and there IS 
aggregationfor datagram mode

> 15:56:40.983883 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801688305:1801692289, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 3984
> 15:56:40.983942 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801692289:1801756033, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 63744
> 15:56:40.984027 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801756033:1801819777, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 63744
> 15:56:40.984079 IP 192.168.20.17.40861 > 192.168.20.18.55714: Flags 
> [.], ack 1801688305, win 1544, options [nop,nop,TS val 305403520 ecr 
> 44014459], length 0
> 15:56:40.984104 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801819777:1801823649, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 3872
> 15:56:40.984159 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801823649:1801883521, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 59872
> 15:56:40.984214 IP 192.168.20.17.40861 > 192.168.20.18.55714: Flags 
> [.], ack 1801819777, win 1009, options [nop,nop,TS val 305403520 ecr 
> 44014459], length 0
> 15:56:40.984241 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags 
> [.], seq 1801883521:1801887393, ack 1, win 220, options [nop,nop,TS 
> val 44014459 ecr 305403520], length 3872


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

AW: AW: ACK behaviour difference LRO/GRO

[Markus Stockhausen]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

> Von: Or Gerlitz [ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
> Gesendet: Dienstag, 29. Oktober 2013 14:58
> An: Erez Shitrit
> Cc: Markus Stockhausen; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Yishai Hadas; s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org; Erez Shitrit; Saeed Mahameed
> Betreff: Re: AW: ACK behaviour difference LRO/GRO
> 
> On 29/10/2013 14:55, Erez Shitrit wrote:
> > In addition to what Or just wrote,
> > GRO currently doesn't work on ipoib interfaces, that according to bad
> > handling mac address that are not 6 bytes (we have plans to fix that
> > in the near future), that is the reason you don't see 64k packets on
> > tcpdump (like you see in LRO).
> 
> I just checked with net-next which is 3.12-rc6+ and there IS
> aggregationfor datagram mode
> 
> > 15:56:40.983883 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801688305:1801692289, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 3984
> > 15:56:40.983942 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801692289:1801756033, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 63744
> > 15:56:40.984027 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801756033:1801819777, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 63744
> > 15:56:40.984079 IP 192.168.20.17.40861 > 192.168.20.18.55714: Flags
> > [.], ack 1801688305, win 1544, options [nop,nop,TS val 305403520 ecr
> > 44014459], length 0
> > 15:56:40.984104 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801819777:1801823649, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 3872
> > 15:56:40.984159 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801823649:1801883521, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 59872
> > 15:56:40.984214 IP 192.168.20.17.40861 > 192.168.20.18.55714: Flags
> > [.], ack 1801819777, win 1009, options [nop,nop,TS val 305403520 ecr
> > 44014459], length 0
> > 15:56:40.984241 IP 192.168.20.18.55714 > 192.168.20.17.40861: Flags
> > [.], seq 1801883521:1801887393, ack 1, win 220, options [nop,nop,TS
> > val 44014459 ecr 305403520], length 3872
> 

Thanks to both of you for that clarification. Nevertheless this
info seems a little contradictionary. Should I exepect GRO to
work on Mellanox IB cards with Linux 3.12 in general? Or is 
this only an effect because you have test cards with "good
mac addresses"?

Markus
=

[-- Attachment #2: InterScan_Disclaimer.txt --]
[-- Type: text/plain, Size: 1650 bytes --]

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

Re: AW: AW: ACK behaviour difference LRO/GRO

[Or Gerlitz]

On 29/10/2013 17:54, Markus Stockhausen wrote:
> Should I exepect GRO to work on Mellanox IB cards with Linux 3.12 in general?

YES
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: AW: AW: ACK behaviour difference LRO/GRO

[Markus Stockhausen]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

> Von: Or Gerlitz [ogerlitz-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org]
> Gesendet: Dienstag, 29. Oktober 2013 16:55
> An: Markus Stockhausen; Erez Shitrit
> Cc: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Yishai Hadas; s.wendy.cheng-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org; Saeed Mahameed
> Betreff: Re: AW: AW: ACK behaviour difference LRO/GRO
> 
> On 29/10/2013 17:54, Markus Stockhausen wrote:
> > Should I exepect GRO to work on Mellanox IB cards with Linux 3.12 in general?
> 
> YES

I'm so sorry to tell you that a jump on the 3.12 rc6 bandwagon did not
gave a better picture. To sum up the most recent situation: 

- NFS server based on a 3.5 kernel 
- Server has an "older" Xeon L5420
- NFS client based on a 3.12 rc6 kernel - NFS version 3/4 does not matter 
- Client has a newer Core i7
- Both ends use datagram mode with 2K MTU 
- Client reads data from NFS (RAM) with dd into null device
- tcpdump is running on the server side.

>From the dump I can see most of the time about 20 incoming ack 
packets for one sent out data packet of size 64k. Statistics from 
/proc/interrupts on the server side give a result that fits the picture. 
~870000 interrupts for 4GB transferred data. That is round about one 
interrupt per 4,5K. Not very scientific but at least an idea of what is 
going on.

With the mac address explanation of Erez in mind I would confirm that
there is still room to improve GRO even in newest kernels.

Markus 


=

[-- Attachment #2: InterScan_Disclaimer.txt --]
[-- Type: text/plain, Size: 1650 bytes --]

****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.

Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

Vorstand:
Kadir Akin
Dr. Michael Höhnerbach

Vorsitzender des Aufsichtsrates:
Hans Kristian Langva

Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.

Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln

executive board:
Kadir Akin
Dr. Michael Höhnerbach

President of the supervisory board:
Hans Kristian Langva

Registry office: district court Cologne
Register number: HRB 52 497

****************************************************************************

Re: AW: AW: ACK behaviour difference LRO/GRO

[Or Gerlitz]

On 29/10/2013 23:17, Markus Stockhausen wrote:
> I would confirm that there is still room to improve GRO even in newest kernels

indeed, so there IS aggregation but the BW isn't as good as it used to 
be with LRO, maybe something
in IPoIB doesn't play well with the core kernel NAPI/GRO/Interrupt 
Coalescing APIs/Layers.

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html