tx checksum offload in rtl8168evl disabled in driver

tx checksum offload in rtl8168evl disabled in driver

[jason.morgan]

Hi,

I'm try to get close to saturating a 1G ethernet.

I'm at 517Mbps and I've found that there seems to be a cpu bottleneck.

I'm using 2k to 4k frames with a rtl8168evl.

I notice from ethtool that tx-checksum is turned off and refuse to turn 
on.

I've found this message
http://www.spinics.net/lists/netdev/msg216530.html

Which indicates the cause being the driver.

I've looked at the driver code rtl8169.c in kernel 3.8 and the line 

        [RTL_GIGA_MAC_VER_34] =
                _R("RTL8168evl/8111evl",RTL_TD_1, FIRMWARE_8168E_3,
                                                        JUMBO_9K, false),

indicates the reason for this.

However the message thread, above indicates that this is not a problem and 

can be changed to make tx-checksum offload possible.

However we are using a newer chip to the on in the message thread.  I've 
tried to find other, more recent citations without success.

So, why is it still turned off?

What will be the effect of turning it on (changing false to true, in the 
driver line) for our chip?

Thanks in advance,
Jason

Re: tx checksum offload in rtl8168evl disabled in driver

[Francois Romieu]

jason.morgan@aveillant.com <jason.morgan@aveillant.com> :
[...]
> I'm at 517Mbps and I've found that there seems to be a cpu bottleneck.

Which kernel ?

> I'm using 2k to 4k frames with a rtl8168evl.
> I've found this message
> http://www.spinics.net/lists/netdev/msg216530.html
[...]
> However the message thread, above indicates that this is not a problem and 
> can be changed to make tx-checksum offload possible.
> 
> However we are using a newer chip to the on in the message thread.  I've 
> tried to find other, more recent citations without success.
> 
> So, why is it still turned off ?

It has been disabled since d58d46b5d85139d18eb939aa7279c160bab70484 ("r8169:
jumbo fixes"). Patch was submitted as a RFC on 2011/07/17 and Hayes was
explicitely requested to comment on the jumbo part if necessary. Patch was
submitted for inclusion on 2011/09/22.

Tx checksumming and jumbo are mutually exclusive in Realtek's driver as well.

It seems no recent gigabit chipset reliably supports it.

> What will be the effect of turning it on (changing false to true, in the 
> driver line) for our chip ?

YMMV.

Hayes may elaborate.

-- 
Ueimor

RE: tx checksum offload in rtl8168evl disabled in driver

[hayeswang]

Francois Romieu [mailto:romieu@fr.zoreil.com] 
[...]
> > I'm using 2k to 4k frames with a rtl8168evl.
> > I've found this message
> > http://www.spinics.net/lists/netdev/msg216530.html
> [...]
> > However the message thread, above indicates that this is 
> not a problem and 
> > can be changed to make tx-checksum offload possible.
> > 
> > However we are using a newer chip to the on in the message 
> thread.  I've 
> > tried to find other, more recent citations without success.
> > 
> > So, why is it still turned off ?
> 
> It has been disabled since 
> d58d46b5d85139d18eb939aa7279c160bab70484 ("r8169:
> jumbo fixes"). Patch was submitted as a RFC on 2011/07/17 and 
> Hayes was
> explicitely requested to comment on the jumbo part if 
> necessary. Patch was
> submitted for inclusion on 2011/09/22.
> 
> Tx checksumming and jumbo are mutually exclusive in Realtek's 
> driver as well.
> 
> It seems no recent gigabit chipset reliably supports it.
> 
> > What will be the effect of turning it on (changing false to 
> true, in the 
> > driver line) for our chip ?

Since RTL8111E, the chips support a feature which we call "tx early".
For jumbo frame, the hw starts to send a packet before getting the whole data.
However, the checksum has to be calculated after the whole data are fetched.
Therefore, the jumbo frame and the tx checksum couldn't be enable at the same
time, otherwise a packet with incorrect checksum would be sent.
 
Best Regards,
Hayes

Re: tx checksum offload in rtl8168evl disabled in driver

[jason.morgan]

Ubuntu 12.04.3 LTS + Kernel 3.8.13-8 64bit

I've patched the driver to allow tx checksum offload for this chip and=20
found the following:

MTU 9000 standard driver:
517Mbps with 2k + header frames

MTU 9000 patched driver:
770Mbps with 2k + header frames

100% transfer without error (1e6 frames)

48% increase in performance combined with a massive decrease in CPU =
effort=20
is not to be sniffed at.

IMO tx offload should be more prevalent as the frames grow, to reduce 
CPU=20
load.

OK, so make the default OFF if there is a silicon error (that spans=20
mulitple chips?) , but why prevent it being turned on in the driver? -=20
even if there is a kernel message that this might cause problems.




From:   Francois Romieu <romieu@fr.zoreil.com>
To:     <jason.morgan@aveillant.com>, 
Cc:     netdev@vger.kernel.org, hayeswang@realtek.com
Date:   04/10/2013 00:01
Subject:        Re: tx checksum offload in rtl8168evl disabled in driver



jason.morgan@aveillant.com <jason.morgan@aveillant.com> :
[...]
> I'm at 517Mbps and I've found that there seems to be a cpu bottleneck.

Which kernel ?

> I'm using 2k to 4k frames with a rtl8168evl.
> I've found this message
> http://www.spinics.net/lists/netdev/msg216530.html
[...]
> However the message thread, above indicates that this is not a problem 
and 
> can be changed to make tx-checksum offload possible.
> 
> However we are using a newer chip to the on in the message thread. I've 
> tried to find other, more recent citations without success.
> 
> So, why is it still turned off ?

It has been disabled since d58d46b5d85139d18eb939aa7279c160bab70484 
("r8169:
jumbo fixes"). Patch was submitted as a RFC on 2011/07/17 and Hayes was
explicitely requested to comment on the jumbo part if necessary. Patch was
submitted for inclusion on 2011/09/22.

Tx checksumming and jumbo are mutually exclusive in Realtek's driver as 
well.

It seems no recent gigabit chipset reliably supports it.

> What will be the effect of turning it on (changing false to true, in the 

> driver line) for our chip ?

YMMV.

Hayes may elaborate.

-- 
Ueimor

RE: tx checksum offload in rtl8168evl disabled in driver

[jason.morgan]

It seems TX Early is a miss-feature, surely this is what needs to be 
turned off by default and not the tx-offload.
Yes it reduces latency, but at the expense of a lower overall throughput.



From:   hayeswang <hayeswang@realtek.com>
To:     'Francois Romieu' <romieu@fr.zoreil.com>, 
<jason.morgan@aveillant.com>, 
Cc:     <netdev@vger.kernel.org>, 'nic_swsd' <nic_swsd@realtek.com>
Date:   04/10/2013 04:10
Subject:        RE: tx checksum offload in rtl8168evl disabled in driver



Francois Romieu [mailto:romieu@fr.zoreil.com] 
[...]
> > I'm using 2k to 4k frames with a rtl8168evl.
> > I've found this message
> > http://www.spinics.net/lists/netdev/msg216530.html
> [...]
> > However the message thread, above indicates that this is 
> not a problem and 
> > can be changed to make tx-checksum offload possible.
> > 
> > However we are using a newer chip to the on in the message 
> thread. I've 
> > tried to find other, more recent citations without success.
> > 
> > So, why is it still turned off ?
> 
> It has been disabled since 
> d58d46b5d85139d18eb939aa7279c160bab70484 ("r8169:
> jumbo fixes"). Patch was submitted as a RFC on 2011/07/17 and 
> Hayes was
> explicitely requested to comment on the jumbo part if 
> necessary. Patch was
> submitted for inclusion on 2011/09/22.
> 
> Tx checksumming and jumbo are mutually exclusive in Realtek's 
> driver as well.
> 
> It seems no recent gigabit chipset reliably supports it.
> 
> > What will be the effect of turning it on (changing false to 
> true, in the 
> > driver line) for our chip ?

Since RTL8111E, the chips support a feature which we call "tx early".
For jumbo frame, the hw starts to send a packet before getting the whole 
data.
However, the checksum has to be calculated after the whole data are 
fetched.
Therefore, the jumbo frame and the tx checksum couldn't be enable at the 
same
time, otherwise a packet with incorrect checksum would be sent.

Best Regards,
Hayes

Re: tx checksum offload in rtl8168evl disabled in driver

[Francois Romieu]

(please don't top post)

jason.morgan@aveillant.com <jason.morgan@aveillant.com> :
> Ubuntu 12.04.3 LTS + Kernel 3.8.13-8 64bit
> 
> I've patched the driver to allow tx checksum offload for this chip and
> found the following:
>
> MTU 9000 standard driver:
> 517Mbps with 2k + header frames
> 
> MTU 9000 patched driver:
> 770Mbps with 2k + header frames
> 
> 100% transfer without error (1e6 frames)

(Ok, so that's 20 ~ 30s worth of traffic)

> 48% increase in performance combined with a massive decrease in CPU
> effort is not to be sniffed at.

*sniff* :o)

It depends on the CPU. You did not specify it and you did not give numbers
for the decrease (did you use 'perf' btw ?). They would be welcome.

> IMO tx offload should be more prevalent as the frames grow, to reduce 
> CPU load.

I can't disagree.

> OK, so make the default OFF if there is a silicon error (that spans
> mulitple chips?),

Yes, I want safe defaults for the kernel.

I give the manufacturer's explanations a lot of credit when they're
related to hardware (up to the point where the marketing or legal dept
kicks in). If we want to balance these with experimental evidences, the
latter must be really, really strong.

> but why prevent it being turned on in the driver? 
> even if there is a kernel message that this might cause problems.

Two points:
- it's a hack: ethtool will return success. A kernel message is not a
  substitute for "Yes, I opt in for problems".
- we can't tell when it's safe and when it isn't.

-- 
Ueimor

Re: tx checksum offload in rtl8168evl disabled in driver

[jason.morgan]

> 
> (please don't top post)

Sorry, I was not aware of the correct netiquette
replies now inline.

> 
> jason.morgan@....> :
> > Ubuntu 12.04.3 LTS + Kernel 3.8.13-8 64bit
> > 
> > I've patched the driver to allow tx checksum offload for this chip and
> > found the following:
> >
> > MTU 9000 standard driver:
> > 517Mbps with 2k + header frames
> > 
> > MTU 9000 patched driver:
> > 770Mbps with 2k + header frames
> > 
> > 100% transfer without error (1e6 frames)
> 
> (Ok, so that's 20 ~ 30s worth of traffic)

Right now this is representative as the network has no other traffic and 
the CPU has no other function.
I will be producing hours of traffic soon in a more realistic network, see 
later....

> 
> > 48% increase in performance combined with a massive decrease in CPU
> > effort is not to be sniffed at.
> 
> *sniff* :o)

Drums.... BurrDumpf!

> 
> It depends on the CPU. You did not specify it and you did not give 
numbers
> for the decrease (did you use 'perf' btw ?). They would be welcome.

As I've patched the kernel driver and build a new kernel, I have no idea 
how to build a perf for my installed kernel.
It appears that there is not one pre-built.
Any help here?
When I learn how to build perf I can reply with CPU performance data.

> 
> > IMO tx offload should be more prevalent as the frames grow, to reduce 
> > CPU load.
> 
> I can't disagree.
> 
> > OK, so make the default OFF if there is a silicon error (that spans
> > mulitple chips?),
> 
> Yes, I want safe defaults for the kernel.

Safe defaults are of course a good thing.

Others have stated that this is not a silicon error, it's a hardware 
feature which it appears
the Linux default for which is non-optimal and there is no means (bar 
hacking) to optimise it.

> 
> I give the manufacturer's explanations a lot of credit when they're
> related to hardware (up to the point where the marketing or legal dept
> kicks in). If we want to balance these with experimental evidences, the
> latter must be really, really strong.

To me it appears the latency stats ( a marketing metric of the MAC ) 
somehow outweigh the load on the CPU
( which is factored out of MAC stats ) so perhaps your comment on the 
marketing dept rings true?

Are there any tests that you can suggest that would provide such evidence?

I am in the process of building a LAN of 18 machines with the intent of
saturating a 1G/10G Ethernet switch, so I have a good test platform - for 
one chip anyway.

> 
> > but why prevent it being turned on in the driver? 
> > even if there is a kernel message that this might cause problems.
> 
> Two points:
> - it's a hack: ethtool will return success. A kernel message is not a
> substitute for "Yes, I opt in for problems".
> - we can't tell when it's safe and when it isn't.

I would agree that generally ethtool can't be trusted, but it is clear 
that
it does work with this driver for this chip for this control so in that 
case surely it can be trusted?

The statement "we can't tell when it's safe and when it isn't." is true 
for almost any hardware interface
Usually we offer safe defaults and a means to trade off safety and 
performance, though that is not always the case, e.g.
It's much safer to run DDR3-1600 as DDR3-1066, but if that were the 
default people would complain.

> 
> -- 
> Ueimor
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html