Question regarding protocol specific mtu for FCoE

Question regarding protocol specific mtu for FCoE

[Zou, Yi]

Hi,
I am trying to figure out a way to support a different MTU for Fiber
Channel over Ethernet (FCoE), where it is preferred to have a baby jumbo
frame with MTU=2158 (14 bytes FCoE header + 24 bytes FC header + 2112
bytes max FC payload + 4 bytes FC CRC + 4 bytes FCoE trailer).

Similar to nedev->mtu being used everywhere for LAN traffic, I wonder if
it is desirable or feasible to add another protocol specific MTU (which
is my case is FCoE) to netdev so any eth driver supporting converged
traffic (e.g. LAN + FCoE) over netdev can make use of it.

Thanks.

Yi Zou

Re: Question regarding protocol specific mtu for FCoE

[Rick Jones]

Zou, Yi wrote:
> Hi,
> I am trying to figure out a way to support a different MTU for Fiber
> Channel over Ethernet (FCoE), where it is preferred to have a baby jumbo
> frame with MTU=2158 (14 bytes FCoE header + 24 bytes FC header + 2112
> bytes max FC payload + 4 bytes FC CRC + 4 bytes FCoE trailer).
> 
> Similar to nedev->mtu being used everywhere for LAN traffic, I wonder if
> it is desirable or feasible to add another protocol specific MTU (which
> is my case is FCoE) to netdev so any eth driver supporting converged
> traffic (e.g. LAN + FCoE) over netdev can make use of it.

Do FCoE upper layers have anything analagous to a TCP_MAXSEG option?  That allows 
an application using TCP to ask for a smaller MSS than TCP might have chosen 
otherwise.

Would a NIC over which FCoE was running be able to be of two minds of what the 
MTU happens to be?  I'd think that if one user of the NIC needed/wanted and MTU > 
foo one would just set the MTU large enough to include foo and be done with it?

In IP networking space at least, if there is a destination for which one does not 
want to use the default MTU, one can set-up a destination-specific (Path) MTU in 
the routing table that will (if smaller) override the link-local MTU for traffic 
going to/from that destination.

rick jones

RE: Question regarding protocol specific mtu for FCoE

[Zou, Yi]

>Zou, Yi wrote:
>> Hi,
>> I am trying to figure out a way to support a different MTU for Fiber
>> Channel over Ethernet (FCoE), where it is preferred to have a baby
>jumbo
>> frame with MTU=2158 (14 bytes FCoE header + 24 bytes FC header + 2112
>> bytes max FC payload + 4 bytes FC CRC + 4 bytes FCoE trailer).
>>
>> Similar to nedev->mtu being used everywhere for LAN traffic, I wonder
>if
>> it is desirable or feasible to add another protocol specific MTU
>(which
>> is my case is FCoE) to netdev so any eth driver supporting converged
>> traffic (e.g. LAN + FCoE) over netdev can make use of it.
>
>Do FCoE upper layers have anything analagous to a TCP_MAXSEG option?
>That allows
>an application using TCP to ask for a smaller MSS than TCP might have
>chosen
>otherwise.
No, FCoE does not have that. The current kernel FCoE initiator initiates
its max frame size based on the associated netdev->mtu, which normally is 1500. So, unless the LAN mtu is changed, FCoE stack is no able to
use baby jumbo frame.

>
>Would a NIC over which FCoE was running be able to be of two minds of
>what the
>MTU happens to be?  I'd think that if one user of the NIC needed/wanted
>and MTU >
>foo one would just set the MTU large enough to include foo and be done
>with it?
For a nic that supports converged traffic, i.e. LAN + FCoE, I think
it's safe to assume the nic has that capability. Setting LAN MTU large
enough will work for FCoE, but it affects everyone using netdev->mtu, so
you may see degraded LAN performance, which is certainly not good.

>
>In IP networking space at least, if there is a destination for which
>one does not
>want to use the default MTU, one can set-up a destination-specific
>(Path) MTU in
>the routing table that will (if smaller) override the link-local MTU
>for traffic
>going to/from that destination.
>
>rick jones
FCoE is on L2 layer, no path specific MTU, everything goes out as
whatever mtu known to the nic. Since the nic is expected to be used for
converged traffic involving multiple traffic types, e.g. LAN, FCoE, I
was wondering if it makes sense to have the additional MTU. Essentially,
the nic driver will be able to setup via netdev for different MTUs for
converged traffic.

Thanks. 

yi

RE: Question regarding protocol specific mtu for FCoE

[Ben Hutchings]

On Wed, 2009-06-03 at 12:16 -0700, Zou, Yi wrote:
[...]
> FCoE is on L2 layer, no path specific MTU, everything goes out as
> whatever mtu known to the nic. Since the nic is expected to be used for
> converged traffic involving multiple traffic types, e.g. LAN, FCoE, I
> was wondering if it makes sense to have the additional MTU. Essentially,
> the nic driver will be able to setup via netdev for different MTUs for
> converged traffic.

Wouldn't you use separate VLANs for FCoE and other traffic?  So maybe we
should allow for per-VLAN rather than per-protocol MTU.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

RE: Question regarding protocol specific mtu for FCoE

[Zou, Yi]

>On Wed, 2009-06-03 at 12:16 -0700, Zou, Yi wrote:
>[...]
>> FCoE is on L2 layer, no path specific MTU, everything goes out as
>> whatever mtu known to the nic. Since the nic is expected to be used
>for
>> converged traffic involving multiple traffic types, e.g. LAN, FCoE, I
>> was wondering if it makes sense to have the additional MTU.
>Essentially,
>> the nic driver will be able to setup via netdev for different MTUs
>for
>> converged traffic.
>
>Wouldn't you use separate VLANs for FCoE and other traffic?  So maybe
>we
>should allow for per-VLAN rather than per-protocol MTU.
>
>Ben.
>
>From what I can tell, you will not be able to set vlan device's MTU
that is larger than the real_dev->mtu, as in vlan_dev_change_mtu().

Thanks.

yi

RE: Question regarding protocol specific mtu for FCoE

[Ben Hutchings]

On Wed, 2009-06-03 at 13:43 -0700, Zou, Yi wrote:
> >On Wed, 2009-06-03 at 12:16 -0700, Zou, Yi wrote:
> >[...]
> >> FCoE is on L2 layer, no path specific MTU, everything goes out as
> >> whatever mtu known to the nic. Since the nic is expected to be used
> >for
> >> converged traffic involving multiple traffic types, e.g. LAN, FCoE, I
> >> was wondering if it makes sense to have the additional MTU.
> >Essentially,
> >> the nic driver will be able to setup via netdev for different MTUs
> >for
> >> converged traffic.
> >
> >Wouldn't you use separate VLANs for FCoE and other traffic?  So maybe
> >we
> >should allow for per-VLAN rather than per-protocol MTU.
> >
> >Ben.
> >
> From what I can tell, you will not be able to set vlan device's MTU
> that is larger than the real_dev->mtu, as in vlan_dev_change_mtu().

Yes, I know this restriction exists at the moment.  But the device MTU
(the limit for dependent virtual devices) could perhaps be decoupled
from the interface MTU (the limit for protocols) and automatically
raised when necessary.  I think something like that is necessary for
your proposal too.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Question regarding protocol specific mtu for FCoE

[Rick Jones]

>> Do FCoE upper layers have anything analagous to a TCP_MAXSEG option? That
>> allows an application using TCP to ask for a smaller MSS than TCP might have
>>  chosen otherwise.
> 
> No, FCoE does not have that. The current kernel FCoE initiator initiates its
> max frame size based on the associated netdev->mtu, which normally is 1500.
> So, unless the LAN mtu is changed, FCoE stack is no able to use baby jumbo
> frame.

So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and send-down 
frames based on that?

>> Would a NIC over which FCoE was running be able to be of two minds of what 
>> the MTU happens to be?  I'd think that if one user of the NIC needed/wanted
>>  and MTU > foo one would just set the MTU large enough to include foo and be
>>  done with it?
> 
> For a nic that supports converged traffic, i.e. LAN + FCoE, I think
> it's safe to assume the nic has that capability. Setting LAN MTU large
> enough will work for FCoE, but it affects everyone using netdev->mtu, so
> you may see degraded LAN performance, which is certainly not good.

There have been some instances of increased MTU running causing applcations to 
run afoul of Nagle and what not, but I don't think it is a general truism that 
using a larger MTU degrades LAN performance.

Perhaps it is just me not getting out enough but I'm still having trouble coming 
to grips with the idea of a NIC that has two (or more) ideas of a valid frame 
size based on ethertype or whatnot.

At the risk of another of my "Emily Litella" moments, it sounds like the NIC HW 
will have to be set to use an MTU that is the max of the desired IP and FCoE MTUs 
and so you are not really talking about adding one MTU concept but two so you have:

1) the "HW MTU" used by the NIC and the guts of the driver
2) the legacy networking (eg IP etc) MTU passed up through the legacy path
3) the fcoe MTU passed up through the fcoe path

where 1 is the max of 2,3

rick jones

Re: Question regarding protocol specific mtu for FCoE

[Waskiewicz Jr, Peter P]

> 1) the "HW MTU" used by the NIC and the guts of the driver
> 2) the legacy networking (eg IP etc) MTU passed up through the legacy path
> 3) the fcoe MTU passed up through the fcoe path
> 
> where 1 is the max of 2,3

Whatever is decided to do is going to be targeted at ixgbe first.  The 
scenario you outlined here is exactly how ixgbe hardware works:

- Set HLREG0 to enable jumbo frames.  This allows larger frames to be 
accepted, but doesn't limit them to any minimum size.

- Set MHADD (for 82598) or MAXFRS (for 82599 - same register, new name) to 
indicate what the absolute maximum frame size you will receive.  Again, 
doesn't limit the minimum size.

- The networking software can send any size up to MAXFRS (maximum frame 
size) and the hardware will happily receive it and pass it up the stack.

Cheers,
-PJ Waskiewicz

Re: Question regarding protocol specific mtu for FCoE

[Roland Dreier]

 > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and
 > send-down frames based on that?

I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a bit
for headers) even when netdev->mtu is 1500.  (eg datacenter network
supports baby jumbo frames so FCoE traffic that stays within the network
should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out onto a
1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
pain)

 - R.

Re: Question regarding protocol specific mtu for FCoE

[Roland Dreier]

 > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and
 > send-down frames based on that?

I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a bit
for headers) even when netdev->mtu is 1500.  (eg datacenter network
supports baby jumbo frames so FCoE traffic that stays within the network
should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out onto a
1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
pain)

 - R.

RE: Question regarding protocol specific mtu for FCoE

[Zou, Yi]

>On Wed, 2009-06-03 at 13:43 -0700, Zou, Yi wrote:
>> >On Wed, 2009-06-03 at 12:16 -0700, Zou, Yi wrote:
>> >[...]
>> >> FCoE is on L2 layer, no path specific MTU, everything goes out as
>> >> whatever mtu known to the nic. Since the nic is expected to be
>used
>> >for
>> >> converged traffic involving multiple traffic types, e.g. LAN, FCoE,
>I
>> >> was wondering if it makes sense to have the additional MTU.
>> >Essentially,
>> >> the nic driver will be able to setup via netdev for different MTUs
>> >for
>> >> converged traffic.
>> >
>> >Wouldn't you use separate VLANs for FCoE and other traffic?  So
>maybe
>> >we
>> >should allow for per-VLAN rather than per-protocol MTU.
>> >
>> >Ben.
>> >
>> From what I can tell, you will not be able to set vlan device's MTU
>> that is larger than the real_dev->mtu, as in vlan_dev_change_mtu().
>
>Yes, I know this restriction exists at the moment.  But the device MTU
>(the limit for dependent virtual devices) could perhaps be decoupled
>from the interface MTU (the limit for protocols) and automatically
>raised when necessary.  I think something like that is necessary for
>your proposal too.
>
>Ben.
Well, for all all NIC devices that are capable of doing converged
traffic, the additional MTU seems to me fits just what you mentioned as
the automatic MTU increase, since a different MTU is checked. Of course,
the vlan above needs to be also fixed to allow this as well. Also, FCoE
should not required to be created only on a VLAN interface.

Thanks.

yi

RE: Question regarding protocol specific mtu for FCoE

[Zou, Yi]

> > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and
> > send-down frames based on that?
>
>I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a
>bit
>for headers) even when netdev->mtu is 1500.  (eg datacenter network
>supports baby jumbo frames so FCoE traffic that stays within the
>network
>should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out onto
>a
>1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
>pain)
>
> - R.

Yes, exactly, thanks for the clarification. Also, since FC max payload
of 2112 is not going to change, I guess the OPTIMAL_FCOEMTU is probably
going to stay unchanged forever.

Thanks.

yi

Re: Question regarding protocol specific mtu for FCoE

[Rick Jones]

Roland Dreier wrote:
>  > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and
>  > send-down frames based on that?
> 
> I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a bit
> for headers) even when netdev->mtu is 1500.  (eg datacenter network
> supports baby jumbo frames so FCoE traffic that stays within the network
> should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out onto a
> 1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
> pain)

Aren't all stations in the same broadcast domain "supposed" to have the same MTU, 
at least down at L2? So, a station in the broadcast domain just doing IP and a 
station in the broadcast domain doing IP+FCoE "should" have the same MTU at the 
HW level right?

I could see where there would be lots of PMTU going-on if the communications were 
to off-campus sites also had an FCoE upping their MTU.  Otherwise, the MSS 
exchange at connection establishment is going to preclude it right?  PMTU only 
"hits" when one has a so called "dumb-bell" network which is "wider" at the ends 
than in the middle.

Re: Question regarding protocol specific mtu for FCoE

[Rick Jones]

Roland Dreier wrote:
>  > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)" and
>  > send-down frames based on that?
> 
> I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a bit
> for headers) even when netdev->mtu is 1500.  (eg datacenter network
> supports baby jumbo frames so FCoE traffic that stays within the network
> should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out onto a
> 1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
> pain)

Aren't all stations in the same broadcast domain "supposed" to have the same MTU, 
at least down at L2? So, a station in the broadcast domain just doing IP and a 
station in the broadcast domain doing IP+FCoE "should" have the same MTU at the 
HW level right?

I could see where there would be lots of PMTU going-on if the communications were 
to off-campus sites also had an FCoE upping their MTU.  Otherwise, the MSS 
exchange at connection establishment is going to preclude it right?  PMTU only 
"hits" when one has a so called "dumb-bell" network which is "wider" at the ends 
than in the middle.

RE: Question regarding protocol specific mtu for FCoE

[Zou, Yi]

>Roland Dreier wrote:
>>  > So FCoE cannot say "fcoe_mtu = min(OPTIMAL_FCOEMTU,netdev->mtu)"
>and
>>  > send-down frames based on that?
>>
>> I think the point is that FCoE wants to use OPTIMAL_FCOEMTU (2KB + a
>bit
>> for headers) even when netdev->mtu is 1500.  (eg datacenter network
>> supports baby jumbo frames so FCoE traffic that stays within the
>network
>> should use OPTIMAL_FCOEMTU, while lots of IP traffic is going out
>onto a
>> 1500-byte MTU campus and having TCP doing lots of PMTU discovery is a
>> pain)
>
>Aren't all stations in the same broadcast domain "supposed" to have the
>same MTU,
>at least down at L2? So, a station in the broadcast domain just doing
>IP and a
>station in the broadcast domain doing IP+FCoE "should" have the same
>MTU at the
>HW level right?
It's still the same MTU to LAN, so two stations communicating by IP
still communicates on the LAN MTU (netdev->mtu) and FCoE using 2158
should be transparent to LAN. Of course, the corresponding NIC driver
needs to have the knowledge to set up correspondingly so HW can do
different MTUs. Ideally, I think this knowledge can be retrieved from
netdev as an additional MTU, say, netdev->fcoe_mtu, or something else,
thus LAN side will not be affected.

Thanks.

yi

Re: Question regarding protocol specific mtu for FCoE

[Roland Dreier]

 > Aren't all stations in the same broadcast domain "supposed" to have
 > the same MTU, at least down at L2? So, a station in the broadcast
 > domain just doing IP and a station in the broadcast domain doing
 > IP+FCoE "should" have the same MTU at the HW level right?
 > 
 > I could see where there would be lots of PMTU going-on if the
 > communications were to off-campus sites also had an FCoE upping their
 > MTU.  Otherwise, the MSS exchange at connection establishment is going
 > to preclude it right?  PMTU only "hits" when one has a so called
 > "dumb-bell" network which is "wider" at the ends than in the middle.

Yes, I think such dumb-bell networks would be pretty common (servers in
two different data centers with FCoE enabled, talking through an
old-school WAN with 1500 MTU).  And even if they're not as common I
think, we probably do want to have some way to handle this.

I'm probably not as up on old and/or obscure networking protocols, but
to me FCoE is the first time I've been forced to think about coexistence
of IP and non-IP protocols through the same netdev.

 - R.

Re: Question regarding protocol specific mtu for FCoE

[Roland Dreier]

 > Aren't all stations in the same broadcast domain "supposed" to have
 > the same MTU, at least down at L2? So, a station in the broadcast
 > domain just doing IP and a station in the broadcast domain doing
 > IP+FCoE "should" have the same MTU at the HW level right?
 > 
 > I could see where there would be lots of PMTU going-on if the
 > communications were to off-campus sites also had an FCoE upping their
 > MTU.  Otherwise, the MSS exchange at connection establishment is going
 > to preclude it right?  PMTU only "hits" when one has a so called
 > "dumb-bell" network which is "wider" at the ends than in the middle.

Yes, I think such dumb-bell networks would be pretty common (servers in
two different data centers with FCoE enabled, talking through an
old-school WAN with 1500 MTU).  And even if they're not as common I
think, we probably do want to have some way to handle this.

I'm probably not as up on old and/or obscure networking protocols, but
to me FCoE is the first time I've been forced to think about coexistence
of IP and non-IP protocols through the same netdev.

 - R.