[PATCH RFC 0/3] Support envelope frames (802.3as)

[PATCH RFC 0/3] Support envelope frames (802.3as)

[Toshiaki Makita]

This patch introduces software implementation of envelope frames defined
in 802.3as[1], which allows encapsulated packets to be received without
expanding MTU for them.


* Envelope frames

Envelope frames are introduces by IEEE 802.3as[1], which has been
incorpolated in IEEE 802.3-2012.

IEEE 802.3-2012 1.4.184 defines envelope frame as:
	A MAC frame that carries a Length/Type field with the Type
	interpretation that may indicate additional encapsulation
	information within the MAC client data and has a maximum length
	of 2000 octets. The envelope frame is intended to allow inclusion
	of additional prefixes and suffixes required by higher layer
	encapsulation protocols. The encapsulation protocols may use up
	to 482 octets.


* Motivation

The intended customer of this feature is mainly vlan, possibly mpls or
other encapsulation protocols.

Vlan is different than other encapsulation protocols in that the packet
size is generally larger than normal packets by vlan-header size (4 bytes).
Thus, most NICs allow packets the size of which is larger by 4 bytes
than MTU (802.3 calls this vlan-tagged packets "Q-tagged frames", whose
MTU is 1504 including vlan header. Most NICs accept Q-tagged frames).

Similarly, when doubly tagged vlan is used leveraging 802.1ad, the packet
size will be larger by vlan-header size * 2 (8 bytes). This packet size is
needed to provide Ethernet VPN transparent to the users. Thus, hardware
switches support 1508 bytes MTU when using 802.1ad, as suggested by MEF[2].
Also, Linux stacked vlan devices have 1500 bytes MTU, which emit 1508
bytes doubly tagged packets. But unfortunately some NICs don't accept 
1508 bytes packets by default, and they are dropped.

+----+ single tag +-------+ double tag +-------+ double tag +------+
|End | 1504 bytes |802.1ad| 1508 bytes |802.1ad| 1508 bytes |Linux |
|User|----------->|Edge SW|----------->|NNI SW |----------->|Server|
+----+            +-------+            +-------+     *drop* +------+
                                                     on NIC

802.3 calls such encapsulated packets larger than 1504 "envelope frames".
Most NICs lack support for envelope frames. But many of them support jumbo
frames, which can be used to implement envelope frames support in Linux.
I'm proposing this envelope frames support to fix problems described above.


* Implementation

Envelope frames require normal packets to use 1500-sized MTU, while
encapsulation headers can be added to the MTU. If we simply increase MTU
of the physical device, it causes jumbo frames as well as envelope frames
(jumbo frames are non-encapsulated packets whose MTU is larger than 1500).
So what we need here is to increase the max acceptable frame size of NICs
without changing dev->mtu.

In order to achieve this, I add a new function pointer,
.ndo_set_env_hdr_len, in net_device_ops, through which kernel can inform
device drivers of needed additional header size of envelope frames
(env_hdr_len).
Implementation in device drivers is as simple as replacing dev->mtu with
dev->mtu + env_hdr_len. This makes devices recognize dev->mtu + env_hdr_len
as MTU, and allow packets with additional header up to env_hdr_len, while
kernel networking stack recognizes dev->mtu as MTU. Thus no packets larger
than MTU will be sent other than those encapsulated by upper devices. This
effectively supports envelope frames.

Userspace API is netlink, the same as MTU. It will be a parameter which
can be configured through "ip link".


* Q&A

** Why not reducing MTU of VLAN devices?

As written in Motivation, in order to achieve transparency of Ethernet VPN,
MTU of vlan device needs to be 1500. Since this is usual in 802.1ad network,
switches in 802.1ad network send 1508-sized tagged packets. Thus, reducing
MTU of vlan device does not change the situation where Linux receives
packets whose MTU is larger than NICs' acceptable size, and does not fix
the issue.

** Why not increasing MTU of physical devices?

Increasing MTU of physical device indeed resolves the problem that NICs
cannot receive doubly tagged packets. However, this effectively allows
devices to send jumbo frames as well as envelope frames, and could cause
packet drops on network elements which does not accept jumbo frames.

** Why is .ndo_set_env_hdr_len needed?
   Why not modifying drivers to accept envelope frames by default?

Some NICs actually support envelope frames by default. One example is igb,
which always accepts packet size up to 9728.
I however don't think all NICs necessarily be able to do that since some
NICs change their behaviour when changing MTU larger than 1500.
For example, e1000e changes usage of descriptors when its MTU gets larger
than 1500. qlge also looks to change its behaviour as far as I can see from
the source code of the driver.
In order to keep the default behaviour when not using 802.1ad or stacked
vlan, some knob is needed.

** Why are drivers notified of header _length_?
   Why not introducing a knob to simply enable envelope mode?

The reason being the same as the previous question. There can be a NIC that
changes its behaviour when changing MTU larger than a certain value (I don't
know any such NICs though). If we do not notify drivers of length, the NIC
should add 482 bytes to MTU in envelope mode as defined in 802.3as, and this
may change NIC's behaviour. Vlan needs only 8 additional bytes and it may
not change NIC's behaviour. I just wanted to make it flexible to handle such
situation.
Another problem when we do not notify drivers of length is how to handle the
case where NICs partially support envelope frames. If a certain NIC does not
support 482 bytes additional header but some size of header smaller than 482,
how to handle this? If we make the .ndo_set_env_hdr_len fail in such a case,
users cannot use its additional header, even if the max acceptable size is
sufficient for them. If we make the op succeed, we need to add another API
to expose the accepted size so that users can know how long header they can
use.


* Examples

I show current behaviour of some NICs and expected behaviour when
setting env_hdr_len.

As far as I can confirm with actual equipments in our lab, there are
at least four types of drivers/devices.

(Type 1) Devices with extra buffer larger than vlan header

(Type 1-1) Devices with small amount of extra buffer
Devices/drivers that already take into account stacked vlan, or have
more extra room than it should be due to alignment restriction, etc.
This type of NICs does not require any additional operation to make
stacked vlan work.
E.g. mlx4_en, sfc

(Type 1-2) Devices with large amount of extra buffer
Devices/drivers that always accept packets up to the maximum configurable
size of MTU with jumbo frames support. This type of NICs has enough size
of extra buffer for envelope frames, the header size of which is defined
as 482 in 802.3as. They accept various types of protocols in addition to
stacked vlan.
E.g. igb

(Type 2) Devices without extra buffer larger than vlan header

(Type 2-1) Devices with generic 4 bytes extra buffer
Devices/drivers that accept MTU + 4 sized packets.
Any packets not larger than MTU + 4 are acceptable but those larger than
the value are dropped.
E.g. e1000e

(Type 2-2) Devices with 4 bytes extra buffer only for vlan
Devices/drivers that accept MTU + 4 sized packets only if it is vlan
tagged. Other packets are dropped if they exceed MTU, even if their size
is less than MTU + 4.
This type of devices even drops 802.1ad single tagged packets, if they
do not support 802.1ad vlan protocol.
E.g. bnx2x, ixgbe


The problematic NICs with stacked vlan are only type 2, but we should
assume all types of NICs can be configured with env_hdr_len, because
users do not know which NIC is which type.

The expected behaviour when users set 8 bytes env_hdr_len for stacked vlan
is as follows:
(Type 1) As they have more room than 8, do nothing.
(Type 2-1) They have 4 bytes env_hdr_len by default. When users specifies 8,
  drivers increase devices' MTU by 4 bytes.
(Type 2-2) They have 0 bytes env_hdr_len by default. When users specifies 8,
  drivers increase devices' MTU by 8 bytes.


* Recommended operations for users

- Set env_hdr_len of the physical device to 8 when they create a 802.1ad
  vlan device or stacked vlan devices.
- If setting of env_hdr_len fails (due to EOPNOTSUPP), try increasing MTU
  of the physical device.
- If increasing MTU fails, there is no other option than reducing MTU of
  vlan devices, unfortunately.

Note that those operations are not needed for 802.1q single vlan devices
at all, even after adding envelope frame support. Adjustment of MTU in case
of single 802.1q vlan should be cared by drivers as before.


* Future Plan

In the future 802.1ad vlan devices should take care of the MTU problem in
kernel and adjust env_hdr_len automatically. This can be achieved by
notifying drivers of env_hdr_len on creating 802.1ad devices.
This notification, however, does not always succeed because of lack of
.ndo support or jumbo frame feature, so this cannot completely remove
the necessity of userspace operation of env_hdr_len. Nevertheless I'm
thinking this would help users to some extent.


* Note

I submitted the previous patch set with title of "Automatic adjustment of
max frame size"[3]. Now I'm not targetting at automation but infrastracture
for the feature, as I think automation is kind of premature when even
manual operation is not possible.

This problem was discussed in Netdev 0.1:
http://www.netdevconf.org/0.1/docs/netdev01_bof_8021ad_makita_150212.pdf
This topic is also going to be discussed in Netdev 1.2:
http://www.netdevconf.org/1.2/session.html?toshiaki-makita


[1] http://www.ieee802.org/3/as/public/0607/802.3as_overview.pdf
[2] https://wiki.mef.net/display/CESG/ENNI+Attributes
[3] https://marc.info/?t=144583097100001&r=1&w=2
    https://marc.info/?t=144583097100005&r=1&w=2
    https://marc.info/?t=144583097100006&r=1&w=2
    https://marc.info/?t=144583097100004&r=1&w=2
    https://marc.info/?t=144583097100002&r=1&w=2

Toshiaki Makita (3):
  net: Add dev_set_env_hdr_len to accept envelope frames
  net: Support IFLA_ENV_HDR_LEN to configure max envelope header length
  e1000e: Add ndo_set_env_hdr_len

 drivers/net/ethernet/intel/e1000e/netdev.c | 84 +++++++++++++++++++++---------
 include/linux/netdevice.h                  | 21 ++++++++
 include/uapi/linux/if_link.h               |  1 +
 net/core/dev.c                             | 32 ++++++++++++
 net/core/rtnetlink.c                       | 16 +++++-
 5 files changed, 128 insertions(+), 26 deletions(-)

-- 
1.8.3.1

[PATCH RFC 1/3] net: Add dev_set_env_hdr_len to accept envelope frames

[Toshiaki Makita]

Currently most NICs support Q-tagged frames[1], i.e. 1522 bytes frames
to handle 4 bytes VLAN header. But some encapsulation protocols like
802.1ad requires them to handle larger frames.
This change introduces dev_set_env_hdr_len() and corresponding drivers'
operation .ndo_set_env_hdr_len(), which notifies drivers of needed
encapsulation header length. This enables devices to accept longer
frames with encapsulation headers, i.e. envelope frames[2], without
expanding MTU size for non-encapsulated frames.

Note 1:
Envelope frames are not jumbo frames. See IEEE 802.3as[3] for detail.
IEEE 802.3-2012 3.2.7 says:
          The envelope frame is intended to allow inclusion of additional
	  prefixes and suffixes required by higher layer encapusulation
	  protocols such as those defined by the IEEE 802.1 working
	  group (such as Provider Bridges and MAC Security), ITU-T or
	  IETF (such as MPLS). The original MAC Client Data field
	  maximum remains 1500 octets while the encapsulation protocols
	  may add up to an additional 482 octets. Use of these extra
	  octets for other purposes is not recommended, and may result
	  in MAC frames being dropped or corrupted as they may violate
	  maximum MAC frame size restrictions if encapsulation protocols
	  are required to operate on them.

Note 2:
Envelope frames in IEEE 802.3 defines the max size of envelope frames
as 2000 bytes. This change is more flexible than 802.3 in terms of max
allowed frame length.

[1] IEEE 802.3-2012, 1.4.334.
[2] IEEE 802.3-2012, 1.4.184.
[3] http://www.ieee802.org/3/as/public/0607/802.3as_overview.pdf

Signed-off-by: Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>
---
 include/linux/netdevice.h | 21 +++++++++++++++++++++
 net/core/dev.c            | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 53 insertions(+)

diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index 136ae6bb..a0ac76a 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1132,6 +1132,10 @@ struct netdev_xdp {
  * int (*ndo_xdp)(struct net_device *dev, struct netdev_xdp *xdp);
  *	This function is used to set or query state related to XDP on the
  *	netdevice. See definition of enum xdp_netdev_command for details.
+ * int (*ndo_set_env_hdr_len)(struct net_device *dev, int hdr_len);
+ *	This function is used to set the maximum header size of envelope
+ *	frames. The device must accept the size of MTU + envelope header
+ *	size on packet reception.
  *
  */
 struct net_device_ops {
@@ -1323,6 +1327,8 @@ struct net_device_ops {
 						       int needed_headroom);
 	int			(*ndo_xdp)(struct net_device *dev,
 					   struct netdev_xdp *xdp);
+	int			(*ndo_set_env_hdr_len)(struct net_device *dev,
+						       int hdr_len);
 };
 
 /**
@@ -1506,6 +1512,7 @@ enum netdev_priv_flags {
  *	@if_port:	Selectable AUI, TP, ...
  *	@dma:		DMA channel
  *	@mtu:		Interface MTU value
+ *	@env_hdr_len:	Additional encapsulation header length to MTU
  *	@type:		Interface hardware type
  *	@hard_header_len: Maximum hardware header length.
  *
@@ -1726,6 +1733,7 @@ struct net_device {
 	unsigned char		dma;
 
 	unsigned int		mtu;
+	unsigned int		env_hdr_len;
 	unsigned short		type;
 	unsigned short		hard_header_len;
 
@@ -3300,6 +3308,7 @@ int dev_change_name(struct net_device *, const char *);
 int dev_set_alias(struct net_device *, const char *, size_t);
 int dev_change_net_namespace(struct net_device *, struct net *, const char *);
 int dev_set_mtu(struct net_device *, int);
+int dev_set_env_hdr_len(struct net_device *, int);
 void dev_set_group(struct net_device *, int);
 int dev_set_mac_address(struct net_device *, struct sockaddr *);
 int dev_change_carrier(struct net_device *, bool new_carrier);
@@ -4233,6 +4242,18 @@ static inline bool netif_reduces_vlan_mtu(struct net_device *dev)
 	return dev->priv_flags & IFF_MACSEC;
 }
 
+/* return envelope header length */
+static inline int netif_get_env_hdr_len(struct net_device *dev)
+{
+	if (dev->netdev_ops->ndo_set_env_hdr_len)
+		return dev->env_hdr_len;
+
+	if (netif_reduces_vlan_mtu(dev))
+		return 0;
+
+	return 4; /* VLAN_HLEN */
+}
+
 extern struct pernet_operations __net_initdata loopback_net_ops;
 
 /* Logging, debugging and troubleshooting/diagnostic helpers. */
diff --git a/net/core/dev.c b/net/core/dev.c
index c0c291f..df75aaa 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -6524,6 +6524,38 @@ int dev_set_mtu(struct net_device *dev, int new_mtu)
 EXPORT_SYMBOL(dev_set_mtu);
 
 /**
+ *	dev_set_env_hdr_len - Set max envelope header length
+ *	@dev: device
+ *	@new_len: new length
+ */
+int dev_set_env_hdr_len(struct net_device *dev, int new_len)
+{
+	const struct net_device_ops *ops = dev->netdev_ops;
+	int err;
+
+	if (!ops->ndo_set_env_hdr_len)
+		return -EOPNOTSUPP;
+
+	if (new_len < 0)
+		return -EINVAL;
+
+	if (!netif_device_present(dev))
+		return -ENODEV;
+
+	if (new_len == dev->env_hdr_len)
+		return 0;
+
+	err = ops->ndo_set_env_hdr_len(dev, new_len);
+	if (err)
+		return err;
+
+	dev->env_hdr_len = new_len;
+
+	return 0;
+}
+EXPORT_SYMBOL(dev_set_env_hdr_len);
+
+/**
  *	dev_set_group - Change group this device belongs to
  *	@dev: device
  *	@new_group: group this device should belong to
-- 
1.8.3.1

[PATCH RFC 2/3] net: Support IFLA_ENV_HDR_LEN to configure max envelope header length

[Toshiaki Makita]

With this change, admin can configure env_hdr_len.

Signed-off-by: Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>
---
 include/uapi/linux/if_link.h |  1 +
 net/core/rtnetlink.c         | 16 ++++++++++++++--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/include/uapi/linux/if_link.h b/include/uapi/linux/if_link.h
index b4fba66..9545ea4 100644
--- a/include/uapi/linux/if_link.h
+++ b/include/uapi/linux/if_link.h
@@ -157,6 +157,7 @@ enum {
 	IFLA_GSO_MAX_SIZE,
 	IFLA_PAD,
 	IFLA_XDP,
+	IFLA_ENV_HDR_LEN,
 	__IFLA_MAX
 };
 
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index 3ac8946..9233709 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -943,7 +943,8 @@ static noinline size_t if_nlmsg_size(const struct net_device *dev,
 	       + nla_total_size(MAX_PHYS_ITEM_ID_LEN) /* IFLA_PHYS_SWITCH_ID */
 	       + nla_total_size(IFNAMSIZ) /* IFLA_PHYS_PORT_NAME */
 	       + rtnl_xdp_size(dev) /* IFLA_XDP */
-	       + nla_total_size(1); /* IFLA_PROTO_DOWN */
+	       + nla_total_size(1) /* IFLA_PROTO_DOWN */
+	       + nla_total_size(4); /* IFLA_ENV_HDR_LEN */
 
 }
 
@@ -1321,7 +1322,8 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, struct net_device *dev,
 	     nla_put_string(skb, IFLA_IFALIAS, dev->ifalias)) ||
 	    nla_put_u32(skb, IFLA_CARRIER_CHANGES,
 			atomic_read(&dev->carrier_changes)) ||
-	    nla_put_u8(skb, IFLA_PROTO_DOWN, dev->proto_down))
+	    nla_put_u8(skb, IFLA_PROTO_DOWN, dev->proto_down) ||
+	    nla_put_u32(skb, IFLA_ENV_HDR_LEN, netif_get_env_hdr_len(dev)))
 		goto nla_put_failure;
 
 	if (rtnl_fill_link_ifmap(skb, dev))
@@ -1458,6 +1460,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = {
 	[IFLA_LINK_NETNSID]	= { .type = NLA_S32 },
 	[IFLA_PROTO_DOWN]	= { .type = NLA_U8 },
 	[IFLA_XDP]		= { .type = NLA_NESTED },
+	[IFLA_ENV_HDR_LEN]	= { .type = NLA_U32 },
 };
 
 static const struct nla_policy ifla_info_policy[IFLA_INFO_MAX+1] = {
@@ -2174,6 +2177,13 @@ static int do_setlink(const struct sk_buff *skb,
 		}
 	}
 
+	if (tb[IFLA_ENV_HDR_LEN]) {
+		err = dev_set_env_hdr_len(dev, nla_get_u32(tb[IFLA_ENV_HDR_LEN]));
+		if (err < 0)
+			goto errout;
+		status |= DO_SETLINK_MODIFIED;
+	}
+
 errout:
 	if (status & DO_SETLINK_MODIFIED) {
 		if (status & DO_SETLINK_NOTIFY)
@@ -2378,6 +2388,8 @@ struct net_device *rtnl_create_link(struct net *net,
 		dev->link_mode = nla_get_u8(tb[IFLA_LINKMODE]);
 	if (tb[IFLA_GROUP])
 		dev_set_group(dev, nla_get_u32(tb[IFLA_GROUP]));
+	if (tb[IFLA_ENV_HDR_LEN])
+		dev->env_hdr_len = nla_get_u32(tb[IFLA_ENV_HDR_LEN]);
 
 	return dev;
 
-- 
1.8.3.1

[PATCH RFC 3/3] e1000e: Add ndo_set_env_hdr_len

[Toshiaki Makita]

e1000e supports generic 1522-sized frames by default, so set the default
env_hdr_len to 4, and replace dev->mtu with dev->mtu + env_hdr_len.
Note that e1000e has adapter->max_frame_size that includes mtu +
env_hdr_len, so I use it where mtu was used to validate frame length.

Signed-off-by: Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>
---
 drivers/net/ethernet/intel/e1000e/netdev.c | 84 +++++++++++++++++++++---------
 1 file changed, 60 insertions(+), 24 deletions(-)

diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index 7017281..4dc9315 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -3033,7 +3033,7 @@ static void e1000_setup_rctl(struct e1000_adapter *adapter)
 	if (hw->mac.type >= e1000_pch2lan) {
 		s32 ret_val;
 
-		if (adapter->netdev->mtu > ETH_DATA_LEN)
+		if (adapter->max_frame_size > VLAN_ETH_FRAME_LEN + ETH_FCS_LEN)
 			ret_val = e1000_lv_jumbo_workaround_ich8lan(hw, true);
 		else
 			ret_val = e1000_lv_jumbo_workaround_ich8lan(hw, false);
@@ -3053,7 +3053,7 @@ static void e1000_setup_rctl(struct e1000_adapter *adapter)
 	rctl &= ~E1000_RCTL_SBP;
 
 	/* Enable Long Packet receive */
-	if (adapter->netdev->mtu <= ETH_DATA_LEN)
+	if (adapter->max_frame_size <= VLAN_ETH_FRAME_LEN + ETH_FCS_LEN)
 		rctl &= ~E1000_RCTL_LPE;
 	else
 		rctl |= E1000_RCTL_LPE;
@@ -3121,7 +3121,8 @@ static void e1000_setup_rctl(struct e1000_adapter *adapter)
 	 * a lot of memory, since we allocate 3 pages at all times
 	 * per packet.
 	 */
-	pages = PAGE_USE_COUNT(adapter->netdev->mtu);
+	pages = PAGE_USE_COUNT(adapter->netdev->mtu +
+			       adapter->netdev->env_hdr_len);
 	if ((pages <= 3) && (PAGE_SIZE <= 16384) && (rctl & E1000_RCTL_LPE))
 		adapter->rx_ps_pages = pages;
 	else
@@ -3191,7 +3192,8 @@ static void e1000_configure_rx(struct e1000_adapter *adapter)
 		    sizeof(union e1000_rx_desc_packet_split);
 		adapter->clean_rx = e1000_clean_rx_irq_ps;
 		adapter->alloc_rx_buf = e1000_alloc_rx_buffers_ps;
-	} else if (adapter->netdev->mtu > ETH_FRAME_LEN + ETH_FCS_LEN) {
+	} else if (adapter->netdev->mtu + adapter->netdev->env_hdr_len >
+		   ETH_FRAME_LEN + ETH_FCS_LEN) {
 		rdlen = rx_ring->count * sizeof(union e1000_rx_desc_extended);
 		adapter->clean_rx = e1000_clean_jumbo_rx_irq;
 		adapter->alloc_rx_buf = e1000_alloc_jumbo_rx_buffers;
@@ -3273,7 +3275,7 @@ static void e1000_configure_rx(struct e1000_adapter *adapter)
 	/* With jumbo frames, excessive C-state transition latencies result
 	 * in dropped transactions.
 	 */
-	if (adapter->netdev->mtu > ETH_DATA_LEN) {
+	if (adapter->max_frame_size > VLAN_ETH_FRAME_LEN + ETH_FCS_LEN) {
 		u32 lat =
 		    ((er32(PBA) & E1000_PBA_RXA_MASK) * 1024 -
 		     adapter->max_frame_size) * 8 / 1000;
@@ -4001,7 +4003,8 @@ void e1000e_reset(struct e1000_adapter *adapter)
 	switch (hw->mac.type) {
 	case e1000_ich9lan:
 	case e1000_ich10lan:
-		if (adapter->netdev->mtu > ETH_DATA_LEN) {
+		if (adapter->max_frame_size > VLAN_ETH_FRAME_LEN +
+					      ETH_FCS_LEN) {
 			pba = 14;
 			ew32(PBA, pba);
 			fc->high_water = 0x2800;
@@ -4020,7 +4023,8 @@ void e1000e_reset(struct e1000_adapter *adapter)
 		/* Workaround PCH LOM adapter hangs with certain network
 		 * loads.  If hangs persist, try disabling Tx flow control.
 		 */
-		if (adapter->netdev->mtu > ETH_DATA_LEN) {
+		if (adapter->max_frame_size > VLAN_ETH_FRAME_LEN +
+					      ETH_FCS_LEN) {
 			fc->high_water = 0x3500;
 			fc->low_water = 0x1500;
 		} else {
@@ -4034,7 +4038,8 @@ void e1000e_reset(struct e1000_adapter *adapter)
 	case e1000_pch_spt:
 		fc->refresh_time = 0x0400;
 
-		if (adapter->netdev->mtu <= ETH_DATA_LEN) {
+		if (adapter->max_frame_size <= VLAN_ETH_FRAME_LEN +
+					       ETH_FCS_LEN) {
 			fc->high_water = 0x05C20;
 			fc->low_water = 0x05048;
 			fc->pause_time = 0x0650;
@@ -4278,7 +4283,7 @@ void e1000e_down(struct e1000_adapter *adapter, bool reset)
 
 	/* Disable Si errata workaround on PCHx for jumbo frame flow */
 	if ((hw->mac.type >= e1000_pch2lan) &&
-	    (adapter->netdev->mtu > ETH_DATA_LEN) &&
+	    (adapter->max_frame_size > VLAN_ETH_FRAME_LEN + ETH_FCS_LEN) &&
 	    e1000_lv_jumbo_workaround_ich8lan(hw, false))
 		e_dbg("failed to disable jumbo frame workaround mode\n");
 
@@ -4391,7 +4396,8 @@ static int e1000_sw_init(struct e1000_adapter *adapter)
 
 	adapter->rx_buffer_len = VLAN_ETH_FRAME_LEN + ETH_FCS_LEN;
 	adapter->rx_ps_bsize0 = 128;
-	adapter->max_frame_size = netdev->mtu + VLAN_ETH_HLEN + ETH_FCS_LEN;
+	adapter->max_frame_size = netdev->mtu + netdev->env_hdr_len +
+				  ETH_HLEN + ETH_FCS_LEN;
 	adapter->min_frame_size = ETH_ZLEN + ETH_FCS_LEN;
 	adapter->tx_ring_count = E1000_DEFAULT_TXD;
 	adapter->rx_ring_count = E1000_DEFAULT_RXD;
@@ -5961,17 +5967,10 @@ struct rtnl_link_stats64 *e1000e_get_stats64(struct net_device *netdev,
 	return stats;
 }
 
-/**
- * e1000_change_mtu - Change the Maximum Transfer Unit
- * @netdev: network interface device structure
- * @new_mtu: new value for maximum frame size
- *
- * Returns 0 on success, negative on failure
- **/
-static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
+static int e1000_change_max_frame(struct net_device *netdev, int max_frame,
+				  int new_mtu)
 {
 	struct e1000_adapter *adapter = netdev_priv(netdev);
-	int max_frame = new_mtu + VLAN_ETH_HLEN + ETH_FCS_LEN;
 
 	/* Jumbo frame support */
 	if ((max_frame > (VLAN_ETH_FRAME_LEN + ETH_FCS_LEN)) &&
@@ -5981,7 +5980,7 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
 	}
 
 	/* Supported frame sizes */
-	if ((new_mtu < (VLAN_ETH_ZLEN + ETH_FCS_LEN)) ||
+	if ((new_mtu && new_mtu < (VLAN_ETH_ZLEN + ETH_FCS_LEN)) ||
 	    (max_frame > adapter->max_hw_frame_size)) {
 		e_err("Unsupported MTU setting\n");
 		return -EINVAL;
@@ -5990,7 +5989,7 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
 	/* Jumbo frame workaround on 82579 and newer requires CRC be stripped */
 	if ((adapter->hw.mac.type >= e1000_pch2lan) &&
 	    !(adapter->flags2 & FLAG2_CRC_STRIPPING) &&
-	    (new_mtu > ETH_DATA_LEN)) {
+	    (max_frame > VLAN_ETH_FRAME_LEN + ETH_FCS_LEN)) {
 		e_err("Jumbo Frames not supported on this device when CRC stripping is disabled.\n");
 		return -EINVAL;
 	}
@@ -5998,9 +5997,14 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
 	while (test_and_set_bit(__E1000_RESETTING, &adapter->state))
 		usleep_range(1000, 2000);
 	/* e1000e_down -> e1000e_reset dependent on max_frame_size & mtu */
+	if (new_mtu) {
+		e_info("changing MTU from %d to %d\n", netdev->mtu, new_mtu);
+		netdev->mtu = new_mtu;
+	} else {
+		e_info("changing max frame size from %d to %d\n",
+		       adapter->max_frame_size, max_frame);
+	}
 	adapter->max_frame_size = max_frame;
-	e_info("changing MTU from %d to %d\n", netdev->mtu, new_mtu);
-	netdev->mtu = new_mtu;
 
 	pm_runtime_get_sync(netdev->dev.parent);
 
@@ -6036,6 +6040,20 @@ static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
 	return 0;
 }
 
+/**
+ * e1000_change_mtu - Change the Maximum Transfer Unit
+ * @netdev: network interface device structure
+ * @new_mtu: new value for maximum frame size
+ *
+ * Returns 0 on success, negative on failure
+ **/
+static int e1000_change_mtu(struct net_device *netdev, int new_mtu)
+{
+	int max_frame = new_mtu + netdev->env_hdr_len + ETH_HLEN + ETH_FCS_LEN;
+
+	return e1000_change_max_frame(netdev, max_frame, new_mtu);
+}
+
 static int e1000_mii_ioctl(struct net_device *netdev, struct ifreq *ifr,
 			   int cmd)
 {
@@ -6925,7 +6943,8 @@ static netdev_features_t e1000_fix_features(struct net_device *netdev,
 	struct e1000_hw *hw = &adapter->hw;
 
 	/* Jumbo frame workaround on 82579 and newer requires CRC be stripped */
-	if ((hw->mac.type >= e1000_pch2lan) && (netdev->mtu > ETH_DATA_LEN))
+	if ((hw->mac.type >= e1000_pch2lan) &&
+	    (adapter->max_frame_size > VLAN_ETH_FRAME_LEN + ETH_FCS_LEN))
 		features &= ~NETIF_F_RXFCS;
 
 	/* Since there is no support for separate Rx/Tx vlan accel
@@ -6977,6 +6996,20 @@ static int e1000_set_features(struct net_device *netdev,
 	return 0;
 }
 
+/**
+ * e1000_set_env_hdr_len - Set max envelope header length
+ * @netdev: network interface device structure
+ * @new_len: new value for maximum envelope header length
+ *
+ * Returns 0 on success, negative on failure
+ **/
+static int e1000_set_env_hdr_len(struct net_device *netdev, int new_len)
+{
+	int max_frame = netdev->mtu + new_len + ETH_HLEN + ETH_FCS_LEN;
+
+	return e1000_change_max_frame(netdev, max_frame, 0);
+}
+
 static const struct net_device_ops e1000e_netdev_ops = {
 	.ndo_open		= e1000e_open,
 	.ndo_stop		= e1000e_close,
@@ -6997,6 +7030,7 @@ static const struct net_device_ops e1000e_netdev_ops = {
 	.ndo_set_features = e1000_set_features,
 	.ndo_fix_features = e1000_fix_features,
 	.ndo_features_check	= passthru_features_check,
+	.ndo_set_env_hdr_len	= e1000_set_env_hdr_len,
 };
 
 /**
@@ -7119,6 +7153,8 @@ static int e1000_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 	netdev->mem_start = mmio_start;
 	netdev->mem_end = mmio_start + mmio_len;
 
+	netdev->env_hdr_len = VLAN_HLEN;
+
 	adapter->bd_number = cards_found++;
 
 	e1000e_check_options(adapter);
-- 
1.8.3.1

[PATCH RFC iproute2] iplink: Support envhdrlen

[Toshiaki Makita]

This adds support for envhdrlen.

Example:
 # ip link set eno1 envhdrlen 8
 # ip link show eno1
 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 envhdrlen 8 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
     link/ether 44:37:e6:6c:69:a4 brd ff:ff:ff:ff:ff:ff

Note:
As an RFC, this includes update for kernel headers.

Signed-off-by: Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>
---
 include/linux/if_link.h |  1 +
 ip/ipaddress.c          |  2 ++
 ip/iplink.c             | 10 ++++++++++
 3 files changed, 13 insertions(+)

diff --git a/include/linux/if_link.h b/include/linux/if_link.h
index b9299e3..46ef8cc 100644
--- a/include/linux/if_link.h
+++ b/include/linux/if_link.h
@@ -157,6 +157,7 @@ enum {
 	IFLA_GSO_MAX_SIZE,
 	IFLA_PAD,
 	IFLA_XDP,
+	IFLA_ENV_HDR_LEN,
 	__IFLA_MAX
 };
 
diff --git a/ip/ipaddress.c b/ip/ipaddress.c
index 76bd7b3..92a472d 100644
--- a/ip/ipaddress.c
+++ b/ip/ipaddress.c
@@ -820,6 +820,8 @@ int print_linkinfo(const struct sockaddr_nl *who,
 
 	if (tb[IFLA_MTU])
 		fprintf(fp, "mtu %u ", *(int *)RTA_DATA(tb[IFLA_MTU]));
+	if (tb[IFLA_ENV_HDR_LEN])
+		fprintf(fp, "envhdrlen %u ", *(int *)RTA_DATA(tb[IFLA_ENV_HDR_LEN]));
 	if (tb[IFLA_QDISC])
 		fprintf(fp, "qdisc %s ", rta_getattr_str(tb[IFLA_QDISC]));
 	if (tb[IFLA_MASTER]) {
diff --git a/ip/iplink.c b/ip/iplink.c
index 6b1db18..4dcb9ac 100644
--- a/ip/iplink.c
+++ b/ip/iplink.c
@@ -50,6 +50,7 @@ void iplink_usage(void)
 		fprintf(stderr, "                   [ address LLADDR ]\n");
 		fprintf(stderr, "                   [ broadcast LLADDR ]\n");
 		fprintf(stderr, "                   [ mtu MTU ] [index IDX ]\n");
+		fprintf(stderr, "                   [ envhdrlen ENVHDRLEN ]\n");
 		fprintf(stderr, "                   [ numtxqueues QUEUE_COUNT ]\n");
 		fprintf(stderr, "                   [ numrxqueues QUEUE_COUNT ]\n");
 		fprintf(stderr, "                   type TYPE [ ARGS ]\n");
@@ -489,6 +490,7 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
 	char abuf[32];
 	int qlen = -1;
 	int mtu = -1;
+	int envhdrlen = -1;
 	int netns = -1;
 	int vf = -1;
 	int numtxqueues = -1;
@@ -547,6 +549,14 @@ int iplink_parse(int argc, char **argv, struct iplink_req *req,
 			if (get_integer(&mtu, *argv, 0))
 				invarg("Invalid \"mtu\" value\n", *argv);
 			addattr_l(&req->n, sizeof(*req), IFLA_MTU, &mtu, 4);
+		} else if (strcmp(*argv, "envhdrlen") == 0) {
+			NEXT_ARG();
+			if (envhdrlen != -1)
+				duparg("envhdrlen", *argv);
+			if (get_integer(&envhdrlen, *argv, 0))
+				invarg("Invalid \"envhdrlen\" value\n", *argv);
+			addattr_l(&req->n, sizeof(*req), IFLA_ENV_HDR_LEN,
+				  &envhdrlen, 4);
 		} else if (strcmp(*argv, "netns") == 0) {
 			NEXT_ARG();
 			if (netns != -1)
-- 
2.5.5

RE: [PATCH RFC 1/3] net: Add dev_set_env_hdr_len to accept envelope frames

[Mintz, Yuval]

> +/* return envelope header length */
> +static inline int netif_get_env_hdr_len(struct net_device *dev) {
> +	if (dev->netdev_ops->ndo_set_env_hdr_len)
> +		return dev->env_hdr_len;
> +
> +	if (netif_reduces_vlan_mtu(dev))
> +		return 0;
> +
> +	return 4; /* VLAN_HLEN */
> +}

Why claim that `4' is the default?
What's the benefit of propagating this value to userspace if
driver doesn't actually support it?

Assuming this *is* a good default [meaning your analysis indicated
basically every driver has at least so many octets reserved]
perhaps we should consider initializing the env_hdr_len in some
common function [e.g., in setup_ether() next to the mtu] to this value?

Re: [PATCH RFC 1/3] net: Add dev_set_env_hdr_len to accept envelope frames

[Toshiaki Makita]

On 2016/09/27 23:23, Mintz, Yuval wrote:
>> +/* return envelope header length */
>> +static inline int netif_get_env_hdr_len(struct net_device *dev) {
>> +	if (dev->netdev_ops->ndo_set_env_hdr_len)
>> +		return dev->env_hdr_len;
>> +
>> +	if (netif_reduces_vlan_mtu(dev))
>> +		return 0;
>> +
>> +	return 4; /* VLAN_HLEN */
>> +}
> 

Thank you for taking a look at this.

> Why claim that `4' is the default?
> What's the benefit of propagating this value to userspace if
> driver doesn't actually support it?

I'm sorry, this is a leftover from previous draft when I was not aware
of type 2-2 drivers (that do not allow over-MTU-sized packets other than
vlan-tagged ones). Now I'm thinking we should not include
IFLA_ENV_HDR_LEN in response of GETLINK if it is not supported.
I'll delete current logic from next version.

> 
> Assuming this *is* a good default [meaning your analysis indicated
> basically every driver has at least so many octets reserved]
> perhaps we should consider initializing the env_hdr_len in some
> common function [e.g., in setup_ether() next to the mtu] to this value?

I actually tried it first and realized it is difficult, since so many
drivers use their own initializer instead of ether_setup(). Anyway, I'm
going not to expose envhdrlen when it is not supported.

Thanks,
Toshiaki Makita

Re: [PATCH RFC iproute2] iplink: Support envhdrlen

[Jiri Benc]

On Tue, 27 Sep 2016 17:55:40 +0900, Toshiaki Makita wrote:
> This adds support for envhdrlen.
> 
> Example:
>  # ip link set eno1 envhdrlen 8

I don't see why this should be user visible, let alone requiring user
to set it. This should be transparent, kernel should compute the value
as needed based on the configuration and set it up. Requiring the
administrator to pick up a calculator and sum up all the vlan, mpls and
whatever header lengths is silly.

I realize that we currently have no easy way to do that. Especially
with lwtunnels and stuff line MPLS where we don't easily know the
number of tags. But every uAPI we introduce will have to be supported
forever and going a particular way just because it is easy to implement
is not sustainable.

At the very least, it should be configurable from the other direction.
I.e. telling which interfaces can be used by vlans or MPLS (if it
cannot be inferred automatically) and configuring maximum number of
tags on the given vlan/mpls/whatever interface/route/whatever.

 Jiri

Re: [PATCH RFC iproute2] iplink: Support envhdrlen

[Toshiaki Makita]

On 16/09/29 (木) 21:49, Jiri Benc wrote:
> On Tue, 27 Sep 2016 17:55:40 +0900, Toshiaki Makita wrote:
>> This adds support for envhdrlen.
>>
>> Example:
>>  # ip link set eno1 envhdrlen 8

Thank you for taking a look at this.

> I don't see why this should be user visible, let alone requiring user
> to set it. This should be transparent, kernel should compute the value
> as needed based on the configuration and set it up. Requiring the
> administrator to pick up a calculator and sum up all the vlan, mpls and
> whatever header lengths is silly.

I'm thinking both in-kernel automation (for VLAN) and users' manual 
operation are necessary. In fact, as I stated in the cover letter, my 
first implementation was an automation-based approach. Actually 
automation is advanced form of this feature so I'm proposing this very 
basic feature first.

> I realize that we currently have no easy way to do that. Especially
> with lwtunnels and stuff line MPLS where we don't easily know the
> number of tags. But every uAPI we introduce will have to be supported
> forever and going a particular way just because it is easy to implement
> is not sustainable.

It may be possible for MPLS lwtunnel to notify underlying device of 
needed header length. But at this point MPLS does not allow encapsulated 
packets to be greater than MTU of underlying device. So how to determine 
if someone wants to leverage envhdrlen instead? Add a knob to lwtunnel 
layer? Then, add another knob to l2tp, iptunnel, or anything like that? 
At least full-automation does not look possible other than VLAN (which 
by default requires envhdrlen expansion), so anyway manual operation is 
needed in some form.
Another use-case is reducing envhdrlen. We can expand it on creating 
VLAN device automatically, but cannot decrease the size because there 
could be other consumers of envhdrlen.

> At the very least, it should be configurable from the other direction.
> I.e. telling which interfaces can be used by vlans or MPLS (if it
> cannot be inferred automatically) and configuring maximum number of
> tags on the given vlan/mpls/whatever interface/route/whatever.

Probably I don't get your point...
Are you suggesting something like this?
$ ip link set eth0.10.20 expand-realdev-envhdrlen
or like this?
$ ip link set eth0 allowed-vlan-tags 2

Toshiaki Makita