[RFC PATCH 0/5] Add GSO Partial support for devices with existing tunnel offloads

[RFC PATCH 0/5] Add GSO Partial support for devices with existing tunnel offloads

[Alexander Duyck]

This patch set is meant to extend tunnel offload support for devices that
already support UDP or GRE tunnels to include support for those type of
tunnels with an outer checksum.  To do this I am enabling GSO partial
support for these types of tunnels which should work assuming the devices
supported by these drivers are either overwriting or ignoring checksum and
length fields contained in the IPv4/v6 and ignoring the checksum contained
in the outer L4 tunnel headers.

I don't have any of the hardware that uses the drivers that these patches
are applied do.  That was one of the motivations for submitting them as RFC.
My hope is that somebody out there with this hardware is willing to pull in
the patch and test it out in order to determine if this approach actually
works for these devices.

---

Alexander Duyck (5):
      mlx4: Add support for UDP tunnel segmentation with outer checksum offload
      mlx5: Add support for UDP tunnel segmentation with outer checksum offload
      bnx2x: Add support for segmentation of tunnels with outer checksums
      bnxt: Add support for segmentation of tunnels with outer checksums
      fm10k: Add support for UDP tunnel segmentation with outer checksum offload


 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c  |   13 ++++++++++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c         |    9 +++++++--
 drivers/net/ethernet/intel/fm10k/fm10k_netdev.c   |   14 ++++++++++----
 drivers/net/ethernet/mellanox/mlx4/en_netdev.c    |   17 +++++++++++++----
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
 5 files changed, 46 insertions(+), 14 deletions(-)

[Intel-wired-lan] [RFC PATCH 0/5] Add GSO Partial support for devices with existing tunnel offloads

[Alexander Duyck]

This patch set is meant to extend tunnel offload support for devices that
already support UDP or GRE tunnels to include support for those type of
tunnels with an outer checksum.  To do this I am enabling GSO partial
support for these types of tunnels which should work assuming the devices
supported by these drivers are either overwriting or ignoring checksum and
length fields contained in the IPv4/v6 and ignoring the checksum contained
in the outer L4 tunnel headers.

I don't have any of the hardware that uses the drivers that these patches
are applied do.  That was one of the motivations for submitting them as RFC.
My hope is that somebody out there with this hardware is willing to pull in
the patch and test it out in order to determine if this approach actually
works for these devices.

---

Alexander Duyck (5):
      mlx4: Add support for UDP tunnel segmentation with outer checksum offload
      mlx5: Add support for UDP tunnel segmentation with outer checksum offload
      bnx2x: Add support for segmentation of tunnels with outer checksums
      bnxt: Add support for segmentation of tunnels with outer checksums
      fm10k: Add support for UDP tunnel segmentation with outer checksum offload


 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c  |   13 ++++++++++---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c         |    9 +++++++--
 drivers/net/ethernet/intel/fm10k/fm10k_netdev.c   |   14 ++++++++++----
 drivers/net/ethernet/mellanox/mlx4/en_netdev.c    |   17 +++++++++++++----
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
 5 files changed, 46 insertions(+), 14 deletions(-)

--

[RFC PATCH 1/5] mlx4: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the mlx4 hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx4 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/mellanox/mlx4/en_netdev.c |   17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index b4b258c8ca47..cb6a8a522857 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -2356,7 +2356,9 @@ out:
 
 	/* set offloads */
 	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL;
+				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL |
+				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				      NETIF_F_GSO_PARTIAL;
 }
 
 static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
@@ -2366,7 +2368,9 @@ static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
 						 vxlan_del_task);
 	/* unset offloads */
 	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL);
+					NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL |
+					NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					NETIF_F_GSO_PARTIAL);
 
 	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
 				  VXLAN_STEER_BY_OUTER_MAC, 0);
@@ -2990,8 +2994,13 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 	}
 
 	if (mdev->dev->caps.tunnel_offload_mode == MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
-		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
-		dev->features    |= NETIF_F_GSO_UDP_TUNNEL;
+		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	mdev->pndev[port] = dev;

[Intel-wired-lan] [RFC PATCH 1/5] mlx4: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the mlx4 hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx4 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/mellanox/mlx4/en_netdev.c |   17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
index b4b258c8ca47..cb6a8a522857 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_netdev.c
@@ -2356,7 +2356,9 @@ out:
 
 	/* set offloads */
 	priv->dev->hw_enc_features |= NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL;
+				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL |
+				      NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				      NETIF_F_GSO_PARTIAL;
 }
 
 static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
@@ -2366,7 +2368,9 @@ static void mlx4_en_del_vxlan_offloads(struct work_struct *work)
 						 vxlan_del_task);
 	/* unset offloads */
 	priv->dev->hw_enc_features &= ~(NETIF_F_IP_CSUM | NETIF_F_RXCSUM |
-				      NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL);
+					NETIF_F_TSO | NETIF_F_GSO_UDP_TUNNEL |
+					NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					NETIF_F_GSO_PARTIAL);
 
 	ret = mlx4_SET_PORT_VXLAN(priv->mdev->dev, priv->port,
 				  VXLAN_STEER_BY_OUTER_MAC, 0);
@@ -2990,8 +2994,13 @@ int mlx4_en_init_netdev(struct mlx4_en_dev *mdev, int port,
 	}
 
 	if (mdev->dev->caps.tunnel_offload_mode == MLX4_TUNNEL_OFFLOAD_MODE_VXLAN) {
-		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
-		dev->features    |= NETIF_F_GSO_UDP_TUNNEL;
+		dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->features    |= NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	mdev->pndev[port] = dev;

[RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx5 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index e0adb604f461..57d8da796d50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
 	netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
 
 	if (mlx5e_vxlan_allowed(mdev)) {
-		netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
+		netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
+					   NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					   NETIF_F_GSO_PARTIAL;
 		netdev->hw_enc_features |= NETIF_F_IP_CSUM;
 		netdev->hw_enc_features |= NETIF_F_RXCSUM;
 		netdev->hw_enc_features |= NETIF_F_TSO;
 		netdev->hw_enc_features |= NETIF_F_TSO6;
 		netdev->hw_enc_features |= NETIF_F_RXHASH;
 		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
+		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					   NETIF_F_GSO_PARTIAL;
+		netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	netdev->features          = netdev->hw_features;

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx5 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index e0adb604f461..57d8da796d50 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
 	netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
 
 	if (mlx5e_vxlan_allowed(mdev)) {
-		netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
+		netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
+					   NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					   NETIF_F_GSO_PARTIAL;
 		netdev->hw_enc_features |= NETIF_F_IP_CSUM;
 		netdev->hw_enc_features |= NETIF_F_RXCSUM;
 		netdev->hw_enc_features |= NETIF_F_TSO;
 		netdev->hw_enc_features |= NETIF_F_TSO6;
 		netdev->hw_enc_features |= NETIF_F_RXHASH;
 		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
+		netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
+					   NETIF_F_GSO_PARTIAL;
+		netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	netdev->features          = netdev->hw_features;

[RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

This patch assumes that the bnx2x hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP and GRE headers.

I have no means of testing this as I do not have any bnx2x hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
index d465bd721146..e3459524fb0e 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
@@ -13258,14 +13258,21 @@ static int bnx2x_init_dev(struct bnx2x *bp, struct pci_dev *pdev,
 		NETIF_F_RXCSUM | NETIF_F_LRO | NETIF_F_GRO |
 		NETIF_F_RXHASH | NETIF_F_HW_VLAN_CTAG_TX;
 	if (!chip_is_e1x) {
-		dev->hw_features |= NETIF_F_GSO_GRE | NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT;
+		dev->hw_features |= NETIF_F_GSO_GRE | NETIF_F_GSO_GRE_CSUM |
+				    NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
+				    NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
 		dev->hw_enc_features =
 			NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_SG |
 			NETIF_F_TSO | NETIF_F_TSO_ECN | NETIF_F_TSO6 |
 			NETIF_F_GSO_IPIP |
 			NETIF_F_GSO_SIT |
-			NETIF_F_GSO_GRE | NETIF_F_GSO_UDP_TUNNEL;
+			NETIF_F_GSO_GRE | NETIF_F_GSO_GRE_CSUM |
+			NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_UDP_TUNNEL_CSUM |
+			NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_GRE_CSUM |
+					    NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	dev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |

[Intel-wired-lan] [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

This patch assumes that the bnx2x hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP and GRE headers.

I have no means of testing this as I do not have any bnx2x hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
index d465bd721146..e3459524fb0e 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
@@ -13258,14 +13258,21 @@ static int bnx2x_init_dev(struct bnx2x *bp, struct pci_dev *pdev,
 		NETIF_F_RXCSUM | NETIF_F_LRO | NETIF_F_GRO |
 		NETIF_F_RXHASH | NETIF_F_HW_VLAN_CTAG_TX;
 	if (!chip_is_e1x) {
-		dev->hw_features |= NETIF_F_GSO_GRE | NETIF_F_GSO_UDP_TUNNEL |
-				    NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT;
+		dev->hw_features |= NETIF_F_GSO_GRE | NETIF_F_GSO_GRE_CSUM |
+				    NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
+				    NETIF_F_GSO_UDP_TUNNEL |
+				    NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_PARTIAL;
 		dev->hw_enc_features =
 			NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_SG |
 			NETIF_F_TSO | NETIF_F_TSO_ECN | NETIF_F_TSO6 |
 			NETIF_F_GSO_IPIP |
 			NETIF_F_GSO_SIT |
-			NETIF_F_GSO_GRE | NETIF_F_GSO_UDP_TUNNEL;
+			NETIF_F_GSO_GRE | NETIF_F_GSO_GRE_CSUM |
+			NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_UDP_TUNNEL_CSUM |
+			NETIF_F_GSO_PARTIAL;
+		dev->gso_partial_features = NETIF_F_GSO_GRE_CSUM |
+					    NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	dev->vlan_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |

[RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

This patch assumes that the bnxt hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP and GRE headers.

I have no means of testing this as I do not have any bnx2x hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 4645c44e7c15..ae668476fff0 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -6194,14 +6194,19 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 			   NETIF_F_TSO | NETIF_F_TSO6 |
 			   NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_GRE |
 			   NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
-			   NETIF_F_RXHASH |
+			   NETIF_F_GSO_UDP_TUNNEL_CSUM | NETIF_F_GSO_GRE_CSUM |
+			   NETIF_F_GSO_PARTIAL | NETIF_F_RXHASH |
 			   NETIF_F_RXCSUM | NETIF_F_LRO | NETIF_F_GRO;
 
 	dev->hw_enc_features =
 			NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_SG |
 			NETIF_F_TSO | NETIF_F_TSO6 |
 			NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_GRE |
-			NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT;
+			NETIF_F_GSO_UDP_TUNNEL_CSUM | NETIF_F_GSO_GRE_CSUM |
+			NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
+			NETIF_F_GSO_PARTIAL;
+	dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_GRE_CSUM;
 	dev->vlan_features = dev->hw_features | NETIF_F_HIGHDMA;
 	dev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HW_VLAN_CTAG_TX |
 			    NETIF_F_HW_VLAN_STAG_RX | NETIF_F_HW_VLAN_STAG_TX;

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

This patch assumes that the bnxt hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP and GRE headers.

I have no means of testing this as I do not have any bnx2x hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c |    9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 4645c44e7c15..ae668476fff0 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -6194,14 +6194,19 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 			   NETIF_F_TSO | NETIF_F_TSO6 |
 			   NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_GRE |
 			   NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
-			   NETIF_F_RXHASH |
+			   NETIF_F_GSO_UDP_TUNNEL_CSUM | NETIF_F_GSO_GRE_CSUM |
+			   NETIF_F_GSO_PARTIAL | NETIF_F_RXHASH |
 			   NETIF_F_RXCSUM | NETIF_F_LRO | NETIF_F_GRO;
 
 	dev->hw_enc_features =
 			NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM | NETIF_F_SG |
 			NETIF_F_TSO | NETIF_F_TSO6 |
 			NETIF_F_GSO_UDP_TUNNEL | NETIF_F_GSO_GRE |
-			NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT;
+			NETIF_F_GSO_UDP_TUNNEL_CSUM | NETIF_F_GSO_GRE_CSUM |
+			NETIF_F_GSO_IPIP | NETIF_F_GSO_SIT |
+			NETIF_F_GSO_PARTIAL;
+	dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				    NETIF_F_GSO_GRE_CSUM;
 	dev->vlan_features = dev->hw_features | NETIF_F_HIGHDMA;
 	dev->hw_features |= NETIF_F_HW_VLAN_CTAG_RX | NETIF_F_HW_VLAN_CTAG_TX |
 			    NETIF_F_HW_VLAN_STAG_RX | NETIF_F_HW_VLAN_STAG_TX;

[RFC PATCH 5/5] fm10k: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the fm10k hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx5 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/intel/fm10k/fm10k_netdev.c |   14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
index 2a08d3f5b6df..dc4901def72d 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
@@ -1417,15 +1417,21 @@ struct net_device *fm10k_alloc_netdev(const struct fm10k_info *info)
 
 	/* Only the PF can support VXLAN and NVGRE tunnel offloads */
 	if (info->mac == fm10k_mac_pf) {
-		dev->hw_enc_features = NETIF_F_IP_CSUM |
+		dev->hw_enc_features = NETIF_F_SG |
+				       NETIF_F_IP_CSUM |
+				       NETIF_F_IPV6_CSUM |
 				       NETIF_F_TSO |
 				       NETIF_F_TSO6 |
 				       NETIF_F_TSO_ECN |
 				       NETIF_F_GSO_UDP_TUNNEL |
-				       NETIF_F_IPV6_CSUM |
-				       NETIF_F_SG;
+				       NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				       NETIF_F_GSO_PARTIAL;
+
+		dev->features |= NETIF_F_GSO_UDP_TUNNEL |
+				 NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				 NETIF_F_GSO_PARTIAL;
 
-		dev->features |= NETIF_F_GSO_UDP_TUNNEL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	/* all features defined to this point should be changeable */

[Intel-wired-lan] [RFC PATCH 5/5] fm10k: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

This patch assumes that the fm10k hardware will ignore existing IPv4/v6
header fields for length and checksum as well as the length and checksum
fields for outer UDP headers.

I have no means of testing this as I do not have any mlx5 hardware but
thought I would submit it as an RFC to see if anyone out there wants to
test this and see if this does in fact enable this functionality allowing
us to to segment UDP tunneled frames that have an outer checksum.

Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
---
 drivers/net/ethernet/intel/fm10k/fm10k_netdev.c |   14 ++++++++++----
 1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
index 2a08d3f5b6df..dc4901def72d 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
@@ -1417,15 +1417,21 @@ struct net_device *fm10k_alloc_netdev(const struct fm10k_info *info)
 
 	/* Only the PF can support VXLAN and NVGRE tunnel offloads */
 	if (info->mac == fm10k_mac_pf) {
-		dev->hw_enc_features = NETIF_F_IP_CSUM |
+		dev->hw_enc_features = NETIF_F_SG |
+				       NETIF_F_IP_CSUM |
+				       NETIF_F_IPV6_CSUM |
 				       NETIF_F_TSO |
 				       NETIF_F_TSO6 |
 				       NETIF_F_TSO_ECN |
 				       NETIF_F_GSO_UDP_TUNNEL |
-				       NETIF_F_IPV6_CSUM |
-				       NETIF_F_SG;
+				       NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				       NETIF_F_GSO_PARTIAL;
+
+		dev->features |= NETIF_F_GSO_UDP_TUNNEL |
+				 NETIF_F_GSO_UDP_TUNNEL_CSUM |
+				 NETIF_F_GSO_PARTIAL;
 
-		dev->features |= NETIF_F_GSO_UDP_TUNNEL;
+		dev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
 	}
 
 	/* all features defined to this point should be changeable */

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Saeed Mahameed]

On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
> header fields for length and checksum as well as the length and checksum
> fields for outer UDP headers.
>
> I have no means of testing this as I do not have any mlx5 hardware but
> thought I would submit it as an RFC to see if anyone out there wants to
> test this and see if this does in fact enable this functionality allowing
> us to to segment UDP tunneled frames that have an outer checksum.
>
> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> index e0adb604f461..57d8da796d50 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>
>         if (mlx5e_vxlan_allowed(mdev)) {
> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
> +                                          NETIF_F_GSO_PARTIAL;
>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>                 netdev->hw_enc_features |= NETIF_F_TSO;
>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
> +                                          NETIF_F_GSO_PARTIAL;
> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>         }
>
>         netdev->features          = netdev->hw_features;
>

Hi Alex,

Adding Matt, VxLAN feature owner from Mellanox,
Matt please correct me if am wrong, but We already tested GSO VxLAN
and we saw the TCP/IP checksum offloads for both inner and outer
headers handled by the hardware.

And looking at mlx5e_sq_xmit:

if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
        eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
if (skb->encapsulation) {
        eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
MLX5_ETH_WQE_L4_INNER_CSUM;
        sq->stats.csum_offload_inner++;
} else {
        eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
}

We enable inner/outer hardware checksumming unconditionally without
looking at the features Alex is suggesting in this patch,
Alex, can you elaborate more on the meaning of those features ? and
why would it work for us without declaring them ?

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Saeed Mahameed]

On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
> header fields for length and checksum as well as the length and checksum
> fields for outer UDP headers.
>
> I have no means of testing this as I do not have any mlx5 hardware but
> thought I would submit it as an RFC to see if anyone out there wants to
> test this and see if this does in fact enable this functionality allowing
> us to to segment UDP tunneled frames that have an outer checksum.
>
> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
> ---
>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> index e0adb604f461..57d8da796d50 100644
> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>
>         if (mlx5e_vxlan_allowed(mdev)) {
> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
> +                                          NETIF_F_GSO_PARTIAL;
>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>                 netdev->hw_enc_features |= NETIF_F_TSO;
>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
> +                                          NETIF_F_GSO_PARTIAL;
> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>         }
>
>         netdev->features          = netdev->hw_features;
>

Hi Alex,

Adding Matt, VxLAN feature owner from Mellanox,
Matt please correct me if am wrong, but We already tested GSO VxLAN
and we saw the TCP/IP checksum offloads for both inner and outer
headers handled by the hardware.

And looking at mlx5e_sq_xmit:

if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
        eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
if (skb->encapsulation) {
        eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
MLX5_ETH_WQE_L4_INNER_CSUM;
        sq->stats.csum_offload_inner++;
} else {
        eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
}

We enable inner/outer hardware checksumming unconditionally without
looking at the features Alex is suggesting in this patch,
Alex, can you elaborate more on the meaning of those features ? and
why would it work for us without declaring them ?

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
<saeedm@dev.mellanox.co.il> wrote:
> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>> header fields for length and checksum as well as the length and checksum
>> fields for outer UDP headers.
>>
>> I have no means of testing this as I do not have any mlx5 hardware but
>> thought I would submit it as an RFC to see if anyone out there wants to
>> test this and see if this does in fact enable this functionality allowing
>> us to to segment UDP tunneled frames that have an outer checksum.
>>
>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>> ---
>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> index e0adb604f461..57d8da796d50 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>
>>         if (mlx5e_vxlan_allowed(mdev)) {
>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>> +                                          NETIF_F_GSO_PARTIAL;
>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>> +                                          NETIF_F_GSO_PARTIAL;
>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>         }
>>
>>         netdev->features          = netdev->hw_features;
>>
>
> Hi Alex,
>
> Adding Matt, VxLAN feature owner from Mellanox,
> Matt please correct me if am wrong, but We already tested GSO VxLAN
> and we saw the TCP/IP checksum offloads for both inner and outer
> headers handled by the hardware.
>
> And looking at mlx5e_sq_xmit:
>
> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
> if (skb->encapsulation) {
>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
> MLX5_ETH_WQE_L4_INNER_CSUM;
>         sq->stats.csum_offload_inner++;
> } else {
>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
> }
>
> We enable inner/outer hardware checksumming unconditionally without
> looking at the features Alex is suggesting in this patch,
> Alex, can you elaborate more on the meaning of those features ? and
> why would it work for us without declaring them ?

Well right now the feature list exposed by the device indicates that
TSO is not used if a VxLAN tunnel has a checksum in an outer header.
Since that is not exposed currently that is completely offloaded in
software via GSO.

What the GSO partial does is allow us to treat GSO for tunnels with
checksum like it is GSO for tunnels without checksum by precomputing
the UDP checksum as though the frame had already been segmented and
restricts us to an even multiple of MSS bytes that are to be segmented
between all the frames.  One side effect though is that all of the IP
and UDP header fields are also precomputed, but from what I can tell
it looks like the values that would be changed by a change in length
are ignored or overwritten by the hardware and driver anyway.

- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
<saeedm@dev.mellanox.co.il> wrote:
> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>> header fields for length and checksum as well as the length and checksum
>> fields for outer UDP headers.
>>
>> I have no means of testing this as I do not have any mlx5 hardware but
>> thought I would submit it as an RFC to see if anyone out there wants to
>> test this and see if this does in fact enable this functionality allowing
>> us to to segment UDP tunneled frames that have an outer checksum.
>>
>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>> ---
>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> index e0adb604f461..57d8da796d50 100644
>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>
>>         if (mlx5e_vxlan_allowed(mdev)) {
>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>> +                                          NETIF_F_GSO_PARTIAL;
>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>> +                                          NETIF_F_GSO_PARTIAL;
>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>         }
>>
>>         netdev->features          = netdev->hw_features;
>>
>
> Hi Alex,
>
> Adding Matt, VxLAN feature owner from Mellanox,
> Matt please correct me if am wrong, but We already tested GSO VxLAN
> and we saw the TCP/IP checksum offloads for both inner and outer
> headers handled by the hardware.
>
> And looking at mlx5e_sq_xmit:
>
> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
> if (skb->encapsulation) {
>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
> MLX5_ETH_WQE_L4_INNER_CSUM;
>         sq->stats.csum_offload_inner++;
> } else {
>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
> }
>
> We enable inner/outer hardware checksumming unconditionally without
> looking at the features Alex is suggesting in this patch,
> Alex, can you elaborate more on the meaning of those features ? and
> why would it work for us without declaring them ?

Well right now the feature list exposed by the device indicates that
TSO is not used if a VxLAN tunnel has a checksum in an outer header.
Since that is not exposed currently that is completely offloaded in
software via GSO.

What the GSO partial does is allow us to treat GSO for tunnels with
checksum like it is GSO for tunnels without checksum by precomputing
the UDP checksum as though the frame had already been segmented and
restricts us to an even multiple of MSS bytes that are to be segmented
between all the frames.  One side effect though is that all of the IP
and UDP header fields are also precomputed, but from what I can tell
it looks like the values that would be changed by a change in length
are ignored or overwritten by the hardware and driver anyway.

- Alex

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
> header fields for length and checksum as well as the length and checksum
> fields for outer UDP and GRE headers.
>
> I have no means of testing this as I do not have any bnx2x hardware but
> thought I would submit it as an RFC to see if anyone out there wants to
> test this and see if this does in fact enable this functionality allowing
> us to to segment tunneled frames that have an outer checksum.
>
> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>

Hi Alex, I just did a very quick test of this patch on our bnxt
hardware and it seemed to work.

I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
getting through.  I've verified that our hardware can be programmed to
either ignore outer UDP checksum or to calculate it.  Current default
is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
Thanks.

> ---
>  drivers/net/ethernet/broadcom/bnxt/bnxt.c |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
> header fields for length and checksum as well as the length and checksum
> fields for outer UDP and GRE headers.
>
> I have no means of testing this as I do not have any bnx2x hardware but
> thought I would submit it as an RFC to see if anyone out there wants to
> test this and see if this does in fact enable this functionality allowing
> us to to segment tunneled frames that have an outer checksum.
>
> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>

Hi Alex, I just did a very quick test of this patch on our bnxt
hardware and it seemed to work.

I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
getting through.  I've verified that our hardware can be programmed to
either ignore outer UDP checksum or to calculate it.  Current default
is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
Thanks.

> ---
>  drivers/net/ethernet/broadcom/bnxt/bnxt.c |    9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
>

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
<michael.chan@broadcom.com> wrote:
> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>> header fields for length and checksum as well as the length and checksum
>> fields for outer UDP and GRE headers.
>>
>> I have no means of testing this as I do not have any bnx2x hardware but
>> thought I would submit it as an RFC to see if anyone out there wants to
>> test this and see if this does in fact enable this functionality allowing
>> us to to segment tunneled frames that have an outer checksum.
>>
>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>
> Hi Alex, I just did a very quick test of this patch on our bnxt
> hardware and it seemed to work.
>
> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
> getting through.  I've verified that our hardware can be programmed to
> either ignore outer UDP checksum or to calculate it.  Current default
> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
> Thanks.

Are you saying you can natively support UDP tunnel with outer checksum
offload then?

I'm just trying to sort out if you actually need to have the partial
segmentation offload support or if we can handle it in hardware.  Also
is there any documentation you could point me to that might help to
clarify what the hardware does/doesn't support so that I could improve
upon this patch in order to make sure we are getting the most bang for
the buck in terms of the features that can be offloaded by hardware?

Thanks.

- Alex

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
<michael.chan@broadcom.com> wrote:
> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>> header fields for length and checksum as well as the length and checksum
>> fields for outer UDP and GRE headers.
>>
>> I have no means of testing this as I do not have any bnx2x hardware but
>> thought I would submit it as an RFC to see if anyone out there wants to
>> test this and see if this does in fact enable this functionality allowing
>> us to to segment tunneled frames that have an outer checksum.
>>
>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>
> Hi Alex, I just did a very quick test of this patch on our bnxt
> hardware and it seemed to work.
>
> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
> getting through.  I've verified that our hardware can be programmed to
> either ignore outer UDP checksum or to calculate it.  Current default
> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
> Thanks.

Are you saying you can natively support UDP tunnel with outer checksum
offload then?

I'm just trying to sort out if you actually need to have the partial
segmentation offload support or if we can handle it in hardware.  Also
is there any documentation you could point me to that might help to
clarify what the hardware does/doesn't support so that I could improve
upon this patch in order to make sure we are getting the most bang for
the buck in terms of the features that can be offloaded by hardware?

Thanks.

- Alex

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
> <michael.chan@broadcom.com> wrote:
>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>> header fields for length and checksum as well as the length and checksum
>>> fields for outer UDP and GRE headers.
>>>
>>> I have no means of testing this as I do not have any bnx2x hardware but
>>> thought I would submit it as an RFC to see if anyone out there wants to
>>> test this and see if this does in fact enable this functionality allowing
>>> us to to segment tunneled frames that have an outer checksum.
>>>
>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>
>> Hi Alex, I just did a very quick test of this patch on our bnxt
>> hardware and it seemed to work.
>>
>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>> getting through.  I've verified that our hardware can be programmed to
>> either ignore outer UDP checksum or to calculate it.  Current default
>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>> Thanks.
>
> Are you saying you can natively support UDP tunnel with outer checksum
> offload then?

Yes.  Calculate or ignore the outer UDP checksum.

>
> I'm just trying to sort out if you actually need to have the partial
> segmentation offload support or if we can handle it in hardware.  Also
> is there any documentation you could point me to that might help to
> clarify what the hardware does/doesn't support so that I could improve
> upon this patch in order to make sure we are getting the most bang for
> the buck in terms of the features that can be offloaded by hardware?

No public documentation yet.  I think the plan is to publish the
programmer's reference on our website at some point in the future.

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
> <michael.chan@broadcom.com> wrote:
>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>> header fields for length and checksum as well as the length and checksum
>>> fields for outer UDP and GRE headers.
>>>
>>> I have no means of testing this as I do not have any bnx2x hardware but
>>> thought I would submit it as an RFC to see if anyone out there wants to
>>> test this and see if this does in fact enable this functionality allowing
>>> us to to segment tunneled frames that have an outer checksum.
>>>
>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>
>> Hi Alex, I just did a very quick test of this patch on our bnxt
>> hardware and it seemed to work.
>>
>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>> getting through.  I've verified that our hardware can be programmed to
>> either ignore outer UDP checksum or to calculate it.  Current default
>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>> Thanks.
>
> Are you saying you can natively support UDP tunnel with outer checksum
> offload then?

Yes.  Calculate or ignore the outer UDP checksum.

>
> I'm just trying to sort out if you actually need to have the partial
> segmentation offload support or if we can handle it in hardware.  Also
> is there any documentation you could point me to that might help to
> clarify what the hardware does/doesn't support so that I could improve
> upon this patch in order to make sure we are getting the most bang for
> the buck in terms of the features that can be offloaded by hardware?

No public documentation yet.  I think the plan is to publish the
programmer's reference on our website at some point in the future.

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Matthew Finlay]

On 4/20/16, 11:06 AM, "Alexander Duyck" <alexander.duyck@gmail.com> wrote:

>On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
><saeedm@dev.mellanox.co.il> wrote:
>> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>>> header fields for length and checksum as well as the length and checksum
>>> fields for outer UDP headers.
>>>
>>> I have no means of testing this as I do not have any mlx5 hardware but
>>> thought I would submit it as an RFC to see if anyone out there wants to
>>> test this and see if this does in fact enable this functionality allowing
>>> us to to segment UDP tunneled frames that have an outer checksum.
>>>
>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>> ---
>>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> index e0adb604f461..57d8da796d50 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>>
>>>         if (mlx5e_vxlan_allowed(mdev)) {
>>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>> +                                          NETIF_F_GSO_PARTIAL;
>>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>> +                                          NETIF_F_GSO_PARTIAL;
>>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>         }
>>>
>>>         netdev->features          = netdev->hw_features;
>>>
>>
>> Hi Alex,
>>
>> Adding Matt, VxLAN feature owner from Mellanox,
>> Matt please correct me if am wrong, but We already tested GSO VxLAN
>> and we saw the TCP/IP checksum offloads for both inner and outer
>> headers handled by the hardware.
>>
>> And looking at mlx5e_sq_xmit:
>>
>> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
>> if (skb->encapsulation) {
>>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
>> MLX5_ETH_WQE_L4_INNER_CSUM;
>>         sq->stats.csum_offload_inner++;
>> } else {
>>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
>> }
>>
>> We enable inner/outer hardware checksumming unconditionally without
>> looking at the features Alex is suggesting in this patch,
>> Alex, can you elaborate more on the meaning of those features ? and
>> why would it work for us without declaring them ?
>
>Well right now the feature list exposed by the device indicates that
>TSO is not used if a VxLAN tunnel has a checksum in an outer header.
>Since that is not exposed currently that is completely offloaded in
>software via GSO.

The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets. 

>
>What the GSO partial does is allow us to treat GSO for tunnels with
>checksum like it is GSO for tunnels without checksum by precomputing
>the UDP checksum as though the frame had already been segmented and
>restricts us to an even multiple of MSS bytes that are to be segmented
>between all the frames.  One side effect though is that all of the IP
>and UDP header fields are also precomputed, but from what I can tell
>it looks like the values that would be changed by a change in length
>are ignored or overwritten by the hardware and driver anyway.
>
>- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Matthew Finlay]

On 4/20/16, 11:06 AM, "Alexander Duyck" <alexander.duyck@gmail.com> wrote:

>On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
><saeedm@dev.mellanox.co.il> wrote:
>> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>>> header fields for length and checksum as well as the length and checksum
>>> fields for outer UDP headers.
>>>
>>> I have no means of testing this as I do not have any mlx5 hardware but
>>> thought I would submit it as an RFC to see if anyone out there wants to
>>> test this and see if this does in fact enable this functionality allowing
>>> us to to segment UDP tunneled frames that have an outer checksum.
>>>
>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>> ---
>>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> index e0adb604f461..57d8da796d50 100644
>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>>
>>>         if (mlx5e_vxlan_allowed(mdev)) {
>>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>> +                                          NETIF_F_GSO_PARTIAL;
>>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>> +                                          NETIF_F_GSO_PARTIAL;
>>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>         }
>>>
>>>         netdev->features          = netdev->hw_features;
>>>
>>
>> Hi Alex,
>>
>> Adding Matt, VxLAN feature owner from Mellanox,
>> Matt please correct me if am wrong, but We already tested GSO VxLAN
>> and we saw the TCP/IP checksum offloads for both inner and outer
>> headers handled by the hardware.
>>
>> And looking at mlx5e_sq_xmit:
>>
>> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
>> if (skb->encapsulation) {
>>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
>> MLX5_ETH_WQE_L4_INNER_CSUM;
>>         sq->stats.csum_offload_inner++;
>> } else {
>>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
>> }
>>
>> We enable inner/outer hardware checksumming unconditionally without
>> looking at the features Alex is suggesting in this patch,
>> Alex, can you elaborate more on the meaning of those features ? and
>> why would it work for us without declaring them ?
>
>Well right now the feature list exposed by the device indicates that
>TSO is not used if a VxLAN tunnel has a checksum in an outer header.
>Since that is not exposed currently that is completely offloaded in
>software via GSO.

The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets. 

>
>What the GSO partial does is allow us to treat GSO for tunnels with
>checksum like it is GSO for tunnels without checksum by precomputing
>the UDP checksum as though the frame had already been segmented and
>restricts us to an even multiple of MSS bytes that are to be segmented
>between all the frames.  One side effect though is that all of the IP
>and UDP header fields are also precomputed, but from what I can tell
>it looks like the values that would be changed by a change in length
>are ignored or overwritten by the hardware and driver anyway.
>
>- Alex

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Thu, Apr 28, 2016 at 2:43 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>
>
>
>
>
> On 4/20/16, 11:06 AM, "Alexander Duyck" <alexander.duyck@gmail.com> wrote:
>
>>On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
>><saeedm@dev.mellanox.co.il> wrote:
>>> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>>>> header fields for length and checksum as well as the length and checksum
>>>> fields for outer UDP headers.
>>>>
>>>> I have no means of testing this as I do not have any mlx5 hardware but
>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>> test this and see if this does in fact enable this functionality allowing
>>>> us to to segment UDP tunneled frames that have an outer checksum.
>>>>
>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>> ---
>>>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> index e0adb604f461..57d8da796d50 100644
>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>>>
>>>>         if (mlx5e_vxlan_allowed(mdev)) {
>>>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>>>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>>>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>>> +                                          NETIF_F_GSO_PARTIAL;
>>>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>>> +                                          NETIF_F_GSO_PARTIAL;
>>>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>         }
>>>>
>>>>         netdev->features          = netdev->hw_features;
>>>>
>>>
>>> Hi Alex,
>>>
>>> Adding Matt, VxLAN feature owner from Mellanox,
>>> Matt please correct me if am wrong, but We already tested GSO VxLAN
>>> and we saw the TCP/IP checksum offloads for both inner and outer
>>> headers handled by the hardware.
>>>
>>> And looking at mlx5e_sq_xmit:
>>>
>>> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>>>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
>>> if (skb->encapsulation) {
>>>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
>>> MLX5_ETH_WQE_L4_INNER_CSUM;
>>>         sq->stats.csum_offload_inner++;
>>> } else {
>>>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
>>> }
>>>
>>> We enable inner/outer hardware checksumming unconditionally without
>>> looking at the features Alex is suggesting in this patch,
>>> Alex, can you elaborate more on the meaning of those features ? and
>>> why would it work for us without declaring them ?
>>
>>Well right now the feature list exposed by the device indicates that
>>TSO is not used if a VxLAN tunnel has a checksum in an outer header.
>>Since that is not exposed currently that is completely offloaded in
>>software via GSO.
>
> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.

The Intel documentation said the same thing.  That was due to the fact
that the hardware didn't computer the outer UDP header checksum.  I
suspect the Mellanox hardware has the same issue.  Also I have tested
on a ConnectX-4 board with the latest firmware and what I am seeing is
that with my patches applied the outer checksum is being correctly
applied for segmentation offloads.

My thought is that that the hardware appears to ignore the UDP
checksum so if it is non-zero you cannot guarantee the checksum would
be correct on the last frame if it is a different size than the rest
of the segments.  In the case of these patches that issue has been
resolved as I have precomputed the UDP checksum for the outer UDP
header and all of the segments will be the same length so there should
be no variation in the UDP checksum of the outer header.  Unless you
can tell my exactly the reason why we cannot provide the outer UDP
checksum I would assume that the reason is due to the fact that the
hardware doesn't compute it so you cannot handle a fragment on the end
which is resolved already via GSO_PARTIAL.

- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Thu, Apr 28, 2016 at 2:43 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>
>
>
>
>
> On 4/20/16, 11:06 AM, "Alexander Duyck" <alexander.duyck@gmail.com> wrote:
>
>>On Wed, Apr 20, 2016 at 10:40 AM, Saeed Mahameed
>><saeedm@dev.mellanox.co.il> wrote:
>>> On Tue, Apr 19, 2016 at 10:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>> This patch assumes that the mlx5 hardware will ignore existing IPv4/v6
>>>> header fields for length and checksum as well as the length and checksum
>>>> fields for outer UDP headers.
>>>>
>>>> I have no means of testing this as I do not have any mlx5 hardware but
>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>> test this and see if this does in fact enable this functionality allowing
>>>> us to to segment UDP tunneled frames that have an outer checksum.
>>>>
>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>> ---
>>>>  drivers/net/ethernet/mellanox/mlx5/core/en_main.c |    7 ++++++-
>>>>  1 file changed, 6 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> index e0adb604f461..57d8da796d50 100644
>>>> --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
>>>> @@ -2390,13 +2390,18 @@ static void mlx5e_build_netdev(struct net_device *netdev)
>>>>         netdev->hw_features      |= NETIF_F_HW_VLAN_CTAG_FILTER;
>>>>
>>>>         if (mlx5e_vxlan_allowed(mdev)) {
>>>> -               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL;
>>>> +               netdev->hw_features     |= NETIF_F_GSO_UDP_TUNNEL |
>>>> +                                          NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>>> +                                          NETIF_F_GSO_PARTIAL;
>>>>                 netdev->hw_enc_features |= NETIF_F_IP_CSUM;
>>>>                 netdev->hw_enc_features |= NETIF_F_RXCSUM;
>>>>                 netdev->hw_enc_features |= NETIF_F_TSO;
>>>>                 netdev->hw_enc_features |= NETIF_F_TSO6;
>>>>                 netdev->hw_enc_features |= NETIF_F_RXHASH;
>>>>                 netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL;
>>>> +               netdev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM |
>>>> +                                          NETIF_F_GSO_PARTIAL;
>>>> +               netdev->gso_partial_features = NETIF_F_GSO_UDP_TUNNEL_CSUM;
>>>>         }
>>>>
>>>>         netdev->features          = netdev->hw_features;
>>>>
>>>
>>> Hi Alex,
>>>
>>> Adding Matt, VxLAN feature owner from Mellanox,
>>> Matt please correct me if am wrong, but We already tested GSO VxLAN
>>> and we saw the TCP/IP checksum offloads for both inner and outer
>>> headers handled by the hardware.
>>>
>>> And looking at mlx5e_sq_xmit:
>>>
>>> if (likely(skb->ip_summed == CHECKSUM_PARTIAL)) {
>>>         eseg->cs_flags = MLX5_ETH_WQE_L3_CSUM;
>>> if (skb->encapsulation) {
>>>         eseg->cs_flags |= MLX5_ETH_WQE_L3_INNER_CSUM |
>>> MLX5_ETH_WQE_L4_INNER_CSUM;
>>>         sq->stats.csum_offload_inner++;
>>> } else {
>>>         eseg->cs_flags |= MLX5_ETH_WQE_L4_CSUM;
>>> }
>>>
>>> We enable inner/outer hardware checksumming unconditionally without
>>> looking at the features Alex is suggesting in this patch,
>>> Alex, can you elaborate more on the meaning of those features ? and
>>> why would it work for us without declaring them ?
>>
>>Well right now the feature list exposed by the device indicates that
>>TSO is not used if a VxLAN tunnel has a checksum in an outer header.
>>Since that is not exposed currently that is completely offloaded in
>>software via GSO.
>
> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.

The Intel documentation said the same thing.  That was due to the fact
that the hardware didn't computer the outer UDP header checksum.  I
suspect the Mellanox hardware has the same issue.  Also I have tested
on a ConnectX-4 board with the latest firmware and what I am seeing is
that with my patches applied the outer checksum is being correctly
applied for segmentation offloads.

My thought is that that the hardware appears to ignore the UDP
checksum so if it is non-zero you cannot guarantee the checksum would
be correct on the last frame if it is a different size than the rest
of the segments.  In the case of these patches that issue has been
resolved as I have precomputed the UDP checksum for the outer UDP
header and all of the segments will be the same length so there should
be no variation in the UDP checksum of the outer header.  Unless you
can tell my exactly the reason why we cannot provide the outer UDP
checksum I would assume that the reason is due to the fact that the
hardware doesn't compute it so you cannot handle a fragment on the end
which is resolved already via GSO_PARTIAL.

- Alex

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Matthew Finlay]

>>
>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>
>The Intel documentation said the same thing.  That was due to the fact
>that the hardware didn't computer the outer UDP header checksum.  I
>suspect the Mellanox hardware has the same issue.  Also I have tested
>on a ConnectX-4 board with the latest firmware and what I am seeing is
>that with my patches applied the outer checksum is being correctly
>applied for segmentation offloads.
>
>My thought is that that the hardware appears to ignore the UDP
>checksum so if it is non-zero you cannot guarantee the checksum would
>be correct on the last frame if it is a different size than the rest
>of the segments.  In the case of these patches that issue has been
>resolved as I have precomputed the UDP checksum for the outer UDP
>header and all of the segments will be the same length so there should
>be no variation in the UDP checksum of the outer header.  Unless you
>can tell my exactly the reason why we cannot provide the outer UDP
>checksum I would assume that the reason is due to the fact that the
>hardware doesn't compute it so you cannot handle a fragment on the end
>which is resolved already via GSO_PARTIAL.

I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.

>
>- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Matthew Finlay]

>>
>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>
>The Intel documentation said the same thing.  That was due to the fact
>that the hardware didn't computer the outer UDP header checksum.  I
>suspect the Mellanox hardware has the same issue.  Also I have tested
>on a ConnectX-4 board with the latest firmware and what I am seeing is
>that with my patches applied the outer checksum is being correctly
>applied for segmentation offloads.
>
>My thought is that that the hardware appears to ignore the UDP
>checksum so if it is non-zero you cannot guarantee the checksum would
>be correct on the last frame if it is a different size than the rest
>of the segments.  In the case of these patches that issue has been
>resolved as I have precomputed the UDP checksum for the outer UDP
>header and all of the segments will be the same length so there should
>be no variation in the UDP checksum of the outer header.  Unless you
>can tell my exactly the reason why we cannot provide the outer UDP
>checksum I would assume that the reason is due to the fact that the
>hardware doesn't compute it so you cannot handle a fragment on the end
>which is resolved already via GSO_PARTIAL.

I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.

>
>- Alex

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>
>
>
>
>
>>>
>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>
>>The Intel documentation said the same thing.  That was due to the fact
>>that the hardware didn't computer the outer UDP header checksum.  I
>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>that with my patches applied the outer checksum is being correctly
>>applied for segmentation offloads.
>>
>>My thought is that that the hardware appears to ignore the UDP
>>checksum so if it is non-zero you cannot guarantee the checksum would
>>be correct on the last frame if it is a different size than the rest
>>of the segments.  In the case of these patches that issue has been
>>resolved as I have precomputed the UDP checksum for the outer UDP
>>header and all of the segments will be the same length so there should
>>be no variation in the UDP checksum of the outer header.  Unless you
>>can tell my exactly the reason why we cannot provide the outer UDP
>>checksum I would assume that the reason is due to the fact that the
>>hardware doesn't compute it so you cannot handle a fragment on the end
>>which is resolved already via GSO_PARTIAL.
>
> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.

Thanks.  Any idea how long it should be.  I know I was getting a
auto-reply about people being out until May 1st due to a holiday so I
am just wondering if we should have Dave drop this patch set and I
submit a v2 when you can get me the feedback next week, or if we run
with the patches as-is for now and be prepared to revert if anything
should come up.

- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>
>
>
>
>
>>>
>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>
>>The Intel documentation said the same thing.  That was due to the fact
>>that the hardware didn't computer the outer UDP header checksum.  I
>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>that with my patches applied the outer checksum is being correctly
>>applied for segmentation offloads.
>>
>>My thought is that that the hardware appears to ignore the UDP
>>checksum so if it is non-zero you cannot guarantee the checksum would
>>be correct on the last frame if it is a different size than the rest
>>of the segments.  In the case of these patches that issue has been
>>resolved as I have precomputed the UDP checksum for the outer UDP
>>header and all of the segments will be the same length so there should
>>be no variation in the UDP checksum of the outer header.  Unless you
>>can tell my exactly the reason why we cannot provide the outer UDP
>>checksum I would assume that the reason is due to the fact that the
>>hardware doesn't compute it so you cannot handle a fragment on the end
>>which is resolved already via GSO_PARTIAL.
>
> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.

Thanks.  Any idea how long it should be.  I know I was getting a
auto-reply about people being out until May 1st due to a holiday so I
am just wondering if we should have Dave drop this patch set and I
submit a v2 when you can get me the feedback next week, or if we run
with the patches as-is for now and be prepared to revert if anything
should come up.

- Alex

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Saeed Mahameed]

On Fri, Apr 29, 2016 at 4:59 AM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>>
>>
>>
>>
>>
>>>>
>>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>>
>>>The Intel documentation said the same thing.  That was due to the fact
>>>that the hardware didn't computer the outer UDP header checksum.  I
>>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>>that with my patches applied the outer checksum is being correctly
>>>applied for segmentation offloads.
>>>
>>>My thought is that that the hardware appears to ignore the UDP
>>>checksum so if it is non-zero you cannot guarantee the checksum would
>>>be correct on the last frame if it is a different size than the rest
>>>of the segments.  In the case of these patches that issue has been
>>>resolved as I have precomputed the UDP checksum for the outer UDP
>>>header and all of the segments will be the same length so there should
>>>be no variation in the UDP checksum of the outer header.  Unless you
>>>can tell my exactly the reason why we cannot provide the outer UDP
>>>checksum I would assume that the reason is due to the fact that the
>>>hardware doesn't compute it so you cannot handle a fragment on the end
>>>which is resolved already via GSO_PARTIAL.
>>
>> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.
>
> Thanks.  Any idea how long it should be.  I know I was getting a
> auto-reply about people being out until May 1st due to a holiday so I
> am just wondering if we should have Dave drop this patch set and I
> submit a v2 when you can get me the feedback next week, or if we run
> with the patches as-is for now and be prepared to revert if anything
> should come up.
>

Alex,

I reviewed your patches and other than  the claim about mlx4,
everything seems to be ok.
I took the liberty to apply your patches and play around with them
only on mlx5 for now,
It seems that it works and HW do correctly calculate the IPv6 outer
UDP sum as illustrated in the log from the tcpdump on the receiver
[1], I also noticed that the GSO also works on the xmitter and the HW
ignores the outer sum as expected.

Matt, I think you still need to do the homework and see if we didn't
miss anything here, but theoretically and practically what Alex did
here should work.
Tariq will also check what went wrong with mlx4 outer ipv6 support,
but this shouldn't block this series.

Side notes:
- Alex, you will need to rebase the series, it didn't apply smoothly.
- BTW mlx5 on RX side will provide csum complete and not csum unnecessary.

Saeed.

[1]
21:28:15.474287 e4:1d:2d:5c:f2:e8 > e4:1d:2d:5c:f2:d4, ethertype IPv6
(0x86dd), length 1514: (hlim 64, next-header UDP (17) payload length:
1460) 2001::37:4.58006 > 2001::36:4.4789: [udp sum ok] VXLAN, flags
[I] (0x08), vni 46
66:cb:6e:a4:31:4e > b6:bf:9b:61:31:62, ethertype IPv4 (0x0800), length
1444: (tos 0x0, ttl 64, id 64920, offset 0, flags [DF], proto TCP (6),
length 1430)
    56.0.37.4.53614 > 56.0.36.4.commplex-link: Flags [.], cksum 0xd6cc
(correct), seq 1911713070:1911714448, ack 1, win 218, options
[nop,nop,TS val 183853011 ecr 183840106], length 1378

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Saeed Mahameed]

On Fri, Apr 29, 2016 at 4:59 AM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>>
>>
>>
>>
>>
>>>>
>>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>>
>>>The Intel documentation said the same thing.  That was due to the fact
>>>that the hardware didn't computer the outer UDP header checksum.  I
>>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>>that with my patches applied the outer checksum is being correctly
>>>applied for segmentation offloads.
>>>
>>>My thought is that that the hardware appears to ignore the UDP
>>>checksum so if it is non-zero you cannot guarantee the checksum would
>>>be correct on the last frame if it is a different size than the rest
>>>of the segments.  In the case of these patches that issue has been
>>>resolved as I have precomputed the UDP checksum for the outer UDP
>>>header and all of the segments will be the same length so there should
>>>be no variation in the UDP checksum of the outer header.  Unless you
>>>can tell my exactly the reason why we cannot provide the outer UDP
>>>checksum I would assume that the reason is due to the fact that the
>>>hardware doesn't compute it so you cannot handle a fragment on the end
>>>which is resolved already via GSO_PARTIAL.
>>
>> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.
>
> Thanks.  Any idea how long it should be.  I know I was getting a
> auto-reply about people being out until May 1st due to a holiday so I
> am just wondering if we should have Dave drop this patch set and I
> submit a v2 when you can get me the feedback next week, or if we run
> with the patches as-is for now and be prepared to revert if anything
> should come up.
>

Alex,

I reviewed your patches and other than  the claim about mlx4,
everything seems to be ok.
I took the liberty to apply your patches and play around with them
only on mlx5 for now,
It seems that it works and HW do correctly calculate the IPv6 outer
UDP sum as illustrated in the log from the tcpdump on the receiver
[1], I also noticed that the GSO also works on the xmitter and the HW
ignores the outer sum as expected.

Matt, I think you still need to do the homework and see if we didn't
miss anything here, but theoretically and practically what Alex did
here should work.
Tariq will also check what went wrong with mlx4 outer ipv6 support,
but this shouldn't block this series.

Side notes:
- Alex, you will need to rebase the series, it didn't apply smoothly.
- BTW mlx5 on RX side will provide csum complete and not csum unnecessary.

Saeed.

[1]
21:28:15.474287 e4:1d:2d:5c:f2:e8 > e4:1d:2d:5c:f2:d4, ethertype IPv6
(0x86dd), length 1514: (hlim 64, next-header UDP (17) payload length:
1460) 2001::37:4.58006 > 2001::36:4.4789: [udp sum ok] VXLAN, flags
[I] (0x08), vni 46
66:cb:6e:a4:31:4e > b6:bf:9b:61:31:62, ethertype IPv4 (0x0800), length
1444: (tos 0x0, ttl 64, id 64920, offset 0, flags [DF], proto TCP (6),
length 1430)
    56.0.37.4.53614 > 56.0.36.4.commplex-link: Flags [.], cksum 0xd6cc
(correct), seq 1911713070:1911714448, ack 1, win 218, options
[nop,nop,TS val 183853011 ecr 183840106], length 1378

Re: [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Fri, Apr 29, 2016 at 12:27 PM, Saeed Mahameed
<saeedm@dev.mellanox.co.il> wrote:
> On Fri, Apr 29, 2016 at 4:59 AM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>>>
>>>
>>>
>>>
>>>
>>>>>
>>>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>>>
>>>>The Intel documentation said the same thing.  That was due to the fact
>>>>that the hardware didn't computer the outer UDP header checksum.  I
>>>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>>>that with my patches applied the outer checksum is being correctly
>>>>applied for segmentation offloads.
>>>>
>>>>My thought is that that the hardware appears to ignore the UDP
>>>>checksum so if it is non-zero you cannot guarantee the checksum would
>>>>be correct on the last frame if it is a different size than the rest
>>>>of the segments.  In the case of these patches that issue has been
>>>>resolved as I have precomputed the UDP checksum for the outer UDP
>>>>header and all of the segments will be the same length so there should
>>>>be no variation in the UDP checksum of the outer header.  Unless you
>>>>can tell my exactly the reason why we cannot provide the outer UDP
>>>>checksum I would assume that the reason is due to the fact that the
>>>>hardware doesn't compute it so you cannot handle a fragment on the end
>>>>which is resolved already via GSO_PARTIAL.
>>>
>>> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.
>>
>> Thanks.  Any idea how long it should be.  I know I was getting a
>> auto-reply about people being out until May 1st due to a holiday so I
>> am just wondering if we should have Dave drop this patch set and I
>> submit a v2 when you can get me the feedback next week, or if we run
>> with the patches as-is for now and be prepared to revert if anything
>> should come up.
>>
>
> Alex,
>
> I reviewed your patches and other than  the claim about mlx4,
> everything seems to be ok.
> I took the liberty to apply your patches and play around with them
> only on mlx5 for now,
> It seems that it works and HW do correctly calculate the IPv6 outer
> UDP sum as illustrated in the log from the tcpdump on the receiver
> [1], I also noticed that the GSO also works on the xmitter and the HW
> ignores the outer sum as expected.
>
> Matt, I think you still need to do the homework and see if we didn't
> miss anything here, but theoretically and practically what Alex did
> here should work.
> Tariq will also check what went wrong with mlx4 outer ipv6 support,
> but this shouldn't block this series.
>
> Side notes:
> - Alex, you will need to rebase the series, it didn't apply smoothly.
> - BTW mlx5 on RX side will provide csum complete and not csum unnecessary.
>
> Saeed.
>
> [1]
> 21:28:15.474287 e4:1d:2d:5c:f2:e8 > e4:1d:2d:5c:f2:d4, ethertype IPv6
> (0x86dd), length 1514: (hlim 64, next-header UDP (17) payload length:
> 1460) 2001::37:4.58006 > 2001::36:4.4789: [udp sum ok] VXLAN, flags
> [I] (0x08), vni 46
> 66:cb:6e:a4:31:4e > b6:bf:9b:61:31:62, ethertype IPv4 (0x0800), length
> 1444: (tos 0x0, ttl 64, id 64920, offset 0, flags [DF], proto TCP (6),
> length 1430)
>     56.0.37.4.53614 > 56.0.36.4.commplex-link: Flags [.], cksum 0xd6cc
> (correct), seq 1911713070:1911714448, ack 1, win 218, options
> [nop,nop,TS val 183853011 ecr 183840106], length 1378


Thanks.  I'll probably resubmit next week after your latest set of 12
patches gets applied.

- Alex

[Intel-wired-lan] [RFC PATCH 2/5] mlx5: Add support for UDP tunnel segmentation with outer checksum offload

[Alexander Duyck]

On Fri, Apr 29, 2016 at 12:27 PM, Saeed Mahameed
<saeedm@dev.mellanox.co.il> wrote:
> On Fri, Apr 29, 2016 at 4:59 AM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Thu, Apr 28, 2016 at 6:18 PM, Matthew Finlay <Matt@mellanox.com> wrote:
>>>
>>>
>>>
>>>
>>>
>>>>>
>>>>> The mlx5 hardware requires the outer UDP checksum is not set when offloading encapsulated packets.
>>>>
>>>>The Intel documentation said the same thing.  That was due to the fact
>>>>that the hardware didn't computer the outer UDP header checksum.  I
>>>>suspect the Mellanox hardware has the same issue.  Also I have tested
>>>>on a ConnectX-4 board with the latest firmware and what I am seeing is
>>>>that with my patches applied the outer checksum is being correctly
>>>>applied for segmentation offloads.
>>>>
>>>>My thought is that that the hardware appears to ignore the UDP
>>>>checksum so if it is non-zero you cannot guarantee the checksum would
>>>>be correct on the last frame if it is a different size than the rest
>>>>of the segments.  In the case of these patches that issue has been
>>>>resolved as I have precomputed the UDP checksum for the outer UDP
>>>>header and all of the segments will be the same length so there should
>>>>be no variation in the UDP checksum of the outer header.  Unless you
>>>>can tell my exactly the reason why we cannot provide the outer UDP
>>>>checksum I would assume that the reason is due to the fact that the
>>>>hardware doesn't compute it so you cannot handle a fragment on the end
>>>>which is resolved already via GSO_PARTIAL.
>>>
>>> I will check internally and verify there are no unforeseen issues with setting the outer UDP checksum in this scenario.
>>
>> Thanks.  Any idea how long it should be.  I know I was getting a
>> auto-reply about people being out until May 1st due to a holiday so I
>> am just wondering if we should have Dave drop this patch set and I
>> submit a v2 when you can get me the feedback next week, or if we run
>> with the patches as-is for now and be prepared to revert if anything
>> should come up.
>>
>
> Alex,
>
> I reviewed your patches and other than  the claim about mlx4,
> everything seems to be ok.
> I took the liberty to apply your patches and play around with them
> only on mlx5 for now,
> It seems that it works and HW do correctly calculate the IPv6 outer
> UDP sum as illustrated in the log from the tcpdump on the receiver
> [1], I also noticed that the GSO also works on the xmitter and the HW
> ignores the outer sum as expected.
>
> Matt, I think you still need to do the homework and see if we didn't
> miss anything here, but theoretically and practically what Alex did
> here should work.
> Tariq will also check what went wrong with mlx4 outer ipv6 support,
> but this shouldn't block this series.
>
> Side notes:
> - Alex, you will need to rebase the series, it didn't apply smoothly.
> - BTW mlx5 on RX side will provide csum complete and not csum unnecessary.
>
> Saeed.
>
> [1]
> 21:28:15.474287 e4:1d:2d:5c:f2:e8 > e4:1d:2d:5c:f2:d4, ethertype IPv6
> (0x86dd), length 1514: (hlim 64, next-header UDP (17) payload length:
> 1460) 2001::37:4.58006 > 2001::36:4.4789: [udp sum ok] VXLAN, flags
> [I] (0x08), vni 46
> 66:cb:6e:a4:31:4e > b6:bf:9b:61:31:62, ethertype IPv4 (0x0800), length
> 1444: (tos 0x0, ttl 64, id 64920, offset 0, flags [DF], proto TCP (6),
> length 1430)
>     56.0.37.4.53614 > 56.0.36.4.commplex-link: Flags [.], cksum 0xd6cc
> (correct), seq 1911713070:1911714448, ack 1, win 218, options
> [nop,nop,TS val 183853011 ecr 183840106], length 1378


Thanks.  I'll probably resubmit next week after your latest set of 12
patches gets applied.

- Alex

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>> <michael.chan@broadcom.com> wrote:
>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>> header fields for length and checksum as well as the length and checksum
>>>> fields for outer UDP and GRE headers.
>>>>
>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>> test this and see if this does in fact enable this functionality allowing
>>>> us to to segment tunneled frames that have an outer checksum.
>>>>
>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>
>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>> hardware and it seemed to work.
>>>
>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>> getting through.  I've verified that our hardware can be programmed to
>>> either ignore outer UDP checksum or to calculate it.  Current default
>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>> Thanks.
>>
>> Are you saying you can natively support UDP tunnel with outer checksum
>> offload then?
>
> Yes.  Calculate or ignore the outer UDP checksum.

I was just thinking about this.  When you say you compute the IPv6
checksum how is it you are specifying to the hardware that you want to
do that?  Is it something you can configure per packet or is it
something that is configured for the VXLAN flow?

I just want to make sure you aren't adding checksums to IPv6 tunnels
that specify that they don't want a checksum, or stripping them from
v4 tunnels that do want a checksum.

Thanks.

- Alex

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>> <michael.chan@broadcom.com> wrote:
>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>> header fields for length and checksum as well as the length and checksum
>>>> fields for outer UDP and GRE headers.
>>>>
>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>> test this and see if this does in fact enable this functionality allowing
>>>> us to to segment tunneled frames that have an outer checksum.
>>>>
>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>
>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>> hardware and it seemed to work.
>>>
>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>> getting through.  I've verified that our hardware can be programmed to
>>> either ignore outer UDP checksum or to calculate it.  Current default
>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>> Thanks.
>>
>> Are you saying you can natively support UDP tunnel with outer checksum
>> offload then?
>
> Yes.  Calculate or ignore the outer UDP checksum.

I was just thinking about this.  When you say you compute the IPv6
checksum how is it you are specifying to the hardware that you want to
do that?  Is it something you can configure per packet or is it
something that is configured for the VXLAN flow?

I just want to make sure you aren't adding checksums to IPv6 tunnels
that specify that they don't want a checksum, or stripping them from
v4 tunnels that do want a checksum.

Thanks.

- Alex

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Fri, Apr 29, 2016 at 2:17 PM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
>> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
>> <alexander.duyck@gmail.com> wrote:
>>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>>> <michael.chan@broadcom.com> wrote:
>>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>>> header fields for length and checksum as well as the length and checksum
>>>>> fields for outer UDP and GRE headers.
>>>>>
>>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>>> test this and see if this does in fact enable this functionality allowing
>>>>> us to to segment tunneled frames that have an outer checksum.
>>>>>
>>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>>
>>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>>> hardware and it seemed to work.
>>>>
>>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>>> getting through.  I've verified that our hardware can be programmed to
>>>> either ignore outer UDP checksum or to calculate it.  Current default
>>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>>> Thanks.
>>>
>>> Are you saying you can natively support UDP tunnel with outer checksum
>>> offload then?
>>
>> Yes.  Calculate or ignore the outer UDP checksum.
>
> I was just thinking about this.  When you say you compute the IPv6
> checksum how is it you are specifying to the hardware that you want to
> do that?  Is it something you can configure per packet or is it
> something that is configured for the VXLAN flow?

In the current version of the hardware, it is a global (chip-wide)
setting.  1 bit to control outer ipv4 vxlan and 1 bit for outer ipv6
vxlan.

>
> I just want to make sure you aren't adding checksums to IPv6 tunnels
> that specify that they don't want a checksum, or stripping them from
> v4 tunnels that do want a checksum.

If the global setting has outer UDP checksum enabled, it will be
calculated no matter what.  If the setting is disabled, it will just
ignore it without overwriting it.

Thanks.

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Fri, Apr 29, 2016 at 2:17 PM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:
> On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
>> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
>> <alexander.duyck@gmail.com> wrote:
>>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>>> <michael.chan@broadcom.com> wrote:
>>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>>> header fields for length and checksum as well as the length and checksum
>>>>> fields for outer UDP and GRE headers.
>>>>>
>>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>>> test this and see if this does in fact enable this functionality allowing
>>>>> us to to segment tunneled frames that have an outer checksum.
>>>>>
>>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>>
>>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>>> hardware and it seemed to work.
>>>>
>>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>>> getting through.  I've verified that our hardware can be programmed to
>>>> either ignore outer UDP checksum or to calculate it.  Current default
>>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>>> Thanks.
>>>
>>> Are you saying you can natively support UDP tunnel with outer checksum
>>> offload then?
>>
>> Yes.  Calculate or ignore the outer UDP checksum.
>
> I was just thinking about this.  When you say you compute the IPv6
> checksum how is it you are specifying to the hardware that you want to
> do that?  Is it something you can configure per packet or is it
> something that is configured for the VXLAN flow?

In the current version of the hardware, it is a global (chip-wide)
setting.  1 bit to control outer ipv4 vxlan and 1 bit for outer ipv6
vxlan.

>
> I just want to make sure you aren't adding checksums to IPv6 tunnels
> that specify that they don't want a checksum, or stripping them from
> v4 tunnels that do want a checksum.

If the global setting has outer UDP checksum enabled, it will be
calculated no matter what.  If the setting is disabled, it will just
ignore it without overwriting it.

Thanks.

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Fri, Apr 29, 2016 at 2:29 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Fri, Apr 29, 2016 at 2:17 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
>>> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
>>> <alexander.duyck@gmail.com> wrote:
>>>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>>>> <michael.chan@broadcom.com> wrote:
>>>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>>>> header fields for length and checksum as well as the length and checksum
>>>>>> fields for outer UDP and GRE headers.
>>>>>>
>>>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>>>> test this and see if this does in fact enable this functionality allowing
>>>>>> us to to segment tunneled frames that have an outer checksum.
>>>>>>
>>>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>>>
>>>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>>>> hardware and it seemed to work.
>>>>>
>>>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>>>> getting through.  I've verified that our hardware can be programmed to
>>>>> either ignore outer UDP checksum or to calculate it.  Current default
>>>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>>>> Thanks.
>>>>
>>>> Are you saying you can natively support UDP tunnel with outer checksum
>>>> offload then?
>>>
>>> Yes.  Calculate or ignore the outer UDP checksum.
>>
>> I was just thinking about this.  When you say you compute the IPv6
>> checksum how is it you are specifying to the hardware that you want to
>> do that?  Is it something you can configure per packet or is it
>> something that is configured for the VXLAN flow?
>
> In the current version of the hardware, it is a global (chip-wide)
> setting.  1 bit to control outer ipv4 vxlan and 1 bit for outer ipv6
> vxlan.
>
>>
>> I just want to make sure you aren't adding checksums to IPv6 tunnels
>> that specify that they don't want a checksum, or stripping them from
>> v4 tunnels that do want a checksum.
>
> If the global setting has outer UDP checksum enabled, it will be
> calculated no matter what.  If the setting is disabled, it will just
> ignore it without overwriting it.

Okay so if that is the case we may want to make it so that we ignore
checksum for both IPv4 and IPv6 and then we can just provide it via
GSO_PARTIAL in the case we want it.  Otherwise you are technically
mangling the frames by inserting a checksum on the outer header even
though the tunnel was not configured for it.  If you can point me
toward the point in the code where that is happening I can probably
make it a part of this patch.

Thanks.

- Alex

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Fri, Apr 29, 2016 at 2:29 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Fri, Apr 29, 2016 at 2:17 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>> On Wed, Apr 27, 2016 at 9:32 PM, Michael Chan <michael.chan@broadcom.com> wrote:
>>> On Wed, Apr 27, 2016 at 8:21 AM, Alexander Duyck
>>> <alexander.duyck@gmail.com> wrote:
>>>> On Tue, Apr 26, 2016 at 10:55 PM, Michael Chan
>>>> <michael.chan@broadcom.com> wrote:
>>>>> On Tue, Apr 19, 2016 at 12:06 PM, Alexander Duyck <aduyck@mirantis.com> wrote:
>>>>>> This patch assumes that the bnxt hardware will ignore existing IPv4/v6
>>>>>> header fields for length and checksum as well as the length and checksum
>>>>>> fields for outer UDP and GRE headers.
>>>>>>
>>>>>> I have no means of testing this as I do not have any bnx2x hardware but
>>>>>> thought I would submit it as an RFC to see if anyone out there wants to
>>>>>> test this and see if this does in fact enable this functionality allowing
>>>>>> us to to segment tunneled frames that have an outer checksum.
>>>>>>
>>>>>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>>>>>
>>>>> Hi Alex, I just did a very quick test of this patch on our bnxt
>>>>> hardware and it seemed to work.
>>>>>
>>>>> I created a vxlan endpoint with udpcsum enabled and I saw TSO packets
>>>>> getting through.  I've verified that our hardware can be programmed to
>>>>> either ignore outer UDP checksum or to calculate it.  Current default
>>>>> is to ignore ipv4 UDP checksum and calculate ipv6 UDP checksum.
>>>>> Thanks.
>>>>
>>>> Are you saying you can natively support UDP tunnel with outer checksum
>>>> offload then?
>>>
>>> Yes.  Calculate or ignore the outer UDP checksum.
>>
>> I was just thinking about this.  When you say you compute the IPv6
>> checksum how is it you are specifying to the hardware that you want to
>> do that?  Is it something you can configure per packet or is it
>> something that is configured for the VXLAN flow?
>
> In the current version of the hardware, it is a global (chip-wide)
> setting.  1 bit to control outer ipv4 vxlan and 1 bit for outer ipv6
> vxlan.
>
>>
>> I just want to make sure you aren't adding checksums to IPv6 tunnels
>> that specify that they don't want a checksum, or stripping them from
>> v4 tunnels that do want a checksum.
>
> If the global setting has outer UDP checksum enabled, it will be
> calculated no matter what.  If the setting is disabled, it will just
> ignore it without overwriting it.

Okay so if that is the case we may want to make it so that we ignore
checksum for both IPv4 and IPv6 and then we can just provide it via
GSO_PARTIAL in the case we want it.  Otherwise you are technically
mangling the frames by inserting a checksum on the outer header even
though the tunnel was not configured for it.  If you can point me
toward the point in the code where that is happening I can probably
make it a part of this patch.

Thanks.

- Alex

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Fri, Apr 29, 2016 at 2:31 PM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:

> Okay so if that is the case we may want to make it so that we ignore
> checksum for both IPv4 and IPv6 and then we can just provide it via
> GSO_PARTIAL in the case we want it.  Otherwise you are technically
> mangling the frames by inserting a checksum on the outer header even
> though the tunnel was not configured for it.  If you can point me
> toward the point in the code where that is happening I can probably
> make it a part of this patch.
>

All the chip settings are controlled by firmware.  I will check with
the firmware team to disable them if they are not already disabled.
When first developing the driver, before all the recent proposals, the
intention was to not advertise NETIF_F_GSO_UDP_TUNNEL_CSUM and not
support TSO with outer UDP checksum enabled.  Thanks.

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Michael Chan]

On Fri, Apr 29, 2016 at 2:31 PM, Alexander Duyck
<alexander.duyck@gmail.com> wrote:

> Okay so if that is the case we may want to make it so that we ignore
> checksum for both IPv4 and IPv6 and then we can just provide it via
> GSO_PARTIAL in the case we want it.  Otherwise you are technically
> mangling the frames by inserting a checksum on the outer header even
> though the tunnel was not configured for it.  If you can point me
> toward the point in the code where that is happening I can probably
> make it a part of this patch.
>

All the chip settings are controlled by firmware.  I will check with
the firmware team to disable them if they are not already disabled.
When first developing the driver, before all the recent proposals, the
intention was to not advertise NETIF_F_GSO_UDP_TUNNEL_CSUM and not
support TSO with outer UDP checksum enabled.  Thanks.

Re: [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Fri, Apr 29, 2016 at 4:29 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Fri, Apr 29, 2016 at 2:31 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>
>> Okay so if that is the case we may want to make it so that we ignore
>> checksum for both IPv4 and IPv6 and then we can just provide it via
>> GSO_PARTIAL in the case we want it.  Otherwise you are technically
>> mangling the frames by inserting a checksum on the outer header even
>> though the tunnel was not configured for it.  If you can point me
>> toward the point in the code where that is happening I can probably
>> make it a part of this patch.
>>
>
> All the chip settings are controlled by firmware.  I will check with
> the firmware team to disable them if they are not already disabled.
> When first developing the driver, before all the recent proposals, the
> intention was to not advertise NETIF_F_GSO_UDP_TUNNEL_CSUM and not
> support TSO with outer UDP checksum enabled.  Thanks.

Right.  But adding a checksum where there wasn't one could
theoretically be problematic if there was a implementation out there
where somebody was mandating that the tunnel checksum must be 0.  The
i40e driver was enabling the same thing in the driver for an upcoming
device that supported an outer checksum offload until I went in and
fixed it.  Generally if we can match what is expected that is
preferred so we don't have any unexpected conflicts in the event that
a VTEP expects the packets to come in with checksums or without.

I have submitted the patch and when the firmware gets updated the
behavior will be cleared up so you can have it either way depending on
what the tunnel itself requested.

Thanks.

- Alex

[Intel-wired-lan] [RFC PATCH 4/5] bnxt: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Fri, Apr 29, 2016 at 4:29 PM, Michael Chan <michael.chan@broadcom.com> wrote:
> On Fri, Apr 29, 2016 at 2:31 PM, Alexander Duyck
> <alexander.duyck@gmail.com> wrote:
>
>> Okay so if that is the case we may want to make it so that we ignore
>> checksum for both IPv4 and IPv6 and then we can just provide it via
>> GSO_PARTIAL in the case we want it.  Otherwise you are technically
>> mangling the frames by inserting a checksum on the outer header even
>> though the tunnel was not configured for it.  If you can point me
>> toward the point in the code where that is happening I can probably
>> make it a part of this patch.
>>
>
> All the chip settings are controlled by firmware.  I will check with
> the firmware team to disable them if they are not already disabled.
> When first developing the driver, before all the recent proposals, the
> intention was to not advertise NETIF_F_GSO_UDP_TUNNEL_CSUM and not
> support TSO with outer UDP checksum enabled.  Thanks.

Right.  But adding a checksum where there wasn't one could
theoretically be problematic if there was a implementation out there
where somebody was mandating that the tunnel checksum must be 0.  The
i40e driver was enabling the same thing in the driver for an upcoming
device that supported an outer checksum offload until I went in and
fixed it.  Generally if we can match what is expected that is
preferred so we don't have any unexpected conflicts in the event that
a VTEP expects the packets to come in with checksums or without.

I have submitted the patch and when the firmware gets updated the
behavior will be cleared up so you can have it either way depending on
what the tunnel itself requested.

Thanks.

- Alex

RE: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Yuval Mintz]

> This patch assumes that the bnx2x hardware will ignore existing IPv4/v6 header
> fields for length and checksum as well as the length and checksum fields for
> outer UDP and GRE headers.
> 
> I have no means of testing this as I do not have any bnx2x hardware but thought
> I would submit it as an RFC to see if anyone out there wants to test this and see
> if this does in fact enable this functionality allowing us to to segment tunneled
> frames that have an outer checksum.
> 
> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>

So it took me some [well, a lot] time to reach this, but I've finally  gave it a try.
I saw a performance boost with the partial support -
Throughput for vxlan tunnels with and without udpcsum were almost identical
after this, whereas without this patch the udpcsum prevented GSO and 
a TCP/IPv4 connection on top of it got roughly half the throughput.

However, I did encounter one oddity I couldn't explain -
After I've disabled tx-udp_tnl-segmentation via ethtool on the base interface,
got left with:
   tx-gso-partial: on
   tx-udp_tnl-segmentation: off
   tx-udp_tnl-csum-segmentation: on

When I ran traffic over both vxlan tunnels the one with the udpcsum was still
Passing gso aggregations to base device to transmit [and the throughput was
same as before], where's the tunnel without the udpcsum showed only
MTU-sized packets reaching the base interface for transmission [which is what
I've expected]

Any idea why that happened?

Re: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Wed, Aug 24, 2016 at 5:33 AM, Yuval Mintz <Yuval.Mintz@qlogic.com> wrote:
>> This patch assumes that the bnx2x hardware will ignore existing IPv4/v6 header
>> fields for length and checksum as well as the length and checksum fields for
>> outer UDP and GRE headers.
>>
>> I have no means of testing this as I do not have any bnx2x hardware but thought
>> I would submit it as an RFC to see if anyone out there wants to test this and see
>> if this does in fact enable this functionality allowing us to to segment tunneled
>> frames that have an outer checksum.
>>
>> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>
> So it took me some [well, a lot] time to reach this, but I've finally  gave it a try.
> I saw a performance boost with the partial support -
> Throughput for vxlan tunnels with and without udpcsum were almost identical
> after this, whereas without this patch the udpcsum prevented GSO and
> a TCP/IPv4 connection on top of it got roughly half the throughput.
>
> However, I did encounter one oddity I couldn't explain -
> After I've disabled tx-udp_tnl-segmentation via ethtool on the base interface,
> got left with:
>    tx-gso-partial: on
>    tx-udp_tnl-segmentation: off
>    tx-udp_tnl-csum-segmentation: on
>
> When I ran traffic over both vxlan tunnels the one with the udpcsum was still
> Passing gso aggregations to base device to transmit [and the throughput was
> same as before], where's the tunnel without the udpcsum showed only
> MTU-sized packets reaching the base interface for transmission [which is what
> I've expected]
>
> Any idea why that happened?

So the way they are implemented tx-udp_tnl-segmentation and
tx-udp_tnl-csum-segmentation are treated as two separate features.
The kernel currently gives them the same treatment as NETIF_F_TSO and
NETIF_F_TSO6.  You can disable one and the other still functions.

Now if you disable tx-gso-partial you should expect to see
tx-udp_tnl-csum-segmentation be disabled because it is dependent on
the partial GSO offload.

- Alex

RE: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Yuval Mintz]

> >> This patch assumes that the bnx2x hardware will ignore existing
> >> IPv4/v6 header fields for length and checksum as well as the length
> >> and checksum fields for outer UDP and GRE headers.
> >>
> >> I have no means of testing this as I do not have any bnx2x hardware
> >> but thought I would submit it as an RFC to see if anyone out there
> >> wants to test this and see if this does in fact enable this
> >> functionality allowing us to to segment tunneled frames that have an outer
> checksum.
> >>
> >> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
> >
> > So it took me some [well, a lot] time to reach this, but I've finally  gave it a try.
> > I saw a performance boost with the partial support - Throughput for
> > vxlan tunnels with and without udpcsum were almost identical after
> > this, whereas without this patch the udpcsum prevented GSO and a
> > TCP/IPv4 connection on top of it got roughly half the throughput.
> >
> > However, I did encounter one oddity I couldn't explain - After I've
> > disabled tx-udp_tnl-segmentation via ethtool on the base interface,
> > got left with:
> >    tx-gso-partial: on
> >    tx-udp_tnl-segmentation: off
> >    tx-udp_tnl-csum-segmentation: on
> >
> > When I ran traffic over both vxlan tunnels the one with the udpcsum
> > was still Passing gso aggregations to base device to transmit [and the
> > throughput was same as before], where's the tunnel without the udpcsum
> > showed only MTU-sized packets reaching the base interface for
> > transmission [which is what I've expected]
> >
> > Any idea why that happened?
> 
> So the way they are implemented tx-udp_tnl-segmentation and tx-udp_tnl-
> csum-segmentation are treated as two separate features.
> The kernel currently gives them the same treatment as NETIF_F_TSO and
> NETIF_F_TSO6.  You can disable one and the other still functions.
> 
> Now if you disable tx-gso-partial you should expect to see tx-udp_tnl-csum-
> segmentation be disabled because it is dependent on the partial GSO offload.
> 
> - Alex

O.k., thanks.
Then I'll run some more testing scenarios, and assuming everything works
fine I'll re-send this. Alex - should I place you at the 'from' field?

Re: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Wed, Aug 24, 2016 at 10:06 PM, Yuval Mintz <Yuval.Mintz@qlogic.com> wrote:
>> >> This patch assumes that the bnx2x hardware will ignore existing
>> >> IPv4/v6 header fields for length and checksum as well as the length
>> >> and checksum fields for outer UDP and GRE headers.
>> >>
>> >> I have no means of testing this as I do not have any bnx2x hardware
>> >> but thought I would submit it as an RFC to see if anyone out there
>> >> wants to test this and see if this does in fact enable this
>> >> functionality allowing us to to segment tunneled frames that have an outer
>> checksum.
>> >>
>> >> Signed-off-by: Alexander Duyck <aduyck@mirantis.com>
>> >
>> > So it took me some [well, a lot] time to reach this, but I've finally  gave it a try.
>> > I saw a performance boost with the partial support - Throughput for
>> > vxlan tunnels with and without udpcsum were almost identical after
>> > this, whereas without this patch the udpcsum prevented GSO and a
>> > TCP/IPv4 connection on top of it got roughly half the throughput.
>> >
>> > However, I did encounter one oddity I couldn't explain - After I've
>> > disabled tx-udp_tnl-segmentation via ethtool on the base interface,
>> > got left with:
>> >    tx-gso-partial: on
>> >    tx-udp_tnl-segmentation: off
>> >    tx-udp_tnl-csum-segmentation: on
>> >
>> > When I ran traffic over both vxlan tunnels the one with the udpcsum
>> > was still Passing gso aggregations to base device to transmit [and the
>> > throughput was same as before], where's the tunnel without the udpcsum
>> > showed only MTU-sized packets reaching the base interface for
>> > transmission [which is what I've expected]
>> >
>> > Any idea why that happened?
>>
>> So the way they are implemented tx-udp_tnl-segmentation and tx-udp_tnl-
>> csum-segmentation are treated as two separate features.
>> The kernel currently gives them the same treatment as NETIF_F_TSO and
>> NETIF_F_TSO6.  You can disable one and the other still functions.
>>
>> Now if you disable tx-gso-partial you should expect to see tx-udp_tnl-csum-
>> segmentation be disabled because it is dependent on the partial GSO offload.
>>
>> - Alex
>
> O.k., thanks.
> Then I'll run some more testing scenarios, and assuming everything works
> fine I'll re-send this. Alex - should I place you at the 'from' field?

If you could do that, then that would be great.  I'm not working at
Mirantis anymore, but I don't believe that the account is setup to
bounce the email so it shouldn't generate any noise for Dave.  If not
you can just leave my original signed off by and add yours if you
want.

- Alex

RE: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Yuval Mintz]

One question I have regarding the feature, regarding the
partial offload compatible with ndo_features_check().

Consider the following example -
Let's assume my adapter is capable of doing outer-csum validation
for vxlan packets, but only if inner network protocol is IPv4,
while at the same time it's capable of doing gso offloading for any
vxlan-encapsulated packet.

In that case, I'll publish NETIF_F_GSO_UDP_TUNNEL_CSUM as
part of my offload capability [& encap capabilities], and in my driver's
implementation of ndo_features_check() I'd remove the feature in case
I'd find out SKB is vxlan whose inner network protocol is ipv6.

Is there a way I could have benefited from the partial offload in that
case? If I understand the feature correctly, if I were to publish
UDP_TUNNEL_CSUM  under partial-gso, it would mean that stack would
peel the feature off until it would decide there's actual need for SW GSO,
thereby denying me the ability of utilizing the HW offload capability for CSUM.

Am I reading it wrong? Or does this trade-off exist?

Re: [RFC PATCH 3/5] bnx2x: Add support for segmentation of tunnels with outer checksums

[Alexander Duyck]

On Wed, Aug 31, 2016 at 4:31 AM, Yuval Mintz <Yuval.Mintz@qlogic.com> wrote:
> One question I have regarding the feature, regarding the
> partial offload compatible with ndo_features_check().
>
> Consider the following example -
> Let's assume my adapter is capable of doing outer-csum validation
> for vxlan packets, but only if inner network protocol is IPv4,
> while at the same time it's capable of doing gso offloading for any
> vxlan-encapsulated packet.

When you say validation do you mean generating an outer checksum?  If
I recall ndo_features_check applies to Tx features not Rx features.
I'll assume you mean Tx checksum offload and not validation.

> In that case, I'll publish NETIF_F_GSO_UDP_TUNNEL_CSUM as
> part of my offload capability [& encap capabilities], and in my driver's
> implementation of ndo_features_check() I'd remove the feature in case
> I'd find out SKB is vxlan whose inner network protocol is ipv6.

Right.  That sounds correct so far.

> Is there a way I could have benefited from the partial offload in that
> case? If I understand the feature correctly, if I were to publish
> UDP_TUNNEL_CSUM  under partial-gso, it would mean that stack would
> peel the feature off until it would decide there's actual need for SW GSO,
> thereby denying me the ability of utilizing the HW offload capability for CSUM.
>
> Am I reading it wrong? Or does this trade-off exist?

I think you may have that correct.  Basically any feature you
advertise as partially supporting via GSO_PARTIAL ends up being tied
to the partial offload after that.  Although from what I have seen the
difference isn't really all that great between GSO_PARTIAL versus TSO.
In many cases the stack generates MSS aligned frames anyway so all
GSO_PARTIAL is really doing is unsharing the header and updating the
header fields.