Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck BAUDIN]

Hello,

This is a repost of https://lists.01.org/pipermail/dpdk-ovs/2014-September/001788.html as the issue might be related to dpdk.org vhost code.

I am using dpdk-ovs 1.1.0 (latest release) as follow :

linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --- Niantic --- > linux-native

[*] commit 8fd8bebc051704d7caecfed8d8a065a79c022329
Author: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Date:   Mon Sep 1 12:31:11 2014 +0200

UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.

Is it supposed to work? If yes, what is the proper qemu version to use and its parameters?

I am using qemu version provided by dpdk-ovs (same issue with PDDK 1.7.0 and latest one) and I followed this documentation: https://github.com/01org/dpdk-ovs/blob/development/docs/04_Sample_Configurations/02_Userspace-vHost.md

Thanks!
Franck



This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Thomas Monjalon]

Hi Franck,

2014-09-02 13:20, Franck BAUDIN:
> I am using dpdk-ovs 1.1.0 (latest release) as follow :
> 
> linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --- Niantic --- > linux-native
> 
[...]
> UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.

Thanks for reporting.
Could you try virtio without vhost?


> This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.
> 
> Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.

This big disclaimer block has no sense on a public mailing list.
Please try to remove it.

Thanks
-- 
Thomas

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi Thomas,

On 09/02/14 15:29, Thomas Monjalon wrote:
> UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.
> Thanks for reporting.
> Could you try virtio without vhost?
Nope, dpdk-ovs supports only vhost.

Thanks!
Franck

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> 
> Hi Franck,
> 
> 2014-09-02 13:20, Franck BAUDIN:
> > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> >
> > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --
> - Niantic --- > linux-native
> >
> [...]
> > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the
> guest are incorrect, as showed with tcpdump on linux-native.
> 
> Thanks for reporting.
> Could you try virtio without vhost?
> 
> 

Looping in the dpdk-ovs list. 

* Does the new vhost API allow a user to know if all the relevant offloads have been
turned on/off for that interface? It seems that this is possible through the virtio_net
structure but it would be good to get some feedback from the relevant person
working on DPDK (Huawei?).

* If this is the case, then it is probably in the realm of the vswitch do the actual 
checksum (for VM-VM) or correctly configure the NIC when sending out through
the physical interface.

Comments?

Mark

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Xie, Huawei]

> -----Original Message-----
> From: Gray, Mark D
> Sent: Wednesday, September 03, 2014 6:01 PM
> To: Thomas Monjalon; Franck BAUDIN; Xie, Huawei
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: RE: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
> 
> >
> > Hi Franck,
> >
> > 2014-09-02 13:20, Franck BAUDIN:
> > > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> > >
> > > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --
> > - Niantic --- > linux-native
> > >
> > [...]
> > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the
> > guest are incorrect, as showed with tcpdump on linux-native.
> >
> > Thanks for reporting.
> > Could you try virtio without vhost?
> >
> >
> 
> Looping in the dpdk-ovs list.
> 
> * Does the new vhost API allow a user to know if all the relevant offloads have
> been
> turned on/off for that interface? It seems that this is possible through the
> virtio_net
> structure but it would be good to get some feedback from the relevant person
> working on DPDK (Huawei?).
> 
> * If this is the case, then it is probably in the realm of the vswitch do the actual
> checksum (for VM-VM) or correctly configure the NIC when sending out through
> the physical interface.
> 
> Comments?


Mark:
So far not supported. This is important as well in VxLan case. For the packet flow  
Guest A->  virtio -> ..->OVDK->.. -> Guest B.
1) If guest A and B are on different host machines, say A and B respectively,  and if the nic on A supports
vxlan checksum offload, then both guest and host needn't generate checksum, the nic will  
generate checksum for both inner and outer packet.
2) In VM2VM case, as it is trusted communication channel, could we negotiate with the guest tcp stack not to verify checksum 
for received packet? 

> 
> Mark

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi,

On 09/03/14 13:13, Xie, Huawei wrote:
> Looping in the dpdk-ovs list.
>
> * Does the new vhost API allow a user to know if all the relevant offloads have
> been
> turned on/off for that interface? It seems that this is possible through the
> virtio_net
> structure but it would be good to get some feedback from the relevant person
> working on DPDK (Huawei?).
>
> * If this is the case, then it is probably in the realm of the vswitch do the actual
> checksum (for VM-VM) or correctly configure the NIC when sending out through
> the physical interface.
>
> Comments?
>
> Mark:
> So far not supported. This is important as well in VxLan case. For the packet flow
> Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> 1) If guest A and B are on different host machines, say A and B respectively,  and if the nic on A supports
> vxlan checksum offload, then both guest and host needn't generate checksum, the nic will
> generate checksum for both inner and outer packet.
> 2) In VM2VM case, as it is trusted communication channel, could we negotiate with the guest tcp stack not to verify checksum
> for received packet?
The problem is that any TCP packet send by a vanilla Linux guest through 
vhost is incorrect (VM to anything, including other colocalied VMs). In 
other words, the VM cannot use TCP. QEMU options and ethtool -K csum off 
tso off ("TCP stack negociation") have no effect, maybe because the 
vhost backend is misbehaving.

Franck

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> 
> Hi,
> 
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant
> > offloads have been turned on/off for that interface? It seems that
> > this is possible through the virtio_net structure but it would be good
> > to get some feedback from the relevant person working on DPDK
> > (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch
> > do the actual checksum (for VM-VM) or correctly configure the NIC when
> > sending out through the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the
> > packet flow Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B
> > respectively,  and if the nic on A supports vxlan checksum offload,
> > then both guest and host needn't generate checksum, the nic will generate
> checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we
> > negotiate with the guest tcp stack not to verify checksum for received
> packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In other
> words, the VM cannot use TCP. QEMU options and ethtool -K csum off tso
> off ("TCP stack negociation") have no effect, maybe because the vhost
> backend is misbehaving.

This sounds like a DPDK issue. However, we did "clone and own" the vhost
code as it is a sample app. It may have changed since we integrated it.

We plan to use the vhost library interface when it is available on dpdk.org. So
that would pick up the latest version of the vhost code.

> 
> Franck

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Xie, Huawei]

> -----Original Message-----
> From: Franck Baudin [mailto:franck.baudin@qosmos.com]
> Sent: Wednesday, September 03, 2014 10:13 PM
> To: Xie, Huawei; Gray, Mark D; Thomas Monjalon
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
> 
> Hi,
> 
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant offloads have
> > been
> > turned on/off for that interface? It seems that this is possible through the
> > virtio_net
> > structure but it would be good to get some feedback from the relevant person
> > working on DPDK (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch do the actual
> > checksum (for VM-VM) or correctly configure the NIC when sending out
> through
> > the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the packet
> flow
> > Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B respectively,
> and if the nic on A supports
> > vxlan checksum offload, then both guest and host needn't generate checksum,
> the nic will
> > generate checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we negotiate
> with the guest tcp stack not to verify checksum
> > for received packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In
> other words, the VM cannot use TCP. QEMU options and ethtool -K csum off
> tso off ("TCP stack negociation") have no effect, maybe because the
> vhost backend is misbehaving.
> 
> Franck

Hi Franck:
I checked your original thread.
root at linux-native:~# tcpdump -vnei eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6), length 60)
   1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd), seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr 0,nop,wscale 7], length 0
17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6), length 40)
    1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack 1963818357, win 0, length 0

The packet from the guest, received on the native machine, has "correct" checksum 0x1db4. The TCP connection is refused is due to this packet has R flag. Could you check if it is the firewall that blocks the 
connection?

Besides, I did ssh tcp test between two guest VMs,  the ssh connection could be established correctly with tx checksum off. I didn't use OVDK, but set the arp table and route table manually.
The packet flow is
	Guest A(virtio)<->vhost example->Guest B(virtio)

Re: Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> > >
> > > Hi Franck,
> > >
> > > 2014-09-02 13:20, Franck BAUDIN:
> > > > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> > > >
> > > > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest
> > > > DPDK [*]) < --
> > > - Niantic --- > linux-native
> > > >
> > > [...]
> > > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by
> > > > the
> > > guest are incorrect, as showed with tcpdump on linux-native.
> > >
> > > Thanks for reporting.
> > > Could you try virtio without vhost?
> > >
> > >
> >
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant
> > offloads have been turned on/off for that interface? It seems that
> > this is possible through the virtio_net structure but it would be good
> > to get some feedback from the relevant person working on DPDK
> > (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch
> > do the actual checksum (for VM-VM) or correctly configure the NIC when
> > sending out through the physical interface.
> >
> > Comments?
> 
> 
> Mark:
> So far not supported. This is important as well in VxLan case. For the packet

Ok 

> flow Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> 1) If guest A and B are on different host machines, say A and B respectively,
> and if the nic on A supports vxlan checksum offload, then both guest and
> host needn't generate checksum, the nic will generate checksum for both
> inner and outer packet.

The vswitch would need to support this

> 2) In VM2VM case, as it is trusted communication channel, could we
> negotiate with the guest tcp stack not to verify checksum for received
> packet?

Perhaps

> 
> >
> > Mark