From: Tom Herbert <therbert@google.com>
To: Or Gerlitz <ogerlitz@mellanox.com>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
John Fastabend <john.r.fastabend@intel.com>,
Jeff Kirsher <jeffrey.t.kirsher@intel.com>
Subject: Re: some failures with vxlan offloads..
Date: Tue, 28 Oct 2014 08:36:11 -0700 [thread overview]
Message-ID: <CA+mtBx_ncYrp_fbp8ucXfBaCoPpv_X+TEsmofCnpzTyPPfLXnw@mail.gmail.com> (raw)
In-Reply-To: <544FB5CA.2060003@mellanox.com>
On Tue, Oct 28, 2014 at 8:27 AM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
> On 10/27/2014 3:23 AM, Tom Herbert wrote:
>>
>> On Sun, Oct 26, 2014 at 3:23 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>
>>> On Sun, Oct 26, 2014 at 5:29 PM, Tom Herbert <therbert@google.com> wrote:
>>>>
>>>> Can you determine what the TSO HW engine is setting in UDP checksum
>>>> field? tcpdump -vv might be able to show this. The symptoms seem to
>>>> indicate that it may not be zero.
>>>
>>> Thanks for the quick response. I'll check what is placed in the UDP
>>> checksum field for packets that went through the offloading HW and let
>>> you know.
>>>
>>> BTW, if following the direction you proposed, I wonder why this works
>>> (e.g the kernel doesn't drops the encapsulated TCP packets) when both
>>> sides are offloaded?
>>>
>> I'm just speculating, but the device may be returning checksum unnecessary
>> for the UDP checksum without actually checking it. Technically, VXLAN
>> RFC7348 allows an implementation to ignore the UDP checksum, although this
>> clearly violates RFC1122 UDP checksum
>> requirements. In the stack we now checksum all non-zero checksums
>> including UDP checksum in VXLAN if it's not marked checksum-unnecessary.
>
>
> OK, I found something (it's always bad habit to try and potentially blame
> someone else for your bugs...) -- as I wrote here earlier, the current HW
> doesn't support checksum generation for both the inner (say TCP) and outer
> (UDP) packet (and indeed we don't advertize SKB_GSO_UDP_TUNNEL_CSUM).
>
> So if we tell them to offload the inner TCP checksum we must **not** tell
> them to attempt and offload the outer checksum too, and I wrongly did
> that... once I stopped doing so, I get mixed configurations (one side
> offloaded the peer not offloaded) to work. I will submit mlx4 fix for that.
>
> I wonder if we have another bug somewhere... when both sides were offloaded,
> it works even with the mlx4 bug, canyou explain that?is it possible that the
> GRO stack somehow covers on the bug when both sides are offloaded and
> GRO/VXLAN comes into play?
>
Look at the receive side. As I mentioned, if the device is returning
checksum-unnecessary and setting csum_level to 1 (inner checksum was
validated) then stack won't try to verify the outer checksum. So in
this case if outer checksum is incorrect nobody complains about it.
> Or.
>
> after the fix, packets sent by the offloaded side (192.168.31.17) carry zero
> udpchecksum
>
> 17:20:44.445866 IP (tos 0x0, ttl 64, id 61275, offset 0, flags [DF], proto
> UDP (17), length 1500)
> 192.168.31.17.45387 > 192.168.31.18.4789: [no cksum] UDP, length 1472
> 0x0000: f452 1401 da82 0002 c9e9 bf32 0800 4500
> 0x0010: 05dc ef5b 4000 4011 8641 c0a8 1f11 c0a8
> 0x0020: 1f12 b14b 12b5 05c8 0000 0800 0000 0000
> 0x0030: 6300 b2c7 81db e850 7a83 2ecb 8c68 0800
> 0x0040: 4500 05aa bb84 4000 4006 9055 c0a8 3411
> 0x0050: c0a8 3412 e479 9116 553a e008 f28e 6268
> 0x0060: 5010 0038 88e3 0000 6600 6e65 7470 6572
> 0x0070: 6600 6e65 7470 6572 6600
> 17:20:44.445871 IP (tos 0x0, ttl 64, id 61276, offset 0, flags [DF], proto
> UDP (17), length 1500)
> 192.168.31.17.45387 > 192.168.31.18.4789: [no cksum] UDP, length 1472
> 0x0000: f452 1401 da82 0002 c9e9 bf32 0800 4500
> 0x0010: 05dc ef5c 4000 4011 8640 c0a8 1f11 c0a8
> 0x0020: 1f12 b14b 12b5 05c8 0000 0800 0000 0000
> 0x0030: 6300 b2c7 81db e850 7a83 2ecb 8c68 0800
> 0x0040: 4500 05aa bb85 4000 4006 9054 c0a8 3411
> 0x0050: c0a8 3412 e479 9116 553a e58a f28e 6268
> 0x0060: 5010 0038 7afc 0000 6e65 7470 6572 6600
> 0x0070: 6e65 7470 6572 6600 6e65
>
>
> before the fix, packets sent by the offloaded side (192.168.31.17) carry
> junkudpchecksum
>
> Also note that on one of the packets sent by the offloaded part, we don't
> see the "bad udp cksum" scream from tcpdump, which is weird...
>
> 17:03:08.765845 IP (tos 0x0, ttl 64, id 52396, offset 0, flags [DF], proto
> UDP (17), length 746)
> 192.168.31.17.56686 > 192.168.31.18.4789: UDP, length 718
> 0x0000: f452 1401 da82 0002 c9e9 bf32 0800 4500
> 0x0010: 02ea ccac 4000 4011 abe2 c0a8 1f11 c0a8
> 0x0020: 1f12 dd6e 12b5 02d6 0c1b 0800 0000 0000
> 0x0030: 6300 b2c7 81db e850 7a83 2ecb 8c68 0800
> 0x0040: 4500 02b8 6357 4000 4006 eb74 c0a8 3411
> 0x0050: c0a8 3412 86f2 3241 d67e f47d e5a9 d041
> 0x0060: 5018 0038 871c 0000 0000 0258 ffff ffff
> 0x0070: 0000 0000 0000 0000 0000
> 17:03:09.336285 IP (tos 0x0, ttl 64, id 52536, offset 0, flags [DF], proto
> UDP (17), length 90)
> 192.168.31.17.56686 > 192.168.31.18.4789: [bad udp cksum 1360!] UDP,
> length 62
> 0x0000: f452 1401 da82 0002 c9e9 bf32 0800 4500
> 0x0010: 005a cd38 4000 4011 ade6 c0a8 1f11 c0a8
> 0x0020: 1f12 dd6e 12b5 0046 139a 0800 0000 0000
> 0x0030: 6300 b2c7 81db e850 7a83 2ecb 8c68 0800
> 0x0040: 4500 0028 6358 4000 4006 ee03 c0a8 3411
> 0x0050: c0a8 3412 86f2 3241 d67e f70d e5a9 d041
> 0x0060: 5011 0038 897b 0000
> 17:03:10.335074 IP (tos 0x0, ttl 64, id 40045, offset 0, flags [DF], proto
> UDP (17), length 98)
> 192.168.31.18.48861 > 192.168.31.17.4789: [no cksum] UDP, length 70
> 0x0000: 0002 c9e9 bf32 f452 1401 da82 0800 4500
> 0x0010: 0062 9c6d 4000 4011 dea9 c0a8 1f12 c0a8
> 0x0020: 1f11 bedd 12b5 004e 0000 0800 0000 0000
> 0x0030: 6300 7a83 2ecb 8c68 b2c7 81db e850 0800
> 0x0040: 4500 0030 0000 4000 4006 5154 c0a8 3412
> 0x0050: c0a8 3411 3241 86f2 e5a9 d040 d67e f47d
> 0x0060: 7012 6e28 f282 0000 0204 0582 0103 0307
> 17:03:10.335110 IP (tos 0x0, ttl 64, id 52764, offset 0, flags [DF], proto
> UDP (17), length 90)
> 192.168.31.17.56686 > 192.168.31.18.4789: [bad udp cksum 1360!] UDP,
> length 62
> 0x0000: f452 1401 da82 0002 c9e9 bf32 0800 4500
> 0x0010: 005a ce1c 4000 4011 ad02 c0a8 1f11 c0a8
> 0x0020: 1f12 dd6e 12b5 0046 139a 0800 0000 0000
> 0x0030: 6300 b2c7 81db e850 7a83 2ecb 8c68 0800
> 0x0040: 4500 0028 6359 4000 4006 ee02 c0a8 3411
> 0x0050: c0a8 3412 86f2 3241 d67e f70e e5a9 d041
> 0x0060: 5010 0038 897b 0000
>
>
next prev parent reply other threads:[~2014-10-28 15:36 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-10-26 13:36 some failures with vxlan offloads Or Gerlitz
2014-10-26 15:29 ` Tom Herbert
2014-10-26 22:23 ` Or Gerlitz
2014-10-27 1:23 ` Tom Herbert
2014-10-28 15:27 ` Or Gerlitz
2014-10-28 15:36 ` Tom Herbert [this message]
2014-10-29 5:50 ` Or Gerlitz
2014-10-29 14:59 ` Tom Herbert
2014-10-29 15:56 ` Or Gerlitz
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CA+mtBx_ncYrp_fbp8ucXfBaCoPpv_X+TEsmofCnpzTyPPfLXnw@mail.gmail.com \
--to=therbert@google.com \
--cc=jeffrey.t.kirsher@intel.com \
--cc=john.r.fastabend@intel.com \
--cc=netdev@vger.kernel.org \
--cc=ogerlitz@mellanox.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.