From: Alexander Duyck <alexander.duyck@gmail.com>
To: Lennart Sorensen <lsorense@csclub.uwaterloo.ca>
Cc: LKML <linux-kernel@vger.kernel.org>,
Netdev <netdev@vger.kernel.org>,
intel-wired-lan <intel-wired-lan@lists.osuosl.org>
Subject: Re: [Intel-wired-lan] i40e X722 RSS problem with NAT-Traversal IPsec packets
Date: Fri, 3 May 2019 10:19:47 -0700 [thread overview]
Message-ID: <CAKgT0UcV2wCr6iUYktZ+Bju_GNpXKzR=M+NLfKhUsw4bsJSiyA@mail.gmail.com> (raw)
In-Reply-To: <20190503151421.akvmu77lghxcouni@csclub.uwaterloo.ca>
On Fri, May 3, 2019 at 8:14 AM Lennart Sorensen
<lsorense@csclub.uwaterloo.ca> wrote:
>
> On Thu, May 02, 2019 at 01:59:46PM -0700, Alexander Duyck wrote:
> > If I recall correctly RSS is only using something like the lower 9
> > bits (indirection table size of 512) of the resultant hash on the
> > X722, even fewer if you have fewer queues that are a power of 2 and
> > happen to program the indirection table in a round robin fashion. So
> > for example on my system setup with 32 queues it is technically only
> > using the lower 5 bits of the hash.
> >
> > One issue as a result of that is that you can end up with swaths of
> > bits that don't really seem to impact the hash all that much since it
> > will never actually change those bits of the resultant hash. In order
> > to guarantee that every bit in the input impacts the hash you have to
> > make certain you have to gaps in the key wider than the bits you
> > examine in the final hash.
> >
> > A quick and dirty way to verify that the hash key is part of the issue
> > would be to use something like a simple repeating value such as AA:55
> > as your hash key. With something like that every bit you change in the
> > UDP port number should result in a change in the final RSS hash for
> > queue counts of 3 or greater. The downside is the upper 16 bits of the
> > hash are identical to the lower 16 so the actual hash value itself
> > isn't as useful.
>
> OK I set the hkey to
> aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55:aa:55
> and still only see queue 0 and 2 getting hit with a couple of dozen
> different UDP port numbers I picked. Changing the hash with ethtool to
> that didn't even move where the tcp packets for my ssh connection are
> going (they are always on queue 2 it seems).
The TCP flow could be bypassing RSS and may be using ATR to decide
where the Rx packets are processed. Now that I think about it there is
a possibility that ATR could be interfering with the queue selection.
You might try disabling it by running:
ethtool --set-priv-flags <iface> flow-director-atr off
> Does it just not hash UDP packets correctly? Is it even doing RSS?
> (the register I checked claimed it is).
The problem is RSS can be bypassed for queue selection by things like
ATR which I called out above. One possibility is that if the
encryption you were using was leaving the skb->encapsulation flag set,
and the NIC might have misidentified the packets as something it could
parse and set up a bunch of rules that were rerouting incoming traffic
based on outgoing traffic. Disabling the feature should switch off
that behavior if that is in fact the case.
> This system has 40 queues assigned by default since that is how many
> CPUs there are. Changing it to a lower number didn't make a difference
> (I tried 32 and 8).
You are probably fine using 40 queues. That isn't an even power of two
so it would actually improve the entropy a bit since the lower bits
don't have a many:1 mapping to queues.
next prev parent reply other threads:[~2019-05-03 17:20 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-01 20:52 i40e X722 RSS problem with NAT-Traversal IPsec packets Lennart Sorensen
2019-05-01 22:52 ` [Intel-wired-lan] " Alexander Duyck
2019-05-02 15:11 ` Lennart Sorensen
2019-05-02 17:03 ` Alexander Duyck
2019-05-02 17:16 ` Lennart Sorensen
2019-05-02 17:28 ` Alexander Duyck
2019-05-02 17:55 ` Lennart Sorensen
2019-05-02 18:52 ` Lennart Sorensen
2019-05-02 20:59 ` Alexander Duyck
2019-05-03 15:14 ` Lennart Sorensen
2019-05-03 17:19 ` Alexander Duyck [this message]
2019-05-03 20:59 ` Lennart Sorensen
2019-05-13 16:55 ` Lennart Sorensen
2019-05-13 19:04 ` Alexander Duyck
2019-05-14 16:34 ` Lennart Sorensen
2019-05-16 17:10 ` Alexander Duyck
2019-05-16 18:34 ` Lennart Sorensen
2019-05-16 18:37 ` Lennart Sorensen
2019-05-16 23:32 ` Alexander Duyck
2019-05-17 16:42 ` Alexander Duyck
2019-05-17 17:23 ` Lennart Sorensen
2019-05-17 22:20 ` Alexander Duyck
2019-05-21 15:15 ` Lennart Sorensen
2019-05-21 16:51 ` Alexander Duyck
2019-05-21 17:54 ` Lennart Sorensen
2019-05-21 23:22 ` Alexander Duyck
2019-05-22 14:39 ` Lennart Sorensen
2019-06-07 14:39 ` Lennart Sorensen
2019-06-07 19:32 ` Alexander Duyck
2019-06-07 20:49 ` [E1000-devel] " Hisashi T Fujinaka
2019-06-07 22:08 ` Fujinaka, Todd
2019-06-10 19:01 ` Lennart Sorensen
2020-02-07 21:51 ` Lennart Sorensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAKgT0UcV2wCr6iUYktZ+Bju_GNpXKzR=M+NLfKhUsw4bsJSiyA@mail.gmail.com' \
--to=alexander.duyck@gmail.com \
--cc=intel-wired-lan@lists.osuosl.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lsorense@csclub.uwaterloo.ca \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).