RSS is not efficient when forwarding

RSS is not efficient when forwarding

[Flavio Leitner]

Hi,

I have a simple router setup which forwards traffic from
one ixgbe 82599ES to another ixgbe of the same model.

kernel: 3.16.0-rc2-00262-ga921e2a

  p2p1: 192.168.155.1/24 is the gateway of the LAN
  p2p2: 192.168.156.1/24 is the gateway of the other LAN

While the ARP is resolving, I can see the packets being spread
among all the 8 queues (8 online CPUs) available and that is fine.

However, as soon as the TCP traffic starts, all streams are
merged to rx-queue-0 which overwhelms one single CPU, so the
total throughput is about 4Gbits/sec.

I can see the driver sending different skb->hash for each stream,
so it can't be the NIC.

Also, if I run a local http on the router, the skb->hash pattern
doesn't change, but the workload is spread among all CPUs.

debug output while reproducing all the streams on rx-queue-0:
[...]
[11685.885093] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xC2AF4A27
[11685.891454] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0x8C5B749D
[11685.897820] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xA33BA6D5
[11690.845032] net_ratelimit: 3276406 callbacks suppressed
[...]

Any idea why all TCP streams are going to a single rx queue?

Thanks,
fbl

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Thu, Jul 03, 2014 at 07:44:08PM -0300, Flavio Leitner wrote:
> 
> Hi,
> 
> I have a simple router setup which forwards traffic from
> one ixgbe 82599ES to another ixgbe of the same model.
> 
> kernel: 3.16.0-rc2-00262-ga921e2a
> 
>   p2p1: 192.168.155.1/24 is the gateway of the LAN
>   p2p2: 192.168.156.1/24 is the gateway of the other LAN
> 
> While the ARP is resolving, I can see the packets being spread
> among all the 8 queues (8 online CPUs) available and that is fine.
> 
> However, as soon as the TCP traffic starts, all streams are
> merged to rx-queue-0 which overwhelms one single CPU, so the
> total throughput is about 4Gbits/sec.
> 
> I can see the driver sending different skb->hash for each stream,
> so it can't be the NIC.
> 
> Also, if I run a local http on the router, the skb->hash pattern
> doesn't change, but the workload is spread among all CPUs.
> 
> debug output while reproducing all the streams on rx-queue-0:
> [...]
> [11685.885093] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xC2AF4A27
> [11685.891454] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0x8C5B749D
> [11685.897820] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xA33BA6D5
> [11690.845032] net_ratelimit: 3276406 callbacks suppressed
> [...]

The command 'ehtool -S p2p1 | grep rx_queue' shows only rx queue #0
receiving packets.  Also, /proc/interrupts shows only p2p1-TxRx-0
generating interrupts.

Nothing changes when I start irqbalance in the middle of the test.
However, if I create new streams, they are correctly distributed
among the NIC queues/irqs/CPUs.

I have tried the same setup and test with another card (bnx2, 1GbE)
and all queues got traffic by default though they were all assigned
to CPU#0 (no irqbalance), so this is expected and makes me think
this is specific to the ixgbe driver.

Any ideas?

Thanks,
fbl

Re: RSS is not efficient when forwarding (ixgbe)

[Tom Herbert]

On Fri, Jul 4, 2014 at 12:36 PM, Flavio Leitner <fbl@redhat.com> wrote:
> On Thu, Jul 03, 2014 at 07:44:08PM -0300, Flavio Leitner wrote:
>>
>> Hi,
>>
>> I have a simple router setup which forwards traffic from
>> one ixgbe 82599ES to another ixgbe of the same model.
>>
>> kernel: 3.16.0-rc2-00262-ga921e2a
>>
>>   p2p1: 192.168.155.1/24 is the gateway of the LAN
>>   p2p2: 192.168.156.1/24 is the gateway of the other LAN
>>
>> While the ARP is resolving, I can see the packets being spread
>> among all the 8 queues (8 online CPUs) available and that is fine.
>>
>> However, as soon as the TCP traffic starts, all streams are
>> merged to rx-queue-0 which overwhelms one single CPU, so the
>> total throughput is about 4Gbits/sec.
>>
>> I can see the driver sending different skb->hash for each stream,
>> so it can't be the NIC.
>>
>> Also, if I run a local http on the router, the skb->hash pattern
>> doesn't change, but the workload is spread among all CPUs.
>>
>> debug output while reproducing all the streams on rx-queue-0:
>> [...]
>> [11685.885093] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xC2AF4A27
>> [11685.891454] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0x8C5B749D
>> [11685.897820] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xA33BA6D5
>> [11690.845032] net_ratelimit: 3276406 callbacks suppressed
>> [...]
>
> The command 'ehtool -S p2p1 | grep rx_queue' shows only rx queue #0
> receiving packets.  Also, /proc/interrupts shows only p2p1-TxRx-0
> generating interrupts.
>
> Nothing changes when I start irqbalance in the middle of the test.
> However, if I create new streams, they are correctly distributed
> among the NIC queues/irqs/CPUs.
>
Could this be interaction with flow director? Is the application
running on just one CPU?

> I have tried the same setup and test with another card (bnx2, 1GbE)
> and all queues got traffic by default though they were all assigned
> to CPU#0 (no irqbalance), so this is expected and makes me think
> this is specific to the ixgbe driver.
>
> Any ideas?
>
> Thanks,
> fbl
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Sat, Jul 05, 2014 at 09:09:34AM -0700, Tom Herbert wrote:
> On Fri, Jul 4, 2014 at 12:36 PM, Flavio Leitner <fbl@redhat.com> wrote:
> > On Thu, Jul 03, 2014 at 07:44:08PM -0300, Flavio Leitner wrote:
> >>
> >> Hi,
> >>
> >> I have a simple router setup which forwards traffic from
> >> one ixgbe 82599ES to another ixgbe of the same model.
> >>
> >> kernel: 3.16.0-rc2-00262-ga921e2a
> >>
> >>   p2p1: 192.168.155.1/24 is the gateway of the LAN
> >>   p2p2: 192.168.156.1/24 is the gateway of the other LAN
> >>
> >> While the ARP is resolving, I can see the packets being spread
> >> among all the 8 queues (8 online CPUs) available and that is fine.
> >>
> >> However, as soon as the TCP traffic starts, all streams are
> >> merged to rx-queue-0 which overwhelms one single CPU, so the
> >> total throughput is about 4Gbits/sec.
> >>
> >> I can see the driver sending different skb->hash for each stream,
> >> so it can't be the NIC.
> >>
> >> Also, if I run a local http on the router, the skb->hash pattern
> >> doesn't change, but the workload is spread among all CPUs.
> >>
> >> debug output while reproducing all the streams on rx-queue-0:
> >> [...]
> >> [11685.885093] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xC2AF4A27
> >> [11685.891454] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0x8C5B749D
> >> [11685.897820] ixgbe_rx_skb:1713 skb(ffff880222a77200) hash: 0xA33BA6D5
> >> [11690.845032] net_ratelimit: 3276406 callbacks suppressed
> >> [...]
> >
> > The command 'ehtool -S p2p1 | grep rx_queue' shows only rx queue #0
> > receiving packets.  Also, /proc/interrupts shows only p2p1-TxRx-0
> > generating interrupts.
> >
> > Nothing changes when I start irqbalance in the middle of the test.
> > However, if I create new streams, they are correctly distributed
> > among the NIC queues/irqs/CPUs.
> >
> Could this be interaction with flow director? Is the application
> running on just one CPU?

It's a router forwarding traffic from one interface to another, so
I guess it's only the kernel. BTW, no firewall.

Flow Director needs to be enabled and I am using defaults.
 
> > I have tried the same setup and test with another card (bnx2, 1GbE)
> > and all queues got traffic by default though they were all assigned
> > to CPU#0 (no irqbalance), so this is expected and makes me think
> > this is specific to the ixgbe driver.
> >
> > Any ideas?
> >
> > Thanks,
> > fbl
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

RE: RSS is not efficient when forwarding (ixgbe)

[Skidmore, Donald C]

> 
> It's a router forwarding traffic from one interface to another, so I guess it's
> only the kernel. BTW, no firewall.
> 
> Flow Director needs to be enabled and I am using defaults.

Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  

Thanks,
-Don Skidmore <donald.c.skidmore@intel.com>

> 
> > > I have tried the same setup and test with another card (bnx2, 1GbE)
> > > and all queues got traffic by default though they were all assigned
> > > to CPU#0 (no irqbalance), so this is expected and makes me think
> > > this is specific to the ixgbe driver.
> > >
> > > Any ideas?
> > >
> > > Thanks,
> > > fbl
> > > --
> > > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > > the body of a message to majordomo@vger.kernel.org More majordomo
> > > info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe netdev" in
> > the body of a message to majordomo@vger.kernel.org More majordomo
> info
> > at  http://vger.kernel.org/majordomo-info.html
> >
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in the body
> of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
> 
> 
> > 
> > It's a router forwarding traffic from one interface to another, so I guess it's
> > only the kernel. BTW, no firewall.
> > 
> > Flow Director needs to be enabled and I am using defaults.
> 
> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  

That was it.
Thanks a lot
fbl

Re: RSS is not efficient when forwarding (ixgbe)

[Carlos Carvalho]

Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
 >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
 >> 
 >> 
 >> > 
 >> > It's a router forwarding traffic from one interface to another, so I guess it's
 >> > only the kernel. BTW, no firewall.
 >> > 
 >> > Flow Director needs to be enabled and I am using defaults.
 >> 
 >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
 >
 >That was it.

We have a similar setup and similar problem. How do we disable ATR? I
tried to set ntuple off but this almost zeroed traffic. I also tried
to change rx-flow-hash but ethtool says it's not possible. The docs
say that one can disable ATR by setting AtrSampleRate to 0 but this
parameter doesn't exist in 3.14.10.

So, how do we disable ATR and keep RSS?

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
>  >> 
>  >> 
>  >> > 
>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
>  >> > only the kernel. BTW, no firewall.
>  >> > 
>  >> > Flow Director needs to be enabled and I am using defaults.
>  >> 
>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
>  >
>  >That was it.
> 
> We have a similar setup and similar problem. How do we disable ATR? I
> tried to set ntuple off but this almost zeroed traffic. I also tried
> to change rx-flow-hash but ethtool says it's not possible. The docs
> say that one can disable ATR by setting AtrSampleRate to 0 but this
> parameter doesn't exist in 3.14.10.
> 
> So, how do we disable ATR and keep RSS?
 
Keep in mind that this is actually 2 problems. One is enabling the
NIC to receive the streams in all queues for this scenario (setting
ntuple off and restarting the traffic works for me). The second problem
is having all the queue interrupts spread among the CPUs. That's what
does irqbalance, tuna, etc...

fbl

Re: RSS is not efficient when forwarding (ixgbe)

[Carlos Carvalho]

Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
 >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
 >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
 >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
 >>  >> 
 >>  >> 
 >>  >> > 
 >>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
 >>  >> > only the kernel. BTW, no firewall.
 >>  >> > 
 >>  >> > Flow Director needs to be enabled and I am using defaults.
 >>  >> 
 >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
 >>  >
 >>  >That was it.
 >> 
 >> We have a similar setup and similar problem. How do we disable ATR? I
 >> tried to set ntuple off but this almost zeroed traffic. I also tried
 >> to change rx-flow-hash but ethtool says it's not possible. The docs
 >> say that one can disable ATR by setting AtrSampleRate to 0 but this
 >> parameter doesn't exist in 3.14.10.
 >> 
 >> So, how do we disable ATR and keep RSS?
 > 
 >Keep in mind that this is actually 2 problems. One is enabling the
 >NIC to receive the streams in all queues for this scenario (setting
 >ntuple off and restarting the traffic works for me). The second problem
 >is having all the queue interrupts spread among the CPUs. That's what
 >does irqbalance, tuna, etc...

Spreading the interrupts among the cpus is not the issue for us. The
problem is that the number of interrupts is *very* different among the
irq's, so no matter how I distribute them among cores there will
always be a few that get saturated while 70% of the machine capacity
remains idle. Your case seems to be the extreme of ours, where all
the flux goes to a single irq.

The problem is in how the NIC distributes traffic among the irq's in
the router. Traffic comes almost only from a single machine and
spreads through several thousand destinations in the internet. That's
why I tried to set the receiving hash mode to the destination IP, but
the NIC or driver refuses. So how do I even out the frequency of irq's?

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Tue, Jul 08, 2014 at 02:32:43PM -0300, Carlos Carvalho wrote:
> Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
>  >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
>  >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
>  >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
>  >>  >> 
>  >>  >> 
>  >>  >> > 
>  >>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
>  >>  >> > only the kernel. BTW, no firewall.
>  >>  >> > 
>  >>  >> > Flow Director needs to be enabled and I am using defaults.
>  >>  >> 
>  >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
>  >>  >
>  >>  >That was it.
>  >> 
>  >> We have a similar setup and similar problem. How do we disable ATR? I
>  >> tried to set ntuple off but this almost zeroed traffic. I also tried
>  >> to change rx-flow-hash but ethtool says it's not possible. The docs
>  >> say that one can disable ATR by setting AtrSampleRate to 0 but this
>  >> parameter doesn't exist in 3.14.10.
>  >> 
>  >> So, how do we disable ATR and keep RSS?
>  > 
>  >Keep in mind that this is actually 2 problems. One is enabling the
>  >NIC to receive the streams in all queues for this scenario (setting
>  >ntuple off and restarting the traffic works for me). The second problem
>  >is having all the queue interrupts spread among the CPUs. That's what
>  >does irqbalance, tuna, etc...
> 
> Spreading the interrupts among the cpus is not the issue for us. The
> problem is that the number of interrupts is *very* different among the
> irq's, so no matter how I distribute them among cores there will
> always be a few that get saturated while 70% of the machine capacity
> remains idle. Your case seems to be the extreme of ours, where all
> the flux goes to a single irq.
> 
> The problem is in how the NIC distributes traffic among the irq's in
> the router. Traffic comes almost only from a single machine and
> spreads through several thousand destinations in the internet. That's
> why I tried to set the receiving hash mode to the destination IP, but
> the NIC or driver refuses. So how do I even out the frequency of irq's?

So you see the traffic going to a few queues only and the rest is
idle, is that correct?  If so, then RSS seems to be working, but
since all the traffic comes from one server and likely one port,
maybe the hash is not good enough to distribute among all queues.
I'd try using software hashing instead of hw hashing to see if it
helps:
# ethtool -K <iface> rxhash off

or using Perfect Filters, but then it depends on the traffic's specifics.

BTW, there was a typo in my previous post, I had to turn on ntuple to
disable ATR.

fbl

Re: RSS is not efficient when forwarding (ixgbe)

[Carlos Carvalho]

Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 02:22:
 >On Tue, Jul 08, 2014 at 02:32:43PM -0300, Carlos Carvalho wrote:
 >> Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
 >>  >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
 >>  >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
 >>  >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
 >>  >>  >> 
 >>  >>  >> 
 >>  >>  >> > 
 >>  >>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
 >>  >>  >> > only the kernel. BTW, no firewall.
 >>  >>  >> > 
 >>  >>  >> > Flow Director needs to be enabled and I am using defaults.
 >>  >>  >> 
 >>  >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
 >>  >>  >
 >>  >>  >That was it.
 >>  >> 
 >>  >> We have a similar setup and similar problem. How do we disable ATR? I
 >>  >> tried to set ntuple off but this almost zeroed traffic. I also tried
 >>  >> to change rx-flow-hash but ethtool says it's not possible. The docs
 >>  >> say that one can disable ATR by setting AtrSampleRate to 0 but this
 >>  >> parameter doesn't exist in 3.14.10.
 >>  >> 
 >>  >> So, how do we disable ATR and keep RSS?
 >>  > 
 >>  >Keep in mind that this is actually 2 problems. One is enabling the
 >>  >NIC to receive the streams in all queues for this scenario (setting
 >>  >ntuple off and restarting the traffic works for me). The second problem
 >>  >is having all the queue interrupts spread among the CPUs. That's what
 >>  >does irqbalance, tuna, etc...
 >> 
 >> Spreading the interrupts among the cpus is not the issue for us. The
 >> problem is that the number of interrupts is *very* different among the
 >> irq's, so no matter how I distribute them among cores there will
 >> always be a few that get saturated while 70% of the machine capacity
 >> remains idle. Your case seems to be the extreme of ours, where all
 >> the flux goes to a single irq.
 >> 
 >> The problem is in how the NIC distributes traffic among the irq's in
 >> the router. Traffic comes almost only from a single machine and
 >> spreads through several thousand destinations in the internet. That's
 >> why I tried to set the receiving hash mode to the destination IP, but
 >> the NIC or driver refuses. So how do I even out the frequency of irq's?
 >
 >So you see the traffic going to a few queues only and the rest is
 >idle, is that correct?  If so, then RSS seems to be working, but
 >since all the traffic comes from one server and likely one port,
 >maybe the hash is not good enough to distribute among all queues.
 >I'd try using software hashing instead of hw hashing to see if it
 >helps:
 ># ethtool -K <iface> rxhash off

I'm all for using software instead of hardware. However, in this case
this is a fundamental function of the NIC, to distribute the load
among cores; if we do it via software, a single core (or subset of
them) will have to do all the work. So in this particular case I think
the correct way is to try to do it in the NIC.

 >BTW, there was a typo in my previous post, I had to turn on ntuple to
 >disable ATR.

Ah. Here it was off by default, which contradicts what Donald said
above... I turned it on, and nothing changed(?!). The nic was reset
but the distribution among queues is the same.

I checked now and in fact the distribution is almost constant among
16 IRQ's. That's the problem, because it leaves the other 24
cores idle. Not completely, but the difference is 4 orders of
magnitude: 5.41e+08 interruptions in the active 16 IRQ's versus
6.36e+04 in the others. So 60% of the machine is just contributing to
global warming and, importantly, limiting our performance :-(

Is there a configuration that expands the used IRQ's?

Re: RSS is not efficient when forwarding (ixgbe)

[Flavio Leitner]

On Wed, Jul 09, 2014 at 09:08:27PM -0300, Carlos Carvalho wrote:
> Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 02:22:
>  >On Tue, Jul 08, 2014 at 02:32:43PM -0300, Carlos Carvalho wrote:
>  >> Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
>  >>  >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
>  >>  >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
>  >>  >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
>  >>  >>  >> 
>  >>  >>  >> 
>  >>  >>  >> > 
>  >>  >>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
>  >>  >>  >> > only the kernel. BTW, no firewall.
>  >>  >>  >> > 
>  >>  >>  >> > Flow Director needs to be enabled and I am using defaults.
>  >>  >>  >> 
>  >>  >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
>  >>  >>  >
>  >>  >>  >That was it.
>  >>  >> 
>  >>  >> We have a similar setup and similar problem. How do we disable ATR? I
>  >>  >> tried to set ntuple off but this almost zeroed traffic. I also tried
>  >>  >> to change rx-flow-hash but ethtool says it's not possible. The docs
>  >>  >> say that one can disable ATR by setting AtrSampleRate to 0 but this
>  >>  >> parameter doesn't exist in 3.14.10.
>  >>  >> 
>  >>  >> So, how do we disable ATR and keep RSS?
>  >>  > 
>  >>  >Keep in mind that this is actually 2 problems. One is enabling the
>  >>  >NIC to receive the streams in all queues for this scenario (setting
>  >>  >ntuple off and restarting the traffic works for me). The second problem
>  >>  >is having all the queue interrupts spread among the CPUs. That's what
>  >>  >does irqbalance, tuna, etc...
>  >> 
>  >> Spreading the interrupts among the cpus is not the issue for us. The
>  >> problem is that the number of interrupts is *very* different among the
>  >> irq's, so no matter how I distribute them among cores there will
>  >> always be a few that get saturated while 70% of the machine capacity
>  >> remains idle. Your case seems to be the extreme of ours, where all
>  >> the flux goes to a single irq.
>  >> 
>  >> The problem is in how the NIC distributes traffic among the irq's in
>  >> the router. Traffic comes almost only from a single machine and
>  >> spreads through several thousand destinations in the internet. That's
>  >> why I tried to set the receiving hash mode to the destination IP, but
>  >> the NIC or driver refuses. So how do I even out the frequency of irq's?
>  >
>  >So you see the traffic going to a few queues only and the rest is
>  >idle, is that correct?  If so, then RSS seems to be working, but
>  >since all the traffic comes from one server and likely one port,
>  >maybe the hash is not good enough to distribute among all queues.
>  >I'd try using software hashing instead of hw hashing to see if it
>  >helps:
>  ># ethtool -K <iface> rxhash off
> 
> I'm all for using software instead of hardware. However, in this case
> this is a fundamental function of the NIC, to distribute the load
> among cores; if we do it via software, a single core (or subset of
> them) will have to do all the work. So in this particular case I think
> the correct way is to try to do it in the NIC.

Actually no, you can use RPS with software hashing to distribute the
workload.  Take a look at Documentation/networking/scaling.txt for
more details.
 
>  >BTW, there was a typo in my previous post, I had to turn on ntuple to
>  >disable ATR.
> 
> Ah. Here it was off by default, which contradicts what Donald said
> above... I turned it on, and nothing changed(?!). The nic was reset
> but the distribution among queues is the same.

ntuple is about Perfect Filters and it's OFF by default which leaves
ATR mode ON. In my case, ATR mode was responsible for directing all
the packets to a single queue. Once I enable nutple, the driver
disables ATR and that works for me.

> I checked now and in fact the distribution is almost constant among
> 16 IRQ's. That's the problem, because it leaves the other 24
> cores idle. Not completely, but the difference is 4 orders of
> magnitude: 5.41e+08 interruptions in the active 16 IRQ's versus
> 6.36e+04 in the others. So 60% of the machine is just contributing to
> global warming and, importantly, limiting our performance :-(

How many NIC queues do you have?  It sounds like you have only 16,
so you're limited by the number of queues which maps to 16 CPUs.
That's one case where RPS can be beneficial.
 
fbl
 

Re: RSS is not efficient when forwarding (ixgbe)

[Carlos Carvalho]

Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 22:14:
 >On Wed, Jul 09, 2014 at 09:08:27PM -0300, Carlos Carvalho wrote:
 >> Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 02:22:
 >>  >On Tue, Jul 08, 2014 at 02:32:43PM -0300, Carlos Carvalho wrote:
 >>  >> Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
 >>  >>  >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
 >>  >>  >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
 >>  >>  >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C wrote:
 >>  >>  >>  >> 
 >>  >>  >>  >> 
 >>  >>  >>  >> > 
 >>  >>  >>  >> > It's a router forwarding traffic from one interface to another, so I guess it's
 >>  >>  >>  >> > only the kernel. BTW, no firewall.
 >>  >>  >>  >> > 
 >>  >>  >>  >> > Flow Director needs to be enabled and I am using defaults.
 >>  >>  >>  >> 
 >>  >>  >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like Tom mentioned the driver will create hash buckets for egress packets.  You could try disabling ATR and just use RSS.  Which would probably be the right thing to do any way since Flow Director isn't very useful for routing scenarios.  
 >>  >>  >>  >
 >>  >>  >>  >That was it.
 >>  >>  >> 
 >>  >>  >> We have a similar setup and similar problem. How do we disable ATR? I
 >>  >>  >> tried to set ntuple off but this almost zeroed traffic. I also tried
 >>  >>  >> to change rx-flow-hash but ethtool says it's not possible. The docs
 >>  >>  >> say that one can disable ATR by setting AtrSampleRate to 0 but this
 >>  >>  >> parameter doesn't exist in 3.14.10.
 >>  >>  >> 
 >>  >>  >> So, how do we disable ATR and keep RSS?
 >>  >>  > 
 >>  >>  >Keep in mind that this is actually 2 problems. One is enabling the
 >>  >>  >NIC to receive the streams in all queues for this scenario (setting
 >>  >>  >ntuple off and restarting the traffic works for me). The second problem
 >>  >>  >is having all the queue interrupts spread among the CPUs. That's what
 >>  >>  >does irqbalance, tuna, etc...
 >>  >> 
 >>  >> Spreading the interrupts among the cpus is not the issue for us. The
 >>  >> problem is that the number of interrupts is *very* different among the
 >>  >> irq's, so no matter how I distribute them among cores there will
 >>  >> always be a few that get saturated while 70% of the machine capacity
 >>  >> remains idle. Your case seems to be the extreme of ours, where all
 >>  >> the flux goes to a single irq.
 >>  >> 
 >>  >> The problem is in how the NIC distributes traffic among the irq's in
 >>  >> the router. Traffic comes almost only from a single machine and
 >>  >> spreads through several thousand destinations in the internet. That's
 >>  >> why I tried to set the receiving hash mode to the destination IP, but
 >>  >> the NIC or driver refuses. So how do I even out the frequency of irq's?
 >>  >
 >>  >So you see the traffic going to a few queues only and the rest is
 >>  >idle, is that correct?  If so, then RSS seems to be working, but
 >>  >since all the traffic comes from one server and likely one port,
 >>  >maybe the hash is not good enough to distribute among all queues.
 >>  >I'd try using software hashing instead of hw hashing to see if it
 >>  >helps:
 >>  ># ethtool -K <iface> rxhash off
 >> 
 >> I'm all for using software instead of hardware. However, in this case
 >> this is a fundamental function of the NIC, to distribute the load
 >> among cores; if we do it via software, a single core (or subset of
 >> them) will have to do all the work. So in this particular case I think
 >> the correct way is to try to do it in the NIC.
 >
 >Actually no, you can use RPS with software hashing to distribute the
 >workload.  Take a look at Documentation/networking/scaling.txt for
 >more details.

Thanks for the pointer. I'll try that if there's no way to do it on
the NIC.

 >>  >BTW, there was a typo in my previous post, I had to turn on ntuple to
 >>  >disable ATR.
 >> 
 >> Ah. Here it was off by default, which contradicts what Donald said
 >> above... I turned it on, and nothing changed(?!). The nic was reset
 >> but the distribution among queues is the same.
 >
 >ntuple is about Perfect Filters and it's OFF by default which leaves
 >ATR mode ON. In my case, ATR mode was responsible for directing all
 >the packets to a single queue. Once I enable nutple, the driver
 >disables ATR and that works for me.

There does seem to be some confusion about this. However this thread
is getting long and since it makes no difference for us I'd rather
focus on our problem.

 >> I checked now and in fact the distribution is almost constant among
 >> 16 IRQ's. That's the problem, because it leaves the other 24
 >> cores idle. Not completely, but the difference is 4 orders of
 >> magnitude: 5.41e+08 interruptions in the active 16 IRQ's versus
 >> 6.36e+04 in the others. So 60% of the machine is just contributing to
 >> global warming and, importantly, limiting our performance :-(
 >
 >How many NIC queues do you have?  It sounds like you have only 16,
 >so you're limited by the number of queues which maps to 16 CPUs.

40 for each NIC. The driver seems to configure the number of queues
equal to the number of cores; in another machine with the same NIC but
32 cores there are 32 queues.

Does hyperthreading make a difference? There are actually only 20
cores. It seems that hyperthreading doesn't help for NIC interrrupt
processing, so the card/kernel could just be ignoring the virtual
cores. However, in this case it should not allocate queues for the
virtual cores. Also, the number of interruptions in the virtual cores
should be zero, but it isn't.

Further, why doesn't the NIC use all of the 20 real cores? Is it
limited to power of 2 cores?

RE: RSS is not efficient when forwarding (ixgbe)

[Skidmore, Donald C]

> -----Original Message-----
> From: Carlos Carvalho [mailto:carlos@fisica.ufpr.br]
> Sent: Thursday, July 10, 2014 5:12 PM
> To: Flavio Leitner
> Cc: Skidmore, Donald C; Tom Herbert; Linux Netdev List
> Subject: Re: RSS is not efficient when forwarding (ixgbe)
> 
> Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 22:14:
>  >On Wed, Jul 09, 2014 at 09:08:27PM -0300, Carlos Carvalho wrote:
>  >> Flavio Leitner (fbl@redhat.com) wrote on 9 July 2014 02:22:
>  >>  >On Tue, Jul 08, 2014 at 02:32:43PM -0300, Carlos Carvalho wrote:
>  >>  >> Flavio Leitner (fbl@redhat.com) wrote on 8 July 2014 14:21:
>  >>  >>  >On Tue, Jul 08, 2014 at 02:09:13PM -0300, Carlos Carvalho wrote:
>  >>  >>  >> Flavio Leitner (fbl@redhat.com) wrote on 7 July 2014 21:28:
>  >>  >>  >>  >On Mon, Jul 07, 2014 at 04:33:24PM +0000, Skidmore, Donald C
> wrote:
>  >>  >>  >>  >>
>  >>  >>  >>  >>
>  >>  >>  >>  >> >
>  >>  >>  >>  >> > It's a router forwarding traffic from one interface to another,
> so I guess it's  >>  >>  >>  >> > only the kernel. BTW, no firewall.
>  >>  >>  >>  >> >
>  >>  >>  >>  >> > Flow Director needs to be enabled and I am using defaults.
>  >>  >>  >>  >>
>  >>  >>  >>  >> Flow Director in ATR mode is on by default for ixgbe.  So like
> Tom mentioned the driver will create hash buckets for egress packets.  You
> could try disabling ATR and just use RSS.  Which would probably be the right
> thing to do any way since Flow Director isn't very useful for routing scenarios.
>  >>  >>  >>  >
>  >>  >>  >>  >That was it.
>  >>  >>  >>
>  >>  >>  >> We have a similar setup and similar problem. How do we disable
> ATR? I  >>  >>  >> tried to set ntuple off but this almost zeroed traffic. I also
> tried  >>  >>  >> to change rx-flow-hash but ethtool says it's not possible. The
> docs  >>  >>  >> say that one can disable ATR by setting AtrSampleRate to 0
> but this  >>  >>  >> parameter doesn't exist in 3.14.10.
>  >>  >>  >>
>  >>  >>  >> So, how do we disable ATR and keep RSS?
>  >>  >>  >
>  >>  >>  >Keep in mind that this is actually 2 problems. One is enabling the  >>
> >>  >NIC to receive the streams in all queues for this scenario (setting  >>  >>
> >ntuple off and restarting the traffic works for me). The second problem  >>
> >>  >is having all the queue interrupts spread among the CPUs. That's what
> >>  >>  >does irqbalance, tuna, etc...
>  >>  >>
>  >>  >> Spreading the interrupts among the cpus is not the issue for us. The
> >>  >> problem is that the number of interrupts is *very* different among
> the  >>  >> irq's, so no matter how I distribute them among cores there will
> >>  >> always be a few that get saturated while 70% of the machine capacity
> >>  >> remains idle. Your case seems to be the extreme of ours, where all  >>
> >> the flux goes to a single irq.
>  >>  >>
>  >>  >> The problem is in how the NIC distributes traffic among the irq's in  >>
> >> the router. Traffic comes almost only from a single machine and  >>  >>
> spreads through several thousand destinations in the internet. That's  >>  >>
> why I tried to set the receiving hash mode to the destination IP, but  >>  >>
> the NIC or driver refuses. So how do I even out the frequency of irq's?
>  >>  >
>  >>  >So you see the traffic going to a few queues only and the rest is  >>
> >idle, is that correct?  If so, then RSS seems to be working, but  >>  >since all
> the traffic comes from one server and likely one port,  >>  >maybe the hash
> is not good enough to distribute among all queues.
>  >>  >I'd try using software hashing instead of hw hashing to see if it  >>
> >helps:
>  >>  ># ethtool -K <iface> rxhash off
>  >>
>  >> I'm all for using software instead of hardware. However, in this case  >>
> this is a fundamental function of the NIC, to distribute the load  >> among
> cores; if we do it via software, a single core (or subset of  >> them) will have
> to do all the work. So in this particular case I think  >> the correct way is to try
> to do it in the NIC.
>  >
>  >Actually no, you can use RPS with software hashing to distribute the
> >workload.  Take a look at Documentation/networking/scaling.txt for  >more
> details.
> 
> Thanks for the pointer. I'll try that if there's no way to do it on the NIC.
> 
>  >>  >BTW, there was a typo in my previous post, I had to turn on ntuple to
> >>  >disable ATR.
>  >>
>  >> Ah. Here it was off by default, which contradicts what Donald said  >>
> above... I turned it on, and nothing changed(?!). The nic was reset  >> but
> the distribution among queues is the same.
>  >
>  >ntuple is about Perfect Filters and it's OFF by default which leaves  >ATR
> mode ON. In my case, ATR mode was responsible for directing all  >the
> packets to a single queue. Once I enable nutple, the driver  >disables ATR
> and that works for me.
> 
> There does seem to be some confusion about this. However this thread is
> getting long and since it makes no difference for us I'd rather focus on our
> problem.
> 
>  >> I checked now and in fact the distribution is almost constant among  >> 16
> IRQ's. That's the problem, because it leaves the other 24  >> cores idle. Not
> completely, but the difference is 4 orders of  >> magnitude: 5.41e+08
> interruptions in the active 16 IRQ's versus  >> 6.36e+04 in the others. So 60%
> of the machine is just contributing to  >> global warming and, importantly,
> limiting our performance :-(  >  >How many NIC queues do you have?  It
> sounds like you have only 16,  >so you're limited by the number of queues
> which maps to 16 CPUs.
> 
> 40 for each NIC. The driver seems to configure the number of queues equal
> to the number of cores; in another machine with the same NIC but
> 32 cores there are 32 queues.
> 
> Does hyperthreading make a difference? There are actually only 20 cores. It
> seems that hyperthreading doesn't help for NIC interrrupt processing, so the
> card/kernel could just be ignoring the virtual cores. However, in this case it
> should not allocate queues for the virtual cores. Also, the number of
> interruptions in the virtual cores should be zero, but it isn't.
> 
> Further, why doesn't the NIC use all of the 20 real cores? Is it limited to
> power of 2 cores?

Hey Guys,

This is getting a bit difficult to read so I will just reply down here. :)

The ixgbe driver by default runs with ATR and RSS enabled.   ATR is a form of flow director the other mode being perfect filter which is enabled, as mentioned above, by ntuple.  ATR stands for application targeted receive, which mean we (the driver) create a hash bucket on ever SYN packet transmitted and returning packets of that flow will be scheduled to the same queue.  Now since that packet could be created on any core we need to have one queue per core (including hyper threads).  Any received packet that does not match an ATR hash bucket is then scheduled via RSS.

I believe you mentioned you were doing forwarding traffic, as you can imagine ATR is not that useful for traffic not tied to user space.  As others suggested you could just turn off ATR and then all packets would be assigned a queue via RSS.  Our RSS hash produces an output that is 4bits long, so the most we can spread out the load is over 16 queues.  This is why you only see 16 interrupts in use if you only use RSS. 

If you wanted to use all the queue and thus all the cores you could create your own filtering rules with perfect filter.  This may be tricky as you would have to have a pretty good understanding of your flow patterns to balance the load evenly.

Thanks,
-Don Skidmore <donald.c.skidmore@intel.com>