* Performance issue with igb with lots of different src-ip addrs.
@ 2017-03-16 22:49 Ben Greear
2017-03-17 2:35 ` Alexander Duyck
0 siblings, 1 reply; 6+ messages in thread
From: Ben Greear @ 2017-03-16 22:49 UTC (permalink / raw)
To: e1000-devel list, netdev
Hello,
We notice that when using two igb ports as a bridge, if we use pktgen to
drive traffic through the bridge and randomize (or use a very large range)
for the source IP addr in pktgen, then performance of igb is very poor (like 150Mbps
throughput instead of 1Gbps). It runs right at line speed if we use same src/dest
IP addr in pktgen. So, seems it is related to lots of src/dest IP addresses.
We see same problem when using pktgen to send to itself, and we see this in
several different kernels. We specifically tested bridge mode in this stock
Fedora kernel:
Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
e1000e does not show this problem in our testing.
Any ideas what the issue might be and how to fix it?
Thanks,
Ben
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Performance issue with igb with lots of different src-ip addrs.
2017-03-16 22:49 Performance issue with igb with lots of different src-ip addrs Ben Greear
@ 2017-03-17 2:35 ` Alexander Duyck
2017-03-17 2:46 ` Ben Greear
0 siblings, 1 reply; 6+ messages in thread
From: Alexander Duyck @ 2017-03-17 2:35 UTC (permalink / raw)
To: Ben Greear; +Cc: e1000-devel list, netdev
Can you include the pktgen script you are running?
Also when you say you are driving traffic through the bridge are you
sending from something external on the system or are you actually
directing the traffic from pktgen into the bridge directly?
- Alex
On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@candelatech.com> wrote:
> Hello,
>
> We notice that when using two igb ports as a bridge, if we use pktgen to
> drive traffic through the bridge and randomize (or use a very large range)
> for the source IP addr in pktgen, then performance of igb is very poor (like
> 150Mbps
> throughput instead of 1Gbps). It runs right at line speed if we use same
> src/dest
> IP addr in pktgen. So, seems it is related to lots of src/dest IP
> addresses.
>
> We see same problem when using pktgen to send to itself, and we see this in
> several different kernels. We specifically tested bridge mode in this stock
> Fedora kernel:
>
> Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC 2017
> x86_64 x86_64 x86_64 GNU/Linux
>
> e1000e does not show this problem in our testing.
>
> Any ideas what the issue might be and how to fix it?
>
> Thanks,
> Ben
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc http://www.candelatech.com
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Performance issue with igb with lots of different src-ip addrs.
2017-03-17 2:35 ` Alexander Duyck
@ 2017-03-17 2:46 ` Ben Greear
2017-03-17 3:05 ` Alexander Duyck
0 siblings, 1 reply; 6+ messages in thread
From: Ben Greear @ 2017-03-17 2:46 UTC (permalink / raw)
To: Alexander Duyck; +Cc: e1000-devel list, netdev
I'm actually using a hacked up version of pktgen nicely driven by our
GUI tool, but the crux is that you need to set min and max src IP to some large
range.
We are driving pktgen from a separate machine. Stock pktgen isn't good at reporting
received pkts last I checked, so it may be more difficult to easily view the problem.
I'll be happy to set up my tool on your Fedora 24 or similar VM or machine if you
want.
Thanks,
Ben
On 03/16/2017 07:35 PM, Alexander Duyck wrote:
> Can you include the pktgen script you are running?
>
> Also when you say you are driving traffic through the bridge are you
> sending from something external on the system or are you actually
> directing the traffic from pktgen into the bridge directly?
>
> - Alex
>
> On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@candelatech.com> wrote:
>> Hello,
>>
>> We notice that when using two igb ports as a bridge, if we use pktgen to
>> drive traffic through the bridge and randomize (or use a very large range)
>> for the source IP addr in pktgen, then performance of igb is very poor (like
>> 150Mbps
>> throughput instead of 1Gbps). It runs right at line speed if we use same
>> src/dest
>> IP addr in pktgen. So, seems it is related to lots of src/dest IP
>> addresses.
>>
>> We see same problem when using pktgen to send to itself, and we see this in
>> several different kernels. We specifically tested bridge mode in this stock
>> Fedora kernel:
>>
>> Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC 2017
>> x86_64 x86_64 x86_64 GNU/Linux
>>
>> e1000e does not show this problem in our testing.
>>
>> Any ideas what the issue might be and how to fix it?
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc http://www.candelatech.com
>>
>
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Performance issue with igb with lots of different src-ip addrs.
2017-03-17 2:46 ` Ben Greear
@ 2017-03-17 3:05 ` Alexander Duyck
2017-03-17 3:51 ` Ben Greear
0 siblings, 1 reply; 6+ messages in thread
From: Alexander Duyck @ 2017-03-17 3:05 UTC (permalink / raw)
To: Ben Greear; +Cc: e1000-devel list, netdev
I'm not really interested in installing a custom version of pktgen.
Any chance you can recreate the issue with standard pktgen?
You might try running perf to get a snapshot of what is using CPU time
on the system. It will probably give you a pretty good idea where the
code is that is eating up all your CPU time.
- Alex
On Thu, Mar 16, 2017 at 7:46 PM, Ben Greear <greearb@candelatech.com> wrote:
> I'm actually using a hacked up version of pktgen nicely driven by our
> GUI tool, but the crux is that you need to set min and max src IP to some
> large
> range.
>
> We are driving pktgen from a separate machine. Stock pktgen isn't good at
> reporting
> received pkts last I checked, so it may be more difficult to easily view the
> problem.
>
> I'll be happy to set up my tool on your Fedora 24 or similar VM or machine
> if you
> want.
>
> Thanks,
> Ben
>
>
> On 03/16/2017 07:35 PM, Alexander Duyck wrote:
>>
>> Can you include the pktgen script you are running?
>>
>> Also when you say you are driving traffic through the bridge are you
>> sending from something external on the system or are you actually
>> directing the traffic from pktgen into the bridge directly?
>>
>> - Alex
>>
>> On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@candelatech.com>
>> wrote:
>>>
>>> Hello,
>>>
>>> We notice that when using two igb ports as a bridge, if we use pktgen to
>>> drive traffic through the bridge and randomize (or use a very large
>>> range)
>>> for the source IP addr in pktgen, then performance of igb is very poor
>>> (like
>>> 150Mbps
>>> throughput instead of 1Gbps). It runs right at line speed if we use same
>>> src/dest
>>> IP addr in pktgen. So, seems it is related to lots of src/dest IP
>>> addresses.
>>>
>>> We see same problem when using pktgen to send to itself, and we see this
>>> in
>>> several different kernels. We specifically tested bridge mode in this
>>> stock
>>> Fedora kernel:
>>>
>>> Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC
>>> 2017
>>> x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> e1000e does not show this problem in our testing.
>>>
>>> Any ideas what the issue might be and how to fix it?
>>>
>>> Thanks,
>>> Ben
>>>
>>> --
>>> Ben Greear <greearb@candelatech.com>
>>> Candela Technologies Inc http://www.candelatech.com
>>>
>>
>
> --
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc http://www.candelatech.com
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Performance issue with igb with lots of different src-ip addrs.
2017-03-17 3:05 ` Alexander Duyck
@ 2017-03-17 3:51 ` Ben Greear
2017-03-24 20:54 ` Ben Greear
0 siblings, 1 reply; 6+ messages in thread
From: Ben Greear @ 2017-03-17 3:51 UTC (permalink / raw)
To: Alexander Duyck; +Cc: e1000-devel list, netdev
I think we can, might take us a day or two to get time to do it.
Thanks,
Ben
On 03/16/2017 08:05 PM, Alexander Duyck wrote:
> I'm not really interested in installing a custom version of pktgen.
> Any chance you can recreate the issue with standard pktgen?
>
> You might try running perf to get a snapshot of what is using CPU time
> on the system. It will probably give you a pretty good idea where the
> code is that is eating up all your CPU time.
>
> - Alex
>
> On Thu, Mar 16, 2017 at 7:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>> I'm actually using a hacked up version of pktgen nicely driven by our
>> GUI tool, but the crux is that you need to set min and max src IP to some
>> large
>> range.
>>
>> We are driving pktgen from a separate machine. Stock pktgen isn't good at
>> reporting
>> received pkts last I checked, so it may be more difficult to easily view the
>> problem.
>>
>> I'll be happy to set up my tool on your Fedora 24 or similar VM or machine
>> if you
>> want.
>>
>> Thanks,
>> Ben
>>
>>
>> On 03/16/2017 07:35 PM, Alexander Duyck wrote:
>>>
>>> Can you include the pktgen script you are running?
>>>
>>> Also when you say you are driving traffic through the bridge are you
>>> sending from something external on the system or are you actually
>>> directing the traffic from pktgen into the bridge directly?
>>>
>>> - Alex
>>>
>>> On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@candelatech.com>
>>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> We notice that when using two igb ports as a bridge, if we use pktgen to
>>>> drive traffic through the bridge and randomize (or use a very large
>>>> range)
>>>> for the source IP addr in pktgen, then performance of igb is very poor
>>>> (like
>>>> 150Mbps
>>>> throughput instead of 1Gbps). It runs right at line speed if we use same
>>>> src/dest
>>>> IP addr in pktgen. So, seems it is related to lots of src/dest IP
>>>> addresses.
>>>>
>>>> We see same problem when using pktgen to send to itself, and we see this
>>>> in
>>>> several different kernels. We specifically tested bridge mode in this
>>>> stock
>>>> Fedora kernel:
>>>>
>>>> Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC
>>>> 2017
>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> e1000e does not show this problem in our testing.
>>>>
>>>> Any ideas what the issue might be and how to fix it?
>>>>
>>>> Thanks,
>>>> Ben
>>>>
>>>> --
>>>> Ben Greear <greearb@candelatech.com>
>>>> Candela Technologies Inc http://www.candelatech.com
>>>>
>>>
>>
>> --
>> Ben Greear <greearb@candelatech.com>
>> Candela Technologies Inc http://www.candelatech.com
>
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Performance issue with igb with lots of different src-ip addrs.
2017-03-17 3:51 ` Ben Greear
@ 2017-03-24 20:54 ` Ben Greear
0 siblings, 0 replies; 6+ messages in thread
From: Ben Greear @ 2017-03-24 20:54 UTC (permalink / raw)
To: Alexander Duyck; +Cc: e1000-devel list, netdev
On 03/16/2017 08:51 PM, Ben Greear wrote:
> I think we can, might take us a day or two to get time to do it.
>
> Thanks,
> Ben
>
> On 03/16/2017 08:05 PM, Alexander Duyck wrote:
>> I'm not really interested in installing a custom version of pktgen.
>> Any chance you can recreate the issue with standard pktgen?
>>
>> You might try running perf to get a snapshot of what is using CPU time
>> on the system. It will probably give you a pretty good idea where the
>> code is that is eating up all your CPU time.
So, I had time to dig into this today.
Turns out that our tool was reporting drops because of sequence number
gaps, which in turn were caused by out-of-order frames...not actually
dropping frames.
If I force the rss_queues to one, the problem goes away.
Sorry for mis-reporting a bug.
Thanks,
Ben
>>
>> - Alex
>>
>> On Thu, Mar 16, 2017 at 7:46 PM, Ben Greear <greearb@candelatech.com> wrote:
>>> I'm actually using a hacked up version of pktgen nicely driven by our
>>> GUI tool, but the crux is that you need to set min and max src IP to some
>>> large
>>> range.
>>>
>>> We are driving pktgen from a separate machine. Stock pktgen isn't good at
>>> reporting
>>> received pkts last I checked, so it may be more difficult to easily view the
>>> problem.
>>>
>>> I'll be happy to set up my tool on your Fedora 24 or similar VM or machine
>>> if you
>>> want.
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> On 03/16/2017 07:35 PM, Alexander Duyck wrote:
>>>>
>>>> Can you include the pktgen script you are running?
>>>>
>>>> Also when you say you are driving traffic through the bridge are you
>>>> sending from something external on the system or are you actually
>>>> directing the traffic from pktgen into the bridge directly?
>>>>
>>>> - Alex
>>>>
>>>> On Thu, Mar 16, 2017 at 3:49 PM, Ben Greear <greearb@candelatech.com>
>>>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> We notice that when using two igb ports as a bridge, if we use pktgen to
>>>>> drive traffic through the bridge and randomize (or use a very large
>>>>> range)
>>>>> for the source IP addr in pktgen, then performance of igb is very poor
>>>>> (like
>>>>> 150Mbps
>>>>> throughput instead of 1Gbps). It runs right at line speed if we use same
>>>>> src/dest
>>>>> IP addr in pktgen. So, seems it is related to lots of src/dest IP
>>>>> addresses.
>>>>>
>>>>> We see same problem when using pktgen to send to itself, and we see this
>>>>> in
>>>>> several different kernels. We specifically tested bridge mode in this
>>>>> stock
>>>>> Fedora kernel:
>>>>>
>>>>> Linux lfo350-59cc 4.9.13-101.fc24.x86_64 #1 SMP Tue Mar 7 23:48:32 UTC
>>>>> 2017
>>>>> x86_64 x86_64 x86_64 GNU/Linux
>>>>>
>>>>> e1000e does not show this problem in our testing.
>>>>>
>>>>> Any ideas what the issue might be and how to fix it?
>>>>>
>>>>> Thanks,
>>>>> Ben
>>>>>
>>>>> --
>>>>> Ben Greear <greearb@candelatech.com>
>>>>> Candela Technologies Inc http://www.candelatech.com
>>>>>
>>>>
>>>
>>> --
>>> Ben Greear <greearb@candelatech.com>
>>> Candela Technologies Inc http://www.candelatech.com
>>
>
--
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc http://www.candelatech.com
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-03-24 20:54 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-16 22:49 Performance issue with igb with lots of different src-ip addrs Ben Greear
2017-03-17 2:35 ` Alexander Duyck
2017-03-17 2:46 ` Ben Greear
2017-03-17 3:05 ` Alexander Duyck
2017-03-17 3:51 ` Ben Greear
2017-03-24 20:54 ` Ben Greear
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.