ixgbe and mac-vlans problem

ixgbe and mac-vlans problem

[Ben Greear]

Hello!

We are seeing a strange problem with mac-vlans on top of an 82599 NIC (ixgbe)
on a 2.6.31.y (plus patches) kernel.

Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential
IP addresses, and set up ip rules properly.

The issue is that only 10 or so of the mac-vlans receive other than
broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.

If we manually turn on PROMISC mode on the physical ixgbe interface,
then everything works properly.  I'm guessing the ixgbe NIC must
have some filters that mac-vlan is trying to enable w/out turning on
full PROMISC mode.  Is there any easy way to debug these filters
to see if they are being properly configured?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: ixgbe and mac-vlans problem

[Patrick McHardy]

Ben Greear wrote:
> Hello!
> 
> We are seeing a strange problem with mac-vlans on top of an 82599 NIC
> (ixgbe)
> on a 2.6.31.y (plus patches) kernel.
> 
> Basically, we create 50 mac-vlans, with sequential MAC addresses and
> sequential
> IP addresses, and set up ip rules properly.
> 
> The issue is that only 10 or so of the mac-vlans receive other than
> broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.

Not showing PROMISC is normal since this is going into promisc
when too many unicast addresses are configured is handled internally
by the drivers.

> If we manually turn on PROMISC mode on the physical ixgbe interface,
> then everything works properly.  I'm guessing the ixgbe NIC must
> have some filters that mac-vlan is trying to enable w/out turning on
> full PROMISC mode.  Is there any easy way to debug these filters
> to see if they are being properly configured?

Adding "#define DEBUG" to ixgbe_common.c should print some information
when unicast addresses are added.

Re: ixgbe and mac-vlans problem

[Arnd Bergmann]

On Friday 30 April 2010 00:27:39 Ben Greear wrote:
> Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential
> IP addresses, and set up ip rules properly.
> 
> The issue is that only 10 or so of the mac-vlans receive other than
> broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.

I just took a brief look at the driver and noticed that 82599 should
be able to handle 128 entries before going into promisc mode, while
82598 (the same driver) does 16.

Maybe the logic for >16 entries is wrong, so you could try forcing
hw->mac.num_rar_entries to 16 for 82599 as well.

	Arnd

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 04/30/2010 11:00 AM, Arnd Bergmann wrote:
> On Friday 30 April 2010 00:27:39 Ben Greear wrote:
>> Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential
>> IP addresses, and set up ip rules properly.
>>
>> The issue is that only 10 or so of the mac-vlans receive other than
>> broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.
>
> I just took a brief look at the driver and noticed that 82599 should
> be able to handle 128 entries before going into promisc mode, while
> 82598 (the same driver) does 16.
>
> Maybe the logic for>16 entries is wrong, so you could try forcing
> hw->mac.num_rar_entries to 16 for 82599 as well.

I think I was actually on an 825998 system when I saw it yesterday,
but I have seen similar issues on 82599, though I didn't take time
to debug it fully, so it could have been something else.

I will double-check the NIC chipset on the system that showed the
problem yesterday.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: ixgbe and mac-vlans problem

[Tantilov, Emil S]

Ben Greear wrote:
> On 04/30/2010 11:00 AM, Arnd Bergmann wrote:
>> On Friday 30 April 2010 00:27:39 Ben Greear wrote:
>>> Basically, we create 50 mac-vlans, with sequential MAC addresses
>>> and sequential IP addresses, and set up ip rules properly.
>>> 
>>> The issue is that only 10 or so of the mac-vlans receive other than
>>> broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.
>> 
>> I just took a brief look at the driver and noticed that 82599 should
>> be able to handle 128 entries before going into promisc mode, while
>> 82598 (the same driver) does 16.
>> 
>> Maybe the logic for>16 entries is wrong, so you could try forcing
>> hw->mac.num_rar_entries to 16 for 82599 as well.
> 
> I think I was actually on an 825998 system when I saw it yesterday,
> but I have seen similar issues on 82599, though I didn't take time
> to debug it fully, so it could have been something else.
> 
> I will double-check the NIC chipset on the system that showed the
> problem yesterday.

I ran a quick test in my setup with 82599 and was able to pass traffic 
on all 50 mac-vlans without issues. This is on net-next.

Thanks,
Emil

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 04/30/2010 02:13 PM, Tantilov, Emil S wrote:

>> I will double-check the NIC chipset on the system that showed the
>> problem yesterday.
>
> I ran a quick test in my setup with 82599 and was able to pass traffic
> on all 50 mac-vlans without issues. This is on net-next.

I attempted this on an 82598 system, and it reproduces the problem
for me, so I think that must be what the customer was using as well
(I haven't gotten access to their system again, yet)

I created 50 mac-vlans on eth11, but it does NOT go PROMISC
mode as far as I can tell.  Only 14 of the mac-vlans are
working.  I'm using 64kbps UDP streams, 1k packets (ie, light load).

[root@i7-1qc-1 lanforge]# cat /sys/class/net/eth11/flags
0x1003

[root@i7-1qc-1 lanforge]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff

I put it into promisc mode and now they all work:

[root@i7-1qc-1 lanforge]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff
[root@i7-1qc-1 lanforge]# cat /sys/class/net/eth11/flags
0x1103


This is on a slightly modified 2.6.31.12 kernel.  I haven't had a chance to try
this on a more recent kernel yet.

lspci:
03:00.0 Ethernet controller: Intel Corporation 82598EB 10 Gigabit AF Dual Port Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82598EB 10 Gigabit AF Dual Port Network Connection

Please let me know if I can offer any additional information.  It will probably take a few days before I can
get the .34 kernel in testing.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 04/30/2010 02:13 PM, Tantilov, Emil S wrote:
> Ben Greear wrote:
>> On 04/30/2010 11:00 AM, Arnd Bergmann wrote:
>>> On Friday 30 April 2010 00:27:39 Ben Greear wrote:
>>>> Basically, we create 50 mac-vlans, with sequential MAC addresses
>>>> and sequential IP addresses, and set up ip rules properly.
>>>>
>>>> The issue is that only 10 or so of the mac-vlans receive other than
>>>> broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.
>>>
>>> I just took a brief look at the driver and noticed that 82599 should
>>> be able to handle 128 entries before going into promisc mode, while
>>> 82598 (the same driver) does 16.
>>>
>>> Maybe the logic for>16 entries is wrong, so you could try forcing
>>> hw->mac.num_rar_entries to 16 for 82599 as well.
>>
>> I think I was actually on an 825998 system when I saw it yesterday,
>> but I have seen similar issues on 82599, though I didn't take time
>> to debug it fully, so it could have been something else.
>>
>> I will double-check the NIC chipset on the system that showed the
>> problem yesterday.
>
> I ran a quick test in my setup with 82599 and was able to pass traffic
> on all 50 mac-vlans without issues. This is on net-next.

For an 82599 system, I can get 127 mac-vlans working out of 500 created.

That NIC also does not go PROMISC with lots (500) of mac-vlans.

Once I put it in promisc mode manually, it works fine.

So, I think whatever logic is supposed to put the NIC into promisc
mode when it overflows it's lookup tables isn't working for ixgbe
in 2.6.31.12.

Thanks,
Ben

>
> Thanks,
> Emil


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: ixgbe and mac-vlans problem

[Tantilov, Emil S]

Ben Greear wrote:
> On 04/30/2010 02:13 PM, Tantilov, Emil S wrote:
>> Ben Greear wrote:
>>> On 04/30/2010 11:00 AM, Arnd Bergmann wrote:
>>>> On Friday 30 April 2010 00:27:39 Ben Greear wrote:
>>>>> Basically, we create 50 mac-vlans, with sequential MAC addresses
>>>>> and sequential IP addresses, and set up ip rules properly.
>>>>> 
>>>>> The issue is that only 10 or so of the mac-vlans receive other
>>>>> than broadcast packets.  The ixgbe NIC doesn't show PROMISC mode.
>>>> 
>>>> I just took a brief look at the driver and noticed that 82599
>>>> should be able to handle 128 entries before going into promisc
>>>> mode, while 82598 (the same driver) does 16.
>>>> 
>>>> Maybe the logic for>16 entries is wrong, so you could try forcing
>>>> hw->mac.num_rar_entries to 16 for 82599 as well.
>>> 
>>> I think I was actually on an 825998 system when I saw it yesterday,
>>> but I have seen similar issues on 82599, though I didn't take time
>>> to debug it fully, so it could have been something else.
>>> 
>>> I will double-check the NIC chipset on the system that showed the
>>> problem yesterday.
>> 
>> I ran a quick test in my setup with 82599 and was able to pass
>> traffic on all 50 mac-vlans without issues. This is on net-next.
> 
> For an 82599 system, I can get 127 mac-vlans working out of 500
> created. 
> 
> That NIC also does not go PROMISC with lots (500) of mac-vlans.
> 
> Once I put it in promisc mode manually, it works fine.
> 
> So, I think whatever logic is supposed to put the NIC into promisc
> mode when it overflows it's lookup tables isn't working for ixgbe
> in 2.6.31.12.

Yeah, you're right. I was able to repro it. 

We'll look into it.

Thanks,
Emil

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 04/30/2010 03:26 PM, Tantilov, Emil S wrote:
> Ben Greear wrote:
>> On 04/30/2010 02:13 PM, Tantilov, Emil S wrote:

>>> I ran a quick test in my setup with 82599 and was able to pass
>>> traffic on all 50 mac-vlans without issues. This is on net-next.
>>
>> For an 82599 system, I can get 127 mac-vlans working out of 500
>> created.
>>
>> That NIC also does not go PROMISC with lots (500) of mac-vlans.
>>
>> Once I put it in promisc mode manually, it works fine.
>>
>> So, I think whatever logic is supposed to put the NIC into promisc
>> mode when it overflows it's lookup tables isn't working for ixgbe
>> in 2.6.31.12.
>
> Yeah, you're right. I was able to repro it.
>
> We'll look into it.

I'd be happy to test out a patch if you have one available.

If you don't expect to have one soon, please let me know and
I'll add work-arounds to my code to throw ixgbe NICs into PROMISC
mode manually.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: ixgbe and mac-vlans problem

[Tantilov, Emil S]

[-- Attachment #1: Type: text/plain, Size: 1210 bytes --]

Ben Greear wrote:
> On 04/30/2010 03:26 PM, Tantilov, Emil S wrote:
>> Ben Greear wrote:
>>> On 04/30/2010 02:13 PM, Tantilov, Emil S wrote:
> 
>>>> I ran a quick test in my setup with 82599 and was able to pass
>>>> traffic on all 50 mac-vlans without issues. This is on net-next.
>>> 
>>> For an 82599 system, I can get 127 mac-vlans working out of 500
>>> created. 
>>> 
>>> That NIC also does not go PROMISC with lots (500) of mac-vlans.
>>> 
>>> Once I put it in promisc mode manually, it works fine.
>>> 
>>> So, I think whatever logic is supposed to put the NIC into promisc
>>> mode when it overflows it's lookup tables isn't working for ixgbe
>>> in 2.6.31.12.
>> 
>> Yeah, you're right. I was able to repro it.
>> 
>> We'll look into it.
> 
> I'd be happy to test out a patch if you have one available.
> 
> If you don't expect to have one soon, please let me know and
> I'll add work-arounds to my code to throw ixgbe NICs into PROMISC
> mode manually.
> 
> Thanks,
> Ben

Hi Ben,

We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.

Thanks,
Emil

[-- Attachment #2: ixgbe_macvlan_v5.patch --]
[-- Type: application/octet-stream, Size: 2680 bytes --]

Introduce uc_set_promisc flag to better handle the enabling of promisc when
the number of RARs is exceeded.

Signed-off-by: Emil Tantilov <emil.s.tantilov@intel.com>

 drivers/net/ixgbe/ixgbe_common.c |    5 ++++-
 drivers/net/ixgbe/ixgbe_main.c   |    8 ++++----
 drivers/net/ixgbe/ixgbe_type.h   |    1 +
 3 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/drivers/net/ixgbe/ixgbe_common.c b/drivers/net/ixgbe/ixgbe_common.c
index ee42fd6..49775b6 100644
--- a/drivers/net/ixgbe/ixgbe_common.c
+++ b/drivers/net/ixgbe/ixgbe_common.c
@@ -1397,14 +1397,17 @@ s32 ixgbe_update_uc_addr_list_generic(struct ixgbe_hw *hw,
 			fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 			fctrl |= IXGBE_FCTRL_UPE;
 			IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);
+			hw->addr_ctrl.uc_set_promisc = true;
 		}
 	} else {
 		/* only disable if set by overflow, not by user */
-		if (old_promisc_setting && !hw->addr_ctrl.user_set_promisc) {
+		if ((old_promisc_setting && hw->addr_ctrl.uc_set_promisc) &&
+		   !(hw->addr_ctrl.user_set_promisc)){
 			hw_dbg(hw, " Leaving address overflow promisc mode\n");
 			fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 			fctrl &= ~IXGBE_FCTRL_UPE;
 			IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);
+			hw->addr_ctrl.uc_set_promisc = false;
 		}
 	}
 
diff --git a/drivers/net/ixgbe/ixgbe_main.c b/drivers/net/ixgbe/ixgbe_main.c
index 577ac72..7bf3b40 100644
--- a/drivers/net/ixgbe/ixgbe_main.c
+++ b/drivers/net/ixgbe/ixgbe_main.c
@@ -2970,8 +2970,8 @@ void ixgbe_set_rx_mode(struct net_device *netdev)
 
 	fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 
-	if (netdev->flags & IFF_PROMISC) {
-		hw->addr_ctrl.user_set_promisc = 1;
+	if (netdev->flags & IFF_PROMISC){
+		hw->addr_ctrl.user_set_promisc = true;	
 		fctrl |= (IXGBE_FCTRL_UPE | IXGBE_FCTRL_MPE);
 		/* don't hardware filter vlans in promisc mode */
 		ixgbe_vlan_filter_disable(adapter);
@@ -2979,11 +2979,11 @@ void ixgbe_set_rx_mode(struct net_device *netdev)
 		if (netdev->flags & IFF_ALLMULTI) {
 			fctrl |= IXGBE_FCTRL_MPE;
 			fctrl &= ~IXGBE_FCTRL_UPE;
-		} else {
+		} else if (!hw->addr_ctrl.uc_set_promisc) {
 			fctrl &= ~(IXGBE_FCTRL_UPE | IXGBE_FCTRL_MPE);
 		}
 		ixgbe_vlan_filter_enable(adapter);
-		hw->addr_ctrl.user_set_promisc = 0;
+		hw->addr_ctrl.user_set_promisc = false;	
 	}
 
 	IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);
diff --git a/drivers/net/ixgbe/ixgbe_type.h b/drivers/net/ixgbe/ixgbe_type.h
index 1c89cbb..38f26bb 100644
--- a/drivers/net/ixgbe/ixgbe_type.h
+++ b/drivers/net/ixgbe/ixgbe_type.h
@@ -2285,6 +2285,7 @@ struct ixgbe_addr_filter_info {
 	u32 mc_addr_in_rar_count;
 	u32 mta_in_use;
 	u32 overflow_promisc;
+	bool uc_set_promisc;
 	bool user_set_promisc;
 };
 

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 05/06/2010 10:51 AM, Tantilov, Emil S wrote:

> Hi Ben,
>
> We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.

It wasn't difficult to backport this patch to 2.6.31.12....

I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.

The NIC doesn't show as PROMISC in any way I can detect, but I guess
it must actually be in PROMISC mode:

[root@i7-1qc-1 ~]# cat /sys/class/net/eth11/flags
0x1003

[root@i7-1qc-1 ~]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
     link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: ixgbe and mac-vlans problem

[Tantilov, Emil S]

Ben Greear wrote:
> On 05/06/2010 10:51 AM, Tantilov, Emil S wrote:
> 
>> Hi Ben,
>> 
>> We do have a patch in testing (see attached). It may not apply
>> cleanly as it is on top of some other patches currently in
>> validation. Let me know if it works for you.  
> 
> It wasn't difficult to backport this patch to 2.6.31.12....
> 
> I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
> 
> The NIC doesn't show as PROMISC in any way I can detect, but I guess
> it must actually be in PROMISC mode:

Yes the interface is in promisc mode. The driver sets the FCTRL.UPE bit
(unicast promisc mode) when the number of allowed rar_entries is exceeded. 

> 
> [root@i7-1qc-1 ~]# cat /sys/class/net/eth11/flags
> 0x1003
> 
> [root@i7-1qc-1 ~]# ip link show dev eth11
> 2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
>      state UP qlen 1000 link/ether 00:e0:ed:11:25:12 brd
> ff:ff:ff:ff:ff:ff 

The IFF_PROMISC flag is not set in this case. That's how the driver knows when the promisc mode is turned by the user.

Thanks,
Emil

Re: ixgbe and mac-vlans problem

[Ben Greear]

On 05/06/2010 05:06 PM, Tantilov, Emil S wrote:
> Ben Greear wrote:
>> On 05/06/2010 10:51 AM, Tantilov, Emil S wrote:
>>
>>> Hi Ben,
>>>
>>> We do have a patch in testing (see attached). It may not apply
>>> cleanly as it is on top of some other patches currently in
>>> validation. Let me know if it works for you.
>>
>> It wasn't difficult to backport this patch to 2.6.31.12....
>>
>> I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
>>
>> The NIC doesn't show as PROMISC in any way I can detect, but I guess
>> it must actually be in PROMISC mode:
>
> Yes the interface is in promisc mode. The driver sets the FCTRL.UPE bit
> (unicast promisc mode) when the number of allowed rar_entries is exceeded.

Is there any way to get this setting from ethtool or similar?  It would be nice
to know the actual PROMISC state of the NIC regardless of what user-space has or has not
configured.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: ixgbe and mac-vlans problem

[Tantilov, Emil S]

Ben Greear wrote:
> On 05/06/2010 05:06 PM, Tantilov, Emil S wrote:
>> Ben Greear wrote:
>>> On 05/06/2010 10:51 AM, Tantilov, Emil S wrote:
>>> 
>>>> Hi Ben,
>>>> 
>>>> We do have a patch in testing (see attached). It may not apply
>>>> cleanly as it is on top of some other patches currently in
>>>> validation. Let me know if it works for you.
>>> 
>>> It wasn't difficult to backport this patch to 2.6.31.12....
>>> 
>>> I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
>>> 
>>> The NIC doesn't show as PROMISC in any way I can detect, but I guess
>>> it must actually be in PROMISC mode:
>> 
>> Yes the interface is in promisc mode. The driver sets the FCTRL.UPE
>> bit (unicast promisc mode) when the number of allowed rar_entries is
>> exceeded. 
> 
> Is there any way to get this setting from ethtool or similar?  It
> would be nice to know the actual PROMISC state of the NIC regardless of
> what user-space has or has not configured.

ethtool -d eth11 | grep -i promisc 

Thanks,
Emil