Bonding problem on Intel X710 hardware

Bonding problem on Intel X710 hardware

[Sven Anders]

Hello!

I'm having problems setting up a bond in adaptive load balancing
mode (balance-alb, mode 6) on an intel X710 network adapter using
the i40e driver connected to an Aruba 2530-48G switch.
The network card has 4 on board ports.
I'm using 2 ports for the bond with 36 VLANs on it.

The setup is correct, because it works without problems, if
I use the same setup with 1GBit Intel hardware (using the
e1000e driver, version 3.2.6-k, firmware 5.10-2).

Data packets are only received sporadically. If I run the same test
with only one slave port, it works without problems.

I debugged it down to the reception of the packets by the
network hardware.

If I remove the number of VLANs under 8, almost all packets are
received. The fewer VLANs the better the receive rate.

I suspected the hardware offloading operations to be the cause, so I
tried to disable them. It resulted in the following:

  If I turn of the "ntuple-filters" with
    ethtool -K eth3 ntuple off
    ethtool -K eth3 ntuple off
  it will work.

  But if I do this I see the following errors in "dmesg":
   i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
   i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on

Disabling any any other offloading operations made no change.

For me it seems, that the hardware filter is dropping packets because they
have the wrong values (mac-address ?).
Turning the "ntuple-filters" off, forces the network adapter to accept
all packets.


My questions:

1. Can anybody explain or confirm this?

2. Is the a correct method to force the adapter in promiscous mode?

3. Are the any special settings needed, if I use ALB bonding, which I missed?


Some details:
-------------

Linux kernel 5.15.35-core2 on x86_64.


This is the hardware:
---------------------
4 port Ethernet controller:
  Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
  8086:15ff (rev 02)

with

  driver: i40e
  version: 5.15.35-core2
  firmware-version: 8.60 0x8000bd80 1.3140.0
  bus-info: 0000:65:00.2
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: yes
  supports-register-dump: yes
  supports-priv-flags: yes


This is current bonding configuration:
--------------------------------------
Ethernet Channel Bonding Driver: v5.15.35-core2

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: eth3
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200
Peer Notification Delay (ms): 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:f8:9c:42
Slave queue ID: 0

Slave Interface: eth4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 68:05:ca:f8:9c:41
Slave queue ID: 0


Regards
  Sven Anders

-- 
  Sven Anders                  () UTF-8 Ribbon Campaign
                               /\ Support plain text e-mail
  ANDURAS intranet security AG
  Messestrasse 3 - 94036 Passau - Germany
  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
   - Benjamin Franklin

Re: Bonding problem on Intel X710 hardware

[Sven Anders]

Hello!

This is a follow up to my question. I did not hear anything so far, but I tried
to find some some information meanwhile.

I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
I'm hitting a limit here.

Somebody other said, that this seems to be an error in the "bonding driver", but
I do not think so. Aside from that, there seem to be no special "bonding" mailing
list anymore. So I will have to ask this questions here anyway...

I want to understand the problem to classify it.

1) Why is this "error" issued? Do I really hit a limit and what is this current limit?
2) Is it really an error or is it more "a warning"?
3) Why is this error triggered only when changing the "ntuples filter" and not during
    the normal adding of VLANs?
    Remark: I can trigger the "ntuples fiilter" later on again and it still works.

I also tried the latest 5.18-rc kernel with the same problem.

Maybe somebody will find time and try to reproduce this?
I will answer any questions...

Regards
  Sven

Am 12.05.22 um 16:05 schrieb Sven Anders:
> Hello!
> 
> I'm having problems setting up a bond in adaptive load balancing
> mode (balance-alb, mode 6) on an intel X710 network adapter using
> the i40e driver connected to an Aruba 2530-48G switch.
> The network card has 4 on board ports.
> I'm using 2 ports for the bond with 36 VLANs on it.
> 
> The setup is correct, because it works without problems, if
> I use the same setup with 1GBit Intel hardware (using the
> e1000e driver, version 3.2.6-k, firmware 5.10-2).
> 
> Data packets are only received sporadically. If I run the same test
> with only one slave port, it works without problems.
> 
> I debugged it down to the reception of the packets by the
> network hardware.
> 
> If I remove the number of VLANs under 8, almost all packets are
> received. The fewer VLANs the better the receive rate.
> 
> I suspected the hardware offloading operations to be the cause, so I
> tried to disable them. It resulted in the following:
> 
>   If I turn of the "ntuple-filters" with
>     ethtool -K eth3 ntuple off
>     ethtool -K eth3 ntuple off
>   it will work.
> 
>   But if I do this I see the following errors in "dmesg":
>    i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
>    i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
> 
> Disabling any any other offloading operations made no change.
> 
> For me it seems, that the hardware filter is dropping packets because they
> have the wrong values (mac-address ?).
> Turning the "ntuple-filters" off, forces the network adapter to accept
> all packets.
> 
> 
> My questions:
> 
> 1. Can anybody explain or confirm this?
> 
> 2. Is the a correct method to force the adapter in promiscous mode?
> 
> 3. Are the any special settings needed, if I use ALB bonding, which I missed?
> 
> 
> Some details:
> -------------
> 
> Linux kernel 5.15.35-core2 on x86_64.
> 
> 
> This is the hardware:
> ---------------------
> 4 port Ethernet controller:
>   Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
>   8086:15ff (rev 02)
> 
> with
> 
>   driver: i40e
>   version: 5.15.35-core2
>   firmware-version: 8.60 0x8000bd80 1.3140.0
>   bus-info: 0000:65:00.2
>   supports-statistics: yes
>   supports-test: yes
>   supports-eeprom-access: yes
>   supports-register-dump: yes
>   supports-priv-flags: yes
> 
> 
> This is current bonding configuration:
> --------------------------------------
> Ethernet Channel Bonding Driver: v5.15.35-core2
> 
> Bonding Mode: adaptive load balancing
> Primary Slave: None
> Currently Active Slave: eth3
> MII Status: up
> MII Polling Interval (ms): 100
> Up Delay (ms): 200
> Down Delay (ms): 200
> Peer Notification Delay (ms): 0
> 
> Slave Interface: eth3
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Permanent HW addr: 68:05:ca:f8:9c:42
> Slave queue ID: 0
> 
> Slave Interface: eth4
> MII Status: up
> Speed: 1000 Mbps
> Duplex: full
> Link Failure Count: 0
> Permanent HW addr: 68:05:ca:f8:9c:41
> Slave queue ID: 0
> 
> 
> Regards
>   Sven Anders
> 


Mit freundlichen Grüßen
  Sven Anders

-- 
  Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
                                                  /\ Support plain text e-mail
  ANDURAS intranet security AG
  Messestraße 3 - 94036 Passau - Germany
  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0

Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB 6032
Vorstand: Marcus Junker
Vorsitzender des Aufsichtsrats: Andreas Wagner

Re: Bonding problem on Intel X710 hardware

[Jakub Kicinski]

CC: intel

On Tue, 17 May 2022 16:23:16 +0200 Sven Anders wrote:
> Hello!
> 
> This is a follow up to my question. I did not hear anything so far, but I tried
> to find some some information meanwhile.
> 
> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
> I'm hitting a limit here.
> 
> Somebody other said, that this seems to be an error in the "bonding driver", but
> I do not think so. Aside from that, there seem to be no special "bonding" mailing
> list anymore. So I will have to ask this questions here anyway...
> 
> I want to understand the problem to classify it.
> 
> 1) Why is this "error" issued? Do I really hit a limit and what is this current limit?
> 2) Is it really an error or is it more "a warning"?
> 3) Why is this error triggered only when changing the "ntuples filter" and not during
>     the normal adding of VLANs?
>     Remark: I can trigger the "ntuples fiilter" later on again and it still works.
> 
> I also tried the latest 5.18-rc kernel with the same problem.
> 
> Maybe somebody will find time and try to reproduce this?
> I will answer any questions...
> 
> Regards
>   Sven
> 
> Am 12.05.22 um 16:05 schrieb Sven Anders:
> > Hello!
> > 
> > I'm having problems setting up a bond in adaptive load balancing
> > mode (balance-alb, mode 6) on an intel X710 network adapter using
> > the i40e driver connected to an Aruba 2530-48G switch.
> > The network card has 4 on board ports.
> > I'm using 2 ports for the bond with 36 VLANs on it.
> > 
> > The setup is correct, because it works without problems, if
> > I use the same setup with 1GBit Intel hardware (using the
> > e1000e driver, version 3.2.6-k, firmware 5.10-2).
> > 
> > Data packets are only received sporadically. If I run the same test
> > with only one slave port, it works without problems.
> > 
> > I debugged it down to the reception of the packets by the
> > network hardware.
> > 
> > If I remove the number of VLANs under 8, almost all packets are
> > received. The fewer VLANs the better the receive rate.
> > 
> > I suspected the hardware offloading operations to be the cause, so I
> > tried to disable them. It resulted in the following:
> > 
> >   If I turn of the "ntuple-filters" with
> >     ethtool -K eth3 ntuple off
> >     ethtool -K eth3 ntuple off
> >   it will work.
> > 
> >   But if I do this I see the following errors in "dmesg":
> >    i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
> >    i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
> > 
> > Disabling any any other offloading operations made no change.
> > 
> > For me it seems, that the hardware filter is dropping packets because they
> > have the wrong values (mac-address ?).
> > Turning the "ntuple-filters" off, forces the network adapter to accept
> > all packets.
> > 
> > 
> > My questions:
> > 
> > 1. Can anybody explain or confirm this?
> > 
> > 2. Is the a correct method to force the adapter in promiscous mode?
> > 
> > 3. Are the any special settings needed, if I use ALB bonding, which I missed?
> > 
> > 
> > Some details:
> > -------------
> > 
> > Linux kernel 5.15.35-core2 on x86_64.
> > 
> > 
> > This is the hardware:
> > ---------------------
> > 4 port Ethernet controller:
> >   Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
> >   8086:15ff (rev 02)
> > 
> > with
> > 
> >   driver: i40e
> >   version: 5.15.35-core2
> >   firmware-version: 8.60 0x8000bd80 1.3140.0
> >   bus-info: 0000:65:00.2
> >   supports-statistics: yes
> >   supports-test: yes
> >   supports-eeprom-access: yes
> >   supports-register-dump: yes
> >   supports-priv-flags: yes
> > 
> > 
> > This is current bonding configuration:
> > --------------------------------------
> > Ethernet Channel Bonding Driver: v5.15.35-core2
> > 
> > Bonding Mode: adaptive load balancing
> > Primary Slave: None
> > Currently Active Slave: eth3
> > MII Status: up
> > MII Polling Interval (ms): 100
> > Up Delay (ms): 200
> > Down Delay (ms): 200
> > Peer Notification Delay (ms): 0
> > 
> > Slave Interface: eth3
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 0
> > Permanent HW addr: 68:05:ca:f8:9c:42
> > Slave queue ID: 0
> > 
> > Slave Interface: eth4
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 0
> > Permanent HW addr: 68:05:ca:f8:9c:41
> > Slave queue ID: 0
> > 
> > 
> > Regards
> >   Sven Anders
> >   
> 
> 
> Mit freundlichen Grüßen
>   Sven Anders
> 

[Intel-wired-lan] Bonding problem on Intel X710 hardware

[Jakub Kicinski]

CC: intel

On Tue, 17 May 2022 16:23:16 +0200 Sven Anders wrote:
> Hello!
> 
> This is a follow up to my question. I did not hear anything so far, but I tried
> to find some some information meanwhile.
> 
> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
> I'm hitting a limit here.
> 
> Somebody other said, that this seems to be an error in the "bonding driver", but
> I do not think so. Aside from that, there seem to be no special "bonding" mailing
> list anymore. So I will have to ask this questions here anyway...
> 
> I want to understand the problem to classify it.
> 
> 1) Why is this "error" issued? Do I really hit a limit and what is this current limit?
> 2) Is it really an error or is it more "a warning"?
> 3) Why is this error triggered only when changing the "ntuples filter" and not during
>     the normal adding of VLANs?
>     Remark: I can trigger the "ntuples fiilter" later on again and it still works.
> 
> I also tried the latest 5.18-rc kernel with the same problem.
> 
> Maybe somebody will find time and try to reproduce this?
> I will answer any questions...
> 
> Regards
>   Sven
> 
> Am 12.05.22 um 16:05 schrieb Sven Anders:
> > Hello!
> > 
> > I'm having problems setting up a bond in adaptive load balancing
> > mode (balance-alb, mode 6) on an intel X710 network adapter using
> > the i40e driver connected to an Aruba 2530-48G switch.
> > The network card has 4 on board ports.
> > I'm using 2 ports for the bond with 36 VLANs on it.
> > 
> > The setup is correct, because it works without problems, if
> > I use the same setup with 1GBit Intel hardware (using the
> > e1000e driver, version 3.2.6-k, firmware 5.10-2).
> > 
> > Data packets are only received sporadically. If I run the same test
> > with only one slave port, it works without problems.
> > 
> > I debugged it down to the reception of the packets by the
> > network hardware.
> > 
> > If I remove the number of VLANs under 8, almost all packets are
> > received. The fewer VLANs the better the receive rate.
> > 
> > I suspected the hardware offloading operations to be the cause, so I
> > tried to disable them. It resulted in the following:
> > 
> >  ?If I turn of the "ntuple-filters" with
> >  ?? ethtool -K eth3 ntuple off
> >  ?? ethtool -K eth3 ntuple off
> >  ?it will work.
> > 
> >  ?But if I do this I see the following errors in "dmesg":
> >  ? i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
> >  ? i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
> > 
> > Disabling any any other offloading operations made no change.
> > 
> > For me it seems, that the hardware filter is dropping packets because they
> > have the wrong values (mac-address ?).
> > Turning the "ntuple-filters" off, forces the network adapter to accept
> > all packets.
> > 
> > 
> > My questions:
> > 
> > 1. Can anybody explain or confirm this?
> > 
> > 2. Is the a correct method to force the adapter in promiscous mode?
> > 
> > 3. Are the any special settings needed, if I use ALB bonding, which I missed?
> > 
> > 
> > Some details:
> > -------------
> > 
> > Linux kernel 5.15.35-core2 on x86_64.
> > 
> > 
> > This is the hardware:
> > ---------------------
> > 4 port Ethernet controller:
> >  ?Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
> >  ?8086:15ff (rev 02)
> > 
> > with
> > 
> >  ?driver: i40e
> >  ?version: 5.15.35-core2
> >  ?firmware-version: 8.60 0x8000bd80 1.3140.0
> >  ?bus-info: 0000:65:00.2
> >  ?supports-statistics: yes
> >  ?supports-test: yes
> >  ?supports-eeprom-access: yes
> >  ?supports-register-dump: yes
> >  ?supports-priv-flags: yes
> > 
> > 
> > This is current bonding configuration:
> > --------------------------------------
> > Ethernet Channel Bonding Driver: v5.15.35-core2
> > 
> > Bonding Mode: adaptive load balancing
> > Primary Slave: None
> > Currently Active Slave: eth3
> > MII Status: up
> > MII Polling Interval (ms): 100
> > Up Delay (ms): 200
> > Down Delay (ms): 200
> > Peer Notification Delay (ms): 0
> > 
> > Slave Interface: eth3
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 0
> > Permanent HW addr: 68:05:ca:f8:9c:42
> > Slave queue ID: 0
> > 
> > Slave Interface: eth4
> > MII Status: up
> > Speed: 1000 Mbps
> > Duplex: full
> > Link Failure Count: 0
> > Permanent HW addr: 68:05:ca:f8:9c:41
> > Slave queue ID: 0
> > 
> > 
> > Regards
> >  ?Sven Anders
> >   
> 
> 
> Mit freundlichen Gr??en
>   Sven Anders
> 

Re: Bonding problem on Intel X710 hardware

[Jesse Brandeburg]

On 5/17/2022 1:45 PM, Jakub Kicinski wrote:
> CC: intel

Thanks for the copy.


> 
> On Tue, 17 May 2022 16:23:16 +0200 Sven Anders wrote:
>> Hello!
>>
>> This is a follow up to my question. I did not hear anything so far, but I tried
>> to find some some information meanwhile.
>>
>> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
>> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
>> I'm hitting a limit here.

Yes, typically this is a response from our firmware that a table is full 
in hardware, and I'm guessing that you might be running into a filter 
limit due to using vlans?

>>
>> Somebody other said, that this seems to be an error in the "bonding driver", but
>> I do not think so. Aside from that, there seem to be no special "bonding" mailing
>> list anymore. So I will have to ask this questions here anyway...

this netdev list is the bonding list since it's part of the networking 
stack in the kernel.

>>
>> I want to understand the problem to classify it.
>>
>> 1) Why is this "error" issued? Do I really hit a limit and what is this current limit?
>> 2) Is it really an error or is it more "a warning"?
>> 3) Why is this error triggered only when changing the "ntuples filter" and not during
>>      the normal adding of VLANs?
>>      Remark: I can trigger the "ntuples fiilter" later on again and it still works.
>>
>> I also tried the latest 5.18-rc kernel with the same problem.
>>
>> Maybe somebody will find time and try to reproduce this?
>> I will answer any questions...

We assigned someone to look into reproduction of this today, and they'll 
get back to you if we have further questions.


>>
>> Regards
>>    Sven
>>
>> Am 12.05.22 um 16:05 schrieb Sven Anders:
>>> Hello!
>>>
>>> I'm having problems setting up a bond in adaptive load balancing
>>> mode (balance-alb, mode 6) on an intel X710 network adapter using
>>> the i40e driver connected to an Aruba 2530-48G switch.
>>> The network card has 4 on board ports.
>>> I'm using 2 ports for the bond with 36 VLANs on it.
>>>
>>> The setup is correct, because it works without problems, if
>>> I use the same setup with 1GBit Intel hardware (using the
>>> e1000e driver, version 3.2.6-k, firmware 5.10-2).
>>>
>>> Data packets are only received sporadically. If I run the same test
>>> with only one slave port, it works without problems.

And there are no counters going up in ethtool -S when you receive/drop 
packets?

>>>
>>> I debugged it down to the reception of the packets by the
>>> network hardware.
>>>
>>> If I remove the number of VLANs under 8, almost all packets are
>>> received. The fewer VLANs the better the receive rate.
>>>
>>> I suspected the hardware offloading operations to be the cause, so I
>>> tried to disable them. It resulted in the following:
>>>
>>>    If I turn of the "ntuple-filters" with
>>>      ethtool -K eth3 ntuple off
>>>      ethtool -K eth3 ntuple off
>>>    it will work.
>>>
>>>    But if I do this I see the following errors in "dmesg":
>>>     i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
>>>     i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
>>>
>>> Disabling any any other offloading operations made no change.
>>>
>>> For me it seems, that the hardware filter is dropping packets because they
>>> have the wrong values (mac-address ?).
>>> Turning the "ntuple-filters" off, forces the network adapter to accept
>>> all packets.
>>>
>>>
>>> My questions:
>>>
>>> 1. Can anybody explain or confirm this?
>>>
>>> 2. Is the a correct method to force the adapter in promiscous mode?
>>>
>>> 3. Are the any special settings needed, if I use ALB bonding, which I missed?
>>>
>>>
>>> Some details:
>>> -------------
>>>
>>> Linux kernel 5.15.35-core2 on x86_64.
>>>
>>>
>>> This is the hardware:
>>> ---------------------
>>> 4 port Ethernet controller:
>>>    Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
>>>    8086:15ff (rev 02)
>>>
>>> with
>>>
>>>    driver: i40e
>>>    version: 5.15.35-core2
>>>    firmware-version: 8.60 0x8000bd80 1.3140.0
>>>    bus-info: 0000:65:00.2
>>>    supports-statistics: yes
>>>    supports-test: yes
>>>    supports-eeprom-access: yes
>>>    supports-register-dump: yes
>>>    supports-priv-flags: yes
>>>
>>>
>>> This is current bonding configuration:
>>> --------------------------------------
>>> Ethernet Channel Bonding Driver: v5.15.35-core2
>>>
>>> Bonding Mode: adaptive load balancing
>>> Primary Slave: None
>>> Currently Active Slave: eth3
>>> MII Status: up
>>> MII Polling Interval (ms): 100
>>> Up Delay (ms): 200
>>> Down Delay (ms): 200
>>> Peer Notification Delay (ms): 0
>>>
>>> Slave Interface: eth3
>>> MII Status: up
>>> Speed: 1000 Mbps
>>> Duplex: full
>>> Link Failure Count: 0
>>> Permanent HW addr: 68:05:ca:f8:9c:42
>>> Slave queue ID: 0
>>>
>>> Slave Interface: eth4
>>> MII Status: up
>>> Speed: 1000 Mbps
>>> Duplex: full
>>> Link Failure Count: 0
>>> Permanent HW addr: 68:05:ca:f8:9c:41
>>> Slave queue ID: 0
>>>
>>>
>>> Regards
>>>    Sven Anders
>>>    
>>
>>
>> Mit freundlichen Grüßen
>>    Sven Anders
>>
> 

[Intel-wired-lan] Bonding problem on Intel X710 hardware

[Jesse Brandeburg]

On 5/17/2022 1:45 PM, Jakub Kicinski wrote:
> CC: intel

Thanks for the copy.


> 
> On Tue, 17 May 2022 16:23:16 +0200 Sven Anders wrote:
>> Hello!
>>
>> This is a follow up to my question. I did not hear anything so far, but I tried
>> to find some some information meanwhile.
>>
>> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
>> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
>> I'm hitting a limit here.

Yes, typically this is a response from our firmware that a table is full 
in hardware, and I'm guessing that you might be running into a filter 
limit due to using vlans?

>>
>> Somebody other said, that this seems to be an error in the "bonding driver", but
>> I do not think so. Aside from that, there seem to be no special "bonding" mailing
>> list anymore. So I will have to ask this questions here anyway...

this netdev list is the bonding list since it's part of the networking 
stack in the kernel.

>>
>> I want to understand the problem to classify it.
>>
>> 1) Why is this "error" issued? Do I really hit a limit and what is this current limit?
>> 2) Is it really an error or is it more "a warning"?
>> 3) Why is this error triggered only when changing the "ntuples filter" and not during
>>      the normal adding of VLANs?
>>      Remark: I can trigger the "ntuples fiilter" later on again and it still works.
>>
>> I also tried the latest 5.18-rc kernel with the same problem.
>>
>> Maybe somebody will find time and try to reproduce this?
>> I will answer any questions...

We assigned someone to look into reproduction of this today, and they'll 
get back to you if we have further questions.


>>
>> Regards
>>    Sven
>>
>> Am 12.05.22 um 16:05 schrieb Sven Anders:
>>> Hello!
>>>
>>> I'm having problems setting up a bond in adaptive load balancing
>>> mode (balance-alb, mode 6) on an intel X710 network adapter using
>>> the i40e driver connected to an Aruba 2530-48G switch.
>>> The network card has 4 on board ports.
>>> I'm using 2 ports for the bond with 36 VLANs on it.
>>>
>>> The setup is correct, because it works without problems, if
>>> I use the same setup with 1GBit Intel hardware (using the
>>> e1000e driver, version 3.2.6-k, firmware 5.10-2).
>>>
>>> Data packets are only received sporadically. If I run the same test
>>> with only one slave port, it works without problems.

And there are no counters going up in ethtool -S when you receive/drop 
packets?

>>>
>>> I debugged it down to the reception of the packets by the
>>> network hardware.
>>>
>>> If I remove the number of VLANs under 8, almost all packets are
>>> received. The fewer VLANs the better the receive rate.
>>>
>>> I suspected the hardware offloading operations to be the cause, so I
>>> tried to disable them. It resulted in the following:
>>>
>>>   ?If I turn of the "ntuple-filters" with
>>>   ?? ethtool -K eth3 ntuple off
>>>   ?? ethtool -K eth3 ntuple off
>>>   ?it will work.
>>>
>>>   ?But if I do this I see the following errors in "dmesg":
>>>   ? i40e 0000:65:00.1: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
>>>   ? i40e 0000:65:00.2: Error I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced on
>>>
>>> Disabling any any other offloading operations made no change.
>>>
>>> For me it seems, that the hardware filter is dropping packets because they
>>> have the wrong values (mac-address ?).
>>> Turning the "ntuple-filters" off, forces the network adapter to accept
>>> all packets.
>>>
>>>
>>> My questions:
>>>
>>> 1. Can anybody explain or confirm this?
>>>
>>> 2. Is the a correct method to force the adapter in promiscous mode?
>>>
>>> 3. Are the any special settings needed, if I use ALB bonding, which I missed?
>>>
>>>
>>> Some details:
>>> -------------
>>>
>>> Linux kernel 5.15.35-core2 on x86_64.
>>>
>>>
>>> This is the hardware:
>>> ---------------------
>>> 4 port Ethernet controller:
>>>   ?Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
>>>   ?8086:15ff (rev 02)
>>>
>>> with
>>>
>>>   ?driver: i40e
>>>   ?version: 5.15.35-core2
>>>   ?firmware-version: 8.60 0x8000bd80 1.3140.0
>>>   ?bus-info: 0000:65:00.2
>>>   ?supports-statistics: yes
>>>   ?supports-test: yes
>>>   ?supports-eeprom-access: yes
>>>   ?supports-register-dump: yes
>>>   ?supports-priv-flags: yes
>>>
>>>
>>> This is current bonding configuration:
>>> --------------------------------------
>>> Ethernet Channel Bonding Driver: v5.15.35-core2
>>>
>>> Bonding Mode: adaptive load balancing
>>> Primary Slave: None
>>> Currently Active Slave: eth3
>>> MII Status: up
>>> MII Polling Interval (ms): 100
>>> Up Delay (ms): 200
>>> Down Delay (ms): 200
>>> Peer Notification Delay (ms): 0
>>>
>>> Slave Interface: eth3
>>> MII Status: up
>>> Speed: 1000 Mbps
>>> Duplex: full
>>> Link Failure Count: 0
>>> Permanent HW addr: 68:05:ca:f8:9c:42
>>> Slave queue ID: 0
>>>
>>> Slave Interface: eth4
>>> MII Status: up
>>> Speed: 1000 Mbps
>>> Duplex: full
>>> Link Failure Count: 0
>>> Permanent HW addr: 68:05:ca:f8:9c:41
>>> Slave queue ID: 0
>>>
>>>
>>> Regards
>>>   ?Sven Anders
>>>    
>>
>>
>> Mit freundlichen Gr??en
>>    Sven Anders
>>
> 

[Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders]

>>> Hello!
>>>
>>> This is a follow up to my question. I did not hear anything so far, but I tried
>>> to find some some information meanwhile.
>>>
>>> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
>>> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
>>> I'm hitting a limit here.
> 
> Yes, typically this is a response from our firmware that a table is full 
> in hardware, and I'm guessing that you might be running into a filter 
> limit due to using vlans?

That's what I suspect, but I did know the hardware and firmware insufficiently
to be sure.
What I'm wondering is: Why this is only triggered if I toggle the "ntuples support"
and not earlier when setting up the interface?

>>> Data packets are only received sporadically. If I run the same test
>>> with only one slave port, it works without problems.
> 
> And there are no counters going up in ethtool -S when you receive/drop 
> packets?

I'm not sure here. I tried to inspect the counters and it seems that the counters
are going up slower in this case, but it's difficult to say, because there is
some "noise" on the line and I do not have direct access to the hardware at the
moment.


If you need any further information or tests, just contact me.

Regards
   Sven Anders

-- 
   Sven Anders                  () UTF-8 Ribbon Campaign
                                                   /\ Support plain text e-mail
   ANDURAS intranet security AG
   Messestrasse 3 - 94036 Passau - Germany
   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
    - Benjamin Franklin

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders]

Hello!

Was anybody able to reproduce the problem?
Do you need assistance or further information?

Regards
  Sven

Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>> Hello!
>>>>
>>>> This is a follow up to my question. I did not hear anything so far, but I tried
>>>> to find some some information meanwhile.
>>>>
>>>> I've got a guess from somebody, that the error message "Error I40E_AQ_RC_EINVAL
>>>> adding RX filters on PF, promiscuous mode forced on" maybe triggered, because
>>>> I'm hitting a limit here.
>>
>> Yes, typically this is a response from our firmware that a table is full in hardware, and I'm guessing that you might be running into a filter limit 
>> due to using vlans?
> 
> That's what I suspect, but I did know the hardware and firmware insufficiently
> to be sure.
> What I'm wondering is: Why this is only triggered if I toggle the "ntuples support"
> and not earlier when setting up the interface?
> 
>>>> Data packets are only received sporadically. If I run the same test
>>>> with only one slave port, it works without problems.
>>
>> And there are no counters going up in ethtool -S when you receive/drop packets?
> 
> I'm not sure here. I tried to inspect the counters and it seems that the counters
> are going up slower in this case, but it's difficult to say, because there is
> some "noise" on the line and I do not have direct access to the hardware at the
> moment.
> 
> If you need any further information or tests, just contact me.

Regards
    Sven Anders
-- 
  Sven Anders                  () UTF-8 Ribbon Campaign
                                                  /\ Support plain text e-mail
  ANDURAS intranet security AG
  Messestrasse 3 - 94036 Passau - Germany
  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
   - Benjamin Franklin

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Switzer, David]

>-----Original Message-----
>From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of
>Sven Anders
>Sent: Monday, June 6, 2022 11:05 PM
>To: intel-wired-lan@osuosl.org
>Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>
>Hello!
Hi Sven!  My apologies on taking so long to reply.
>
>Was anybody able to reproduce the problem?
I haven't been able to reproduce your issue yet, but with your description it sounds that the issue could be related to the X710 hardware LLDP engine consuming the LLDP packets.  To disable the hardware engine, user the following command on each of the ports that you're using in the bond:
ethtool --set-priv-flags <interface name> disable-fw-lldp on

Then the LLDP packets that the bonding modules relies on will be forwarded by the hardware to the OS network stack.

I hope you have a great weekend!
Dave

>Do you need assistance or further information?
>
>Regards
>  Sven
>
>Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>> Hello!
>>>>>
>>>>> This is a follow up to my question. I did not hear anything so far,
>>>>> but I tried to find some some information meanwhile.
>>>>>
>>>>> I've got a guess from somebody, that the error message "Error
>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced
>>>>> on" maybe triggered, because I'm hitting a limit here.
>>>
>>> Yes, typically this is a response from our firmware that a table is
>>> full in hardware, and I'm guessing that you might be running into a filter
>limit due to using vlans?
>>
>> That's what I suspect, but I did know the hardware and firmware
>> insufficiently to be sure.
>> What I'm wondering is: Why this is only triggered if I toggle the "ntuples
>support"
>> and not earlier when setting up the interface?
>>
>>>>> Data packets are only received sporadically. If I run the same test
>>>>> with only one slave port, it works without problems.
>>>
>>> And there are no counters going up in ethtool -S when you receive/drop
>packets?
>>
>> I'm not sure here. I tried to inspect the counters and it seems that
>> the counters are going up slower in this case, but it's difficult to
>> say, because there is some "noise" on the line and I do not have
>> direct access to the hardware at the moment.
>>
>> If you need any further information or tests, just contact me.
>
>Regards
>    Sven Anders
>--
>  Sven Anders                  () UTF-8 Ribbon Campaign
>                                                  /\ Support plain text e-mail
>  ANDURAS intranet security AG
>  Messestrasse 3 - 94036 Passau - Germany
>  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55
>
>Those who would give up essential Liberty, to purchase a little temporary
>Safety, deserve neither Liberty nor Safety.
>   - Benjamin Franklin
>
>_______________________________________________
>Intel-wired-lan mailing list
>Intel-wired-lan@osuosl.org
>https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders]

Hallo Dave,

thanks for your answer.

I tried to disable the LLDP engine, as suggested, but it did not change anything.

To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
which works with ARP negotiation.
If I'm using the "active-backup" bonding mode, it all works fine. But this
is nearly the same as using only one interface, which works too.

In my test setup, I'm using two "HPE 2530-48G-PoE+" switches (YA.16.10.0002),
each is connected with one cable to the network card.
My test is very simple:
After configuring the bond, I used a continous ping to the Linux server and
I see many drops. If I toggle the "ntuples support", the drops disappear.

Can we compare the settings of your setup with mine? Maybe you missed
something...
How can I assist further?

Regards
  Sven

Am 11.06.22 um 03:27 schrieb Switzer, David:
>> -----Original Message-----
>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of
>> Sven Anders
>> Sent: Monday, June 6, 2022 11:05 PM
>> To: intel-wired-lan@osuosl.org
>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>
>> Hello!
> Hi Sven!  My apologies on taking so long to reply.
>>
>> Was anybody able to reproduce the problem?
> I haven't been able to reproduce your issue yet, but with your description it sounds that the issue could be related to the X710 hardware
> LLDP engine consuming the LLDP packets.  To disable the hardware engine, user the following command on each of the ports that you're using
> in the bond:
 >
> ethtool --set-priv-flags <interface name> disable-fw-lldp on
> 
> Then the LLDP packets that the bonding modules relies on will be forwarded by the hardware to the OS network stack.
> 
> I hope you have a great weekend!
> Dave
> 
>> Do you need assistance or further information?
>>
>> Regards
>>   Sven
>>
>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>> Hello!
>>>>>>
>>>>>> This is a follow up to my question. I did not hear anything so far,
>>>>>> but I tried to find some some information meanwhile.
>>>>>>
>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode forced
>>>>>> on" maybe triggered, because I'm hitting a limit here.
>>>>
>>>> Yes, typically this is a response from our firmware that a table is
>>>> full in hardware, and I'm guessing that you might be running into a filter
>> limit due to using vlans?
>>>
>>> That's what I suspect, but I did know the hardware and firmware
>>> insufficiently to be sure.
>>> What I'm wondering is: Why this is only triggered if I toggle the "ntuples
>> support"
>>> and not earlier when setting up the interface?
>>>
>>>>>> Data packets are only received sporadically. If I run the same test
>>>>>> with only one slave port, it works without problems.
>>>>
>>>> And there are no counters going up in ethtool -S when you receive/drop
>> packets?
>>>
>>> I'm not sure here. I tried to inspect the counters and it seems that
>>> the counters are going up slower in this case, but it's difficult to
>>> say, because there is some "noise" on the line and I do not have
>>> direct access to the hardware at the moment.
>>>
>>> If you need any further information or tests, just contact me.
>>
>> Regards
>>     Sven Anders
>> --
>>   Sven Anders                  () UTF-8 Ribbon Campaign
>>                                                   /\ Support plain text e-mail
>>   ANDURAS intranet security AG
>>   Messestrasse 3 - 94036 Passau - Germany
>>   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55
>>
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>>    - Benjamin Franklin
>>
>> _______________________________________________
>> Intel-wired-lan mailing list
>> Intel-wired-lan@osuosl.org
>> https://eu-central-1.protection.sophos.com?d=osuosl.org&u=aHR0cHM6Ly9saXN0cy5vc3Vvc2wub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtd2lyZWQtbGFu&i=NWRkZTc5MjQzMjM2YzExNjg0ZTI4MjM4&t=WHZUTUdjNVkySTVVcWVjUEpVTGlkaXV3NkxVQm1LSGdUOXMxeVVEQ1pyOD0=&h=4ed5ca43b14147eb8ae4891cf2a54f70


Mit freundlichen Grüßen
  Sven Anders

-- 
  Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
                                                  /\ Support plain text e-mail
  ANDURAS intranet security AG
  Messestraße 3 - 94036 Passau - Germany
  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0

Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB 6032
Vorstand: Marcus Junker
Vorsitzender des Aufsichtsrats: Andreas Wagner
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Switzer, David]

>-----Original Message-----
>From: Sven Anders <sven.anders@anduras.de>
>Sent: Monday, June 13, 2022 12:44 AM
>To: Switzer, David <david.switzer@intel.com>; intel-wired-lan@osuosl.org
>Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>
>Hallo Dave,
Hi Sven,

>
>thanks for your answer.
You're very welcome!

>
>I tried to disable the LLDP engine, as suggested, but it did not change
>anything.
>
>To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
>which works with ARP negotiation.
>If I'm using the "active-backup" bonding mode, it all works fine. But this is
>nearly the same as using only one interface, which works too.
>
>In my test setup, I'm using two "HPE 2530-48G-PoE+" switches
>(YA.16.10.0002), each is connected with one cable to the network card.
>My test is very simple:
>After configuring the bond, I used a continous ping to the Linux server and I
>see many drops. If I toggle the "ntuples support", the drops disappear.
>
>Can we compare the settings of your setup with mine? Maybe you missed
>something...
Just realized we missed the 2nd switch(we're working on setting that up now)

>How can I assist further?
Can you enable source pruning on the ports that are part of your bond?  There might be some confusion in the fw/sw with the two interfaces and packets are getting ignored:
ethtool --set-priv-flags <interface name> disable-source-pruning on

Let me know if that helps.  It sounds like we'll be getting our two switches later today so we can make sure we have our environment matching yours.

Have a great day!
Dave


>
>Regards
>  Sven
>
>Am 11.06.22 um 03:27 schrieb Switzer, David:
>>> -----Original Message-----
>>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
>>> Of Sven Anders
>>> Sent: Monday, June 6, 2022 11:05 PM
>>> To: intel-wired-lan@osuosl.org
>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>>
>>> Hello!
>> Hi Sven!  My apologies on taking so long to reply.
>>>
>>> Was anybody able to reproduce the problem?
>> I haven't been able to reproduce your issue yet, but with your
>> description it sounds that the issue could be related to the X710
>> hardware LLDP engine consuming the LLDP packets.  To disable the
>hardware engine, user the following command on each of the ports that
>you're using in the bond:
> >
>> ethtool --set-priv-flags <interface name> disable-fw-lldp on
>>
>> Then the LLDP packets that the bonding modules relies on will be forwarded
>by the hardware to the OS network stack.
>>
>> I hope you have a great weekend!
>> Dave
>>
>>> Do you need assistance or further information?
>>>
>>> Regards
>>>   Sven
>>>
>>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>>> Hello!
>>>>>>>
>>>>>>> This is a follow up to my question. I did not hear anything so
>>>>>>> far, but I tried to find some some information meanwhile.
>>>>>>>
>>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode
>>>>>>> forced on" maybe triggered, because I'm hitting a limit here.
>>>>>
>>>>> Yes, typically this is a response from our firmware that a table is
>>>>> full in hardware, and I'm guessing that you might be running into a
>>>>> filter
>>> limit due to using vlans?
>>>>
>>>> That's what I suspect, but I did know the hardware and firmware
>>>> insufficiently to be sure.
>>>> What I'm wondering is: Why this is only triggered if I toggle the
>>>> "ntuples
>>> support"
>>>> and not earlier when setting up the interface?
>>>>
>>>>>>> Data packets are only received sporadically. If I run the same
>>>>>>> test with only one slave port, it works without problems.
>>>>>
>>>>> And there are no counters going up in ethtool -S when you
>>>>> receive/drop
>>> packets?
>>>>
>>>> I'm not sure here. I tried to inspect the counters and it seems that
>>>> the counters are going up slower in this case, but it's difficult to
>>>> say, because there is some "noise" on the line and I do not have
>>>> direct access to the hardware at the moment.
>>>>
>>>> If you need any further information or tests, just contact me.
>>>
>>> Regards
>>>     Sven Anders
>>> --
>>>   Sven Anders                  () UTF-8 Ribbon Campaign
>>>                                                   /\ Support plain text e-mail
>>>   ANDURAS intranet security AG
>>>   Messestrasse 3 - 94036 Passau - Germany
>>>   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4
>>> 90 50-55
>>>
>>> Those who would give up essential Liberty, to purchase a little
>>> temporary Safety, deserve neither Liberty nor Safety.
>>>    - Benjamin Franklin
>>>
>>> _______________________________________________
>>> Intel-wired-lan mailing list
>>> Intel-wired-lan@osuosl.org
>>> https://eu-central-
>1.protection.sophos.com?d=osuosl.org&u=aHR0cHM6Ly9
>>>
>saXN0cy5vc3Vvc2wub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtd2lyZWQtb
>GFu&i=N
>>>
>WRkZTc5MjQzMjM2YzExNjg0ZTI4MjM4&t=WHZUTUdjNVkySTVVcWVjUEpVT
>GlkaXV3Nkx
>>>
>VQm1LSGdUOXMxeVVEQ1pyOD0=&h=4ed5ca43b14147eb8ae4891cf2a54f70
>
>
>Mit freundlichen Grüßen
>  Sven Anders
>
>--
>  Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
>                                                  /\ Support plain text e-mail
>  ANDURAS intranet security AG
>  Messestraße 3 - 94036 Passau - Germany
>  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0
>
>Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB 6032
>Vorstand: Marcus Junker
>Vorsitzender des Aufsichtsrats: Andreas Wagner
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Switzer, David]

>-----Original Message-----
>From: Switzer, David
>Sent: Thursday, June 16, 2022 1:31 PM
>To: Sven Anders <sven.anders@anduras.de>; intel-wired-lan@osuosl.org
>Subject: RE: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>
>>-----Original Message-----
>>From: Sven Anders <sven.anders@anduras.de>
>>Sent: Monday, June 13, 2022 12:44 AM
>>To: Switzer, David <david.switzer@intel.com>;
>>intel-wired-lan@osuosl.org
>>Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>
>>Hallo Dave,
>Hi Sven,

Hello Sven!
>
>>
>>thanks for your answer.
>You're very welcome!
>
>>
>>I tried to disable the LLDP engine, as suggested, but it did not change
>>anything.
>>
>>To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
>>which works with ARP negotiation.
>>If I'm using the "active-backup" bonding mode, it all works fine. But
>>this is nearly the same as using only one interface, which works too.
>>
>>In my test setup, I'm using two "HPE 2530-48G-PoE+" switches
>>(YA.16.10.0002), each is connected with one cable to the network card.
>>My test is very simple:
>>After configuring the bond, I used a continous ping to the Linux server
>>and I see many drops. If I toggle the "ntuples support", the drops disappear.
>>
>>Can we compare the settings of your setup with mine? Maybe you missed
>>something...
>Just realized we missed the 2nd switch(we're working on setting that up now)
I got my 2nd switch configured with 36 vlans on the bond, I still can't reproduce the issue.

I'm in a test environment so I don't have much of a "network" between the two switches.  I was wondering how your switches were configured and if there was anything else going on with  the network(people streaming, backups occurring, etc).

>
>>How can I assist further?
>Can you enable source pruning on the ports that are part of your bond?  There
>might be some confusion in the fw/sw with the two interfaces and packets
>are getting ignored:
>ethtool --set-priv-flags <interface name> disable-source-pruning on
>
I was reading over this and realized I contradicted myself above.  I would like to see if disabling source pruning would help.

I hope you're having a good week!
Dave

>Let me know if that helps.  It sounds like we'll be getting our two switches
>later today so we can make sure we have our environment matching yours.
>
>Have a great day!
>Dave
>
>
>>
>>Regards
>>  Sven
>>
>>Am 11.06.22 um 03:27 schrieb Switzer, David:
>>>> -----Original Message-----
>>>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
>>>> Of Sven Anders
>>>> Sent: Monday, June 6, 2022 11:05 PM
>>>> To: intel-wired-lan@osuosl.org
>>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710
>>>> hardware
>>>>
>>>> Hello!
>>> Hi Sven!  My apologies on taking so long to reply.
>>>>
>>>> Was anybody able to reproduce the problem?
>>> I haven't been able to reproduce your issue yet, but with your
>>> description it sounds that the issue could be related to the X710
>>> hardware LLDP engine consuming the LLDP packets.  To disable the
>>hardware engine, user the following command on each of the ports that
>>you're using in the bond:
>> >
>>> ethtool --set-priv-flags <interface name> disable-fw-lldp on
>>>
>>> Then the LLDP packets that the bonding modules relies on will be
>>> forwarded
>>by the hardware to the OS network stack.
>>>
>>> I hope you have a great weekend!
>>> Dave
>>>
>>>> Do you need assistance or further information?
>>>>
>>>> Regards
>>>>   Sven
>>>>
>>>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>>>> Hello!
>>>>>>>>
>>>>>>>> This is a follow up to my question. I did not hear anything so
>>>>>>>> far, but I tried to find some some information meanwhile.
>>>>>>>>
>>>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode
>>>>>>>> forced on" maybe triggered, because I'm hitting a limit here.
>>>>>>
>>>>>> Yes, typically this is a response from our firmware that a table
>>>>>> is full in hardware, and I'm guessing that you might be running
>>>>>> into a filter
>>>> limit due to using vlans?
>>>>>
>>>>> That's what I suspect, but I did know the hardware and firmware
>>>>> insufficiently to be sure.
>>>>> What I'm wondering is: Why this is only triggered if I toggle the
>>>>> "ntuples
>>>> support"
>>>>> and not earlier when setting up the interface?
>>>>>
>>>>>>>> Data packets are only received sporadically. If I run the same
>>>>>>>> test with only one slave port, it works without problems.
>>>>>>
>>>>>> And there are no counters going up in ethtool -S when you
>>>>>> receive/drop
>>>> packets?
>>>>>
>>>>> I'm not sure here. I tried to inspect the counters and it seems
>>>>> that the counters are going up slower in this case, but it's
>>>>> difficult to say, because there is some "noise" on the line and I
>>>>> do not have direct access to the hardware at the moment.
>>>>>
>>>>> If you need any further information or tests, just contact me.
>>>>
>>>> Regards
>>>>     Sven Anders
>>>> --
>>>>   Sven Anders                  () UTF-8 Ribbon Campaign
>>>>                                                   /\ Support plain text e-mail
>>>>   ANDURAS intranet security AG
>>>>   Messestrasse 3 - 94036 Passau - Germany
>>>>   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49
>>>> (0)851-4
>>>> 90 50-55
>>>>
>>>> Those who would give up essential Liberty, to purchase a little
>>>> temporary Safety, deserve neither Liberty nor Safety.
>>>>    - Benjamin Franklin
>>>>
>>>> _______________________________________________
>>>> Intel-wired-lan mailing list
>>>> Intel-wired-lan@osuosl.org
>>>> https://eu-central-
>>1.protection.sophos.com?d=osuosl.org&u=aHR0cHM6Ly9
>>>>
>>saXN0cy5vc3Vvc2wub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtd2lyZWQt
>b
>>GFu&i=N
>>>>
>>WRkZTc5MjQzMjM2YzExNjg0ZTI4MjM4&t=WHZUTUdjNVkySTVVcWVjUEpV
>T
>>GlkaXV3Nkx
>>>>
>>VQm1LSGdUOXMxeVVEQ1pyOD0=&h=4ed5ca43b14147eb8ae4891cf2a54f70
>>
>>
>>Mit freundlichen Grüßen
>>  Sven Anders
>>
>>--
>>  Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
>>                                                  /\ Support plain text
>>e-mail
>>  ANDURAS intranet security AG
>>  Messestraße 3 - 94036 Passau - Germany
>>  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0
>>
>>Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB
>>6032
>>Vorstand: Marcus Junker
>>Vorsitzender des Aufsichtsrats: Andreas Wagner
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders]

Hallo Dave,

I will setup a new test environment these days to go deeper in the
analysis. Then I can compare different kernel versions and I will
try to reproduce this issue with an small script. I will check, if
I can reproduce this even with a major linux distri (Ubuntu?).

If I have any result, I will get back to you...

Regards
  Sven

Am 21.06.22 um 23:18 schrieb Switzer, David:
> 
> 
>> -----Original Message-----
>> From: Switzer, David
>> Sent: Thursday, June 16, 2022 1:31 PM
>> To: Sven Anders <sven.anders@anduras.de>; intel-wired-lan@osuosl.org
>> Subject: RE: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>
>>> -----Original Message-----
>>> From: Sven Anders <sven.anders@anduras.de>
>>> Sent: Monday, June 13, 2022 12:44 AM
>>> To: Switzer, David <david.switzer@intel.com>;
>>> intel-wired-lan@osuosl.org
>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>>
>>> Hallo Dave,
>> Hi Sven,
> 
> Hello Sven!
>>
>>>
>>> thanks for your answer.
>> You're very welcome!
>>
>>>
>>> I tried to disable the LLDP engine, as suggested, but it did not change
>>> anything.
>>>
>>> To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
>>> which works with ARP negotiation.
>>> If I'm using the "active-backup" bonding mode, it all works fine. But
>>> this is nearly the same as using only one interface, which works too.
>>>
>>> In my test setup, I'm using two "HPE 2530-48G-PoE+" switches
>>> (YA.16.10.0002), each is connected with one cable to the network card.
>>> My test is very simple:
>>> After configuring the bond, I used a continous ping to the Linux server
>>> and I see many drops. If I toggle the "ntuples support", the drops disappear.
>>>
>>> Can we compare the settings of your setup with mine? Maybe you missed
>>> something...
>> Just realized we missed the 2nd switch(we're working on setting that up now)
> I got my 2nd switch configured with 36 vlans on the bond, I still can't reproduce the issue.
> 
> I'm in a test environment so I don't have much of a "network" between the two switches.  I was wondering how your switches were configured and if there was anything else going on with  the network(people streaming, backups occurring, etc).
> 
>>
>>> How can I assist further?
>> Can you enable source pruning on the ports that are part of your bond?  There
>> might be some confusion in the fw/sw with the two interfaces and packets
>> are getting ignored:
>> ethtool --set-priv-flags <interface name> disable-source-pruning on
>>
> I was reading over this and realized I contradicted myself above.  I would like to see if disabling source pruning would help.
> 
> I hope you're having a good week!
> Dave
> 
>> Let me know if that helps.  It sounds like we'll be getting our two switches
>> later today so we can make sure we have our environment matching yours.
>>
>> Have a great day!
>> Dave
>>
>>
>>>
>>> Regards
>>>   Sven
>>>
>>> Am 11.06.22 um 03:27 schrieb Switzer, David:
>>>>> -----Original Message-----
>>>>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
>>>>> Of Sven Anders
>>>>> Sent: Monday, June 6, 2022 11:05 PM
>>>>> To: intel-wired-lan@osuosl.org
>>>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710
>>>>> hardware
>>>>>
>>>>> Hello!
>>>> Hi Sven!  My apologies on taking so long to reply.
>>>>>
>>>>> Was anybody able to reproduce the problem?
>>>> I haven't been able to reproduce your issue yet, but with your
>>>> description it sounds that the issue could be related to the X710
>>>> hardware LLDP engine consuming the LLDP packets.  To disable the
>>> hardware engine, user the following command on each of the ports that
>>> you're using in the bond:
>>>>
>>>> ethtool --set-priv-flags <interface name> disable-fw-lldp on
>>>>
>>>> Then the LLDP packets that the bonding modules relies on will be
>>>> forwarded
>>> by the hardware to the OS network stack.
>>>>
>>>> I hope you have a great weekend!
>>>> Dave
>>>>
>>>>> Do you need assistance or further information?
>>>>>
>>>>> Regards
>>>>>    Sven
>>>>>
>>>>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> This is a follow up to my question. I did not hear anything so
>>>>>>>>> far, but I tried to find some some information meanwhile.
>>>>>>>>>
>>>>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode
>>>>>>>>> forced on" maybe triggered, because I'm hitting a limit here.
>>>>>>>
>>>>>>> Yes, typically this is a response from our firmware that a table
>>>>>>> is full in hardware, and I'm guessing that you might be running
>>>>>>> into a filter
>>>>> limit due to using vlans?
>>>>>>
>>>>>> That's what I suspect, but I did know the hardware and firmware
>>>>>> insufficiently to be sure.
>>>>>> What I'm wondering is: Why this is only triggered if I toggle the
>>>>>> "ntuples
>>>>> support"
>>>>>> and not earlier when setting up the interface?
>>>>>>
>>>>>>>>> Data packets are only received sporadically. If I run the same
>>>>>>>>> test with only one slave port, it works without problems.
>>>>>>>
>>>>>>> And there are no counters going up in ethtool -S when you
>>>>>>> receive/drop
>>>>> packets?
>>>>>>
>>>>>> I'm not sure here. I tried to inspect the counters and it seems
>>>>>> that the counters are going up slower in this case, but it's
>>>>>> difficult to say, because there is some "noise" on the line and I
>>>>>> do not have direct access to the hardware at the moment.
>>>>>>
>>>>>> If you need any further information or tests, just contact me.
>>>>>
>>>>> Regards
>>>>>      Sven Anders
>>>>> --
>>>>>    Sven Anders                  () UTF-8 Ribbon Campaign
>>>>>                                                    /\ Support plain text e-mail
>>>>>    ANDURAS intranet security AG
>>>>>    Messestrasse 3 - 94036 Passau - Germany
>>>>>    Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49
>>>>> (0)851-4
>>>>> 90 50-55
>>>>>
>>>>> Those who would give up essential Liberty, to purchase a little
>>>>> temporary Safety, deserve neither Liberty nor Safety.
>>>>>     - Benjamin Franklin
>>>>>
>>>>> _______________________________________________
>>>>> Intel-wired-lan mailing list
>>>>> Intel-wired-lan@osuosl.org
>>>>> https://eu-central-
>>> 1.protection.sophos.com?d=osuosl.org&u=aHR0cHM6Ly9
>>>>>
>>> saXN0cy5vc3Vvc2wub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtd2lyZWQt
>> b
>>> GFu&i=N
>>>>>
>>> WRkZTc5MjQzMjM2YzExNjg0ZTI4MjM4&t=WHZUTUdjNVkySTVVcWVjUEpV
>> T
>>> GlkaXV3Nkx
>>>>>
>>> VQm1LSGdUOXMxeVVEQ1pyOD0=&h=4ed5ca43b14147eb8ae4891cf2a54f70
>>>
>>>
>>> Mit freundlichen Grüßen
>>>   Sven Anders
>>>
>>> --
>>>   Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
>>>                                                   /\ Support plain text
>>> e-mail
>>>   ANDURAS intranet security AG
>>>   Messestraße 3 - 94036 Passau - Germany
>>>   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0
>>>
>>> Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB
>>> 6032
>>> Vorstand: Marcus Junker
>>> Vorsitzender des Aufsichtsrats: Andreas Wagner


Mit freundlichen Grüßen
  Sven Anders

-- 
  Sven Anders <anders@anduras.de>                 () UTF-8 Ribbon Campaign
                                                  /\ Support plain text e-mail
  ANDURAS intranet security AG
  Messestraße 3 - 94036 Passau - Germany
  Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0

Rechtsform: Aktiengesellschaft - Sitz: Passau - Amtsgericht: Passau HRB 6032
Vorstand: Marcus Junker
Vorsitzender des Aufsichtsrats: Andreas Wagner
_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan

Re: [Intel-wired-lan] Bonding problem on Intel X710 hardware

[Sven Anders]

Hello Dave,

I want so report back that I solved the problem.
The problem was not in the i40e driver.

Explanation:
We upgraded a server with the new 10G (quad) network card.
The server was working with kernel in version 4.14.238.
Due to the new network card, we were forced to upgrade to a newer kernel and we used 5.15.38.
The network configuration stayed the same, but we had the reported problem.

Due to a mysterious fix with an older version of the kernel, there was a part of
the network configuration that set the mac address on all vlans.
This was the cause of the problem. The older kernel ignored this settings but
the newer kernel performed (correctly) some actions on it (and maybe sent some instructions
to the network card). Forcing the promiscous mode by triggering the error must disabled some
previous settings of the network card and made it working again.

Nevertheless, thanks for your time testing it. It helped me to eliminate the driver as the
cause and forced me to dig deeper on my side.
Removing the part, which was setting the mac address, elimiated the errounous behaviour
and the network card is not working.

Regards
   Sven

Am 21.06.22 um 23:18 schrieb Switzer, David:
> 
>> -----Original Message-----
>> From: Switzer, David
>> Sent: Thursday, June 16, 2022 1:31 PM
>> To: Sven Anders <sven.anders@anduras.de>; intel-wired-lan@osuosl.org
>> Subject: RE: [Intel-wired-lan] Bonding problem on Intel X710 hardware
>>
>>> Hallo Dave,
>> Hi Sven,
> 
> Hello Sven!
>>
>>>
>>> thanks for your answer.
>> You're very welcome!
>>
>>>
>>> I tried to disable the LLDP engine, as suggested, but it did not change
>>> anything.
>>>
>>> To remark: I do not use LACP, I'm trying to use "Adaptive Load Balancing"
>>> which works with ARP negotiation.
>>> If I'm using the "active-backup" bonding mode, it all works fine. But
>>> this is nearly the same as using only one interface, which works too.
>>>
>>> In my test setup, I'm using two "HPE 2530-48G-PoE+" switches
>>> (YA.16.10.0002), each is connected with one cable to the network card.
>>> My test is very simple:
>>> After configuring the bond, I used a continous ping to the Linux server
>>> and I see many drops. If I toggle the "ntuples support", the drops disappear.
>>>
>>> Can we compare the settings of your setup with mine? Maybe you missed
>>> something...
>> Just realized we missed the 2nd switch(we're working on setting that up now)
> I got my 2nd switch configured with 36 vlans on the bond, I still can't reproduce the issue.
> 
> I'm in a test environment so I don't have much of a "network" between the two switches.  I was wondering how your switches were configured and if there was anything else going on with  the network(people streaming, backups occurring, etc).
> 
>>
>>> How can I assist further?
>> Can you enable source pruning on the ports that are part of your bond?  There
>> might be some confusion in the fw/sw with the two interfaces and packets
>> are getting ignored:
>> ethtool --set-priv-flags <interface name> disable-source-pruning on
>>
> I was reading over this and realized I contradicted myself above.  I would like to see if disabling source pruning would help.
> 
> I hope you're having a good week!
> Dave
> 
>> Let me know if that helps.  It sounds like we'll be getting our two switches
>> later today so we can make sure we have our environment matching yours.
>>
>> Have a great day!
>> Dave
>>
>>
>>>
>>> Regards
>>>   Sven
>>>
>>> Am 11.06.22 um 03:27 schrieb Switzer, David:
>>>>> -----Original Message-----
>>>>> From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf
>>>>> Of Sven Anders
>>>>> Sent: Monday, June 6, 2022 11:05 PM
>>>>> To: intel-wired-lan@osuosl.org
>>>>> Subject: Re: [Intel-wired-lan] Bonding problem on Intel X710
>>>>> hardware
>>>>>
>>>>> Hello!
>>>> Hi Sven!  My apologies on taking so long to reply.
>>>>>
>>>>> Was anybody able to reproduce the problem?
>>>> I haven't been able to reproduce your issue yet, but with your
>>>> description it sounds that the issue could be related to the X710
>>>> hardware LLDP engine consuming the LLDP packets.  To disable the
>>> hardware engine, user the following command on each of the ports that
>>> you're using in the bond:
>>>>
>>>> ethtool --set-priv-flags <interface name> disable-fw-lldp on
>>>>
>>>> Then the LLDP packets that the bonding modules relies on will be
>>>> forwarded
>>> by the hardware to the OS network stack.
>>>>
>>>> I hope you have a great weekend!
>>>> Dave
>>>>
>>>>> Do you need assistance or further information?
>>>>>
>>>>> Regards
>>>>>    Sven
>>>>>
>>>>> Am 30.05.22 um 17:53 schrieb Sven Anders:
>>>>>>>>> Hello!
>>>>>>>>>
>>>>>>>>> This is a follow up to my question. I did not hear anything so
>>>>>>>>> far, but I tried to find some some information meanwhile.
>>>>>>>>>
>>>>>>>>> I've got a guess from somebody, that the error message "Error
>>>>>>>>> I40E_AQ_RC_EINVAL adding RX filters on PF, promiscuous mode
>>>>>>>>> forced on" maybe triggered, because I'm hitting a limit here.
>>>>>>>
>>>>>>> Yes, typically this is a response from our firmware that a table
>>>>>>> is full in hardware, and I'm guessing that you might be running
>>>>>>> into a filter
>>>>> limit due to using vlans?
>>>>>>
>>>>>> That's what I suspect, but I did know the hardware and firmware
>>>>>> insufficiently to be sure.
>>>>>> What I'm wondering is: Why this is only triggered if I toggle the
>>>>>> "ntuples
>>>>> support"
>>>>>> and not earlier when setting up the interface?
>>>>>>
>>>>>>>>> Data packets are only received sporadically. If I run the same
>>>>>>>>> test with only one slave port, it works without problems.
>>>>>>>
>>>>>>> And there are no counters going up in ethtool -S when you
>>>>>>> receive/drop
>>>>> packets?
>>>>>>
>>>>>> I'm not sure here. I tried to inspect the counters and it seems
>>>>>> that the counters are going up slower in this case, but it's
>>>>>> difficult to say, because there is some "noise" on the line and I
>>>>>> do not have direct access to the hardware at the moment.
>>>>>>
>>>>>> If you need any further information or tests, just contact me.
>>>>>
>>>>> Regards
>>>>>      Sven Anders

Regards
   Sven Anders

-- 
   Sven Anders                  () UTF-8 Ribbon Campaign
                                                   /\ Support plain text e-mail
   ANDURAS intranet security AG
   Messestrasse 3 - 94036 Passau - Germany
   Web: www.anduras.de - Tel: +49 (0)851-4 90 50-0 - Fax: +49 (0)851-4 90 50-55

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
    - Benjamin Franklin

_______________________________________________
Intel-wired-lan mailing list
Intel-wired-lan@osuosl.org
https://lists.osuosl.org/mailman/listinfo/intel-wired-lan