Instability of i40e driver on 4.9 kernel

Instability of i40e driver on 4.9 kernel

[Pavlos Parissis]

[-- Attachment #1.1: Type: text/plain, Size: 1573 bytes --]

Hi all,

We have been running 4.9 kernels for several months on CentOS 7.3 and for few weeks on CentOS 7.4,
and, after we replaced 10GbE cobber cards(X540-AT2 with ixgbe driver) with X710 10GbE SFP cards
using i40e driver, we noticed sever instabilities on our servers.

On several servers the links were marked down and up again, without any obvious reasons expect a lot
of errors on kernel.log. We run Bird Internet daemon on our servers in order to establish BGP
peerings with routers and we have observed flapping on BGP peerings. At the same time we had BGP
peering stabilities issues we had kernel errors. We decided to go back to 3.10 kernel from CentOS,
but that process wasn't smooth as latest firmware gave us problems with speed detection. We rolled
back to two version old and speed detection issue was resolved. We have been running 3.10 several
weeks without any problems. Even we want certain functionality from kernel 4.9, we decided to switch
back to 3.10 as stability of our systems has higher priority.

I need to mention that in all occurrences of the issue we didn't see any anomalies, such DDOS
attacks and etc.

I have opened https://communities.intel.com/message/501682#501682 and there you can find all the
error messages and other information.

Since we noticed the issues, I have been following netdev ML and I know that there are a lot of
improvements/patched queued up for 4.14 and I am hoping those patches fix our issue and most
importantly are sent to linux-stable for inclusion in 4.9 kernel.

Cheers,
Pavlos



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Pavlos Parissis]

Hi all,

We have been running 4.9 kernels for several months on CentOS 7.3 and for few weeks on CentOS 7.4,
and, after we replaced 10GbE cobber cards(X540-AT2 with ixgbe driver) with X710 10GbE SFP cards
using i40e driver, we noticed sever instabilities on our servers.

On several servers the links were marked down and up again, without any obvious reasons expect a lot
of errors on kernel.log. We run Bird Internet daemon on our servers in order to establish BGP
peerings with routers and we have observed flapping on BGP peerings. At the same time we had BGP
peering stabilities issues we had kernel errors. We decided to go back to 3.10 kernel from CentOS,
but that process wasn't smooth as latest firmware gave us problems with speed detection. We rolled
back to two version old and speed detection issue was resolved. We have been running 3.10 several
weeks without any problems. Even we want certain functionality from kernel 4.9, we decided to switch
back to 3.10 as stability of our systems has higher priority.

I need to mention that in all occurrences of the issue we didn't see any anomalies, such DDOS
attacks and etc.

I have opened https://communities.intel.com/message/501682#501682 and there you can find all the
error messages and other information.

Since we noticed the issues, I have been following netdev ML and I know that there are a lot of
improvements/patched queued up for 4.14 and I am hoping those patches fix our issue and most
importantly are sent to linux-stable for inclusion in 4.9 kernel.

Cheers,
Pavlos


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20171020/5c41f224/attachment.asc>

RE: [Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Fujinaka, Todd]

You picked a bunch of places to post this, and you really should've used a different place: e1000-devel@lists.sourceforge.net

Also, since you flagged the "communities" post as "answered", you're not likely to get any follow-up. The Intel communities are also not monitored as much by the wired networking people at Intel.

Please let us know if you have any specific issues, and please provide exact reproduction steps so we can investigate your issues, and please use e1000-devel.

Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka@intel.com


-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces@osuosl.org] On Behalf Of Pavlos Parissis
Sent: Thursday, October 19, 2017 4:03 PM
To: netdev@vger.kernel.org; intel-wired-lan@lists.osuosl.org
Subject: [Intel-wired-lan] Instability of i40e driver on 4.9 kernel

Hi all,

We have been running 4.9 kernels for several months on CentOS 7.3 and for few weeks on CentOS 7.4, and, after we replaced 10GbE cobber cards(X540-AT2 with ixgbe driver) with X710 10GbE SFP cards using i40e driver, we noticed sever instabilities on our servers.

On several servers the links were marked down and up again, without any obvious reasons expect a lot of errors on kernel.log. We run Bird Internet daemon on our servers in order to establish BGP peerings with routers and we have observed flapping on BGP peerings. At the same time we had BGP peering stabilities issues we had kernel errors. We decided to go back to 3.10 kernel from CentOS, but that process wasn't smooth as latest firmware gave us problems with speed detection. We rolled back to two version old and speed detection issue was resolved. We have been running 3.10 several weeks without any problems. Even we want certain functionality from kernel 4.9, we decided to switch back to 3.10 as stability of our systems has higher priority.

I need to mention that in all occurrences of the issue we didn't see any anomalies, such DDOS attacks and etc.

I have opened https://communities.intel.com/message/501682#501682 and there you can find all the error messages and other information.

Since we noticed the issues, I have been following netdev ML and I know that there are a lot of improvements/patched queued up for 4.14 and I am hoping those patches fix our issue and most importantly are sent to linux-stable for inclusion in 4.9 kernel.

Cheers,
Pavlos

[Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Fujinaka, Todd]

You picked a bunch of places to post this, and you really should've used a different place: e1000-devel at lists.sourceforge.net

Also, since you flagged the "communities" post as "answered", you're not likely to get any follow-up. The Intel communities are also not monitored as much by the wired networking people at Intel.

Please let us know if you have any specific issues, and please provide exact reproduction steps so we can investigate your issues, and please use e1000-devel.

Todd Fujinaka
Software Application Engineer
Datacenter Engineering Group
Intel Corporation
todd.fujinaka at intel.com


-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at osuosl.org] On Behalf Of Pavlos Parissis
Sent: Thursday, October 19, 2017 4:03 PM
To: netdev@vger.kernel.org; intel-wired-lan at lists.osuosl.org
Subject: [Intel-wired-lan] Instability of i40e driver on 4.9 kernel

Hi all,

We have been running 4.9 kernels for several months on CentOS 7.3 and for few weeks on CentOS 7.4, and, after we replaced 10GbE cobber cards(X540-AT2 with ixgbe driver) with X710 10GbE SFP cards using i40e driver, we noticed sever instabilities on our servers.

On several servers the links were marked down and up again, without any obvious reasons expect a lot of errors on kernel.log. We run Bird Internet daemon on our servers in order to establish BGP peerings with routers and we have observed flapping on BGP peerings. At the same time we had BGP peering stabilities issues we had kernel errors. We decided to go back to 3.10 kernel from CentOS, but that process wasn't smooth as latest firmware gave us problems with speed detection. We rolled back to two version old and speed detection issue was resolved. We have been running 3.10 several weeks without any problems. Even we want certain functionality from kernel 4.9, we decided to switch back to 3.10 as stability of our systems has higher priority.

I need to mention that in all occurrences of the issue we didn't see any anomalies, such DDOS attacks and etc.

I have opened https://communities.intel.com/message/501682#501682 and there you can find all the error messages and other information.

Since we noticed the issues, I have been following netdev ML and I know that there are a lot of improvements/patched queued up for 4.14 and I am hoping those patches fix our issue and most importantly are sent to linux-stable for inclusion in 4.9 kernel.

Cheers,
Pavlos

Re: [Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Pavlos Parissis]

[-- Attachment #1.1: Type: text/plain, Size: 1048 bytes --]

On 21/10/2017 02:07 πμ, Fujinaka, Todd wrote:
> You picked a bunch of places to post this, and you really should've used a
> different place: e1000-devel@lists.sourceforge.net
> 

Just subscribed to that ML and mailed about it.

> Also, since you flagged the "communities" post as "answered", you're not likely
> to get any follow-up. The Intel communities are also not monitored as much by
> the wired networking people at Intel.

I don't see it as "answered" when I visit the page, maybe the fact I have replied
with extra information is confusing something. Anyway, it isn't that important
since the "communities" posts aren't monitored by Intel people, which makes sense
as it is very time consuming to monitor mailing lists and web forums at the same time.

> 
> Please let us know if you have any specific issues, and please provide exact
> reproduction steps so we can investigate your issues, and please use
> e1000-devel.

I hope the information I provided in my mail to e1000-devel is enough.

Thanks,
Pavlos


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Pavlos Parissis]

On 21/10/2017 02:07 ??, Fujinaka, Todd wrote:
> You picked a bunch of places to post this, and you really should've used a
> different place: e1000-devel at lists.sourceforge.net
> 

Just subscribed to that ML and mailed about it.

> Also, since you flagged the "communities" post as "answered", you're not likely
> to get any follow-up. The Intel communities are also not monitored as much by
> the wired networking people at Intel.

I don't see it as "answered" when I visit the page, maybe the fact I have replied
with extra information is confusing something. Anyway, it isn't that important
since the "communities" posts aren't monitored by Intel people, which makes sense
as it is very time consuming to monitor mailing lists and web forums at the same time.

> 
> Please let us know if you have any specific issues, and please provide exact
> reproduction steps so we can investigate your issues, and please use
> e1000-devel.

I hope the information I provided in my mail to e1000-devel is enough.

Thanks,
Pavlos

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.osuosl.org/pipermail/intel-wired-lan/attachments/20171025/1ae4d80a/attachment-0001.asc>

Re: [Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[Paweł Staszewski]

e1000 sourceforge is a bad place to make anything with Your problems


Just checked this now if something changed :)

But when I post reply to some bug that i was have same problem somebody 
closed the ticket and delete my message :)


So rly :)




W dniu 2017-10-25 o 23:49, Pavlos Parissis pisze:
> On 21/10/2017 02:07 πμ, Fujinaka, Todd wrote:
>> You picked a bunch of places to post this, and you really should've used a
>> different place: e1000-devel@lists.sourceforge.net
>>
> Just subscribed to that ML and mailed about it.
>
>> Also, since you flagged the "communities" post as "answered", you're not likely
>> to get any follow-up. The Intel communities are also not monitored as much by
>> the wired networking people at Intel.
> I don't see it as "answered" when I visit the page, maybe the fact I have replied
> with extra information is confusing something. Anyway, it isn't that important
> since the "communities" posts aren't monitored by Intel people, which makes sense
> as it is very time consuming to monitor mailing lists and web forums at the same time.
>
>> Please let us know if you have any specific issues, and please provide exact
>> reproduction steps so we can investigate your issues, and please use
>> e1000-devel.
> I hope the information I provided in my mail to e1000-devel is enough.
>
> Thanks,
> Pavlos
>

[Intel-wired-lan] Instability of i40e driver on 4.9 kernel

[=?unknown-8bit?q?Pawe=C5=82?= Staszewski]

e1000 sourceforge is a bad place to make anything with Your problems


Just checked this now if something changed :)

But when I post reply to some bug that i was have same problem somebody 
closed the ticket and delete my message :)


So rly :)




W dniu 2017-10-25 o?23:49, Pavlos Parissis pisze:
> On 21/10/2017 02:07 ??, Fujinaka, Todd wrote:
>> You picked a bunch of places to post this, and you really should've used a
>> different place: e1000-devel at lists.sourceforge.net
>>
> Just subscribed to that ML and mailed about it.
>
>> Also, since you flagged the "communities" post as "answered", you're not likely
>> to get any follow-up. The Intel communities are also not monitored as much by
>> the wired networking people at Intel.
> I don't see it as "answered" when I visit the page, maybe the fact I have replied
> with extra information is confusing something. Anyway, it isn't that important
> since the "communities" posts aren't monitored by Intel people, which makes sense
> as it is very time consuming to monitor mailing lists and web forums at the same time.
>
>> Please let us know if you have any specific issues, and please provide exact
>> reproduction steps so we can investigate your issues, and please use
>> e1000-devel.
> I hope the information I provided in my mail to e1000-devel is enough.
>
> Thanks,
> Pavlos
>