Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

All of lore.kernel.org
 help / color / mirror / Atom feed

* Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
@ 2021-08-05 20:53 Martin Zaharinov
  2021-08-06  4:40 ` Greg KH
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-05 20:53 UTC (permalink / raw)
  To: netdev, gregkh, Eric Dumazet

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .
And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected


Martin

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-05 20:53 Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected Martin Zaharinov
@ 2021-08-06  4:40 ` Greg KH
  2021-08-06  5:40   ` Martin Zaharinov
  2021-08-08 15:14   ` Martin Zaharinov
  0 siblings, 2 replies; 23+ messages in thread
From: Greg KH @ 2021-08-06  4:40 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: netdev, Eric Dumazet

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
> Hi Net dev team
> 
> 
> Please check this error :
> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
> 
> But not find any solution.
> 
> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
> 
> Server is work fine users is down/up 500+ users .
> But in one moment server make spike and affect other vlans in same server .
> And in accel I see many row with this error.
> 
> Is there options to find and fix this bug.
> 
> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
> 
> 
> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

thanks,

greg k-h

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-06  4:40 ` Greg KH
@ 2021-08-06  5:40   ` Martin Zaharinov
  2021-08-08 15:14   ` Martin Zaharinov
  1 sibling, 0 replies; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-06  5:40 UTC (permalink / raw)
  To: Greg KH; +Cc: netdev, Eric Dumazet

Hi Greg

Latest kernel 5.13.8.

I try old version from 5.10 to 5.13 and its same error.

Martin

> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>> Hi Net dev team
>> 
>> 
>> Please check this error :
>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>> 
>> But not find any solution.
>> 
>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>> 
>> Server is work fine users is down/up 500+ users .
>> But in one moment server make spike and affect other vlans in same server .
>> And in accel I see many row with this error.
>> 
>> Is there options to find and fix this bug.
>> 
>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>> 
>> 
>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> 
> These are userspace error messages, not kernel messages.
> 
> What kernel version are you using?
> 
> thanks,
> 
> greg k-h


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-06  4:40 ` Greg KH
  2021-08-06  5:40   ` Martin Zaharinov
@ 2021-08-08 15:14   ` Martin Zaharinov
  2021-08-08 15:23     ` Pali Rohár
  1 sibling, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-08 15:14 UTC (permalink / raw)
  To: Greg KH; +Cc: netdev, Eric Dumazet, pali

Add Pali Rohár,

If have any idea .

Martin

> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>> Hi Net dev team
>> 
>> 
>> Please check this error :
>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>> 
>> But not find any solution.
>> 
>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>> 
>> Server is work fine users is down/up 500+ users .
>> But in one moment server make spike and affect other vlans in same server .
>> And in accel I see many row with this error.
>> 
>> Is there options to find and fix this bug.
>> 
>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>> 
>> 
>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> 
> These are userspace error messages, not kernel messages.
> 
> What kernel version are you using?
> 
> thanks,
> 
> greg k-h


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-08 15:14   ` Martin Zaharinov
@ 2021-08-08 15:23     ` Pali Rohár
  2021-08-08 15:29       ` Martin Zaharinov
  0 siblings, 1 reply; 23+ messages in thread
From: Pali Rohár @ 2021-08-08 15:23 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Greg KH, netdev, Eric Dumazet

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
> Add Pali Rohár,
> 
> If have any idea .
> 
> Martin
> 
> > On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
> > 
> > On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
> >> Hi Net dev team
> >> 
> >> 
> >> Please check this error :
> >> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
> >> 
> >> But not find any solution.
> >> 
> >> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
> >> 
> >> Server is work fine users is down/up 500+ users .
> >> But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

> >> And in accel I see many row with this error.
> >> 
> >> Is there options to find and fix this bug.
> >> 
> >> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
> >> 
> >> 
> >> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> > 
> > These are userspace error messages, not kernel messages.
> > 
> > What kernel version are you using?

Yes, we need to know, what kernel version are you using.

> > thanks,
> > 
> > greg k-h
> 

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-08 15:23     ` Pali Rohár
@ 2021-08-08 15:29       ` Martin Zaharinov
  2021-08-09 15:15         ` Pali Rohár
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-08 15:29 UTC (permalink / raw)
  To: Pali Rohár; +Cc: Greg KH, netdev, Eric Dumazet

Hi Pali

Kernel 5.13.8


The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12

I use accel-pppd daemon (not pppd) .

And yes after users started to connecting .

When system boot and connect first time all user connect without any problem .
In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.

Martin

> On 8 Aug 2021, at 18:23, Pali Rohár <pali@kernel.org> wrote:
> 
> Hello!
> 
> On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
>> Add Pali Rohár,
>> 
>> If have any idea .
>> 
>> Martin
>> 
>>> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
>>> 
>>> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>>>> Hi Net dev team
>>>> 
>>>> 
>>>> Please check this error :
>>>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>>>> 
>>>> But not find any solution.
>>>> 
>>>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>>>> 
>>>> Server is work fine users is down/up 500+ users .
>>>> But in one moment server make spike and affect other vlans in same server .
> 
> When this error started to happen? After kernel upgrade? After pppd
> upgrade? Or after system upgrade? Or when more users started to
> connecting?
> 
>>>> And in accel I see many row with this error.
>>>> 
>>>> Is there options to find and fix this bug.
>>>> 
>>>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>>>> 
>>>> 
>>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> 
>>> These are userspace error messages, not kernel messages.
>>> 
>>> What kernel version are you using?
> 
> Yes, we need to know, what kernel version are you using.
> 
>>> thanks,
>>> 
>>> greg k-h
>> 
> 
> And also another question, what version of pppd daemon are you using?
> 
> Also, are you able to dump state of ppp channels and ppp units? It is
> needed to know to which tty device, file descriptor (or socket
> extension) is (or should be) particular ppp channel bounded.


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-08 15:29       ` Martin Zaharinov
@ 2021-08-09 15:15         ` Pali Rohár
  2021-08-10 18:27           ` Martin Zaharinov
  2021-08-11 11:10           ` Martin Zaharinov
  0 siblings, 2 replies; 23+ messages in thread
From: Pali Rohár @ 2021-08-09 15:15 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Greg KH, netdev, Eric Dumazet

On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:
> Hi Pali
> 
> Kernel 5.13.8
> 
> 
> The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12
> 
> I use accel-pppd daemon (not pppd) .

I'm not using accel-pppd, so cannot help here.

I would suggest to try "git bisect" kernel version which started to be
problematic for accel-pppd.

Providing state of ppp channels and ppp units could help to debug this
issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
process which has ppp file descriptors can retrieve and dump this
information.

> And yes after users started to connecting .
> 
> When system boot and connect first time all user connect without any problem .
> In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
> And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.
> 
> Martin
> 
> > On 8 Aug 2021, at 18:23, Pali Rohár <pali@kernel.org> wrote:
> > 
> > Hello!
> > 
> > On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
> >> Add Pali Rohár,
> >> 
> >> If have any idea .
> >> 
> >> Martin
> >> 
> >>> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
> >>> 
> >>> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
> >>>> Hi Net dev team
> >>>> 
> >>>> 
> >>>> Please check this error :
> >>>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
> >>>> 
> >>>> But not find any solution.
> >>>> 
> >>>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
> >>>> 
> >>>> Server is work fine users is down/up 500+ users .
> >>>> But in one moment server make spike and affect other vlans in same server .
> > 
> > When this error started to happen? After kernel upgrade? After pppd
> > upgrade? Or after system upgrade? Or when more users started to
> > connecting?
> > 
> >>>> And in accel I see many row with this error.
> >>>> 
> >>>> Is there options to find and fix this bug.
> >>>> 
> >>>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
> >>>> 
> >>>> 
> >>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> 
> >>> These are userspace error messages, not kernel messages.
> >>> 
> >>> What kernel version are you using?
> > 
> > Yes, we need to know, what kernel version are you using.
> > 
> >>> thanks,
> >>> 
> >>> greg k-h
> >> 
> > 
> > And also another question, what version of pppd daemon are you using?
> > 
> > Also, are you able to dump state of ppp channels and ppp units? It is
> > needed to know to which tty device, file descriptor (or socket
> > extension) is (or should be) particular ppp channel bounded.
> 

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-09 15:15         ` Pali Rohár
@ 2021-08-10 18:27           ` Martin Zaharinov
  2021-08-11 16:40             ` Guillaume Nault
  2021-08-11 11:10           ` Martin Zaharinov
  1 sibling, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-10 18:27 UTC (permalink / raw)
  To: Pali Rohár; +Cc: Greg KH, netdev, Eric Dumazet, Guillaume Nault

Add Guillaume Nault

> On 9 Aug 2021, at 18:15, Pali Rohár <pali@kernel.org> wrote:
> 
> On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:
>> Hi Pali
>> 
>> Kernel 5.13.8
>> 
>> 
>> The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12
>> 
>> I use accel-pppd daemon (not pppd) .
> 
> I'm not using accel-pppd, so cannot help here.
> 
> I would suggest to try "git bisect" kernel version which started to be
> problematic for accel-pppd.
> 
> Providing state of ppp channels and ppp units could help to debug this
> issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
> process which has ppp file descriptors can retrieve and dump this
> information.
> 
>> And yes after users started to connecting .
>> 
>> When system boot and connect first time all user connect without any problem .
>> In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
>> And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.
>> 
>> Martin
>> 
>>> On 8 Aug 2021, at 18:23, Pali Rohár <pali@kernel.org> wrote:
>>> 
>>> Hello!
>>> 
>>> On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
>>>> Add Pali Rohár,
>>>> 
>>>> If have any idea .
>>>> 
>>>> Martin
>>>> 
>>>>> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
>>>>> 
>>>>> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>>>>>> Hi Net dev team
>>>>>> 
>>>>>> 
>>>>>> Please check this error :
>>>>>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>>>>>> 
>>>>>> But not find any solution.
>>>>>> 
>>>>>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>>>>>> 
>>>>>> Server is work fine users is down/up 500+ users .
>>>>>> But in one moment server make spike and affect other vlans in same server .
>>> 
>>> When this error started to happen? After kernel upgrade? After pppd
>>> upgrade? Or after system upgrade? Or when more users started to
>>> connecting?
>>> 
>>>>>> And in accel I see many row with this error.
>>>>>> 
>>>>>> Is there options to find and fix this bug.
>>>>>> 
>>>>>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>>>>>> 
>>>>>> 
>>>>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>> 
>>>>> These are userspace error messages, not kernel messages.
>>>>> 
>>>>> What kernel version are you using?
>>> 
>>> Yes, we need to know, what kernel version are you using.
>>> 
>>>>> thanks,
>>>>> 
>>>>> greg k-h
>>>> 
>>> 
>>> And also another question, what version of pppd daemon are you using?
>>> 
>>> Also, are you able to dump state of ppp channels and ppp units? It is
>>> needed to know to which tty device, file descriptor (or socket
>>> extension) is (or should be) particular ppp channel bounded.
>> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-10 18:27           ` Martin Zaharinov
@ 2021-08-11 16:40             ` Guillaume Nault
  0 siblings, 0 replies; 23+ messages in thread
From: Guillaume Nault @ 2021-08-11 16:40 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

On Tue, Aug 10, 2021 at 09:27:14PM +0300, Martin Zaharinov wrote:
> Add Guillaume Nault
> 
> > On 9 Aug 2021, at 18:15, Pali Rohár <pali@kernel.org> wrote:
> > 
> > On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:
> >>>>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>>>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

The PPPIOCCONNECT ioctl returns -ENOTCONN if the ppp channel has been
unregistered.

From a user space point of view, this means that accel-ppp establishes
PPPoE sessions, starts negociating PPP connection parameters on top of
them (LCP and authentication) and finally the PPPoE sessions get
disconnected before accel-ppp connects them to ppp units (units are
roughly the "pppX" network devices).

Unregistration of PPPoE channels can happen for the following reasons:

  * Changing some parameters of the network interface used by the
    PPPoE connection: MAC address, MTU, bringing the device down.

  * Reception of a PADT (PPPoE disconnection message sent from the peer).

  * Closing the PPPoE socket.

  * Re-connecting a PPPoE socket with a different session ID (this
    unregisters the previous channel and creates a new one, so that
    shouldn't be the problem you're facing here).

Given that this seems to affect all PPPoE connections, I guess
something happened to the underlying network interface (1st bullet
point).


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-09 15:15         ` Pali Rohár
  2021-08-10 18:27           ` Martin Zaharinov
@ 2021-08-11 11:10           ` Martin Zaharinov
  2021-08-11 16:48             ` Guillaume Nault
  1 sibling, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-08-11 11:10 UTC (permalink / raw)
  To: Pali Rohár, Guillaume Nault; +Cc: Greg KH, netdev, Eric Dumazet

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

Martin

> On 9 Aug 2021, at 18:15, Pali Rohár <pali@kernel.org> wrote:
> 
> On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:
>> Hi Pali
>> 
>> Kernel 5.13.8
>> 
>> 
>> The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12
>> 
>> I use accel-pppd daemon (not pppd) .
> 
> I'm not using accel-pppd, so cannot help here.
> 
> I would suggest to try "git bisect" kernel version which started to be
> problematic for accel-pppd.
> 
> Providing state of ppp channels and ppp units could help to debug this
> issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
> process which has ppp file descriptors can retrieve and dump this
> information.
> 
>> And yes after users started to connecting .
>> 
>> When system boot and connect first time all user connect without any problem .
>> In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
>> And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.
>> 
>> Martin
>> 
>>> On 8 Aug 2021, at 18:23, Pali Rohár <pali@kernel.org> wrote:
>>> 
>>> Hello!
>>> 
>>> On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:
>>>> Add Pali Rohár,
>>>> 
>>>> If have any idea .
>>>> 
>>>> Martin
>>>> 
>>>>> On 6 Aug 2021, at 7:40, Greg KH <gregkh@linuxfoundation.org> wrote:
>>>>> 
>>>>> On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:
>>>>>> Hi Net dev team
>>>>>> 
>>>>>> 
>>>>>> Please check this error :
>>>>>> Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html
>>>>>> 
>>>>>> But not find any solution.
>>>>>> 
>>>>>> Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.
>>>>>> 
>>>>>> Server is work fine users is down/up 500+ users .
>>>>>> But in one moment server make spike and affect other vlans in same server .
>>> 
>>> When this error started to happen? After kernel upgrade? After pppd
>>> upgrade? Or after system upgrade? Or when more users started to
>>> connecting?
>>> 
>>>>>> And in accel I see many row with this error.
>>>>>> 
>>>>>> Is there options to find and fix this bug.
>>>>>> 
>>>>>> With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.
>>>>>> 
>>>>>> 
>>>>>> [2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>>> [2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>>>> 
>>>>> These are userspace error messages, not kernel messages.
>>>>> 
>>>>> What kernel version are you using?
>>> 
>>> Yes, we need to know, what kernel version are you using.
>>> 
>>>>> thanks,
>>>>> 
>>>>> greg k-h
>>>> 
>>> 
>>> And also another question, what version of pppd daemon are you using?
>>> 
>>> Also, are you able to dump state of ppp channels and ppp units? It is
>>> needed to know to which tty device, file descriptor (or socket
>>> extension) is (or should be) particular ppp channel bounded.
>> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-11 11:10           ` Martin Zaharinov
@ 2021-08-11 16:48             ` Guillaume Nault
  2021-09-07  6:16               ` Martin Zaharinov
  0 siblings, 1 reply; 23+ messages in thread
From: Guillaume Nault @ 2021-08-11 16:48 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
> And one more that see.
> 
> Problem is come when accel start finishing sessions,
> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
> May be kernel destroy old session slow and entrained other users by locking other sessions.
> is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-08-11 16:48             ` Guillaume Nault
@ 2021-09-07  6:16               ` Martin Zaharinov
  2021-09-07  6:42                 ` Martin Zaharinov
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-07  6:16 UTC (permalink / raw)
  To: Guillaume Nault; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

> On 11 Aug 2021, at 19:48, Guillaume Nault <gnault@redhat.com> wrote:
> 
> On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
>> And one more that see.
>> 
>> Problem is come when accel start finishing sessions,
>> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
>> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
>> May be kernel destroy old session slow and entrained other users by locking other sessions.
>> is there a way to speed up the closing of stopped/dead sessions.
> 
> What are the CPU stats when that happen? Is it users space or kernel
> space that keeps it busy?
> 
> One easy way to check is to run "mpstat 1" for a few seconds when the
> problem occurs.
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-07  6:16               ` Martin Zaharinov
@ 2021-09-07  6:42                 ` Martin Zaharinov
  2021-09-11  6:26                   ` Martin Zaharinov
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-07  6:42 UTC (permalink / raw)
  To: Guillaume Nault; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

Perf top from text


PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
     9.73%  [kernel]                 [k] mutex_spin_on_owner
     9.07%  [pppoe]                  [k] pppoe_rcv
     2.77%  [nf_nat]                 [k] device_cmp
     1.66%  [kernel]                 [k] osq_lock
     1.65%  [kernel]                 [k] _raw_spin_lock
     1.61%  [kernel]                 [k] __local_bh_enable_ip
     1.35%  [nf_nat]                 [k] inet_cmp
     1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
     1.16%  [kernel]                 [k] menu_select
     0.99%  [kernel]                 [k] cpuidle_enter_state
     0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
     0.86%  [kernel]                 [k] __dev_queue_xmit
     0.70%  [kernel]                 [k] __cond_resched
     0.69%  [sch_cake]               [k] cake_dequeue
     0.67%  [nf_tables]              [k] nft_do_chain
     0.63%  [kernel]                 [k] rcu_all_qs
     0.61%  [kernel]                 [k] fib_table_lookup
     0.57%  [kernel]                 [k] __schedule
     0.57%  [kernel]                 [k] skb_release_data
     0.54%  [kernel]                 [k] sched_clock
     0.54%  [kernel]                 [k] __copy_skb_header
     0.53%  [kernel]                 [k] dev_queue_xmit_nit
     0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
     0.50%  [kernel]                 [k] kmem_cache_free
     0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
     0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
     0.45%  [kernel]                 [k] timerqueue_add
     0.45%  [kernel]                 [k] lapic_next_deadline
     0.45%  [kernel]                 [k] csum_partial_copy_generic
     0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
     0.44%  [kernel]                 [k] kmem_cache_alloc
     0.44%  [nf_conntrack]           [k] nf_conntrack_lock

> On 7 Sep 2021, at 9:16, Martin Zaharinov <micron10@gmail.com> wrote:
> 
> Hi 
> Sorry for delay but not easy to catch moment .
> 
> 
> See this is mpstatl 1 :
> 
> Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)
> 
> 11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
> 11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
> 11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
> 11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
> 11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
> 11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
> 11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
> 11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
> 11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
> 11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
> 11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
> 11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
> 11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
> 11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
> 11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
> 11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
> 11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
> Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26
> 
> 
> I attache and one screenshot from perf top (Screenshot is send on preview mail)
> 
> And I see in lsmod 
> 
> pppoe                  20480  8198
> pppox                  16384  1 pppoe
> ppp_generic            45056  16364 pppox,pppoe
> slhc                   16384  1 ppp_generic
> 
> To slow remove pppoe session .
> 
> And from log : 
> 
> [2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> [2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> 
>> On 11 Aug 2021, at 19:48, Guillaume Nault <gnault@redhat.com> wrote:
>> 
>> On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
>>> And one more that see.
>>> 
>>> Problem is come when accel start finishing sessions,
>>> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
>>> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
>>> May be kernel destroy old session slow and entrained other users by locking other sessions.
>>> is there a way to speed up the closing of stopped/dead sessions.
>> 
>> What are the CPU stats when that happen? Is it users space or kernel
>> space that keeps it busy?
>> 
>> One easy way to check is to run "mpstat 1" for a few seconds when the
>> problem occurs.
>> 
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-07  6:42                 ` Martin Zaharinov
@ 2021-09-11  6:26                   ` Martin Zaharinov
  2021-09-14  6:16                     ` Martin Zaharinov
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-11  6:26 UTC (permalink / raw)
  To: Guillaume Nault; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

Hi Guillaume

Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
But how to find where is a problem any locking or other.
And is there options to make fast remove ppp interface from kernel to reduce this load.


Martin

> On 7 Sep 2021, at 9:42, Martin Zaharinov <micron10@gmail.com> wrote:
> 
> Perf top from text
> 
> 
> PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
>    17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
>     9.73%  [kernel]                 [k] mutex_spin_on_owner
>     9.07%  [pppoe]                  [k] pppoe_rcv
>     2.77%  [nf_nat]                 [k] device_cmp
>     1.66%  [kernel]                 [k] osq_lock
>     1.65%  [kernel]                 [k] _raw_spin_lock
>     1.61%  [kernel]                 [k] __local_bh_enable_ip
>     1.35%  [nf_nat]                 [k] inet_cmp
>     1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
>     1.16%  [kernel]                 [k] menu_select
>     0.99%  [kernel]                 [k] cpuidle_enter_state
>     0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
>     0.86%  [kernel]                 [k] __dev_queue_xmit
>     0.70%  [kernel]                 [k] __cond_resched
>     0.69%  [sch_cake]               [k] cake_dequeue
>     0.67%  [nf_tables]              [k] nft_do_chain
>     0.63%  [kernel]                 [k] rcu_all_qs
>     0.61%  [kernel]                 [k] fib_table_lookup
>     0.57%  [kernel]                 [k] __schedule
>     0.57%  [kernel]                 [k] skb_release_data
>     0.54%  [kernel]                 [k] sched_clock
>     0.54%  [kernel]                 [k] __copy_skb_header
>     0.53%  [kernel]                 [k] dev_queue_xmit_nit
>     0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
>     0.50%  [kernel]                 [k] kmem_cache_free
>     0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
>     0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
>     0.45%  [kernel]                 [k] timerqueue_add
>     0.45%  [kernel]                 [k] lapic_next_deadline
>     0.45%  [kernel]                 [k] csum_partial_copy_generic
>     0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
>     0.44%  [kernel]                 [k] kmem_cache_alloc
>     0.44%  [nf_conntrack]           [k] nf_conntrack_lock
> 
>> On 7 Sep 2021, at 9:16, Martin Zaharinov <micron10@gmail.com> wrote:
>> 
>> Hi 
>> Sorry for delay but not easy to catch moment .
>> 
>> 
>> See this is mpstatl 1 :
>> 
>> Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)
>> 
>> 11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
>> 11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
>> 11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
>> 11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
>> 11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
>> 11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
>> 11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
>> 11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
>> 11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
>> 11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
>> 11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
>> 11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
>> 11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
>> 11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
>> 11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
>> 11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
>> 11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
>> Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26
>> 
>> 
>> I attache and one screenshot from perf top (Screenshot is send on preview mail)
>> 
>> And I see in lsmod 
>> 
>> pppoe                  20480  8198
>> pppox                  16384  1 pppoe
>> ppp_generic            45056  16364 pppox,pppoe
>> slhc                   16384  1 ppp_generic
>> 
>> To slow remove pppoe session .
>> 
>> And from log : 
>> 
>> [2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> [2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>> 
>>> On 11 Aug 2021, at 19:48, Guillaume Nault <gnault@redhat.com> wrote:
>>> 
>>> On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
>>>> And one more that see.
>>>> 
>>>> Problem is come when accel start finishing sessions,
>>>> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
>>>> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
>>>> May be kernel destroy old session slow and entrained other users by locking other sessions.
>>>> is there a way to speed up the closing of stopped/dead sessions.
>>> 
>>> What are the CPU stats when that happen? Is it users space or kernel
>>> space that keeps it busy?
>>> 
>>> One easy way to check is to run "mpstat 1" for a few seconds when the
>>> problem occurs.
>>> 
>> 
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-11  6:26                   ` Martin Zaharinov
@ 2021-09-14  6:16                     ` Martin Zaharinov
  2021-09-14  8:02                       ` Guillaume Nault
  0 siblings, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-14  6:16 UTC (permalink / raw)
  To: Guillaume Nault; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

Hi Nault

See this stats :

Linux 5.14.2 (testb)   09/14/21        _x86_64_        (12 CPU)

11:33:44     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:33:45     all    1.75    0.00   18.85    0.00    0.00    5.00    0.00    0.00    0.00   74.40
11:33:46     all    1.74    0.00   17.88    0.00    0.00    4.72    0.00    0.00    0.00   75.66
11:33:47     all    2.23    0.00   17.62    0.00    0.00    5.05    0.00    0.00    0.00   75.10
11:33:48     all    1.82    0.00   13.64    0.00    0.00    5.70    0.00    0.00    0.00   78.84
11:33:49     all    1.50    0.00   13.46    0.00    0.00    5.15    0.00    0.00    0.00   79.90
11:33:50     all    3.06    0.00   13.96    0.00    0.00    4.79    0.00    0.00    0.00   78.20
11:33:51     all    1.40    0.00   16.53    0.00    0.00    5.21    0.00    0.00    0.00   76.86
11:33:52     all    4.43    0.00   19.44    0.00    0.00    6.56    0.00    0.00    0.00   69.57
11:33:53     all    1.51    0.00   16.40    0.00    0.00    4.77    0.00    0.00    0.00   77.32
11:33:54     all    1.51    0.00   16.55    0.00    0.00    4.71    0.00    0.00    0.00   77.23
11:33:55     all    1.00    0.00   13.21    0.00    0.00    5.90    0.00    0.00    0.00   79.90
Average:     all    2.00    0.00   16.14    0.00    0.00    5.23    0.00    0.00    0.00   76.63


  PerfTop:   28046 irqs/sec  kernel:96.3%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    23.37%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
    17.76%  [kernel]                 [k] mutex_spin_on_owner
     9.47%  [pppoe]                  [k] pppoe_rcv
     7.71%  [kernel]                 [k] osq_lock
     2.77%  [nf_nat]                 [k] inet_cmp
     2.59%  [nf_nat]                 [k] device_cmp
     2.55%  [kernel]                 [k] __local_bh_enable_ip
     2.04%  [kernel]                 [k] _raw_spin_lock
     1.23%  [kernel]                 [k] __cond_resched
     1.16%  [kernel]                 [k] rcu_all_qs
     1.13%  libfrr.so.0.0.0          [.] 0x00000000000ce970
     0.79%  [nf_conntrack]           [k] nf_conntrack_lock
     0.75%  libfrr.so.0.0.0          [.] 0x00000000000ce94e
     0.53%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
     0.46%  [kernel]                 [k] fib_table_lookup
     0.46%  [ip_tables]              [k] ipt_do_table
     0.45%  [ixgbe]                  [k] ixgbe_clean_rx_irq
     0.37%  [kernel]                 [k] __dev_queue_xmit
     0.34%  [nf_conntrack]           [k] __nf_conntrack_find_get.isra.0
     0.33%  [ixgbe]                  [k] ixgbe_clean_tx_irq
     0.30%  [kernel]                 [k] menu_select
     0.25%  [kernel]                 [k] vlan_do_receive
     0.21%  [kernel]                 [k] ip_finish_output2
     0.21%  [ixgbe]                  [k] ixgbe_poll
     0.20%  [kernel]                 [k] _raw_spin_lock_irqsave
     0.19%  [kernel]                 [k] get_rps_cpu
     0.19%  libc.so.6                [.] 0x0000000000186afa
     0.19%  [kernel]                 [k] queued_read_lock_slowpath
     0.19%  [kernel]                 [k] do_poll.constprop.0
     0.19%  [kernel]                 [k] cpuidle_enter_state
     0.18%  [kernel]                 [k] dev_hard_start_xmit
     0.18%  [kernel]                 [k] ___slab_alloc.constprop.0
     0.17%  zebra                    [.] 0x00000000000b9271
     0.16%  [kernel]                 [k] csum_partial_copy_generic
     0.16%  zebra                    [.] 0x00000000000b91f1
     0.16%  [kernel]                 [k] page_frag_free
     0.16%  [kernel]                 [k] kmem_cache_alloc
     0.15%  [kernel]                 [k] __skb_flow_dissect
     0.15%  [kernel]                 [k] sched_clock
     0.15%  libc.so.6                [.] 0x00000000000965a2
     0.15%  [kernel]                 [k] kmem_cache_free_bulk.part.0
     0.15%  [pppoe]                  [k] pppoe_flush_dev
     0.15%  [ixgbe]                  [k] ixgbe_tx_map
     0.14%  [kernel]                 [k] _raw_spin_lock_bh
     0.14%  [kernel]                 [k] fib_table_flush
     0.14%  [kernel]                 [k] native_irq_return_iret
     0.14%  [kernel]                 [k] __dev_xmit_skb
     0.13%  [kernel]                 [k] nf_hook_slow
     0.13%  [kernel]                 [k] fib_lookup_good_nhc
     0.12%  [kernel]                 [k] __fget_files
     0.12%  [kernel]                 [k] process_backlog
     0.12%  [xt_dtvqos]              [k] 0x00000000000008d1
     0.12%  [kernel]                 [k] __list_del_entry_valid
     0.12%  [kernel]                 [k] skb_release_data
     0.12%  [kernel]                 [k] ip_route_input_slow
     0.11%  [kernel]                 [k] netif_skb_features
     0.11%  [kernel]                 [k] sock_poll
     0.11%  [kernel]                 [k] __schedule
     0.11%  [kernel]                 [k] __softirqentry_text_start


And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

In case need to know why system is overloaded when deconfig ppp interface.


Best regards,
Martin




> On 11 Sep 2021, at 9:26, Martin Zaharinov <micron10@gmail.com> wrote:
> 
> Hi Guillaume
> 
> Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
> But how to find where is a problem any locking or other.
> And is there options to make fast remove ppp interface from kernel to reduce this load.
> 
> 
> Martin
> 
>> On 7 Sep 2021, at 9:42, Martin Zaharinov <micron10@gmail.com> wrote:
>> 
>> Perf top from text
>> 
>> 
>> PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>> 
>>   17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
>>    9.73%  [kernel]                 [k] mutex_spin_on_owner
>>    9.07%  [pppoe]                  [k] pppoe_rcv
>>    2.77%  [nf_nat]                 [k] device_cmp
>>    1.66%  [kernel]                 [k] osq_lock
>>    1.65%  [kernel]                 [k] _raw_spin_lock
>>    1.61%  [kernel]                 [k] __local_bh_enable_ip
>>    1.35%  [nf_nat]                 [k] inet_cmp
>>    1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
>>    1.16%  [kernel]                 [k] menu_select
>>    0.99%  [kernel]                 [k] cpuidle_enter_state
>>    0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
>>    0.86%  [kernel]                 [k] __dev_queue_xmit
>>    0.70%  [kernel]                 [k] __cond_resched
>>    0.69%  [sch_cake]               [k] cake_dequeue
>>    0.67%  [nf_tables]              [k] nft_do_chain
>>    0.63%  [kernel]                 [k] rcu_all_qs
>>    0.61%  [kernel]                 [k] fib_table_lookup
>>    0.57%  [kernel]                 [k] __schedule
>>    0.57%  [kernel]                 [k] skb_release_data
>>    0.54%  [kernel]                 [k] sched_clock
>>    0.54%  [kernel]                 [k] __copy_skb_header
>>    0.53%  [kernel]                 [k] dev_queue_xmit_nit
>>    0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
>>    0.50%  [kernel]                 [k] kmem_cache_free
>>    0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
>>    0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
>>    0.45%  [kernel]                 [k] timerqueue_add
>>    0.45%  [kernel]                 [k] lapic_next_deadline
>>    0.45%  [kernel]                 [k] csum_partial_copy_generic
>>    0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
>>    0.44%  [kernel]                 [k] kmem_cache_alloc
>>    0.44%  [nf_conntrack]           [k] nf_conntrack_lock
>> 
>>> On 7 Sep 2021, at 9:16, Martin Zaharinov <micron10@gmail.com> wrote:
>>> 
>>> Hi 
>>> Sorry for delay but not easy to catch moment .
>>> 
>>> 
>>> See this is mpstatl 1 :
>>> 
>>> Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)
>>> 
>>> 11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
>>> 11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
>>> 11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
>>> 11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
>>> 11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
>>> 11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
>>> 11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
>>> 11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
>>> 11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
>>> 11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
>>> 11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
>>> 11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
>>> 11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
>>> 11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
>>> 11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
>>> 11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
>>> 11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
>>> Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26
>>> 
>>> 
>>> I attache and one screenshot from perf top (Screenshot is send on preview mail)
>>> 
>>> And I see in lsmod 
>>> 
>>> pppoe                  20480  8198
>>> pppox                  16384  1 pppoe
>>> ppp_generic            45056  16364 pppox,pppoe
>>> slhc                   16384  1 ppp_generic
>>> 
>>> To slow remove pppoe session .
>>> 
>>> And from log : 
>>> 
>>> [2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> [2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
>>> 
>>>> On 11 Aug 2021, at 19:48, Guillaume Nault <gnault@redhat.com> wrote:
>>>> 
>>>> On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
>>>>> And one more that see.
>>>>> 
>>>>> Problem is come when accel start finishing sessions,
>>>>> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
>>>>> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
>>>>> May be kernel destroy old session slow and entrained other users by locking other sessions.
>>>>> is there a way to speed up the closing of stopped/dead sessions.
>>>> 
>>>> What are the CPU stats when that happen? Is it users space or kernel
>>>> space that keeps it busy?
>>>> 
>>>> One easy way to check is to run "mpstat 1" for a few seconds when the
>>>> problem occurs.
>>>> 
>>> 
>> 
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14  6:16                     ` Martin Zaharinov
@ 2021-09-14  8:02                       ` Guillaume Nault
  2021-09-14  9:50                         ` Florian Westphal
  0 siblings, 1 reply; 23+ messages in thread
From: Guillaume Nault @ 2021-09-14  8:02 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Pali Rohár, Greg KH, netdev, Eric Dumazet

On Tue, Sep 14, 2021 at 09:16:55AM +0300, Martin Zaharinov wrote:
> Hi Nault
> 
> See this stats :
> 
> Linux 5.14.2 (testb)   09/14/21        _x86_64_        (12 CPU)
> 
> 11:33:44     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
> 11:33:45     all    1.75    0.00   18.85    0.00    0.00    5.00    0.00    0.00    0.00   74.40
> 11:33:46     all    1.74    0.00   17.88    0.00    0.00    4.72    0.00    0.00    0.00   75.66
> 11:33:47     all    2.23    0.00   17.62    0.00    0.00    5.05    0.00    0.00    0.00   75.10
> 11:33:48     all    1.82    0.00   13.64    0.00    0.00    5.70    0.00    0.00    0.00   78.84
> 11:33:49     all    1.50    0.00   13.46    0.00    0.00    5.15    0.00    0.00    0.00   79.90
> 11:33:50     all    3.06    0.00   13.96    0.00    0.00    4.79    0.00    0.00    0.00   78.20
> 11:33:51     all    1.40    0.00   16.53    0.00    0.00    5.21    0.00    0.00    0.00   76.86
> 11:33:52     all    4.43    0.00   19.44    0.00    0.00    6.56    0.00    0.00    0.00   69.57
> 11:33:53     all    1.51    0.00   16.40    0.00    0.00    4.77    0.00    0.00    0.00   77.32
> 11:33:54     all    1.51    0.00   16.55    0.00    0.00    4.71    0.00    0.00    0.00   77.23
> 11:33:55     all    1.00    0.00   13.21    0.00    0.00    5.90    0.00    0.00    0.00   79.90
> Average:     all    2.00    0.00   16.14    0.00    0.00    5.23    0.00    0.00    0.00   76.63
> 
> 
>   PerfTop:   28046 irqs/sec  kernel:96.3%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
>     23.37%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
>     17.76%  [kernel]                 [k] mutex_spin_on_owner
>      9.47%  [pppoe]                  [k] pppoe_rcv
>      7.71%  [kernel]                 [k] osq_lock
>      2.77%  [nf_nat]                 [k] inet_cmp
>      2.59%  [nf_nat]                 [k] device_cmp
>      2.55%  [kernel]                 [k] __local_bh_enable_ip
>      2.04%  [kernel]                 [k] _raw_spin_lock
>      1.23%  [kernel]                 [k] __cond_resched
>      1.16%  [kernel]                 [k] rcu_all_qs
>      1.13%  libfrr.so.0.0.0          [.] 0x00000000000ce970
>      0.79%  [nf_conntrack]           [k] nf_conntrack_lock
>      0.75%  libfrr.so.0.0.0          [.] 0x00000000000ce94e
>      0.53%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
>      0.46%  [kernel]                 [k] fib_table_lookup
>      0.46%  [ip_tables]              [k] ipt_do_table
>      0.45%  [ixgbe]                  [k] ixgbe_clean_rx_irq
>      0.37%  [kernel]                 [k] __dev_queue_xmit
>      0.34%  [nf_conntrack]           [k] __nf_conntrack_find_get.isra.0
>      0.33%  [ixgbe]                  [k] ixgbe_clean_tx_irq
>      0.30%  [kernel]                 [k] menu_select
>      0.25%  [kernel]                 [k] vlan_do_receive
>      0.21%  [kernel]                 [k] ip_finish_output2
>      0.21%  [ixgbe]                  [k] ixgbe_poll
>      0.20%  [kernel]                 [k] _raw_spin_lock_irqsave
>      0.19%  [kernel]                 [k] get_rps_cpu
>      0.19%  libc.so.6                [.] 0x0000000000186afa
>      0.19%  [kernel]                 [k] queued_read_lock_slowpath
>      0.19%  [kernel]                 [k] do_poll.constprop.0
>      0.19%  [kernel]                 [k] cpuidle_enter_state
>      0.18%  [kernel]                 [k] dev_hard_start_xmit
>      0.18%  [kernel]                 [k] ___slab_alloc.constprop.0
>      0.17%  zebra                    [.] 0x00000000000b9271
>      0.16%  [kernel]                 [k] csum_partial_copy_generic
>      0.16%  zebra                    [.] 0x00000000000b91f1
>      0.16%  [kernel]                 [k] page_frag_free
>      0.16%  [kernel]                 [k] kmem_cache_alloc
>      0.15%  [kernel]                 [k] __skb_flow_dissect
>      0.15%  [kernel]                 [k] sched_clock
>      0.15%  libc.so.6                [.] 0x00000000000965a2
>      0.15%  [kernel]                 [k] kmem_cache_free_bulk.part.0
>      0.15%  [pppoe]                  [k] pppoe_flush_dev
>      0.15%  [ixgbe]                  [k] ixgbe_tx_map
>      0.14%  [kernel]                 [k] _raw_spin_lock_bh
>      0.14%  [kernel]                 [k] fib_table_flush
>      0.14%  [kernel]                 [k] native_irq_return_iret
>      0.14%  [kernel]                 [k] __dev_xmit_skb
>      0.13%  [kernel]                 [k] nf_hook_slow
>      0.13%  [kernel]                 [k] fib_lookup_good_nhc
>      0.12%  [kernel]                 [k] __fget_files
>      0.12%  [kernel]                 [k] process_backlog
>      0.12%  [xt_dtvqos]              [k] 0x00000000000008d1
>      0.12%  [kernel]                 [k] __list_del_entry_valid
>      0.12%  [kernel]                 [k] skb_release_data
>      0.12%  [kernel]                 [k] ip_route_input_slow
>      0.11%  [kernel]                 [k] netif_skb_features
>      0.11%  [kernel]                 [k] sock_poll
>      0.11%  [kernel]                 [k] __schedule
>      0.11%  [kernel]                 [k] __softirqentry_text_start
> 
> 
> And on time of problem when try to write : ip a 
> to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

Probably some contention on the rtnl lock.

> In case need to know why system is overloaded when deconfig ppp interface.

Does it help if you disable conntrack?

> 
> Best regards,
> Martin
> 
> 
> 
> 
> > On 11 Sep 2021, at 9:26, Martin Zaharinov <micron10@gmail.com> wrote:
> > 
> > Hi Guillaume
> > 
> > Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
> > But how to find where is a problem any locking or other.
> > And is there options to make fast remove ppp interface from kernel to reduce this load.
> > 
> > 
> > Martin
> > 
> >> On 7 Sep 2021, at 9:42, Martin Zaharinov <micron10@gmail.com> wrote:
> >> 
> >> Perf top from text
> >> 
> >> 
> >> PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
> >> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> >> 
> >>   17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
> >>    9.73%  [kernel]                 [k] mutex_spin_on_owner
> >>    9.07%  [pppoe]                  [k] pppoe_rcv
> >>    2.77%  [nf_nat]                 [k] device_cmp
> >>    1.66%  [kernel]                 [k] osq_lock
> >>    1.65%  [kernel]                 [k] _raw_spin_lock
> >>    1.61%  [kernel]                 [k] __local_bh_enable_ip
> >>    1.35%  [nf_nat]                 [k] inet_cmp
> >>    1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
> >>    1.16%  [kernel]                 [k] menu_select
> >>    0.99%  [kernel]                 [k] cpuidle_enter_state
> >>    0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
> >>    0.86%  [kernel]                 [k] __dev_queue_xmit
> >>    0.70%  [kernel]                 [k] __cond_resched
> >>    0.69%  [sch_cake]               [k] cake_dequeue
> >>    0.67%  [nf_tables]              [k] nft_do_chain
> >>    0.63%  [kernel]                 [k] rcu_all_qs
> >>    0.61%  [kernel]                 [k] fib_table_lookup
> >>    0.57%  [kernel]                 [k] __schedule
> >>    0.57%  [kernel]                 [k] skb_release_data
> >>    0.54%  [kernel]                 [k] sched_clock
> >>    0.54%  [kernel]                 [k] __copy_skb_header
> >>    0.53%  [kernel]                 [k] dev_queue_xmit_nit
> >>    0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
> >>    0.50%  [kernel]                 [k] kmem_cache_free
> >>    0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
> >>    0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
> >>    0.45%  [kernel]                 [k] timerqueue_add
> >>    0.45%  [kernel]                 [k] lapic_next_deadline
> >>    0.45%  [kernel]                 [k] csum_partial_copy_generic
> >>    0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
> >>    0.44%  [kernel]                 [k] kmem_cache_alloc
> >>    0.44%  [nf_conntrack]           [k] nf_conntrack_lock
> >> 
> >>> On 7 Sep 2021, at 9:16, Martin Zaharinov <micron10@gmail.com> wrote:
> >>> 
> >>> Hi 
> >>> Sorry for delay but not easy to catch moment .
> >>> 
> >>> 
> >>> See this is mpstatl 1 :
> >>> 
> >>> Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)
> >>> 
> >>> 11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
> >>> 11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
> >>> 11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
> >>> 11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
> >>> 11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
> >>> 11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
> >>> 11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
> >>> 11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
> >>> 11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
> >>> 11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
> >>> 11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
> >>> 11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
> >>> 11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
> >>> 11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
> >>> 11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
> >>> 11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
> >>> 11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
> >>> Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26
> >>> 
> >>> 
> >>> I attache and one screenshot from perf top (Screenshot is send on preview mail)
> >>> 
> >>> And I see in lsmod 
> >>> 
> >>> pppoe                  20480  8198
> >>> pppox                  16384  1 pppoe
> >>> ppp_generic            45056  16364 pppox,pppoe
> >>> slhc                   16384  1 ppp_generic
> >>> 
> >>> To slow remove pppoe session .
> >>> 
> >>> And from log : 
> >>> 
> >>> [2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> [2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
> >>> 
> >>>> On 11 Aug 2021, at 19:48, Guillaume Nault <gnault@redhat.com> wrote:
> >>>> 
> >>>> On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:
> >>>>> And one more that see.
> >>>>> 
> >>>>> Problem is come when accel start finishing sessions,
> >>>>> Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
> >>>>> And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
> >>>>> May be kernel destroy old session slow and entrained other users by locking other sessions.
> >>>>> is there a way to speed up the closing of stopped/dead sessions.
> >>>> 
> >>>> What are the CPU stats when that happen? Is it users space or kernel
> >>>> space that keeps it busy?
> >>>> 
> >>>> One easy way to check is to run "mpstat 1" for a few seconds when the
> >>>> problem occurs.
> >>>> 
> >>> 
> >> 
> > 
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14  8:02                       ` Guillaume Nault
@ 2021-09-14  9:50                         ` Florian Westphal
  2021-09-14 10:01                           ` Martin Zaharinov
  2021-09-14 10:53                           ` Martin Zaharinov
  0 siblings, 2 replies; 23+ messages in thread
From: Florian Westphal @ 2021-09-14  9:50 UTC (permalink / raw)
  To: Guillaume Nault
  Cc: Martin Zaharinov, Pali Rohár, Greg KH, netdev, Eric Dumazet

Guillaume Nault <gnault@redhat.com> wrote:
> > And on time of problem when try to write : ip a 
> > to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.
> 
> Probably some contention on the rtnl lock.

Yes, I'll create a patch.

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14  9:50                         ` Florian Westphal
@ 2021-09-14 10:01                           ` Martin Zaharinov
  2021-09-14 11:00                             ` Florian Westphal
  2021-09-14 10:53                           ` Martin Zaharinov
  1 sibling, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-14 10:01 UTC (permalink / raw)
  To: Florian Westphal
  Cc: Guillaume Nault, Pali Rohár, Greg KH, netdev, Eric Dumazet

Hi Nault and Florian

Nault : 

No not test need conntrack to log user traffic.

Florian: 

If you make patch send to test please.


Martin

> On 14 Sep 2021, at 12:50, Florian Westphal <fw@strlen.de> wrote:
> 
> Guillaume Nault <gnault@redhat.com> wrote:
>>> And on time of problem when try to write : ip a 
>>> to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.
>> 
>> Probably some contention on the rtnl lock.
> 
> Yes, I'll create a patch.


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14 10:01                           ` Martin Zaharinov
@ 2021-09-14 11:00                             ` Florian Westphal
  2021-09-15 14:25                               ` Martin Zaharinov
  2021-09-16 20:00                               ` Martin Zaharinov
  0 siblings, 2 replies; 23+ messages in thread
From: Florian Westphal @ 2021-09-14 11:00 UTC (permalink / raw)
  To: Martin Zaharinov; +Cc: Florian Westphal, Guillaume Nault, netdev

[-- Attachment #1: Type: text/plain, Size: 238 bytes --]

Martin Zaharinov <micron10@gmail.com> wrote:

[ Trimming CC list ]

> Florian: 
> 
> If you make patch send to test please.

Attached.  No idea if it helps, but 'ip' should stay responsive
even when masquerade processes netdevice events.

[-- Attachment #2: defer_masq_work.diff --]
[-- Type: text/x-diff, Size: 6674 bytes --]

diff --git a/net/netfilter/nf_nat_masquerade.c b/net/netfilter/nf_nat_masquerade.c
index 8e8a65d46345..50c6d6992ed6 100644
--- a/net/netfilter/nf_nat_masquerade.c
+++ b/net/netfilter/nf_nat_masquerade.c
@@ -9,8 +9,19 @@
 
 #include <net/netfilter/nf_nat_masquerade.h>
 
+struct masq_dev_work {
+	struct work_struct work;
+	struct net *net;
+	union nf_inet_addr addr;
+	int ifindex;
+	int (*iter)(struct nf_conn *i, void *data);
+};
+
+#define MAX_MASQ_WORKER_COUNT	16
+
 static DEFINE_MUTEX(masq_mutex);
 static unsigned int masq_refcnt __read_mostly;
+static atomic_t masq_worker_count __read_mostly;
 
 unsigned int
 nf_nat_masquerade_ipv4(struct sk_buff *skb, unsigned int hooknum,
@@ -63,13 +74,68 @@ nf_nat_masquerade_ipv4(struct sk_buff *skb, unsigned int hooknum,
 }
 EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv4);
 
-static int device_cmp(struct nf_conn *i, void *ifindex)
+static void iterate_cleanup_work(struct work_struct *work)
+{
+	struct masq_dev_work *w;
+
+	w = container_of(work, struct masq_dev_work, work);
+
+	nf_ct_iterate_cleanup_net(w->net, w->iter, (void *)w, 0, 0);
+
+	put_net(w->net);
+	kfree(w);
+	atomic_dec(&masq_worker_count);
+	module_put(THIS_MODULE);
+}
+
+/* Iterate conntrack table in the background and remove conntrack entries
+ * that use the device/address being removed.
+ *
+ * In case too many work items have been queued already or memory allocation
+ * fails iteration is skipped, conntrack entries will time out eventually.
+ */
+static void nf_nat_masq_schedule(struct net *net, union nf_inet_addr *addr,
+				 int ifindex,
+				 int (*iter)(struct nf_conn *i, void *data),
+				 gfp_t gfp_flags)
+{
+	struct masq_dev_work *w;
+
+	net = maybe_get_net(net);
+	if (!net)
+		return;
+
+	if (!try_module_get(THIS_MODULE))
+		goto err_module;
+
+	w = kzalloc(sizeof(*w), gfp_flags);
+	if (w) {
+		/* We can overshoot MAX_MASQ_WORKER_COUNT, no big deal */
+		atomic_inc(&masq_worker_count);
+
+		INIT_WORK(&w->work, iterate_cleanup_work);
+		w->ifindex = ifindex;
+		w->net = net;
+		w->iter = iter;
+		if (addr)
+			w->addr = *addr;
+		schedule_work(&w->work);
+		return;
+	}
+
+	module_put(THIS_MODULE);
+ err_module:
+	put_net(net);
+}
+
+static int device_cmp(struct nf_conn *i, void *arg)
 {
 	const struct nf_conn_nat *nat = nfct_nat(i);
+	const struct masq_dev_work *w = arg;
 
 	if (!nat)
 		return 0;
-	return nat->masq_index == (int)(long)ifindex;
+	return nat->masq_index == w->ifindex;
 }
 
 static int masq_device_event(struct notifier_block *this,
@@ -85,8 +151,8 @@ static int masq_device_event(struct notifier_block *this,
 		 * and forget them.
 		 */
 
-		nf_ct_iterate_cleanup_net(net, device_cmp,
-					  (void *)(long)dev->ifindex, 0, 0);
+		nf_nat_masq_schedule(net, NULL, dev->ifindex,
+				     device_cmp, GFP_KERNEL);
 	}
 
 	return NOTIFY_DONE;
@@ -94,35 +160,45 @@ static int masq_device_event(struct notifier_block *this,
 
 static int inet_cmp(struct nf_conn *ct, void *ptr)
 {
-	struct in_ifaddr *ifa = (struct in_ifaddr *)ptr;
-	struct net_device *dev = ifa->ifa_dev->dev;
 	struct nf_conntrack_tuple *tuple;
+	struct masq_dev_work *w = ptr;
 
-	if (!device_cmp(ct, (void *)(long)dev->ifindex))
+	if (!device_cmp(ct, ptr))
 		return 0;
 
 	tuple = &ct->tuplehash[IP_CT_DIR_REPLY].tuple;
 
-	return ifa->ifa_address == tuple->dst.u3.ip;
+	return nf_inet_addr_cmp(&w->addr, &tuple->dst.u3);
 }
 
 static int masq_inet_event(struct notifier_block *this,
 			   unsigned long event,
 			   void *ptr)
 {
-	struct in_device *idev = ((struct in_ifaddr *)ptr)->ifa_dev;
-	struct net *net = dev_net(idev->dev);
+	const struct in_ifaddr *ifa = ptr;
+	const struct in_device *idev;
+	const struct net_device *dev;
+	union nf_inet_addr addr;
+
+	if (event != NETDEV_DOWN)
+		return NOTIFY_DONE;
 
 	/* The masq_dev_notifier will catch the case of the device going
 	 * down.  So if the inetdev is dead and being destroyed we have
 	 * no work to do.  Otherwise this is an individual address removal
 	 * and we have to perform the flush.
 	 */
+	idev = ifa->ifa_dev;
 	if (idev->dead)
 		return NOTIFY_DONE;
 
-	if (event == NETDEV_DOWN)
-		nf_ct_iterate_cleanup_net(net, inet_cmp, ptr, 0, 0);
+	memset(&addr, 0, sizeof(addr));
+
+	addr.ip = ifa->ifa_address;
+
+	dev = idev->dev;
+	nf_nat_masq_schedule(dev_net(idev->dev), &addr, dev->ifindex,
+			     inet_cmp, GFP_KERNEL);
 
 	return NOTIFY_DONE;
 }
@@ -136,8 +212,6 @@ static struct notifier_block masq_inet_notifier = {
 };
 
 #if IS_ENABLED(CONFIG_IPV6)
-static atomic_t v6_worker_count __read_mostly;
-
 static int
 nat_ipv6_dev_get_saddr(struct net *net, const struct net_device *dev,
 		       const struct in6_addr *daddr, unsigned int srcprefs,
@@ -187,40 +261,6 @@ nf_nat_masquerade_ipv6(struct sk_buff *skb, const struct nf_nat_range2 *range,
 }
 EXPORT_SYMBOL_GPL(nf_nat_masquerade_ipv6);
 
-struct masq_dev_work {
-	struct work_struct work;
-	struct net *net;
-	struct in6_addr addr;
-	int ifindex;
-};
-
-static int inet6_cmp(struct nf_conn *ct, void *work)
-{
-	struct masq_dev_work *w = (struct masq_dev_work *)work;
-	struct nf_conntrack_tuple *tuple;
-
-	if (!device_cmp(ct, (void *)(long)w->ifindex))
-		return 0;
-
-	tuple = &ct->tuplehash[IP_CT_DIR_REPLY].tuple;
-
-	return ipv6_addr_equal(&w->addr, &tuple->dst.u3.in6);
-}
-
-static void iterate_cleanup_work(struct work_struct *work)
-{
-	struct masq_dev_work *w;
-
-	w = container_of(work, struct masq_dev_work, work);
-
-	nf_ct_iterate_cleanup_net(w->net, inet6_cmp, (void *)w, 0, 0);
-
-	put_net(w->net);
-	kfree(w);
-	atomic_dec(&v6_worker_count);
-	module_put(THIS_MODULE);
-}
-
 /* atomic notifier; can't call nf_ct_iterate_cleanup_net (it can sleep).
  *
  * Defer it to the system workqueue.
@@ -233,36 +273,19 @@ static int masq_inet6_event(struct notifier_block *this,
 {
 	struct inet6_ifaddr *ifa = ptr;
 	const struct net_device *dev;
-	struct masq_dev_work *w;
-	struct net *net;
+	union nf_inet_addr addr;
 
-	if (event != NETDEV_DOWN || atomic_read(&v6_worker_count) >= 16)
+	if (event != NETDEV_DOWN)
 		return NOTIFY_DONE;
 
 	dev = ifa->idev->dev;
-	net = maybe_get_net(dev_net(dev));
-	if (!net)
-		return NOTIFY_DONE;
-
-	if (!try_module_get(THIS_MODULE))
-		goto err_module;
 
-	w = kmalloc(sizeof(*w), GFP_ATOMIC);
-	if (w) {
-		atomic_inc(&v6_worker_count);
+	memset(&addr, 0, sizeof(addr));
 
-		INIT_WORK(&w->work, iterate_cleanup_work);
-		w->ifindex = dev->ifindex;
-		w->net = net;
-		w->addr = ifa->addr;
-		schedule_work(&w->work);
+	addr.in6 = ifa->addr;
 
-		return NOTIFY_DONE;
-	}
-
-	module_put(THIS_MODULE);
- err_module:
-	put_net(net);
+	nf_nat_masq_schedule(dev_net(dev), &addr, dev->ifindex, inet_cmp,
+				 GFP_ATOMIC);
 	return NOTIFY_DONE;
 }
 

^ permalink raw reply related	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14 11:00                             ` Florian Westphal
@ 2021-09-15 14:25                               ` Martin Zaharinov
  2021-09-15 14:37                                 ` Martin Zaharinov
  2021-09-16 20:00                               ` Martin Zaharinov
  1 sibling, 1 reply; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-15 14:25 UTC (permalink / raw)
  To: Florian Westphal; +Cc: Guillaume Nault, netdev

Hey Florian

make test in lab and look much better that before.

see this perf 

 PerfTop:    6551 irqs/sec  kernel:77.8%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    15.70%  [ixgbe]           [k] ixgbe_read_reg
    13.33%  [kernel]          [k] mutex_spin_on_owner
     7.65%  [kernel]          [k] osq_lock
     2.85%  libfrr.so.0.0.0   [.] 0x00000000000ce970
     1.94%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
     1.19%  libc.so.6         [.] 0x0000000000186afa
     1.15%  [kernel]          [k] do_poll.constprop.0
     0.99%  [kernel]          [k] inet_dump_ifaddr
     0.94%  libteam.so.5.6.1  [.] 0x0000000000006470
     0.79%  libc.so.6         [.] 0x0000000000186e57
     0.71%  [ixgbe]           [k] ixgbe_update_mc_addr_list_generic
     0.65%  [kernel]          [k] __fget_files
     0.61%  [kernel]          [k] sock_poll
     0.57%  libteam.so.5.6.1  [.] 0x0000000000009e7d
     0.51%  perf              [.] 0x00000000000bc7b3
     0.51%  libteam.so.5.6.1  [.] 0x0000000000006501
     0.48%  [kernel]          [k] next_uptodate_page
     0.46%  [kernel]          [k] _raw_read_lock_bh
     0.43%  libc.so.6         [.] 0x0000000000186eac
     0.42%  bgpd              [.] 0x0000000000070a46
     0.41%  [pppoe]           [k] pppoe_flush_dev
     0.39%  [kernel]          [k] zap_pte_range


This happened when remove and add new interface on time of drop and reconnect users.


now : ip a command work fine !


Martin


> On 14 Sep 2021, at 14:00, Florian Westphal <fw@strlen.de> wrote:
> 
> Martin Zaharinov <micron10@gmail.com> wrote:
> 
> [ Trimming CC list ]
> 
>> Florian: 
>> 
>> If you make patch send to test please.
> 
> Attached.  No idea if it helps, but 'ip' should stay responsive
> even when masquerade processes netdevice events.
> <defer_masq_work.diff>


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-15 14:25                               ` Martin Zaharinov
@ 2021-09-15 14:37                                 ` Martin Zaharinov
  0 siblings, 0 replies; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-15 14:37 UTC (permalink / raw)
  To: Florian Westphal; +Cc: Guillaume Nault, netdev

and this : 

  PerfTop:   26378 irqs/sec  kernel:61.4%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

     5.65%  libfrr.so.0.0.0   [.] 0x00000000000ce970
     5.56%  [kernel]          [k] osq_lock
     5.22%  [kernel]          [k] mutex_spin_on_owner
     3.66%  [pppoe]           [k] pppoe_flush_dev
     3.01%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
     1.98%  libc.so.6         [.] 0x00000000000965a2
     1.84%  libc.so.6         [.] 0x0000000000186afa
     1.55%  libc.so.6         [.] 0x0000000000186e57
     1.54%  zebra             [.] 0x00000000000b9271
     1.46%  zebra             [.] 0x00000000000b91f1
     1.46%  libteam.so.5.6.1  [.] 0x0000000000006470
     1.44%  libc.so.6         [.] 0x00000000000965a0
     1.30%  libteam.so.5.6.1  [.] 0x0000000000009e7d
     1.08%  [kernel]          [k] fib_table_flush
     1.02%  libc.so.6         [.] 0x0000000000186eac
     0.93%  [kernel]          [k] do_poll.constprop.0
     0.85%  libc.so.6         [.] 0x0000000000186afe
     0.80%  dtvbras           [.] 0x0000000000014be8
     0.78%  [kernel]          [k] queued_read_lock_slowpath
     0.72%  [kernel]          [k] next_uptodate_page
     0.64%  [kernel]          [k] zap_pte_range
     0.64%  bgpd              [.] 0x0000000000070a46
     0.61%  [kernel]          [k] fib_table_insert

> On 15 Sep 2021, at 17:25, Martin Zaharinov <micron10@gmail.com> wrote:
> 
> Hey Florian
> 
> make test in lab and look much better that before.
> 
> see this perf 
> 
> PerfTop:    6551 irqs/sec  kernel:77.8%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
>    15.70%  [ixgbe]           [k] ixgbe_read_reg
>    13.33%  [kernel]          [k] mutex_spin_on_owner
>     7.65%  [kernel]          [k] osq_lock
>     2.85%  libfrr.so.0.0.0   [.] 0x00000000000ce970
>     1.94%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
>     1.19%  libc.so.6         [.] 0x0000000000186afa
>     1.15%  [kernel]          [k] do_poll.constprop.0
>     0.99%  [kernel]          [k] inet_dump_ifaddr
>     0.94%  libteam.so.5.6.1  [.] 0x0000000000006470
>     0.79%  libc.so.6         [.] 0x0000000000186e57
>     0.71%  [ixgbe]           [k] ixgbe_update_mc_addr_list_generic
>     0.65%  [kernel]          [k] __fget_files
>     0.61%  [kernel]          [k] sock_poll
>     0.57%  libteam.so.5.6.1  [.] 0x0000000000009e7d
>     0.51%  perf              [.] 0x00000000000bc7b3
>     0.51%  libteam.so.5.6.1  [.] 0x0000000000006501
>     0.48%  [kernel]          [k] next_uptodate_page
>     0.46%  [kernel]          [k] _raw_read_lock_bh
>     0.43%  libc.so.6         [.] 0x0000000000186eac
>     0.42%  bgpd              [.] 0x0000000000070a46
>     0.41%  [pppoe]           [k] pppoe_flush_dev
>     0.39%  [kernel]          [k] zap_pte_range
> 
> 
> This happened when remove and add new interface on time of drop and reconnect users.
> 
> 
> now : ip a command work fine !
> 
> 
> Martin
> 
> 
>> On 14 Sep 2021, at 14:00, Florian Westphal <fw@strlen.de> wrote:
>> 
>> Martin Zaharinov <micron10@gmail.com> wrote:
>> 
>> [ Trimming CC list ]
>> 
>>> Florian: 
>>> 
>>> If you make patch send to test please.
>> 
>> Attached.  No idea if it helps, but 'ip' should stay responsive
>> even when masquerade processes netdevice events.
>> <defer_masq_work.diff>
> 


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14 11:00                             ` Florian Westphal
  2021-09-15 14:25                               ` Martin Zaharinov
@ 2021-09-16 20:00                               ` Martin Zaharinov
  1 sibling, 0 replies; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-16 20:00 UTC (permalink / raw)
  To: Florian Westphal; +Cc: Guillaume Nault, netdev

Small Updates

After switch from frr to bird bgp reduce load from frr
but still when have disconnect 5k+ users have slow pppoe_flush_dev




   PerfTop:   15606 irqs/sec  kernel:77.7%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

     8.24%  [kernel]              [k] osq_lock
     7.55%  [kernel]              [k] mutex_spin_on_owner
     7.04%  [pppoe]               [k] pppoe_flush_dev
     2.77%  libteam.so.5.6.1      [.] 0x0000000000009e7d
     2.67%  libteam.so.5.6.1      [.] 0x0000000000006470
     1.90%  [kernel]              [k] fib_table_flush
     1.73%  [kernel]              [k] queued_read_lock_slowpath
     1.68%  [kernel]              [k] next_uptodate_page
     1.36%  ip                    [.] 0x0000000000011b74
     1.23%  ip                    [.] 0x00000000000121b0
     1.09%  [kernel]              [k] zap_pte_range
     0.99%  libteam.so.5.6.1      [.] 0x0000000000006501
     0.88%  dtvbras               [.] 0x0000000000014be8
     0.87%  [kernel]              [k] inet_dump_ifaddr
     0.74%  [kernel]              [k] filemap_map_pages
     0.72%  [kernel]              [k] neigh_flush_dev.isra.0
     0.66%  [kernel]              [k] snmp_get_cpu_field
     0.65%  [kernel]              [k] fib_table_insert
     0.63%  [kernel]              [k] native_irq_return_iret
     0.63%  libteam.so.5.6.1      [.] 0x0000000000005c78
     0.60%  [kernel]              [k] copy_page
     0.52%  libteam.so.5.6.1      [.] 0x000000000000647f
     0.50%  [kernel]              [k] _raw_spin_lock
     0.48%  libc.so.6             [.] 0x00000000000965a2
     0.45%  [kernel]              [k] _raw_read_lock_bh
     0.44%  [kernel]              [k] release_pages
     0.42%  [kernel]              [k] clear_page_erms
     0.42%  [kernel]              [k] page_remove_rmap
     0.41%  [kernel]              [k] queued_spin_lock_slowpath
     0.38%  [kernel]              [k] kmem_cache_alloc
     0.36%  [kernel]              [k] vma_interval_tree_insert
     0.36%  libteam.so.5.6.1      [.] 0x0000000000009e6f
     0.36%  [kernel]              [k] do_set_pte


sessions:
  starting: 296
  active: 3868
  finishing: 6748



> On 14 Sep 2021, at 14:00, Florian Westphal <fw@strlen.de> wrote:
> 
> Martin Zaharinov <micron10@gmail.com> wrote:
> 
> [ Trimming CC list ]
> 
>> Florian: 
>> 
>> If you make patch send to test please.
> 
> Attached.  No idea if it helps, but 'ip' should stay responsive
> even when masquerade processes netdevice events.
> <defer_masq_work.diff>


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: Urgent  Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected
  2021-09-14  9:50                         ` Florian Westphal
  2021-09-14 10:01                           ` Martin Zaharinov
@ 2021-09-14 10:53                           ` Martin Zaharinov
  1 sibling, 0 replies; 23+ messages in thread
From: Martin Zaharinov @ 2021-09-14 10:53 UTC (permalink / raw)
  To: Florian Westphal
  Cc: Guillaume Nault, Pali Rohár, Greg KH, netdev, Eric Dumazet

Florian Hi 
One more 

please see i try to remove nf_nat and xt_MASQUERADE 

and on time of problem need 50-80 sec to remove and overload system .

see perf from this moment: 




 PerfTop:    1738 irqs/sec  kernel:85.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    40.63%  [nf_conntrack]    [k] nf_ct_iterate_cleanup
    21.23%  [kernel]          [k] __local_bh_enable_ip
    10.93%  [kernel]          [k] __cond_resched
     9.20%  [kernel]          [k] _raw_spin_lock
     8.91%  [kernel]          [k] rcu_all_qs
     5.83%  [nf_conntrack]    [k] nf_conntrack_lock
     0.10%  [kernel]          [k] mutex_spin_on_owner
     0.08%  telegraf          [.] 0x0000000000021bf0
     0.06%  [kernel]          [k] osq_lock
     0.06%  [kernel]          [k] kallsyms_expand_symbol.constprop.0
     0.05%  [kernel]          [k] format_decode
     0.04%  [kernel]          [k] rtnl_fill_ifinfo.constprop.0.isra.0
     0.04%  perf              [.] 0x00000000000bc7b3
     0.04%  [kernel]          [k] memcpy_erms
     0.03%  [kernel]          [k] string
     0.03%  [kernel]          [k] menu_select
     0.03%  [kernel]          [k] nla_put
     0.03%  [kernel]          [k] vsnprintf



Martin

> On 14 Sep 2021, at 12:50, Florian Westphal <fw@strlen.de> wrote:
> 
> Guillaume Nault <gnault@redhat.com> wrote:
>>> And on time of problem when try to write : ip a 
>>> to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.
>> 
>> Probably some contention on the rtnl lock.
> 
> Yes, I'll create a patch.


^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, other threads:[~2021-09-16 20:00 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-05 20:53 Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected Martin Zaharinov
2021-08-06  4:40 ` Greg KH
2021-08-06  5:40   ` Martin Zaharinov
2021-08-08 15:14   ` Martin Zaharinov
2021-08-08 15:23     ` Pali Rohár
2021-08-08 15:29       ` Martin Zaharinov
2021-08-09 15:15         ` Pali Rohár
2021-08-10 18:27           ` Martin Zaharinov
2021-08-11 16:40             ` Guillaume Nault
2021-08-11 11:10           ` Martin Zaharinov
2021-08-11 16:48             ` Guillaume Nault
2021-09-07  6:16               ` Martin Zaharinov
2021-09-07  6:42                 ` Martin Zaharinov
2021-09-11  6:26                   ` Martin Zaharinov
2021-09-14  6:16                     ` Martin Zaharinov
2021-09-14  8:02                       ` Guillaume Nault
2021-09-14  9:50                         ` Florian Westphal
2021-09-14 10:01                           ` Martin Zaharinov
2021-09-14 11:00                             ` Florian Westphal
2021-09-15 14:25                               ` Martin Zaharinov
2021-09-15 14:37                                 ` Martin Zaharinov
2021-09-16 20:00                               ` Martin Zaharinov
2021-09-14 10:53                           ` Martin Zaharinov

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.