* Re: SCTP unable to bind after restart application
2019-11-25 13:57 SCTP unable to bind after restart application Emanuel Freitas
@ 2019-11-25 14:16 ` Neil Horman
2019-11-25 15:19 ` Emanuel Freitas
2019-11-25 21:00 ` Neil Horman
2 siblings, 0 replies; 4+ messages in thread
From: Neil Horman @ 2019-11-25 14:16 UTC (permalink / raw)
To: linux-sctp
On Mon, Nov 25, 2019 at 01:57:00PM +0000, Emanuel Freitas wrote:
> Hello,
>
> I had an application running on SCTP port 3890 for a few days and I
> stopped it (kill <pid>) for maintenance purposes. After that I’m not
> able to bind on port.
> The same situation happened in the past and the only way that I found
> to fix it was to restart the server. I was hoping that you can help me
> fixing this issue without restart.
>
> The application is not running and the port is not used by anything else:
> [root@server1 /]# netstat -lanp | grep 3890
> [root@server1 /]#
>
> I tried to use the sctp_test in order to exclude any issue on the
> application and it also cannot bind on that port (my IP address is
> replaced with <IPv4>):
>
> [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3890 -l
> local:addr=<IPv4>, port=ndsconnect, family=2
> seed = 1574684002
> Starting tests...
> socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 1/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 2/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 3/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 4/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 5/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 6/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 7/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 8/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 9/10
> bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 10/10
> Maximum bind() attempts. Die now...
>
> I have no issues while binding on other ports:
> [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3891 -l
> local:addr=<IPv4>, port=rtc-pm-port, family=2
> seed = 1574684925
> Starting tests...
> socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> bind(sk=3, [a:<IPv4>,p:rtc-pm-port]) -- attempt 1/10
> listen(sk=3,backlog\x100)
> Server: Receiving packets.
> recvmsg(sk=3) ^C
>
> There are no active SCTP associations:
> [root@server1 log]# tail /proc/net/sctp/* -n 10000
> => /proc/net/sctp/assocs <=
> ASSOC SOCK STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE
> LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC
> => /proc/net/sctp/eps <=
> ENDPT SOCK STY SST HBKT LPORT UID INODE LADDRS
> => /proc/net/sctp/remaddr <=
> ADDR ASSOC_ID HB_ACT RTO MAX_PATH_RTX REM_ADDR_RTX START
> => /proc/net/sctp/snmp <=
> SctpCurrEstab 0
> SctpActiveEstabs 0
> SctpPassiveEstabs 602
> SctpAborteds 13
> SctpShutdowns 589
> SctpOutOfBlues 29128
> SctpChecksumErrors 0
> SctpOutCtrlChunks 891800
> SctpOutOrderChunks 135693
> SctpOutUnorderChunks 0
> SctpInCtrlChunks 941831
> SctpInOrderChunks 122325
> SctpInUnorderChunks 13931
> SctpFragUsrMsgs 0
> SctpReasmUsrMsgs 0
> SctpOutSCTPPacks 1027573
> SctpInSCTPPacks 1035656
> SctpT1InitExpireds 0
> SctpT1CookieExpireds 0
> SctpT2ShutdownExpireds 0
> SctpT3RtxExpireds 81
> SctpT4RtoExpireds 0
> SctpT5ShutdownGuardExpireds 0
> SctpDelaySackExpireds 57489
> SctpAutocloseExpireds 0
> SctpT3Retransmits 80
> SctpPmtudRetransmits 0
> SctpFastRetransmits 0
> SctpInPktSoftirq 1035623
> SctpInPktBacklog 19
> SctpInPktDiscards 29139
> SctpInDataChunkDiscards 27869
>
>
> Other useful information:
> [root@server1 log]# uname -a
> Linux server1 2.6.32-754.11.1.el6.x86_64 #1 SMP Tue Feb 26 15:38:56
> UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
>
It looks kind of like theres a leak in endpoints here, but you are on a VERY old
kernel. The first thing you need to do is retest this on the latest upstream
kernel to see if the problem persists.
Neil
> [root@server1 log]# cat /etc/redhat-release
> CentOS release 6.10 (Final)
>
> [root@server1 log]# rpm -qa | grep sctp
> lksctp-tools-1.0.10-7.el6.x86_64
>
> I don’t find any relevant information on /var/log
> I also disabled IPv6 (although I’m not using it) in an attempt to
> isolate the issue but there was no difference.
>
> Thanks in advance!
>
> Kind regards,
>
> Emanuel
>
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: SCTP unable to bind after restart application
2019-11-25 13:57 SCTP unable to bind after restart application Emanuel Freitas
2019-11-25 14:16 ` Neil Horman
@ 2019-11-25 15:19 ` Emanuel Freitas
2019-11-25 21:00 ` Neil Horman
2 siblings, 0 replies; 4+ messages in thread
From: Emanuel Freitas @ 2019-11-25 15:19 UTC (permalink / raw)
To: linux-sctp
Hi Neil,
Thanks for the fast answer.
I'm using kernel-2.6.32-754 which is the latest official version
available for Centos 6 (and also RHEL Server release 6). If this is a
known issue it would be a good "excuse" for a major upgrade but this
operation has multiple impacts on the applications and I want to avoid
that unless it is really necessary.
On Mon, Nov 25, 2019 at 2:16 PM Neil Horman <nhorman@tuxdriver.com> wrote:
>
> On Mon, Nov 25, 2019 at 01:57:00PM +0000, Emanuel Freitas wrote:
> > Hello,
> >
> > I had an application running on SCTP port 3890 for a few days and I
> > stopped it (kill <pid>) for maintenance purposes. After that I’m not
> > able to bind on port.
> > The same situation happened in the past and the only way that I found
> > to fix it was to restart the server. I was hoping that you can help me
> > fixing this issue without restart.
> >
> > The application is not running and the port is not used by anything else:
> > [root@server1 /]# netstat -lanp | grep 3890
> > [root@server1 /]#
> >
> > I tried to use the sctp_test in order to exclude any issue on the
> > application and it also cannot bind on that port (my IP address is
> > replaced with <IPv4>):
> >
> > [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3890 -l
> > local:addr=<IPv4>, port=ndsconnect, family=2
> > seed = 1574684002
> > Starting tests...
> > socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 1/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 2/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 3/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 4/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 5/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 6/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 7/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 8/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 9/10
> > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 10/10
> > Maximum bind() attempts. Die now...
> >
> > I have no issues while binding on other ports:
> > [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3891 -l
> > local:addr=<IPv4>, port=rtc-pm-port, family=2
> > seed = 1574684925
> > Starting tests...
> > socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> > bind(sk=3, [a:<IPv4>,p:rtc-pm-port]) -- attempt 1/10
> > listen(sk=3,backlog=100)
> > Server: Receiving packets.
> > recvmsg(sk=3) ^C
> >
> > There are no active SCTP associations:
> > [root@server1 log]# tail /proc/net/sctp/* -n 10000
> > ==> /proc/net/sctp/assocs <==
> > ASSOC SOCK STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE
> > LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC
> > ==> /proc/net/sctp/eps <==
> > ENDPT SOCK STY SST HBKT LPORT UID INODE LADDRS
> > ==> /proc/net/sctp/remaddr <==
> > ADDR ASSOC_ID HB_ACT RTO MAX_PATH_RTX REM_ADDR_RTX START
> > ==> /proc/net/sctp/snmp <==
> > SctpCurrEstab 0
> > SctpActiveEstabs 0
> > SctpPassiveEstabs 602
> > SctpAborteds 13
> > SctpShutdowns 589
> > SctpOutOfBlues 29128
> > SctpChecksumErrors 0
> > SctpOutCtrlChunks 891800
> > SctpOutOrderChunks 135693
> > SctpOutUnorderChunks 0
> > SctpInCtrlChunks 941831
> > SctpInOrderChunks 122325
> > SctpInUnorderChunks 13931
> > SctpFragUsrMsgs 0
> > SctpReasmUsrMsgs 0
> > SctpOutSCTPPacks 1027573
> > SctpInSCTPPacks 1035656
> > SctpT1InitExpireds 0
> > SctpT1CookieExpireds 0
> > SctpT2ShutdownExpireds 0
> > SctpT3RtxExpireds 81
> > SctpT4RtoExpireds 0
> > SctpT5ShutdownGuardExpireds 0
> > SctpDelaySackExpireds 57489
> > SctpAutocloseExpireds 0
> > SctpT3Retransmits 80
> > SctpPmtudRetransmits 0
> > SctpFastRetransmits 0
> > SctpInPktSoftirq 1035623
> > SctpInPktBacklog 19
> > SctpInPktDiscards 29139
> > SctpInDataChunkDiscards 27869
> >
> >
> > Other useful information:
> > [root@server1 log]# uname -a
> > Linux server1 2.6.32-754.11.1.el6.x86_64 #1 SMP Tue Feb 26 15:38:56
> > UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> >
> It looks kind of like theres a leak in endpoints here, but you are on a VERY old
> kernel. The first thing you need to do is retest this on the latest upstream
> kernel to see if the problem persists.
>
> Neil
>
> > [root@server1 log]# cat /etc/redhat-release
> > CentOS release 6.10 (Final)
> >
> > [root@server1 log]# rpm -qa | grep sctp
> > lksctp-tools-1.0.10-7.el6.x86_64
> >
> > I don’t find any relevant information on /var/log
> > I also disabled IPv6 (although I’m not using it) in an attempt to
> > isolate the issue but there was no difference.
> >
> > Thanks in advance!
> >
> > Kind regards,
> >
> > Emanuel
> >
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: SCTP unable to bind after restart application
2019-11-25 13:57 SCTP unable to bind after restart application Emanuel Freitas
2019-11-25 14:16 ` Neil Horman
2019-11-25 15:19 ` Emanuel Freitas
@ 2019-11-25 21:00 ` Neil Horman
2 siblings, 0 replies; 4+ messages in thread
From: Neil Horman @ 2019-11-25 21:00 UTC (permalink / raw)
To: linux-sctp
On Mon, Nov 25, 2019 at 03:19:10PM +0000, Emanuel Freitas wrote:
> Hi Neil,
>
> Thanks for the fast answer.
> I'm using kernel-2.6.32-754 which is the latest official version
> available for Centos 6 (and also RHEL Server release 6). If this is a
> known issue it would be a good "excuse" for a major upgrade but this
> operation has multiple impacts on the applications and I want to avoid
> that unless it is really necessary.
>
I don't know of any existing problem on that kernel, only that I haven't tested
with it in a few years now.
I can tell you that testing on my local fedora 30 system with sctp_darn by
establishing a connection between two localhost ports, then kill -9-ing both
processes leads to no such problem. I'm able to restablish the connection on
the previously used ports without issue, but running the same commands again.
I can setup a CentOS 6 VM if you like and try again, but 6 is effectively EOL at
this point, in that RHEL won't make updates for anything but critical security
bugs, and this doesn't fit the bill. So I would suggest that you test on a
later kernel, and if the problem abates, that should be your motivation to
upgrade.
Best
Neil
>
>
>
>
> On Mon, Nov 25, 2019 at 2:16 PM Neil Horman <nhorman@tuxdriver.com> wrote:
> >
> > On Mon, Nov 25, 2019 at 01:57:00PM +0000, Emanuel Freitas wrote:
> > > Hello,
> > >
> > > I had an application running on SCTP port 3890 for a few days and I
> > > stopped it (kill <pid>) for maintenance purposes. After that I’m not
> > > able to bind on port.
> > > The same situation happened in the past and the only way that I found
> > > to fix it was to restart the server. I was hoping that you can help me
> > > fixing this issue without restart.
> > >
> > > The application is not running and the port is not used by anything else:
> > > [root@server1 /]# netstat -lanp | grep 3890
> > > [root@server1 /]#
> > >
> > > I tried to use the sctp_test in order to exclude any issue on the
> > > application and it also cannot bind on that port (my IP address is
> > > replaced with <IPv4>):
> > >
> > > [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3890 -l
> > > local:addr=<IPv4>, port=ndsconnect, family=2
> > > seed = 1574684002
> > > Starting tests...
> > > socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 1/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 2/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 3/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 4/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 5/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 6/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 7/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 8/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 9/10
> > > bind(sk=3, [a:<IPv4>,p:ndsconnect]) -- attempt 10/10
> > > Maximum bind() attempts. Die now...
> > >
> > > I have no issues while binding on other ports:
> > > [root@server1 /]# /usr/bin/sctp_test -H <IPv4> -P 3891 -l
> > > local:addr=<IPv4>, port=rtc-pm-port, family=2
> > > seed = 1574684925
> > > Starting tests...
> > > socket(SOCK_SEQPACKET, IPPROTO_SCTP) -> sk=3
> > > bind(sk=3, [a:<IPv4>,p:rtc-pm-port]) -- attempt 1/10
> > > listen(sk=3,backlog\x100)
> > > Server: Receiving packets.
> > > recvmsg(sk=3) ^C
> > >
> > > There are no active SCTP associations:
> > > [root@server1 log]# tail /proc/net/sctp/* -n 10000
> > > => /proc/net/sctp/assocs <=
> > > ASSOC SOCK STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE
> > > LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC
> > > => /proc/net/sctp/eps <=
> > > ENDPT SOCK STY SST HBKT LPORT UID INODE LADDRS
> > > => /proc/net/sctp/remaddr <=
> > > ADDR ASSOC_ID HB_ACT RTO MAX_PATH_RTX REM_ADDR_RTX START
> > > => /proc/net/sctp/snmp <=
> > > SctpCurrEstab 0
> > > SctpActiveEstabs 0
> > > SctpPassiveEstabs 602
> > > SctpAborteds 13
> > > SctpShutdowns 589
> > > SctpOutOfBlues 29128
> > > SctpChecksumErrors 0
> > > SctpOutCtrlChunks 891800
> > > SctpOutOrderChunks 135693
> > > SctpOutUnorderChunks 0
> > > SctpInCtrlChunks 941831
> > > SctpInOrderChunks 122325
> > > SctpInUnorderChunks 13931
> > > SctpFragUsrMsgs 0
> > > SctpReasmUsrMsgs 0
> > > SctpOutSCTPPacks 1027573
> > > SctpInSCTPPacks 1035656
> > > SctpT1InitExpireds 0
> > > SctpT1CookieExpireds 0
> > > SctpT2ShutdownExpireds 0
> > > SctpT3RtxExpireds 81
> > > SctpT4RtoExpireds 0
> > > SctpT5ShutdownGuardExpireds 0
> > > SctpDelaySackExpireds 57489
> > > SctpAutocloseExpireds 0
> > > SctpT3Retransmits 80
> > > SctpPmtudRetransmits 0
> > > SctpFastRetransmits 0
> > > SctpInPktSoftirq 1035623
> > > SctpInPktBacklog 19
> > > SctpInPktDiscards 29139
> > > SctpInDataChunkDiscards 27869
> > >
> > >
> > > Other useful information:
> > > [root@server1 log]# uname -a
> > > Linux server1 2.6.32-754.11.1.el6.x86_64 #1 SMP Tue Feb 26 15:38:56
> > > UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > It looks kind of like theres a leak in endpoints here, but you are on a VERY old
> > kernel. The first thing you need to do is retest this on the latest upstream
> > kernel to see if the problem persists.
> >
> > Neil
> >
> > > [root@server1 log]# cat /etc/redhat-release
> > > CentOS release 6.10 (Final)
> > >
> > > [root@server1 log]# rpm -qa | grep sctp
> > > lksctp-tools-1.0.10-7.el6.x86_64
> > >
> > > I don’t find any relevant information on /var/log
> > > I also disabled IPv6 (although I’m not using it) in an attempt to
> > > isolate the issue but there was no difference.
> > >
> > > Thanks in advance!
> > >
> > > Kind regards,
> > >
> > > Emanuel
> > >
>
^ permalink raw reply [flat|nested] 4+ messages in thread