Heartbeat on closed SCTP sockets?

Heartbeat on closed SCTP sockets?

[Andreas Fink]

Hello all,

We are trying to debug a very strange case here and would like to hear your input.

Here is what we have

1. we have a application which listens on a  point to multipoint SCTP socket
2. when a incoming connection comes in and it matches a preconfigured one, it peels of that socket and a separate thread is starting communication on the upper layer.
3. when it doesnt match, an abort is triggered (that part might not work yet though).


Now we have multiple connections to different vendors and we have traces where we can see that there was a temporary issue on the IP layer and associations get shutdown and restarted.
After the IP layer resolved, all connection came up except two which go to the same peer and vendor.

What we now see in netstat --sctp is:

we have a LISTEN on port 2010
we have a  association from port 2010 to the remote in status CLOSED

in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.

We now kill the application

What we now see in netstat --sctp is:
we no longer listen on port 2010
we have a closed association from port 2010 to the remote.

in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
We do not see any SHUTDOWN or INIT even if we restart the application.

Can anyone explain how this can be?

We are using kernel linux-image-5.4.0-0.bpo.4-amd64 from the Debian Backport repositiory on Debian 10.

The issue seems to be related that the remote side never closes the SCTP assoc but simply tries to restart the upper layers while other vendors time out on upper layers and restart the SCTP assoc.
Restarting it from my application outbound also didnt help. Kernel somehow still remembers there's something up where theres clearly not.

The only solution to get this assoc back alive is to reboot the whole machine it seems.

Thanks for any input.

Heartbeat on closed SCTP sockets?

[Andreas Fink]

Hello all,

We are trying to debug a very strange case here and would like to hear your input.

Here is what we have

1. we have a application which listens on a  point to multipoint SCTP socket
2. when a incoming connection comes in and it matches a preconfigured one, it peels of that socket and a separate thread is starting communication on the upper layer.
3. when it doesnt match, an abort is triggered (that part might not work yet though).


Now we have multiple connections to different vendors and we have traces where we can see that there was a temporary issue on the IP layer and associations get shutdown and restarted.
After the IP layer resolved, all connection came up except two which go to the same peer and vendor.

What we now see in netstat --sctp is:

we have a LISTEN on port 2010
we have a  association from port 2010 to the remote in status CLOSED

in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.

We now kill the application

What we now see in netstat --sctp is:
we no longer listen on port 2010
we have a closed association from port 2010 to the remote.

in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
We do not see any SHUTDOWN or INIT even if we restart the application.

Can anyone explain how this can be?

We are using kernel linux-image-5.4.0-0.bpo.4-amd64 from the Debian Backport repositiory on Debian 10.

The issue seems to be related that the remote side never closes the SCTP assoc but simply tries to restart the upper layers while other vendors time out on upper layers and restart the SCTP assoc.
Restarting it from my application outbound also didnt help. Kernel somehow still remembers there's something up where theres clearly not.

The only solution to get this assoc back alive is to reboot the whole machine it seems.

Thanks for any input.

Re: Heartbeat on closed SCTP sockets?

[Marcelo Ricardo Leitner]

Hi,

On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
...
> What we now see in netstat --sctp is:
> 
> we have a LISTEN on port 2010
> we have a  association from port 2010 to the remote in status CLOSED
> 
> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.

That's weird. If it is in CLOSED, then the stack should be handling
it as an OOTB packet and trigger an Abort.

> 
> We now kill the application
> 
> What we now see in netstat --sctp is:
> we no longer listen on port 2010
> we have a closed association from port 2010 to the remote.
> 
> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
> We do not see any SHUTDOWN or INIT even if we restart the application.
> 
> Can anyone explain how this can be?

Please check the assoc status as well, via 'ss -a --sctp' and
/proc/net/sctp/assocs . Maybe it got out of sync of the socket status.

  Marcelo

Re: Heartbeat on closed SCTP sockets?

[Marcelo Ricardo Leitner]

Hi,

On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
...
> What we now see in netstat --sctp is:
> 
> we have a LISTEN on port 2010
> we have a  association from port 2010 to the remote in status CLOSED
> 
> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.

That's weird. If it is in CLOSED, then the stack should be handling
it as an OOTB packet and trigger an Abort.

> 
> We now kill the application
> 
> What we now see in netstat --sctp is:
> we no longer listen on port 2010
> we have a closed association from port 2010 to the remote.
> 
> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
> We do not see any SHUTDOWN or INIT even if we restart the application.
> 
> Can anyone explain how this can be?

Please check the assoc status as well, via 'ss -a --sctp' and
/proc/net/sctp/assocs . Maybe it got out of sync of the socket status.

  Marcelo

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

here are the matching entries:

ss -a --sctp
State                        Recv-Q                      Send-Q                                                         Local Address:Port                                                            Peer Address:Port
LISTEN                       0                           128                                                   [::ffff:194.0.137.189]:2020                                                                       *:*
UNCONN                       0                           0                                                     [::ffff:194.0.137.189]:2020                                                                       *:1202
SYN-RECV                     0                           0                                                [::ffff:194.0.137.234]%eth0:2020                                                                       *:1202

from /proc/net/sctp/assocs

 ASSOC     SOCK   STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC wmema wmemq sndbuf rcvbuf
6947dc99 7e08fe9b 1   7   3  0       5        0        0       0     0 2020   1202  194.0.137.234 194.0.137.189 <-> *93.95.48.82 93.95.48.90        7500     2    10   10    0    2        9        1        0   212992   212992

The application opens up a IPv6 socket with IPv4 conversion.
It opens a listener socket on 2020 and had a connection with the indicated IPs (its multihomed with two IPs).
The entry in /proc/net/sctp/assocs refers to the association which is being kept alive by the remote
but in ss -a --sctp you can see its "unconnected" and listening. So the connection above should be handed over to the listener port and a SCTP UP message should be delivered to the application upon which my application would call sctp_peeloff and everything would restart.


quite confusing.


> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> 
> Hi,
> 
> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
> ...
>> What we now see in netstat --sctp is:
>> 
>> we have a LISTEN on port 2010
>> we have a  association from port 2010 to the remote in status CLOSED
>> 
>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
> 
> That's weird. If it is in CLOSED, then the stack should be handling
> it as an OOTB packet and trigger an Abort.
> 
>> 
>> We now kill the application
>> 
>> What we now see in netstat --sctp is:
>> we no longer listen on port 2010
>> we have a closed association from port 2010 to the remote.
>> 
>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>> We do not see any SHUTDOWN or INIT even if we restart the application.
>> 
>> Can anyone explain how this can be?
> 
> Please check the assoc status as well, via 'ss -a --sctp' and
> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
> 
>  Marcelo

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

here are the matching entries:

ss -a --sctp
State                        Recv-Q                      Send-Q                                                         Local Address:Port                                                            Peer Address:Port
LISTEN                       0                           128                                                   [::ffff:194.0.137.189]:2020                                                                       *:*
UNCONN                       0                           0                                                     [::ffff:194.0.137.189]:2020                                                                       *:1202
SYN-RECV                     0                           0                                                [::ffff:194.0.137.234]%eth0:2020                                                                       *:1202

from /proc/net/sctp/assocs

 ASSOC     SOCK   STY SST ST HBKT ASSOC-ID TX_QUEUE RX_QUEUE UID INODE LPORT RPORT LADDRS <-> RADDRS HBINT INS OUTS MAXRT T1X T2X RTXC wmema wmemq sndbuf rcvbuf
6947dc99 7e08fe9b 1   7   3  0       5        0        0       0     0 2020   1202  194.0.137.234 194.0.137.189 <-> *93.95.48.82 93.95.48.90        7500     2    10   10    0    2        9        1        0   212992   212992

The application opens up a IPv6 socket with IPv4 conversion.
It opens a listener socket on 2020 and had a connection with the indicated IPs (its multihomed with two IPs).
The entry in /proc/net/sctp/assocs refers to the association which is being kept alive by the remote
but in ss -a --sctp you can see its "unconnected" and listening. So the connection above should be handed over to the listener port and a SCTP UP message should be delivered to the application upon which my application would call sctp_peeloff and everything would restart.


quite confusing.


> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> 
> Hi,
> 
> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
> ...
>> What we now see in netstat --sctp is:
>> 
>> we have a LISTEN on port 2010
>> we have a  association from port 2010 to the remote in status CLOSED
>> 
>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
> 
> That's weird. If it is in CLOSED, then the stack should be handling
> it as an OOTB packet and trigger an Abort.
> 
>> 
>> We now kill the application
>> 
>> What we now see in netstat --sctp is:
>> we no longer listen on port 2010
>> we have a closed association from port 2010 to the remote.
>> 
>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>> We do not see any SHUTDOWN or INIT even if we restart the application.
>> 
>> Can anyone explain how this can be?
> 
> Please check the assoc status as well, via 'ss -a --sctp' and
> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
> 
>  Marcelo

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

by reading the linux diver source I discovered this code segment in input.c around line 188


/*
	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
	 * An SCTP packet is called an "out of the blue" (OOTB)
	 * packet if it is correctly formed, i.e., passed the
	 * receiver's checksum check, but the receiver is not
	 * able to identify the association to which this
	 * packet belongs.
	 */
	
if (!asoc) {
	if (sctp_rcv_ootb(skb)) {
		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
		goto discard_release;
	}
}

This means out of the blue packets are always ignored and dropped.

the RFC however says:

   8) The receiver should respond to the sender of the OOTB packet with
      an ABORT.  When sending the ABORT, the receiver of the OOTB packet
      MUST fill in the Verification Tag field of the outbound packet
      with the value found in the Verification Tag field of the OOTB
      packet and set the T-bit in the Chunk Flags to indicate that no
      TCB was found.  After sending this ABORT, the receiver of the OOTB
      packet shall discard the OOTB packet and take no further action.

I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
There must be a second issue that the socket structures are not in sync up.


> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> 
> Hi,
> 
> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
> ...
>> What we now see in netstat --sctp is:
>> 
>> we have a LISTEN on port 2010
>> we have a  association from port 2010 to the remote in status CLOSED
>> 
>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
> 
> That's weird. If it is in CLOSED, then the stack should be handling
> it as an OOTB packet and trigger an Abort.
> 
>> 
>> We now kill the application
>> 
>> What we now see in netstat --sctp is:
>> we no longer listen on port 2010
>> we have a closed association from port 2010 to the remote.
>> 
>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>> We do not see any SHUTDOWN or INIT even if we restart the application.
>> 
>> Can anyone explain how this can be?
> 
> Please check the assoc status as well, via 'ss -a --sctp' and
> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
> 
>  Marcelo

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

by reading the linux diver source I discovered this code segment in input.c around line 188


/*
	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
	 * An SCTP packet is called an "out of the blue" (OOTB)
	 * packet if it is correctly formed, i.e., passed the
	 * receiver's checksum check, but the receiver is not
	 * able to identify the association to which this
	 * packet belongs.
	 */
	
if (!asoc) {
	if (sctp_rcv_ootb(skb)) {
		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
		goto discard_release;
	}
}

This means out of the blue packets are always ignored and dropped.

the RFC however says:

   8) The receiver should respond to the sender of the OOTB packet with
      an ABORT.  When sending the ABORT, the receiver of the OOTB packet
      MUST fill in the Verification Tag field of the outbound packet
      with the value found in the Verification Tag field of the OOTB
      packet and set the T-bit in the Chunk Flags to indicate that no
      TCB was found.  After sending this ABORT, the receiver of the OOTB
      packet shall discard the OOTB packet and take no further action.

I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
There must be a second issue that the socket structures are not in sync up.


> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> 
> Hi,
> 
> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
> ...
>> What we now see in netstat --sctp is:
>> 
>> we have a LISTEN on port 2010
>> we have a  association from port 2010 to the remote in status CLOSED
>> 
>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
> 
> That's weird. If it is in CLOSED, then the stack should be handling
> it as an OOTB packet and trigger an Abort.
> 
>> 
>> We now kill the application
>> 
>> What we now see in netstat --sctp is:
>> we no longer listen on port 2010
>> we have a closed association from port 2010 to the remote.
>> 
>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>> We do not see any SHUTDOWN or INIT even if we restart the application.
>> 
>> Can anyone explain how this can be?
> 
> Please check the assoc status as well, via 'ss -a --sctp' and
> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
> 
>  Marcelo

RE: Heartbeat on closed SCTP sockets?

[David Laight]

From: Andreas Fink
> Sent: 08 October 2020 07:40
> 
> by reading the linux diver source I discovered this code segment in input.c around line 188
> 
> 
> /*
> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
> 	 * An SCTP packet is called an "out of the blue" (OOTB)
> 	 * packet if it is correctly formed, i.e., passed the
> 	 * receiver's checksum check, but the receiver is not
> 	 * able to identify the association to which this
> 	 * packet belongs.
> 	 */
> 
> if (!asoc) {
> 	if (sctp_rcv_ootb(skb)) {
> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
> 		goto discard_release;
> 	}
> }
> 
> This means out of the blue packets are always ignored and dropped.
> 
> the RFC however says:
> 
>    8) The receiver should respond to the sender of the OOTB packet with
>       an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>       MUST fill in the Verification Tag field of the outbound packet
>       with the value found in the Verification Tag field of the OOTB
>       packet and set the T-bit in the Chunk Flags to indicate that no
>       TCB was found.  After sending this ABORT, the receiver of the OOTB
>       packet shall discard the OOTB packet and take no further action.
> 
> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means
> the remote doesnt reset the connection.
> There must be a second issue that the socket structures are not in sync up.

But I'm not sure sending an ABORT is right.
You need to stop the remote sending messages to your IP address
not abort the SCTP connection itself.
Especially on an unverified path.

It is pretty easy to get an IP address that is just 'wrong'.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

RE: Heartbeat on closed SCTP sockets?

[David Laight]

From: Andreas Fink
> Sent: 08 October 2020 07:40
> 
> by reading the linux diver source I discovered this code segment in input.c around line 188
> 
> 
> /*
> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
> 	 * An SCTP packet is called an "out of the blue" (OOTB)
> 	 * packet if it is correctly formed, i.e., passed the
> 	 * receiver's checksum check, but the receiver is not
> 	 * able to identify the association to which this
> 	 * packet belongs.
> 	 */
> 
> if (!asoc) {
> 	if (sctp_rcv_ootb(skb)) {
> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
> 		goto discard_release;
> 	}
> }
> 
> This means out of the blue packets are always ignored and dropped.
> 
> the RFC however says:
> 
>    8) The receiver should respond to the sender of the OOTB packet with
>       an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>       MUST fill in the Verification Tag field of the outbound packet
>       with the value found in the Verification Tag field of the OOTB
>       packet and set the T-bit in the Chunk Flags to indicate that no
>       TCB was found.  After sending this ABORT, the receiver of the OOTB
>       packet shall discard the OOTB packet and take no further action.
> 
> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means
> the remote doesnt reset the connection.
> There must be a second issue that the socket structures are not in sync up.

But I'm not sure sending an ABORT is right.
You need to stop the remote sending messages to your IP address
not abort the SCTP connection itself.
Especially on an unverified path.

It is pretty easy to get an IP address that is just 'wrong'.

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)

Re: Heartbeat on closed SCTP sockets?

[Michael Tuexen]

> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
> 
> by reading the linux diver source I discovered this code segment in input.c around line 188
> 
> 
> /*
> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
> 	 * An SCTP packet is called an "out of the blue" (OOTB)
> 	 * packet if it is correctly formed, i.e., passed the
> 	 * receiver's checksum check, but the receiver is not
> 	 * able to identify the association to which this
> 	 * packet belongs.
> 	 */
> 	
> if (!asoc) {
> 	if (sctp_rcv_ootb(skb)) {
> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
> 		goto discard_release;
> 	}
> }
The above code looks good. Have a look at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666

RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.
> 
> This means out of the blue packets are always ignored and dropped.
> 
> the RFC however says:
> 
>   8) The receiver should respond to the sender of the OOTB packet with
>      an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>      MUST fill in the Verification Tag field of the outbound packet
>      with the value found in the Verification Tag field of the OOTB
>      packet and set the T-bit in the Chunk Flags to indicate that no
>      TCB was found.  After sending this ABORT, the receiver of the OOTB
>      packet shall discard the OOTB packet and take no further action.
> 
> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
What are those OOTB messages? Which chunks do they contain?

Bes regards
Michael
> There must be a second issue that the socket structures are not in sync up.
> 
> 
>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>> 
>> Hi,
>> 
>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>> ...
>>> What we now see in netstat --sctp is:
>>> 
>>> we have a LISTEN on port 2010
>>> we have a  association from port 2010 to the remote in status CLOSED
>>> 
>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>> 
>> That's weird. If it is in CLOSED, then the stack should be handling
>> it as an OOTB packet and trigger an Abort.
>> 
>>> 
>>> We now kill the application
>>> 
>>> What we now see in netstat --sctp is:
>>> we no longer listen on port 2010
>>> we have a closed association from port 2010 to the remote.
>>> 
>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>> 
>>> Can anyone explain how this can be?
>> 
>> Please check the assoc status as well, via 'ss -a --sctp' and
>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>> 
>> Marcelo
> 
> 

Re: Heartbeat on closed SCTP sockets?

[Michael Tuexen]

> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
> 
> by reading the linux diver source I discovered this code segment in input.c around line 188
> 
> 
> /*
> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
> 	 * An SCTP packet is called an "out of the blue" (OOTB)
> 	 * packet if it is correctly formed, i.e., passed the
> 	 * receiver's checksum check, but the receiver is not
> 	 * able to identify the association to which this
> 	 * packet belongs.
> 	 */
> 	
> if (!asoc) {
> 	if (sctp_rcv_ootb(skb)) {
> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
> 		goto discard_release;
> 	}
> }
The above code looks good. Have a look at
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666

RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.
> 
> This means out of the blue packets are always ignored and dropped.
> 
> the RFC however says:
> 
>   8) The receiver should respond to the sender of the OOTB packet with
>      an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>      MUST fill in the Verification Tag field of the outbound packet
>      with the value found in the Verification Tag field of the OOTB
>      packet and set the T-bit in the Chunk Flags to indicate that no
>      TCB was found.  After sending this ABORT, the receiver of the OOTB
>      packet shall discard the OOTB packet and take no further action.
> 
> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
What are those OOTB messages? Which chunks do they contain?

Bes regards
Michael
> There must be a second issue that the socket structures are not in sync up.
> 
> 
>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>> 
>> Hi,
>> 
>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>> ...
>>> What we now see in netstat --sctp is:
>>> 
>>> we have a LISTEN on port 2010
>>> we have a  association from port 2010 to the remote in status CLOSED
>>> 
>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>> 
>> That's weird. If it is in CLOSED, then the stack should be handling
>> it as an OOTB packet and trigger an Abort.
>> 
>>> 
>>> We now kill the application
>>> 
>>> What we now see in netstat --sctp is:
>>> we no longer listen on port 2010
>>> we have a closed association from port 2010 to the remote.
>>> 
>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>> 
>>> Can anyone explain how this can be?
>> 
>> Please check the assoc status as well, via 'ss -a --sctp' and
>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>> 
>> Marcelo
> 
> 

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

> On 8 Oct 2020, at 11:08, Michael Tuexen <michael.tuexen@lurchi.franken.de> wrote:
> 
>> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
>> 
>> by reading the linux diver source I discovered this code segment in input.c around line 188
>> 
>> 
>> /*
>> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
>> 	 * An SCTP packet is called an "out of the blue" (OOTB)
>> 	 * packet if it is correctly formed, i.e., passed the
>> 	 * receiver's checksum check, but the receiver is not
>> 	 * able to identify the association to which this
>> 	 * packet belongs.
>> 	 */
>> 	
>> if (!asoc) {
>> 	if (sctp_rcv_ootb(skb)) {
>> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
>> 		goto discard_release;
>> 	}
>> }
> The above code looks good. Have a look at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666

So where does it generate the ABORT? I don't see it. I can only it checks for an incoming ABORT but its not generating an outgoing ABORT for the packets which are not handled.

> 
> RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.

Its ok to drop the packet and not hand it over to any socket (after all theres' no open socket at that moment around to handle it anyway). But we should inform the remote that this packet is not being processed by sending ABORT. That way the remote would start with INIT the next time and things probably come back to live.

>> 
>> This means out of the blue packets are always ignored and dropped.
>> 
>> the RFC however says:
>> 
>>  8) The receiver should respond to the sender of the OOTB packet with
>>     an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>>     MUST fill in the Verification Tag field of the outbound packet
>>     with the value found in the Verification Tag field of the OOTB
>>     packet and set the T-bit in the Chunk Flags to indicate that no
>>     TCB was found.  After sending this ABORT, the receiver of the OOTB
>>     packet shall discard the OOTB packet and take no further action.
>> 
>> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
> What are those OOTB messages? Which chunks do they contain?

They contain HEARTBEAT for example.

But because we are indeed sending HEATBEAT ACK back, they are handled. 
This means the kernel must think they are not OOTB but some established assoc. Only its status is CLOSED.


> 
> Bes regards
> Michael
>> There must be a second issue that the socket structures are not in sync up.
>> 
>> 
>>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>>> ...
>>>> What we now see in netstat --sctp is:
>>>> 
>>>> we have a LISTEN on port 2010
>>>> we have a  association from port 2010 to the remote in status CLOSED
>>>> 
>>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>>> 
>>> That's weird. If it is in CLOSED, then the stack should be handling
>>> it as an OOTB packet and trigger an Abort.
>>> 
>>>> 
>>>> We now kill the application
>>>> 
>>>> What we now see in netstat --sctp is:
>>>> we no longer listen on port 2010
>>>> we have a closed association from port 2010 to the remote.
>>>> 
>>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>>> 
>>>> Can anyone explain how this can be?
>>> 
>>> Please check the assoc status as well, via 'ss -a --sctp' and
>>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>>> 
>>> Marcelo
>> 
>> 
> 

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

> On 8 Oct 2020, at 11:08, Michael Tuexen <michael.tuexen@lurchi.franken.de> wrote:
> 
>> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
>> 
>> by reading the linux diver source I discovered this code segment in input.c around line 188
>> 
>> 
>> /*
>> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
>> 	 * An SCTP packet is called an "out of the blue" (OOTB)
>> 	 * packet if it is correctly formed, i.e., passed the
>> 	 * receiver's checksum check, but the receiver is not
>> 	 * able to identify the association to which this
>> 	 * packet belongs.
>> 	 */
>> 	
>> if (!asoc) {
>> 	if (sctp_rcv_ootb(skb)) {
>> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
>> 		goto discard_release;
>> 	}
>> }
> The above code looks good. Have a look at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666

So where does it generate the ABORT? I don't see it. I can only it checks for an incoming ABORT but its not generating an outgoing ABORT for the packets which are not handled.

> 
> RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.

Its ok to drop the packet and not hand it over to any socket (after all theres' no open socket at that moment around to handle it anyway). But we should inform the remote that this packet is not being processed by sending ABORT. That way the remote would start with INIT the next time and things probably come back to live.

>> 
>> This means out of the blue packets are always ignored and dropped.
>> 
>> the RFC however says:
>> 
>>  8) The receiver should respond to the sender of the OOTB packet with
>>     an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>>     MUST fill in the Verification Tag field of the outbound packet
>>     with the value found in the Verification Tag field of the OOTB
>>     packet and set the T-bit in the Chunk Flags to indicate that no
>>     TCB was found.  After sending this ABORT, the receiver of the OOTB
>>     packet shall discard the OOTB packet and take no further action.
>> 
>> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
> What are those OOTB messages? Which chunks do they contain?

They contain HEARTBEAT for example.

But because we are indeed sending HEATBEAT ACK back, they are handled. 
This means the kernel must think they are not OOTB but some established assoc. Only its status is CLOSED.


> 
> Bes regards
> Michael
>> There must be a second issue that the socket structures are not in sync up.
>> 
>> 
>>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>>> ...
>>>> What we now see in netstat --sctp is:
>>>> 
>>>> we have a LISTEN on port 2010
>>>> we have a  association from port 2010 to the remote in status CLOSED
>>>> 
>>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>>> 
>>> That's weird. If it is in CLOSED, then the stack should be handling
>>> it as an OOTB packet and trigger an Abort.
>>> 
>>>> 
>>>> We now kill the application
>>>> 
>>>> What we now see in netstat --sctp is:
>>>> we no longer listen on port 2010
>>>> we have a closed association from port 2010 to the remote.
>>>> 
>>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>>> 
>>>> Can anyone explain how this can be?
>>> 
>>> Please check the assoc status as well, via 'ss -a --sctp' and
>>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>>> 
>>> Marcelo
>> 
>> 
> 

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

PS wheres the repo of the latest version hosted?
I want to check it out and do some debugging

> On 8 Oct 2020, at 11:08, Michael Tuexen <michael.tuexen@lurchi.franken.de> wrote:
> 
>> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
>> 
>> by reading the linux diver source I discovered this code segment in input.c around line 188
>> 
>> 
>> /*
>> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
>> 	 * An SCTP packet is called an "out of the blue" (OOTB)
>> 	 * packet if it is correctly formed, i.e., passed the
>> 	 * receiver's checksum check, but the receiver is not
>> 	 * able to identify the association to which this
>> 	 * packet belongs.
>> 	 */
>> 	
>> if (!asoc) {
>> 	if (sctp_rcv_ootb(skb)) {
>> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
>> 		goto discard_release;
>> 	}
>> }
> The above code looks good. Have a look at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666
> 
> RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.
>> 
>> This means out of the blue packets are always ignored and dropped.
>> 
>> the RFC however says:
>> 
>>  8) The receiver should respond to the sender of the OOTB packet with
>>     an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>>     MUST fill in the Verification Tag field of the outbound packet
>>     with the value found in the Verification Tag field of the OOTB
>>     packet and set the T-bit in the Chunk Flags to indicate that no
>>     TCB was found.  After sending this ABORT, the receiver of the OOTB
>>     packet shall discard the OOTB packet and take no further action.
>> 
>> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
> What are those OOTB messages? Which chunks do they contain?
> 
> Bes regards
> Michael
>> There must be a second issue that the socket structures are not in sync up.
>> 
>> 
>>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>>> ...
>>>> What we now see in netstat --sctp is:
>>>> 
>>>> we have a LISTEN on port 2010
>>>> we have a  association from port 2010 to the remote in status CLOSED
>>>> 
>>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>>> 
>>> That's weird. If it is in CLOSED, then the stack should be handling
>>> it as an OOTB packet and trigger an Abort.
>>> 
>>>> 
>>>> We now kill the application
>>>> 
>>>> What we now see in netstat --sctp is:
>>>> we no longer listen on port 2010
>>>> we have a closed association from port 2010 to the remote.
>>>> 
>>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>>> 
>>>> Can anyone explain how this can be?
>>> 
>>> Please check the assoc status as well, via 'ss -a --sctp' and
>>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>>> 
>>> Marcelo
>> 
>> 
> 

Re: Heartbeat on closed SCTP sockets?

[Andreas Fink]

PS wheres the repo of the latest version hosted?
I want to check it out and do some debugging

> On 8 Oct 2020, at 11:08, Michael Tuexen <michael.tuexen@lurchi.franken.de> wrote:
> 
>> On 8. Oct 2020, at 08:40, Andreas Fink <afink@list.fink.org> wrote:
>> 
>> by reading the linux diver source I discovered this code segment in input.c around line 188
>> 
>> 
>> /*
>> 	 * RFC 2960, 8.4 - Handle "Out of the blue" Packets.
>> 	 * An SCTP packet is called an "out of the blue" (OOTB)
>> 	 * packet if it is correctly formed, i.e., passed the
>> 	 * receiver's checksum check, but the receiver is not
>> 	 * able to identify the association to which this
>> 	 * packet belongs.
>> 	 */
>> 	
>> if (!asoc) {
>> 	if (sctp_rcv_ootb(skb)) {
>> 		__SCTP_INC_STATS(net, SCTP_MIB_OUTOFBLUES);
>> 		goto discard_release;
>> 	}
>> }
> The above code looks good. Have a look at
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/sctp/input.c?h=v5.9-rc8#n666
> 
> RFC 4960 requires also to drop some OOTB packets. This is what sctp_rcv_ootb() checks for.
>> 
>> This means out of the blue packets are always ignored and dropped.
>> 
>> the RFC however says:
>> 
>>  8) The receiver should respond to the sender of the OOTB packet with
>>     an ABORT.  When sending the ABORT, the receiver of the OOTB packet
>>     MUST fill in the Verification Tag field of the outbound packet
>>     with the value found in the Verification Tag field of the OOTB
>>     packet and set the T-bit in the Chunk Flags to indicate that no
>>     TCB was found.  After sending this ABORT, the receiver of the OOTB
>>     packet shall discard the OOTB packet and take no further action.
>> 
>> I think this is what I am seeing. The remote sends OOTB messages, we dont reply with abort which means the remote doesnt reset the connection.
> What are those OOTB messages? Which chunks do they contain?
> 
> Bes regards
> Michael
>> There must be a second issue that the socket structures are not in sync up.
>> 
>> 
>>> On 5 Oct 2020, at 19:16, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> On Mon, Oct 05, 2020 at 06:39:22PM +0200, Andreas Fink wrote:
>>> ...
>>>> What we now see in netstat --sctp is:
>>>> 
>>>> we have a LISTEN on port 2010
>>>> we have a  association from port 2010 to the remote in status CLOSED
>>>> 
>>>> in tcpdump we see packets coming in from the remote and heartbeat being acknowledged. However our application is not answering to these packets and the status of the application shows SCTP being down.
>>>> In other words, my application sees the association down. Netstat shows the association as being closed but the kernel seems to continue to entertain this association by continue to send heartbeat ACK and not sending ABORT.
>>> 
>>> That's weird. If it is in CLOSED, then the stack should be handling
>>> it as an OOTB packet and trigger an Abort.
>>> 
>>>> 
>>>> We now kill the application
>>>> 
>>>> What we now see in netstat --sctp is:
>>>> we no longer listen on port 2010
>>>> we have a closed association from port 2010 to the remote.
>>>> 
>>>> in tcpdump we however we STILL see packets coming in from the remote and heartbeat being acknowledged, even though no application is listening on this port and no userspace application is using that port.
>>>> We do not see any SHUTDOWN or INIT even if we restart the application.
>>>> 
>>>> Can anyone explain how this can be?
>>> 
>>> Please check the assoc status as well, via 'ss -a --sctp' and
>>> /proc/net/sctp/assocs . Maybe it got out of sync of the socket status.
>>> 
>>> Marcelo
>> 
>> 
>