SCTP nailed down connections trouble

SCTP nailed down connections trouble

[Andreas Fink]

Hello all

I have detected that the Linux SCTP implementation behaves differently than the FreeBSD and MacOS X SCTP implementations and I am trying to find the proper way to work around this issue.
What I have is nailed down peer to peer SCTP associations as it is mandatory in the Sigtran world. This means the source and destination ports are preconfigured on both sides and both sides can establish the connection at any time. So there is no client and no server. 

For this to work I normally do

	socket()
	fcntl() to set to non blocking
	setsockopt() to set various options such as linger time, nodelay and reuseaddr and enable sctp events
	bind() to define the local IP's and ports to use
	sctp_connectx() to connect to the remote ip and port

After this a SCTP INIT is sent to the remote.

	if the remote is a Cisco ITP; the connection comes up normally and all works.
	If the remote is another linux box with the same software it fails because the Linux kernel sends back a SCTP ABORT. and sctp_connectx fails instead of waiting for the other side to come up.
	So we end up in a race condition that both sides have to send the SCTP INIT at the same time to have this working. Whenever one is faster, the other side generates ABORT and it fails again.


Under FreeBSD and MacOS, the sctp_connectx returns without an error (as its a non blocking socket) and my subsequent code then calls poll to wait for events or data to be delivered with a follow up call to sctp_recvmsg to process that data and/or events
I would expect  a SCTP UP message  to appear after the other side has done the same after the SCTP handshake completes.

Under Linux however sctp_connectx does immediately return with connection refused and we are stuck.
I have not figured out how this can be achieved reliably. constantly calling sctp_connectx can not be the solution as it would create a busy loop and probably a packet storm.

I have a minimalized code example on https://github.com/andreasfink/sctp-test.

That example  you run with 
	./sctp-test <localip> <localport> <remoteip> <remoteport>

when run under MacOS with a non existing host I get

./sctp-test 10.0.67.209 7000 1.2.3.4 7000
socket() successful
setting socket to non blocking
setting socket to blocking=0
fcntl successful
setsockopt(IPPROTO_SCTP,SCTP_EVENTS) successful
setsockopt(SOL_SOCKET,SO_LINGER,5) successful
setsockopt(SOL_SOCKET,SO_REUSEADDR) successful
setsockopt(IPPROTO_SCTP,SCTP_REUSE_PORT) successful
setsockopt(SCTP_NODELAY) successful
bind() successful
sctp_connectx() successful
 poll returns 0
 poll returns 0
 poll returns 0
 poll returns 0
 poll returns 0
 poll returns 0
 poll returns 0

when under Linux

to a non existing hist
./sctp-test 10.99.3.27 7000 1.2.3.4 7000
socket() successful
setting socket to non blocking
setting socket to blocking=0
fcntl successful
setsockopt(IPPROTO_SCTP,SCTP_EVENTS) successful
setsockopt(SOL_SOCKET,SO_LINGER,5) successful
setsockopt(SOL_SOCKET,SO_REUSEADDR) successful
setsockopt(SCTP_NODELAY) successful
bind() successful

(meaning sctp_connectx doesn't return here and locks)


under Linux to a existing host where the software is not started yet.

./sctp-test x.x.x.x 7000 y.y.y.y 8000
socket() successful
setting socket to non blocking
setting socket to blocking=0
fcntl successful
setsockopt(IPPROTO_SCTP,SCTP_EVENTS) successful
setsockopt(SOL_SOCKET,SO_LINGER,5) successful
setsockopt(SOL_SOCKET,SO_REUSEADDR) successful
setsockopt(IPPROTO_SCTP,SCTP_REUSE_PORT) not supported
setsockopt(SCTP_NODELAY) successful
bind() successful
sctp_connectx failed (111 Connection refused)

Note: the error is immediately returned. not after timeout.


I test with the lksctp version delivered with Debian 9 - strech. If theres some changes since that version in this area, let me know.

A second area where I found issues is on how to configure a specific port to be used for multiple connections. For example my local port 2000 can be used for a connection to host 1 or to host 2 and the individual sessions which are created as above will however not allow me to use the same port in a second socket even though the bind is not follwed with a listen and the sctp_connectx goes to  a different port. Workaround to this is to use different ports for every connection (which is not really nice). Maybe there's another way?




A. FInk

Re: SCTP nailed down connections trouble

[Marcelo Ricardo Leitner]

Hi,

On Wed, May 16, 2018 at 02:02:46PM +0200, Andreas Fink wrote:
> Under FreeBSD and MacOS, the sctp_connectx returns without an error
> (as its a non blocking socket) and my subsequent code then calls
> poll to wait for events or data to be delivered with a follow up
> call to sctp_recvmsg to process that data and/or events
> I would expect  a SCTP UP message  to appear after the other side
> has done the same after the SCTP handshake completes.
>
> Under Linux however sctp_connectx does immediately return with
> connection refused and we are stuck.

Right. This (the returning) is because on Linux the connect operation
always blocks, even if the socket is configured to not do so. It's a
known issue, we're working on it. Xin Long has some patches already
and hopefully we can post them soon.

On the simultaenous open, when you connect to Cisco ITP or the others,
doesn't it send an Abort right after the INIT request reaches it? If
they don't, I find it odd.

> I have not figured out how this can be achieved reliably. constantly
> calling sctp_connectx can not be the solution as it would create a
> busy loop and probably a packet storm.

Maybe by using some iptables rule to drop Abort chunks, using a socket
filter (SO_ATTACH_FILTER) to do it (I never did it, btw, just sharing
ideas).

...
> A second area where I found issues is on how to configure a specific
> port to be used for multiple connections. For example my local port
> 2000 can be used for a connection to host 1 or to host 2 and the
> individual sessions which are created as above will however not
> allow me to use the same port in a second socket even though the
> bind is not follwed with a listen and the sctp_connectx goes to  a
> different port. Workaround to this is to use different ports for
> every connection (which is not really nice). Maybe there's another
> way?

Needs to check on this.

  Marcelo

Re: SCTP nailed down connections trouble

[Andreas Fink]

> On 16 May 2018, at 18:29, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> 
> Hi,
> 
> On Wed, May 16, 2018 at 02:02:46PM +0200, Andreas Fink wrote:
>> Under FreeBSD and MacOS, the sctp_connectx returns without an error
>> (as its a non blocking socket) and my subsequent code then calls
>> poll to wait for events or data to be delivered with a follow up
>> call to sctp_recvmsg to process that data and/or events
>> I would expect  a SCTP UP message  to appear after the other side
>> has done the same after the SCTP handshake completes.
>> 
>> Under Linux however sctp_connectx does immediately return with
>> connection refused and we are stuck.
> 
> Right. This (the returning) is because on Linux the connect operation
> always blocks, even if the socket is configured to not do so. It's a
> known issue, we're working on it. Xin Long has some patches already
> and hopefully we can post them soon.
> 
> On the simultaenous open, when you connect to Cisco ITP or the others,
> doesn't it send an Abort right after the INIT request reaches it? If
> they don't, I find it odd.

actually I played around with my test programm and figured I had the block/no block logic inverted by mistake.
Right now I have sctp_connectx returning error 115 (EINPROGRESS) in errno which is not 0 as on MacOS X or FreeBSD but at least it can be handled properly.
The second problem I spotted was that poll was returning data and a error at the same time. pollfds[0].revents = 9 is what I saw. Because my code bails out if it sees bit 8 set (POLLERR), it never got to the point to actually read and process the data. And because errno was actually set to 0, my code was thinking its time to immediately read again, creating a busy loop. poll returns instantly, not after timeout.
I'm just changing that. I however find it odd that poll can return an error flag ,errno being 0 and data. Somehow this doesn't make much sense to me. But it could be related to the fact that EINPROGRESS was set before maybe?


>> I have not figured out how this can be achieved reliably. constantly
>> calling sctp_connectx can not be the solution as it would create a
>> busy loop and probably a packet storm.
> 
> Maybe by using some iptables rule to drop Abort chunks, using a socket
> filter (SO_ATTACH_FILTER) to do it (I never did it, btw, just sharing
> ideas).
> 

Uh that would be a quite ugly hack :)
The ABORT's seem however to be no longer be the issue once the socket is in non blocking.

> ...
>> A second area where I found issues is on how to configure a specific
>> port to be used for multiple connections. For example my local port
>> 2000 can be used for a connection to host 1 or to host 2 and the
>> individual sessions which are created as above will however not
>> allow me to use the same port in a second socket even though the
>> bind is not follwed with a listen and the sctp_connectx goes to  a
>> different port. Workaround to this is to use different ports for
>> every connection (which is not really nice). Maybe there's another
>> way?
> 
> Needs to check on this.

Could that be related to the missing SCTP_REUSE_PORT socket option?

> 
>  Marcelo

Re: SCTP nailed down connections trouble

[Marcelo Ricardo Leitner]

On Wed, May 16, 2018 at 07:17:07PM +0200, Andreas Fink wrote:
>
>
> > On 16 May 2018, at 18:29, Marcelo Ricardo Leitner <marcelo.leitner@gmail.com> wrote:
> >
> > Hi,
> >
> > On Wed, May 16, 2018 at 02:02:46PM +0200, Andreas Fink wrote:
> >> Under FreeBSD and MacOS, the sctp_connectx returns without an error
> >> (as its a non blocking socket) and my subsequent code then calls
> >> poll to wait for events or data to be delivered with a follow up
> >> call to sctp_recvmsg to process that data and/or events
> >> I would expect  a SCTP UP message  to appear after the other side
> >> has done the same after the SCTP handshake completes.
> >>
> >> Under Linux however sctp_connectx does immediately return with
> >> connection refused and we are stuck.
> >
> > Right. This (the returning) is because on Linux the connect operation
> > always blocks, even if the socket is configured to not do so. It's a
> > known issue, we're working on it. Xin Long has some patches already
> > and hopefully we can post them soon.
> >
> > On the simultaenous open, when you connect to Cisco ITP or the others,
> > doesn't it send an Abort right after the INIT request reaches it? If
> > they don't, I find it odd.
>
> actually I played around with my test programm and figured I had the
> block/no block logic inverted by mistake.  Right now I have
> sctp_connectx returning error 115 (EINPROGRESS) in errno which is
> not 0 as on MacOS X or FreeBSD but at least it can be handled
> properly.

Ah right. The issue I mentioned happens only with connect calls from
within the kernel itself. My bad.

Ok. EINPROGRESS is expected.

> The second problem I spotted was that poll was returning data and a
> error at the same time. pollfds[0].revents = 9 is what I saw.
> Because my code bails out if it sees bit 8 set (POLLERR), it never
> got to the point to actually read and process the data. And because
> errno was actually set to 0, my code was thinking its time to
> immediately read again, creating a busy loop. poll returns
> instantly, not after timeout.
> I'm just changing that. I however find it odd that poll can return
> an error flag ,errno being 0 and data. Somehow this doesn't make
> much sense to me. But it could be related to the fact that
> EINPROGRESS was set before maybe?

EINPROGRESS is described in connect() man page. If the socket faced
some issue after returning it, you need to fetch the error code using
getsockopt(fd, SOL_SOCKET, SO_ERROR, ...) instead and not errno.

errno itself is more to the syscall you just called, poll in the case,
and poll worked.

>
>
> >> I have not figured out how this can be achieved reliably. constantly
> >> calling sctp_connectx can not be the solution as it would create a
> >> busy loop and probably a packet storm.
> >
> > Maybe by using some iptables rule to drop Abort chunks, using a socket
> > filter (SO_ATTACH_FILTER) to do it (I never did it, btw, just sharing
> > ideas).
> >
>
> Uh that would be a quite ugly hack :)

Agreed :)

> The ABORT's seem however to be no longer be the issue once the socket is in non blocking.

Interesting. So then you just ignore the error the socket reports
(SO_ERROR above)?


  Marcelo

Re: SCTP nailed down connections trouble

[Xin Long]

On Thu, May 17, 2018 at 1:17 AM, Andreas Fink <afink@list.fink.org> wrote:
>
>> ...
>>> A second area where I found issues is on how to configure a specific
>>> port to be used for multiple connections. For example my local port
>>> 2000 can be used for a connection to host 1 or to host 2 and the
>>> individual sessions which are created as above will however not
>>> allow me to use the same port in a second socket even though the
>>> bind is not follwed with a listen and the sctp_connectx goes to  a
>>> different port. Workaround to this is to use different ports for
>>> every connection (which is not really nice). Maybe there's another
>>> way?
>>
>> Needs to check on this.
>
> Could that be related to the missing SCTP_REUSE_PORT socket option?
>
That's right, but SO_REUSEADDR works as the equivalent of
SCTP_REUSE_PORT for SCTP in linux, and for now you can work around by:

        if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &val, sizeof(val)) = -1) {
                perror("reuse port failed");
                exit(-1);
        }

I'll add SCTP_REUSE_PORT sockopt in linux SCTP soon. Thanks.

RE: SCTP nailed down connections trouble

[David Laight]

From: Marcelo Ricardo Leitner
> Sent: 16 May 2018 17:29
...
> On the simultaenous open, when you connect to Cisco ITP or the others,
> doesn't it send an Abort right after the INIT request reaches it? If
> they don't, I find it odd.

I thought that simultaenous connect requests (colliding INIT)
were expected to result in an actual connection.

So two hosts can establish a connection by sending INIT (etc)
with both the source and destination port numbers set to the
same (well known) value.

	David

Re: SCTP nailed down connections trouble

[Andreas Fink]

> On 17 May 2018, at 18:58, David Laight <David.Laight@ACULAB.COM> wrote:
> 
> From: Marcelo Ricardo Leitner
>> Sent: 16 May 2018 17:29
> ...
>> On the simultaenous open, when you connect to Cisco ITP or the others,
>> doesn't it send an Abort right after the INIT request reaches it? If
>> they don't, I find it odd.
> 

No they dont send abort because they are configured to establish a connection. This means they start handshaking with INIT_ACK and COOKIE etc. However they also send INIT to you all the time (couple of times per minute).

> I thought that simultaenous connect requests (colliding INIT)
> were expected to result in an actual connection.

this is true. I have seen this when I run my test program at the very same moment.

> 
> So two hosts can establish a connection by sending INIT (etc)
> with both the source and destination port numbers set to the
> same (well known) value.
> 
> 	David

The problem is not really on the network level but on the understanding of how the API was meant to be.
The mystery has been solved now. The Userspace API for SCTP is not as clear as it could be (at least to me).
From a network perspective (looking at wireshark traces etc), you simply see a connection being established between two nodes. Either side can be first to send INIT's and it doesnt matter. At the end both sides have a working pipe they can stuff data into. So it was expected to see this metapher in the userspace API as well but instead, we have client/server again there even though the underlying network protocol is true peer to peer.

So you do 

	sctp_connectx,()

to create an outbound connection. It will succeed if the other side is ready to listen. If not, you receive ABORT and youre dead. And if the other side starts up  half a second earlier, it doesn't matter, you are not listening hence the kernel doesnt care if you are having established that exact session. It only worked if the INIT's collide.

To listen for an inbound connection you simply call

	listen()

(followed by potential accept() etc)

When you do a peer to peer connection where it doesn't matter which side establishes first (a nailed down connection where source and destination port and IP's are predefined), one could expect you could do this with async listen() and sctp_connectx(). However this fails. sctp_connectx() complains that the socket is in use already. By calling listen() you say this is inbound only. On the other hand by calling sctp_connectx() you tell the system it is outbound *only*. This was my wrong assumption that this is all it takes. But because this responds incoming INITS from the other side with ABORT's instead of accepting, you basically kill the other side and vice versa.
Incoming INIT's will cause an abort instead of being accepted and handed to this peer to peer connection as it would make sense in my eyes.

This is what was puzzling me. Before I had hardware I can always connect to (sctp_connectx *always* succeded) or I was in passive mode (which means I only did listen and waited for someone to connect to me). Trying to do both at the same time as it should be, opened the pandoras box.

The solution is to use two sockets. One to listen, and another one for sctp_connectx. This sounds like pseudo peer to peer in a old TCP world again.
This way the listen eliminates the ABORT going back and the sctp_connectx makes sure there's INIT's going out. The liste one you never do anything with it. You only use it to tell the kernel that it should not send ABORTs for that part and voila things start magically working again. The SCTP API brings so many new options that they are confusing on how to be used.

The subtle differences between the Linux implementation and the FreeBSD or MacOS X implementation then added to further confusion. But finally my connections to other vendor equipment works again. And when I have finished restructuring my own stuff  a bit, true peer to peer should finally start working again.


Thanks for your hints. A few tipps where very useful


Andreas