RE: connect() does not return ETIMEDOUT

RE: connect() does not return ETIMEDOUT

[Alessandro Motter Ren]

	I had to set connect to do non blocking connections in order to
avoid this problem, the 4 minutes timeout and use getsockopt for low level
error check on the socket.
	[]s.

-----Original Message-----
From: Manfred Bartz [mailto:mbartz@optushome.com.au]
Sent: Wednesday, August 15, 2001 11:34 AM
To: linux-kernel@vger.kernel.org
Subject: Re: connect() does not return ETIMEDOUT


Andi Kleen <ak@suse.de> writes:

> It works correct here (2.4.7-something)

Ok, after running tcpdump, it now looks the problem is not with
connect() but elsewhere.  Linux 2.4.8, same results on 2.4.2

The programs which demonstrate the problem, pcap file and
other files are at <http://logi.cc/linux/tcp2a/>.

My experimental server will accept() two connections and the 
backlog is set to 1 (one) in the call to listen().

My experimental client will open up to 100 connections to 
the server.  The first 6 connections succeed immediately.
Subsequent connections succeed at a rate of 2 connections 
every 9 seconds.

Under Solaris, the client succeeds with 4 connections,
the 5th connect() gives ETIMEDOUT after about 4 minutes.

server side:
    /* tcp socket bound to port 7000 */
    listen(listen_fd, 1);  /* backlog set to 1 */
    accept() called twice only.

commented and compacted tcpdump:

The server has now accept()ed 2 connections and does no 
further calls to accept().

--------------------------------------------------
Connection 3, backlog, ok
10:22:01.258195 192.168.1.1.51834 > 192.168.1.11.7000: S 
10:22:01.258283 192.168.1.11.7000 > 192.168.1.1.51834: S ack
10:22:01.258970 192.168.1.1.51834 > 192.168.1.11.7000: . ack

10:22:01.024577 Connection[3] established.  connect()==0

--------------------------------------------------
Connection 4, ?
10:22:01.265549 192.168.1.1.51835 > 192.168.1.11.7000: S 
10:22:01.265634 192.168.1.11.7000 > 192.168.1.1.51835: S ack
10:22:01.266324 192.168.1.1.51835 > 192.168.1.11.7000: . ack

10:22:01.031926 Connection[4] established.  connect()==0

The backlog was set to one, but it completes and queues 
two connections.  Is that because the kernel maintains two 
queues (one each for incomplete and complete connections)?

--------------------------------------------------
Connection 5, ?  should fail
10:22:01.272881 192.168.1.1.51836 > 192.168.1.11.7000: S 
10:22:01.272958 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:01.273650 192.168.1.1.51836 > 192.168.1.11.7000: . ack

10:22:01.039251 Connection[5] established.  connect()==0

10:22:04.866558 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:10.866629 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:22.866793 192.168.1.11.7000 > 192.168.1.1.51836: S ack

--------------------------------------------------
Connection 6, ?  should fail
10:22:01.280139 192.168.1.1.51837 > 192.168.1.11.7000: S 
10:22:01.280223 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:01.280917 192.168.1.1.51837 > 192.168.1.11.7000: . ack

10:22:01.046514 Connection[6] established.  connect()==0

10:22:05.666558 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:11.666637 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:23.666789 192.168.1.11.7000 > 192.168.1.1.51837: S ack

etc. etc...

connect() keeps returning 0 (success) because the client
side sees the full 3-way handshake.

Questions:

Once the backlog is exhausted, shouldn't the server side 
TCP stop sending SYNs and either be quiet or perhaps send 
an appropriate ICMP response?

Why does the server side keep sending SYNs after the connection
handshake was apparently completed?

-- 
Manfred

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: connect() does not return ETIMEDOUT

[Alexey Kuznetsov]

Hello!

> 	I had to set connect to do non blocking connections in order to
> avoid this problem,

Sorry, what "this" problem? :-)

You have made something surely not related to the subject.

* non-blocking connect succeeds instantly, of course, like blocking one. 
* error never appears in this case, because connection is succesful.

It was original observation.

Alexey

Re: connect() does not return ETIMEDOUT

[Andi Kleen]

Manfred Bartz <mbartz@optushome.com.au> writes:
> 
> But in reality and going by the tcpdump, an unlimited number of
> connections is accepted because the server side completes the 3-way
> handshake regardless.  The connections are then lost later (with a
> different error message).  This does not look right to me.

The default is upto 128 queued syns before stopping acks, or an unlimited 
number when syncookies are turned on. You can change both with the 
tcp_max_syn_backlog or tcp_syncookies sysctls. 

-Andi

Re: connect() does not return ETIMEDOUT

[Manfred Bartz]

Sridhar Samudrala <samudrala@us.ibm.com> writes:

> linux has 2 queues associated with a listening socket...
> The second one is called accept queue which hold the complete
> connections ... The length of this queue ... is actually backlog+1.
> 
> When the accept queue is full, new incoming SYNs are accepted in a burst of 
> 2 at a time. These are put in the SYN table expecting that the accept queue
> will open up by the time we receive the ACK. If the accept queue doesn't get
> open up, the ACK is simply dropped and SYN-ACK is sent again after a certain
> timeout period. 

Thanks for the explanation, I appreciate it.

> In your example, 6 connections succeed immediately.
> But only 4 enter ESTABLISHED state.
> 2 are accepted by the server. 2 remain in the accept queue (backlog+1).

Ok, that part makes sense.

> 2 of them are added to the SYN table. 

So, the connections in the SYN table should be half open?
No server SYN should have been returned?
I just re-discovered ``TCP/IP Illustrated'' Vol.1 by W.R.Stevens and
it confirms this in section 18.11 under ``Incoming Connection Request
Queue''.  The book says:

    5. If there is not room on the queue for the new connection, TCP
    just ignores the received SYN.  Nothing is sent back (i.e. no RST
    segment).  ...

But in reality and going by the tcpdump, an unlimited number of
connections is accepted because the server side completes the 3-way
handshake regardless.  The connections are then lost later (with a
different error message).  This does not look right to me.

Comments?

-- 
Manfred

Re: connect() does not return ETIMEDOUT

[Sridhar Samudrala]

linux has 2 queues associated with a listening socket to maintain incoming 
connections before they are accepted.  The first one is syntable which holds 
the partial connections (SYN received and SYN-ACK sent). This is a hash table 
and the maximum length is limited by /proc/sys/net/ipv4/tcp_syn_max_backlog.
The second one is called accept queue which hold the complete connections (3 
way handshake is complete). The length of this queue is limited by the backlog value specfied with listen(). It is actually backlog+1.

When the accept queue is full, new incoming SYNs are accepted in a burst of 
2 at a time. These are put in the SYN table expecting that the accept queue
will open up by the time we receive the ACK. If the accept queue doesn't get
open up, the ACK is simply dropped and SYN-ACK is sent again after a certain
timeout period. 

In your example, 6 connections succeed immediately.
But only 4 enter ESTABLISHED state.
2 are accepted by the server. 2 remain in the accept queue (backlog+1).
2 of them are added to the SYN table. 

Thanks
Sridhar


> Questions:

> Once the backlog is exhausted, shouldn't the server side 
> TCP stop sending SYNs and either be quiet or perhaps send 
> an appropriate ICMP response?

> Why does the server side keep sending SYNs after the connection
> handshake was apparently completed?

> -- 
> Manfred

Re: connect() does not return ETIMEDOUT

[Manfred Bartz]

Andi Kleen <ak@suse.de> writes:

> It works correct here (2.4.7-something)

Ok, after running tcpdump, it now looks the problem is not with
connect() but elsewhere.  Linux 2.4.8, same results on 2.4.2

The programs which demonstrate the problem, pcap file and
other files are at <http://logi.cc/linux/tcp2a/>.

My experimental server will accept() two connections and the 
backlog is set to 1 (one) in the call to listen().

My experimental client will open up to 100 connections to 
the server.  The first 6 connections succeed immediately.
Subsequent connections succeed at a rate of 2 connections 
every 9 seconds.

Under Solaris, the client succeeds with 4 connections,
the 5th connect() gives ETIMEDOUT after about 4 minutes.

server side:
    /* tcp socket bound to port 7000 */
    listen(listen_fd, 1);  /* backlog set to 1 */
    accept() called twice only.

commented and compacted tcpdump:

The server has now accept()ed 2 connections and does no 
further calls to accept().

--------------------------------------------------
Connection 3, backlog, ok
10:22:01.258195 192.168.1.1.51834 > 192.168.1.11.7000: S 
10:22:01.258283 192.168.1.11.7000 > 192.168.1.1.51834: S ack
10:22:01.258970 192.168.1.1.51834 > 192.168.1.11.7000: . ack

10:22:01.024577 Connection[3] established.  connect()==0

--------------------------------------------------
Connection 4, ?
10:22:01.265549 192.168.1.1.51835 > 192.168.1.11.7000: S 
10:22:01.265634 192.168.1.11.7000 > 192.168.1.1.51835: S ack
10:22:01.266324 192.168.1.1.51835 > 192.168.1.11.7000: . ack

10:22:01.031926 Connection[4] established.  connect()==0

The backlog was set to one, but it completes and queues 
two connections.  Is that because the kernel maintains two 
queues (one each for incomplete and complete connections)?

--------------------------------------------------
Connection 5, ?  should fail
10:22:01.272881 192.168.1.1.51836 > 192.168.1.11.7000: S 
10:22:01.272958 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:01.273650 192.168.1.1.51836 > 192.168.1.11.7000: . ack

10:22:01.039251 Connection[5] established.  connect()==0

10:22:04.866558 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:10.866629 192.168.1.11.7000 > 192.168.1.1.51836: S ack
10:22:22.866793 192.168.1.11.7000 > 192.168.1.1.51836: S ack

--------------------------------------------------
Connection 6, ?  should fail
10:22:01.280139 192.168.1.1.51837 > 192.168.1.11.7000: S 
10:22:01.280223 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:01.280917 192.168.1.1.51837 > 192.168.1.11.7000: . ack

10:22:01.046514 Connection[6] established.  connect()==0

10:22:05.666558 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:11.666637 192.168.1.11.7000 > 192.168.1.1.51837: S ack
10:22:23.666789 192.168.1.11.7000 > 192.168.1.1.51837: S ack

etc. etc...

connect() keeps returning 0 (success) because the client
side sees the full 3-way handshake.

Questions:

Once the backlog is exhausted, shouldn't the server side 
TCP stop sending SYNs and either be quiet or perhaps send 
an appropriate ICMP response?

Why does the server side keep sending SYNs after the connection
handshake was apparently completed?

-- 
Manfred

Re: connect() does not return ETIMEDOUT

[Andi Kleen]

Manfred Bartz <mbartz@optushome.com.au> writes:

> When a TCP client opens multiple connections beyond the server's
> backlog capacity, the call to connect() returns a value of zero 
> (success) after 9 seconds. Toggling SYN-cookies makes no difference.

It works correct here (2.4.7-something)

# ipchains -A input -j DENY -p tcp --dport 10000
# strace -e connect telnet localhost 10000
...
connect(3, {sin_family=AF_INET, sin_port=htons(10000), sin_addr=inet_addr("127.0.0.1")}}, 16) = -1 ETIMEDOUT (Connection timed out)

If you think something is wrong you should probably send a strace that
shows it.

-Andi

connect() does not return ETIMEDOUT

[Manfred Bartz]

When a TCP client opens multiple connections beyond the server's
backlog capacity, the call to connect() returns a value of zero 
(success) after 9 seconds. Toggling SYN-cookies makes no difference.

Is this intended behaviour?

Under Solaris I get ETIMEDOUT (Connection timed out) after a little 
less than 4 minutes.

Presumably I could set the socket to non-blocking and select() for
writability...  (I have not tested that yet.)

But if I want to keep it simple, what is the recommended solution?

-- 
Manfred Bartz